1

Topic: Can it be used to safely extract the intro of blog articles?

First I like to thank you for making this software available to the public free of charge.

As for my question, it may have come up before, but if so,  couldn't find it in the forum. What I'm looking for, like so many others along with me, is a safe way to extract the intro (or teaser if you will) from a html blog article. There are two obvious problems:

  • Determine the proper position to split the text, preferably on a word boundry.

  • Make sure the extracted intro is still valid html, preferably containing any embedded elements like images etc.

Can htmLawed be of help for this purpose and if so, can you give a few pointers how to put it to use?

2

Re: Can it be used to safely extract the intro of blog articles?

Of the two aspects you mention, htmLawed can address only the second one.

The first, extracting the teaser/intro. part of the article, would depend on what the input is. If it is an entire web-page (equivalently, the URL of the page), how would one separate out, say, the heading/title, any banner, menu items, etc.? If the web-pages one uses are similarly structured, this may not be that difficult and simple regular expression/sub-string-identifying PHP code can be used to pick out the article text.

With the extract available, htmLawed can do almost any specific filtering. At a simple level, one would use the $config parameters 'elements' and 'deny_attribute'. Other parameters, and $spec, can be used for more advanced work. Refer to the documentation to get an idea of what all is possible.

Irrespective of the length of the extract or where the HTML/DOM structure is cut off because of the extraction, htmLawed will ensure the filtered output is proper HTML, with balanced and properly nested tags. So, what is left is to specify elements/attributes to be allowed/denied.

E.g., to ensure there is no conflict for unique IDs or with style on the web-page displaying the filtered extract, one may want to remove attributes 'style', 'class' and 'id' from the extract's HTML markup. For security one would like to remove 'script', and 'object' and 'embed' elements. Full tables may be allowed but not interrupted ones. And, finally, one may want to convert 'h1' to 'h3' header tags to simple 'div' and 'strong'. The filtering code may thus look something like this:

$config['deny_attribute'] = 'class, id, style';
$config['elements'] = '* -embed -object -script';
if(substr_count($in, '<table') != substr_count($in, '</table>')){
  $config['elements'] .= '-table';
}
$out = htmLawed($in, $config);
$out = str_replace(
  array('<h1', '<h2', '<h3', '</h1>', '</h2>', '</h3>' ), 
  array('<div><strong>', '<div><strong>', '<div><strong>', '</div></strong>'), 
  $out;
);

3

Re: Can it be used to safely extract the intro of blog articles?

Thanks for your swift and extensive reply. Especially for the suggestions you make for cleaning up in the last part.

The kind of input I'm thinking of, are not complete HTML pages but rather 'simple' blog like articles. Usually title and various other elements are entered separately, so I will only be parsing the 'body' of the article. That body may or may not contain tables, images and other HTML elements though.

What I think I need, is a way to first extract the plain text, use that to determine the cut off position for the intro and subsequently translate that position back into the corresponding position in the HTML document. I can then extract the intro end have it 'repaired' by htmLawed. I could use PHP's strip_tags() to simply strip all tags and return me my plain text, but I would loose my reference to the original HTML document completely. I haven't figured out a way to solve that yet. Any suggestions?

4

Re: Can it be used to safely extract the intro of blog articles?

There may be a number of ways to achieve this, some requiring more processing. One simplistic, hastily-thought-of way can be:

// Since intro. will be <500 'displayed' characters,
// we reduce input size to fasten the process.

$in = substr($in, 0, 2000);

// Optional. We ensure that input is correct HTML.
// E.g., '>' when used as the 'greater than' symbol should be '&gt;'.
// We also compact the HTML so useless but confounding
// white-spaces are disregarded.

$in = htmLawed($in, array('tidy'=>-1));

// As any markup is within '<' and '>',
// we use those characters to break-up the text.
// $pieces is an array of 'real' text, with each element being an array with
// offset 0 being the text and 1 being the position of the first character.
// Thus, 'I said <em>hello</em>!' becomes:
// array(array('I said ', 0), array('hello', 11), array('!', 21));

$pieces = preg_split('`<[^>]+>`', $in, -1, PREG_SPLIT_OFFSET_CAPTURE);

// Our custom code in my_extractor_function accepts the $pieces array
// It assembles the plain text, extracts the excerpt from it,
// and identifies the corresponding cut-off position in the original input.

// The 'my_extractor_function' has to be coded :)

$cutoff = my_extractor_function($pieces);
$in = substr($in, 0, $cutoff);

// We re-run htmLawed with the filtering criteria for excerpt display.

$config = array(...);
$out = htmLawed($in, $config);

5

Re: Can it be used to safely extract the intro of blog articles?

I'm not such an experienced PHP programmer as you are, but I think this will do the trick indeed. That is to say, with proper implementation of my_extractor_function() of course :D But that shouldn't be too hard.

Thank you very much and I'll keep you posted about the result.

6

Re: Can it be used to safely extract the intro of blog articles?

The cut HTML string (http://code.google.com/p/cut-html-string/) PHP class can be used to extract a sub-string of text of certain length that doesn't count the characters of the HTML markup.