DIHtmlParser

• Full Unicode support (UnicodeString or WideString, depending on Delphi version).
• Reads and writes over 70 character sets natively (independent of the OS). More than 150 are supported with the help of DIConverters.
• Operates on TStreams, memory buffers or strings.
• Returns a single piece of HTML to the application at a time.
• Extends easily via the TDIHtmlParserPlugin interface.

DIHtmlParser recognizes 10 pieces of HTML plus 4 pieces of Non-HTML.

The HTML pieces are:

• CData Sections: CData Sections, found in XML, are used to escape blocks of text containing characters which would otherwise be recognized as markup. A CData section begins with <![CDATA[ and ends with ]]>.

• Comments: The Comments' contents are returned readily stripped of the comment markers. A comment starts with <!– and ends with –>.

• Document Type Definitions: A Document Type Definition defines the syntax of markup constructs. It begins with <!DOCTYPE and ends with >.

• HTML Processing Instructions: HTML Processing Instructions are a mechanism to capture platform-specific idioms. They start with <? and end with >.

• HTML-Tags: HTML-Tags are readily parsed into Name, Attributes and Values. DIHtmlParser recognizes Start Tags, End Tags and Empty Element Tags. Example: <TagName Attribute=“Value” />.

• Scripts: DIHtmlParser returns the contents between the <SCRIPT> and </SCRIPT> tags as simple text. The surrounding HTML tags are reported separately.

• Styles: DIHtmlParser returns the contents between the <STYLE> and </STYLE> tags as simple text. The surrounding HTML tags are reported separately.

• Text: Text is everything which is not markup. If the NormalizeWhiteSpace option is enabled, DIHtmlParser reduces multiple white space to a single character. Preformatted text wrapped by <PRE> and </PRE> is never normalized.

• Titles: DIHtmlParser returns the contents between the <TITLE> and </TITLE> tags as simple text. Titles are not normal text because they are parsed differently.

• XML Processing Instructions: XML Processing Instructions are similar to the HTML Processing Instructions with a slightly different syntax: They begin with <?XML and end with ?>.

The Non-HTML pieces are:

• Active Server Pages (ASP): Active Server Page markup is often used to enclose scripting macros. It begins with <% and runs up to %>.

• Custom-Tags: Custom Tags are similar to HTML-Tags and to what Delphi's Help calls Transparent Tags. For DIHtmlParser, a Custom-Tags' name must begin with a user-define start character just as # like in <#Name Attribute=“Value” />.

• PHP: PHP is a powerful and popular scripting language. Its markup begins with <?PHP and ends with ?>.

• Server Side Includes (SSI): SSI, an extension of the Apache Web Server, starts with <!–# and continues up to –>. It allows to insert include files and other data into HTML documents on the fly.

DIHtmlParser is extremely fast, especially when parsing huge files. Thanks to the internal buffer mechanism, it does not need to load the entire file into memory at once but can read one small chunk after the other at a single time only. DIHtmlParser parses up to 50 000 tags per second even with an outdated 166 MHz processor. On modern machines the score goes up to more than 15 MB of HTML data per second.

DIHtmlParser only parses what it needs to parse. Thanks to its filtering mechanism, the parser can skip all pieces of HTML which the application did not request. Even though the parser must eventually touch each single character of a HTML document, it might only need to store a fraction of that data for further processing. We call this “Smart Parsing”, as not storing unnecessary data is one of the greatest time savers.

Another trick of “Smart Parsing” is to convert relevant tag and attribute strings into ordinal number IDs. As a result, the parser never needs to compare lengthy strings consisting of many characters but can easily get away with one simple number comparison instead. This improves performance and reduces processor load. Your own coding benefits from this technique, too, as tag and attribute IDs are part of the DIHtmlParser interface.

Tag filtering forwards the general filtering to individual tags. It enables the programmer to instruct the parser to hold back all tags which are not relevant to the application. Why bother with <TABLE> tags if you are only interested in the images of a HTML document? Instead of having the application check each tag for an <IMG> tag, simply instruct the parser only to report <IMG> tags in the first place. This allows DIHtmlParser to optimize its parsing and your application no longer has to worry about unwanted tags.

DIHtmlParser Plugins are the next step to customized HTML parsing. A single instance ot TDIHtmlParser can run any number of parsing processes in parallel to the its main parsing process. Each plugin features its own flexible filtering mechanism just as the main parser. The plugin architecture keeps overhead to a minimum, as each of them informs the parser about its requirements ahead of the parsing. So even with many plugins in effect, DIHtmlParser will never parse more than what your application actually asks for.

More information on DIHtmlParser Plugins is available on this page.