Our project has two parts:

• Some lines of code
  - A class library for C++ implementing streaming XML.
• Survey
  - A look at existing technologies

- whenever possible!

Not always possible, however. Here are some guidelines traditionally used:

You should probably use streams if:

• You are parsing large files.
• You want to be able to abort parsing.
• You only need to collect small amounts of data.
• You cannot afford DOM

You should probably use DOM if:

• You need random access to the whole file.
• You need to perform XSLT transformations.
• You need to do complex XML queries.
• You want to modify and save XML

You should probably use lazy DOM if:

• You need random access to small part of the file.
• You are dealing with somewhat large files.
• You would like to use SAX, but you have to use DOM.

Our streaming library implements a pull parser; the user initiates the reading and extraction of stream elements. This strategy is contrary to most current parsers, which implement push parsers, that have total control of what code is executed. Once started, the push parser controls the control flow of the program.

A push parser is easily implemented using a pull parser.

Streams are used to extract elements. In case of XML streams the elements consists of several different kinds of tokens:

• begining of document
• processing instructions
• DOCTYPE declaration
• comments
• start tag
• character data
• entities
• CDATA sections
• end tags
• end of document

Examples of other XML input streams:

• Socket/network based
• Source of binary objects

Examples of other XML output streams:

• Socket/network based
• Binary object generator
• SAX event generator
• DOM tree builder
• GSM WAP configuration

Examples of other XML filters:

• Validation
• Multithreaded buffer
• Sorting
• XML Encryption

Using the classes

> elmt; !elmt.is_enddoc(); xin >> elmt )
            xout << elmt;
        xout << elmt; // Output the enddoc element
    }
    catch( exception e )
    {
        std::cerr << "Exception caught in main(): " << e << std::endl;
        return 1;
    }

    return 0;
}
					]]>

XPath algorithm

We can recognize some XPath expressions

While traversing the input stream the state is updated and matching nodes are send to the output stream.

The state consists of

• A pointer to a step in the XPath
• A nesting level counter
• A stack of counters to keep track of positions at the different levels

The state is updated every time we encounter a new node.

If the XPath pointer is at the end of the expression the current node matches.

Here is some code using the XPath filter (and the | operator):

 xp("/slidecollection/section[2]/slide[@title=\"XMLS\"]");

    xin | xp | xout;

    return 0;
}
					]]>

We've made preliminary benchmarks of XMLS compared to Apache's Xalan/Xerces and DOM4J (only available in Java) DOM-based implementation of XPath filters. We ran the following queries:

Q1: /dblp/inproceedings/title
Q4: /datasets/dataset[@subject="astronomy"]/reference/source/other/name
Q5: /root/Entry/Ref/Cite
Q6: /ProteinDatabase/ProteinEntry/protein/name

With these results:

Most streaming implementations are based on the SAX standard API.

• Xerces from Apache implements SAX and SAX2, and is a mature library
• Xalan, also from Apache, builds on top of Xerces (or some other SAX parser) to provide streaming XPath processing and XSLT transformation
• XAOS specializes in streaming XPath processing, and claims to perform better than Xalan - But no XSLT, and only part of XPath
• MSXML4 from Microsoft implements SAX and SAX2 and is very mature
• Joost - Implementation of STX (Streaming Transformations for XML) which is a high speed, low memory alternative to XSLT
• XMLTK - UNIX command-line XML processing tools

XSQ: Streaming XPath Queries

Allows child and descendant-or-self axes, and a wide range of predicates.

• /Entry/Ref/Cite
• //article[@key]//title/text()
• /pub[year]/book[@id=3]/author

They use Hierarchically structured Push Down Transducers.

• Parts of the stream is buffered
• An automaton decides when to insert, query, remove from or clear the buffer
• a HDPT consists of smaller PDTs each with their own buffer

An Algorithm for Streaming XPath Processing with Forward and Backward Axes

They allow the backward axes parent and ancestor together with the forward axes child and descendant. Predicates can only query existence of descendant nodes.

• /Person/Adress/City
• //pub[year]/book/title
• //listitem/ancestor::category//name
• /X[parent::Y/Z]/W//Z

They use X-dags and X-trees to represent XPaths.

• X-tree nodes represent node tests and edges represent axis
• X-dags are used to convert backwards constraints into forward constraints

General Parsers

• alphaWorks XML Parser for Java
• MSXML 4.0 SDK
• SourceForge.net Project Info - NQXML
• SourceForge.net Project Info - Streaming XML -- DOM event processing
• Xerces2 Java Parser Readme
• XML Streaming Library (serialize objects)
• XML Toolkit
• xmlhack Streaming API for XML (StAX)

Info

• Introduction to the DOM (MSDN, with diagrams of typical DOM and SAX parsers)
• Official SAX Website
• What is SAX (MSDN)
• When Should I Use SAX (MSDN)
• xml.apache.org

STX

• SourceForge.net Project Info - Joost STX processor
• SourceForge.net Project Info - Streaming Transformations for XML (STX)

IBM Research - XAOS (XML Analysis, Optimization, and Stuff)