Querying JSON Streams

Abstract: A data stream management system (DSMS) is similar to a database management system (DBMS) but can search data directly in on-line streams. Using its mediator-wrapper approach, the extensible database system, Amos II, allows different kinds of distributed data resource to be queried. It has been extended with a stream datatype to query possibly infinite streams, which provides DSMS functionality. Nowadays, more and more web applications start to offer their services in JSON format which is a text-based, human comprehendible format for representing simple data structures and associative arrays. For example, one of the most popular websites worldwide, Twitter, has developed a stream interface that one can register to receive large amounts of Twitter messages in JSON format. Another popular website Facebook and some weather services also provide stream interfaces through JSON. The objective of the project is to develop a general JSON stream reader as an Amos II wrapper, allowing such streams to be easily queried and managed by a DSMS. To implement such a system, three primitive foreign functions are defined to allow queries to JSON streams using the query language AmosQL of Amos II. The usefulness of the developed wrapper is demonstrated on the popular Twitter social network data stream.