UNIVERSITY of GLASGOW

Computing at Glasgow University
 
Paper ID: 7750
DCS Tech Report Number: TR-2004-179

Using database technologies to transform low-level stream data for client analysis
Mancy,R. Thomas,R. Draper,S. Gray,P. Kennedy,G.

Publication Type: Tech Report (internal)
Appeared in:
Page Numbers : 7
Publisher: N/A
Year: 2004
Abstract:

GRUMPS, a Generic Remote Usage Measurement Production System, collects streams of computer-generated events representing low-level actions and stores these in a database. We have successfully created repositories of 5 million events in about 2GB of semi-structured data. These generic data represent a valuable source for multiple investigations but require significant transformation before domain-level analysis can begin. This is a difficult and timeconsuming stage with this approach, and requires the correct choice of tool. Because of the large volumes of data, we chose to carry out transformations within the database. To benefit from previous experience effectively, techniques and queries need to be optimised for both re-usability and speed of execution with the chosen tool. We have found it useful to group investigations according to the highlevel data required, using an intermediate representation from which data for multiple investigations can be generated, and developing techniques using re-usable stored procedures.


Bibtex entry Endnote XML