Matthew Chalmers, University of Glasgow
www.dcs.gla.ac.uk/~matthew, matthew@dcs.gla.ac.uk
In this way, path systems aim to do more than make personal or individual recommendations. They also aim to be contextually specific in their recommendations. An issue here is what one records in the path. This may be addressed by focusing on fine-grained actions such as activity within and between web pages (Villa 2001). Another approach is to focus on the media one tracks and records. As we describe below, we have enriched paths by including file activity in the xemacs editor so that, for example, recommendations are made on the basis of a combination of web pages and images browsed along with the code files recently read or edited. Recommendations can therefore be ‘documents’ of various types such as HTML pages, JPEG images, Java source code, Emacs Lisp, and so forth. Recommender systems’ ability to handle heterogeneous data lets us bridge between tools and types usually treated in isolation. We can record and combine a range of tools and media, and use the temporal patterns of user activity that interweave them to make recommendations that span them.
We are now broadening our range of tools and media significantly further. As part of Equator, a large interdisciplinary research project funded by the UK government, we are developing path systems that record user activity in terms of physical, informational and virtual context. Equator’s aim is the bridging physical and digital media in order to improve and extend interactive information systems. We are combining the physical location, gained via technologies such as GPS and ultrasound tracking, informational location in terms of recently used documents, and virtual location in terms of position and motion in maps and virtual environments. For example, someone using a mobile computer in a museum may browse web documents about exhibits nearby. These documents may link through to a 3D model of related cultural institutions, which the user might then use to find out about examples of interesting architectural design in the same city. In the system we are developing, that user would gain recommendations based on their location in the museum, the documents they have just read and the places they have examined in the virtual city. This paper outlines this system, and sets it in the context of Equator’s aims.
A path is a time–ordered history of a person's use of information. Each person's use of symbols, such as URLs accessed in a web browser, is logged over time, and this path serves as the system's representation of the person's activity. Unlike most recommender systems, the context of activity is an essential part of Recer’s approach. Each path entry, i.e. each recorded instance of a symbol's use, can be associated with a context in the form of the temporally neighbouring entries in that path. We then treat a person's most recent path entries as a request for recommendations i.e. as a possibly implicit statement of current context and of information need. As we describe below, this current context drives the recommendation process in that we use the pattern of past occurrences of recent symbols and the contexts of those past occurrences as the source of recommendations.
Recer is dependent on external systems to initialise and extend the set of symbols recorded in paths and used in recommendations. Paths would be useless without people's use of Web search engines and file browsers, and indeed editors, mail tools, magazines, journals and so forth. We see this not as a weakness, but as realistic and pragmatic acceptance of the interdependence and holism of tools, and their embedding in the wider world. We interleave information tools in everyday use, and the sets of objects they operate on overlap. One of the improvements to the system of (Chalmers 1998) described in this paper is that we now record activity in the xemacs text editor, in the same path as that of the web browser. Web activity can thus potentially trigger recommendations for local files, and vice versa, and we are looking towards further expansion of the range of tools and media Recer works with. A later section describes this in more detail.
We have experimented with two modes of activity logging. The first records only the URLs and filenames that identify the documents active in an editor and a browser i.e. no document content. In the xemacs editor, a timer triggers a check every five seconds for a change between editor buffers. When one is detected, and there is a filename associated with the new buffer, the filename is written out to a temporary log file along with a timestamp roughly representing the time of the buffer switch. A shell script, emacsWatcher, periodically pipes any new lines added to the log file into a Java program that enters a path entry for each via a JDBC connection. The name of the database table to write to, e.g. path_matthew, is determined by the username property of the Java environment. In the Muffin web proxy, a Java program freely available from muffin.doit.org, each time a new page is accessed we insert a path entry for the URL in question.
In the second logging mode we create a line of text once every second, containing the nearest complete word or symbol to the xemacs insertion point, the current time, and the filename associated with the current editor buffer. This is only written to the log when the symbol differs from the previous recorded symbol. In the Web browser, the proxy records the URL and all unique terms in the page, including contained URLs, each time a new page is accessed. Again, all path entries for the same person are written to the same table. Words and URLs inside documents clearly may be significant, but we have suspended experimentation with this logging mode for the moment. The primary issue was that the computational load of logging interfered with interaction with the web browser and editor. Another issue was that of finding which words are worth logging and worth recommending. Using recent symbols helps to filter unhelpful recommendations, and we are interested to see if we can use this approach as a less blunt instrument than IR’s traditional stop lists.
In a path table, each entry or row is a triple: (timestamp, symbol, source). In the case of a content symbol, the second element represents the symbol itself while the third element represents the document or file it was found in. Otherwise, the third is a duplicate of the second. For efficiency and to save space, in both logging modes we use the hash function of (Pearson 1990) to make 32 bit fingerprints for symbols and sources. A separate lookup table stores (fingerprint, string) pairs. Also, each path table has a primary key based on both the timestamp and the symbol, and an index based on this key.
We treat the user's most recent path entries as an implicit request for recommendations. Periodically, the recommender takes these entries as a set, and searches the database for past occurrences of each one. (At this stage it also collects the set of symbols used in the later ‘exclusion step’.) This search can cover one's own path and/or a selection of others' i.e. the set of paths is a resource shared amongst those who contribute to paths. By offering selectivity of which paths (or rather, whose paths) to draw from, we allow people to use knowledge of their colleagues’ skills and interests to steer the recommendation process. In another change to the system of (Chalmers 1998), we no longer do a brute force search across the entire path database, sliding windows from the start to the end of each path table. The key and index for each table lets us narrow our search to where at least one past occurrence of a recent symbol exists, which greatly aids speed and scalability.
The system collects the context of each past occurrence of each recent symbol i.e. a window containing the path entries immediately following the past occurrence. It tallies the symbols collected from these windows and then, in a final exclusion step, discards any symbols that were recently used. Symbols used recently thus form a dynamic stop list, helping to filter out unnecessary and commonly used symbols. Symbols not filtered out are presented to the user as recommendations. The system then sleeps for a time before waking, collecting a new set of recent symbols and beginning again. To afford some privacy, path logging can be turned on and off at will.
The recommender’s interface is split into two panels. The first displays recommended symbols, split into three subgroups: URLs, local filenames, and 'everything else'. (In the case of no content being logged, the third list is empty.) The second panel has buttons for selecting paths to draw from and for triggering an immediate recommendation, as well as four sliders which set periods of time: the recent period used to define the current context, the window after each past occurrence, the recent period used for excluding symbols from recommendations, and the sleep period between recommendations.
Clicking on a URL sends it to the web browser to be loaded (and logged). Clicking on a local filename triggers xemacs to load and log it. Note that a URL or filename can be recommended even if no–one has actually visited it. If a URL occurs frequently enough in the content of files or pages associated with recent work then it can gain enough tallies to be recommended. In the Everything else list, clicking pops up a window which shows a list of the URLs and filenames which were recorded as containing the selected symbol. A click in this pop–up list triggers display as before. We do not identify the paths (and people) that contributed most to recommendations. No–one has so far objected to the idea, but then no–one has experienced it. Less conservative trade–offs are possible, of course, balancing the utility of finding out about contributors and the potential invasiveness in making explicit exactly who used what, and when (Bellotti 1993). Indeed, we are negotiating with a governmental organisation to run a large–scale trial of Recer, and they require that identities are revealed in order to support the sharing of expertise and experts across the organisation.
When past occurrences of recently used symbols are temporally near to each other, their contexts or windows overlap. Symbols within overlaps then get higher tallies. Highly ranked symbols tend to be, therefore, those symbols most consistently used just after the periods most similar to recent activity. We assume that consistency of use in similar contexts is a good indicator of appropriate (literally, utile) recommendations. Just as the stereotypical recommender system draws recommendations only from the most ‘similar’ people to oneself, path systems draw recommendations from the most similar contexts to one’s own. Statistics for recommendation and visualisation thus form categorisations or abstractions over path entries that are not fixed a priori. Instead, recurrence statistics are made anew for each person at the time of, and based on the context of, each recommendation operation. They are also ephemeral, in that once a new path entry is made after a recommendation e.g. upon clicking on a recommended URL, the old recurrence statistics are no longer valid. Finally, we note that adaptation can occur even if one refrains from logged activity. One’s recommendations may vary as other people’s activities extend their paths.
As with (Chalmers 1998), co-occurrence statistics were used in the layout algorithm of (Chalmers 1996), and the results browsed using the visualisation tool of (Brodbeck 1996). The intention is to show more of the context and structure which lies behind recommendation lists. Layouts involve a relatively small number of symbols, which tend to form ‘strands.’ Each strand is a sequence of symbols collected prior to symbol tallying i.e. one of the periods of past activity used in the recommendation process. Strands cross at multiply accessed symbols, and so symbols used at diverse periods of time, but adjacent to these recurrences, are brought near to each other. Such a visualisation is made in under 30 seconds on a Silicon Graphics O2.
In a second example, recommendations were made via the second logging mode while working with xemacs on the recommender itself. URL recommendations were for JDK and Swing user interface toolkit documentation read during previous editing. Files recommended for reuse were other components of the recommender system, and were of various types: Java files, a shell script, a file of SQL statements and an Emacs lisp file. The Everything else recommendations were for the class System, often used in println statements to output (copious) debugging information, and variable names used elsewhere in the recer class such as doc and recentSymbols. If several programmers' paths were combined in this way, we believe the current system would recommend to each user, in his or her current work, software components that were used by others in similar work but that the user did not know about. Our system would then help support software reuse within work groups.
One theme within Equator is the City, and Mackintosh is one of three current City projects. The Mackintosh project explores new devices and information systems that support mobility, adaptation to context, and access to a mixture of information types. It is a vehicle for an approach to information and computation that takes fuller account of subjectivity and context than most. The project will involve images, textual descriptions, and references to locations in the city, to artefacts in museums and exhibitions, and to electronic resources such as web sites. Beyond the storage of individual items, paths and narratives through them will initially be made ‘by hand’, via a simple text editor, but we will soon have more sophisticated software tools to build up narratives and paths. We do not intend to make these paths ourselves, but will invite a number of parties to contribute their own interpretation of the ‘raw’ data and of existing paths. We plan for these editors or authors to include curators of various Mackintosh resources, people who work with cultural information and institutions in Glasgow, members of design groups such as commercial design firms in Glasgow and local art schools, and residents of the city from various communities and backgrounds.
We propose to support information access via audio, wearable computers, tablets and VRs. In the first case, we would generate a stream of verbal and non–verbal audio, supplied to a phone or to a computer. In the second case, we refer to a wearable computer with relatively small graphical displays, audio interfaces and a variety of context sensors integrated into a 'CyberJacket', as developed by U. Bristol. These wearables are of particular applicability to places such as the Mackintosh Room, where we aim to make available fine–grained tracking of location and to adapt system responses accordingly. By fine–grained we mean to within around 15cm, as we are installing an ultrasound–based tracking system. A tablet refers to a laptop–like computer with wireless network communications, and it may have greater computation and display facilities than a wearable. In VRs, we refer to more high–end graphics systems that convey sophisticated 3D graphical display of virtual environments, including CAVEs (such as that of UCL and another local Equator affiliate, the Glasgow Science Centre).
An important aspect of this variety of media and devices is that we expect people’s activities in each one to be perceptible in the others. This would certainly mean that use of each one would be recorded in the central information resource, and made available for interpretation by people using the system later. It should also mean that synchronous users, possibly using different media, can be made aware of each other and can communicate with each other. An example would be that of a person walking along the physical Buchanan Street, with a wearable that is tracked. That person’s activity would be represented inside a virtual model of the city. Someone in London, exploring that virtual city, would have his or her activity tracked too. The first person’s wearable could show if the Londoner was approaching for a chat, and similarly the Londoner could see and make contact with the Glaswegian.
Tracking activity in the virtual city, the physical city and the information about the city means that the patterns of stored symbols, the analysis of recent activity and the consequent recommendations will span all such media. Recer can be extended to recommend physical objects and locations as well as digital objects as long as these objects have persistent identifiers akin to URLs that we can track and log. As someone changes location, their wearable can log the new location and obtain recommendations for people, places and things to go on to. As someone moves around a virtual world, their location can similarly be logged and woven into the system of paths and associations between objects of heterogeneous types. Recommendations can be made of any or all types, because the recommendation algorithm does not depend on types but instead relies on activity patterns.
Another aspect of paths arises here. When people know they are being logged, they will shape their actions to suit how other people will perceive them. They will change their behaviour, because they know other people may perceive that behaviour later on. This is not so unusual, especially in the case of guides and curators, who often lay out a path through a building or museum collection in order to provide visitors with an interesting or useful experience. This is an important aspect of paths: people can design them. People can either shape their actions to be a path for others to use, or they may use a tool that lets them edit and design a path more explicitly.
This interweaving of physical and digital media, using activity to associate and design text, building, video, museum display, web page, room location, audio track and so on has been called monomedia. This stands in deliberate contrast to the rather wan term ‘multimedia’ that too often involves synchronised use of just two media, audio and video. (Thanks to Willem Velthoven and monomedia.org for this.) Activity stems from previous understanding, but also feeds back into understanding by creating or reinforcing associations between individual objects, individual spaces and individual people. A person’s movement through data, through the city and through society adds to his or her understanding of information, places and people. One aim of this project is to support this interpretive process, improving information and information systems by representing and adapting with real use, making manifest more of the information and understanding that turn city spaces into ‘places’, and offering people useful and interesting ways to interact with each other.