wiki:UserObservationHub

Version 10 (modified by schwarz, 17 years ago) (diff)

--

UserObservationHub (UOH)

Check out the UOH from this SVN repository: 

https://usercontext.opendfki.de/repos/trunk/UserObservationHub

Important classes:

The UserObservationHub - shortly: UOH (pronounced "uh-oh!" ;-)) is a compact project and service catching several registered user observation notifications (e.g., from ​http://dragontalk.opendfki.de) and passes them on to interested listeners. For one user, one user observation hub is running on his or her desktop. There are multiple input sources that actually observe the user - these observers are implemented as plugins to applications or as listeners to operating system events. They connect to the User observation hub, or more precisely to collectors which are started together with the user observation hub. Each collector provides a different kind of interface, the differences can be both on communication layer (SOAP, REST, XML-RPC) or on the message layer. Some observers report messages to the collectors using serialized RDF conforming to the NOP ontology, others call methods of a collector and pass information as method arguments. Therefore, multiple collectors exist providing the needed interfaces. Each collector standardizes the input observations to RDF, which is then passed to the user observation hub.

The interfaces of the collectors are, in their respective protocol, similar to this: 

/* Collect a reported observation, protocol is REST, message is already in RDF, 
   the rdf can be parsed and passed to the UOH */
RESTCollector.userobservation(String uri, String observationAsRDFXML);

/* Collect a reported Browser Navigate To URL Operation,
   Protocol is XML-RPC, message format is specific to the NOP ontology.
   A RDF has to be created from the parameters, more values of the observation are added
   (such as type of operation and current time) and then passed on to UOH */
eventBrowserNavigateToURL(String currentURI, String toURI);

The methods of the collectors are called from the application plugins via remote protocols. The main functionality of a collector is to transform the input to RDF and invoke the fireObservedNopGraph method of the UOH. All observed native operations pass through this method of the UOH. 

/*
  public void fireObservedNopGraph( String nopUri, String modelAsRdfXml );
*/

As the name indicates, the UOH receives messages and notifies other components about them. The listeners can be registered with the UOH on startup. All registered listeners are called in sequence. In the default implementation, there are multiple listeners:

  • OutputObservedNops - prints the message to system-out for debugging purposes
  • NepomukNopForwarder - sends the message to NEPOMUK's User Context service which will compute the current work context
  • RemoteUserObsListener - sends the message to MyMory's User Context service which will compute the current work context
  • SimpleUrlCounter - an example implementation that counts URIs of navigated web resources.

Message format and mapping

The passed messages often consist of two parts:

  • What native operation was observed? This has to be described using the NOP ontology and captures the class of operation, time, and application used.
  • On what data object (file, e-mail, webpage, etc) did the operation work? This can be expressed using NOP, which only provides basic classes for data objects, or other vocabularies.

Before passing the RDF message to the listeners, a mapping of ontologies is done.

The NOP ontology focuses to describe the domain of native operations. Data Objects is not intended to be complete but to be minimal. We acknowledge that other endavours, such as ​NIE (with its more detailed sub-ontologies NCO, NMO, ...) or the W3C ontologies for calendaring are more detailed description of such data objects. However, the NOP Data Objects provide a simple schema to be understood and interpretable by all listeners to a user observation hub, independent of detailed modelling. To realize a standardization of vocabulary, we require observers to describe and pass on observed user actions using NOP's native operation classes. However they are allowed to describe Data Objects either using UOH/NOP DataObjects or in their proprietary data format. As output, the original input messages will be passed on, with data objects transformed to NOP's dataobject vocabulary if needed. So the UOH is liberal in what it accepts and conservative in what it passes on.

The UserObservationHub will translate DataObjects (in any input form) to UOH schema while keeping the "native" (richer) data. For example, the incoming NOP contains a richer data object modeling representing an e-mail using NIE. The message will then be enriched with the simpler UOH data object modeling but the original richer modeling is not removed but remains in the RDF model. A DataObject then may have two types: their original type (nmo:Email in the example) and also the DataObject type (nop:Email).

Classes providing such a mapping have to implement a special interface (MappingToNop) and can be registered as modules with the User Observation Hub. For NEPOMUK's NIE ontology, such a mapping was implemented. It is based on the Jena rule engine and uses horn clauses to generate the additional data in the UOH vocabulary based on data objects represented in NIE. User who want to extend the implementation and pass RDF messages using other vocabularies to express DataObjects, they must provide a mapping module for the UserObservationHub to the UOH vocabulary.

Usage of the UserObservationHub

To use the UserObservationHub in your project, checkout the project from SVN and export a JAR file of it using the packaged build.xml file. Note that the build.xml file does not make a compilation but depends on eclipse to compile the classes.

  • You need all dependencies that are included in the lib folder of the project.
  • To start the hub, call UserObservationHubLauncher.bat, which uses ant to gather the dependencies.
  • If you have startup scripts for other operating systems, please send them to the involved persons or commit them to the project.

The UserObservationHub runs as a standalone application and will open listening ports to receive messages via the registered collectors. It will also test if NEPOMUK is running and forward messages to the NEPOMUK server, as with Mymory. This behavior is determined by the StartUserObservationHub class, other configurations require another startup class.

Once started, the UserObservationHub listens to incoming messages. Typically, plugins added to existing desktop applications will observe the user and once interesting user operations occur, generate a message and fire them to the collector that can retrieve the message. One way of using the Hub is to write more plugins for desktop applications and send more detailled messages about the user operations. This would be the side of creating messages. If a programmer wants to register a new listener, for example to generate log data for long-term data gathering experiments, this would involve writing a new Output Listener and registering it at the UserObservationHub during the startup phase.

End-users never interact with the hub. Applications that want to query the current work context usually contact the Current Work Context service provided by NEPOMUK or MyMory. Indirectly, the last operations and relevant resources can be accessed via these services and shown to the user for context-sensitive support.

List of user observation providers working with UOH

  • ​Dragontalk (DFKI): an open source project hosted by DFKI provides user observation plugins for the Mozilla Thunderbird e-mail client and the Mozilla Firefox web-browser.
  • File System Observer (DFKI): a project hosted by DFKI providing file system observations - currently closed-source because it's using JNIWrapper, a commercial product. You can have the DFKI sources, of course, without the JNIWrapper - send us an email!
  • ​User Activity Logger (L3S): an activity logging tool that creates a log based on common events triggered by the Microsoft Windows user. These data (timestamps, application names, currently used files, application states).
  • A UOH wrapper for ​Real-Time Document Image Retrieval with LLAH (DFKI, Osaka Prefecture University): a new method of real-time document image retrieval which takes as input images captured by a web camera and retrieves their corresponding pages from a large-scale document image database. The core of the method is LLAH ― the algorithm called "Locally Likely Arrangement Hashing" invented at Osaka Prefecture University. Currently closed-source because of the copyright. However, you can get the wrapper source code from DFKI (send us an email) and download the libraries from the LLAH website (see above).

NOP ontology (vocabulary)

The NOP ontology (nop.rdfs) is hosted at these two positions:

Goals:

  • Important is, that the input to the UOH is in a language that can be understood by all listerners, i.e, they all should use the same schema for observed NOPS.
  • On the other hand, information loss is not really acceptable, that is, down-translating to simple UOH language may remove interesting data.
  • The UOH language captures:
    • Native Operations (NOPs), a tree of classes. They are used to express observed user actions, such as navigating in a browser or opening a file.
    • DataObjects, which are a second structure, describe and reference the touched objects, such as files, e-mails, webpages.
  • NOPs are the primary information captured, namely what the user did. The UOH vocabulary aims at providing a thorough and exhaustive description of the domain of NOPs.
  • Data Objects is not intended to be complete but to be minimal. We acknowledge that other endavours, such as ​NIE (with its more detailled sub-ontologies NCO, NMO, ...) or the W3C ontologies for calendaring are more detailled description of such objects. However, we provide a shared schema to be understood and interpretable by all listeners to a user observation hub, independent of detailled modelling.
  • To realize these goals, we require observers to pass on observed user actions using NOP. However they are allowed to describe Data Objects either using UOH/NOP DataObjects or in their proprietary data format.
  • The UserObservationHub will translate DataObjects (in any input form) to UOH schema while KEEPING the "native" (richer) data. That means: The incoming NOP contains for example a richer data object modeling. This will then be enriched with the simpler UOH data object modeling BUT the original richer modeling still stays in the RDF. DataObject then may have two types: their original type (for example, NIE:Document) and also the NOP:DataObject type.
  • Some translation modules are included in the User Observation Hub (for example for NIE). To model DataObjects with unknown schemes, you must provide a translation module for the UserObservationHub to the UOH vocabulary.

Extending the UOH:

  • You may add more subclasses to allow more NOPs or data objects. For example, add a SwitchOnLights NOP.
  • You should inform the UserObservationHub team, such that, the new NOP can be included in the nops.rdfs and distributed to other researchers listening to UOH.

Involved Persons

  • Sven Schwarz (DFKI GmbH, Kaiserslautern) -> for Projects Mymory and Nepomuk
  • Leo Sauermann (DFKI GmbH) -> for Project Nepomuk
  • Julien Gaugaz (L3S, Hannover) -> for Project Nepomuk
  • Gianluca Demartini (L3S, Hannover) -> for Project Nepomuk

Attachments (2)

Download all attachments as: .zip