DWARF - Distributed Wearable Augmented Reality Framework

Technische Universität München,
Chair for Applied Software Engineering

Prof. Bernd Bruegge Ph.D.,
Prof. Gudrun Klinker, Ph.D.

(view as PDF file)

The Chair for Applied Software Engineering at the Technische Universität München is doing research on the field of Augmented Reality systems. Augmented Reality (AR) is a technology by which a user?s view of the real world is augmented with additional information from a computer model. Users can work with and examine real world objects while receiving additional information about those objects or the task at hand. Rather than pulling the user into the computer?s virtual world, Augmented Reality brings information into the user?s real world, thereby building upon people?s visual and spatial skills. Augmented Reality constitutes a very promising new user interface concept for many applications, e.g., in medicine, exterior construction, interior design, the assembly, maintenance, and repair of complex technical objects, and games. From a software architecture point of view Augmented Reality systems can be divided into a set of components that each contribute a dedicated functionality to the whole system.

The task was to find a solution where rapid prototyping with several researchers on dierent computers should be possible. This way we wanted to accommodate the heterogeneous infrastructure of the chair consisting of several Intel-based PCs and laptops with Windows and Linux, Apple workstations and laptops, and Compaq iPAQ PDAs. Additionally it should be possible to integrate third party components such as external trackers into the system. The solution to these requirements a framework for component-based peerto- peer systems, called Dwarf. We model Augmented Reality applications as distributed systems which form spontaneously from mobile and stationary components. Such a system can reconfigure itself at runtime by exchanging components and changing component configurations. As communication infrastructure we chose CORBA.

Architecture.

The Dwarf framework is based on the concept of collaborating distributed services. The services are interdependent and expose their requirements, called Needs, and oers, called Abilities, with the help of service managers. On each network node of a Dwarf system, there is one service manager; there is no central component. The service manager controls its local services and maintains descriptions of them. Each service manager cooperates with the others in the network to set up connections between services. Each service has an xml description of its Abilities, Needs and connectors (communication protocols). This concept extends the approach of the CORBA Component Model (CCM) of required and oered interfaces respectively event sinks and sources towards more flexibility in the inter-service communication and is part of the research eorts. Each Ability has a set of attributes describing quality-of-service parameters of that service. Likewise, each Need specifies a predicate about the quality of service it expects. This predicate is used by the service manager to select abilities that can provide a sucient quality of service to satisfy a given need. This predicate can also be used at runtime to ensure that the desired quality of service is still provided.

Implementation.

The stationary computers are connected together using standard 100 megabit ethernet; the mobile computers (laptops and palmtop) are connected using IEEE 802.11b wireless ethernet. The Dwarf services are realized as separate processes and threads within single processes. Distributed middleware, consisting of CORBA and several Dwarf service managers, connect the services together. Upon startup, each service registers itself, via CORBA, with its service manager running on the local machine, which listens for connections on a well-known TCP port. The middleware is lightweight enough to run on the Linux-based palmtop as well as the fullsized machines.

The service managers running on the different machines communicate with one another using SLP and CORBA and set up connections between the services. The services then use CORBA method calls or CORBA notification service events to communicate. The format of communication between the services is specified in CORBA IDL. Some services have CORBA interfaces with specified methods that are called by other services; most, however, communicate using CORBA structured events.

The implementation of the Dwarf service manager uses the open-source OmniORB CORBA implementation, which is also used for the services written in C++. Additionally, it uses OmniNotify, a CORBA Notification Service implementation based on OmniORB. Both OmniORB and OmniNotify were cross-compiled to Linux StrongARM so that the middleware could run on the palmtop. Services written in Java use OpenORB, an open-source Java CORBA implementation, and JavaORB, its predecessor, which was necessary for the Java 1.1 virtual machine needed to control the Cortona VRML browser under Windows. To find services managed by other service managers, the service managers use OpenSLP, an implementation of the Service Location Protocol SLP. This allows services to be found based on type and named attributes. OpenSLP can evaluate predicates based on boolean expressions, including wild cards, which is used to connect the matching needs and abilities based on their predicates and attributes.

Tool support.

As a general-purpose monitoring and debugging tool, we implemented a tool that presents a graphical view of the network of interconnected services that dynamically changes when the system con- figuration changes. It also allows developers to view arbitrary event streams in the system, which proved invaluable for debugging.

Using Dwarf, we have previously built Pathfinder, an experimental system for campus navigation; Fatamorgana for the visualization of prototype automobile designs; Tramp for automobile repair;FixIt^? for machine maintenance; and Paarti, prototype vehicle construction. We also built Sheep, a multiplayer collaborative game (figure 1) with tangible user interfaces. Finally, new projects based on Dwarf^? are underway. At the time of writing, these include Archie, a collaborative architectural design system, Heart, a system for cardiac surgery, and Pong, a set of simple Augmented Reality games authored with a Python-based scripting interface.

Benefits

Our project is the first that uses CORBA to realize an Augmented Reality system. Since that first prototype in 2000 several others have been built based on the described concept. To use CORBA as the underlying middleware had several advantages over alternatives such as socket communication, Java RMI, SOAP, DCOM, or proprietary solutions. Platform and language independence. CORBA allows us to develop our services in a broad range of programming languages. In particular, we use C++, Java, and Python. With CORBA we are free to chose the development language from case to case. Usually we chose programmin language and operating system because we need to build against a third-party library to implement a service. Currently we have a mixture of services deployed on workstations and laptops running Apple Mac OS X, Linux, and Microsoft Windows. We even have services running on a Compaq iPAQ with Linux.

Rapid prototyping.

As we used the concepts of the Dwarf framework, all developers were forced to use components running as separate processes. This lead to a clear and lean definition of the communication between dierent components in CORBA IDL files. In addition, the relationship between components of a running system could be displayed dynamically. This allowed us the very late integration of several student developers in the overall process without the need for lengthy documentation of the current system state, a simple look at the debugging tool suced. In fact, several people unfamiliar with the overall system were given and completed successfully self-contained tasks such as adding a collision detection component two weeks before the final deadline of the overall system. Use available hardware. CORBA is running on every hardware and operating system that is already available at our chair. So we can concentrate on developing the service functionality instead of thinking about the best hardware/operating system combination for a new system.

Integration of third-party hard- and software.

The implementation of Augmented Reality systems is very laborious if every component should be built from scratch. On the other side, there are several components available from other research groups that could be reused. Quite often these components require a fixed runtime environment, porting would be to costly. CORBA helps us to develop wrappers around these components in the required programming language and operating system.

Remote monitoring.

As a general-purpose monitoring and debugging tool, we implemented a tool called the Dwarf Interactive Visualization Environment. In analogy to visual programming interfaces in systems such as AVS, this tool presents a graphical view of the network of interconnected services that dynamically changes when the system configuration changes. It also allows developers to view arbitrary event streams in the system, which proved invaluable for debugging. In addition, the tool let us explain the running system to technically interested spectators and illustrate the distributed nature of Dwarf.