Progress Report

Title: A Collaboration Testbed in Medical Image-Based Examination, Diagnosis, and Treatment
NSF Grant ECS-94-22701
PI: Terry E Weymouth
Co-PIs: Charles Meyer, Atul Prakash, Thomas Finholt, Ronald Adler, Michael D. Cohen; University of Michigan.

Post questions and comments to: Terry E Weymouth.

Contents:

• Introduction
• Project Summary
• Behavioral Science - Observations of actual procedures
• Real-time Collaboration - Development of technology
• Asynchronous Collaboration - Development of technology
• Tech Transfer from the UARC project
• Students
• Papers

Introduction

• In the initial four month period, we have accomplished the following:
• Initiated an in-depth study of the structure and flow of image-based consulting in the ultra-sound and radiology laboratories at the University of Michigan's teaching hospital.
• Initiated work on computer tools for real-time and asynchronous collaboration.
• Used tech transfer from a related project.
• Started a number of graduate students on project.
• Published two papers, with others "in the works."
• Set up infrastructure: interviewed programmers (one to be hired in the next week or two), acquired workstations, video and sound recording and replay equipment, and C-Video software (for the behavioral studies), looked at tech-transfer from a closely related project (the UARC project, NSF cooperative agreement, IRI-9216848, Dan Atkins, PI).

Here are the details.

Project Summary.

In this project we explore, develop, implement and test a series of prototypes for collaboration centered around the viewing of images and video over a distance. Each prototype supports remote healthcare, linking primary care facilities with a world-class hospital: The University of Michigan Medical Center. The project will simultaneously address issues on three fronts: the development of new collaboration technology, the development of a toolkit for a medical application domain, and the systematic evaluation of the effect of the introduction of collaboration technology on the current practice of consulting. These three aspects of the study reinforce each other: the development of new technologies and their underlying principles is informed by the needs of users in an application area; the social science studies provide clear methods to probe and define user needs; the introduction of a new technology provides an opportunity to study of the effects of technology on a working group, and the medical community gets a new set of tools. In addition to the research on collaboration using images and voice, this project is an extension and further verification of the collaboration technology developed for scientific collaboration in the UARC Project.

We employ a rapid prototyping strategy. This style of development incorporates a careful study of the user community with research and development of toolkits for collaboration. This approach puts actual working systems into the hands of the users in the early stages of the project and maintains a closely coupled and frequent cycle of user feedback, responsive redesign and systematic evaluation. The rapid prototyping forces an evolution of collaboration tools that are useful as well as innovative.

Behavioral Science - Observations of actual procedures.

In the proposal, we identified the need for a focused study of an actual user community. The behavior science component of the NSF Medical Collaboratory Grant is proceeding along several fronts.

First, an exhaustive literature search has been completed and over 200 books and articles on radiological practice (including interpretation of images) today and studies of radiologists are being analyzed and entered into a database.

Second, baseline data collection focusing on radiological practice and consultations with clinicians and primary care physicians is nearing completion. We have gained access to and have made observations in four different radiological services: ultrasound, in-patient bone, out-patient bone, and chest. Additionally, we have made some limited observations in magnetic resonance imaging. The ultrasound data, in particular, is rich in consultations between clinicians/primary care physicians and radiologists.

The data that we have collected is in the form of written notes and videotapes which document a variety of routine activities as well as exception handling in these areas. This is the result of approximately 80 hours of observation with a resulting 35 hours of videotaped information.

Third, the data analysis has also begun in terms of both coding the data (using C-Video software) and in extracting and writing up important elements for consideration in designing the first prototypes of the new system. Approximately 10 hours of videotape has been completely analyzed at this point, and another 5 hours has received an initial review. Several procedural documents regarding radiological practice are also now in draft form. Finally, there have been two sessions in which participants from all aspects of the project: radiologists, behavioral scientists, and computer scientists have watched the videos together and discussed which elements must be incorporated into the system.

In the next few months, we will focus on finishing baseline data collection and primarily devote our energies to intensive analysis of the data collected, and communication with the radiologists and the computer scientists.

Real-time collaboration.

In the proposal we identified the need to:

• develop mechanisms to take advantage of caching of images/video during a collaboration session, so as to reduce processor loads, network bandwidth requirements, and latency.
• provide protocols for synchronizing continuous media (audio) with discrete media (pointer movements and manipulation of images/video). Variations in network delays and differences at the collaborating sites in terms of system load, processing speeds, and access times for loading image/video data need to be taken into account.
• provide mechanisms to arbitrate access to shared workspace, so that simultaneous interactions can be permitted without losing consistency of views.
• do all the above in a manner low latency and adequate response time can be ensured during interactions.

The progress to-date is as follows:

• We have designed and evaluated communication protocols for synchronizing audio with actions in shared workspaces. An earlier version of the protocol and the experimental results were described in a paper that appears in the ACM Multimedia'94 conference. An enhanced version of that protocol has been designed. The protocol uses end-to-end feedback to attempt to meet user-specified quality-of-service requirements, expressed in terms of maximum latency, jitter, and asynchrony between streams. We are currently in the process of writing a paper on the enhanced protocol.
• To experiment with synchronized interactions with video data over a network, we have implemented an early prototype of a collaborative video viewer. The video viewer plays back video between two sites. To reduce network latencies and ensure good response times, video is cached at both the sending site and the receiving site prior to the start of the collaborative session.

Work remains to be done on evaluating the collaborative video viewer, doing dynamic caching during viewing if one site does not have the copy of the video prior to the start of the collaborative session, and integrating the video viewing with live audio and telepointing services.

Asynchronous Collaboration.

The key issues that we proposed to address include the following:

1. mechanisms for recording interactive sessions, which involves both continuous and discrete media, and saving the recording into file(s);
2. mechanisms for mailing such session recordings efficiently over a network;
3. mechanisms for faithful playback of the recording at the receiver's workstation, with the receiver having control over the playback; and
4. mechanisms to allow a back-and-forth exchange of messages that include session recordings and refer to earlier recordings.

The progress on the above issues is as follows:

• We have designed methods for representing a session recording and making such recordings persistent. The methods will be presented in a forthcoming paper at the European Computer-Supported Cooperative Work conference.
• To facilitate mailing of such recordings over a network, we have concentrated on minimizing the space requirements for representing multimedia session recordings. We have decided to record user-events digitally and play them back to playback a recording. Such recordings are typically 10-100 times more compact than a compressed video recording of a session and do not have any loss of resolution. Furthermore, we store each media stream separately so that appropriate compression techniques can be applied to each stream individually.
• We have begun to explore representation of video artifacts in session
• recordings.

Tech Transfer from the UARC project.

Software to support collaboration had been developed under the UARC project, the "Upper Atmospheric Research Collaboratory," supported by NSF cooperative agreement, IRI-9216848. Four of the students and two of the PI from the Medical Collaboration project (Terry E. Weymouth and Atul Prakash) have also worked on, or are being partially supported by, the UARC project. In both projects we are researching and developing tools for computer supported collaboration. This synergy has netted a gain for both of these closely related projects.

In particular, work on asynchronous collaboration was initiated under UARC but now continues in this project, work on an object-oriented toolkit for building shared shared-windows was supported by UARC but is now being used in this project, and both projects will profit (in the long run) by the more general development of collaboration tools that comes from forcing the tools to work in multiple domains.

Graduate Students.

Following computer science graduate students are working the project(+ indicated those students that are directly supported by the projects, other students are supported on closely related projects):

• Nelson Manohar(+): working on asynchronous collaboration using session record/replay paradigm.
• Amit Mathur(+): working on communication infrastructure for collaborative systems
• Hyong Shim: working on an object-oriented toolkit for building shared window applications.
• Gwo-ba Wu: addressing concurrence control issues in collaborative systems.
• Trent Jaeger: addressing security issues in collaborative systems.
• Elizabeth Yakel(+): collecting, indexing and analyzing behavioral observations.

Papers.

T. Jaeger and A. Prakash, "Implementation of a Discretionary Access Control Model for Script-based Systems," Proc. of the 8th IEEE Computer Security Foundations Workshop, County Kerry, Ireland, June 1995.

N. R. Manohar and A. Prakash, "The Session Capture and Replay Paradigm for Asynchronous Collaboration," to be published in the Proceedings of European Conference on Computer-supported Cooperative Work, Stockholm, Sweden, September 1995.