This paper describes recent progress in the development of the Immersive Segmentation environment. This environment allows users to exploit their expert and contextual knowledge in visualizing and segmenting volumetric data sets such as the Visible Human cryosection images. This is accomplished by placing the user at the center of a tight computational loop, with direct control of the segmentation and visualization algorithms. By hiding the technical aspects of image segmentation the system allows the user to focus on the visual interpretation of anatomical features using their domain knowledge.

      The environment is computationally intensive, supporting a fine-grained user interaction. As a client/server application it provides a unique form of interaction between a low-cost client and remote high performance computing resources. Previous attempts to support visualization with remote computing resources have focused on a relatively coarse-grained request/response paradigm ([1], [2]), where the user invokes the application of large computational operations. Current development of the Immersive Segmentation system is focusing on providing access to the system from low-cost stereo-capable PC's over Next Generation Internets.

      The Immersive Segmentation environment provides the user with a stereoscopic view of the data set. The environment produces the illusion that the data set occupies physical space. Imaging of the data set is performed using a high quality ray-casting algorithm that supports a full lighting model and a flexible use of partial transparency.

      All interactions with the data set occur through the use of a 3D position tracked probe. It is through this probe that the user controls the various segmentation algorithms continuously effecting their operation. The application was originally implemented using the Fakespace Immersive Workbench (see Figure 1).

      A fundamental principle of the environment is the application of computation in small units. This supports a fine-grained interaction paradigm by allowing high-cost computations to be performed in interactive time with continuous user interaction. The algorithms used by the system are parallelized to exploit shared memory multi-processor servers.

      All visualization and segmentation algorithms employed by the system operate on local regions of the data set. The region of operation is determined by the position of the user's probe. Within this local region, the system applies several data clustering algorithms. The results of cluster analysis are used to identify and visualize anatomical structures. All segmentation and visualization parameters are determined from the cluster analysis through heuristic rules. The technical details of the algorithms are never apparent to the user. The local region of operation is continuously re-visualized as the segmentation work progresses.

      The local cluster analysis is used to drive two modes of segmentation. In the first mode, a primary cluster is identified and visualized within the local region. In the second mode, a digital stain in injected at the user's position, and propagates through the data set. The stain exhibits a preference for regions similar to the primary cluster. Figure 2 shows a stereo-pair image of the system in operation.

      In this system the client tracks user input and displays left/right image pairs of the scene. The server responds to user direction by performing data segmentation and visualization. The result is a stream of pixel updates that are transmitted to the client for display. The server and client are loosely coupled in the sense that the server operates continuously based on the last known user state. It does not require an explicit acknowledgment from the client before continuing with an operation. Depending on the visualization mode, pixel updates are either persistent or transient lasting only until the next update. In the persistent mode, pixel updates remain in the scene until re-visualized by the user. In the transient mode, pixel updates are discarded and the previous values recovered at explicit frame transitions. Movie 1 provides an example of both visualization patterns.

      The responsiveness of the system is directly related to the effective network bandwidth and latency between the client and server. The client/server communication protocol is implemented at the UDP (user datagram protocol) layer in order to maximize available bandwidth. Each data packet sent between the client and server is independently interpretable and is stamped with a sequence number and other identifying information. The data streams to and from the server are independent with no explicit acknowledgment of data packets. Instead the client periodically requests the server to re-transmit lost packets as they are discovered through missing sequence numbers. Transient data, no longer required by the scene is discarded. This design allows the system to tolerate a modest level of data loss. The bandwidth required by the system varies substantially with the intensity of the activity but ranges between 5 and 40 Mbps per visualization stream. A round- trip latency of 50-100ms provides adequate responsiveness.

      The system has been tested with connections between the University of Wisconsin - La Crosse, the NASA Ames Research Center and the NASA/Stanford Biocomputation Center. The networks traversed in these tests include WiscNet (Wisconsin State education network), Nap.net, NISN (the NASA production network) and CalRen (California State Research and Education network). The roundtrip latency of these connections is approximately 70ms but varies widely depending on the traffic load of the commercial internet.

      The client/server UDP protocol has been tested in two forms. As a fully reliable standalone implementation, the protocol is able to achieve data transfer rates of 25 - 60 Mbps between Stanford and La Crosse. When embedded in the Immersive Segmentation environment (in a semi-reliable form) the application is able to effectively use data rates up to 40 Mbps. These attributes fit well with the quality of service (QoS) and bandwidth reservation protocols under development for Next Generation Internets and will allow the application the rely on a predictable communication channel between the client and server.

      The UDP layer implementation of the client/server protocol also allows the server data stream to be multi-cast. This means that the server's data stream can be received at multiple locations without requiring explicit action on the part of the server. This capability can be used to support multiple simultaneous users. In this model, geographically separated clients can connect to a shared server and view the same pixel update stream. The server would track the probe of each client and perform the segmentation work for each. The server could also generate multiple data streams to support different views of the work for different users. This extension results in an environment that intuitively supports collaboration in segmenting and visualizing data sets.

      The goal of this environment is to allow a user to interactively apply substantial computing power to the task of segmenting and visualizing data sets such as the Visible Human cryosection images. By implementing this system using a client/server design with specific QoS requirements for the network connectivity, the power of this environment can be made available to remote users over an NGI network.

      The Immersive Segmentation environment was originally begun at the NASA Ames Biocomputation Center (Dr. Muriel Ross, Director). Work on its adaptation to NGI networks is continuing in conjunction with a National Library of Medicine contract (# NO1-LM-3506), to develop medical applications of the NGI, awarded to the SUMMIT group (Dr. Parvati Dev, Director) at the Stanford Medical School.