VITA: Visual Interpretation with Three-Dimensional Annotations

Sharmili Roy Michael Brown George Shih

(JDI'13, RSNA'11)


Imaging is a vital component of modern medicine. A typical clinical workflow, pictorially depicted in Figure 1, involves a referring physician who requests an image exam and a radiologist who interprets this exam and prepares an exam report. The exam interpretation often involves using image-based tools to markup annotations on the exam’s images, for example, drawing geometric primitives to denote lengths and volumes of interest, drawing arrows with text annotations, and selecting key images from the exam that represent images of clinical significance. A final text-based report is prepared to summarize the key findings. The benefits of these radiological visual annotations are well known in the medical community. Unfortunately however, this information is not always easily available to the referring physician. This could be due to several reasons starting from software incompatibilities between physician’s office desktops and radiologist’s workstations to tedious workflows which require the physicians to sift through exams that often contain thousands of images to find and match radiological annotations with the findings described in the report. In addition, because the annotations are made only on a few key images, the context of the annotation within the 3D volume may not always be clear, especially to the patient.

Figure 1: This figure gives an overview of a typical clinical workflow where a patient undergoes a radiological exam which is interpreted by a radiologist. The radiological findings are summarized in a text report and sent to the doctor for diagnosis. The radiological exam data and the annotations are lost in the workflow.


In this project, we propose a three-dimensional radiological reporting framework which enhances the archaic paper-based radiological reports with three-dimensional animated visual reports. The visual reports are generated by volume rendering of the exam pixel data with the radiological visual markups embedded and clearly highlighted as the volume spins to provide a comprehensive summary of the important clinical content.


Figure 2 presents a typical radiology reporting workflow employed in clinical practice and how our visual reporting framework is integrated into the workflow. Central to this workflow is the Picture Archiving and Communication System (PACS) server which is used as the central repository of image-based studies and the Radiology Information System (RIS) which is typically where the text-based report of the radiology exam is stored. Medical images are stored in the PACS archive in the Digital Imaging and Communications in Medicine (DICOM) format. Under this framework, a technologist performs an image-based exam on a patient that is sent to the PACS server where a radiologist accesses the exam images and prepares a report for the referring physician. This exam report is sent back to the PACS server with the text-based portion also sent to the RIS server. The referring physician typically receives the report via RIS. The text-report in the RIS system can sometimes refer back to the original study in the PACS system. VITA uses the exam images and annotations employed by the radiologist during exam interpretation to produce a visual summary. This visual summary is sent to the PACS server as a new DICOM series which can be downloaded by the referring physicians for diagnosis.

Figure 2: This figure gives an overview of a typical radiological reporting IT set-up and explains how our visual report module can be integrated into the existing workflow. Our framework can work either directly with PACS, RIS, or even an external database that is cross-referenced via RIS.

Since the annotations are now embedded in the DICOM pixel data of the visual report, many issues of software incompatibilities with regards to annotation implementation across PACS vendors are avoided. This also allows our VITA framework to seamlessly integrate within the existing workflow as no additional input is needed from the radiologists and the results are available in PACS. In addition, we have the ability to produce video versions (e.g., AVI, MOV, or MPEG) of the VITA summary for sharing with patients and for situations when access to PACS or DICOM viewers is not readily available.

While PACS and DICOM are supported by all vendor software, the manner in which proprietary software encodes annotations and markup is often a source of incompatibility. Most software, however, use a structured format like Extendible Markup Language (XML) to implement annotations. Further, in order to unify annotations across PACS frameworks, the National Institutes of Health Cancer Biomedical Informatics Grid (NIH caBIG) has initiated the Annotation and Image Markup (AIM) project which provides a standards-based annotation format that can be shared between different PACS.

Our VITA system is developed in C++ using Nokia’s Qt cross-platform application and UI framework. VITA can be used on standalone computer or with ClearCanvas PACS workstation. ClearCanvas workstation allows radiologists to draw geometric shapes over images, e.g., lines, ellipses etc and associate text with the drawn geometry. This markup is saved either as ClearCanvas study file in the XML format or in the unified AIM schema in both XML and Digital Imaging and Communication in Medicine Structured Reporting (DICOM-SR) format.

VITA has a built-in XML parser module that mines ClearCanvas study files and AIM XML files associated with a medical exam to extract the annotated geometric primitives, observations and text tags. The geometric primitives are sent to the GPU-based rendering engine of VITA which produces a visual summary in the form of a 3D volume animation using ray casting that renders the volume as it spins 360° around the spinal axis. The geometric primitives are distinctly highlighted in the volume. Figure 3 captures this pipeline. As the volume spins, the visual report is saved by generating image files in the DICOM format at every 10° rotation of the volume. The DICOM images are composed as a DICOM stack and pushed to the PACS archive as an additional DICOM object to the existing study.

Figure 3: The VITA framework uses radiologist annotations prepared using a structured format (e.g., XML, AIM). Geometric primitives are extracted from the annotation encodings and used to produce visual summary in the form of a rotating 3D volume rendering.


A demo on how to use VITA software framework is shown in the following video.

The following videos show some rendering results of VITA and how the VITA visual report appears at the Physician's office desktop.

VITA Physician Evaluation

So far seven physicians based out of Singapore have participated in a user satisfaction study of VITA. During the study, the participants were shown three anonymized radiological reports along with the corresponding key images that had the graphical overlays of the annotations. The first case reported a solitary pulmonary nodule in a lung CT exam, second case reported tumor in brain MR images and the third case was about calcified lesion in the liver diagnosed in abdominal CT. For these three cases participants were also given the visual reports generated by VITA. Physicians were asked three questions: 1) whether the VITA summaries improve clarity of communication between referring physicians and radiologists; 2) whether the VITA summaries would be useful in assisting physicians in communicating diagnosis to patients and; 3) whether they would be willing to use the VITA service in their clinical routine if made available. The first two questions asked the participants to rate their answers on a 5-point Likert scale with one being strongly disagree and five being strongly agree. The following figure shows the results of the evaluation survey.

Figure 4: This figure shows the results of a user satisfaction study performed with seven referring physicians.

Six out of seven participating referring physicians strongly agreed that 3D visual reports improve clarity of communication between radiologists and physicians and also strongly agreed that 3D visual reports would aid patient communication. One physician agreed that visual report improves clarity of communication between radiologists and physicians and was neutral on whether visual summary aided patient communication. Six participants were willing to use the system in their routine clinical practice. Comments from participants were also positive, samples include “it is a new brilliant concept for patient understanding” and “this is an excellent intervention which helps in better collaboration between physician and radiologist.” The only physician who was neutral on using VITA commented that “3D rendering does not add additional information for clinician. It looks nice for the layperson i.e., patient, but clinical use is very limited.”

Help us improve VITA by taking the survey at the following link:


  1. Sharmili Roy, Michael S. Brown, George Shih, "Visual Interpretation with Three-Dimensional Annotations (VITA): Three-Dimensional Image Interpretation Tool for Radiological Reporting," Journal of Digital Imaging (JDI'13) [PDF] [Slides]

  2. Sharmili Roy, Miaojun Yao, Michael S. Brown, George Shih, "Visual Interpretation with Three-Dimensional Annotations (VITA): Open Source Automated 3D Visual Summary Application Using AIM (Annotation Imaging Markup) Enabled PACS Based On Radiologist Annotations," Radiological Society of North America (RSNA'11), Nov-Dec 2011 [PDF] [Slides]


© Copyright 2014, Sharmili Roy