Management of Medical Multimedia Data

Digital production of medical data in almost all medical institutionsis increasing exponentially (Geneva UniversityHospitals’ radiology produced on average 12,000 images per day in 2004, 40,000 in 2006, and over 70,000 in 2007). These data are an integral part of aiding diagnosis and treatment planning.

At the same time as data production is increasing, the variety of data produced and diagnostic tests available is increasing as well, creating sometimes an overload of information for the clinicians.

All data are now directly accessible within the electronic patient record and are not mostly reviewed by specialists in radiology or nuclear medicine as was the case in the past [1].

All patient data are now accessible in digital form, so it can also be reused to exploit the important information stored in it and help clinicians for similar patients in the future [2]. This secondary use of medical data is currently a hot and high potential topic in medical informatics. Legal guidelines still need to be developed for data reuse respecting the private sphere of patient.

Multimedia Data Sources

When thinking about the electronic patient record, structured textual data come tomind first. However, the situation is much more complex as graphics, images, 3D volumes and video streams are equally available and need to be analysed. Here is a short list of sources commonly stored inmedical patient records that could benefit from automatic visual analysis, here ordered by increasing complexity:

1.        free text (release letter, anamnesis, …) often stored in pdf format, sometimes scanned;

2.        one-dimensional signals that are often time-based such as ECG (Electrocardiogram) and EEG (Electroencephalogram);

3.        two-dimensional signals or images: x-rays, dermatology images, pathology images, …;

4.        series of images belonging together such as several photographs of the entire body for dermatology;

5.        series of pseudo-3D image slices such as tomographies (CT, MRI, PET, SPECT);

6.        videos: sleeping laboratory, cardiology, endoscopy, …;

3-dimensional images: reconstructions from tomographic images or ultrasound creating surface or volume images;

4-dimensional images: flow simulations based on 3D datasets, for example to show flow in aneurisms, or functionalMRI (fMRI);n-dimensional combinations of modalities: PET/CT combinedmodalities, for example.

This list can only give starting points and the problem of treating videos in the Picture Archival and Communication System(PACS) is currently only starting.

The Geneva sleeping laboratory, as an example, produces currently over 1,000 DVDs of video data per year and no automatic analysis of these data is performed at themoment.

Content-Based Information Search

Content-based image retrieval (CBIR) has been an extremely active research domain [3] in the non-medical field as data production has risen strongly through the availability of digital cameras. Many of the image archives had little or no annotation, creating the need to navigate among large data sets directly by viewing the visual content of the images and not through textual annotation.

In themedical field CBIR was proposed very early [4]. Nevertheless, and even aftermany years of research, only a few prototypes exist in clinical practice, although a first clinical study showed a significant gain in diagnostic quality [5].

An extensive review of current image retrieval techniques can be found in [6].Generally, visualmethods attempt to find visually similar images to one or several example image(s) supplied by clinicians. This allows searching for similar cases based on visual data or automatically pre-classifying images for further analysis. Other clinical data, or the context in which the image was taken, also needs to be taken into account.

Teaching searches such as “Show mex-ray images similar totuberculosis” caneasilybeperformedwitha visual example but hardly with text.

Visual Features

Instead of using textual data for retrieval, CBIR automatically analyses the image content and extracts features that represent the images for retrieval. These are supposed to be similar to words extracted fromfree text but in general some information is lostwhenautomatically extracting visual features.

The most commonly used visual features are:

color or grey level features globally in the formof a histogramor locally in image regions;

texture features describing the repetitiveness in local homogeneous patterns, for example for describing the texture of lung CTs to aid the diagnosis of interstitial lung diseases;

shape features describing the form of identified objects often after a segmentation of the image into homogeneous regions, and salient points or interest points have emerged as a powerful feature over the last years including invariance with respect to small changes in the images, for example rotations, shifts, or in intensity.

All these features describe the content of the images itself, albeit with an information loss. To obtain good retrieval results other clinical data or the context in which the images were taken need to be taken into account, such as the age of the patient, the weight, the reasons for taking the image, or themedication use.

User Interfaces

An extremely important aspect of visual search is the user interface with which the clinician communicates. Inmost interfaces a query is performed with an example image or case, and then themost similar images are shown in decreasing order of similarity comparable to a Web search engine. Newer interfaces also allowmarking regions of interest in the images to concentrate the analysis and search on a small part of the image.

Conclusions and Outlook

The electronic patient record is increasingly becoming amultimedia patient record. These new data sources need to be included into an automatic data analysis circle to fully exploit the knowledge stored in them.

Content-based image retrieval in connection with clinical data has the potential to help particularly - less experienced clinicians in the decision making process and to exploit knowledge stored inherently in past cases in an efficient way. Still, tomake these tools a success, access to large data repositories needs to be made possible and this includes organisational as well as legal changes in the system.

Once these barriers are overcome, image retrieval need to take into account visual features as well as the entire clinical context of the patient to retrieve similar cases in order to aid diagnosis and plan treatment by extracting knowledge from the patient record including visual information.

Acknowledgements

This work was partially funded by the European Union through the KnowARC project (grant 032691) and the Swiss National Science Foundation (FNS, grant 205321-109304/1).

References available upon request at [email protected]

Authors:

Henning Müller (ab),

David Bandon (a),

a.        Antoine Geissbuhler (a),

Medical Informatics, University and Hospitals of Geneva, Switzerland

(b) Business Information Systems, University of Applied Sciences, Sierre, Switzerland

E-mail: [email protected]