Visual Search in Radiology Archives and the Open-Access Medical Literature
Dimitrios Markonis, MSc 1
René Donner, MSc 2
Markus Holzer, MSc 2
Thomas Schlegl, MSc 2
Roger Schaer, MSc 1
Georg Langs, PhD 2
Henning Müller, PhD 1
1 University of Applied Sciences, Western Switzerland, Sierre, Switzerland
2 UCIR Lab
Department of Biomedical Imaging and Image-guided Therapy Medical University of Vienna
- Radiologists are overloaded with large amounts of visual data;
- Current searches for images often fail, or return results for questionable quality, particularly on the Internet where there is little control on quality;
- The KHRESMOI project is developing a medical visual information search engine for radiologists;
- The system contains novel information retrieval features such as search by image example and makes use of semantics to improve the retrieval performance;
- Search into internal hospital visual data by registered users is supported, and related cases from the open access medial literature are linked automatically;
- The project follows a user-oriented design process, and the prototypes are user-tested by radiologists.
Radiology is a field strongly linked to medical imaging. The size of visual data being produced in hospitals increases exponentially, and new imaging techniques are being introduced regularly. An EU report estimated medical imaging to occupy 30% of world storage in 2010 (High Level Expert Group 2010). However, after being used once for diagnosis, typically this enormous amount of visual information in hospitals remains unexploited. Recent studies have shown that, even though visual information search is a task performed daily, radiologists fail to find relevant images one out of four times using existing solutions (Markonis et al., 2012). Access to electronic records is often solely patient-based, while image search on the Internet is mostly performed using general-purpose search engines that return results of questionable quality.
The KHRESMOI Project
“Knowledge Helper for Medical and Other Information users” (KHRESMOI*) is a four-year European project funded since 2010 in the 7th Framework programme, consisting of 12 partners from academia and industry that originate from 9 European countries. KHRESMOI aims to create a multi-lingual, multi-modal search and access system for biomedical information and documents. Radiologists are among the three target user groups of the project (see Figure 1), as they often search for images during their clinical work, teaching duties and research activities. The system uses tools that exploit state-of-the-art techniques in information retrieval to assist radiologists in demanding information need scenarios. The two other user groups of patients and general practitioners concentrate rather on text and semantic search and prototypes have been developed for these groups. This article details the radiology search system.
A user-oriented approach was followed to develop and evaluate the entire system. Following a user-centred design the system is more likely to correspond to realistic requirements of the target user groups. Observation of the radiology clinical workflow and interviews with radiologists assisted in identifying the information needs of radiologists and creating the portrait of an ideal system (Markonis et al., 2012). The development was performed in an iterative manner, having radiologists test the prototypes (Markonis et al., 2013), and modifying the system according to the feedback from the user tests. Currently, a new version of user tests is being prepared for obtaining additional feedback before the final prototypes are implemented.
Content-Based Image Retrieval
In clinical work in radiology, such as differential diagnosis or when an unknown abnormality is found in an image, search by keywords for finding answers to an unknown visual finding is not the optimal or even an applicable solution. Being able to mark a region of interest in an image can be much more efficient and effective than comparing an image on screen with images printed in books. Apart from conventional text-based search, the KHRESMOI system makes use of content-based image retrieval (CBIR) to address these common yet often unaddressed scenarios. CBIR is an information retrieval technique that allows querying by using an image example or image regions to retrieve visually similar results without the need of text. In some cases images are combined with text to leverage additional information available at query time. CBIR has been proposed as a promising field in medical applications (Müller et al., 2004; Aisen et al., 2003), but only a few applications have yet reached the clinical environment. Many of the problems are linked to low retrieval performance when only visual information is used and the combination of text with visual attributes seems most promising. In Figure 2, an example screenshot of an application developed in KHRESMOI can be seen.
Search in Hospital Imaging Databases
The KHRESMOI system allows for CBIR search in databases of radiology reports and imaging data such as magnetic resonance imaging (MRI) or computed tomography (CT) volumes (see Figure 3). A typical user assesses an individual case by analysing the imaging data (e.g., a lung CT) and uses basic image manipulation to improve visualisation (zoom, level window settings). If an unknown abnormality is found, he/she can mark a region of interest (ROI) and initiate a search. The system will automatically extract visual characteristics of the ROI and search the database for volumes of the same anatomic location that contain ROIs with similar visual features. The results are returned in the form of a ranked list, and the user can select individual result cases to be displayed in full size. In these cases ROIs that are similar to the query are highlighted in the images.
The radiology reports associated with the volumes are also available. Radlex terms (Langlox, 2006) relating to anatomical regions, pathological observations and other aspects mentioned in the report are highlighted, while filtering of the results by Radlex terms is supported to focus search results. Finally, the system automatically analyses the top ranked results to find diagnosis groupings relevant for differential diagnosis. The proposed diagnoses are then shown in decreasing order with the goal to help differential diagnosis. The physician is still taking decisions and creating the radiology report, but the system provides the expert with relevant information and possible pointers for difficult-to-diagnose abnormalities. The search can be extended to the open access medical literature using as queries the diagnosis terms and slices of interest. Again, visual information and textual data is used to provide specificity during an extended content-based information query.
Figure 2. Screenshot of a combined text and visual search interface that allows keyword search and a search for similar images to an example
Figure 3. 3D search interfacein images from a routine PACS and radiology reports
The primary objective of the system is to provide radiologists with efficient access to information in the hospital image storage system (PACS) together with corresponding reports when preparing a new radiology report. This allows use of the knowledge of other physicians who described similar cases in the past. The system was demonstrated on a collection of 3876 volumes of CTs and MRIs extracted from the PACS of the General Hospital of Vienna, and scalability has been taken into account in order to be able to handle larger datasets that occur in all big hospitals.
Assessing Images from the Open Access Medical Literature
The biomedical literature contains a large amount of academically interesting or rare cases. Apart from the obvious uses in teaching and research activities this information can also be used in differential diagnosis. The KHRESMOI system allows image search into the open access medical literature and straightforward connection with the corresponding articles (see Figure 2).
A search may be initiated by keywords and/or image examples or as a succession of the search in the internal hospital PACS described in the previous section. Filtering results by specifying the imaging modality is available, while a number of images in the result list can be marked as relevant or non-relevant to quickly reformulate the query and obtain better results. Images can be selected from the results view to be displayed in full size along with their details. Once an image is selected, the system provides the user with easy access to the details of its corresponding article and also the figure caption to quickly see the context of a figure.
The goal of this functionality is to connect the search in hospital PACS to trustworthy (peer-reviewed) external sources, and exploit the visual information found in the open access medical literature. For this purpose, an image dataset of 1.7 million images included in PubMed Central articles was indexed, demonstrating also the scalability of the system. The reader may find the algorithmic methods used and technical details of the system in Khresmoi publications (Langs et al., 2012a; 2012b). A specific challenge in figures in the literature is compound figures that contain several sub-figures. For separating these sub-figures automatically while keeping the context, tools have also been developed to focus the search (Chhatkuli et al., 2013).