Main Research Topics:
Medical Image Analysis: Visualization,
Segmentation, Quantification
- MRI Brain image analysis
- Coronary analysis
- Echo-Doppler
- Bacteria and Cellular imaging
Statistical
models for image representation, segmentation and matching
Medical
image indexing and retrieval
Extended
Description:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
Statistical Models for Image Representation, Segmentation and Matching in Content-Based Image Retrieval (CBIR)
With the growing size of digital image libraries, arises the need for
tools that can analyze the image content and represent it in a way that can be
efficiently searched and compared. This research aims at providing a unified
approach to image content representation, segmentation and matching, to advance
content-based image retrieval (CBIR). Region-based representations are used and
currently developed to enable two scenarios for CBIR: the well known
'search-by-query', where a query image, in its entirety, is compared to images
in the archive, on an image-to-image basis; and a ‘search-by-region’ scenario,
in which the user marks a region-of-interest (ROI) on a query image, and images
with similar regions are retrieved. The goal is to enable the focus on
particular details within an image and to progress towards ROI-matching and
object-level retrieval.
Contributions (see refs in list below):
·
The GMM-KL framework has been
developed in the Lab, as a statistical framework for image representation,
segmentation and matching. It involves the use of Gaussian mixture modeling
(GMM) for a continuous region-based statistical representation of the image-content, and information-theoretic criteria of Kullback-Leibler (KL) as a probabilistic measure of
image-similarity that defines distances between continuous distributions. This
formalism extends the Blobworld work, and was shown
to provide state-of-the-art image content-retrieval results (Greenspan etal. 2001).
·
The GMM framework has been extended to support the
representation of sequences in time and provides a novel space-time analysis of
image sequences and video (Greenspan et al. IEEE-PAMI 2004). In the proposed
methodology, unsupervised clustering via Gaussian mixture modeling (GMM)
extracts coherent space-time regions in a predefined feature space, and
corresponding coherent video-region segments in the video content. The
parameters of the GMM are determined via the maximum likelihood principle and
the Expectation-Maximization (EM) algorithm. A key feature of the system is the
analysis of video input as a single entity as opposed to a sequence of separate
frames. Space and time are thus treated uniformly. The extracted space-time
regions allow for the detection and recognition of video events.
·
The GMM-KL is currently being
revisited and modified in several ways. In (Gordon etal.
ICCV-03) initial investigation into making the GMM-KL framework more efficient
computationally is conducted. We are working on introducing novel efficient
approximations for the KL measure. In addition, we are focusing on a “reduced
GMM” model that is a clustering of GMM components, a topic of interest for
modeling image categories or in hierarchical archive modeling.
·
Utilizing the information
bottleneck (IB) method, the GMM-KL framework has recently been applied to
unsupervised image-set clustering and archive modeling (Goldberger etal. IEEE_IP 2006).
·
Utilizing context for image and
video segmentation (Greenspan et al CVIU 2004, Goldberger and Greenspan,
IEEE-PAMI 2006)
Journal Contributions in general-image modeling, segmentation
and matching:
·
C. Carson, S. Belongie,
H. Greenspan and J. Malik, “Blobworld:
Image Segmentation using Expectation-Maximization and its Application to Image
Querying”, IEEE Transactions on Pattern Analysis and Machine Intelligence
(IEEE-PAMI), Vol. 24, No. 8, pp. 1026-1038, August 2002.
·
H. Greenspan, J. Goldberger and Lenny Ridel, “A Continuous Probabilistic Framework for Image
Matching,” Journal of Computer Vision and Image Understanding(CVIU),
Vol. 84, No. 3, pp. 384-406, Dec. 2001.
·
H. Greenspan, G. Dvir, Y.
Rubner, “Context Dependent Segmentation and Matching
in Image Databases,” Journal of Computer Vision and Image Understanding (CVIU),
online publication: October 2003; printed version: Vol
93, pp. 86-109, January 2004.
·
H. Greenspan, J. Goldberger and A. Mayer, “Probabilistic
Space-Time Video Modeling via Piecewise GMM”, IEEE Pattern Analysis and
Machine Intelligence (IEEE-PAMI), Vol 26, No. 3,
pp. 384-396, 2004.
·
J. Goldberger, S. Gordon and H. Greenspan,
“Unsupervised Image-Set Clustering Using an Information Theoretic Framework,” IEEE
Trans on Image Processing (IEEE-IP), to be published 2006.
·
J.
Goldberger and H. Greenspan. “Context-based Segmentation of Image and Video
data,” IEEE Trans on Pattern Analysis and Machine Intelligence (IEEE-PAMI),
To be published 2006.
Related Conferences
·
H. Greenspan, S. Gordon and J. Goldberger,
Probabilistic models for generating, modeling and matching image categories, International
Conference on Pattern Recognition (ICPR'02),
·
G. Dvir, H. Greenspan,
and Y. Rubner, Context-Based Image Modeling, International
Conference on Pattern Recognition (ICPR'02),
·
Jacob Goldberger, Hayit
Greenspan, Shiri Gordon, Unsupervised Image
Clustering using the Information Bottleneck Method, The annual symposium for
Pattern Recognition of the DAGM02 , Zurich, September 2002.
·
S. Gordon, H. Greenspan, J. Goldberger, Applying
the Information Bottleneck Principle to Unsupervised Clustering of Discrete and
Continuous Image Representations, Oral presentation in the International
Conference on Computer Vision (ICCV-03),
·
S. Gordon,
J. Goldberger, H. Greenspan, An Efficient Image Similarity Measure Based on
Approximations of KL-Divergence Between Two Gaussian Mixtures, International
Conference on Computer Vision (ICCV-03), Nice, France, 2003.
Medical Image Analysis: Visualization, Segmentation, Quantification
New methodologies are being developed for image visualization as well as advanced segmentation and analysis algorithms for bio-medical applications. The general goal is to develop novel technology to support the diagnosis and the clinical practice in both augmenting the visual input presented to the human expert as well as by extracting objective quantitative measures from the data. In this summary we focus on brain image analysis, from the visualization to segmentation and lesion detection.
Superresolution
in MRI: Resolution
augmentation is important for visualization and early diagnosis.
In this research, MRI reconstruction using super-resolution is investigated. In
2-D multislice MRI, the resolution
in the slice direction is often lower than the in-plane resolution.
For certain diagnostic imaging applications, isotropic resolution
is necessary but true 3-D acquisition methods are not practical.
The novel contribution of this research is to show that if the imaging volume
is acquired two or more times,
with small spatial shifts between acquisitions combination
of the data sets using an iterative superresolution algorithm
gives improved resolution and better edge definition in the
slice-select direction. The method also improves
the signal-to-noise efficiency of the data acquisition. The superresolution framework is currently being extended to
additional modalities, such as MRA and CT. The research has been presented at
international conferences and has resulted in several publications (Greenspan
et al . Magnetic Resonance Imaging 2002)
.
Output following Superresolution
alg.
Figure 1: Phantom input. Output of MRI todate (left); (b) Superresolution output (right).
Image Segmentation: Automated segmentation of medical imagery is an important tool to support the radiologist needs and as a means for extracting desired quantitative measures. The research focus in this domain is to introduce statistical modeling tools for the segmentation process, for detection of lesions and the tracking in time. Statistical models include parametric as well as non-parametric models. We also focus on combining across region-based models and edge-based models.
A novel automated algorithm for tissue segmentation of noisy, low contrast magnetic resonance (MR) images of the brain was recently published at MICCAI and is being revised for publication in IEEE-TMI (Ruf et al, MICCAI 2005, Greenspan et al. IEEE_TMI in revision). A mixture model composed of a large number of Gaussians is used to represent the brain image. Each tissue is represented by a large number of Gaussian components to capture the complex tissue spatial layout. The intensity of a tissue is considered a global feature and is incorporated into the model through tying of all the related Gaussian parameters. The Expectation-Maximization (EM) algorithm is utilized to learn the parameter-tied, constrained Gaussian mixture model (CGMM). An elaborate initialization scheme is suggested to link the set of Gaussians per tissue type, such that each Gaussian in the set has similar intensity characteristics with minimal overlapping spatial supports. Segmentation of the brain image is achieved by the affiliation of each voxel to the component of the model that maximized the a-posteriori probability. The proposed framework combines global intensity modeling with localized spatial processing, within a coherent statistical model. The presented algorithm is used to segment 3D, T1-weighted, simulated and real MR images of the brain into three different tissues, under varying noise conditions. Results advance the state-of-the-art in the literature. An example is shown in Fig, 2. The presented framework provides an alternative for MRF based approaches. In addition, the algorithm is adaptive to the data at hand and does not require the use of an atlas for initialization or parameter learning. Registration processes are therefore not required and the applicability of the framework can be extended to cases in which no atlas exists, such as diseased brains and neonatal brains.
Future work entails
extending the framework to multi-modal analysis, incorporating bias correction
and handling multiple-sclerosis lesions.
Non-parametric modeling using Adaptive mean shift clustering (AMS) is an
alternative approach to the parametric models used, which we are experimenting
with. Here again, intensity is combined with spatial features. In Figure 3 we
show the automated segmentation result for a multi-modal slice consisting of
PD, T2, T1 and Fast Flair modalities. For comparison, we implemented the
classical GMM EM (intensity only) with Kmeans
clustering as initialization. The segmentation by GMM
Multiple
Sclerosis detection and analysis:
The application of a novel statistical
image-sequence (video) modeling scheme to sequences of multiple sclerosis (MS)
images taken over time was demonstrated in Shahar and
Greenspan, SPIE 04, ISBI 04. A unique key feature of the proposed framework is
the analysis of the image-sequence input as a single entity as opposed to a
sequence of separate frames. The extracted space-time regions allow for the
detection and identification of disease events and processes, such as the
appearance and progression of lesions. According
to the proposed methodology, coherent space-time regions in the feature space,
and corresponding coherent segments in the video content are extracted by
unsupervised clustering via Gaussian mixture modeling. The parameters of the
GMM are determined via the maximum likelihood principle and the
Expectation-Maximization algorithm. The clustering of the image sequence yields
a collection of regions (blobs) in a four-dimensional feature space (including
intensity, position (x,y), and time). Regions
corresponding to MS lesions are automatically identified based on criteria
regarding the mean intensity and the size variability over time. The proposed
methodology was applied to a registered sequence of 24 T2-weighted MR images
acquired from an MS patient over a period of approximately a year. Qualitative
and quantitative results were shown. An example result is presented in Figure 4.
Figure 2: Comparison of CGMM vs. state-of the-art algorithm for
segmentation of slice 95 from BrainWEB simulator with
9% noise level and no bias (a) Original image (b) state-of-the-art (c) CGMM
algorithm.
Figure 3. Four MRI modalities,
(a)Pd,(b)T2,(c)T1,(d)Fast Flair;
(e) GMM
EM segmentation (intensity only); (f) AMS segmentation.
Figure 4: Space-time
segmentation of lesions; each active lesion is marked by a different color.
Static lesions are in black.
·
H. Greenspan, G. Oz,
· Greenspan H, Oz G, Kiryati N, PeledS: MRI Inter-Slice Reconstruction Using Super Resolution. Magnetic Resonance Imaging 20: 437-446, 2002.
·
Ruf, J.
Goldberger, H. Greenspan. Tissue Classification of Noisy MR Brain Image Using
Constrained GMM. 8th International Conference on Medical Image
Computing and Computer Assisted Intervention (MICCAI-05),
· H. Greenspan, A. Ruf and J. Goldberger, “Tissue Classification of Noisy MR Brain Images Using Constrained GMM,” Submitted to IEEE Trans on Medical Imaging (IEEE-TMI) July 2005.
·
Allon Shahar and Hayit Greenspan, A Probabilistic framework for the detection and tracking in
time of multiple sclerosis lesions. IEEE International Symposium on
Biomedical Imaging (ISBI-04),
· H. Greenspan and A. Shahar, “Probabilistic Space-Time Modeling for the Detection and Tracking in Time of Active Lesions in MRI Images,” Submitted to Journal of Pattern Analysis and Applications, September 2004. In revision cycle.
Medical Image Analysis (cont):
Coronary Analysis Research
The field of Quantitative
Coronary Angiography (QCA) entails a wide variety of research problems, such as
geometrical modeling of vessels, stenosis computations,
and blood flow measurements. A major component of any coronary analysis tool is
the accurate estimation of the vessel centerline. Currently no standard tool is
available for extracting the centerline; moreover, testing of centerline
extraction accuracy in QCA algorithms is a very difficult task. In a major
focus of the research todate, a simulation tool was
developed in the lab to generate synthetic angiographic images of a single
coronary artery with predetermined centerline and diameter function. This
simulation tool was used to create a library of images for the objective
comparison and evaluation of QCA algorithms. The developed tool provides the
means for understanding the relationship between QCA algorithms’ performance
and vessel’s geometrical parameters. This research has matured into several
conference publications and a major journal publication. Further research in
the area includes the evaluation of the effect of vessel geometry as well as stenosis geometry on the flow within the vessel.
H.
Greenspan, M. Laifenfeld, S. Einav
and O. Barnea, ``Center-line
Extraction in Quantitative Coronary Angiography as a Function of the Coronary
Geometrical Complexity," in IEEE Transactions on Medical Imaging IEEE-TMI
20(9): 928-941, 2001
Automatic Identification of Bacterial Types using
Statistical Imaging Methods
Automated tools are developed in this work to identify microbiological data types using computer-vision and statistical modeling techniques. Bacteriophage (phage)-typing methods are used to identify and extract representative profiles of bacterial types, such as the Staphylococcus Aureus. Current systems rely on the subjective reading of profiles by a human expert. This process is time-consuming and prone to errors, especially as technology is enabling the increase in the number of phages used for typing. The statistical methodology presented in this work, provides for an automated, objective and robust analysis of visual data. The statistical framework enables the signature extraction (or profiling) of visual data, along with the ability to cope with increasing data volumes. It is applicable in many additional related domains, such as microarray data.
S. Trattner, H. Greenspan, G. Tepper
and
Bacterial
Types using Statistical Imaging Methods," IEEE Trans on Medical Imaging
(IEEE-TMI),
Vol 23, No. 7, pp. 807-820, July 2004.
Doppler Echocardiography Flow-Velocity Image Analysis
for Patients with Atrial
Fibrillation
The
aim of this study was to develop an automated method for Doppler analysis based
on image processing and computer vision algorithms. Collaboration
with Prof. Feinberg, head of non-invasive cardiac clinic, at Tel-hashomer hospital. Currently, Doppler echocardiography analysis
is performed manually. An automated method that analyzes the Doppler signal can
potentially improve accuracy and result in a powerful tool for noninvasive
evaluation of cardiac hemodynamics, especially for
patients with atrial fibrillation (AF) where multiple
samples are needed to obtain an accurate averaged measurement.
Images
were obtained from the mitral valve (MV) and the
tricuspid valve (TV) Doppler tracings from 45 patients; 20 with normal sinus
rhythm and 25 with atrial fibrillation. The proposed
algorithm automatically detects the maximal velocity envelope (MVE) of the
spectral Doppler ultrasound tracings. Averaged values for the time velocity
integral, peak mitral inflow velocity and peak
tricuspid regurgitation velocity are calculated for multiple beats available in
a single screen frame. Measurements extracted automatically from the MVE were
compared to measurements obtained manually by 2 expert technicians. High linear
correlation (r) was found between the automatically and the manually extracted
parameters (
). A smaller variation was found in most cases between the
manually calculated average beat and the automated average beat (bias value
between 3.8% and 5.2%) than between the manually calculated average beat and
the selection of a representative beat (bias value between 6.2% and -2.6%). The newly developed automated algorithm offers a new
accurate and reliable clinical tool particularly for the assessment of patients
with irregular heart rate.
H. Greenspan, O. Schechner,
M. Feinberg and M. Sheinovitz, “Automatic method forEchocardiography Analysis in Patients with Atrial Fibrillation”, Journal of Ultrasound in Medicine and
Biology, 2005
Medical Image Indexing and Retrieval
In addition to developing new CBIR tools, a major focus is on testing their applicability to medical image archives, as one of the pioneer groups in this domain.
There is an increasing trend towards the digitization of medical imagery
and the formation of adequate archives, as medical image databases are becoming
a key component in diagnosis and preventive medicine. The resulting picture
archiving and communication systems (PACS) are available across wards within a
hospital setting and allow global access to shared resources. Visual-based
(i.e. content-based) indexing and retrieval based on the information contained
in medical images is expected to have a great impact on medical image
databases. The unique characteristics of medical images hinder the direct
adaptation of content-based retrieval approaches, which are already in use for
unstructured collections of images. Novel methodologies thus need to be
developed. A few research groups in the world are focused on this domain of
research, including our group.
Several works have matured in this domain. From the analysis of white
blood cells (collaboration with Prof. Z. Malik of
Bar-Ilan University), thru x-ray archive
categorization (collaboration with Prof. Lehmann at Aachen University), and finally in a major collaboration
with NIH, National Library of Medicine (NLM) and National Cancer Institute
(NCI) - on analyzing images of the uterine cervix for supporting research and
detection of cancer (the 2nd most deadly cancer for women
worldwide).
Advanced Image Processing and Retrieval Systems for Pathology and
Histology
In a particular project in this field, spectral imaging and microscopy were combined for the representation of white blood cells, towards the goal of detecting and classifying Leukemia.
Computational
analysis of the spectral maps allows for the objective quantification of a set
of parameters, or features, representing the cell. The features used in this
work include the area and perimeter of the nucleus, circularity, edginess and
the spectral pattern. The analysis pursued showed that each class of cells is associated
with a set of unique parameters. We conclude that spectral analysis combined
with feature analysis provides significant information in the analysis of lymphoproliferative disorders and may serve as an
additional tool for the histopathological evaluation
of disease (Greenspan et al, Journal of Histology and Histopathology, 2002).
Medical
Image Categorization and Retrieval for PACS Using the GMM-KL Framework
This work presents
an image representation and matching framework for image categorization in
medical image archives. Categorization enables to determine automatically,
based on the image content, the examined body region and imaging modality. It
is a basic step in content-based image retrieval (CBIR) systems, the goal of
which is to augment text-based search with visual information analysis. CBIR
systems are currently being integrated with Picture-Archiving and Communication
Systems (PACS) for increasing the overall search capabilities and tools
available to radiologists. The GMM-KL framework is used for matching and
categorizing x-ray images by body regions. A multi-dimensional feature space is
used to represent the x-ray image input, including intensity, texture and
spatial information. Unsupervised clustering
via the GMM is used to extract coherent regions (“blobs”) in feature space
which are then used in the matching process. A dominant characteristic of the
radiological images is their poor contrast and large intensity variations. This
presents a challenge to matching between images and is handled via an
illumination invariant representation. The GMM-KL framework is evaluated for
image categorization and image retrieval on a dataset of 1500 radiological
images. A classification rate of 97.5% was achieved. The classification results
compare favorably with reported global and local representation schemes.
Precision vs. Recall curves indicate a strong retrieval result as compared with
other state-of-the-art retrieval techniques. Finally, category models are
learned and results are presented for comparing images to learned category
models (Pinhas and Greenspan, SPIE 2003, Lehmann et al, 2003, Greenspan and Pinhas,
Submitted IEEE Transactions on
Information Technology in BioMedicine, January 2005).
Image
Content Indexing of Uterine Cervix Images
The National
Cancer Institute (NCI) has collected a large database of digitized 35mm slides
of the uterine cervix, the idea being to build a system enabling to study the
evolution of lesions related to cervical cancer. In taking the first few steps
towards this goal, the objective of our research is to develop and evaluate
methodologies required for visual-based (i.e. content-based) indexing and
retrieval that substantially improve information management of such a database.
Methods: Using theoretical contributions from image
processing and visual data analysis, the images are first preprocessed to focus
on the cervix region within the image, remove artifacts and noise and correct uneven
illumination. Image segmentation
targeted on the identification of various landmarks and tissues within the
cervix is then performed. Important tissue features are extracted and used for
the image indexing. Image retrieval can
then be automatically performed, based on the image content, using the indexed
representations and appropriate distance measures. On the algorithmic front,
methodologies for image segmentation, indexing and retrieval will be developed,
headed by the group at
Significance: An important contribution will be made in
introducing novel algorithms for the uterine cervix image analysis and
indexing. The combination of expertise in the medical CBIR domains, with
medical experts in cervicography, who cooperate and
combine their know-how in the proposed research, along with the availability of
the large database of cervical images, is unique and stands in the cutting edge
of medical imaging research. In particular, the computerized system capabilities (such as image landmark
identification and indexing) will serve as a tool for cervical cancer
investigation as part of the on-going research conducted by NCI to validate
existing cervicographic technology for cervical
cancer screening.
The
developed tools can benefit the general field of computer-assisted diagnosis.
The medical archive generated as part of the research can be used as a large
test bed for research and development in both medical and computer vision
communities.
Publications: Initial work has been published in
international conferences (CBMS 2004, SPIE 2006). A journal paper is in
preparation.
Journal Contributions in Medical image indexing and
retrieval:
·
H. Greenspan, C. Rothmann, T. Cycowitz, Y. Nissan, A. Cohen and Z. Malik,
Classification of Lymphoproliferative Disorders by
Spectral Imaging of the Nucleus", Journal of Histology and
Histopathology, Vol. 17, No. 3, pp. 767-73, 2002.
·
H. Greenspan and A. Pinhas, “Medical Image
Categorization and Retrieval for PACS using the GMM-KL framework,” Submitted to
IEEE Transactions on Information Technology in BioMedicine,
January 2005. In revision cycle.
Related Conferences
·
Pinhas A, Greenspan H: A continuous and probabilistic framework for medical
image representation and categorization.
Proceedings of SPIE International Symposium on Medical Imaging,
·
Lehmann TM, Wein BB, Greenspan H: Integration of
Content-based Image Retrieval to Picture Archiving and Communication Systems. Procs Medical Informatics
·
Shiri Gordon, Gali Zimmerman and Hayit
Greenspan, Image Segmentation of Uterine Cervix Images for Indexing in PACS,
Proceedings of the Seventeenth IEEE Symposium on Computer-Based Medical
Systems, pp. 298-303,
·
G. Zimmerman, S. Gordon and H.
Greenspan. "Content-based indexing and retrieval of uterine cervix
images". In Proc. of 23rd IEEE
Convention of Electrical and Electronics Engineers in Israel 2004, 181-185,
Tel-Aviv, Israel, 2004.
·
S. Gordon , G.
Zimmerman, R. Long and J. Jeronimo, H. Greenspan. "Content Analysis of Uterine Cervix Images: Initial steps towards Content
Based Indexing and Retrieval of Cervigrams". To appear in Proc. of SPIE medical imaging, 2006.