by Michael D. Doyle, Cheong Ang, Rakesh Raju, *Gary Klein, **Betsey S. Williams, Thomas DeFanti, Ardeshir Goshtasby, ***Robert Grzesczuk and ****Adrianne Noe
The University of Illinois at Chicago, *Amoco Oil Co., **Harvard Medical School, ***the University of Chicago, and ****the National Museum of Health and Medicine, AFIP
Abstract
A project is described (The Visible Embryo Project) to develop software strategies for the creation of large-scale databases of 3-dimensional image data on human developmental anatomy. The issues discussed relate to the processing of serial-section image data for the purpose of reconstructing volumetric models of individual embryos from museum specimens. Attempts were made to fully automate the processes of image registration and artifact correction, in order to allow for the eventual unattended reconstruction of thousands of such embryos from such resources as the Carnegie Collection of Human Embryology. Tools were also developed to allow the real-time interactive visualization of the massive databases that this type of reconstruction project creates. The implementation of a 3-D model of a human embryo in a viewer-centric virtual reality environment is also described.
Introduction
Congenital malformations have challenged the humanistic and intellectual resources of mankind for ages. Generations of scientists have labored to understand their causes and effect their treatments. Despite this legacy, there are a variety of problems currently plaguing the discipline of human embryology. Although many remarkable collections of human embryological materials have been built up over the last century, the nature of these materials dictates that they be housed in museums under tight lock and key. The wealth of knowledge represented by such archives is largely accessible only through the publications of earlier researchers who studied various aspects of the collections. New methods for computer-based 3-dimensional morphometric analysis are enabling developmental biologists to study the dynamics of human development in ways that were not possible until only a few years ago. The publications of early researchers in the field are of little value to these new quantitative biologists. Yet these researchers have been severely limited by a lack of free access to large populations of embryological specimens. New technologies in high performance computing and communications are beginning to enable museums and research centers to open up their collections, in electronic form, to researchers, educators and students throughout the nation without endangering the fragile specimens themselves. The presently-described research is part of an effort to develop the technologies necessary for such a "museum without walls" to take form.
The Center for Human Developmental Anatomy
For over a century, scholars have perceived a need for an internationally accessible center for research in human developmental anatomy. Consistent with its missions to support consultation, education, and research in medicine and related disciplines, the staff at the United States Armed Forces Institute of Pathology began in 1990 to develop such a center at its National Museum of Health and Medicine. Its long-time experience in both conducting and supporting collections-based scientific research have allowed the staff and associates to identify prospectively the collections that will serve critical medical and scientific needs. Beginning with the acquisition of the world-renowned Carnegie Human Embryology Collection (O'Rahilly, 1987), the Institute continues to identify and collect well-documented materials and carefully houses them for perpetual availability to scholars. The Institute's mission is to preserve these and other materials relating to developmental anatomy in a careful, organized manner, which will eventually include extensive computer-based image archiving and the development of large-scale,
remotely accessible, database on the Center's collections. The Center relies on supported research agendas to activate these materials and make them useful around the globe.
The Visible Embryo Project
A multi-institutional, multidisciplinary research project (the Visible Embryo Project) is being led by the Developmental Anatomy Center and the Biomedical Visualization Laboratory at the University of Illinois at Chicago to develop software strategies for the development of distributed biostructural databases using current technologies for high-performance computing and communications (HPCC), and to implement these tools in the creation of a large-scale digital archive of multidimensional data on normal and abnormal human development (Doyle, et al., 1992).. This project relates to BVL's long-term activity in the areas of health informatics, educational multimedia, and biomedical imaging science (Doyle, et al., 1990-92, Carlbom, et al., 1992).
Serial-section reconstruction has long been a tool used by morphologists, and especially embryologists, to study the three-dimensional structure of microscopic anatomy. The early workers typically hand traced the contours of microscopic sections projected through a camera lucida, cut out the contours from pieces of cardboard of a thickness corresponding to the scale of the reconstruction, and carefully glued them together, in register, to form a model of the 3-D surface of the structure of interest (O'Rahilly,1987).
Several workers have attempted to extend this approach into computer environments by tracing the contours of consecutive sections on a digitizing tablet, and then using computer graphics to render the 3D surface described by the stacked contours (Prothero,1986, Wilkinson,1990). Two embryos from the Blechschmidt collection have been reconstructed in this manner and visualized using a polygonal surface-based reconstruction approach (personal communication Cornelius Rosse). In the mouse, Lawson of the Hubrecht Laboratory, has attempted reconstruction from plastic histologic sections. These two approaches have in common the limitation that only the surface of the structure can be studied from a given model. If a reconstruction of the exterior of the heart were performed, for example, entirely separate reconstructions would have to be performed in order to visualize the chambers, or the myocardial vascular patterns.
Most of the early work in the Visible Embryo project has involved using serial section reconstruction from microscope slides, however we extracted volumetric data from these specimens, rather than just surface data. Sets of serial microscopic cross-sections through human embryos (prepared between the 1890s and 1970s) are being used as sample image data around which to design and implement various components of the system. These images are digitized and processed to create both 3D voxel datasets and polygonal models representing embryonic anatomy. Standard techniques for 3D volume visualization could then be applied to these data (Carlbom 1992, Doyle 1989-92, Karssemeijer 1988, Leichtman 1990, Levoy 1990, Lorenson 1987, McClean 1991, Prothero 1986, Terzopoulos 1987, Udupa 1990, Wilkinson 1990, Yuille 1986). Processing of these artifacts was required to correct for certain artifacts that were found in the original microscope sections from routine histological techniques of tissue preparation.
This processing includes, as will be discussed below, registration of adjacent serial sections, specialized methods of histogram equalization to correct for staining irregularities, and various levels of conformal warping to correct for dimensional instability (stretching, etc.) in the embedding material. Predominantly-automatic software tools have been developed by the UIC BVL team to accomplish these tasks, thereby enabling the future processing of huge numbers of images to create large-scale datasets describing human development through the first trimester. Remote-access visualization and database tools are under development to allow real-time interaction with these enormous datasets by distributing certain computational tasks to super computers.
Materials and Methods
A human embryo specimen was acquired, on loan, from the collection of the Human Developmental Anatomy Center. This specimens comprised a set of serial transverse sections through a 7-week-old embryo. The sectioned specimen was in the form of paraffin embedded 10mm-thick sections stained with haemotoxylin and eosin and mounted on glass slides with coverslips. Figure 1 shows a stereo pair image of a human embryo of similar age.
Figure 1: Stereo-pair image of a 7 to 8 week old human embryo from the Carnegie
Collection of Human Embryology.
The microscopic sections through the embryo specimen were transilluminated with a voltage-stabilized dichroic light source. Video images were created with a Sony XC-77 CCD monochrome video camera through a Micro Nikkor 105mm macro lens with extension tubes, and then digitized with a Jovian Logic frame grabber at a spatial resolution of 640x480 pixels with 256 gray levels.
Images were stored in the TIF format with LZW (lossless) compression on 3.5" magneto-optical disks. A total of 1297 images were thereby obtained of the embryo, with each image corresponding to an individual physical section through the embryo. The magnification of the imaging system was such that the digitized images were made up of square pixels with a side length of 9mm.
Image Processing
The serial cross-section images were first processed in order to isolate the embryonic tissue from the surrounding area. This was accomplished through the use of the automatic edge-tracing feature of Image Pro Plus from Media Cybernetics Corp. Image Pro Plus also allowed the x,y coordinates of the external contour of each embryo section to be saved to an ASCII file. This proved to be very useful in the later stage of the project devoted to creating a surface-based polygonal model of the embryo exterior.
Registration of Images
Since the embryo specimen had been embedded and sectioned in the 1930's, fiducial markers were not included in the embedment to assist in the computer reconstruction of the embryo. It was therefore necessary to process the image data in order to restore topological relationships between tissue elements, and to restore dimensional accuracy in the direction perpendicular to the plane of section. We were therefore forced to "reverse engineer" proper image registration based solely upon intrinsic image features. This was accomplished through the creation of an algorithm by one of us (Klein) that used an error minimization technique he calls "iterative chasing." This technique attempts to find the "best" difference result between two adjacent images in a series, such that the following value is minimized:
H(x,y) = SS |r(x,y) - R(x,y)|
xy
where r(x,y) represents the first image (the reference image) and R(x,y) represents the second image (the mobile image) in a sequential pair. The program iteratively performs transformations in terms of x and y translations and rotations of the mobile image, while holding the position of the reference image stationary. The assumption is made that the closer one gets to the optimal transformation, the lower will be the value H. The 3-dimensional matrix of h (x translation), k (y translation), and q (rotation), one dimension at a time. First, the best h transformation is found while k and q are kept at zero. Then the best k transformation is sought while h = h-best and q=0. A third set of transformations are then done to find q while h = h-best and k = k-best. This sequence is then repeated continuously until H(x,y) does not decrease any more. We then use h-best, k-best and q-best as the coefficients of transformation for the mobile image. The program then saves the transformed mobile image to disk, replaces the image in the reference buffer with the mobile image, and then loads the next sequential serial-section image into the mobile image buffer. The registration procedure then begins again. This process is repeated automatically until all of the images of sequential cross-sections are processed and saved to disk.
Once proper registration of adjacent images was restored to the embryo data, these data were loaded into a 3 dimensional array representing voxels where each voxel had the dimensions 9mm x 9mm (the pixel size) x 10 mm (the standard section thickness). Standard techniques for volume visualization could then be used to examine the data.
Figure 3 shows a para-saggital section through the 3-D embryo dataset in the region of the thorax, at a plane perpendicular to the original plane of sectioning. Two important categories of image artifacts can be seen in this image, both a result of the original histological techniques used when the embryo was sectioned. First, although the heart (arrow) looks reasonably-well reconstructed, a closer examination shows a horizontal banding resulting from both staining irregularities and dimensional instability (shrinkage variations) among the set of serial cross sections. This can be seen throughout the sagittal image, but is especially evident in the area of the liver, which is immediately below the heart. There is a strong periodicity to this effect which makes the image appear as if it was made up of a set of "slabs" which were found to correspond to roughly 12 cross-sections each in the original data. The source of the periodicity of this effect may be clearer when one considers the routine techniques that were probably used by the 1930's era histology technician that prepared the original specimen.
Figure 2: Parasagittal section through the thorax region of the auto-registered data.
The arrow points toward the heart area. The liver can be seen below the heart.
It can be assumed that the embryo was first dehydrated with a graded alcohol series and then embedded in paraffin before sectioning began. The sectioning occurred on a guillotine-style microtome that made precise 10 mm-thick sections perpendicular to the long axis of the embryo. As the sections were cut, they formed ribbons that floated out onto a "boat" filled with water. They were then lifted out of the water directly onto a glass microscope slide and placed on a slide-warmer to dry. Once a number of slides were prepared in this way, they were probably then placed, end up, into a typical "staining rack" before being
immersed in a series of solutions for staining of the important structures. They were then probably left to dry in the staining rack and then removed from the rack to be examined or stored.
When we examined the positions of the sections on the microscope slides, we found that they were placed on the glass in a rectangular grid with 4 to 5 rows of 12 section ribbons. If these slides had been dried in a vertical position after staining, then the sections at one end of a row would have dried faster than those at the other end of the row. The sections that dried faster may have shrunk at a different rate that those that spent more time being wet. In addition some rows of sections may have been exposed to stain for a longer time than others, and this would result in those rows staining darker than the others. Therefore, the handling of the specimens during histological processing may account for the banding artifacts in the original dataset. This is supported by the fact that the 12-section thickness of the bands correlates with the 12 sections per row found on the glass slides.
Staining irregularities were corrected for by histogram equalization. Optimal histograms were calculated for every 26th image among the 1300 images from the embryo; 50 such histograms in all. Histogram equalization curves were saved to disk for each of these optimal histograms. Each of the equalization curves were then used for the correction of the following 26 images. Each of those images were processed so that their histogram matched the optimal histogram for that 26 image segment. This produced an image dataset with minimized banding and enhanced contrast, yet in a way that was sensitive to the topographical variations in image character throughout the embryo. Dimensional instability was corrected for through a the use of an algorithm (Goshtasby,1986) that conformally-warped the images based upon a pixel-to-pixel comparison between adjacent images, and that was "aware" of the periodicity of the shrinkage artifact measured in the original data.
It was found that additional artifacts were introduced into the data by the registration program itself. Due to the nature of the registration algorithm, it tended to "straighten out" any global curvatures or twists that existed in the original embryo. These "global artifacts" were partially corrected by interactively defining a "correction spline" in a direction parallel to the cranial-caudal axis of the body. The correction spline was then used as a guide for a smooth progression of x and y translations that restored a natural-appearing curve to the embryo.
3-D Visualization Tools
Software tools were developed to allow the interactive three-dimensional visualization of the embryo reconstruction in real time. Figure 3 shows the display of the application as it appeared at the
Figure 3: Screen image of the 3-D visualization program demonstrated at SIGGRAPH '92, in Chicago. This application was distributed between a Silicon Graphics Crimson VGXT workstation and a Convex supercomputer.
SIGGRAPH '92 conference in Chicago (Doyle, et al., 1992). The left of the screen shows a surface-based
model of the embryo's exterior. This model was built from data which was derived, through three-dimensional interpolation, from the original embryo dataset . Two-hundred volume slices of the embryo (stored as texture maps) can be interactively displayed at this lower resolution while the model is rotated freely in three dimensions. A cutting-plane can be seen to intersect the surface-based model. This cutting plane can be interactively controlled to intersect with the embryo model at any arbitrary angle and position. To the right of the screen, one can see a window that displays a high-resolution image of the oblique section through the embryo as indicated by the interactive cutting plane. In order to maintain the quick response needed for effective real-time interaction, the computational load of this application was distributed so that the interface panel, seen at the bottom of the screen, and the 3-D surface model were running on the CPU of the Silicon Graphics workstation. Computation of the high-resolution oblique section image displayed in the right window took place on the Convex supercomputer. Both of these operations occurred simultaneously, communicating through a high-speed fiber optic network.
Current efforts are being directed towards the development of a very portable tool for viewing arbitrary oblique slices through such data. This program allows interactive display of orthogonal and oblique slices through volumetric data without using any machine-specific functions. The application is written in pure ANSI-standard C and uses the X Window (Motif) toolkit for its interface. It has already been successfully tested on workstations from Silicon Graphics, Sun, and IBM.
Virtual Reality
The surface-based embryo-model described above was also implemented within a virtual reality environment, called "The Cave," at the 1992 SIGGRAPH conference. The Cave was a 10' x 10' room made up of back-projection screens upon which were projected stereo views of three dimensional data that a viewer, wearing LCD stereo glasses and a 3-D tracking device, could move around in as the system tracked his or her motion. One could walk around the data and receive the sensation that one was actually "in" the computer graphic environment. This system was also shown at the Supercomputing '92 conference and the 1992 meeting of the Radiological Society of North America.
Summary
The work described here represents the very earliest stages of the Visible Embryo project. Its long term goals are simply to create a comprehensive multimedia database that documents and describes both normal and abnormal human morphological development in multiple dimensions. This is in all respects a long term endeavor. Yet even the earliest stages of progress towards this goal will be of importance to a variety of investigators, educators and students of embryology and many ancillary fields. Among the obvious areas that will benefit from this work are medical education, cell and developmental biology, biomedical informatics, biomedical imaging science and high performance computing. Perhaps the most important benefit will be that it will form a model for similar projects in other fields ranging from the life sciences to the humanities.
Bibliography
1) Atlas, S.W., Braffman, B.H., LoBrutto, R., Elder, D.E. and Herlyn, D. Human malignant melanomas with varying degrees of melanin content in nude mice: MR imaging, histopathology, and electron paramagnetic resonance; J. Comp. Asst. Tomog. 14/4: 547-554 (1990)
.
2) Carlbom, I., Hsu, W.M., Klinker, G., Szelski, R., Waters, K., Doyle, M.D., Gettys, J., Harris, K.M., Levergood, T.M., Palmer, R., Palmer, L., Picart, M., Terzopoulos, D., Tonnessen, D., Vannier, M., and Wallace, G. Modeling and Analysis of Empiracle Data in Collaborative Environments, Communications of the ACM, 33/6: 75-84, (1992).
3) Cersen, S., Lotscher, H.R., Kummecke, B. and Seelig, J. Monoclonal antibody-coated magnetite particles as contrast agents in magnetic resonance imaging of tumors Magnetic Resonance in Medicine 12:151-163 (1989).
4) Doyle, M.D., Raju,R., Ang,C., Goshtasby,A., and DeFanti, T. The Virtual Embryo: real-time viewer-centric 4-D visualization of human embryonic anatomy, presented at the High Performance Computing and Communication Showcase, SIGGRAPH '92, the annual meeting of the Association for Computing Machinery's Special Interest Group for Graphics, Chicago. (August 1992).
5) Doyle, M.D., Raju, R., Ang, C., Klein,G., Goshtasby, A., DeFanti, T., and Noe, A.: The Visible Embryo distributed (workstation/supercomputer) interactive visualization of high-density 3D image data of embryonic anatomy, presented at the High Performance Computing and Communication
Showcase, SIGGRAPH '92, the annual meeting of the Association for Computing Machinery's Special Interest Group for Graphics, Chicago. (August,1992).
6) Doyle, M.D. and Sadler, L.L. Health Informatics: MetaMap indexing and retrieval of image-based data for an institutional PACS system, presented at the 1991 Forum in Cellular and Organ Biology, American Association of Anatomists, Chicago, (April 1991).
7) Doyle, M.D. Palette Segmentation Indexing: The MetaMap Process, SIGBIO Newsletter (The Journal of the ACM Special Interest Group for Biological Computing), In Press (1992).
8) Doyle, M.D. The MetaMap Process: A New Approach to the Creation of Object-oriented Image Databases for Medical Education, Proceedings of the 13th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, IEEE Press (1991).
9) Doyle, M.D., Raju, R., Ang, C., Klein, G., Goshtasby, A.,, and DeFanti, T. Multidimensional reconstruction of human embryonic anatomy from museum-archived serial section collections, to be submitted to Medical Imaging (1993).
10) Doyle, M.D., Williams,B.S., Noe, A. The Visible Embryo Project: high performance computing, vizualization, and .communications for developmental anatomy research, to be submitted to Anatomical Record (1992)
11) Goshtasby, A.: A region based approach to digital image registration with subpixel accuracy, IEEE Transactions on Geoscience and Remote Sensing, GE-24, V03, May (1986)
12) Karssemeijer, N. Three-dimensional stochastic organ-models for segmentation in CT-scans SPIE 1030: 177-184 (1988).
13) Leichtman, G.S. and Anderson, D.J. Linking a relational database of biological features to computer-aided reconstruction of tissue. Proc. 1st Conf. Visualization in Biomed. Computing. IEEE Computer Society Press 288-294 (1990).
14) Levoy, M. A hybrid ray tracer for rendering polygon and volume data IEEE Comp. Graphics and Applic. 33-40 (1990).
15) Lorensen, W.E. Marching cubes: A high resolution 3-d surface construction algorithm. ACM SIGGRAPH 21/4: 38-44 (1987).
16) McLean, M. and Prothero, J.W. Three-dimensional reconstruction from serial sections. V. Calibration of dimensional changes incurred during tissue preparation and data processing. Anal. and Quant. Cytol. and Hist. 13: 269-278 (1991).
17) O"Rahilly, R. and Muller, F. Developmental stages in human embryos, Carnegie Institution of Washington, Publication no. 637. Carnegie Institute, Washington, DC. (1987).
18) Prothero, J.S. and Prothero, J.W. Three-dimensional reconstruction from serial sections. IV: The reassembly problem. Computers and Biomed. Res., 19: 361-373 (1986).
19) Terzopoulos, D. On matching deformable models to images. Topical Meeting on Machine Vision, Technical Digest Series (Optical Society of America, Washington, DC.) 12: 160-163 (1987).
20) Udupa, J.K. Course notes: Visualization of biomedical data: principles and algorithms. 1st Conf. on Visualization in Biomed. Computing. IEEE Computer Society Press, p14-16 (1990).
21) Wilkinson, D.G. and Green, J. In situ hybridization and the three-dimensional reconstruction of serial sections. IN: Copp, A.J. and Cockroft, D.L. eds. Postimplantation Mammalian Embryos: A Practical Approach. IRL Press, Oxford University. pp155-171(1990).
22) Yuille, A.L., Cohen, D.S., and Hallinan, P.W. Feature extraction from faces using deformable templates. Harvard Robotics Lab. Tech. Rep. 88/2: 104-109 (1988).
