Principles and Practices of Robust, Photography-based Digital Imaging Techniques for Museums Mark Mudge, Carla Schroer, Graeme Earl, Kirk Martinez, Hembo Pagi, Corey Toler-Franklin, Szymon Rusinkiewicz, Gianpaolo Palma, Melvin Wachowiak, Michael Ashley, Neffra Matthews, Tommy Noble, Matteo Dellepiane Abstract This full day tutorial will use lectures and demonstrations from leading researchers and museum practitioners to present the principles and practices for robust photography-based digital techniques in museum contexts. The tutorial will present many examples of existing and cutting-edge uses of photography-based imaging including Reflectance Transformation Imaging (RTI), Algorithmic Rendering (AR), camera calibration, and methods of imaged-based generation of textured 3D geometry. The tutorial will also explore a framework for Leading museums are now adopting the more mature members of this family of robust digital imaging practices. These practices are part of the emerging science known as Computational Photography (CP). The imaging family’s common feature is the purpose-driven selective extraction of information from sequences of standard digital photographs. The information is extracted from the photographic sequences by computer algorithms. The extracted information is then integrated into a new digital representations containing knowledge not present in the original photogs, examined either alone or sequentially. The tutorial will examine strategies that promote widespread museum adoption of empirical acquisition technologies, generate scientifically reliable digital representations that are ‘born archival’, assist this knowledge’s long-term digital preservation, enable its future reuse for novel purposes, aid the physical conservation of the digitally represented museum materials, and enable public access and research. Keywords: Reflectance transformation imaging, empirical provenance, photogrammetry, non-photorealistic rendering, digital preservation, cultural heritage The 11th International Symposium on Virtual reality, Archaeology and Cultural Heritage VAST (2010) pp A. Artusi, M. Joly-Parvex,G. Lucet, A. Ribes, and D. Pitzalis (Editors) 1. Tutorial Overview Today, leading museums are adopting a new family of robust digital imaging practices. This imaging family’s common feature is the purpose-driven selective extraction of information from a sequence of standard digital photographs. The information extracted from the photographic sequence is selected by computer algorithms. The extracted information is then integrated into a new digital representation containing knowledge not present in the original photographs, examined either alone or sequentially. These practices are part of the emerging science known as Computational Photography. The algorithms are embedded in software tools that keep the computer science ‘under the hood’ and allow the user to ‘drive’ the tools in service of their customary working culture. No ongoing assitance from outside digital imaging technologists is necessary. The imaging family is able to process the information from the photographs with only minor user involvement. This highly automatic operation permits the writing of a scientific ‘lab notebook’ chronicling each of the means and circumstances of the new digital representation’s generation. This machine readable log permits qualitative evaluation of the representation’s reliability and suitability for its original and potential novel purposes both now and in the future. Following international metadata standards, the lab notebook, bundled with the original photographs and the newly generated representations form a ‘born archival’ package ready for ingest into the world’ knowledge base and the museum-library-archive long-term digital preservation environment. The following presentations describe the practices of Reflectance Transformation Imaging, Algorithmic Rendering, dense, close range Photogrammetry, semantic knowledge management, long term digital preservation, and the application of these tools within museums and cultural heritage environments. 1.1 Sequence of Presentations Mark Mudge and Carla Schroer from Cultural Heritage Imaging will present an overview of the themes uniting the tutorial’s presentations. They will explore issues that influence technology adoption decisions and the advantages that can be realized when image-based empirical information acquisition is organized in conformance with the fundamental principles of science. They will also present a unified photographic data capture strategy that acquires all the information necessary to enable Reflectance Transformation Imaging, Algorithmic Rendering and Photogrammetry. M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums Graeme Earl, Kirk Martinez, and Hembo Pagi from Southampton University will provide a summary of their uses of reflectance transformation imaging in archaeological contexts. they will also introduce the UK Arts and Humanities Research Council funded Reflectance Transformation Imaging (RTI) System for Ancient Documentary Artefacts project. The AHRC RTI project is a collaboration with Alan Bowman, Charles Crowther and Jacob Dahl at the University of Oxford. Corey Toler-Franklin and Szymon Rusinkiewicz from Princeton University will discuss Algorithmic Rendering (AR). Their AR work takes photographic image sequences containing reflective spheres, such as the RTI data set, and generates RGBN images with per-pixel color and surface shape information, in the form of surface normals. These RGBN images are powerful tools for documenting complex real-world objects because they are easy to capture at a high resolution, and readily extendible to processing tools originally developed for full 3D models. Most state-of-the- art nonphotorealistic rendering algorithms are simply functions of the surface normal, lighting and viewing directions. Simple extensions to signal processing tools can preserve the integrity of the normals, while introducing a wide range of control for a variety of stylistic effects. RGBN images are more efficient to process than full 3D geometry, requiring less storage and computation time. Functions are computed in image space producing powerful 3D results with simpler 2D methods. Gianpaolo Palma from the Visual Computing Lab, from the Italian National Research Council’s (CNR) Institute for Information Science and Technology (ISTI) will present two tools to visualize and analyze RTI images in an interactive way. The first one is a multi-platform viewer, RTIViewer, developed also to work remotely through HTTP, that allows the user to apply a set of new shading enhancement techniques improving the virtual examination and interpretation of details of the artifact. The second is a web application based on SpiderGL [DBPGS10], a JavaScript 3D graphics library which relies on WebGL, which permits the realtime rendering of huge RTIs with a multi-resolution encoding in the next generation of web browser. Mel Wachowiak from the Smithsonian Institution’s Museum Conservation Institute (MCI) will describe some museum uses of RTI and its place among photographic capture and 3D scanning at the Smithsonian Institution (SI). MCI has a central role as a research unit and collaborator in analysis of heritage objects and sites. MCI’s part in the digitization of collections is to offer an expanded vision of the application of appropriate technologies. He will show how RTI fills a niche that other imaging solutions can’t fill by offering an immersive, near 3D experience and image processing tools as well as accurately document features that are impossible to acquire with 3D scanning. He will also show a broad range of RTI projects. These have ranged in size and scope from tiny natural history specimens to large artworks, both in the studio and on location. Buttons, jewelry, fossils, prehistoric stone tools and many other materials will demonstrate the strengths and weaknesses of the current RTI technology and software. Michael Ashley from Cultural Heritage Imaging will discuss and demonstrate practical digital preservation frameworks that protect images throughout the entire production life-cycle. Using off the shelf and open source software coupled with a basic understanding of metadata, he will show it is possible to produce and manage high value digital representations of physical objects that are born archive-ready and long-term sustainable. He will also demystify the alphabet soup of file formats, data standards, and parametric imaging, and demonstrate proven workflows that can be deployed in any museum production environment, scalable from the individual part time shooter to full fledged imaging departments. Neffra Matthews and Tommy Noble from the U.S. Department of the Interior, Bureau of Land Management’s, National Operations Center will present the principles of photogrammetry, deriving measurements from photographs. They will demonstrate that by following the photogrammetric fundamentals, mathematically sound and highly accurate textured 3D geometric results may be achieved. they will also show how technological advances in digital cameras, computer processors, and computational techniques, such as sub-pixel image matching, make photogrammetry an even more portable and powerful tool. Extremely dense and accurate 3D surface data can be created with a limited number of photos, equipment, and image capture time. Matteo Dellepiane from the Visual Computing Lab of the Italian National Research Council’s (CNR) Institute for Information Science and Technology (ISTI) will present 2 applications. The first is an alternate method for generating textured 3D geometry for interpretive purposes using the ARC3D web service. The Arc3D web service, inputs user uploaded uncalibrated photographic sequences to generate and then return the 3D model. The second application, Meshlab is and open source tool for processing 3D data from a wide variety of 3D scanning and image-based sources into high quality 3D geometric models. The tutorial will also include a live demonstration by Mark Mudge and Carla Schroer of the Highlight RTI image acquisition process along with the capture of a camera calibration and photogrammetric image sequences. ********************************************* 2. Integrated Capture Methods for the Generation of Multiple Scientifically Reliable Digital Representations for Museums Tutorial Presenters: Mark Mudge, Carla Schroer Additional Author: Marlin Lum Cultural Heritage Imaging, USA M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums Adoption of RTI tools is underway at leading museums including the Smithsonian Institution, the Museum of Modern Art, the Metropolitan Museum, the Fine Arts Museums of San Francisco, the Los Angeles County Museum of Art, and the Worcester Art Museum. The lessons learned by CHI and its collaborators, which established the sufficient conditions for this adoption, can guide the development of emerging technologies and the adaptation of existing technologies to the adoption requirements of the museum community and cultural heritage activities generally. Figure 1: Unified photo sequence capture of the Sennenjem Lintel from the collection of the Phoebe A. Hearst Museum of Anthropology at the University of California Berkeley. Data to generate RTIs, ARs, and dense textured 3D geometry was acquired during the session 2.1 Factors influencing widespread adoption of digital imaging practices CHI and our collaborators have extensively discussed the obstacles to widespread adoption of robust digital documentary technologies by cultural heritage professionals and the means to remove these obstacles in prior literature. [MMSL06] [MMC*08] [RSM09]. The following material reviews the central themes of this analysis. 2.1.1 Ease of use for museum professionals Designed from the beginning through intensive collaboration with cultural heritage practitioners, Reflectance Transformation Imaging (RTI), and related emerging technologies such as Algorithmic Rendering (AR) along with its next generation Collaborative Algorithmic Rendering Engine (CARE) tool, are crafted to be compatible with current working cultures and digital-imaging skill sets. The goal is to democratize technology and foster widespread adoption of robust digital documentary methods by greatly reducing the barriers of cost and technological complexity that characterize many current 3D methodologies. Until recently, adoption of robust digital practices was slow in museum contexts, largely because many of today’s legacy digital practices, required museum workers to seek help from expensive digital imaging experts, or to learn complex computer programs themselves. For successful widespread adoption, practices must be not require extensive technical re-education, and must remain within the scope of restrictive budgets. The key design insight behind Cultural Heritage Imaging’s (CHI’s) international RTI software research development collaborations and now the AR-based emerging CARE tool is that automation of digital processing tasks can put the computer-science complexity and genius “under the hood,” leaving humanities users free to explore in the direction that accomplishes their primary objectives, using their knowledge more effectively. This strategy overcomes the the ‘hard to learn’ ‘hard to use’ obstacles to digital technology adoption and greatly enhances the effective use of work and research time among domain experts. 2.1.2 Scientific reliability Over the past eight years, CHI's discussions with numerous humanities and natural science professionals revealed that widespread adoption of digital representations in all fields, including the multi-disciplinary study of cultural heritage, requires confidence that the data they represent are reliable. This confidence requires means to qualitatively evaluate the digital representation. For scholars to use digital representations built by someone else, they need to know that what is represented in the digital surrogate is truly what is observed on the physical original. If archaeologists are relying on digital representations to study Paleolithic stone tools, they must be able to judge the likelihood that a feature on the representation is also on the original and vice versa. For scholars to adopt widespread use of digital representations, they must be able to have absolute trust in the representation’s quality and authenticity. RTIs and the CARE tool are designed to record the same information that a scientist records in a lab notebook or an archaeologist records in field notes. The RTI and CARE tools are and will be based on digital photography, capable of automatic post-processing and automatic recording of image generation process history in a machine readable log. Additional software features are under construction. They will automatically map this log to a semantically robust information architecture. Once the mapping process has been completed, digital processing can automatically record empirical provenance information into these semantic architectures as new RTI digital surrogates are “born archival.” We will document process history within CARE using the same robust semantic knowledge management common language, the International Council of Museums’ (ICOM) Committee on Documentation’s (CIDOC) Conceptual Reference Model (CRM) Special Interest Group’s ISO standard 21127 [CCRMweb], including its most recent extension CRM Digital [TTD*10]. M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums This work will build upon CHI’s deep involvement in the CRM, including the recent amendment to permit its use to record process-history provenance during the digitization process of “digital objects.” Incorporation of semantic knowledge management greatly simplifies long-term preservation, permits concatenation of RTI information and information related to its real-world subjects archived in many collections using dissimilar metadata architectures, and demystify queries of vast amounts of information to efficiently find relevant material. Semantically managed archives remove physical barriers to scholarly and public access and foster widespread information re-purposing, future re-use of previously collected information, public access, and distributed scholarship. Each RTI and AR records the access path to the original empirical data — in this case, the raw photographs and processing files. RTIs and ARs are constructed to contain links to their raw data, and are bundled with the raw data when archived. As we have seen, because the processing of the raw photos into RTIs and ARs is performed automatically, we can automatically save the history of this process, each step of the way. The CARE tool will display a visual gallery of different graphic possibilities by performing mathematical transformations not only on the color information, but also on the rich 3D surface-structure information derived from the originally captured photo sequence. Such a gallery of surface-feature depiction and visual emphasis can disclose both anticipated information and accidental discoveries uncovered by processing pipeline options never imagined by the user. Because the record of the processing pipeline settings is compact (in contrast to saving the entire image at each stage), we will be able to transmit the settings over the network to another user who has the same original data, enabling collaborative, interactive visualization design. Any changes performed by one user will be instantly visible to the other. This will allow for easy interaction among multiple domain experts, image-processing experts, and professional illustrators, working together to produce the most effective possible visualizations. The CARE tool will record this entire history of parameter configuration, sample AR generation, user evaluation in the form of AR selection and parameter reconfiguration, subsequent AR generation, evaluation, further AR generation, and so on, until the final AR is created from the RTI capture set. This construction history will be available during and after the AR creation, and can be shared, in real time, with collaborators anywhere in the world. This distributed interplay of creation and discovery will become part of the AR record, enabling others to relive moments of discovery and learn from successful practices. Over time, as computer scientists continue to develop processing possibilities and humanities users continue to describe features that are useful to abstract and emphasize, the number of available processing pipelines is likely to grow very large and the opportunities for serendipitous discovery will increase accordingly. Anyone can view the final AR in an associated viewer and replay this history of discovery and decision. In summary, the RTIs and ARs build in the ability for anyone to access both the original image data and the complete AR generation process history, in order to track and reconfirm the quality and authenticity of the data. Both current and future users of these digital surrogates can decide for themselves whether the RTI or AR is appropriate for their research. 2.1.3 Usefulness for the museum community The documentary usefulness of RTI technology has been demonstrated in many natural science and cultural heritage subject areas [MVSL05] and offers significant advantages, suggesting widespread future adoption. RTI enables robust “virtual” examination and interpretation of real-world subjects that possess surface relief features. An enormous benefit of the technology is the fact that RTI information can be mathematically enhanced to disclose surface features that are impossible to discern under direct physical examination.including raking light photography and microscopy [CHIweb1]. There is a growing family of enhancement functions that use RTI color and 3D shape data to aid the examination, analysis, communication and interpretation of scholarly material. The enhanced interplay of light and shadow in the image interacts with the human perceptual system to reveal fine details of a subject’s 3D surface form. This ability to efficiently communicate both color and true 3D shape information is the source of RTI’s documentary power. For many documentary purposes, RTI also offers cost and precision advantages over other 3D scanning methods. Reflectance information can be captured with widely available and relatively inexpensive digital photographic equipment. CHI has developed techniques for capturing RTIs over a large size range, from a few millimeters to several square meters, and for acquiring a sample density and precision that most 3D scanners are unable to reach. RTIs can capture the surface features of a wide variety of material types, including highly specular reflective material such as jade or gold. The CARE tool will offer significant advantages to museum operations including documentation, curation, conservation, and public outreach. Museum professionals will be able to generate high-quality, comprehensible illustrations for scientific papers and books, with control over selective emphasis, contrast, attention, and abstraction. The process will have lower cost, greater flexibility, and more precise archival documentation than is available with hand-drawn or Photoshopped illustrations. 2.2 Unified capture methodology Today we know how to capture the digital photography image sequences that enables the integrated acquisition of M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums Reflectance Transformation Imaging (RTI), Algorithmic Rendering (AR), digital camera calibration, and the generation of measurable, dense, textured 3D geometry. There are 3 photographic image sets required for the integrated capture process. The first sequence, the RTI and AR data acquisition set, requires a fixed camera to subject alignment. At least 2 black reflective spheres are placed near the subject in the camera’s field of view. The subject is then illuminated from 24 to 72 evenly distributed lighting directions with a fixed light to subject distance. The second photographic sequence captures the information necessary to calibrate the camera. This sequence requires 6 to 8 overlapping photos with the camera positioned in different horizontal and vertical orientations. A detailed discussion of this procedure is found in Section 8. The camera calibration permits optical distortion correction and ortho rectification of the RTI and AR data set. It also lays the ground work for the third photographic sequence, the 66% overlapping set of photos covering the subject that will be used to generate dense, textured 3D geometry. An example of this unified capture method is CHI’s documentation of the Sennenjem Lintel from the collection of the Phoebe A. Hearst Museum of Anthropology at the University of California Berkeley, depicted in Figure 1. The RTI, and dense textured 3D geometry results of the unified capture method are seen in Figures 2, 3, and 4 below. Images of the AR results from the lintel can be seen in Section 4, Figure 7. Figure 2: RTI representation of the Sennenjem Lintel showing the effects of interactive relighting and mathematical enhancement Figure 3: Textured 3D geometry of the Sennenjem Lintel Figure 4: Un-textured 3D geometry of the Sennenjem Lintel 2.3 Conclusion Experience from the RTI and AR software architecture design process has provided a road map to produce scientifically reliable, ‘Born Archival’ knowledge. built for long-tern digital preservation, that fosters widespread adoption within the museum and cultural heritage community. Currently, the tools producing the highest quality 3D textured geometry from photographic images are less likely to see widespread adoption. They are proprietary, expensive, or closely held. Process history logs from these tools are also incomplete or non-existent. These attributes make their long-term preservation less likely. Nonetheless, It is now understood how to capture the image sequences we needed to archive the information necessary to insure a subject’s reflectance properties, 3D geometry, and registered texture is well documented. To the extent that the current textured 3D geometry processing software is difficult to adopt, practically out of reach, and offers less than the desired level of scientific reliability and long-term preservation prospects, capture of the documentary photographic sequences today will make the information available for processing in the future with more affordable, easier to use, more scientifically reliable, preservation friendly, and widely adoptable tools. For museums, this means that collection materials can now be imaged once and returned to an optimized physical preservation environment without fear that they will need to be re-imaged in the near future. The information present in the archived photo sequences will increase in value and descriptive power as the Computational Photography tools designed to exploit them increase in power and practical adoptability. References [CCRMweb] CIDOC Conceptual Reference Model (accessed August, 2010). http://cidoc.ics.forth.gr [CHIweb1] Art Conservation and Reflectance Transformation Imaging, Video. (accessed August 2010) http://c-h-i.org/conservation M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums [TTD*10] Theodoridou, M., Tzitzikas, Y., Doerr, M., Marketakis, Y., Melessanakis, V., 2010. Modeling and querying provenance by extending CIDOC CRM, Distributed and Parallel Databases, Volume 27, Number 2 / April, 2010, pp. 169-210 [MMC*08] Mudge, M., Malzbender, T., Chalmers, A., Scopigno, R., Davis, J., Wang, O., Gunawardane, P., Ashley, M., Doerr, M., Proenca, A., Barbosa, J., 2008. Image-based Empirical Acquisition, Scientific Reliability, and Long-term Digital Preservation for the Natural Sciences and Cultural Heritage. Eurographics Tutorial Notes, 2008. [MMSL06] Mudge M., Malzbender T., Schroer C., Lum M., 2006. New Reflection Transformation Imaging Methods for Rock Art and Multiple-Viewpoint Display. VAST: International Symposium on Virtual Reality, Archaeology and Intelligent Cultural Heritage (Nicosia, Cyprus, 2006), Ioannides M., Arnold D., Niccolucci F., Mania K., (Eds.), Eurographics Association, pp. 195–202. [MVSL05] Mudge M., Voutaz J.P., Schroer C. and Lum M., 2005. Reflection Transformation Imaging and Virtual Representations of Coins from the Hospice of the Grand St. Bernard. Proceedings of 6th International Symposium on Virtual Reality, Archaeology and Cultural Heritage (VAST2005), Mudge M., Ryan N., Scopigno R. (Eds.), Eurographics Association, pp. 29– 39, 2005. [RSM09] Rabinowitz, A., Schroer, C., Mudge, M., 2009. Grass-roots Imaging: A Case-study in Sustainable Heritage Documentation at Chersonesos, Ukraine, Proceedings of the CAA Conference March 22-26, 2009 Williamsburg Virginia, pp. 320-328 ********************************************* 3. Reflectance Transformation Imaging (RTI) System for Ancient Documentary Artefacts Tutorial Presenters: Graeme Earl1, Kirk Martinez2 and Hembo Pagi1 1 Archaeological Computing Research Group, School of Humanities, University of Southampton, UK 2 School of Electronics and Computer Science, University of Southampton, UK 3.1 Introduction This tutorial will provide a summary of our uses of reflectance transformation imaging in archaeological contexts. It also introduces the UK Arts and Humanities Research Council funded Reflectance Transformation Imaging (RTI) System for Ancient Documentary Artefacts project. Some of the case studies and methodologies introduced here are explored in more detail in [EBMP10] and [EMM10]. The AHRC RTI project is a collaboration with Alan Bowman, Charles Crowther and Jacob Dahl at the University of Oxford. 3.2 Recent applications and lessons learned Over the past five years we have been undertaking RTI data capture in a broad range of cultural heritage contexts. In each case the capture technologies employed have been adapted as far as possible to suit specific needs. Experiences from this process have fed directly into the RTI DEDEFI project. 3.2.1 Conservation recording We have applied RTI techniques in a range of conservation contexts. For example, on projects with English Heritage and the Hampshire and Wight Trust for Maritime Archaeology we are using RTI datasets alongside non-contact digitizing via a Minolta laser scanner to provide an emerging conservation record of wooden artefacts recovered from shipwreck and submerged landscape contexts. RTI in particular has proved itself an invaluable interrogative tool for conservators and artefact specialists. In the first case the RTI data produced provide a low cost, easy, portable and interactive means for engaging with fine surface detail. Secondly, comparisons between RTI datasets pre- and post-conservation identify clear transformations in the morphology of the wooden objects as a consequence of the conservation techniques employed, including reburial (Figure 1). Figure 1: Representation of the RTI normal maps as model geometry (left), and a subsequent metric comparison of these representations. Conservation applications have also been demonstrated in ceramic assemblages. Figure 2 shows the subtle surface details made visible by RTI captures. In addition to cracking and repaired fractures on ceramics the technique clearly identified scratched initial sketches on a Greek bowl fragment. This application of the technique at the Fitzwilliam Museum also showed the ability of RTI datasets to reveal small changes in surface reflectance as a function M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums of successive modifications to the glaze of some medieval ceramics. Figure 2: Ceramic viewed under normal lighting (left) and with specular highlights in a captured RTI dataset using the HP PTM fitter (right). The application of RTI captures to complex, irregular solids presents a range of problems. These are well demonstrated in our work to provide a complete conservation record of a Bronze ship ram (Figure 3). A number of RTI datasets were produced at different scales and from different orientations. Figure 3: RTI captures of a Bronze ship ram from a maritime context Problems developing an holistic understanding of the object in part prompted the development of the virtual PTM rig described below, where the photographic coverage is used to derive the camera and light positions for each RTI capture in 3D space. What remains is a need for an RTI viewer that provides a transition between RTI datasets in a fully three-dimensional space, in a way analogous to the Microsoft Photosynth browser. A final conservation application has been in the recording of trial RTI datasets at Herculaneum. Here the technique has provided a good record of the surface morphology of Roman wall painting fragments visible on site (Figure 4). 3.2.2 Analysis of archaeological materials In addition to provision of conservation records our applications of RTI have been driven by specific research needs. In our case studies to date these have focussed on the reading of ancient texts and graffiti, visualization and interpretation of rock art, identification of diagnostic traits in osteo-archaeological materials, reading of ceramic stamps, inscriptions and coins, definition of tool marks in wood and stone, and working practices in ceramics and lithics (Figure 5). In these and other applications it has been clear that whilst the RTI approach is ideal for recording the state of a surface and for sharing this representation with a wide public, the possibilities of the viewer are paramount. In a recent case study in recording medieval pictorial graffiti it was only through the capture of detailed RTI datasets that the full scale of the material was revealed. The practicalities of moving around and changing the illumination within archaeological sites preclude extended visual engagements with the material in a physical context. Conversely the digital analogue provided by the RTI dataset offers limitless, comfortable possibilities for adjusting parameters both within and beyond physical ranges and provides a wholly new form of engagement. . Figure 4: Highlight RTI capture underway on site at Herculaneum Figure 5: RTI dataset of an eroded brick stamp excavated by the AHRC Portus Project M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums 3.2.3 Representation of archaeological data RTI datasets provide an excellent record of surface morphology, reflectance and color for use in the development of computer graphic simulations of archaeological data. Plug-ins enabling direct rendering of RTI data within modelling environments have been limited and short lived and we would welcome the ability to use the gathered data directly in interactive and offline rendering. In the short term we have identified a simple technique for direct comparison of captured RTI and digital geometry, using the camera matching algorithms included in Autodesk 3ds Max to define the three-dimensional locations of RTI capture planes in space (Figure 6). The RTI approach has also been shown to offer a potential as a viewing format for photographic datasets illustrating properties other than reflectance. [MGW01] provide examples of illustrating changing times of day and focal plane via their PTM viewer. We have produced a virtual RTI capture rig in software that enables other digital datasets to be loaded. To date we have used this to represent laser scan datasets and topographic models derived from Geographic Information Systems. The approach also works as an effective means to blend representative styles, for example as a means to demonstrate the data underlying a digital reconstruction. Thus, in our online viewer we are able to load laser scan datasets represented simultaneously as meshes and rendered surfaces (Figure 6). Finally we have used the HP PTM fitter to combine multiple GIS-based viewshed calculations providing an interactive cumulative viewshed viewer Figure 6: Automatic generation of a virtual RTI capture rig from a single camera match . Figure 7: Using an embedded online HP PTM viewer to interrogate a laser scan dataset 3.3 The AHRC RTI DEDEFI project During the course of this tutorial we have described a number of areas of focus for the AHRC RTI research project. To date the project has collated a comprehensive repository of publications relating to RTI, brought together many of its developers and practitioners, designed and built a new capture system, begun to write new capture, fitting and viewing software and captured new RTI datasets from various domains. In addition the project WIKI forms the focus of discussions relating to ongoing and future software and hardware developments. There is not scope in this tutorial to cover all areas but we have identified some crucial steps in what follows, many of which are being explored under the aegis of the project. We very much welcome input from new partners. The needs of conservators to produce records enabling both qualitative and quantitative comparisons have prompted discussions concerning RTI enhancements. Firstly, the need to derive true three-dimensional data via a photogrammetric or other technique is clear. Whilst RTI fulfils a different role to such techniques the need to measure values such as lost volume is considerable. Secondly, formal modes for registering and comparing normal maps need to be integrated into the RTI viewer. This would enable habitual application of the comparisons shown in [DCCS06]. Thirdly, the automated or manual calculation of a per-pixel scale factor per RTI should be incorporated into the fitting and viewing process. Similarly an automated process for removing metric distortion and for application of color calibration across datasets tied to individual light sources is needed. The RTI viewing experience remains limited. While the fully three-dimensional viewer described above is an ideal, simpler paths to the development of 3D RTI datasets are needed. C-H-I and others have proposed potential solutions to this problem. Of further significance is the ability to annotate RTI datasets, including the ability to associate annotations with RTI viewer parameters such as light position and image processing parameters. These are core M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums requirements for our Oxford University colleagues and others working on transcription of ancient document artefacts, and ones with a considerable extant body of literature. Furthermore, the dataset loaded into the viewer needs to be potentially far larger, with the ideal being a seamless tiling of multiple RTI datasets, in addition to the tiled delivery of single high resolution RTIs. RTI capture hardware continues to improve with a number of groups developing dome and other rig systems. Our own project is developing a series of systems. The first completed dome is 1m in diameter and divided into four portable segments, with a current maximum of 76 light positions. The system uses a Nikon D3X SLR. Our next capture dome will fit on a standard archaeological microscope enabling rapid, very high resolution RTI capture. 3.4 Conclusions The RTI technique remains under-utilized. Whilst we continue to come across new and exciting applications of RTI it is surprising the extent to which colleagues in archaeology, conservation science, museum studies, art history, epigraphy and ancient document studies remain ignorant of the technique. Above all other challenges our RTI project and this VAST workshop must seek to induce step changes in the technology, the awareness of its potential, and crucially the further development of a shared community of practice. 3.5 Acknowledgements Ongoing work in Reflectance Transformation Imaging is funded under the AHRC DEDEFI programme. It is a collaboration between the University of Southampton and the University of Oxford. The project draws on the generous contributions of a great many partners via our project WIKI. We would very much welcome new members to join this group. We are particularly grateful to input from Tom Malzbender and from Cultural Heritage Imaging who have been instrumental in our work with RTI and in our ability to develop this tutorial. Further details of the project are available at: http:// www.southampton.ac.uk/archaeology/acrg/ acrg_research_DEDEFI.html References [DCCS06] Dellepiane, M., Corsini, M., Callieri, M., Scopigno, R., 2006. High quality PTM acquisition: Reflection Transformation Imaging for large objects in: Ioannides, M., Arnold, D., Niccolucci, F., Mania, K. (eds) VAST06: Proceedings of the.7th International Symposium on Virtual Reality, Archaeology and Cultural Heritage (Cyprus, 2006), pp. 179-86 [EBMP10] Earl, G., Beale, G., Martinez, K. and Pagi, H. 2010. Polynomial texture mapping and related imaging technologies for the recording, analysis and presentation of archaeological materials. Proceedings of ISPRS Newcastle 2010 Commission V, WG VI/4. Available from: http://eprints.soton.ac.uk/153235/ [EMM10] Earl, G., Martinez, K. and Malzbender, T. 2010. Archaeological applications of polynomial texture mapping: analysis, conservation and representation. Journal of Archaeological Science, 37. Available from: http://eprints.soton.ac.uk/156253/ [MGW01] Malzbender, T., Gelb, D., Wolters, H., Polynomial Texture Maps, Proceedings of ACM Siggraph 2001, pp. 519-528. ********************************************* 4. Visualizing and Re-Assembling Cultural Heritage Artifacts Using Images with Normals Tutorial Presenters: Corey Toler-Franklin, Szymon Rusinkiewicz Princeton University, USA 4.1 Introduction Images with normals (RGBN images[TFFR07] are a type of data that lies between simple 2D images and full 3D models: images with both a color and a surface normal (orientation) stored at each pixel. RGBN images are powerful tools for documenting complex real-world objects because they are easy to capture at a high resolution, and readily extendible to processing tools originally developed for full 3D models. Several characteristics of RGBN images make them practical solutions for illustrating artifacts of cultural heritage significance: Easy to Acquire: The process for capturing RGBN data is only mildly more complex than taking a digital photograph. Low-cost, off-the-shelf capture devices (digital cameras and 2D scanners), make the process practical and significantly easier than 3D scanning. For example, complex shapes with significant occlusion, like the pinecone in Figure 1, would require the alignment of dozens of 3D scans to create a hole- free model (even from a single viewpoint). High Resolution: RGBN images are more informative than traditional color images because they store some information about the object’s shape. In addition, they have higher resolution color and normal maps (Figure 2) than 3D geometry from 3D laser scanners, giving us the ability to document, visualize, and analyze fine surface detail. Easily Extended For Stylized Rendering: Most state-of- the-art nonphotorealistic [GG01] rendering algorithms are simply functions of the surface normal, lighting and viewing directions. Simple extensions to signal processing tools can preserve the integrity of the normals, while introducing a wide range of control for a variety of stylistic effects. M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums Simple and Efficient: RGBN images are more efficient to process than full 3D geometry, requiring less storage and computation time. Functions are computed in image space producing powerful 3D results with simpler 2D methods. Figure 1: Capturing RGBN images using a digital SLR camera and hand-held flash. White and mirror spheres are used to find the flash intensity and position. Right: The original image, extracted normals, colors, and depth discontinuities . Figure 2: Capturing RGBN images using a high resolution 2D flat-bed scanner. Left: The object is scanned at multiple orientations. The scanner’s light source is linear (Top Right); a calibration step is used to measure I(n). The output is a high resolution color texture and normal map. 4.2 Capturing RGBN Datasets There are several methods for acquiring RGBN datasets. We use photometric stereo [Woo80], a process whereby normals are inferred from several images (captured from a single camera position) of an object illuminated from different directions. We assume a perfectly diffuse object with equal brightness in all directions. Under these conditions, the observed intensities are given by the Lambertian lighting law where a is the albedo of a point, n is the surface normal, and li is each lighting direction. With at least 3 (preferably more) such observations, we can solve for the normal n using linear least squares. Our set-up is depicted in Figure 2. When there is less control over background lighting, or objects are small and flat, a 2D scanner is a more effective capture device for recording fine details. Brown et al.[BTFN*08] deployed this technique (Figure 2) to archive fragments of the Theran frescos at the archaeological site of ancient Akroriti, (modern day Santorini, Greece). Although we use photometric stereo, we cannot use the traditional formulation of the Lambertian Lighting Law because the scanner’s light source is linear (rather than a point source). We introduce a one-time calibration phase to measure I(n), the observed brightness as a function of the surface normal. This is achieved by sampling intensities over a wide range of known normal orientations. We then fit a first-order spherical harmonic model to the sampled data to obtain this parametric representation Fragments are scanned at multiple orientations (typically four). Given a set of scans a0 a1 a2 a3 , we invert I to solve for the normal, n. Figures 3 and 4 show the results. Figure 3: Computed normals (top right) reveal more surface detail than those extracted from the geometry (top left). Extracted RGB color (bottom right) has a higher resolution than color maps from the 3D scanner (bottom left). Figure 4: String impressions, most clearly visible in the computed normals, are important cues for reconstruction [TFBW*10], restoration, and archaeological study. M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums 4.3 Tools for RGBN Processing Nonphotorealistic rendering algorithms rely on fundamental signal processing tools that are easily adaptable for use with RGBN images. Filtering: Smoothing is important for de-noising and scale- space analysis of images. However, we cannot naively convolve an RGBN image with a smoothing kernel.We must account for foreshortening: over-smoothing in regions where normals tilt away from the view direction.We assume a constant view direction (along z) and scale the contribution of each normal by secq, transforming the vector (nx,ny,nz) into (nx/nz,ny/nz,1). To avoid blurring across depth discontinuities, we adopt the bilateral filter [TM98] which is edge-preserving. Specifically, we augment the bilateral filter with a term that reduces the influence of samples on the basis of differences in normals: where ci and xi are the color and location of pixel i, g is a Gaussian, and the sum is over all pixels j in the image. In this equation, σx and σc are the widths of the domain and range filters, respectively; decreasing σc leads to better preservation of edges. The normal differences |ni - nj| are computed using the foreshortening correction, as above. Figure 5 shows the effects of adjusting the bilateral filter parameters. Segmentation: One illustration technique separates regions of an image and renders them in different shading styles. RGBN segmentation extends the graph-cut segmentation algorithm of Felzenszwalb et al. [FH04] to consider not only color, but normals. RGBN pixels are continually clustered to form components, such that edges between components in the graph have larger weights (larger dissimilarity values) than edges within components. Figure 9 shows how segmentation by color and shape can be more effective than segmentation by color alone. Curvature Estimation: Several stylization techniques use surface curvature to convey shape. The normal curvature of a surface is the reciprocal of the radius of the circle that best approximates a normal slice of surface in the given direction. By tracking the changes in direction of the normals over the surface, we can compute properties such as mean curvature, Gaussian curvature or the principal curvatures. However, we must account for the foreshortening effect. Refer to Toler-Franklin et al. [TFFR07] for the details. Figure 5: The RGBN bilateral filter is capable of producing different results, depending on the settings of the domain and range filter widths. For large σc and, σn there is little edge preservation, and the filter resembles a simple Gaussian. Making σc small preserves color detail, such as that around the eye, while making σn small as well preserves both color and geometric edges. 4.4 Depiction Styles We can apply the signal processing framework for manipulating RGBN images to several stylization techniques. Toon shading: Cartoon shading consists of quantizing the amount of diffuse shading (i.e., n·l) and mapping each discrete value to a different color. This technique is effective because it abstracts shading while conveying information about geometry (the boundaries between toon shading regions are isophotes, curves of constant illumination that have been shown to convey shape). Because toon shading only depends on the surface normals, it easily extends to RGBN images. Figure 6 is an example of how toon shading is used to enhance surface features not apparent in the color image. Line drawings: Many line drawing algorithms are computed directly on the normal maps. For example, discontinuity lines mark locations where there are significant changes in depth. They occur where there are sharp changes in normal direction among neighboring normals, and at least one normal is nearly orthogonal to the viewing direction. Figure 6 combines discontinuity lines with toon shading to define silhouette edges. Suggestive contours are similar to lines that artists draw. They are found by calculating n·v, (where v is the viewing direction) over the entire intensity map and then searching for local valleys in intensity. M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums Figure 6: Locations of depth discontinuities overlaid on toon shading. Exaggerated Shading: Exaggerated shading [RBD] considers all possible lighting positions to find the maximum contrast at all scales over multiple orientations. The result reveals fine surface details (Figure 7 top left) that are not readily apparent in color only. Figure 7: Sennedjem Lintel from the Phoebe A. Hearst Museum of Anthropology: A variety of stylization techniques can be used to reveal more information than is readily apparent in the color-only image. (Top Left) Exaggerated shading reveals fine surface detail. Details are further enhanced by darkening groves and emphasizing large features (Top Right). Lambertian shading (Bottom Right) can be computed on the grey scale image by combining the normal map (Bottom Left) with a lighting direction to convey shape. Curvature Shading and Shadows: Shadows are important for conveying shape. Because RGBN images have no depth information, we must simulate shadowing effects. Multi- scale mean curvature shading works by darkening regions with negative mean curvature and brightening those with positive mean curvature. The result is averaged over multiple scales to reduce high-frequency noise (Figure 8). . Figure 8: Multi-scale curvature shading closely resembles ambient occlusion, revealing shape over local neighborhoods 4.5 Cultural Heritage Applications RGBN images are suitable for many cultural heritage applications. High quality renderings generated with flexible signal processing tools are ideal for textbook illustrations. RGBN datasets are suitable for art historical and scientific study. Figure 11 uses exaggerated and mean curvature shading to analyze a petroglyph. The nonphotorealistic visualization (Bottom) reveals inscriptions that are fairly deep, and almost invisible in the color photograph (Top). Fine surface markings on fragments (Figure 4) are important cues for matching and re- assembling fragments of objects. The 2D acquisition pipeline and the resulting high fidelity data would be suitable for applications in forensics, where surface cues are important. M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums . Figure 9: RGBN segmentation produced accurate results without visible color edges. The hammer has been segmented into multiple facets . Figure 10: Illustration of tools reveals fine details, such as the maker’s stamp on the shears. References [BTFN*08] BROWN B., TOLER-FRANKLIN C., NEHAB D., BURNS M., DOBKIN D., VLACHOPOULOS A., DOUMAS C., RUSINKIEWICZ S., WEYRICH T.: A system for high-volume acquisition and matching of fresco fragments: Reassembling Theran wall paintings. In ACM Transactions on Graphics (Proc. SIGGRAPH) (Aug. 2008), vol. 27. [FH04] FELZENSZWALB P., HUTTENLOCHER D.: Efficient graph-based image segmentation. International Journal of Computer Vision 59, 2 (Sept. 2004). Figure 11: Color image and nonphotorealistic rendering (with mean curvature shading and exaggerated shading) of the Legend Rock archaeological site. [GG01] GOOCH B., GOOCH A.: Non-Photorealistic Rendering. A. K. Peters Ltd., 2001. [RBD] RUSINKIEWICZ S., BURNS M., DECARLO D.: Exaggerated shading for depicting shape and detail. In ACM Transactions on Graphics (Proc. SIGGRAPH), vol. 25. [TFBW*10] TOLER-FRANKLIN C., BROWN B., WEYRICH T., FUNKHOUSER T., RUSINKIEWICZ S.: Multi-feature matching of fresco fragments. In ACM Transactions on Graphics (Proc. SIGGRAPH Asia) (Dec. 2010). [TFFR07] TOLER-FRANKLIN C., FINKELSTEIN A., RUSINKIEWICZ S.: Illustration of complex real-world objects using images with normals. In International M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums Symposium on Non-Photorealistic Animation and Rendering (NPAR) (2007). [TM98] TOMASI C., MANDUCHI R.: Bilateral filtering for gray and color images. Proc. ICCV (1998). [Woo80] WOODHAM R.: Photometric method for determining surface orientation from multiple images. Optical Engineering 19, 1 (1980), 139–144. ********************************************* 5. Visualization of RTI images Tutorial Presenter: Gainpaolo Palma Additional author: Massimiliano Corsini Visual Computing Lab, ISTI - CNR, Italy 5.1 Introduction Reflectance Transformation Images (RTI) have significant potential in the Cultural Heritage (CH) field, where the way light interacts with the geometry is important in the visual examination of the artifact. The characteristics of the material, the reflectance behavior, and the texture offer major perceptual and cognitive hints for the study of these kind of objects with respect to the simple 3D geometry. To further improve the user ability to interactively inspect the content of the RTI media, several shading enhancement techniques have been proposed for improving the perception of the details and the shape characteristics. We present two tools to visualize and analyze RTI images in an interactive way. The first one is a multi-platform viewer, RTIViewer [CHI], developed also to work remotely through HTTP, that allows the user to apply a set of new shading enhancement techniques improving the virtual examination and interpretation of several details of the artifact. The second is a web application based on SpiderGL [DBPGS10], a JavaScript 3D graphics library which relies on WebGL, which permits the realtime rendering of huge RTIs with a multiresolution encoding in the next generation of web browser. 5.2 RTIViewer RTIViewer is a multi-platform tool to load and examine images created with RTI techniques. The tool supports several formats, collectively called RTI files: Polynomial Texture Maps (PTM files) [MGW01]; Hemispherical Harmonics Maps (HSH files) [GWS09]; Universal Reflectance Transformation Imaging (URTI files). The viewer can display both single-view and multi-view images; a multi-view RTI [GWS09] is a collection of single-view images together with optical flow data that generates intermediate views. . Figure 1: High-relief in gilded wood representing a kiss between Corsica and Elba islands from Isola D’Elba museum; (top) standard rendering; (middle) specular enhancement; (bottom) static multi-light enhancement. M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums The tool is capable to visualize a RTI image loading it from a local hard disk or from a remote server through HTTP connection. In order to handle the remote loading, the original image (usually of huge resolution) has to be processed by a command line tool to prepare a multi- resolution encoding. The tool allows also the interactive changing of several rendering parameters, like the zoom factor, the light Figure 2: Sumerian cuneiform tablet: (Left) standard rendering; (Center) diffuse gain; (Right) normal unsharp masking Figure 3: Roman sarcophagus in the Camposanto Monumentale of Pisa: (Left) standard rendering; (Center) luminance unsharp masking; (Right) coefficient unsharp masking Figure 4: Tomb of the Archbishop Giovanni Scherlatti in the Opera Primaziale of Pisa: (Top) standard rendering; (Bottom) dynamic multi-light enhancement. M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums direction, the shading enhancement technique to apply to the image and its settings, and, only for multi-view format, the viewpoint around the object. Several shading enhancement methods are available: • Diffuse Gain [MGW01], that enhances the perception of the surface shape by increasing the curvature of the reflectance function (Figure 2); • Specular Enhancement [MGW01] to add a specular effect to the surface by a Phong/Blinn shading (Figure 1); • Normal Unsharp Masking [MWGA06] [PCC10] that enhances the high frequency details of the normals by unsharp masking (Figure 2); • Luminance and Coefficient Unsharp Masking [PCC10] to enhance the high frequency details of the luminance channel of the LRGB RTI or of the basis coefficients of the polynomial by unsharp masking (Figure 3); • Multi-Light Detail Enhancement [PCC10] that uses different light directions to create a virtual lighting environment that maximizes the sharpness of the image and at the same time preserves the global brightness. There exist two versions: dynamic enhancement where the light chosen by the user is locally perturbed (Figure 4); static enhancement that produces an automatic high-contrast and well-illuminated single image by sampling all possible light directions (Figure 1). Some methods are based on a per-pixel surface normal estimated by photometric stereo methods or by computing the light direction that maximizes the reflectance function used in the PTM images [MGW01] assuming a lambertian material. 5.3 RTI on the web Thanks to WebGL, graphics API specification for the JavaScript programming language, it is possible to use GPU capabilities in a next generation of web browser without the need for an ad-hoc plug-in. SpiderGL is a JavaScript library for developing 3D graphics web applications based on WebGL, providing a set of data structures and algorithms to ease the development of WebGL application, to define and manipulate shapes, to import 3D models in various formats, to handle asynchronous data loading. These characteristics can be exploited even for the visualization of huge RTI image in the web (see [Ben] for a demo) with a multi-resolution encoding. This encoding needs a hierarchical layout of the data to prepare the data to store in a web server, an algorithm to visit such hierarchy and determine the nodes to use for producing the current viewport, and the ability to load the nodes of the hierarchy asynchronously, i.e. to proceed with rendering while missing data are being fetched. The hierarchical layout is made with a quadtree where for each nodes we save a number of images to store the RTI data, i.e 3 PNG images for a LRGB PTM. References [Ben] BENEDETTO M. D.: SpiderGL - PTM example. http://www.spidergl.org/example.php?id=9. [CHI] C-H-I: Cultural Heritage Imaging - RTIViewer. http://www.c-h-i.org/learn/index.html. [DBPGS10] DI BENEDETTO M., PONCHIO F., GANOVELLI F., SCOPIGNO R.: Spidergl: A javascript 3d graphics library for next-generation www. In Web3D 2010. 15th Conference on 3D Web technology (2010). [GWS09] GUNAWARDANE P., WANG O., SCHER S., DAVIS J., RICKARD I., MALZBENDER T.: Optimized image sampling for view and light interpolation, 2009. [MGW01] MALZBENDER T., GELB D., WOLTERS H.: Polynomial texture maps. In SIGGRAPH’01 (2001), pp. 519–528. [MWGA06] MALZBENDER T., WILBURN B., GELB D., AMBRISCO B.: Surface enhancement using real-time photometric stereo and reflectance transformation. In Eurographics Workshop/ Symposium on Rendering (2006), pp. 245–250. [PCC10] PALMA G., CORSINI M., CIGNONI P., SCOPIGNO R., MUDGE M.: Dynamic shading enhancement for reflectance transformation imaging. Journal on Computing and Cultural heritage (2010). ********************************************** 6. Museum uses of RTI at the Smithsonian Institution Tutorial Presenter: Mel Wachowiak Additional Author: Elizabeth Keats Webb Museum Conservation Institute (MCI), Smithsonian Institution Washington, D.C. USA 6.1 Introduction This section will describe some museum uses of RTI and its place among photographic capture and 3D scanning at the Smithsonian Institution (SI). The SI is the world’s largest museum complex and has an incredible breadth of collections. MCI has a central role as a research unit and collaborator in analysis of heritage objects and sites. Imaging is part of the routine research, diagnostic, and documentation conducted at MCI. The SI has only recently begun a major examination of digitization of collections, which can include data, still images, video and other motion picture, sound, and associated metadata. MCI’s part in the digitization of collections is to offer a more expanded vision of the application of appropriate technologies. M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums While RTI is the focus of this presentation, it is but one part of imaging technology used by MCI. It should be no surprise that there are overlapping techniques, in terms of data collected and the scale of the objects. Work can include microstructure to macrostructure, two and three dimensions, and wavelengths beyond the visible. Several methods are based on the computational advances that digital photography offers. For example, high dynamic range imaging (HDRI) offers an enormous increase in dynamic range compared to 8-bit images. Multispectral imaging is possible with modified digital cameras, and can be improved by sensors with increased or more specific ranges (such as infrared sensors). Laser or structured light scanners can capture extremely high-resolution data in three dimensions, and some capture color for each point in the spatial data. Multifocus montage, or extended depth of field, has added a third dimension to microscopy in a practical solution. As application specialists, not developers, we in the "Imaging Group" at MCI have a unique responsibility. We conduct research and interpret objects using various technologies. Our task is often to find the correct solution from among available technologies, or collaborate with specialists. One interesting fact about RTI is that it fills a niche that other imaging solution can’t fill. In particular, it bridges the gap between photography and 3D scanning. However, it is more than a "2-dimensional" solution. It has been amply demonstrated that it offers an immersive, near 3D experience and image processing tools. It should also be pointed out that it can accurately document features that are impossible to acquire with 3D scanning. As such, it is an important addition to the cultural heritage community. After receiving training by CHI staff in 2009, lead by Mark Mudge and Carla Schroer, we have compiled a fairly broad range of RTI projects. These have ranged in size and scope from tiny natural history specimens to large artworks, both in the studio and on location. Buttons, jewelry, fossils, prehistoric stone tools and many other materials have helped us understand the strengths and weaknesses of the current RTI technology and software. Several examples below will illustrate RTI results and some creative solutions to problems encountered. 6.2 Preparation for RTI It is certainly worth mentioning again the importance of understanding the digital camera and good workflow practices. The best images are those that require no post- processing! For this reason we spend appropriate time with color balance and exposure conditions. We also have a responsibility as stewards of the collection to take great care in handling and positioning objects, and to limit light exposure. 6.3 Examples of RTI: Easel Paintings Two recent projects have demonstrated the great value of RTI for documentation and investigation of paintings. One painting is Italian from the late 15th century, the other by an American working in Paris the early 20thcentury. The 20th century oil painting was created with heavy impasto to accentuate light and shadow effects. RTI was an excellent was method to reveal the texture, as well as later defects. The conservators and curators are also interested in creating interactive display stations for the museum visitors. While single images can capture the general effect of RTI images, the great strength of the technique is rapid, nondestructive processing. By making the RTI files available to the public, we greatly enhance their appreciation and understanding of the object. Conservation and preservation are also better understood since it is easy to demonstrate both the subtlety of the art and the fragile conditions of some objects. Digital surrogates made by RTI are excellent preservation and research tools Figure 1: RTI of painting showing normal lighting (left) and specular enhancement (right). The specular enhancement shows the surface texture without distraction of pictorial elements. The earlier painting is painted on a panel of edge-jointed wood. It is very thinly painted and almost transparent in some areas. The surface is somewhat glossy, which precludes most 3D scanning techniques. The conservator investigating the painting was most interested in the scribe lines in the preparatory layer used to layout the single point perspective of the painting. While they are evident in raking light, the indented scribe lines are difficult to image and study. M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums Figure 2: RTI set up for large panel painting: reference spheres at top, scale and color target below. In our preparation for imaging the painting, which is nearly seven feet (2.13 meters) wide, we tested the ability of our camera to resolve the scribe lines. The sub-millimeter wide indentations made by the stylus are too small to be seen in an image of the entire painting. Therefore, we needed to divide the imaging into three sections; all with about 25% overlap. The images below show fine details of the painting made visible using the specular enhancement algorithm. Note the alteration of the original composition of the building on right (arrow in specular enhancement image) Figure 3: Panel painting detail approximately 30cm wide (normal view above); specular enhancement shows alternation of original building composition (upper part of central area). Note that the black painted areas create a false impression of texture. RTI of other paintings has revealed features that were intriguing glimpses into their histories, including subtle deformations of the canvas occurring during storage. 6.4 Daguerreotype This mid-19th century photograph proved another excellent project for RTI. Daguerreotypes are notoriously difficult to light and photograph for documentation. In addition, the surface is essentially polished metal and would therefore be a poor subject for 3D scanning. We were able to successfully do RTI with only slight modification of our typical method. A custom-made velvet lined snoot for the flash made a significant improvement in the RTI. One of the great advantages of RTI is in the creation of excellent documentation for condition examination. The many scratches and accretions are quite apparent, as is the general deformation of the sheet metal. The level of detail is impressive in an RTI averaging 4000 x 5000 pixels. Many observers have remarked that it is similar to examining the object with a stereomicroscope. Figure 4: Daguerreotype above, and 2cm high detail below; specular enhancement at left, image unsharp mask to right. 6.5 Ebony door The RTI of a pair of ebony and ivory veneered doors was prompted by technological limitations of other 3D scanning M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums and photography technologies. The conservator’s wish was to document the condition before treatment. These doors are part of a very glossy French polished cabinet. Three aspects of the surface geometry eliminated other imaging techniques. 1.The highly reflective surface cannot be imaged by 3D scanning because of scattering of light, and the surface could not be modified. 2. Black surfaces or white surfaces cause related problems (high absorption or reflection). 3. Black and white material adjacent to one another is even more problematic. Not surprisingly, attempts made using a structure light scanner were unsuccessful. RTI was very effective, as seen in the figure below. Figure 5: Detail of French polished ebony and ivory veneered door (approx. 30cm x 30cm). RTI image at left "illuminated" by source perpendicular to surface. At right, specular enhancement and raking angle shows surface deformation and Shreger lines in ivory. Inset circle at lower left is from an actual image processed for the RTI and shows high reflection from flash at 65¡. 6.6 Lenape Indian bandolier bag This leather object is Native American and dates from approximately 1820. The curator could decipher seven words from an inscription on one side of the bag. The ink was dark brown in areas, and apparently faded in others. Multispectral imaging was not particularly helpful, but did reinforce an earlier assumption. Since the brown ink completely disappeared in infrared bands, we concluded it was probably iron gall ink. The ink had eroded some of the leather, leaving a slight impression, which led us to attempt RTI. The RTI was not particular successful, most likely due to the overall suede-like surface texture. However, we were able to use the RTI and many of the individual images used to create it. By comparison of the nearly 50 images, as well as the RTI, we were able to determine that there were 21 words in the inscription and could decipher 19. Figure 6: Lenape Indian leather bandolier bag (top); detail showing inscription (bottom) 6.7 Summary This last example of the uses of RTI is a bit of a cautionary tale. RTI is certainly not the only method for documenting the surfaces of objects. But it has earned an important place among our practical imaging tools. With our practical imaging experience, we have been able to use RTI at the SI almost immediately. We have appreciated this powerful new tool and have addressed several problems in order to maximize the results. We are especially looking forward to improvements and new developments in the software. All of the examples shown here share a common trait: they are generally flat, with only a slight waviness. They also have a finer scale texture that comprises the surface of interest. These flat objects show the subtlest detail to advantage. They were chosen for their high contrast of features are not meant to misrepresent the possibilities. We have successfully completed RTI of many other object types including larger carvings, human teeth, and other highly textured objects. Just as in the in the case of the Lenape bandolier bag, the combination of tools, not exclusive use of one, will give the best result. 6.8 Acknowledgments We greatly appreciate the guidance of the staff at Cultural Heritage imaging, especially Mark Mudge and Carla Schroer, and Tom Malzbender of Hewlett-Packard Laboratories. ********************************************* M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums 7. Digital Preservation Workflows for Museum Imaging Environments Tutorial Presenter: Michael Ashley Cultural Heritage Imaging, USA 7.1 Introduction We discuss and demonstrate practical digital preservation frameworks that protect images throughout the entire production life-cycle. Using off the shelf and open source software coupled with a basic understanding of metadata, it is possible to produce and manage high value digital representations of physical objects that are born archive- ready and long-term sustainable. We demystify the alphabet soup of file formats, data standards, and parametric imaging, and demonstrate proven workflows that can be deployed in any museum production environment, scalable from the individual part time shooter to full fledged imaging departments. 7.2 The iPad Effect and Museum Imaging The world of imaging is going through its next paradigm shift, and it requires radically rethinking how digital curators work with their media collections. Mobile and cloud computing is application based, not file based, and the tendency is to hide the file system from users in favor of media libraries held within and accessed through applications. "Apple is dramatically rethinking how applications organize their documents on iPad, ... Rather than iPad apps saving their documents into a wide open file system, apps on iPad save all their documents within their own installation directory. Delete the app and you’ll clean out all of its related files." [Dil10] The divide between professional producers/managers of content and consumers has grown, but convergence is on the way. So much attention (and financial development resources) are trained on mobile computing that we are seeing amazing applications for creating high definition media that are much ‘smarter’ than their predecessors. This includes wifi transfer of images in realtime, built-in GPS, 3D video, and native DNG shooting (see below). For digital imaging professionals and enthusiasts, it is an exciting but confusing moment in history. This technology shift has direct implications for digital preservation and access workflows. We can prepare our digital assets to not only withstand the transformations they go through as they move from one application and platform, but to become more valuable through context aware metadata embedding. You can safeguard your media collection while taking advantage of the positive impacts mobile computing is having on cultural heritage imaging. Small tweaks to well defined image production workflows can yield dramatic dividends both in digital asset management/preservation and in public access/enjoyment. This tutorial focuses on straightforward steps that will help museum imaging professionals produce born archival media that can survive the iPad effect by strategically embedding essential metadata within files. Following these steps will save you time and help to future proof your image collection investment. This tutorial relies heavily on the corpus of excellent materials available on digital asset management for professionals and attempts to augment rather than repeat. We provide essential references throughout the guide. 7.3 Basics of Born Archival Imaging As Doerr et al. argue, the CIDOC-CRM provides a nearly generic information model for handling cultural heritage events, documents, places and people [DI08]. Born archival imaging can be described as a method for implementing the CRM. This simple approach requires three components, defined as a framework of things, people, places, and media, meeting in space and time [ATP10]. Uniquely Identified Entities. Treat every digital original with the same respect as museum objects. Give every person, place and object and media item in your museum a unique identifier. A majority of this effort can be automated using non-destructive parametric image editing (PIE) software [Kro09]. Refined Relations Between Entities. The list of potential relations between people, places, things and media is comprehensive and manageable. This TIF (media) is a photo of a painting (thing). The Night Cafe (thing) was painted by Van Gogh (person) in Arles, France (place) in 1887 (event), and is now located in the Yale University Art Gallery in New Haven (place). We will explain the process of defining provenance information for imaging below. Parameterize Properties. There is a fine balance between exhaustive controlled vocabularies and Google’s text indexing algorithm. Within museum contexts, it is relatively easy to define ‘local’ terms that people can agree upon and use in their daily practice. Defining a list of properties to describe the people, places, things and media limited to a particular museum context will produce incredibly accurate search and browse capabilities, whether using Google or a desktop file browser. This localization can be articulated with other standardized efforts, such as Metaweb’s Freebase, to link up your data and media to the world’s knowledgebase. 7.4 Born-Archival Implications What do we mean by ‘born-archival’? John Kunze, preservation specialist for the California Digital Library, calls for ‘born-archival’ media that is fully accessible and preservable at every stage, throughout the lifecycle of this data, from birth through pre-release to publication to revision to relative dis-use and later resurgence. Data that is born-archival can remain long-term viable at significantly reduced preservation cost [Kun08]. M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums Figure 1: Seeing Double: Standards for media and museums require mapping and translation. Born archival images require the medium and the message to be openly readable and stable, akin to a digital Rosetta Stone. The medium is the file format. The message is the raw image data and associated metadata. The challenge is to produce a resilient digital surrogate that can withstand limitless media transfers and potentially destructive metadata modifications over the span of its life. Standards. There are hundreds of metadata standards to choose from, and they all have their virtues. We are not addressing metadata standards in this tutorial, except to say that it is vital to follow a standard, documented protocol. Jenn Riley and designer Devin Becker recently mapped the myriad of standards that are applied to worldwide information (Figure 1) [RB10]. Whatever standards you are using to manage your collections, born archival imaging simply requires the best practice of explicitly stating what standards you are using and how you are implementing them. We call this the desk instructions, an essential component for describing the provenance of image processing. Born-archival imaging requires the desk instructions be accessible to the viewer, either through URL linking or direct embedding within the asset. File Media. What good is metadata if the medium they and the images they describe are stored on will not last? This is a reality of digital storage. While stone tablets can last millennia and silver based film for hundreds of years, digital media is trending toward shorter and shorter lifespans [DB10]. We overcome this risk by creating a structured network of pointers from derivatives and data files to the original source image, no matter where it is stored, or in what format. We advocate for a two-pronged approach to media management, described by Peter Krogh as the ‘truth is in the catalog’ vs. ‘the truth is in the file’ [Kro09]. The idea is that you push the best, most current and accurate information into your image files as you can, a combination of descriptive and technical metadata, a snapshot at the end of the image’s production. You keep a separate, external database of the metadata. This is the definitive informational source. If anything happens to the image or its data, the provenance can be reconstituted from the database. We describe this as the passport and the bureau model. Figure 2: Media through the millennia, from analog to hybrid to virtual, the trend is toward short-term lifespans. 7.5 The Passport (file) and the Bureau (library) The passport is the data in the file and the bureau is the full metadata, stored in the catalog. Image files, from RAW to TIF and JPG, can include embedded data in XMP format. XMP is a derivative of XML, an open standard created by Adobe, and is held within the DNA of every image file. Backed by camera manufacturers, international standards bodies and software companies, XMP is the Esperanto of photographic metadata. XMP is a loose standard, providing a rich framework for you to embed your metadata in the format that best fits your needs. There are many open source, inexpensive and professional options for embedding data into images through XMP (Figure 3). With careful planning, XMP will accommodate even the most complex metadata schema. Because it is simply structured XML, it is a highly resilient preservation format. The library can be a catalog file, such as an Adobe Lightroom database, to a full-fledged XML or relational database, a simple photo log saved as an Excel spreadsheet, or a saved query from your collection management system. M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums Figure 3: XMP is an emerging open standard, supported by Adobe and embraced by camera and software companies. 7.6 Binding message, media and objects in perpetuity The metadata stored within a file or externally in a database is only as useful as the bound relationship between both. How can you assure a connection? Embed within the file the links to unique identifiers associated with data records and original surrogates. Once this connection is made, it is much more trivial to link derivative files and subsequent comments and notes. Example: In a shooting session at the deYoung museum, the staff images a painted frame attributed to Domenico Ghirlandaio, accession number A387967. The image, A387967_AT_V1_001_O.tif, has a data record stored in the deYoung image database, http://A387967_AT_V1.html. This URL provides links to the metadata on the masterpiece, the image production, and all known derivative works, as well as information on the photo sessions associated with the painting. Filename: The filename is comprised of the object ID, photo session ID, a serial number and file version. In this example, AT = after treatment, V1 = version 1, O = digital original. The session ID = A387967_AT. In the simplest system, you would swap the O for T (thumbnail) or F (full resolution JPEG). We have bound the physical object to its shooting event, version and file type in a compact format. At the absolute minimum, you would embed the data URL in the file. Ideally, you would add descriptive metadata about the subject, technical metadata about the file, and additional information about who produced the image and how. This example assumes that an established museum such as the deYoung will continue to have robust data infrastructure, thus the amount of carried metadata can be quite minimal. To assure maximum reusability, we would embed as much provenance information in the XMP of the file as possible, describing who helped create the image, where it was shot and when, what was imaged and how. . Figure 4: Smartphones can automatically capture place/ time data for sessions in images, essential on location. 7.7 Reducing Risk Through Chained Embedded Provenance We have avoided describing backup strategies or file management in this tutorial, as these topics are amply covered elsewhere. You will need to have a robust backup system in place that covers each stage in production, from camera to final archive. For an excellent description of backup strategies and PIE workflows, see The Dam Book by Peter Krogh [Kro09]. We conclude by describing a real world example dealing with the challenge of image sequences in order to produce a robust digital surrogate of a painting (Figure 5). We are producing an RTI, a Reflectance Transformation Image, from a series of 48 images (for a description of RTI, see [MMSL06]. This RTI was produced in partnership with Cultural Heritage Imaging (CHI) and the Fine Arts Museums of San Francisco (FAMSF). The resulting RTI is derived from the image data in the raw captures, therefore we want to embed pointers to all of the documents required to reproduce the RTI in the future, including the images, processing logs, object information and desk instructions. M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums The Camera RAW files are converted to Digital Negative Format (DNG) using Adobe Camera Raw (see Figure 3) and assigned a unique filename. We embed the sequence series IDs into each image, binding them together. All process history, processing information and museum object information available is structured and embedded in the image XMP. Provenance documents are gathered in one location. The documents are parsed into a single XML document, from which we can extract information blocks to embed in the DNG files through XMP. At this stage, each image in the sequence contains a light but coherent set of information about its sibling files in the sequence, plus preliminary information about the resulting RTI. The RTI is processed, and its processing notes are synced with the provenance documents, and pushed into the DNG images. The finalized XML is linked to the image sequence and RTI, and the RTI XMP points to the XML file available online. As additional derivatives are created – JPEGs, movies, related RTI files – or more information about the object is discovered, the XML document is updated, providing future users with up-to-date information and trustworthy provenance of the file they are evaluating. Most importantly, the files are carriers of their own life histories, securely weaved into the XMP and relatively safe from the deleterious effects of the application-centric iPad effect. References [ATP10] Ashley, M., R. Tringham, C. Perlingieri: Last House on the Hill: Digitally Remediating Data and Media for Preservation and Access. Journal on Computing and Cultural Heritage. Friedlander. Ed. (in press) [DB10] Dunne, T., Bollacker, K.: Avoiding a Digital Dark Age. In American Scientist. Accessed Aug 2, 2010. http://www.americanscientist.org/issues/num2/2010/3/ avoiding-a-digital-dark-age/1 [Dil10] "Apple reinventing file access, wireless sharing for iPad" Retrieved, Aug 3 2010, from http://www.roughlydrafted.com/2010/01/29/apple- reinventing-file-access-wireless-sharing-for-ipad/ [DI08] Doerr, M. and D. Iorizzo: The dream of a global knowledge network—A new approach. ACM Journal on Computing and Cultural Heritage, Vol. 1, No. 1. 2008. [Kro09] Krogh, P.: The Dam Book. Digital Asset Management for Photographers. O’Reilly Press. 2009 [Kun08] Kunze, J.: New Knowledge and Data Preservation Initiative. California Digital Library (2008) [RB10] Riley, J., Becker, D: Seeing Standards. Accessed Aug 2, 2010. http://www.dlib.indiana.edu/~jenlrile/metadatamap/ [MMSL06] Mudge M., Malzbender T., Schroer C., Lum M.: New reflection transformation imaging methods for rock art and multiple-viewpoint display. In VAST: International Symposium on Virtual Reality, Archaeology and Intelligent Cultural Heritage (Nicosia, Cyprus, 2006), Ioannides M., Arnold D., Niccolucci F., Mania K., (Eds.), Eurographics Association, pp. 195–202. ********************************************* M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums 8. Photogrammetric Principles, Examples and Demonstration Tutorial Presenters: Neffra Matthews, Tommy Noble U.S. Department of the Interior, Bureau of Land Management, National Operations Center 8.1 Introduction Since its inception the principles of photogrammetry, deriving measurements from photographs, have remained constant. Even today, when following the fundamentals, mathematically sound and highly accurate results may be achieved. While requirements, such as overlapping (stereoscopic) images remain, technological advances in digital cameras, computer processors, and computational techniques, such as sub-pixel image matching, make photogrammetry an even more portable and powerful tool. Extremely dense and accurate 3D surface data can be created with a limited number of photos, equipment, and image capture time. An overlap of 60% has traditionally been required for analytical photogrammetry providing a very strong base to height ratio. Now, because of the highly automatic image correlation algorithms available today, a perfect photogrammetry sequence of photos would be 66% overlapping images. Points matched in at least three images, provide a high level of redundancy and a more robust solution, tri-lap. While there are a number of commercial software products available (3DM, PhotoModeler, 2D3, Alice labs PhotoStruct, to name a few) the basic principles for capturing robust stereoscopic images and the photos needed for camera calibration remain consistent. 8.2 Basics of Stereoscopic Photogrammetry A crucial element of successful photogrammetric process is obtaining “good” photographs. Herein the term good refers to a series of sharp pictures that have uniform exposure, high contrast, and fill the frame with the subject. The final accuracy of the resulting dense surface model is governed by the image resolution, or ground sample distance (GSD). The GSD is a result of the resolution of the camera sensor (higher is better), the focal length of the lens, and the distance from the subject (closer is better). The resolution of the images is governed by the number of pixels per given area and the size of the sensor. The camera should be set to aperture priority (preferable F8) and the ISO, shutter speed, white balance, and other settings be adjusted to achieve properly exposed images. To obtain the highest order results it is necessary to ensure that focal distance and zoom do not change for a given sequence of photos. This can be achieved by taking a single photo, at the desired Figure 1: Sequence of images needed to capture a basic stereoscopic project and perform a camera calibration. M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums distance, using the autofocus function, than turn the camera to manual focus and tape the focus ring in place. To maintain a consistent 66% overlap, the camera must be moved a distance equivalent to 34% of a single photo field of view (Figure 1). To ensure the entire subject is covered by at least two overlapping photos, position the left extent of the subject in the center of the first frame. Proceed systematically from left to right along the length of the subject and take as many photos as necessary to ensure complete stereo coverage. 8.3 Camera Calibration Sequence The main purpose of the camera calibration is to determine and map the distortions in the lens with respect to the sensor location. This can be accomplished most effectively when there are a large number of auto-correlated points in common between the stereoscopic images and the additional set of calibration photographs. The camera calibration photographs must be captured at the same settings as the stereo photos described in section 2. At least four additional photos are required; two taken with the camera physically rotated 90° to the previous line of stereoscopic photos and two additional photos with the camera rotated 270° (Figure 1). The additional four camera calibration photos may be taken at any location along the line of stereo photographs, however the best results occur in areas where the greatest number of autocorrelated points may be generated. Figure 1 illustrates the sequence of images needed to capture a basic stereoscopic project and perform a camera calibration. The dashed outlines highlight a pair of photographs that overlap each other by 66%. Arrows, indicating the rotation at which photos were taken, show a 0 degree (or landscape) orientation. The solid outlines highlight the 4 photos required for the camera calibration process. These photos are taken at 90 degrees (or portrait) orientation. By stacking the camera calibration photos over the previously taken stereoscopic photos, maximum benefit will be achieved. Areas of minimum overlap are illustrated by shading of the photographs. Note the calibrated target sticks positioned along the subject. 8.4 Adding Measurability In addition to maintaining a proper base to height for 66% overlap and the camera calibration photo sequence, the next most important component needed to acquire geometrically correct dense surface models is the ability to introduce real world values, or scale, to a project. This is accomplished by simply adding an object of known dimension (meter stick or other object) that is visible in at least two stereo models (three photos). It is preferable to have two or more such objects, to ensure visibility and for accuracy assessment. Calibrated target sticks may be used in addition to, or in place of, the object of known dimension. These objects may then be assigned their proper length during processing. Most photogrammetrically based software products conduct a mathematical procedure knows as a bundle adjustment. Once an object length is established the bundle adjustment passes those measurements to all photos and reduces error in the project. As a result high accuracy may be extended for a long distance along a series of photos allowing for the object of know dimension to be placed so as to not detract visually from the subject. 8.5 Conclusion Capturing photographs for stereoscopic photogrammetric processing may be accomplished in as few as 6 photos for a small subject and can provide extremely dense, high resolution, geometric and orthometrically correct 3D digital data sets. Because of the flexibility of this technique, it is possible to obtain high accuracy 3D data from subjects that are at almost any orientation (horizontal, vertical, above, or below) the camera position. However, it is important to keep the plane of the sensor and lens parallel to the subject and to maintain a consistent height (or distance) from the subject. Although currently, low- or no-cost automatic image matching alternatives are not available for typical analytical photogrammetric processing, the same sub-pixel matching algorithms used in structure for motion and gigapixel panoramas will undoubtedly lead to this niche being filled. Regardless, incorporating the basic photogrammetric image capture steps described above (correct base to height, addition of camera calibration photos, and adding an object of know dimension) to other capture methods will undoubtedly increase their geometric accuracy. In addition, photogrammetric image capture steps can be done in concert with the RTI image capture process[MMSL06]. This combination will result in a geometrically correct RTI when the photogrammetric preprocessing is completed. In addition, as the dense surface model is derived directly from the images, there is no need for further registration processing. The resulting dense surface model and image texture may be output directly to an OBJ or XYZRGB format. Additional processing of these files may produce .ply, .las, .stl,, as well as, a variety of solid model printouts. It is also possible to use gigapixel techniques to produce mosaiced stereoscopic "pairs" of images resulting in spectacularly dense surface models. These very dense surface models may be processed through the chain resulting in Algorithmic Rendering entire rock art panels. References [Mathews08] Matthews, N. A. 2008. Aerial and Close- Range Photogrammetric Technology: Providing Resource Documentation, Interpretation, and Preservation. Technical Note 428. U.S. Department of the Interior, Bureau of Land Management, National Operations Center, Denver, Colorado. 42 pp. M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums [MMSL06] MUDGE M., MALZBENDER T., SCHROER C., LUM M.: New reflection transformation imaging methods for rock art and multiple-viewpoint display. In VAST: International Symposium on Virtual Reality, Archaeology and Intelligent Cultural Heritage (Nicosia, Cyprus, 2006), Ioannides M., Arnold D., Niccolucci F., Mania K.,(Eds.), Eurographics Association, pp. 195–202. ********************************************* 9. 3D models from un-calibrated images and use of MeshLab Tutorial Presenter: Matteo Dellepiane Visual Computing Lab, ISTI - CNR, Italy 9.1 Introduction In the last few years, the advancement of active acquisition systems has been impressive. This has happened in terms of both hardware and software improvements. Nevertheless, there is still a tradeoff between the costs and the accuracy in the acquisition of the geometry. This prevents from having a technology which is able to cover a wide range of possible objects. The best solution would be to be able to obtain a dense reconstruction of a scene with a complete "passive" approach. This means that only images of the object could be used to automatically infer a dense geometric information of the object of interest. This would overcome the problem of the need of expertise for the photogrammetric approach. Recently, dense stereo matching approaches have been made fast and robust enough to handle a large number of images, obtaining accurate results. This brings the possibility to apply dense stereo matching in the context of a lot of applications, from archeological excavation to visualization and restoration support, in Cultural Heritage 9.2 Dense stereo matching Dense stereo matching approaches derive from the success of structure-from-motion techniques [BL05, SSS06] which produce sparse feature points, and have recently been demonstrated to operate effectively. Instead of obtaining a discrete depth map, as is common in many stereo methods [SS01], dense stereo matching aims at reconstructing a sub- pixel-accurate continuous depth map. Essentially, stereo matching is applied on each pixel of the images considering all the possible couples among them, starting from an initial estimate provided by some descriptors (e.g. SIFT features). The method proved to be robust enough to be applied also to community photo collections [GSC07], obtaining impressive results. Moreover, it was recently shown that it can obtain a high degree of accuracy [FP10]. While usually completely automatic, dense stereo matching systems are computationally intensive. Recently, the code for some of them has been made available [SSS06], but most of the practical solutions for the reconstruction of geometry from images are based on Web services [VG06]. This permits to use the resources of big clusters of computers, so that the whole matching operation can be completed within minutes. Although reconstruction from un-calibrated images can’t be considered as a measurement system, it’s becoming a widely used approach, especially in the context of archaeological excavations [HPA10], where the interpretation of data is more important than the geometric accuracy. Figure 1: A snapshot of the interface which loads Arc3D data in Meshlab 9.3 Geometry from images using a Web Service and an Open Source Tool The Arc3D web-service (http://www.arc3d.be) [VG06] was one of the result of the Epoch Network of excellence, which aimed at improving the quality and effectiveness of the use of Information and Communication Technology for Cultural Heritage. Using a simple upload tool, and following some guidelines about how to acquire the data, it’s possible to obtain a 3D model of an object of interest. Moreover, the output of the web-service (which is made of a group of depth maps associated to each uploaded image, together with a quality map related to each pixel of each image), can be loaded and processed through MeshLab (http:// meshlab.sourceforge.net/) [CCC08], an open source, portable, and extensible system for the processing and editing of unstructured 3D triangular meshes. Figure 1 shows how the Arc3D output data are visualized in the context of MeshLab. Using the interface, it’s possible to select only a subgroup of images from which the final geometry, and define a number of useful operations (masking, hole filling, smoothing) to improve the quality of the result. Figure 2 top shows the output of the geometry reconstruction. Using MeshLab, it’s also possible to further improve the quality of the 3D model by applying a simple pipeline, made of three steps: • Mesh Cleaning: with simple editing and processing tool, unwanted data and main geometric artifacts are removed. M. Mudge, C. Schroer, et al. / Photography-based Digital Imaging Techniques for Museums • Remeshing: using a reconstruction method (e.g. Poisson approach) the geometry is re-created, by preserving the geometric features and removing the typical noise generated by the generation from images. • Color transfer: a filter permits to transfer the color information from the original to the reconstructed model. Figure 2: The model produced using a subset of the original images, before and after the processing pipeline of MeshLab Figure 2 bottom shows the model after applying the pipeline: while the geometric features are preserved, the mesh is cleaner and more complete. 9.4 Conclusion The use of dense stereo matching techniques for the reconstruction of 3D models from images is a very promising direction of work especially for the application in the context of Cultural Heritage. Preliminary but structured test cases are currently under development, especially for the support of archaeological excavations. Nevertheless, the limitations of the approach must be made clear, in order to understand the possible applicability a usefulness of the method. The main limitation are essentially: • Kind of objects and acquisition environment: in order to exploit dense stereo matching at its best, there are several prerequisites on the material of the object, the photographic campaign strategy and the external environment. This can prevent from being able to acquire certain kind of objects. • Accuracy and scaling: since the system starts from uncalibrated images, the scale of the object isn’t known in advance: hence, a scaling operation is needed. Moreover, in- depth testing of the accuracy of the method should be necessary to show that this can be a low-cost alternative to 3D Scanning. In conclusion, geometry reconstruction from images will be probably applied massively in the next future in the context of Cultural Heritage. The availability of open source tools to process and present the data can boost its usefulness for the whole community. References [BL05] BROWN M., LOWE D. G.: Unsupervised 3d object recognition nd reconstruction in unordered datasets. In 3DIM’05: Proceedings of the Fifth International Conference on 3-D Digital Imaging and Modeling (Washington, DC, USA, 2005), IEEE Computer Society, pp. 56–63. [CCC08] CIGNONI P., CALLIERI M., CORSINI M., DELLEPIANE M., GANOVELLI F., RANZUGLIA G.: Meshlab: an open-source mesh processing tool. In Sixth Eurographics Italian Chapter Conference (2008), pp. 129–136. [FP10] FURUKAWA Y., PONCE J.: Accurate, dense, and robust multiview stereopsis. Pattern Analysis and Machine Intelligence, IEEE Transactions on 32, 8 (Aug. 2010), 1362 –1376. [GSC07] GOESELE M., SNAVELY N., CURLESS B., HOPPE H., SEITZ S.: Multi-view stereo for community photo collections. pp. 1–8. [HPA10] HERMON S., PILIDES D., AMICO N., DŠANDREA A., IANNONE G., CHAMBERLAIN M.: Arc3d and 3d laser scanning a comparison of two alternate technologies for 3d data acquisition. In CAA 2010, Granada, Spain (2010). [ SS01] SCHARSTEIN D., SZELISKI R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision 47 (2001), 7–42. [SSS06] SNAVELY N., SEITZ S. M., SZELISKI R.: Photo tourism: exploring photo collections in 3d. In SIGGRAPH’06: ACM SIGGRAPH 2006 Papers (New York, NY, USA, 2006), ACM, pp. 835–846. [VG06] VERGAUWEN M., GOOL L. V.: Web-based 3d reconstruction service. Mach. Vision Appl. 17, 6 (2006), 411–426.