The 4th International Laboratory for the History of Science
Art, Science and Techniques of Drafting in the Renaissance
24 May - 1 June 2001
Florence and Vinci, Italy

Organized by Istituto e Museo di Storia della Scienza

ANTONIO CRIMINISI
Microsoft Corporation, One Microsoft Way, Redmond, WA 98052-6399

The Virtual Trinity


 Abstract

This presentation describes new computer techniques to construct realis-tic, three-dimensional models from paintings. Once constructed the 3D environments can be interactively experienced, in a virtual reality fashion.
     The algorithms described are drawn from the vast Computer Vision literature, notably from the work on Single View Metrology and Three-Dimensional Visual Reconstruction. We show how those algorithms can be easily adapted to the needs of History of Art, providing it with novel, rigorous and powerful ways of analyzing the geometry of the painted scene and interactively explore its third dimension. We, also, demostrate simple geometric methods to evaluate the artist’s perspective skills.
       The reconstruction techniques described in this presentation have been applied to many paintings from the Italian Renaissance and later periods. However, here, we focus on reconstructing Masaccio’s florentine fresco “Trinità”. We also address the problem of the ambiguous reconstruction of the Trinity’s architectonical structure, namely the computation of the correct shape of the vault coffers and the base plane.
       Convincing virtual walks and fly-throughs have been created and will be shown during the presentation.


Computer Vision and Art History

Possibly Masaccio’s most famous fresco, “Trinità” has been analysed over and over again by many in the past. This presentation inves-tigates new, fast and accurate ways of modelling the three-dimensional geometry of the represented scene. Once modelled, the scene is automatically reconstruc-tructed into a virtual reality model.

Projective Geometry is the powerful mathematical theory at the basis of our techniques. In the past twenty years the Computer Vision community has made heavy use of Projective Geometry to develop flexible and robust algorithms ca-pable of generating three-dimensional models of objects and large-scale envi-ronments directly from input photographs . Projective Geometry can be interpreted as an algebraic modelling of Linear Perspective. For instance, Alberti’s window intuition can be translated directly into an algebraic projective mapping of three-dimensional points onto a planar surface (e.g. the canvas, the retina in our eyes or the camera film).

Traditionally, researchers in Computer Vision have concentrated on the idea of generating 3D models from two or more photographs of a real scene (stere-oscopy). More recent research has led to new algorithms for recovering 3D shape from single input images. This presentation focuses on this spe-cific technique and applies it to the particular case of paintings.

Single View Metrology comprises non-invasive, rigorous mathematical tech-niques to measure dimensions of objects and distances on planar surfaces, as well as automatic detection of vanishing points (and vantage points) and van-ishing lines (e.g. the horizon). Thanks to their algebraic nature these algorithms have been incorporated in an user-friendly, interactive software ap-plication capable of generating geometrically-consistent models from an input image of a painting.

Once the virtual model has been constructed it can be observed from different viewpoints; convincing animations can be created and 3D environments explored in a virtual reality fashion (e.g. inside a “virtual cave” or wearing a “virtual reality helmet” etc.). This provides a new and exciting way of experiencing art, by diving into the paintings third dimension.

Vision-based algorithms extract the geometric information and the related “texture mapping” directly from the input image (the painting). Therefore, the generated models show a very natural and realistic appearance, as supposed to the cold and artificial look of CAD models.

Our techniques apply to any perspective image (e.g. a photograph or a per-spective painting). Consequently, when applied to non-perspective paintings such as cubist or abstract art it cannot generate a geometrically-consistent 3D model. This feature provides a powerful way of assessing to what extent an artist’s perspectival skills conform with the mathematical rules of Linear Per-spective.


The Trinity’s third dimension

Our reconstruction algorithms have been applied to many paintings from the Italian Renaissance and later periods. Here we focus on Masaccio’s Trinity.

Particular attention is paid to the accuracy of the computed geometry. A new way of interpreting and visualizing the ambiguity in the Trinity’s depth is described. There is an infinite number of geometrically-consistent re-constructions of the Trinity; amongst those, two solutions are more likely than others. We show how the shape of vault coffers and the floor change according to the assumptions made and that all the possible solutions are related to each other by a simple geometric transformation: an affine mapping along the direction per-pendicular to the plane of the fresco. This transformation can be implemented algebraically with a simple matrix product.

The last part of this talk presents examples of 3D virtual models and ani-mations generated from Masaccio’s “Trinità” as well as Piero della Francesca’s “Flagellazione”, Steinwick’s “St Jerome” and Vermeer’s “The Music Lesson”. It is very interesting to compare our results with the ones obtained by manual techniques or CAD tools.


Conclusions and work ahead

Without any doubt Computer Vision can provide artists and art historians with new powerful tools for analyzing and understanding visual art. A mathematical representation of Linear Perspective has allowed us an accurate three-dimensional reconstruction of paintings and a rigorous analysis of the perspectival skills of the artist.

Currently we are aiming at increasing the algorithms level of automation and improving the shape and look of reconstructed human figures. Other interesting and challenging ideas for future research include: analysis of colour, textures and shading, automatic filling of occluded regions, automatic localization of light sources.

Hopefully these preliminary research results will pave the way for a new, enriching co-operation between two very different fields such as History of Art and Computer Vision.


References

A. Criminisi. Accurate Visual Metrology from Single and Multiple Uncalibrated Images. Distinguished Dissertation Series. Springer-Verlag London Ltd., June 2001. to appear.

A Criminisi, I. Reid, and A Zisserman. A plane measuring device. Image and Vision Computing, 17(8):625–634, 1999.

A. Criminisi, I. Reid, and A. Zisserman. Single view metrology. International Journal of Computer Vision, 40(2), November 2000. ISSN: 0920-5691.

M. De Mey. Perspektief in 3-D. Gent Universiteit, 8ste jaargang(3):14–17, Decem-ber 1993.

O. D. Faugeras. Three-Dimensional Computer Vision: a Geometric Viewpoint. MIT Press, 1993.

J. V. Field. The Invention of Infinity, mathematics and arts in the Renaissance. Oxford University Press, 1997.

J. V. Field, R. Lunardi, and T. B. Settle. The Perspective Scheme of Masaccio’s Trinity Fresco. Leo S. Olschki Editore, Firenze, 1989.

Y. Horry, K. Anjyo, and K. Arai. Tour into the picture: Using a spidery mesh interface to make animation from a single image. In Proceedings of the ACM SIG-GRAPH Conference on Computer Graphics, pages 225–232, 1997.

R. I. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. Cam-bridge University Press, ISBN: 0521623049, 2000.

M. Kemp. The Science of Art. Yale University Press, New Haven and London, 1989.

D. Liebowitz, A. Criminisi, and A. Zisserman. Creating architectural models from images. In Proc. EuroGraphics, volume 18, pages 39–50, September 1999.