In this paper we present a sketch modelling system which we call Stilton. The program resembles a desktop VRML browser, allowing a user to navigate a 3D model in a perspective projection, or panoramic photographs, which the program maps onto the scene as a ‘floor ’ and ‘walls’. We place an imaginary 2D drawing plane in front of the user, and any geometric information that user sketches onto this plane may be reconstructed to form solid objects through an optimisation process. We show how the system can be used to reconstruct geometry from panoramic images, or to add new objects to an existing model. While panoramic imaging can greatly assist with some aspects of site familiarisation and qualitative assessment of a site, without the addition of some foreground geometry they offer only limited utility in a design context. Therefore, we suggest that the system may be of use in ‘just-intime’ CAD recovery of complex environments, such as shop floors, or construction sites, by recovering objects through sketched overlays, where other methods such as automatic line-retrieval may be impossible. The result of using the system in this manner is the ‘sketching of space ’ — sketching out a volume around the user — and once the geometry has been recovered, the designer is free to quickly sketch design ideas into the newly constructed context, or analyse the space around them. Although end-user trials have not, as yet, been undertaken we believe that this implementation may afford a user-interface that is both accessible and robust, and that the rapid growth of pen-computing devices will further stimulate activity in this area. Key words: Sketch design, Line-drawing interpretation, Image-based rendering 1 Introduction — Greenfields and Brownfields? Sophisticated three-dimensional CAD modelling systems are now routinely employed to ensure the cost effective design and construction of a wide range of commercial and industrial facilities. The spatially indexed database systems