WO2007015059A1

WO2007015059A1 - Method and system for three-dimensional data capture

Info

Publication number: WO2007015059A1
Application number: PCT/GB2006/002715
Authority: WO
Inventors: James Paterson; Andrew Fitzgibbon
Original assignee: Isis Innovation Limited
Priority date: 2005-08-02
Filing date: 2006-07-20
Publication date: 2007-02-08
Also published as: GB0515915D0

Abstract

There is described a method of obtaining three-dimensional data relating to a physical scene, comprising (a) projecting a predetermined two-dimensional finite array of shapes onto the scene, the projected array having uniqueness properties in at least one dimension thereof; (b) capturing an image of the array projected onto the scene; (c) deriving correspondences between the shapes in the captured image to the finite array of projected shapes, based upon the uniqueness properties; and (d) obtaining three-dimensional data points from the correspondence between the projected array and the captured image array. There is also described a system for obtaining three-dimensional data relating to a physical scene.

Description

METHOD AND SYSTEM FOR THREE-DIMENSIONAL DATA CAPTURE

FIELD OF THE INVENTION

The present invention relates to a method and system for obtaining three-dimensional data relating to a physical scene .

BACKGROUND OF THE INVENTION

Three-dimensional capture techniques are used to obtain three-dimensional data relating to a physical scene or object based on two-dimensional images of the scene or object. These techniques are becoming increasingly important in the field of computer graphics for applications such as virtual reality and the film industry. Three-dimensional capture techniques can be conveniently categorised into techniques designed for the reconstruction of static scenes, and those designed to capture moving objects (dynamic scenes) . Furthermore, some systems provide near instantaneous (real-time) reconstruction for continual feedback, whilst others rely on offline processing of a captured image sequence.

In some current schemes for three-dimensional capture, multiple images of the scene must be captured under different illuminations. The requirement for multiple images means that corresponding image points must be identified in each of the multiple images (i.e. the stereo correspondence problem) . Furthermore, the need for multiple images with different illuminations imposes the limitation that the scene must remain static or move very slowly during capture. Thus, such techniques are usually only suitable for the reconstruction of static scenes. Furthermore, the use of high resolution capture devices, such as current digital stills cameras, is precluded because of the long delay between exposures.

It is known in the art to employ synchronised stereo cameras which view a structured light pattern projected onto a scene to allow three-dimensional capture of dynamic scenes. In such a scheme, the stereo correspondence problem is simplified because the structured light pattern codes the scene surface points nearly uniquely. However, the difficulty of generating a truly unique coding for every surface point using bar codes or random dots means that strategies such as space-time correlation must be employed to resolve the inevitable ambiguities.

Many of the recent schemes for dynamic capture make use of stripe encoding, where a series of horizontal or vertical lines are projected. However, a difficulty with such schemes is that the codes are of necessity very limited. In order to combat ambiguities caused by surface normal and reflectance variations, only a small number of distinct codes can be formed, so that stripe encoding schemes are prone to considerable ambiguity when obtaining three- dimensional data from only a single two-dimensional image (so called "one-shot operation") .

Other known schemes involve projecting a pattern consisting of a limited number of differently coloured spots, each of which can each be uniquely identified via its near neighbours. However, the projection of coloured spots makes it difficult to acquire data from coloured objects.

The present invention seeks to address these and other such problems with the art . SUMMARY OF THE INVENTION

According to a first aspect of the present invention, there is provided a method of obtaining three-dimensional data relating to a physical scene, comprising (a) projecting a predetermined two-dimensional finite array of shapes onto the scene, the projected array having uniqueness properties in at least one dimension thereof; (b) capturing an image of the array projected onto the scene; (c) deriving correspondences between the shapes in the captured image to the finite array of projected shapes, based upon the uniqueness properties; and (d) obtaining three-dimensional data points from the correspondence between the projected array and the captured image array.

The method therefore enables three-dimensional data relating to the scene to be captured from just one two- dimensional image. The method may be used to capture dynamic scenes since there is no need for camera synchronisation and the method is not limited by slow shutter speeds. Furthermore, the method requires only a single camera and projector so the system is relatively cheap as well as being easy to set up. The step of projecting an array of shapes onto the scene means that data may be acquired from coloured objects. A further benefit of the current invention is that high resolution data is acquired around the edges of each shape.

Advantageously, the projected array has uniqueness properties along mutually non-parallel lines. More advantageously, the mutually non-parallel lines are epipolar lines . By projecting an array of shapes having uniqueness properties along epipolar lines of the camera-projector system, the present invention is able to powerfully disambiguate the correspondence problem (i.e. the array need only have uniqueness properties in one direction) .

Advantageously, the method further comprises obtaining calibration data. More advantageously, the calibration data comprises a fundamental matrix. Still more advantageously, the step of obtaining calibration data further comprises resolving a projective ambiguity.

Advantageously, the step (c) further comprises detecting edges of the shapes from the captured image array. More advantageously, the step of detecting edges of the shapes comprises determining edgels corresponding to intensity gradients within the captured image. Still more advantageously, the step (c) further comprises representing the edges of the imaged shapes as shape vectors . More advantageously again, the step (c) further comprises classifying the shapes using the shape vectors.

Advantageously, the method further comprises projecting training arrays of shapes onto a training scene; capturing training images of the training arrays projected onto the training scene; and comparing the training images with the training arrays to obtain training data. More advantageously, the step (c) further comprises classifying the shapes using the training data.

Advantageously, the step (c) further comprises rectifying the projected array and the captured image.

Advantageously, the step (c) further comprises grouping the imaged shapes according to respective lines of shapes in the projected image, the lines of projected shapes being oriented along lines having uniqueness properties. More advantageously, the method further comprises ordering the groups of shapes. Still more advantageously, the method further comprises aligning the ordered groups of imaged shapes with respective lines of projected shapes using the uniqueness properties. In a preferred embodiment , dynamic programming is used to align the ordered groups of imaged shapes with respective lines of projected shapes. Advantageously, the three dimensional data points are obtained by triangulation.

Advantageously, the method further comprises regularising the three-dimensional data points using a weak smoothness constraint. Advantageously, the shapes are selected from the group consisting of circles, triangles, squares, diamonds and stars .

According to a second aspect of the present invention, there is provided a system for obtaining three-dimensional data relating to a physical scene, the system comprising a projector for projecting a predetermined two-dimensional finite array of shapes onto the scene, the projected array having uniqueness properties in at least one dimension thereof; a camera for capturing an image of the array projected onto the scene; and processing means for obtaining the three-dimensional data from the image.

BRIEF DESCRIPTION OF THE DRAWINGS

An embodiment of the present invention will now be described by way of example with reference to the accompanying drawings in which:

Figure 1 is a camera-projector system according to one embodiment of the present invention;

Figure 2 shows a flow chart describing a method of obtaining three-dimensional data relating to a physical scene in accordance with the present invention; Figure 3 shows a flow chart describing one of the steps in the method illustrated in Figure 2, in further detail;

Figure 4a shows an example of a captured image of an array of shapes projected by the projector of Figure 1; Figure 4b shows the image of Figure 4a, as captured by the camera of Figure 1, the image having been processed to show detected shape edges;

Figure 5 shows three examples of shape vectors obtained from captured images of shapes; and Figure 6 shows a flow chart describing a further one of the steps in the method illustrated in Figure 2, in more detail .

DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT Figure 1 shows an embodiment of a camera-projector system 10 for obtaining three-dimensional data relating to a three-dimensional physical scene 12, in accordance with an embodiment of the present invention. A projector 14 projects a known array of shapes 16 onto the scene 12. A camera 18 captures an image 20 of the array of shapes 16 projected onto the scene 12. Given the correspondence between the shapes in the projected array 16 and the shapes in the captured image 20, it is possible to obtain three- dimensional data relating to the scene 12. The projector 14 and the camera 18 are directed towards the scene 12 at an acute angle relative to one another. In this embodiment, the projector 14 and the camera 18 are located in the same vertical plane. In an alternative embodiment, the camera 18 is displaced horizontally with respect to the projector 14. Figure 2 shows a flow chart of the steps in the method of obtaining the three-dimensional data relating to the scene 12.

As an overview of the method steps, at step 30 of Figure 2, the camera-projector system 10 is calibrated to obtain calibration data concerning the positions and internal parameters of the projector 14 and camera 18. At 32, the known array of shapes 16 is projected onto the scene 12 with the projector 14. At 34, an image 20 of the projected array of shapes 16 is captured with the camera 18. At 36, the shapes in the captured image 20 are classified into shape types (e.g. circle, triangle, etc) . At 38, one- to-one shape correspondences are identified between shapes in the captured image 20 and the shapes in the projected array 16. At 40, the one-to-one shape correspondences are used to obtain three-dimensional data relating to the physical scene 12.

These method steps will now be described in more detail, still referring to Figure 2. At 30, the system is calibrated to determine calibration data comprising positions, orientations and internal parameters (e.g. focal length) of the projector 14 and the camera 18 as 3x4 projection matrices P_P and P_c respectively. The calibration step 30 need only be performed once for a given set up of the camera-projector system 10. The calibration data may be determined by computing a fundamental matrix F of the system and by then resolving a projective ambiguity which remains after computation of the fundamental matrix F. In this embodiment, the fundamental matrix F is computed by projecting a sequence of images in which only a single pixel is illuminated. This enables the camera 18 to capture a corresponding sequence of images of single scene points. The coordinates of the captured image points are determined by image processing. In the i^th such image, the projector pixel at pi=(pxi,pyi) and the scene point detected in the camera 18 at c±=(cxi,cyi) are related by

Thus, a linear constraint is provided on the 3x3 fundamental matrix F. Each captured image provides another such p_± <-» c_± correspondence so that F may be determined from several such correspondences. P_P and Pc are then determined from F up to a projective ambiguity which comprises a choice of coordinates in projective space. The projective ambiguity may be understood by considering the projector 14 and the camera 18 together being moved relative to the scene 12 such that their projective relationship remains the same while their positions and orientations are altered in real space. The projective ambiguity is resolved by locating particular scene points in real space. For example, the projective ambiguity may be resolved by imaging a scene consisting of a calibration object having identifiable features across three axes, e.g. a planar target of known dimensions oriented at varying depths from the camera. A two-dimensional homography H is first derived mapping from the imaged plane of the target to the projector view. Next, using F, the three-dimensional location of, for example, the imaged corners of the planes is resolved up to the unknown projective ambiguity. A three-dimensional homography H_SPACK is then computed between the projectively ambiguous space and the known coordinate frame of the calibration object, which provides a general mapping from ambiguous space to the real space of the planar targets.

After the system has been calibrated at 30, a known array of shapes 16 is projected onto the scene 12 at 32, and an image 20 of the array 16 is captured by the camera 18 at 34. The known array of shapes 16 comprises a predetermined two-dimensional pattern of a finite array of shapes, the array having uniqueness properties along the epipolar lines of the camera projector system. In this embodiment, the epipolar lines are approximately vertical so that the array of shapes 16 has column uniqueness properties. In an alternative embodiment, the camera and projector are located in the same horizontal plane and the epipolar lines of the system are approximately horizontal so that the array of shapes 16 has row uniqueness properties. Further alternative arrangements are also possible.

The shapes in the finite array are selected from a finite number of shape types. The projected shapes are sufficiently distinct that they can be distinguished under moderate to severe distortion. Furthermore, the shapes are simple enough that blurring during image capture does not disguise small details of the shapes. In this embodiment, the shape types are circles, diamonds and triangles. In an alternative embodiment, squares and stars may additionally be used. It will be appreciated that other shape types are also possible. For example, other geometric shapes may be used, or the finite array of shapes may be made up from letters and/or numbers.

After the image 20 has been captured at 34, the shapes in the image 20 are classified into shape types at 36. The step 36 of classifying the shapes in the image 20 into shape types is shown in more detail in Figure 3. At 50, the edges of the imaged shapes are detected. Then, at 52, the edges of the imaged shapes are represented as shape vectors . At 54, the shape vectors are classified into shape types using training data obtained at 56. It will be appreciated that the steps shown in Figure 3 provide one possible method of classifying the shapes into shape types . In alternative embodiments, morphological operators or patch comparison may be used to detect and vectorize the imaged shapes. Nonetheless, the steps shown in Figure 3 are described in more detail below.

The image processing to detect edges of the imaged shapes at 50 is invariant to variations caused by surface reflectance changes and allows accurate localisation of the shape boundaries. One particular edge detection method is described below, however it will be appreciated that other known edge detection methods could be used in alternative embodiments. In this embodiment, a local implementation of the Canny operator is applied to the captured image 20 to determine edgels corresponding to intensity gradients within the captured image 20. The edgels comprise sub-pixel accurate positions, directions and magnitudes. In this embodiment, an array of solid white shapes on a black background is projected to give maximum contrast between the shapes and the background. Alternatively, an array of solid black shapes on a white background may be used. The projection of a black and white array of shapes ensures that the method may be used to obtain three-dimensional data relating to a coloured scene.

Having applied the Canny operator to the captured image 20, the output is an unconnected, non-ordered list of edgels. The next step is to link the edgels into groups corresponding to individual shapes in the captured image 20. Linking edgels into shape edges is a common process which will not be described here, but about 95% of shapes are cleanly detected on average. As an example, Figure 4a is an image 60 of a projected array of shapes, and Figure 4b is a corresponding processed image 62 showing detected shape edges. In this example, the physical scene is a human face. Many image shapes 64 are cleanly detected, other image shapes 66 merge over depth discontinuities, and an image shape 68 near an eyebrow is not detected due to the noisy three-dimensional surface of the eyebrow.

Having detected the edges of the imaged shapes at 50, the edges are next represented as one-dimensional shape vectors at 52. Figure 5 illustrates the step 52 for three example imaged shapes 70, 72 and 74. The groups of edgels corresponding to the individual shapes 70, 72 and 74 are fitted to ellipses which are then transformed into transformed ellipses 76, 78 and 80 respectively by mapping to the unit circle 82. The transformed two-dimensional locations of the points on the transformed ellipses 76, 78 and 80 are then converted to polar coordinates (r,θ). The right hand column of Figure 5 shows graphs 84, 86 and 88 of r against θ for the three transformed ellipses 76, 78 and 80 respectively. For the graphs 84, 86 and 88, θ=0 is chosen to be the point at which r is a maximum. The graphs of r are then sampled at regular intervals in θ to obtain D- dimensional shape vectors. Values of D from 10 to 36 provide reasonable results, but it will be appreciated that other values of D are also possible.

At 54, the shape vectors are classified into shape types using training data obtained at 56. The training data are obtained by using the projector 14 to project training arrays of shapes onto a training scene. For example, the training arrays may comprise projected arrays of shapes of a single shape type. In this embodiment, three training arrays are used: a first training array comprising only circles, a second training array comprising only triangles, and a third training array comprising only diamonds. The camera 18 is used to capture training images of the training arrays projected onto the training scene. Shape vectors are calculated for the imaged training shapes in the same way as described above . Image shape vectors are classified into shape types using a nearest-neighbour classifier based on the shape vectors for the training images. In this embodiment, shapes are labelled with their shape type if a consensus among five nearest neighbours can be reached, or as "unknown" otherwise. Typically, about 95% of imaged shapes are correctly identified in this way.

Having classified the shapes in the captured image 20 into shape types at 36, one-to-one correspondences are identified between shapes in the captured image 20 and shapes in the projected array 16 at 38. The step 38 of classifying the shapes in the image 20 into shape types is shown in more detail in Figure 6. At 90, the projected array 16 and the captured image 20 are rectified. At 92, imaged shapes are grouped according to respective columns of the projected image. At 94, the groups of shapes are ordered into lists. At 96, the ordered lists of shapes are aligned with the known columns of the array of projected shapes 16. Thus, one-to-one shape correspondences are identified between the projected array 16 and the captured image 20.

At 90, the calibrated fundamental matrix F is used to rectify the projected array 16 and the captured image 20 so that the columns of the projected array 16 fall along epipolar lines of the camera-projector system 10. Epipolar lines corresponding to the horizontally central points of the columns in the projected array 16 are then identified. Shapes are then assigned to columns at 92 by scanning along the central column epipolar lines and finding shapes which ^■ intersect the scan lines. Shapes are then ordered along the scan lines by sorting on the vertical position of the shapes ' centroids . Shapes are therefore able to be ordered along scan lines at 94 using only the column uniqueness properties of the array 16. Thus, due to the projected array 16 having uniqueness properties along the epipolar lines of the camera-projector system, and due to the rectification step 90, the method powerfully disambiguates the correspondence problem because the array of shapes 16 only requires uniqueness properties in one dimension thereof. Furthermore, the requirement for uniqueness properties in only one dimension considerably simplifies the construction of the projected array of shapes 16. A small percentage of imaged shapes may have been misclassified, and some imaged shapes may have been classified as "unknown" . In addition, occlusion and/or the limitations of the shape classification step 36 may lead to missing shapes in the image 20. Therefore, the extraction of shapes along scan lines is usually imperfect. Therefore, at 96, the ordered lists of imaged shapes are aligned with known lists of shapes corresponding to the columns of the array of projected shapes 16. The alignment optimisation problem is well known. In this embodiment, the alignment is optimised using a dynamic programming technique. This dynamic programming technique does not form a part of the present invention and will therefore be described only briefly here. The DP problem is visualized in this application via a 2D graph structure, with the list of observed shapes on one axis and known projected shapes on the other. A correspondence between observed and known shape type provides a point on the graph, and typically the point is assigned a score indicating the confidence of the match; the DP task is thus to find the highest scoring path from approximately lower left to the upper right sides of the graph. As no explicit score function is proposed from our classifier, the dynamic programming technique is enhanced using the uniqueness properties of the projected array 16. The uniqueness properties mean that aligned chains of adjacent shapes become less likely to occur randomly as the length of the chain increases. Therefore, the dynamic programming algorithm is written so as to be biased towards longer aligned chains of shapes. In this embodiment, the algorithm is biased towards aligned chains of length greater than two. The output of the dynamic programming algorithm is an optimised set of one-to-one correspondences between shapes in the projected array 16 and shapes in the captured image 20.

At 40, the set of one-to-one shape correspondences is used to obtain three-dimensional data relating to the physical scene 12. This is done by using the one-to-one shape correspondences to obtain one-to-one point correspondences between points in the projected array 16 and points in the captured image 20. Due to the rectification step 90, the epipolar lines intersect the projected shapes at only two points per shape. Similarly, the epipolar lines intersect the imaged shapes at only two points per shape.

The epipolar lines which intersect the projected and imaged shapes are densely sampled to obtain two point correspondences per shape per epipolar line. The point correspondences are then triangulated using the projection matrices P_P and P_c to obtain a dense three-dimensional representation of the shape boundaries up to the projective ambiguity. The three-dimensional representation may be transformed from ambiguous space coordinates to real space coordinates by multiplying by the three-dimensional homography H_SPACE- Thus, the method enables high resolution three-dimensional data to be obtained from a single two- dimensional image.

Having extracted dense three-dimensional data around the boundaries of the shapes, the surface of the scene visible to the camera-projector system is known within the constraints of imaging noise. Some noise may be removed by regularising the three dimensional data using a weak smoothness constraint. Alternatively, the smoothness constraint need not be applied.

As the three-dimensional data is reconstructed, shape- by-shape triangulation is easily performed by connecting three-dimensional data only within shapes, giving a partial surface suitable for rendering on PC graphics hardware. An alternative is to triangulate the complete point set, in which case a complete surface area manifold is presented. In either case, the data presented can be considered as a height map or as a range/depth image.

Although a preferred embodiment of the invention has been described, it is to be understood that this is by way of example only and that various modifications may be contemplated.

Claims

CLAIMS :

1. A method of obtaining three-dimensional data relating to a physical scene, comprising: (a) projecting a predetermined two-dimensional finite array of shapes onto the scene, the projected array having uniqueness properties in at least one dimension thereof;

(b) capturing an image of the array projected onto the scene_/ (c) deriving correspondences between the shapes in the captured image to the finite array of projected shapes, based upon the uniqueness properties; and

(d) obtaining three-dimensional data points from the correspondence between the projected array and the captured image array.

2. The method of claim 1 wherein the projected array has uniqueness properties along mutually non-parallel lines.

3. The method of claim 2 wherein the mutually non- parallel lines are epipolar lines.

4. The method of any preceding claim wherein the method further comprises obtaining calibration data.

5. The method of claim 4 wherein the calibration data comprises a fundamental matrix.

6. The method of claim 5 wherein the step of obtaining calibration data further comprises resolving a projective ambiguity.

7. The method of any preceding claim wherein the step (c) further comprises detecting edges of the shapes from the captured image array.

8. The method of claim 7 wherein the step of detecting edges of the shapes comprises determining edgels corresponding to intensity gradients within the captured image .

9. The method of claim 7 or claim 8 wherein the step (c) further comprises representing the edges of the imaged shapes as shape vectors.

10. The method of claim 9 wherein the step (c) further comprises classifying the shapes using the shape vectors.

11. The method of any preceding claim wherein the method further comprises: projecting training arrays of shapes onto a training scene; capturing training images of the training arrays projected onto the training scene; and comparing the training images with the training arrays to obtain training data.

12. The method of claim 11 wherein the step (c) further comprises classifying the shapes using the training data.

13. The method of any preceding claim wherein the step (c) further comprises rectifying the projected array and the captured image .

14. The method of any preceding claim wherein the step (c) further comprises grouping the imaged shapes according to respective lines of shapes in the projected image, the lines of projected shapes being oriented along lines having uniqueness properties.

15. The method of claim 14 wherein the method further comprises ordering the groups of shapes.

16. The method of claim 15 wherein the method further comprises aligning the ordered groups of imaged shapes with respective lines of projected shapes using the uniqueness properties .

17. The method of claim 16 wherein dynamic programming is used to align the ordered groups of imaged shapes with respective lines of projected shapes.

18. The method of any preceding claim wherein the three dimensional data points are obtained by triangulation.

19. The method of any preceding claim wherein the method further comprises regularising the three-dimensional data points using a weak smoothness constraint.

20. The method of any preceding claim wherein the shapes are selected from the group consisting of circles, triangles, squares, diamonds and stars.

21. A system for obtaining three-dimensional data relating to a physical scene, the system comprising: a projector for projecting a predetermined two- dimensional finite array of shapes onto the scene, the projected array having uniqueness properties in at least one dimension thereof; a camera for capturing an image of the array projected onto the scene; and processing means for obtaining the three-dimensional data from the image .