CN102200881A - Image processing apparatus, image processing method and program - Google Patents

Image processing apparatus, image processing method and program Download PDF

Info

Publication number
CN102200881A
CN102200881A CN201110069689XA CN201110069689A CN102200881A CN 102200881 A CN102200881 A CN 102200881A CN 201110069689X A CN201110069689X A CN 201110069689XA CN 201110069689 A CN201110069689 A CN 201110069689A CN 102200881 A CN102200881 A CN 102200881A
Authority
CN
China
Prior art keywords
unit
image
stack
operating body
processing apparatus
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201110069689XA
Other languages
Chinese (zh)
Other versions
CN102200881B (en
Inventor
柏谷辰起
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Corp
Original Assignee
Sony Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from JP2010068269A external-priority patent/JP2011203823A/en
Priority claimed from JP2010068270A external-priority patent/JP2011203824A/en
Application filed by Sony Corp filed Critical Sony Corp
Publication of CN102200881A publication Critical patent/CN102200881A/en
Application granted granted Critical
Publication of CN102200881B publication Critical patent/CN102200881B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Abstract

The invention provides an image processing apparatus, an image processing method and a program. The image processing apparatus comprises an overlay display location determination unit which determines the location of an object having a predetermined plane or a curved surface in imaging objects of input images based on an environmental map, an overlay display image generating unit which arranges overlay display data and generates overlay display images at a position determined by the overlay display location determination unit, an image overlay unit which overlays the overlay display images on the viewing field of a user, an operation body identification unit which identifies an operation body imaged in the input image, and processing execution unit which conducts a corresponding processing on a project selected based on the location of the operation body identified by the operation body identification unit.

Description

Image processing apparatus, image processing method and program
Technical field
The present invention relates to a kind of image processing apparatus, image processing method and program.
Background technology
Recently, the technology that is called as augmented reality (AR) arouses attention, and is presented to the user by real space being carried out image that imaging obtains and that revise by particular procedure thus.In the AR technology, for example, the useful information about the object in the real space shown in the input picture can be inserted in this image, to export as output image.That is, in the AR technology, usually, the major part of presenting to user's image illustrates real space, and can handle the some parts of image according to application purpose.This specific character forms contrast with the virtual reality that the figure (CG) that uses a computer constitutes the integral body (or most of) of output image.By using the AR technology, for example, can provide advantage such as the situation of understanding real space based on the output image user easily or work support.
In addition, in the AR technology, except relating to the useful information about the object in the real space is inserted through real space is carried out imaging and the technology in the image that obtains, also be present on the user's who watches real space the visual field and will present to user's technology about the useful information of the object in the real space in the mode of stack.In this technology, wait with the user's who watches real space visual field optics synthetic by using half-mirror (half mirror), present useful information to the user about the object in the real space.Equally, under the situation of using this AR technology, for example, can provide such as making the user understand the situation of real space or the advantage that work is supported easily.
In the AR technology, in order to present useful information truly to the user, the situation that computing machine is understood real space exactly is very important.Therefore, developed the technology of the situation that is intended to understand real space, it is as the basis of AR technology.For example, Japanese patent application in early days openly No.2008-304268 following method is disclosed: by use the position that can estimate camera simultaneously and posture and the position of the unique point shown in the image of camera, be called as the technology of location and map structuring (SLAM) simultaneously, dynamically generate the environmental map that expression is present in the three-dimensional position of the object in the real space.Note, at " Real-Time Simultaneous Localization and Mapping with a Single Camera " (Andrew J.Davison, Proceedings of the 9th IEEE International Conference on Computer Vision Volume 2,2003, the ultimate principle of using the SLAM technology of monocular camera is disclosed in pp.1403-1410).
Summary of the invention
Simultaneously, as the compact display apparatus that is installed on the head, head mounted display (HMD) is widely used.Be equipped with the technology of the HMD of camera as use, can illustrate following technology: it is used to use image processing apparatus, by the AR technology modification by the image of camera imaging and show by the amended image of HMD, so that the user watches amended image.The function that this HMD has can for example realize by video transmission-type HMD.In addition, for example, also have following technology: it is used for the image by camera imaging is used as the source, generates the additional information image by image processing apparatus by the AR technology, and in the visual field, use the synthetic additional information images that generated of optics such as half-mirror, so that the user watches image.For example, this function of HMD can realize by optical transmission type HMD.By using these technology, the user can easily understand the situation of real space, and the work support based on output image can be provided.As being used to import the device that makes image processing apparatus carry out the operation of handling, for example can suppose input media such as keyboard or mouse.
Yet, though for example understand the advantage of using HMD, promptly, can save the trouble of user's input operation, but have following problem: when with input media (such as, keyboard or mouse) when acting on the device of input operation, cause bringing the trouble of input operation to the user.Thereby, reduced the advantage of using HMD.
In view of the above, be desirable to provide a kind of image processing apparatus, image processing method and program, it is novel and improved, and it is being convenient to user input operation in the configuration with the stacked system display message by HMD on by the visual field of AR technology the user.
According to embodiments of the invention, a kind of image processing apparatus is provided, comprising: characteristic storage unit, the characteristic of the feature of the outward appearance of its storage representation object; Stack video data storage unit, its storage stack video data and item location, wherein, the stack video data is the source that is superimposed upon the image on user's the visual field, and item location is the position that constitutes the project of stack video data; The environmental map generation unit, it generates the environmental map that expression is present in the position of the one or more objects in the real space based on by using imaging device that real space is carried out imaging input picture that obtains and the characteristic that is stored in the characteristic storage unit; Stack display position determining unit, it is based on environmental map, determines to have the position of the object of predetermined plane or curved surface from the object of imaging input picture; Stack display image generation unit, it is provided with the stack video data by the position at the object of being determined by stack display position determining unit, generates the stack display image; The image overlay unit, its display image that will superpose is superimposed upon on user's the visual field; The operating body recognition unit, it is identified in the operating body of imaging in the input picture; And the processing execution unit, it is carried out and the corresponding processing of selecting based on the position of the operating body that is identified by the operating body recognition unit of project.
The operating body recognition unit can be identified as operating body with the foot of imaging in input picture.
The operating body recognition unit can be carried out as the footwear registration image of the image of the footwear of registering in advance and the coupling between the input picture, and determine footwear with footwear registrations images match in input picture during imaging when the operating body recognition unit, the operating body recognition unit is identified as operating body with these footwear.
When the user is worn over imaging device on his/her head, the operating body recognition unit can determine that whether the foot of imaging in input picture is from entering with the immediate limit of user among each limit that constitutes input picture, and determine foot when entering with the immediate limit of user when the operating body recognition unit, the operating body recognition unit is identified as operating body with foot.
The operating body recognition unit can determine to have passed through in advance whether imaging in input picture of footwear predetermined labels, that have mark, and when the operating body recognition unit determine have mark footwear in input picture during imaging, the operating body recognition unit is identified as operating body with these footwear.
The processing execution unit can determine whether the touch sensor that is attached to foot detects contact, and detects when contact when touch sensor, and the processing execution unit can be carried out and the corresponding processing of selecting based on the position of foot of project.
The processing execution unit can determine whether the operating body that is identified by the operating body recognition unit has stopped scheduled time slot in essentially identical position, and when the processing execution unit is determined operating body and has been stopped described scheduled time slot in essentially identical position, the processing execution unit can carry out with based on the position of operating body and the corresponding processing of selecting of project.
Stack video data storage unit can be stored in other stack video datas that show after the stack video data.When by the option of processing execution unit, stack display image generation unit can generate new stack display image by other stack video datas further are set.When having selected project by the processing execution unit, the image overlay unit video data that other can be superposeed further is superimposed upon on the new stack display image.
The operating body recognition unit can be operating body with the hard recognition of imaging in input picture.
The image overlay unit can be superimposed upon the stack display image on user's the visual field, and can make display unit show the stack display image.
Detect the sensor of the degree of tilt of imaging device by use when the image overlay unit, detect with imaging device be worn over user on his/her head make head with the degree of tilt that surpasses predetermined value when downward direction tilts, the image overlay unit can make display unit show the stack display image, and detect the sensor of the degree of tilt of imaging device by use when the image overlay unit, do not detect with imaging device be worn over user on his/her head make head with the degree of tilt that surpasses predetermined value when downward direction tilts, the image overlay unit can limit and show the stack display image, wherein, this demonstration is carried out by display unit.
Stack display position determining unit can will have the position of the object that is defined as having predetermined plane or curved surface in the position of the object on the upwardly extending plane, side of basic horizontal.
Image processing apparatus can also comprise the location estimation unit, and it is based on the position of the point on the object surfaces of being represented by environmental map, estimates the ground in the real space or the position of metope.Stack display position determining unit can be defined as the position on ground the position of object further based on the position by ground in the real space of location estimation unit estimation or metope.
Characteristic can comprise at the one or more points on each object surfaces, represent each point whether probably with real space in ground or the data that contact of metope.The location estimation unit can be further based on the ground in the characteristic estimation real space or the position of metope.
In addition, according to another embodiment of the present invention, a kind of image processing method of being carried out by image processing apparatus is provided, wherein, this image processing apparatus comprises the storage unit of characteristic of characteristic of feature of the outward appearance of storage representation object, the stack video data storage unit of storage stack video data and item location, the environmental map generation unit, stack display position determining unit, the image overlay unit, the operating body recognition unit, and processing execution unit, wherein, this stack video data is the source that is superimposed upon the image on user's the visual field, and this item location is the position that constitutes the project of described stack video data, this image processing method may further comprise the steps: based on by using imaging device that real space is carried out imaging input picture that obtains and the characteristic that is stored in the characteristic storage unit, generate the environmental map that expression is present in the position of the one or more objects in the real space by the environmental map generation unit; Based on environmental map, from the object of imaging input picture, determine to have the position of the object of predetermined plane or curved surface by stack display position determining unit; By position the stack video data is set by stack display image generation unit, generates the stack display image at the object of determining by stack display position determining unit; Be superimposed upon on user's the visual field by the image overlay unit display image that will superpose; Be identified in the operating body of imaging in the input picture by the operating body recognition unit; And carry out and the corresponding processing of selecting based on the position of the operating body that identifies by the operating body recognition unit of project by the processing execution unit.
In addition, according to another embodiment of the present invention, provide a kind of and be used to make computing machine to play the program of the effect of image processing apparatus, this image processing apparatus comprises: characteristic storage unit, the characteristic of the feature of the outward appearance of its storage representation object; Stack video data storage unit, its storage stack video data and item location, wherein, this stack video data is the source that is superimposed upon the image on user's the visual field, and this item location is the position that constitutes the project of the video data that superposes; The environmental map generation unit, it generates the environmental map that expression is present in the position of the one or more objects in the real space based on by using imaging device that real space is carried out imaging input picture that obtains and the characteristic that is stored in the characteristic storage unit; Stack display position determining unit, it is based on environmental map, determines to have the position of the object of predetermined plane or curved surface from the object of imaging input picture; Stack display image generation unit, it is provided with the stack video data by the position at the object of being determined by stack display position determining unit, generates the stack display image; The image overlay unit, its display image that will superpose is superimposed upon on user's the visual field; The operating body recognition unit, it is identified in the operating body of imaging in the input picture; And the processing execution unit, it is carried out and the corresponding processing of selecting based on the position of the operating body that is identified by the operating body recognition unit of project.
According to image processing apparatus, image processing method and the program of above-mentioned present embodiment, can be convenient to user's input operation in the configuration of stacked system display message on by the visual field of AR technology the user by HMD.
Simultaneously, as the compact display apparatus that is installed on the head, head mounted display (HMD) is widely used.Be equipped with the technology of the HMD of camera as use, can illustrate following technology: it is used to use image processing apparatus, by the AR technology modification by the image of camera imaging and show the image of revising by HMD, so that the user watches amended image.The function that this HMD has can for example realize by video transmission-type HMD.In addition, for example, also have following technology: it is used for the image by camera imaging is used as the source, generates the additional information image by image processing apparatus by the AR technology, and in the visual field, use the synthetic additional information images that generated of optics such as half-mirror, so that the user watches image.For example, this function of HMD can realize by optical transmission type HMD.By using these technology, the user can easily understand the situation of real space, and the work support based on input picture can be provided.As the treatment technology that is used for photographic images, can illustrate the information about buildings is superimposed upon technology on the buildings of imaging in the photographic images.
Here, exist near the object (such as, buildings) of imaging photographic images positive usually, and according to this reason, usually near output image shown on the HMD positive with the information of stacked system demonstration about object, such as, about the information of buildings.Therefore, when near photographic images positive with the stacked system demonstration be not directly dependent on imaging in the photographic images object (such as, menu, advertisement, schedule and p.m.entry) data (being hereinafter referred to as " stack video data ") time, the situation that the video data that may superpose is overlapped or approaching with the information relevant with the object of imaging in captured image.When this happens, exist the user to become and be difficult to watch the problem that is superimposed upon the superimposed image that obtains on the photographic images by the video data that will superpose.
In view of the above, be desirable to provide a kind of image processing apparatus, image processing method and program, it is novel and improved, and it can make it possible to watch easily by the video data that will superpose and is superimposed upon the superimposed image that obtains on user's the visual field.
According to embodiments of the invention, a kind of image processing apparatus is provided, comprising: characteristic storage unit, the characteristic of the feature of the outward appearance of its storage representation object; Stack video data storage unit, its storage is as the stack video data in the source of the image on the visual field that is superimposed upon the user; The environmental map generation unit, it generates the environmental map that expression is present in the position of the one or more objects in the real space based on by using imaging device that real space is carried out imaging input picture that obtains and the characteristic that is stored in the characteristic storage unit; Stack display position determining unit, it is based on environmental map, determines to have the position of the object of predetermined plane or curved surface from the object of imaging input picture; Stack display image generation unit, it is provided with the stack video data by the position at the object of being determined by stack display position determining unit, generates the stack display image; And the image overlay unit, its display image that will superpose is superimposed upon on user's the visual field and makes display unit show the stack display image.
When the sensor of the degree of tilt of image overlay unit by use detecting imaging device go out the user who imaging device is worn on his head make head with the degree of tilt that surpasses predetermined value when downward direction tilts, the image overlay unit can make display unit show the stack display image, and when the sensor of the degree of tilt of image overlay unit by use detecting imaging device do not detect the user who imaging device is worn on his head make head with the degree of tilt that surpasses predetermined value when downward direction tilts, the image overlay unit can limit and show the stack display image, wherein, this demonstration is carried out by display unit.
Image processing apparatus can also comprise the self-position detecting unit, and it is based on input picture and characteristic, dynamically the position of detected image treating apparatus.When from by the position of the detected image processing apparatus of self-position detecting unit when the distance of the position of the object of being determined by stack display position determining unit surpasses predetermined value, the image overlay unit can limit and show the stack display image, wherein, this demonstration is carried out by display unit.
Stack display position determining unit can will have the position of the object that is defined as having predetermined plane or curved surface in the position of the object on the upwardly extending plane, side of basic horizontal.
Stack display position determining unit can be defined as at least one the position in ground, desktop and the stair having the position at the object on the upwardly extending plane, side of basic horizontal.
Detect the sensor of the rotation of imaging device by use when stack display image generation unit, detect when imaging device is worn over user on his/her head head is rotated on the direction of basic horizontal, stack display image generation unit can change the position that is provided with of stack video data by the degree according to rotation, moves the stack video data that is provided with in the stack display image.
Image processing apparatus can also comprise the location estimation unit, and it is based on the position of the point on the object surfaces of being represented by environmental map, estimates the ground in the real space or the position of metope.Stack display position determining unit can be defined as the position on ground the position of object further based on the position by ground in the real space of location estimation unit estimation or metope.
Characteristic can comprise at the one or more points on each object surfaces represent each point whether probably with real space in ground or the data that contact of metope.The location estimation unit can be further based on the ground in the characteristic estimation real space or the position of metope.
In addition, according to another embodiment of the present invention, a kind of image processing method of being carried out by image processing apparatus is provided, image processing apparatus comprises the characteristic storage unit of characteristic of feature of the outward appearance of storage representation object, storage is as the stack video data storage unit of the stack video data in the source of the image on the visual field that is superimposed upon the user, the environmental map generation unit, stack display position determining unit, stack display image generation unit and image overlay unit, this image processing method may further comprise the steps: based on by using imaging device that real space is carried out imaging input picture that obtains and the characteristic that is stored in the characteristic storage unit, generate the environmental map that expression is present in the position of the one or more objects in the real space by the environmental map generation unit; Based on environmental map, from the object of imaging input picture, determine to have the position of the object of predetermined plane or curved surface by stack display position determining unit; By position the stack video data is set by stack display image generation unit, generates the stack display image at the object of determining by stack display position determining unit; And be superimposed upon on user's the visual field, and make display unit show the stack display image by the image overlay unit by the image overlay unit display image that will superpose.
In addition, according to another embodiment of the present invention, provide a kind of and be used to make computing machine to play the program of the effect of image processing apparatus, this image processing apparatus comprises: characteristic storage unit, the characteristic of the feature of the outward appearance of its storage representation object; Stack video data storage unit, its storage is as the stack video data in the source of the image on the visual field that is superimposed upon the user; The environmental map generation unit, it generates the environmental map that expression is present in the position of the one or more objects in the real space based on by using imaging device that real space is carried out imaging input picture that obtains and the characteristic that is stored in the characteristic storage unit; Stack display position determining unit, it is based on environmental map, determines to have the position of the object of predetermined plane or curved surface from the object of imaging input picture; Stack display image generation unit, it is provided with the stack video data by the position at the object of being determined by stack display position determining unit, generates the stack display image; The image overlay unit, its display image that will superpose is superimposed upon on user's the visual field, and makes display unit show the stack display image.
According to image processing apparatus, image processing method and the program of above-mentioned present embodiment, can make it possible to watch easily by the video data that will superpose be superimposed upon the superimposed image that obtains on user's the visual field.
Description of drawings
Fig. 1 is the synoptic diagram that illustrates according to the image processing apparatus of embodiment;
Fig. 2 is the key diagram that illustrates according to the example of the input picture that is used for Flame Image Process of embodiment;
Fig. 3 is the block diagram that illustrates according to the ios dhcp sample configuration IOS DHCP of the image processing apparatus of first embodiment;
Fig. 4 illustrates the process flow diagram that detects the example of the flow process of handling according to the self-position of first embodiment;
Fig. 5 is the key diagram that the unique point that is provided with on object is described;
Fig. 6 is the key diagram that unique point is added in explanation;
Fig. 7 is the key diagram of the example of explanation forecast model;
Fig. 8 is the key diagram of the ios dhcp sample configuration IOS DHCP of characterization data;
Fig. 9 is the process flow diagram that the flow example of handling according to the object identification of first embodiment is shown;
Figure 10 A is the diagrammatic sketch that the example of the output image that is generated by image processing apparatus during towards the front as the user who has on imaging device is shown;
Figure 10 B is the diagrammatic sketch of the example of the output image that illustrates when the user looks down to be generated;
Figure 10 C is the diagrammatic sketch that the example of the output image that the project choice by first stratum generates is shown;
Figure 10 D is the diagrammatic sketch that is right after the example of the output image before the project of selecting second stratum;
Figure 10 E is the diagrammatic sketch that the example of the output image that the project choice by second stratum generates is shown;
Figure 11 is the diagrammatic sketch of the example of the data of being stored in the stack video data storage unit that is illustrated in according to first embodiment;
Figure 12 illustrates the process flow diagram that generates the example of the flow process of handling according to the output image of first embodiment;
Figure 13 is the process flow diagram that the example of the flow process of handling according to the project choice of first embodiment is shown;
Figure 14 is the block diagram that illustrates according to the example of the configuration of the image processing apparatus of second embodiment;
Figure 15 is the key diagram of another example of the configuration of characterization data;
Figure 16 is the key diagram of the explanation polygonal example relevant with the characteristic shown in Figure 15; And
Figure 17 is the block diagram of example that the hardware configuration of multi-purpose computer is shown.
Embodiment
Hereinafter, incite somebody to action the preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings.Notice that in this instructions and accompanying drawing, the structural detail with essentially identical function and structure is represented with same reference numerals, and omission is to the repeat specification of these structural details.
In addition, " embodiment " described in the following order.
1. according to the general introduction of the image processing apparatus of embodiment
2. first embodiment
2-1. image-generating unit
2-2. environmental map generation unit
2-3. output image generation unit
2-4. the summary of first embodiment
3. second embodiment
3-1. environmental map generation unit
3-2. output image generation unit
3-3. the summary of second embodiment
4. hardware configuration
<1. according to the general introduction of the image processing apparatus of embodiment 〉
Fig. 1 illustrates the synoptic diagram of image processing apparatus according to an embodiment of the invention.Fig. 1 illustrates environment 1 according to an embodiment of the invention, wherein, exists image processing apparatus 100 is worn over user on his/her head.
With reference to figure 1, in environment 1, there are three object Obj01, Obj02 and Obj03, metope W01 and W02 and ground F0.The corner of object Obj01 between metope W01 and W02.In addition, along metope W01, object Obj02 is in close proximity to object Obj01 and places, and object Obj03 is in close proximity to object Obj02 placement.At environment 1 is under the situation of the Room in the house, and for example, object Obj01, Obj02 and Obj03 are corresponding to many furniture, such as drawer.
Image processing apparatus 100 imaging in environment 1 (it is a real space), and according to the present embodiment carries out image processing of describing after a while.Though the example as the image processing apparatus among Fig. 1 100 shows with lower device, but image processing apparatus 100 is not limited to such example: this device is equipped with the camera on the head that is installed in the user, and revises by the image of camera imaging and with image and export display device D such as head mounted display (HMD) to.For example, image processing apparatus 100 can be a signal conditioning package, such as, can be from obtain personal computer (PC), portable terminal or the digital home appliance of image such as the imaging device of video camera.In addition, in the image processing apparatus 100 needn't and the head of access customer on the camera installed.That is, camera must be held by the user, and still the configuration the camera in being included in image processing apparatus 100 needn't be held by the user, as shown in Figure 1.In addition, environment 1 is not limited to the example shown in Fig. 1, and can be indoor environment or outdoor environment.
Fig. 2 illustrates as the input picture Im01 that passes through the example of image processing apparatus 100 imagings in the environment 1 in Fig. 1.Three object Obj01, Obj02 shown in Fig. 1 and Obj03, metope W01 and W02 and ground F0 appear in input picture Im01.Image processing apparatus 100 obtains for example this input picture, and generates the output image that is not directly dependent on data (hereinafter being called " stack the video data ") acquisition of the object of imaging in the input picture by stack on input picture.As the example of stack video data, can suppose data such as menu, advertisement, schedule, p.m.entry.
Usually exist near imaging input picture positive object (such as, buildings), and according to this reason, usually near output image shown on the display device D positive with the information of stacked system demonstration about object, such as, about the information of buildings.Therefore, when showing when superposeing video data with stacked system near photographic images positive, video data and information mutual superposition relevant with the object of imaging in taken image or approaching situation may superpose.When this happens, the user is difficult to watch by the video data that will superpose and is superimposed upon the output image that obtains on the input picture.In this instructions, to describe for example following technology in detail: watch in order to make that to be superimposed upon the output image that obtains on the input picture by the video data that will superpose easier, the position on identification ground from input picture, and the video data that will superpose is superimposed upon on the position on the ground in the input picture.Because ground exists everywhere, and the unlikely stack of the information relevant with object on the ground, so can improve the situation that is difficult to watch output image.
In addition, illustrate the advantage of using HMD can save the trouble of user's input operation, but in the time will acting on the device of input operation, for the user, cause the trouble of input operation such as the input media usefulness of keyboard or mouse.Thereby, reduced the advantage of using HMD.In this manual, for the processing that makes that the user can easily select the user to want image processing apparatus 100 to carry out, to describe for example following technology in detail: when the user watched stack his/her foot to be positioned at video data the time on the position of expectation project, image processing apparatus 100 was carried out the corresponding processing of project of locating with foot based on input picture.According to this technology, can save the trouble of user's input operation.
In addition, this technology can be applied to image processing apparatus 100, and it shows stack video data with stacked system by the synthetic stack of optics in user's visual field video data on real space.Self-evident, in this case, realized the effect that the user watches composograph to become easier effect and can save the trouble of user's input operation.Hereinafter, by following image processing apparatus is made description as example: its demonstration is superimposed upon the output image that obtains on the input picture by the video data that will superpose.
<2. first embodiment 〉
Fig. 3 is the block diagram that illustrates according to the ios dhcp sample configuration IOS DHCP of the image processing apparatus 100 of first embodiment.With reference to Fig. 3, image processing apparatus 100 comprises image-generating unit 102, environmental map generation unit 110 and output image generation unit 180.
[2-1. image-generating unit]
For example, image-generating unit 102 can be embodied as and have image-forming component (such as, charge-coupled device (CCD) or complementary metal oxide semiconductor (CMOS) (CMOS)) imaging device.Though the part of image-generating unit 102 composing images treating apparatus 100 in the present embodiment, image-generating unit 102 can be arranged on image processing apparatus 100 outsides.Image-generating unit 102 will export environmental map generation unit 110 and output image generation unit 180 to as input picture by the image that generates becoming real space (such as, the environment 1 shown in Fig. 1) to carry out imaging.
[2-2. environmental map generation unit]
Environmental map generation unit 110 build environment maps, this environmental map is illustrated in the position of the one or more objects that exist in the real space etc. based on input picture and the characteristic that is stored in the object of describing subsequently the characteristic storage unit 130 from image-generating unit 102 inputs.As shown in Figure 3, in this embodiment, environmental map generation unit 110 comprises self-position detecting unit 120, characteristic storage unit 130, image identification unit 140, environmental map construction unit 150 and environmental map storage unit 152.
(1) self-position detecting unit
Self-position detecting unit 120 dynamically detects the position of the imaging device of having taken input picture based on input picture and the characteristic that is stored in the characteristic storage unit 130 from image-generating unit 102 inputs.For example, also have under the situation of monocular camera at imaging device, self-position detecting unit 120 can be by being applied in " Real-Time Simultaneous Localization and Mapping with a Single Camera " (Andrew J.Davison, Proceedings of the 9th IEEE International Conference on Computer Vision Volume 2,2003, pp.1403-1410) the SLAM technology of describing in is dynamically determined position and the posture and the position of unique point on the imaging plane of camera of camera at every frame.
The overall flow of the self-position detection processing of the self-position detecting unit 120 that passes through application SLAM technology at first, is described with reference to Fig. 4.Next, describe self-position in detail with reference to Fig. 5 to Fig. 7 and detect processing.
Fig. 4 illustrates the process flow diagram that the self-position that is undertaken by the self-position detecting unit 120 of using the SLAM technology detects the example of the flow process of handling.In Fig. 4, when self-position detected the processing beginning, self-position detecting unit 120 is init state variable (step S102) at first.In this embodiment, state variable is to comprise following vector as element: the translational speed of the position of camera and posture (rotation angle), camera and the position of angular velocity and one or more unique points.Then, self-position detecting unit 120 sequentially obtains input picture (step S112) from image-generating unit 102.Can repeat processing for each input picture (that is every frame) from step S112 to step S118.
At step S114, self-position detecting unit 120 is followed the tracks of the unique point that occurs in input picture.For example, self-position detecting unit 120 detects the sheet (patch) (for example, the little image of 3 * 3=9 pixel around the unique point) that is stored in each unique point the characteristic storage unit 130 in advance from input picture.When the update mode variable, use subsequently in this position of detected, that is, and the position of unique point.
At step S116, self-position detecting unit 120 for example generates the predicted value of the state variable of next frame based on predetermined forecast model.And at step S118, self-position detecting unit 120 uses the predicted value of the state variable that is generated at step S116 and according to the observed reading in the position of the detected unique point of step S114, update mode variable.Self-position detecting unit 120 is carried out processing at step S116 and S118 based on the principle of extended Kalman filter.
As the result of this processing, the value of the state variable that output needle upgrades every frame.The content of each processing of tracking characteristics point (step S114), predicted state variable (step S116) and update mode variable (step S118) is hereinafter more specifically described.
(1-1) tracking characteristics point
In the present embodiment, characteristic storage unit 130 is stored characteristic in advance, and this characteristic is represented the feature of the object corresponding with being present in physical objects in the real space.For example, characteristic comprises little image, that is, the sheet relevant with one or more unique points, each unique point is all represented the feature of the outward appearance of each object.For example, sheet can be the little image that is made of 3 * 3=9 pixel around the unique point.
Fig. 5 illustrates two examples of object and is arranged on unique point (FP) on each object and the example of sheet.Left side object among Fig. 5 be the expression drawer object (a) with reference to Fig. 6.The a plurality of unique points that comprise unique point FP1 are set on this object.And, about unique point FP1 definition sheet Pth1.On the other hand, the right side object of Fig. 5 is the object (with reference to Fig. 6 b) of expression calendar.The a plurality of unique points that comprise unique point FP2 are set on this object.And, about unique point FP2 definition sheet Pth2.
When image-generating unit 102 obtains input pictures, the sheet that self-position detecting unit 120 will be included in parts of images and each unique point shown in the Fig. 6 that is used for being stored in advance characteristic storage unit 130 in the input picture mates.Then, as matching result, self-position detecting unit 120 is specified the position (for example, the position of detected center pixel) that is included in each unique point in the input picture.
Should be noted that for tracking characteristics point (the step S114 among Fig. 4), will not be stored in the characteristic storage unit 130 about the data in advance of all unique points that will follow the tracks of.For example, in the example shown in Fig. 6,, in input picture, detect six unique points (a) with reference to Fig. 7 at time T=t-1 place.Next, when in the position of time T=t place camera or posture when changing, in input picture, only appear at time T=t-1 place and appear at two unique points in six unique points in the input picture.In this case, self-position detecting unit 120 can be reset unique point in the position that the exemplary pixels pattern (pattern) of input picture exists, and is being used for self detecting to handle and using the new feature point of frame subsequently.For example, in the example shown in Fig. 6,, four new feature points (with reference to Fig. 7 b) are set on object at time T=t.This is the feature of SLAM technology, and according to this technology, can reduce the cost that sets in advance all unique points, and can use the unique point of accelerating to improve the accuracy of handling.
(1-2) prediction of state variable
In the present embodiment, the self-position detecting unit 120 state variable X that will express in following equation is as the state variable that is applied to extended Kalman filter.
[equation 1]
X = x ω x · ω · p 1 · · · p N - - - ( 1 )
(z) three-dimensional position in is as expressed in the following equation for x, y at the global coordinate system as the coordinate system that is provided with in real space for the first element representation camera of state variable X in the equation (1).
[equation 2]
X = x c y c z c - - - ( 2 )
In addition, second element of state variable is four-vector ω, and its hypercomplex number (quaternion) that makes corresponding with rotation matrix, as to represent camera posture is as element.Note, can use Eulerian angle to replace hypercomplex number to represent the posture of camera.And, translational speed and angular velocity that the element of state variable and quaternary element are represented camera respectively.
In addition, the The Fifth Element of state variable and element representation subsequently are expressed in following equation, unique point FP i(i=1 ... N) the three-dimensional position p in global coordinate system iNotice that as mentioned above, the quantity N of unique point can change during handling.
[equation 3]
p i = x i y i z i - - - ( 3 )
Self-position detecting unit 120 based on step S102 the value of initialized state variable X or the value of the state variable X that formerly upgraded in the frame, generate the predicted value of the state variable of latest frame.According to the equation of state of the extended Kalman filter of the multidimensional normal distribution as shown in following equation, generate the predicted value of state variable.
[equation 4]
The predicted state variable
Figure BSA00000456633400152
Here, F represents the forecast model relevant with the state transitions of coordinate system, and " a " expression predicted condition.In addition, w represents Gaussian noise and for example can comprise model approximation error, observational error etc.Usually, the mean value of Gaussian noise w is 0.
Fig. 6 is the key diagram that is used to illustrate according to the example of the forecast model of present embodiment.With reference to Fig. 6, illustrate according to two predicted condition in the forecast model of present embodiment.At first, as first condition, suppose that the three-dimensional position of unique point in global coordinate system do not change.That is, the three-dimensional position of supposing the unique point FP1 at time T place is p T, then satisfy following relation.
[equation 5]
p t=p t-1 (5)
Next, as second condition, the motion of supposing camera is a uniform motion.That is,, satisfy following relation for the speed and the angular velocity of camera from time T=t-1 to time T=t.
[equation 6]
X · t = X · t - 1 - - - ( 6 )
ω · t = ω · t - 1 - - - ( 7 )
Self-position detecting unit 120 generates the predicted value of the state variable of latest frame based on the equation of state of expressing in this forecast model and the equation (4).
(1-3) renewal of state variable
Then, self-position detecting unit 120 uses the observation equatioies, for example estimate the observation information predicted from the predicted value of state variable and the actual observation information that obtains as the result of feature point tracking between error.Notice that the v in the equation (8) is an error.
[equation 7]
Observation information
Figure BSA00000456633400163
The observation information of prediction
Figure BSA00000456633400164
Here, H represents observation model.For example, as expressed in the following equation, feature points FP iPosition on imaging surface (u-v face).
[equation 8]
FP iPosition on imaging surface
Figure BSA00000456633400165
Here, provide the position x of camera, the posture ω and the unique point FP of camera iThree-dimensional position P iIn whole elements as state variable X.Then, according to pin hole (pinhole) model, use the following equation unique point FP that derives iPosition on imaging surface.
[equation 9]
λ p ~ i = A R ω ( p i - x ) - - - ( 11 )
Here, λ represents normalized parameter, and A represents camera inner parameter, R ωExpression and the corresponding rotation matrix of hypercomplex number ω, wherein, this hypercomplex number ω represents to be included in the posture of the camera among the state variable X.Characteristic according to the imaging device of taking input picture as expressed in the following equation, provides camera inner parameter A in advance.
[equation 10]
A = - f · k u f · k u · cot θ u o 0 - f · k v sin θ v o 0 0 1 - - - ( 12 )
Here, f represents focal length, the orthogonality of θ presentation video axle (ideal value is 90 degree), k uExpression is along the ratio of the longitudinal axis of imaging surface (being tied to the rate of change of ratio of the coordinate system of imaging surface from world coordinates), k vExpression is along the ratio of the transverse axis of imaging surface, and (u o, v o) be expressed as the center of image planes.
Therefore, can obtain feasible last state variable X by the search condition variable X, error minimum between the prediction observation information (that is, each unique point is in the position on the imaging surface) that this feasible use equation (11) is derived and the result of the step S114 place feature point tracking in Fig. 4.
[equation 11]
The last state variable
Figure BSA00000456633400172
Self-position detecting unit 120 will export environmental map construction unit 150 and output image generation unit 180 to by position x and the posture ω that uses the camera (imaging device) that the SLAM technology dynamically updates in this mode.
(2) characteristic storage unit
Characteristic storage unit 130 is used the storage medium such as hard disk or semiconductor memories, comes the characteristic of the feature of the object corresponding with being present in physical objects in the real space of storage representation in advance.Though the storage unit of characteristic shown in Fig. 3 130 is examples of the part of environmental map generation unit 110, is not limited to this example, and characteristic storage unit 130 can be arranged on environmental map generation unit 110 outsides.Fig. 8 is the key diagram that is used for the ios dhcp sample configuration IOS DHCP of characterization data.
With reference to Fig. 8, the characteristic FD1 of conduct about the example of object Obj1 is shown.Characteristic FD1 comprises object names FD11, from six direction shot image data FD12, sheet data FD13, three-dimensional shape data FD14 and body (ontology) data FD15.
Object names FD11 is the title that can specify respective objects, such as " coffee cup A ".
For example, view data FD12 comprises six sections view data that obtain by the image of taking respective objects from six direction (front, back, left, right, up and following).Sheet data FD13 be in the one or more unique points that are arranged on each object each, the set of little image around each unique point.The object identification that view data FD12 and sheet data FD13 can be used for being undertaken by the image identification unit of describing subsequently 140 is handled.In addition, sheet data FD13 can be used for detecting processing by the above-mentioned self-position that self-position detecting unit 120 carries out.
Three-dimensional shape data FD14 comprises the polygon information of the shape that is used to discern respective objects and the three dimensional local information of unique point.Three-dimensional shape data FD14 can be used for describing subsequently passes through environmental map that environmental map construction unit 150 carries out and makes up to handle and generate for the CG image of each object and handle.
For example, ontology data FD15 is that the environmental map that can be used for assisting to be undertaken by environmental map construction unit 150 makes up the data of handling.In the example depicted in fig. 8, ontology data FD15 represents to contact with object corresponding to desk or dish-washing machine probably as the object Obj1 of coffee cup, and can not contact with the object corresponding to bookshelf.
(3) image identification unit
Image identification unit 140 is used the above-mentioned characteristic that is stored in the characteristic storage unit 130, specified object and appear at corresponding relation between the physical objects in the input picture.
Fig. 9 is the process flow diagram that the example of the flow process that the object identification undertaken by image identification unit 140 handles is shown.With reference to Fig. 9, image identification unit 140 at first obtains input picture (step S212) from image-generating unit 102.Next, the parts of images that image identification unit 140 will be included in the input picture mates with the sheet that is included in one or more unique points of each object in the characteristic, is included in unique point (step S214) in the input picture with extraction.It is not necessarily identical to should be noted that in the object identification undertaken by image identification unit 140 is handled employed unique point and the self-position that is being undertaken by self-position detecting unit 120 detect in the processing employed unique point.Yet when all using public characteristic point in two kinds of processing, image identification unit 140 can be reused the result of the feature point tracking that is undertaken by self-position detecting unit 120.
Next, image identification unit 140 is specified the object (step S216) that appears in the input picture based on the extraction result of unique point.For example, when extracting when belonging to the unique point of an object with high density in certain zone, image identification unit 140 can identify object and be present in this zone.Then, image identification unit 140 exports the object names (perhaps identifier) of specified object and the position of unique point on imaging surface that belong to this object to environmental map construction unit 150 (step S218).
(4) environmental map construction unit
Environmental map construction unit 150 use from the position of the camera of self-position detecting unit 120 inputs and posture, from the unique point of image identification unit 140 inputs on imaging surface the position and be stored in characteristic the characteristic storage unit 130, the build environment map.In this manual, environmental map is the set of data that expression is present in the position (and posture) of the one or more objects in the real space.For example, environmental map can comprise object names corresponding to object, belong to the three-dimensional position of unique point of object and the polygon information that constitutes the shape of object.For example, can come the constructing environment map by from position on imaging surface of the unique point of image identification unit 140 input, obtain the three-dimensional position of each unique point according to above-mentioned pin-hole model.
By making the relational equation distortion of the pin-hole model of expressing in the equation (11), can obtain unique point FP by following equation iThree-dimensional position p in global coordinate system i
[equation 12]
p i = x + λ · R ω T · A - 1 · p ~ i = x + d · R ω T A - 1 · p ~ i | | A - 1 · p ~ i | | - - - ( 14 )
Here, d represents the distance in global coordinate system between camera and each unique point.Environmental map construction unit 150 can calculate this apart from d based on the distance of at least four unique points between the unique point of the position on the imaging surface and each object.As with reference to the three-dimensional shape data FD14 that is included in the characteristic shown in Figure 8, the distance between the unique point is stored in the characteristic storage unit 130 in advance.Should be noted that at Japanese patent application and disclose the computing that discloses equation (14) among the No.2008-304268 in detail in early days apart from d.
After calculating apart from d, the remaining variables on the right side of equation (14) is from the position of the camera of self-position detecting unit 120 input and posture and from the position of unique point on imaging surface of image identification unit 140 inputs, and all these is known.Then, environmental map construction unit 150 calculates from the three-dimensional position of each unique point global coordinate system of image identification unit 140 inputs according to equation (14).Then, environmental map construction unit 150 three-dimensional positions according to each unique point that is calculated make up up-to-date environmental map, and allow the constructed environmental map of environmental map storage unit 152 storages.Should be noted that this moment, environmental map construction unit 150 can use the ontology data FD15 that is included in reference in the characteristic shown in Figure 8, improves the accuracy of the data of environmental map.
Environmental map storage unit 152 is used such as the storage medium of hard disk or semiconductor memory and is stored the environmental map that is made up by environmental map construction unit 150.
[2-3. output image generation unit]
Output image generation unit 180 is by being provided with the stack video data based on the position that has the object of predetermined plane or curved surface among the object of environmental map in input picture, generate the stack display image, and the display image that will superpose is superimposed upon on the input picture, thereby generates output image.As shown in Figure 3, in the present embodiment, output image generation unit 180 comprises stack video data storage unit 181, processing execution unit 182, operating body recognition unit 183, stack display image generation unit 184, stack display position determining unit 186 and image overlay unit 188.
Figure 10 A is the diagrammatic sketch that the example of the output image that is generated by image processing apparatus 100 during towards the front as the user who has on imaging device is shown.Figure 10 B is the diagrammatic sketch of the example of the output image that illustrates when the user looks down to be generated.Figure 10 C is the diagrammatic sketch that the example of the output image that the project choice by first stratum generates is shown.Figure 10 D is the diagrammatic sketch that the example of the output image before the project that is right after in selecting second stratum is shown.Figure 10 E is the diagrammatic sketch that the example of the output image that the project choice by second stratum generates is shown.By using Figure 10 A to Figure 10 E, will the function of each piece that constitutes output image generation unit 180 be described.
(1) stack video data storage unit
Stack video data storage unit 181 is by using storage medium stores stack video data and the item location such as hard disk or semiconductor memory, wherein, this stack video data is the source that is superimposed upon the image on the input picture, and item location is the position of the project of formation stack video data.With reference to Figure 11 the ios dhcp sample configuration IOS DHCP that is stored in the polytype data in the stack video data storage unit 181 is described subsequently.
(2) stack display position determining unit
Stack display position determining unit 186 is based on the environmental map that is stored in the environmental map storage unit 152, determines to have in the object of imaging in input picture the position of the object of predetermined plane or curved surface.For example, stack display position determining unit 186 will have the position of the object that is defined as having predetermined plane or curved surface in the position of the object on the upwardly extending plane, side of basic horizontal.As shown in Figure 10 A, in this embodiment, stack display position determining unit 186 positions with ground F0 are defined as having the position at the object on the upwardly extending plane, side of basic horizontal, but the object that has on the upwardly extending plane, side of basic horizontal is not limited thereto, and can be desktop, stair etc.Note, in the output image Ima shown in Figure 10 A, in boundary B0, extend as ground F0 in the example on the upwardly extending plane, side of basic horizontal.Stack display position determining unit 186 exports determined position to stack display image generation unit 184.
(3) stack display image generation unit
Stack display image generation unit 184 is provided with the stack video data in the position of the object of being determined by stack display position determining unit 186, generates the stack display image thus.Be under the situation of position of ground F0 in the position of the object of determining by stack display position determining unit 186, stack display image generation unit 184 generates the stack display image by in the position of ground F0 the stack video data being set by stack display image generation unit 184.Because the stack display image that is generated by stack display image generation unit 184 is used to generate output image, so stack display image generation unit 184 exports the stack display image that is generated to image overlay unit 188.
(4) image overlay unit
Image overlay unit 188 is superimposed upon on the input picture by the display image that will superpose, and generates output image.Output image Imb shown in Figure 10 B obtains like this: be superimposed upon on the input picture that comprises such as the image of ground F0 by comprising stack display image such as the image of " Email ", " navigation " and " recreation ".Output image Imb shown in Figure 10 B comprises the image as the foot of the example of user's he operating body OP itself.Then, image overlay unit 188 will export display device D (when perhaps needing, exporting another functional unit to) with the output image that this mode generates to, as the result of the processing of being undertaken by image processing apparatus 100.Display device D is the example of display unit and shows output image.
Image overlay unit 188 can be exported output image detecting the stage that the user looks down with the degree of tilt that surpasses predetermined value.Promptly, the sensor of the degree of tilt that detects imaging device lift one's head the user that wears imaging device make head with the degree of tilt that surpasses predetermined value under acclivitous situation downwards, image overlay unit 188 can make display device D show the output image that is generated.In addition, the sensor of the degree of tilt that detects imaging device do not detect lift one's head the user that wears imaging device make head with the degree of tilt that surpasses predetermined value under acclivitous situation downwards, image overlay unit 188 can limit the output image that demonstration generates, wherein, this demonstration is carried out by display device D.More specifically, the sensor of the degree of tilt that detects imaging device do not detect lift one's head the user that wears imaging device make head with the degree of tilt that surpasses predetermined value under acclivitous situation downwards, image overlay unit 188 can not make display device D show the output image that is generated, but can make display device D show input picture.Like this, for example the user towards positive and not have to plan to watch under the situation of stack video data, do not generate the output image that is superimposed with the video data that superposes.
On the plane that superposes the destination (such as, ground F0) under the situation away from user's head, because hypothesis time stack video data on the plane that the stack video data is superimposed upon the stack destination becomes very little, so can the stack video data be superimposed upon on the plane of the destination that superposes.Therefore, for example, surpassing under the situation of predetermined value to the distance the position of the object of determining by stack display position determining unit 186 from position by self-position detecting unit 120 detected image processing apparatus 100, image overlay unit 188 can limit the demonstration output image, wherein, this demonstration is carried out by display device D.More specifically, surpassing under the situation of predetermined value to the distance the position of the object of determining by stack display position determining unit 186 from position by self-position detecting unit 120 detected image processing apparatus 100, image overlay unit 188 can not make display device D show output image, and can carry out such as making display device D show the control of input picture.
Figure 11 is the diagrammatic sketch that the example that is stored in the data in the stack video data storage unit 181 is shown.As shown in figure 11, except the stack video data, stack video data storage unit 181 is gone back storage representation constitutes the content of the processing that item location information and the expression of position of each project of stack video data will carry out when having selected project one or more combinations of contents processing, wherein, each in item location information and the contents processing all is associated with the stack video data.
Figure 12 illustrates the process flow diagram that the output image of being carried out by output image generation unit 180 generates the example of the flow process of handling.In Figure 12, when output image generated the processing beginning, at first, stack display position determining unit 186 obtained environmental maps (step S302) from environmental map construction unit 150.Then, stack display position determining unit 186 determines whether to detect the conversion operations (step S304) of display mode.Though in above-mentioned example, when the user who has on imaging device looks down, carry out the conversion operations of display mode, when the user is input to input media etc. with operation, also can carry out the conversion operations of display mode.Can repeat processing at each input picture (that is every frame) from step S302 to step S314.
Determined not detect in stack display position determining unit 186 under the situation of conversion operations of display mode (in step S304, being "No"), processing proceeds to step S314, and determined to detect in stack display position determining unit 186 under the situation of conversion operations of display mode (being "Yes" in step S304), whether stack display position determining unit 186 is determined to exist in the environmental map that is obtained and be can be used for the plane (step S306) that superposes and show.Though determine in the environmental map that is obtained, whether to exist the plane that can be used for superposeing and show, as mentioned above, also can determine in the environmental map that is obtained, whether to exist to can be used for superposeing the curved surface of demonstration.
Determined in the environmental map that is obtained, not exist in stack display position determining unit 186 and can be used for superposeing (in step S306, being "No") under the situation on the plane that shows, processing proceeds to step S314, and determined to exist in the environmental map that is obtained in stack display position determining unit 186 to can be used for superposeing (being "Yes" in step S306) under the situation on the plane that shows, whether stack display position determining unit 186 determine to can be used for the superposeing area on the plane that shows surpasses threshold value (step S308).Surpass under the situation of threshold value and (in step S308, be "No") at stack display position determining unit 186 determined to can be used for the superposeing area on the plane that shows, processing proceeds to step S314, and surpass under the situation of threshold value and (in step S308, be "Yes") at stack display position determining unit 186 determine to can be used for the superposeing area on the plane that shows, handle proceeding to step S310.
In step S310, stack display image generation unit 184 obtains the stack video data from stack video data storage unit 181, image overlay unit 188 is superimposed upon by the video data that will superpose and generates output image (step S312) on the input picture, and the output image generation is finished dealing with.In step S314, input picture is used as output image (step S314), and the output image generation is finished dealing with.
(5) operating body recognition unit
Operating body recognition unit 183 is identified in the user's of imaging in the input picture operating body OP.In this embodiment, operating body recognition unit 183 will be identified as the example of user's operating body OP by the foot of imaging in input picture, but the part except foot can be identified as user's operating body OP.For example, under stack display position determining unit 186 was defined as the position of desktop having situation in the position of the object on the upwardly extending plane, side of basic horizontal, operating body recognition unit 183 can be user's operating body OP with the hard recognition of imaging in input picture.This is because during imaging, can suppose that the user can easily will be placed on the desktop at his/her hand when desktop in input picture.
In addition, can be the technology of operating body recognition unit 183 user's that is identified in imaging in the input picture operating body OP with the multiple other technologies of hypothesis.For example, coupling between the footwear registration image of the image of the footwear that operating body recognition unit 183 execution input pictures and conduct are registered in advance, and under operating body recognition unit 183 has been determined situation with footwear imaging in input picture of footwear registrations images match, footwear can be identified as user's operating body.
In addition, imaging device is worn under the overhead situation the user, operating body recognition unit 183 can determine that whether the foot of imaging in input picture is from entering with the immediate limit of user among each limit that constitutes input picture.In the output image Imb shown in Figure 10 B, show the state of foot (example of operating body OP) among each limit that constitutes input picture, entering of imaging in input picture with the immediate limit of user (limit on the bottom direction of Figure 10 B).Determine foot under situation about entering with the immediate limit of user at operating body recognition unit 183, operating body recognition unit 183 can be identified as foot user's operating body OP.
In addition, operating body recognition unit 183 also can determine to have the whether imaging in input picture of footwear (it has passed through predetermined labels in advance) of mark.Determine at operating body recognition unit 183 under the situation of the footwear imaging in input picture with mark, operating body recognition unit 183 can be identified as footwear user's operating body OP.
Operating body recognition unit 183 can be discerned user's operating body OP by using normal image recognition technology for example.When identifying user's operating body OP, operating body recognition unit 183 can be grasped the position of user's operating body OP.That is, as the result of the operating body OP that discerns the user, operating body recognition unit 183 exports the position of user's operating body OP to processing execution unit 182.
(6) processing execution unit
Processing execution unit 182 is carried out and the corresponding processing of selecting based on the position of the operating body OP that is identified by operating body recognition unit 183 of project.Project constitutes the stack video data, and in the output image Imb shown in Figure 10 B, each in " Email ", " navigation " and " recreation " is all represented project.The item location information of the position of expression project is used as the item location information shown in Figure 11 and is stored in the stack video data storage unit 181, and in output image, each project has been present in apart from the displacement of stack video data the position by the position of item location information representation.The project corresponding with the position of operating body OP selected in processing execution unit 182 among the project that exists as mentioned above.Output image Imc shown in Figure 10 C is by such acquisition: when the user moves to the position of project " navigation " with user's operating body OP, as with the corresponding processing of project " navigation ", by processing execution unit 182 carry out will be used for option (such as, " website recently " or " convenience store ") the stack video data be superimposed upon processing on the input picture.
Can be that the timing of handling is carried out in processing execution unit 182 with multiple timing hypothesis.The user foot is used as under the situation of operating body OP, processing execution unit 182 can determine whether the touch sensor that is attached to foot detects contact.In that case, detect at touch sensor under the situation of contact, processing execution unit 182 can carry out with based on the position of foot and the corresponding processing of selecting of project.
In addition, processing execution unit 182 can determine whether the operating body OP that is identified by operating body recognition unit 183 has stopped scheduled time slot in essentially identical position.In that case, determine operating body OP in processing execution unit 182 and stopped in essentially identical position under the situation of scheduled time slot, processing execution unit 182 can carry out with based on the position of operating body OP and the corresponding processing of selecting of project.In addition, processing execution unit 182 can determine also whether the operating body OP that is identified by operating body recognition unit 183 is present in the presumptive area that is provided with the corresponded manner with project.In that case, determine operating body OP in processing execution unit 182 and be present under the situation in the presumptive area, for example, processing execution unit 182 can carry out with corresponding to the corresponding processing of the project of presumptive area.
In the output image Imb shown in Figure 10 B, what superpose is the stack video data (comprising the project such as " Email ", " navigation " and " recreation ") of first stratum, and in the output image Imc shown in Figure 10 C, stack be the stack video data (comprising project) of second stratum such as " convenience store " and " recently website ".In this way, the stack video data can be superimposed upon on the input picture with classifying.In this case, stack video data storage unit 181 is stored in other stack video datas that show after the stack video data, and when by 182 options of processing execution unit, stack display image generation unit 184 generates new stack display image by other stack video datas further are set.Then, when by 182 options of processing execution unit, image overlay unit 188 further is superimposed upon other stack video datas on the new stack display image, generates new output image thus.
Output image Imd shown in Figure 10 D is by such acquisition: when the user moves to the position of project " website recently " with user's operating body OP, as with the corresponding processing of project " recently website ", carry out to carry out and be used for the processing of search to the application program of the route of " nearest website " by processing execution unit 182.Output image Ime shown in Figure 10 E illustrates the example of Search Results.
In addition, all items that can also suppose to constitute the stack video data is not suitable for the situation of output image.In this case, the part of stack video data can be superimposed upon on the output image, and under the situation of rotation head on the direction of user in basic horizontal, the extendible portion of stack video data can be superimposed upon on the output image.Promptly, for example, under the situation of imaging device being worn his/her head of rotation on the direction of overhead user in basic horizontal, stack display image generation unit 184 can change the position that is provided with of stack video data according to the degree of rotation in the sensor of the rotation that detects imaging device.Correspondingly, stack display image generation unit 184 can move set stack video data in the stack display image.
Figure 13 is the process flow diagram that the example of the flow process of being handled by the project choice of output image generation unit 180 execution is shown.In Figure 13, when project choice is handled beginning, at first, operating body recognition unit 183 identifying operation body OP (step S402).Then, operating body recognition unit 183 determines whether operating body OP is user's operating body OP (step S404).Can repeat processing at each input picture (that is every frame) from step S402 to step S410.
Operating body recognition unit 183 determines whether the operating body OP that is identified is user's operating body OP (step S404).Determined that at operating body recognition unit 183 the operating body OP that is identified is not under user's the situation of operating body OP (being "No" in step S404), handles being back to step S402.Determined that at operating body recognition unit 183 the operating body OP that is identified is under user's the situation of operating body OP (being "Yes" in step S404), the position (step S406) of operating body recognition unit 183 assigned operation body OP.
Subsequently, processing execution unit 182 determines whether operating body OP has carried out the operation (step S408) of option.Determine under the situation of operation that operating body OP do not carry out option (in step S408, being "No") in processing execution unit 182, handle being back to step S406.Determine under the situation of operation that operating body OP carried out option (being "Yes" in step S408) in processing execution unit 182, the processing (step S410) corresponding with selected item carried out in processing execution unit 182, and project choice is finished dealing with.
[summary of 2-4. first embodiment]
According to the corresponding image processing apparatus 100 of present embodiment, based on expression be present in real space in the environmental map of the corresponding object dimensional position of object, generate by the stack output image that video data obtains that superposes.In view of the above, can generate the output image that the user watches easily.
In addition,, can discern the position of user's operating body OP, and can select to constitute the project that is superimposed upon the stack video data on the input picture based on the position of being identified according to present embodiment.Therefore, pass through in AR technology modification and the configuration by the HMD demonstration at photographic images, can be so that user's operation input.
<3. second embodiment 〉
In first embodiment, description also is identified as metope in the real space and ground the example of object.On the other hand, in advance not under qualification and metope or the ground characteristic of correspondence data conditions, metope or ground are not included in the environmental map.In this case, preferred identification metope or ground in addition are to generate output image according to recognition result.Therefore, in this part, will work as the ios dhcp sample configuration IOS DHCP that to discern the image processing apparatus on metope or ground when metope or ground are not included in the environmental map in addition and be described as the second embodiment of the present invention.
Figure 14 is the block diagram that illustrates according to the ios dhcp sample configuration IOS DHCP of the image processing apparatus 200 of second embodiment.With reference to Figure 14, image processing apparatus 200 comprises image-generating unit 102, environmental map generation unit 210 and output image generation unit 280.
[3-1. environmental map generation unit]
In the present embodiment, environmental map generation unit 210 comprises self-position detecting unit 120, characteristic storage unit 230, image identification unit 140, environmental map construction unit 150 and environmental map storage unit 152.
(1) characteristic storage unit
Characteristic storage unit 230 is used and is stored characteristic in advance such as the storage medium of hard disk or semiconductor memory, and this characteristic is represented the feature of the object corresponding with being present in physical objects in the real space.In the present embodiment, except that the data shown in Fig. 8, characteristic also comprises the additional data whether polygonal summit of each object of expression formation contacts with ground or metope probably.Figure 15 is the key diagram of the ios dhcp sample configuration IOS DHCP of this characteristic of explanation.
With reference to Figure 15, as the characteristic FD2 of example comprise object names FD21, from six direction shot image data FD22, sheet data FD23, three-dimensional shape data FD24, ontology data FD25 and additional data FD26.
Additional data FD26 has two signs, these two mark needles represent to polygonal each summit of each object of being limited by the polygon information that is included among the three-dimensional shape data FD24 whether the summit may contact with ground and whether the summit contacts with metope probably.For example, in the example depicted in fig. 15, additional data FD26 represents that but the polygonal summit A of the object corresponding with characteristic FD2 contacts with ground probably can not contact with metope.In addition, additional data FD26 represents that but the polygonal summit B of the object corresponding with characteristic FD2 can not contact with ground contacts with metope probably.Should be noted that polygonal summit can be an employed unique point in the above-mentioned processing of being undertaken by self-position detecting unit 120 or image identification unit 140, perhaps can be some points except that unique point.
Figure 16 is the key diagram of the explanation polygonal example relevant with the characteristic shown in Figure 15.With reference to Figure 16, three object Obj21, Obj22 and Obj23 are shown.Therein, object Obj21 represents chair.In the polygonal summit corresponding with object Obj21, six summits of the part in the chair legs contact with ground probably.In addition, object Obj22 represents calendar.Eight summits in the polygonal summit corresponding with object Obj22 contact with metope probably.In addition, object Obj23 represents drawer.Four summits that are positioned in the polygonal summit corresponding with object Obj23 on the bottom surface of drawer contact with ground probably.Four summits that are positioned in the polygonal summit corresponding with object Obj23 on the rear surface of drawer contact with metope probably.Additional data FD26 shown in Figure 15 limits this attribute on each summit.
In the present embodiment, characteristic storage unit 230 storages of environmental map generation unit 210 comprise the characteristic of above-mentioned additional data, and according to the request output additional data from location estimation unit 281.
[3-2. output image generation unit]
As shown in figure 14, in the present embodiment, output image generation unit 280 comprises location estimation unit 281, stack video data storage unit 181, processing execution unit 182, operating body recognition unit 183, stack display image generation unit 184, stack display position determining unit 186 and image overlay unit 188.
(1) location estimation unit
Location estimation unit 281 is based on the position and the above-mentioned characteristic of the point on the object surfaces of being represented by environmental map, estimates the ground in the real space or the position of metope.In this embodiment, the point on object surfaces can be and each corresponding polygonal summit in the above-mentioned object.
More specifically, for example, location estimation unit 281 extracts the set of vertices that contacts with ground probably, represented by above-mentioned characteristic from the polygonal set of vertices that is included in the object from the environmental map of environmental map construction unit 150 input.Then, the plane corresponding with ground estimated based on the three-dimensional position of the set of vertices of being extracted in global coordinate system in location estimation unit 281.For example, but location estimation unit 281 can use well-known Hough transformation method to estimate to comprise the parallel planes of set of vertices from the three-dimensional position of set of vertices.
Similarly, for example, location estimation unit 281 extracts the set of vertices that contacts with metope probably, represented by above-mentioned characteristic from the polygonal set of vertices that is included in the object from the environmental map of environmental map construction unit 150 input.Then, the plane corresponding with metope estimated based on the three-dimensional position of the set of vertices of being extracted in global coordinate system in location estimation unit 281.Should be noted that location estimation unit 281 can be divided into plural set with set of vertices according to the three-dimensional position of set of vertices, estimates the plane corresponding with the metope that is used for each set thus under two above metopes may be present in situation in the real space.
The ground that location estimation unit 281 will be estimated in this way and/or the position of metope export stack display position determining unit 186 to.
[summary of 3.3. second embodiment]
According to the corresponding image processing apparatus 200 of present embodiment, be present in the environmental map of the three-dimensional position of the object in the real space based on expression, generate the output image that obtains by stack stack video data.Here, based on the position of the point that contacts with ground or metope probably in the point on the object surfaces in being included in environmental map, estimate the position of ground or metope.
<4. hardware configuration 〉
Noting, is to realize it being inessential by hardware or software according to a series of processing of above-mentioned first embodiment and second embodiment.For example, when a series of processing or a part are wherein carried out by software, use computing machine or the multi-purpose computer shown in Figure 17 incorporated in the specialized hardware to carry out the program that constitutes software.
In Figure 17, the integrated operation of CPU (central processing unit) (CPU) 902 control multi-purpose computers.A part or whole program or the data of describing a series of processing are stored in the ROM (read-only memory) (ROM) 904.Be temporarily stored in random-access memory (ram) 906 when carrying out program and the data used by CPU 902 when handling.
CPU 902, ROM 904 and RAM 906 interconnect by bus 910.Input/output interface 912 further is connected to bus 910.
Input/output interface 912 is interfaces that CPU 902, ROM 904 and RAM 906 are connected with input media 920, display device 922, memory storage 924, imaging device 926 and driver 930.
For example, input media 920 accept from user's instruction and by input interface (such as, button, switch, control lever (lever), mouse and keyboard) information of input.Yet when imaging device 926 existed, input media 920 can not exist.For example, display device 922 is by cathode ray tube (CRT), LCD, Organic Light Emitting Diode formations such as (OLED), and on its screen display image.In the above-described embodiments, the display unit of assembling HMD is corresponding to display device 922.
For example, memory storage 924 is made of hard disk drive or semiconductor memory, and stored programme and data.Imaging device 926 is corresponding to the hardware of above-mentioned image-generating unit 102, and the image-forming component of use such as CCD or CMOS carries out imaging to real space.If desired driver 930 is arranged on the multi-purpose computer, and for example removable medium 932 is installed on the driver 930.
When a series of processings carried out by software according to first embodiment and second embodiment, the program in the ROM 904 shown in Figure 17, memory storage 924 or the removable medium 932 of for example being stored in is read among the RAM 906 and by CPU 902 execution.
It will be understood by those skilled in the art that and in the scope of claims or its equivalent, to carry out various modifications, combination, sub-portfolio and change according to designing requirement and other factors.
The application comprise with on March 24th, 2010 to Japanese priority patent application JP 2010-068269 that Jap.P. office submits to and on March 24th, 2010 the relevant subject content of disclosed subject content in the Japanese priority patent application JP 2010-068270 that Jap.P. office submits to, by reference it is herein incorporated in full at this.

Claims (26)

1. image processing apparatus comprises:
The characteristic storage unit, the characteristic of the feature of the outward appearance of its storage representation object;
Stack video data storage unit, its storage stack video data and item location, wherein, described stack video data is the source that is superimposed upon the image on user's the visual field, and described item location is the position that constitutes the project of described stack video data;
The environmental map generation unit, it generates the environmental map that expression is present in the position of the one or more objects in the described real space based on by using imaging device that real space is carried out imaging input picture that obtains and the described characteristic that is stored in the described characteristic storage unit;
Stack display position determining unit, it is based on described environmental map, determines to have the position of the object of predetermined plane or curved surface from the object of imaging described input picture;
Stack display image generation unit, it is provided with described stack video data by the position at the object of being determined by described stack display position determining unit, generates the stack display image;
The image overlay unit, it is superimposed upon described stack display image on described user's the visual field;
The operating body recognition unit, it is identified in the operating body of imaging in the described input picture; And
The processing execution unit, it is carried out and the corresponding processing of selecting based on the position of the described operating body that is identified by described operating body recognition unit of described project.
2. image processing apparatus according to claim 1,
Wherein, the foot that described operating body recognition unit will imaging in described input picture is identified as described operating body.
3. image processing apparatus according to claim 2,
Wherein, described operating body recognition unit is carried out as the footwear registration image of the image of the footwear of registering in advance and the coupling between the described input picture, and determine footwear with described footwear registration images match in described input picture during imaging when described operating body recognition unit, described operating body recognition unit is identified as described operating body with these footwear.
4. image processing apparatus according to claim 2,
Wherein, when the user is worn over described imaging device on his/her head, described operating body recognition unit determines whether the foot of imaging in described input picture enters with the immediate limit of described user among each limit that constitutes described input picture, and determine described foot when entering with the immediate limit of described user when described operating body recognition unit, described operating body recognition unit is identified as described operating body with described foot.
5. image processing apparatus according to claim 2,
Wherein, described operating body recognition unit determines to have passed through in advance whether imaging in described input picture of footwear predetermined labels, that have mark, and determine described footwear with mark in described input picture during imaging when described operating body recognition unit, described operating body recognition unit is identified as described operating body with these footwear.
6. image processing apparatus according to claim 2,
Wherein, described processing execution unit determines whether the touch sensor that is attached to described foot detects contact, and when described touch sensor detected described contact, described processing execution unit was carried out and the corresponding processing of selecting based on the position of described foot of described project.
7. image processing apparatus according to claim 2,
Wherein, described processing execution unit determines whether the described operating body that is identified by described operating body recognition unit has stopped scheduled time slot in essentially identical position, and when described processing execution unit was determined described operating body and stopped scheduled time slot in essentially identical position, described processing execution unit was carried out and the corresponding processing of selecting based on the position of described operating body of described project.
8. image processing apparatus according to claim 1,
Wherein, other stack video datas that described stack video data cell stores shows after described stack video data,
Wherein, when having selected described project by described processing execution unit, described stack display image generation unit generates new stack display image by described other stack video datas further are set, and
Wherein, when having selected described project by described processing execution unit, described image overlay unit further is superimposed upon described other stack video datas on the described new stack display image.
9. image processing apparatus according to claim 1,
Wherein, the hard recognition that described operating body recognition unit will imaging in described input picture is described operating body.
10. image processing apparatus according to claim 1,
Wherein, described image overlay unit is superimposed upon described stack display image on described user's the visual field, and makes display unit show described stack display image.
11. image processing apparatus according to claim 10,
Wherein, detect the sensor of the degree of tilt of described imaging device by use when described image overlay unit, detect with described imaging device be worn over user on his/her head make described head with the degree of tilt that surpasses predetermined value when downward direction tilts, described image overlay unit makes described display unit show described stack display image, and detect the described sensor of the degree of tilt of described imaging device by use when described image overlay unit, do not detect with described imaging device be worn over user on his/her head make described head with the degree of tilt that surpasses described predetermined value when downward direction tilts, the restriction of described image overlay unit shows described stack display image, wherein, described demonstration is carried out by described display unit.
12. image processing apparatus according to claim 1,
Wherein, described stack display position determining unit will have the position of the object that is defined as having described predetermined plane or curved surface in the position of the object on the upwardly extending plane, side of basic horizontal.
13. image processing apparatus according to claim 1 also comprises:
The location estimation unit, it is based on the position of the point on the object surfaces of being represented by described environmental map, estimates the ground in the described real space or the position of metope,
Wherein, described stack display position determining unit further based on by the ground in the described real space of described location estimation unit estimation or the position of metope, is defined as the position on ground the position of described object.
14. image processing apparatus according to claim 13,
Wherein, described characteristic comprise at the one or more points on each object surfaces represent each point whether probably with described real space in ground or the data that contact of metope, and
Wherein, the ground in the described real space or the position of metope are further estimated based on described characteristic in described location estimation unit.
15. image processing method, it is carried out by image processing apparatus, described image processing apparatus comprises the characteristic storage unit of characteristic of feature of the outward appearance of storage representation object, the stack video data storage unit of storage stack video data and item location, the environmental map generation unit, stack display position determining unit, the image overlay unit, the operating body recognition unit, and processing execution unit, wherein, described stack video data is the source that is superimposed upon the image on user's the visual field, and described item location is the position that constitutes the project of described stack video data, and described image processing method may further comprise the steps:
Based on by using imaging device that real space is carried out imaging input picture that obtains and the described characteristic that is stored in the described characteristic storage unit, generate the environmental map that expression is present in the position of the one or more objects in the described real space by described environmental map generation unit;
Based on described environmental map, from the object of imaging described input picture, determine to have the position of the object of predetermined plane or curved surface by described stack display position determining unit;
By position described stack video data is set by described stack display image generation unit, generates the stack display image at the object of determining by described stack display position determining unit;
By described image overlay unit described stack display image is superimposed upon on described user's the visual field;
Be identified in the operating body of imaging in the described input picture by described operating body recognition unit; And
Carry out and the corresponding processing of selecting based on the position of the described operating body that identifies by described operating body recognition unit of described project by described processing execution unit.
16. one kind is used to make computing machine to play the program of the effect of image processing apparatus, described image processing apparatus comprises:
The characteristic storage unit, the characteristic of the feature of the outward appearance of its storage representation object,
Stack video data storage unit, its storage stack video data and item location, wherein, described stack video data is the source that is superimposed upon the image on user's the visual field, and described item location is the position that constitutes the project of described stack video data,
The environmental map generation unit, it is based on by using imaging to adorn real space being carried out imaging input picture that obtains and the described characteristic that is stored in the described characteristic storage unit, generate the environmental map that expression is present in the position of the one or more objects in the described real space
Stack display position determining unit, it is based on described environmental map, determines to have the position of the object of predetermined plane or curved surface from the object of imaging described input picture,
Stack display image generation unit, it is provided with described stack video data by the position at the object of being determined by described stack display position determining unit, generates the stack display image,
The image overlay unit, it is superimposed upon described stack display image on described user's the visual field,
The operating body recognition unit, it is identified in the operating body of imaging in the described input picture, and
The processing execution unit, it is carried out and the corresponding processing of selecting based on the position of the described operating body that is identified by described operating body recognition unit of described project.
17. an image processing apparatus comprises:
The characteristic storage unit, the characteristic of the feature of the outward appearance of its storage representation object;
Stack video data storage unit, its storage is as the stack video data in the source of the image on the visual field that is superimposed upon the user;
The environmental map generation unit, it generates the environmental map that expression is present in the position of the one or more objects in the described real space based on by using imaging device that real space is carried out imaging input picture that obtains and the described characteristic that is stored in the described characteristic storage unit;
Stack display position determining unit, it is based on described environmental map, determines to have the position of the object of predetermined plane or curved surface from the object of imaging described input picture;
Stack display image generation unit, it is provided with described stack video data by the position at the object of being determined by described stack display position determining unit, generates the stack display image; And
The image overlay unit, it is superimposed upon described stack display image on described user's the visual field, and makes display unit show described stack display image.
18. image processing apparatus according to claim 17,
Wherein, detect the sensor of the degree of tilt of described image device by use when described image overlay unit, detect with described imaging device be worn over user on his/her head make described head with the degree of tilt that surpasses predetermined value when downward direction tilts, described image overlay unit makes described display unit show described stack display image, and detect the described sensor of the degree of tilt of described image device by use when described image overlay unit, do not detect with described imaging device be worn over user on his/her head make described head with the degree of tilt that surpasses described predetermined value when downward direction tilts, the restriction of described image overlay unit shows described stack display image, wherein, described demonstration is carried out by described display unit.
19. image processing apparatus according to claim 18 also comprises
The self-position detecting unit, it dynamically detects the position of described image processing apparatus based on described input picture and described characteristic,
Wherein, when surpassing predetermined value from the distance of position to the position of the object of determining by described stack display position determining unit by the detected described image processing apparatus of described self-position detecting unit, the restriction of described image overlay unit shows described stack display image, wherein, described demonstration is carried out by described display unit.
20. image processing apparatus according to claim 17,
Wherein, described stack display position determining unit will have the position of the object that is defined as having described predetermined plane or curved surface in the position of the object on the upwardly extending plane, side of basic horizontal.
21. image processing apparatus according to claim 20,
Wherein, described stack display position determining unit is defined as having position at the object on the upwardly extending plane, side of basic horizontal with at least one the position in ground, desktop and the stair.
22. image processing apparatus according to claim 17,
Wherein, detect the sensor of the rotation of described imaging device by use when described stack display image generation unit, detect when described imaging device is worn over user on his/her head described head is rotated on the direction of basic horizontal, described stack display image generation unit changes the position that is provided with of described stack video data by the degree according to described rotation, moves the described stack video data that is provided with in described stack display image.
23. image processing apparatus according to claim 17 also comprises
The location estimation unit, it is based on the position of the point on the object surfaces of being represented by described environmental map, estimates the ground in the described real space or the position of metope,
Wherein, described stack display position determining unit further based on by the ground in the described real space of described location estimation unit estimation or the position of metope, is defined as the position on ground the position of described object.
24. image processing apparatus according to claim 23,
Wherein, described characteristic comprise at the one or more points on each object surfaces represent each point whether probably with described real space in ground or the data that contact of metope, and
Wherein, the ground in the described real space or the position of metope are further estimated based on described characteristic in described location estimation unit.
25. image processing method, it is carried out by image processing apparatus, described image processing apparatus comprises the characteristic storage unit, storage of characteristic of feature of outward appearance of storage representation object as stack video data storage unit, environmental map generation unit, stack display position determining unit, stack display image generation unit and the image overlay unit of the stack video data in the source that is superimposed upon the image on user's the visual field, and described image processing method may further comprise the steps:
Based on by using imaging device that real space is carried out imaging input picture that obtains and the described characteristic that is stored in the described characteristic storage unit, generate the environmental map that expression is present in the position of the one or more objects in the described real space by described environmental map generation unit;
Based on described environmental map, from the object of imaging described input picture, determine to have the position of the object of predetermined plane or curved surface by described stack display position determining unit;
By position described stack video data is set by described stack display image generation unit, generates the stack display image at the object of determining by described stack display position determining unit; And
By described image overlay unit described stack display image is superimposed upon on described user's the visual field, and makes display unit show described stack display image by described image overlay unit.
26. one kind is used to make computing machine to play the program of the effect of image processing apparatus, described image processing apparatus comprises:
The characteristic storage unit, the characteristic of the feature of the outward appearance of its storage representation object;
Stack video data storage unit, it stores the stack video data as the source of the image on the visual field that is superimposed upon the user,
The environmental map generation unit, it is based on by the use imaging device real space being carried out imaging input picture that obtains and the described characteristic that is stored in the described characteristic storage unit, generate the environmental map that expression is present in the position of the one or more objects in the described real space
Stack display position determining unit, it is based on described environmental map, determines to have the position of the object of predetermined plane or curved surface from the object of imaging described input picture,
Stack display image generation unit, it is provided with described stack video data by the position at the object of being determined by described stack display position determining unit, generates the stack display image, and
The image overlay unit, it is superimposed upon described stack display image on described user's the visual field, and makes display unit show described stack display image.
CN201110069689.XA 2010-03-24 2011-03-17 Image processing apparatus and image processing method Active CN102200881B (en)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
JP2010068269A JP2011203823A (en) 2010-03-24 2010-03-24 Image processing device, image processing method and program
JP2010068270A JP2011203824A (en) 2010-03-24 2010-03-24 Image processing device, image processing method and program
JP2010-068270 2010-03-24
JP2010-068269 2010-03-24

Publications (2)

Publication Number Publication Date
CN102200881A true CN102200881A (en) 2011-09-28
CN102200881B CN102200881B (en) 2016-01-13

Family

ID=44661594

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201110069689.XA Active CN102200881B (en) 2010-03-24 2011-03-17 Image processing apparatus and image processing method

Country Status (1)

Country Link
CN (1) CN102200881B (en)

Cited By (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014101606A1 (en) * 2012-12-28 2014-07-03 腾讯科技(深圳)有限公司 Trigger and control method and system for human-computer interaction operation
CN104102344A (en) * 2013-04-15 2014-10-15 奥林巴斯株式会社 Wearable device, and display controlling method of wearable device
CN104103032A (en) * 2013-04-05 2014-10-15 欧姆龙株式会社 Image processing device, control method, and program
CN104620212A (en) * 2012-09-21 2015-05-13 索尼公司 Control device and recording medium
CN104683104A (en) * 2013-12-03 2015-06-03 腾讯科技(深圳)有限公司 Identity identification method, identity identification device and identity identification system
CN104798109A (en) * 2012-11-13 2015-07-22 高通股份有限公司 Modifying virtual object display properties
CN105278670A (en) * 2014-06-26 2016-01-27 Lg电子株式会社 Eyewear-type terminal and method for controlling the same
CN105683900A (en) * 2013-12-04 2016-06-15 英特尔公司 Wearable map and image display
CN105825468A (en) * 2015-01-26 2016-08-03 联发科技(新加坡)私人有限公司 Graphics processing unit and graphics processing method thereof
CN105872526A (en) * 2015-01-21 2016-08-17 成都理想境界科技有限公司 Binocular AR (Augmented Reality) head-mounted display device and information display method thereof
WO2017054420A1 (en) * 2015-09-30 2017-04-06 深圳多新哆技术有限责任公司 Method and device for determining position of virtual object in virtual space
CN106575184A (en) * 2014-07-30 2017-04-19 索尼公司 Information processing device, information processing method and program
CN106874594A (en) * 2017-02-13 2017-06-20 云南电网有限责任公司电力科学研究院 One introduces a collection lotus control method for coordinating
CN107038746A (en) * 2017-03-27 2017-08-11 联想(北京)有限公司 A kind of information processing method and electronic equipment
CN108664124A (en) * 2018-05-08 2018-10-16 北京奇艺世纪科技有限公司 A kind of control method and device based on attitude information
CN109032330A (en) * 2017-06-12 2018-12-18 赫克斯冈技术中心 Seamless bridge joint AR device and AR system
CN109171780A (en) * 2018-10-10 2019-01-11 宝鸡市中医医院 A kind of medical visiting pen
CN109496433A (en) * 2016-05-27 2019-03-19 乐威指南公司 Method and system for selecting that the supplemental content shown near user equipment during media asset is presented on a user device
CN109756672A (en) * 2018-11-13 2019-05-14 深圳艺达文化传媒有限公司 Short-sighted frequency animal model stacking method and Related product

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO1998011528A1 (en) * 1997-05-09 1998-03-19 Remec Inc. Computer control device
US6050822A (en) * 1997-10-01 2000-04-18 The United States Of America As Represented By The Secretary Of The Army Electromagnetic locomotion platform for translation and total immersion of humans into virtual environments
WO2002073287A2 (en) * 2001-03-13 2002-09-19 Canon Kabushiki Kaisha Mixed reality system which reduces measurement errors of viewpoint position and direction of an observer
EP1244003A2 (en) * 2001-03-09 2002-09-25 Samsung Electronics Co., Ltd. Information input system using bio feedback and method thereof
US6771294B1 (en) * 1999-12-29 2004-08-03 Petri Pulli User interface
CN1696606A (en) * 2004-05-14 2005-11-16 佳能株式会社 Information processing method and apparatus for finding position and orientation of targeted object
US20070132722A1 (en) * 2005-12-08 2007-06-14 Electronics And Telecommunications Research Institute Hand interface glove using miniaturized absolute position sensors and hand interface system using the same

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO1998011528A1 (en) * 1997-05-09 1998-03-19 Remec Inc. Computer control device
US6050822A (en) * 1997-10-01 2000-04-18 The United States Of America As Represented By The Secretary Of The Army Electromagnetic locomotion platform for translation and total immersion of humans into virtual environments
US6771294B1 (en) * 1999-12-29 2004-08-03 Petri Pulli User interface
EP1244003A2 (en) * 2001-03-09 2002-09-25 Samsung Electronics Co., Ltd. Information input system using bio feedback and method thereof
WO2002073287A2 (en) * 2001-03-13 2002-09-19 Canon Kabushiki Kaisha Mixed reality system which reduces measurement errors of viewpoint position and direction of an observer
CN1696606A (en) * 2004-05-14 2005-11-16 佳能株式会社 Information processing method and apparatus for finding position and orientation of targeted object
US20070132722A1 (en) * 2005-12-08 2007-06-14 Electronics And Telecommunications Research Institute Hand interface glove using miniaturized absolute position sensors and hand interface system using the same

Cited By (33)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104620212A (en) * 2012-09-21 2015-05-13 索尼公司 Control device and recording medium
CN104620212B (en) * 2012-09-21 2018-09-18 索尼公司 Control device and recording medium
US9727996B2 (en) 2012-11-13 2017-08-08 Qualcomm Incorporated Modifying virtual object display properties to increase power performance of augmented reality devices
CN104798109A (en) * 2012-11-13 2015-07-22 高通股份有限公司 Modifying virtual object display properties
CN104798109B (en) * 2012-11-13 2018-07-03 高通股份有限公司 For in the method and apparatus of augmented reality display inner tissue virtual objects
US9619911B2 (en) 2012-11-13 2017-04-11 Qualcomm Incorporated Modifying virtual object display properties
WO2014101606A1 (en) * 2012-12-28 2014-07-03 腾讯科技(深圳)有限公司 Trigger and control method and system for human-computer interaction operation
US9829974B2 (en) 2012-12-28 2017-11-28 Tencent Technology (Shenzhen) Company Limited Method for controlling triggering of human-computer interaction operation and apparatus thereof
CN104103032A (en) * 2013-04-05 2014-10-15 欧姆龙株式会社 Image processing device, control method, and program
CN104102344A (en) * 2013-04-15 2014-10-15 奥林巴斯株式会社 Wearable device, and display controlling method of wearable device
CN104683104A (en) * 2013-12-03 2015-06-03 腾讯科技(深圳)有限公司 Identity identification method, identity identification device and identity identification system
CN104683104B (en) * 2013-12-03 2018-11-23 腾讯科技(深圳)有限公司 The method, apparatus and system of authentication
CN105683900B (en) * 2013-12-04 2019-09-17 英特尔公司 Wearable map and image display
CN105683900A (en) * 2013-12-04 2016-06-15 英特尔公司 Wearable map and image display
CN105278670B (en) * 2014-06-26 2020-06-05 Lg电子株式会社 Eyeglasses-type terminal and method of controlling the same
CN105278670A (en) * 2014-06-26 2016-01-27 Lg电子株式会社 Eyewear-type terminal and method for controlling the same
CN106575184B (en) * 2014-07-30 2020-05-19 索尼公司 Information processing apparatus, information processing method, and computer readable medium
CN106575184A (en) * 2014-07-30 2017-04-19 索尼公司 Information processing device, information processing method and program
CN105872526B (en) * 2015-01-21 2017-10-31 成都理想境界科技有限公司 Binocular AR wears display device and its method for information display
CN105872526A (en) * 2015-01-21 2016-08-17 成都理想境界科技有限公司 Binocular AR (Augmented Reality) head-mounted display device and information display method thereof
CN105825468A (en) * 2015-01-26 2016-08-03 联发科技(新加坡)私人有限公司 Graphics processing unit and graphics processing method thereof
WO2017054420A1 (en) * 2015-09-30 2017-04-06 深圳多新哆技术有限责任公司 Method and device for determining position of virtual object in virtual space
US10957065B2 (en) 2015-09-30 2021-03-23 Shenzhen Dlodlo Technologies Co., Ltd. Method and device for determining position of virtual object in virtual space
CN109496433A (en) * 2016-05-27 2019-03-19 乐威指南公司 Method and system for selecting that the supplemental content shown near user equipment during media asset is presented on a user device
CN106874594A (en) * 2017-02-13 2017-06-20 云南电网有限责任公司电力科学研究院 One introduces a collection lotus control method for coordinating
CN107038746A (en) * 2017-03-27 2017-08-11 联想(北京)有限公司 A kind of information processing method and electronic equipment
CN107038746B (en) * 2017-03-27 2019-12-24 联想(北京)有限公司 Information processing method and electronic equipment
CN109032330A (en) * 2017-06-12 2018-12-18 赫克斯冈技术中心 Seamless bridge joint AR device and AR system
US10950054B2 (en) 2017-06-12 2021-03-16 Hexagon Technology Center Gmbh Seamless bridging AR-device and AR-system
CN109032330B (en) * 2017-06-12 2022-02-11 赫克斯冈技术中心 AR system, AR device and method for establishing and seamlessly bridging reference states thereof
CN108664124A (en) * 2018-05-08 2018-10-16 北京奇艺世纪科技有限公司 A kind of control method and device based on attitude information
CN109171780A (en) * 2018-10-10 2019-01-11 宝鸡市中医医院 A kind of medical visiting pen
CN109756672A (en) * 2018-11-13 2019-05-14 深圳艺达文化传媒有限公司 Short-sighted frequency animal model stacking method and Related product

Also Published As

Publication number Publication date
CN102200881B (en) 2016-01-13

Similar Documents

Publication Publication Date Title
CN102200881A (en) Image processing apparatus, image processing method and program
US10521085B2 (en) Image processing device, image processing method, and program for displaying an image in accordance with a selection from a displayed menu and based on a detection by a sensor
CN102129290B (en) Image processing device, object selection method and program
CN102054164B (en) Image processing device, image processing method and program
JP5799521B2 (en) Information processing apparatus, authoring method, and program
Hile et al. Landmark-based pedestrian navigation from collections of geotagged photos
CN102141885B (en) Image processing device and image processing method
JP5471626B2 (en) Information processing apparatus, map update method, program, and information processing system
US8751969B2 (en) Information processor, processing method and program for displaying a virtual image
JP5920352B2 (en) Information processing apparatus, information processing method, and program
US9965682B1 (en) System and method for determining position of a device
Schmalstieg et al. The world as a user interface: Augmented reality for ubiquitous computing
Hertzberg et al. Experiences in building a visual SLAM system from open source components
Hile et al. Information overlay for camera phones in indoor environments
Wientapper et al. Composing the feature map retrieval process for robust and ready-to-use monocular tracking
López-Nicolás et al. Spatial layout recovery from a single omnidirectional image and its matching-free sequential propagation
Hu et al. Computer vision for sight: Computer vision techniques to assist visually impaired people to navigate in an indoor environment
Pintore et al. Interactive mapping of indoor building structures through mobile devices
KR20140001152A (en) Areal-time coi registration system and the method based on augmented reality
Beleznai et al. Pose-aware object recognition on a mobile platform via learned geometric representations
Zhang et al. ARCargo: Multi-Device Integrated Cargo Loading Management System with Augmented Reality
Chen et al. Research on smart navigation system based on AR technology
Yun et al. Multi-hypothesis localization with a rough map using multiple visual features for outdoor navigation
Hu et al. Computer Vision for Sight
Peng Design and Application of Cultural Digital Display System based on Interactive Technology

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
C14 Grant of patent or utility model
GR01 Patent grant