US8195386B2 - Movable-body navigation information display method and movable-body navigation information display unit - Google Patents
Movable-body navigation information display method and movable-body navigation information display unit Download PDFInfo
- Publication number
- US8195386B2 US8195386B2 US11/663,909 US66390905A US8195386B2 US 8195386 B2 US8195386 B2 US 8195386B2 US 66390905 A US66390905 A US 66390905A US 8195386 B2 US8195386 B2 US 8195386B2
- Authority
- US
- United States
- Prior art keywords
- road
- data
- navigation information
- movable body
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related, expires
Links
Images
Classifications
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/09—Arrangements for giving variable traffic instructions
- G08G1/0962—Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages
- G08G1/0968—Systems involving transmission of navigation instructions to the vehicle
- G08G1/0969—Systems involving transmission of navigation instructions to the vehicle having a display in the form of a map
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/26—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
- G01C21/34—Route searching; Route guidance
- G01C21/36—Input/output arrangements for on-board computers
- G01C21/3626—Details of the output of route guidance instructions
- G01C21/3647—Guidance involving output of stored or live camera images or video streams
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
- G01C21/26—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
- G01C21/34—Route searching; Route guidance
- G01C21/36—Input/output arrangements for on-board computers
- G01C21/3626—Details of the output of route guidance instructions
- G01C21/365—Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/09—Arrangements for giving variable traffic instructions
- G08G1/0962—Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages
- G08G1/0968—Systems involving transmission of navigation instructions to the vehicle
- G08G1/096855—Systems involving transmission of navigation instructions to the vehicle where the output is provided in a suitable form to the driver
- G08G1/096861—Systems involving transmission of navigation instructions to the vehicle where the output is provided in a suitable form to the driver where the immediate route instructions are output to the driver, e.g. arrow signs for next turn
Definitions
- the present invention relates to a movable-body navigation information display method and movable-body navigation information display unit for, for example, collecting, processing, and displaying information on a current position of a movable body and information for navigation related thereto, such as a so-called car navigation system and a display method of car navigation information thereby.
- movable-body navigation system such as a so-called car navigation system has been developed.
- a current position of a movable body such as a motor vehicle is detected, the detected current position is displayed on a display screen together with a road map of surroundings, various guide information such as a destination and a route is displayed, and voice output is made.
- an autonomous-navigation controlling section performs tracing on a map previously stored as data in a storing means such as a CD-ROM (Compact Disc Read Only Memory) based on the speed pulses from a speed sensor and a direction obtained by earth magnetism from a geomagnetic sensor.
- a GPS signal transmitted from a GPS satellite is used.
- an image showing the current position and the road map in a given range in the vicinity thereof can be displayed as a plane map-like two-dimensional image on a screen of an image display unit such as a liquid crystal display unit.
- navigation information such as a vehicle position, a traveling direction, speed, map information, a course guide, the shortest route, and road information by being superimposed on the two-dimensional image map, or by being inserted in an appropriate position in the two-dimensional image map.
- an image of information for example, a traveling position and a traveling direction indicated by an icon of an arrow or the like
- a recommended route (optimal route) from the current position to the destination is displayed by being superimposed on a two-dimensional image road planar map or on a road map that shows the surrounding landscape in a state of a three-dimensional CG.
- traffic jam information or the like obtained by a VICS sensor or the like is displayed with a text label on the screen.
- Patent document 1 Patent document 2 and the like, a navigation system using a real picture in addition to the map information is proposed.
- Patent document 1 a current position and a traveling direction of a movable body are detected, and navigation information to be displayed according to the detected current position and the detected traveling direction is selected.
- An image of the navigation information read according to the current position of the vehicle is superimposed on a real picture captured by an image pickup device from a camera angle approximately similar to a landscape ahead of the vehicle in the traveling direction seen through the windshield of the vehicle, and the superimposed result is displayed on a screen or the like of a liquid crystal display panel.
- a relation between the picture of the real road, the surrounding landscape thereof and the navigation information can be displayed easily recognizable visually.
- the navigation information is mainly picture information displayed on the picture display section. Therefore, there is a possibility that a driver gets distracted easily while driving.
- Patent document 2 the following technique is proposed. That is, an image pickup device such as a CCD camera that captures a landscape ahead of a body for motor vehicle is provided, for example, in a position near the ceiling of the windshield of the vehicle or in the vicinity of the dashboard. A picture (image) of the landscape including a road ahead of the body for motor vehicle captured by the image pickup device is displayed by being inserted in a given position of the screen displaying map information as a sub-screen. According to such a technique, even when the driver of the vehicle views navigation information such as a map displayed on the screen of the image display unit while driving, the driver views the real picture of the landscape ahead of the vehicle displayed as the sub-image in the given position on the screen. It is stated in Patent document 2 that the driver can thereby comprehend a state in front of the vehicle without returning her/his attention to the landscape ahead of the vehicle.
- an image pickup device such as a CCD camera that captures a landscape ahead of a body for motor vehicle is provided, for example, in a position near the ceiling of the windshield
- a three-dimensional road shape is estimated by using image road information and map road information.
- the image road shape extracted from image data of the landscape ahead of the vehicle and map data around the vehicle are projected in one two-dimensional or three-dimensional logical space. Based on an overlapping state in the logical space of both of the projected road shapes, estimation is made for the road shape, a posture of the vehicle to the road surface, an absolute position of the vehicle and the like.
- a sufficiently accurate road shape is estimated, and accurate determination of the leading vehicle and automatic steering are realized.
- the image information is only used as a background picture.
- the image is firstly displayed in the central portion of the screen, and then is displayed to move linearly from the original position to the backward landscape (in general, from the upper side to the lower side of the display screen) in the central section of the screen as the vehicle travels.
- the posture parameter of the camera is previously fixed as the angle of the camera optical axis to the ground surface (or horizontal direction). Therefore, for example, when the posture of the camera is changed due to vibration of the running vehicle, rolling and pitching of motor vehicle by steering, or tilt of the vehicle on the uphill slope and the downhill slope, a navigation information image of route such as right turning position and indicated by an arrow or the like of a traveling direction is largely deviated to the picture obtained by capturing the real landscape. Consequently, the arrow indicates a wrong direction substantially, and the right turning position is displayed unclearly.
- an arrow 901 as a navigation information image appears to indicate the left turning position on the road in the landscape ahead of the vehicle clearly.
- a general passenger car or even in the case of a vehicle such as a bus and a track with the driver seat located in a high position, there is no possibility that the road landscape actually seen through the windshield from the driver seat of the vehicle becomes a bird's-eye view down from a high position as shown in FIG. 13 .
- the road picture often results in a landscape (and real picture) in which anteroposterior density is particularly high and a road that crosses the road in the traveling direction is not easily seen since such a road is hidden by the buildings and the like along the road in the traveling direction.
- FIGS. 15A and 15B show comparative schematic views.
- FIG. 15A shows the degree of change ( ⁇ LA) of a projection image onto a road surface with respect to the unit angle change (posture to the road surface) ⁇ e in the visual line at about 1 m high from a driver seat of a general vehicle.
- FIG. 15B shows the degree of change ( ⁇ LB) of a projection image onto a road surface with respect to the unit angle change (posture to the road surface) ⁇ e in the visual line at about 10 m high, which is higher than the height in FIG. 15A .
- ⁇ LA degree of change
- ⁇ LB degree of change
- Patent document 1 in which the navigation information such as the arrow is located in a fixed position such as on the central line on the screen of the real picture of the landscape, even if the posture of the vehicle is changed slightly, the deviation between the real picture of the landscape and the image of the navigation information becomes large.
- the landscape located near the vehicle along the road such as buildings and trees blocks the landscape located farther than the foregoing landscape. Therefore, to address such a problem, it is necessary to display the image of the navigation information on a more accurate position.
- Patent document 2 there is the following problem in addition to the problems similar to those of the Patent document 1 described above. That is, in Patent document 2, the real picture of the road landscape ahead of the vehicle is only displayed in the sub-screen. Therefore, matching of the road map information and the real picture should be made by the driver in his/her mind. Therefore, for example, when the driver drives a road with many crossings and forks in an unfamiliar (or firstly driving) place, it is difficult to intuitively comprehend the navigation information. In the result, there is a possibility that the user takes a wrong route, falsely recognizes the content indicated by the navigation information, and is not able to understand the content.
- Patent document 3 has the following problems. That is, in such a technique, there is no specific road shape model. Therefore, in the case of a multilane road, there is a possibility that large deviation is caused between the central line of the driving lane of the vehicle extracted from the image data and the road centerline estimated from the road map data. Further, it is not possible to estimate information of the driving lane of the vehicle. Thus, accurate navigation information is not able to be provided in changing the lane and in turning left or right.
- the estimation result of the road shape is not feed backed, and the section with large luminance change is simply extracted as a white line by a differential filter for every frame.
- the estimation result obtained by the foregoing method is largely subject to influences of various environmental factors such as weather change and shadow or dirt on the road surface. Therefore, the road shape model expressed by the data obtained by extracting the road characteristics may become an incorrect model that is significantly deviated from the road shape of the real object.
- the image of the navigation information is composited and displayed in the real picture in front of the vehicle, there is a possibility as follows.
- the image of the navigation information is superimposed on a caution-needed picture such as a pedestrian and other obstruction in the real picture. Thereby, the visibility of the caution-needed picture may be inhibited.
- navigation information display method and a movable-body navigation information display unit In the navigation information display method and the movable-body navigation information display unit, navigation information such as a course guide, a vehicle position, and map information is accurately projected and displayed in an appropriate position in a real picture or in a real landscape of the road ahead of the movable body. Thereby, a driver can intuitively and accurately recognize a relation between the navigation information and the real picture or the real landscape. In addition, visibility of a caution-needed picture such as a pedestrian in the real picture and a real picture of a road construction site is not blocked by the image of the navigation information.
- a caution-needed picture such as a pedestrian in the real picture and a real picture of a road construction site is not blocked by the image of the navigation information.
- a first movable-body navigation information display method includes the steps of detecting a current position of a movable body, and capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data; creating a road shape model of a road assumed to be captured from the current position, based on the current position of the movable body and the road map data; extracting road shape data as image data of the road included in the landscape from the real picture; matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, and determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data; and displaying an image in which the read navigation
- a first movable-body navigation information display unit includes: a current position detecting means for detecting a current position of a movable body; a capturing means for capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; a data processing means for reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data, creating a road shape model of a road assumed to be captured from the current position based on the current position of the movable body and the road map data, extracting road shape data as image data of the road included in the landscape from the real picture, matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based
- the road shape model of the road assumed to be captured from the current position is created based on the current position of the movable body such as a motor vehicle and the road map data including the current position.
- the road shape data as the image data of the road included in the landscape ahead of the vehicle or in the traveling direction is extracted from the real picture.
- the data of the road shape model is matched with the road shape data to estimate the posture data of the in-vehicle camera or the movable body to the road in the landscape as the object.
- the appropriate display position in the captured real picture of the navigation information read according to the current position of the movable body is determined. In the result, the image in which the read navigation information is composited with the display dimensions and the tone set by the user is displayed in the determined appropriate position of the captured real picture.
- the appropriate composite position of the navigation information in the real picture is determined by matching the data of the road shape model with the road shape data.
- display is enabled so that the navigation information such as a course guide, a position of a vehicle, and map information is projected accurately in an appropriate position in a real picture of a road ahead of the movable body or a real landscape.
- the display dimensions and the tone of the image (generically referred to as image, including character information and the like) of the navigation information composited in the real picture can be set as appropriate according to the user's option. Therefore, deterioration of visibility of the entire image in which the image of the navigation information is composited in the real picture is avoided.
- the user her/himself can customize the image of the navigation information to obtain the image with higher visibility for the user.
- a second movable-body navigation information display method includes the steps of: detecting a current position of a movable body, and capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data; creating a road shape model of a road assumed to be captured from the current position, based on the current position of the movable body and the road map data; extracting road shape data as image data of the road included in the landscape from the real picture; matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, and determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data; and previously defining three classifications of warning
- a second movable-body navigation information display unit includes: a current position detecting means for detecting a current position of a movable body; a capturing means for capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; a data processing means for reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data, creating a road shape model of a road assumed to be captured from the current position based on the current position of the movable body and the road map data, extracting road shape data as image data of the road included in the landscape from the real picture, matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based
- the appropriate composite position of the navigation information in the real picture is determined by matching the data of the road shape model with the road shape data.
- display is enabled so that the navigation information such as a course guide, a position of a vehicle, and map information is projected accurately in an appropriate position in a real picture of a road ahead of the movable body or a real landscape.
- a driver can intuitively and accurately recognize a relation between the navigation information and the real picture or the real landscape.
- the image of the read navigation information is composited and displayed according to the order corresponding to the classification based on the priority described above. Therefore, many navigation information images are inhibited from being displayed in one real picture. Consequently, deterioration of visibility of the entire image in which the image of the navigation information is composited in the real picture is avoided.
- the navigation information other than the foregoing warning information may be displayed on and after when the current position of the movable body to a warning target in the real picture enters within a given distance. Thereby, the data processing amount for creating the image of the navigation information is saved. In addition, a navigation information image necessary for the user can be displayed timely when such navigation information becomes really necessary for the user.
- a third movable-body navigation information display method includes the steps of: detecting a current position of a movable body, and capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data; creating a road shape model of a road assumed to be captured from the current position, based on the current position of the movable body and the road map data; extracting road shape data as image data of the road included in the landscape from the real picture; matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, and determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data; and displaying an image in which the read
- a third movable-body navigation information display unit includes: a current position detecting means for detecting a current position of a movable body; a capturing means for capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; a data processing means for reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data, creating a road shape model of a road assumed to be captured from the current position based on the current position of the movable body and the road map data, extracting road shape data as image data of the road included in the landscape from the real picture, matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based
- the appropriate composite position of the navigation information in the real picture is determined by matching the data of the road shape model with the road shape data.
- display is enabled so that the navigation information such as a course guide, a position of a vehicle, and map information is projected accurately in an appropriate position in a real picture of a road ahead of the movable body or a real landscape.
- a driver can intuitively and accurately recognize a relation between the navigation information and the real picture or the real landscape.
- the navigation information other than the graphic display for guiding the traveling direction is not displayed in the region under the virtual vanishing point or the virtual horizon line in the real picture but in the region above the virtual vanishing point or the virtual horizon line. Consequently, visibility of the real picture of a road and a surface ground is not inhibited by display of the navigation information.
- a fourth movable-body navigation information display method includes the steps of detecting a current position of a movable body, and capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data; creating a road shape model of a road assumed to be captured from the current position, based on the current position of the movable body and the road map data; extracting road shape data as image data of the road included in the landscape from the real picture; matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, and determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data; and compositing and displaying the read
- a fourth movable-body navigation information display unit includes: a current position detecting means for detecting a current position of a movable body; a capturing means for capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; a data processing means for reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data, creating a road shape model of a road assumed to be captured from the current position based on the current position of the movable body and the road map data, extracting road shape data as image data of the road included in the landscape from the real picture, matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based
- the appropriate composite position of the navigation information in the real picture is determined by matching the data of the road shape model with the road shape data.
- display is enabled so that the navigation information such as a course guide, a position of a vehicle, and map information is projected accurately in an appropriate position in a real picture of a road ahead of the movable body or a real landscape.
- a driver can intuitively and accurately recognize a relation between the navigation information and the real picture or the real landscape.
- the image of the navigation information is composited and displayed in the real picture as the rotatable three-dimensional icon that rotates centering on the vertical axis or the horizontal axis. Consequently, visibility of the real picture of a road and a surface ground is not inhibited by display of the navigation information.
- the foregoing navigation information may be, for example, at least one of a course guide of a route to reach a destination of the movable body, a position of a vehicle, a driving lane of the vehicle, and a building as an eyemark for a driver of the movable body to check the course guide or the position of the vehicle.
- the navigation information is information of a character, a symbol, or a number
- the information may be iconified, and the icon image may be composited and displayed in the captured real picture.
- the navigation information may be expressed as a virtual object in the three-dimensional augmented reality space, and a corresponding position in the road shape data converted into the two-dimensional characteristics space may be assigned to the expressed navigation information based on already obtained posture data.
- the image of the navigation information can be composited in the real picture as a virtual object.
- the navigation information composed of a character, a symbol, or a number for a building as an eyemark of the course guide can be expressed visually and intuitively, even when the building as the eyemark is hidden by the building or the like in front of the building as the eyemark or is hidden by a curved road.
- the road shape data is converted to image data in a perspective two-dimensional characteristics space and the data of the road shape model is converted to image data in the perspective two-dimensional characteristics space, and the two-dimensional road shape data is matched with the two-dimensional data of the road shape model in the two-dimensional characteristics space to estimate the posture data of the in-vehicle camera or the movable body to the road surface as the object.
- matching the road shape data with the data of the road shape model performed for estimating the posture data can be made with the use of 2 pieces of the two-dimensional data in the pseudo-three-dimensional two-dimensional characteristics space, without use of a method assumed to lead to the case that the information amount is enormously large and high speed processing of data becomes difficult such as a method using 2 pieces of three-dimensional data in the three-dimensional logical space. Therefore, the matching process can be simplified and the speed thereof can be accelerated.
- the road shape model of a multilane road may be created by performing modeling while considering a road structure in which one-way grade of a traveling road surface in a curved road is set to change according to a horizontal curvature of the curved road.
- an existence probability of a road white line included in the landscape is obtained from the real picture and an RSL value is calculated by using a road shaped look-up table (RSL), the posture data is obtained so that an evaluation value based on the RSL value becomes the maximum, and thereby the data of the road shape model is matched with the road shape data.
- RSL road shaped look-up table
- the foregoing step of displaying the image or the foregoing image display means it is possible to display the image in which the read navigation information is composited in the determined appropriate position in the captured real picture on a given display screen of the display unit such as a liquid crystal display panel for car navigation set in the almost central portion of the dashboard or the like.
- the image in which the read navigation information is composited in the determined position in the captured real picture is projected and displayed on the internal surface of a transparent window in front of the driver seat by a display unit such as a so-called HUD (Head Up Display) projection device.
- a display unit such as a so-called HUD (Head Up Display) projection device.
- the navigation information corresponding to the detected current position of the movable body is read from the navigation information previously stored in association with the road map data
- the road shape model of the road assumed to be captured from the current position is created based on the current position of the movable body and the road map data
- the road shape data as the image data of the road included in the landscape is extracted from the real picture
- the data of the road shape model is matched with the road shape data to estimate the posture data of the in-vehicle camera or the movable body to the road as the object
- the display position in the captured real picture of the navigation information read according to the current position of the movable body is determined based on the posture data
- the data for displaying the image of the read navigation information in the determined position is outputted.
- the image of the navigation information is projected and displayed on the internal surface of the transparent window in front of the driver seat of the movable body, and thereby the image of the navigation information is composited and displayed in the landscape seen through the transparent window in front of the driver seat.
- the road shape model of the road assumed to be captured from the current position is created based on the current position of the movable body such as a motor vehicle and the road map data including the current position
- the road shape data as the image data of the road included in the landscape ahead of the vehicle or in the traveling direction is extracted from the real picture
- the data of the road shape model is matched with the road shape data to estimate the posture data of the in-vehicle camera or the movable body to the road in the landscape as the object
- the appropriate display position in the captured real picture of the navigation information read according to the current position of the movable body is determined based on the posture data
- the image in which the read navigation information is composited in the determined appropriate position in the captured real image is displayed.
- the navigation information such as a course guide, a vehicle position, and map information can be accurately projected and displayed in an appropriate position in the real picture or in the real landscape of the road ahead of the movable body.
- the driver can intuitively and accurately recognize a relation between the navigation information and the real picture or the real landscape.
- visibility of a caution-needed picture such as a pedestrian in the real picture and a real picture of a road construction site is not inhibited by the image of the navigation information. Consequently, the visibility can be favorable over the entire display image.
- FIG. 1 is a diagram showing an outline structure of a movable-body navigation information display unit according to an embodiment of the invention
- FIGS. 2A to 2D are views showing a relative position relation among three-dimensional vehicle coordinate system VCS, three-dimensional camera coordinate system CCS, and two-dimensional projection image coordinate system ICS;
- FIG. 3 is a view showing an example of mapping composed of dots and lines expressed by road map data
- FIG. 4 is a diagram showing a road segment horizontal shape model obtained by approximation with the clothoid curved line
- FIG. 5 is a view showing an example of a road horizontal shape model used in 2D-2D matching
- FIG. 6 is a flowchart showing a flow of a series of main processing including extracting road shape data, creating a road shape model, estimating a camera posture parameter and the like in a central processing section;
- FIG. 7 is a diagram showing various mathematical formulas together that are used for various calculations performed in the central processing section
- FIG. 8 is a view showing an example of an image finally displayed
- FIGS. 9A and 9B are schematic views showing rotation of a three-dimensional icon displayed in a composite image
- FIG. 10 is a view showing an example of a screen structure in the case that a character information exclusive display region is previously provided on the periphery of a real picture;
- FIG. 11 is a view showing another example (different from the view of FIG. 8 ) of the image finally displayed;
- FIG. 12 is a schematic view showing a state that 2 three-dimensional icons are displayed while being rotated at a phase different from each other;
- FIG. 13 is a view showing an example of a picture on which an image of navigation information is superimposed, which is stated displayable in Patent document 1;
- FIG. 14 is a view showing an example of a landscape ahead of a real motor vehicle that is seen through a windshield from a driver seat of the real motor vehicle and;
- FIGS. 15A and 15B are schematic views to compare the degree of change of a projection image onto a road surface with respect to unit angle change in the visual line from a driver seat at about 1 m high of a general vehicle ( FIG. 15A ) to the degree of change of a projection image onto a road surface with respect to unit angle change in the visual line at about 10 m high, which is higher than the height in FIG. 15A ( FIG. 15B ).
- FIG. 1 shows an outline structure of a movable-body navigation information display unit of the invention.
- a movable-body navigation information display method according to an embodiment of the invention is embodied by operations or functions of the movable-body navigation information display unit. Therefore, a description will be given of both the movable-body navigation information display unit and the movable-body navigation information display method together.
- the movable-body navigation information display unit includes a sensor input section 1 , an operating section 2 , a map managing section 3 , a central processing section 4 , a picture display section 5 , and a controlling section 6 as the main sections.
- the sensor input section 1 includes a CCD (solid-state image sensing device) camera 101 , a GPS sensor 102 , an INS sensor 103 , and a VICS 104 .
- the CCD camera 101 is set on the dashboard of the driver seat or in the vicinity of the ceiling (not shown) in a movable body such as a motor vehicle (hereinafter referred to as vehicle or the movable body) in which the movable-body navigation information display unit is mounted, so that the CCD camera 101 shoots (captures) a landscape ahead of the vehicle at the camera angle almost similar to the visual line of a driver seeing the landscape through the windshield.
- vehicle or the movable body a motor vehicle
- the CCD camera 101 is, for example, a fixed focal length monocled type camera.
- the CCD camera 101 captures an image of the landscape ahead of the vehicle including a road, and takes in an image signal thereof.
- the taken image signal is transmitted as data to an image memory (not shown) in the central processing section 4 .
- Traveling direction data and speed data of the movable body acquired by the GPS sensor 102 and the INS sensor 103 is transmitted to the central processing section 4 , while being synchronized with the image data acquired by the CCD camera 101 .
- Data received by the VICS 104 as a road traffic information receiver is also transmitted to the central processing section 4 .
- the operating section 2 transmits a command such as system setting and mode change to the central processing section 4 in response to an operation instruction inputted by a user through a button operation, a remote controller input apparatus (not shown) or the like.
- the map managing section 3 reads various information of a road position designated by a command input through the operating section 2 from map data CD 301 that previously records map data in a given geographic region, and transmits read data to the central processing section 4 .
- the central processing section 4 includes four modules composed of an image processing module 401 , a positioning processing module 402 , a picture output processing module 403 , and a control output processing module 404 and an image data creating section 405 as main sections thereof.
- the image processing module 401 performs, for example, posture estimation of the mounted CCD camera 101 , tracing of a driving lane, detection of an obstruction, and calculation of the distance between two vehicles.
- the positioning processing module 402 matches the direction and the speed sent from the sensor input section 1 with the road map data of the map managing section 3 , calculates accurate road position information, and outputs data thereof.
- the picture output processing module 403 expresses a course guide, a vehicle position, and map information in the picture display section 5 as a virtual object in a three-dimensional augmented reality space, projects the virtual object on a two-dimensional road picture with the use of the posture parameter of the CCD camera 101 obtained by the after-mentioned estimation method, and integrates (composites) the virtual object in a real picture of the landscape ahead of the movable body. Further, the picture output processing module 403 creates data for, for example, emphatically displaying road sectional lines in the bad weather and displaying dangerous substances such as an obstruction.
- the picture output processing module 403 iconifies information of objects to be able to become an eyemark in route guide such as a landmark, a railroad station, a hospital, and a gas station as information to be added to the road map, and projects the iconified information on the real picture of the road by using the camera posture parameter.
- the control output processing module 404 comprehensively determines each analysis result, and gives an alarm output instruction for outputting an alarm or the like corresponding to the degree of danger for the vehicle to the controlling section 6 .
- the image data creating section 405 Based on the data mainly outputted from the image processing module and the positioning processing module and the map data read by the map managing section 3 , for example, the image data creating section 405 recognizes the driving lane of the vehicle, a road shape, and an obstruction of currently driving vehicle, recognizes a vehicle's absolute on-road position, and estimates the camera posture. The image data creating section 405 thereby creates data for compositing and displaying the navigation information on an appropriate position in the real picture. More specifically, such display data is created based on the following three-dimensional modeling rules in the virtual space:
- the picture display section 5 displays the picture (image) in which the navigation information is composited in the appropriate position of the real picture on the screen such as a liquid crystal display panel.
- the controlling section 6 controls outputting the alarm or the like corresponding to the foregoing alarm output instruction, controls voice output corresponding to the analysis result by the control output module, controls a brake, and controls steering by, for example, controlling operations of each servo motor system provided for adjusting the control amount thereof.
- the image data creating section 405 creates a road shape model for a road assumed to be captured from the current position, based on current position data detected by the GPS sensor 102 , the INS sensor 103 , and the positioning processing module.
- road shape data as image data of a road included in a landscape in the traveling direction is extracted from the real picture, based on, for example, white line image data of a road traveling sectional line and the like.
- the image data creating section 405 matches the road shape data with data of the road shape model data. Then, the image data creating section 405 estimates posture data of the CCD camera 101 (or posture data of the vehicle) to the road in the landscape as an object captured by the CCD camera 101 .
- an appropriate display position in the captured real picture of the navigation information read according to the current position of the movable body is determined.
- image data that enables displaying an image in which the read navigation information is composited in the determined appropriate position in the captured real picture is created.
- the image data creating section 405 determines what position in the real picture the navigation information is appropriately composited by matching the data of the road shape model with the road shape data.
- the picture display section 5 can display the image in which the navigation information such as the route guide, the vehicle position, and the map information is appropriately composited in the appropriate position in the real picture of the road ahead of the movable body or in the real landscape seen through the windshield.
- the movable-body navigation information display unit of this embodiment it is possible to display the picture with which the relation between the navigation information and the real picture/real landscape can be intuitively and accurately recognized by the driver.
- the foregoing navigation information presents at least one of a course guide for a route to arrive at the destination of the movable body, the vehicle position, the driving lane of the vehicle, and the building as an eyemark for the driver of the movable body to check the course guide or the vehicle position.
- the navigation information is information of a character, a symbol, or a number, it is desirable that the information is iconified, and the icon image is composited and displayed in the captured real picture.
- the image data creating section 405 expresses the navigation information as a virtual object in the three-dimensional augmented reality space, and assigns a corresponding position in the road shape data converted to the two-dimensional characteristics space to the expressed navigation information based on already obtained posture data and the like. Thereby, the image data creating section 405 composites the image of the navigation information as a virtual object in an appropriate position of the real picture.
- the navigation information composed of a character, a symbol, or a number for a building as an eyemark of the course guide can be expressed visually and intuitively, even when the building as the eyemark is hidden by the building or the like in front of the building as the eyemark or is hidden by a curved road.
- the image data creating section 405 converts the road shape data to image data of the perspective two-dimensional characteristics space, and converts the data of the road shape model to image data of the perspective two-dimensional characteristics space.
- the image data creating section 405 matches the foregoing 2 pieces of two-dimensional data each other in the two-dimensional characteristics space, and estimates the posture data of the in-vehicle camera or the movable body to the road surface as an object. Since matching the road shape data with the data of the road shape model performed for estimating the posture data is made with the use of 2 pieces of the two-dimensional data in the pseudo-three-dimensional two-dimensional characteristics space, the matching process is simplified and the speed thereof is accelerated.
- the road shape model is created, modeling is performed by considering the road structure in which the one-way grade of the traveling road surface in the curved road is set to change according to the horizontal curvature of the curved road.
- the posture data of the movable body may be obtained as follows. That is, a road look-up table is used (RSL, reference: “Research on extracting and simultaneous tracing multiple movable bodies by visual motion interpretation of an in-vehicle camera,” H U Zhencheng, doctoral thesis, graduate school of Science and Technology, National University Corporation Kumamoto University). The existence probability of a road white line included in the landscape is obtained from the real picture, and thereby the RSL value is calculated. The posture data of the movable body is obtained so that the evaluation value based on the RSL value becomes the maximum. Thereby, it is possible to always extract accurate road shape data without being adversely influenced by various environmental factors such as weather change and shadow or dirt of the road surface. In the result, an accurate posture data can be estimated by using the extracted data.
- RSL road look-up table
- the image in which the navigation information is composited in the determined appropriate position in the captured real picture on a given display screen of the display unit such as a liquid crystal display panel for car navigation set in the almost central section of the dashboard or the like.
- the image in which the read navigation information is composited in the appropriate position in the captured real picture determined by the foregoing matching is projected and displayed on the internal surface of a transparent window in front of the driver seat by a display unit such as a so-called HUD projection device.
- the image of the navigation information is HUD-projected and displayed on the internal surface of the windshield in front of the driver seat corresponding to the foregoing position. Thereby, the image of the navigation information is composited and displayed in the landscape seen through the transparent window in front of the driver seat.
- FIG. 2 shows a relative position relation among three-dimensional vehicle coordinate system VCS (X v , Y v , Z v ), three-dimensional camera coordinate system CCS (X c , Y c , Z c ), and two-dimensional projected image coordinate system ICS (x i , y i ).
- the origin of the three-dimensional vehicle coordinate system VCS is located at the midpoint of the central line of the vehicle rear wheels.
- the axis Z v is set on the central line, the axis X v is set to be oriented leftward, and the axis Y v is set to be oriented upward, respectively.
- the origin of the three-dimensional camera coordinate system CCS is located at the central point of the lens of the CCD camera.
- the axis Z c is set to be overlapped on the optical axis of the camera.
- the conversion relation from the camera coordinate system CCS to the image coordinate system ICS is orthogonal projection. Therefore, the relation can be described as the relational expression based on a matrix as shown in Formula 1 of FIG. 7 .
- P is the coordinates [X c , Y c , Z c , 1] in the CCS coordinate system
- p is the coordinates [x i , y i , 1] in the ICS coordinate system.
- A is a 3 ⁇ 4 projection matrix, and can be generally decomposed as shown in Formula 2 of FIG. 7 .
- K is called the parameter matrix inside the camera, and is determined by the image horizontal and vertical direction transformation ratio (S x , Sy), the image central point (uo, vo), and the rotation transformation ratio S ⁇ . K is expressed as Formula 3 in FIG. 7 .
- the camera posture matrix M is called the parameter matrix outside the camera, and represents a conversion relation from a view point to the target model coordinate system.
- the camera posture matrix M can be expressed as Formula 4 of FIG. 7 by three-dimensional translation and rotational conversion of a rigid body.
- R11 to R33 (all elements of R) represent the rotation parameters
- T x , T y , and T z , (all elements of T) represent the translation parameters.
- the camera magnification can be approximated with 1. Therefore, based on formula 1 to formula 4, the restraint formula as shown in Formula 5 of FIG. 7 is established.
- the projection relation between the image coordinate system and the vehicle coordinate system is expressed by the formula shown in Formula 6 of FIG. 7 . That is, according to Formula 6, one pair of corresponding points in the 2D-3D space (p, P) determines one restriction formula as shown in Formula 6 of FIG. 7 to the camera posture data. Theoretically, such 6 pairs of corresponding points are enough to estimate the camera posture.
- a multilane road shape model is estimated from road map information by avoiding the matching in 2D-3D (matching of the two-dimensional space with the three-dimensional space).
- the estimated multilane road shape model is matched with multilane road shape data extracted from real picture data in the 2D-2D characteristics space.
- the camera posture data is estimated by such matching in the two-dimensional space to the two-dimensional space.
- the matching is not limited to the matching in the 2D-2D characteristics space.
- the depth data in the three-dimensional space is accurately and surely estimated from an information source other than the real image data of the landscape ahead of the vehicle, and matching of the two-dimensional space to the three-dimensional space is made by using such data.
- the data amount to be processed tends to become larger than that of the case of matching in the 2D-2D characteristics space.
- FIG. 3 shows an example of a mapping with the use of dots and lines expressed by the road map data.
- the road map data three-dimensional position information such as the latitude, the longitude, and the altitude of a road segment generally called a node, a road name, a road class, the number of lanes, a crossing state and the like are recorded.
- the road width can be estimated based on the road class.
- a node position described in the map data is located on the road centerline.
- the structure of the road is composed of a complex curved surface using the horizontal curvature and the vertical curvature.
- FIG. 4 shows a road segment horizontal shape model obtained by approximation with the clothoid curved line.
- a road segment horizontal shape model can be modeled by using the mathematical formula as shown in Formula 7 of FIG. 7 .
- c 0 and c 1 in Formula 7 respectively represent the change parameters of the initial curvature and the curvature of the horizontal curved line.
- nli represents the number of up lanes
- nri represents the number of down lanes
- wi represents the average road width between segments.
- Li represents the segment length.
- the traveling position of vehicles is not on the road centerline, but is often deviated leftward from the road centerline.
- the road horizontal shape model of the road centerline corresponding to the fact that the actual traveling position is deviated to the one lane side can be converted to a new model based on the vehicle coordinate system VCS.
- the projection formula of the road horizontal shape model as shown in Formula 8 of FIG. 7 is estimated from Formula 6 of FIG. 7 as the projection conversion formula.
- the three-dimensional road shape is projected perspectively and two dimensionally as it were. Therefore, the 2D-3D matching having a possibility to result in tangled data processing as a matching method to estimate the camera posture can be simplified to the 2D-2D matching.
- the road shape model estimated from the road map data and the road shape data extracted from the real picture are 2D-2D matched in the two-dimensional characteristics space, and the optimal posture vector is estimated.
- FIG. 5 shows an example of the road horizontal shape model used for the 2D-2D matching.
- the road white line, the sectional line, and a borderline prospect of the road region are extracted as a characteristics region, and the image is binarized (a pixel belonging to the characteristics region is 1, and the other pixels are 0). Then, by using Formula 9 shown in FIG. 7 , the RSL value of each pixel is calculated. ⁇ , x, and y are binarized pixel values. ⁇ , i, and j are the kernel coefficient for the RSL. To inhibit noise, the kernel size is generally set to 5 or 7. Each coefficient is determined by the Gaussin distribution formula. The final evaluation formula of the camera posture estimation is shown in Formula 10 of FIG. 7 . ⁇ is a set of two-dimensional projection points of the road horizontal shape model. By using Formula 10, it is possible to obtain the best RSL evaluation value for totally matching the road shape model created based on the road map data with the road shape data extracted from the real picture.
- the camera posture data obtained as above is used for data matching for determining the position in which the navigation information should be displayed.
- the camera posture data is also used for the next estimation as the feedback amount.
- FIG. 6 is a flowchart showing a flow of a series of main processing including extracting the road shape date, creating the road shape model, estimating the camera posture parameter and the like in the central processing section.
- the real picture data of the landscape including the road ahead of the vehicle took by the CCD camera is captured (S 1 ).
- the data acquired by the GPS sensor and the INS sensor is captured (S 2 ).
- a white line of the road traveling sectional line or the like and a sectional line region of a lane of a paved face or the like are extracted from the real picture data, and the RSL value is calculated (S 3 ).
- the current position (absolute position) of the vehicle is obtained (detected) by so-called map matching, and the related map data corresponding to the current position is read out from information stored in the map data CD (S 4 ).
- the road horizontal shape model for the road in the landscape captured as the real picture is constructed (S 5 ).
- the road horizontal shape model is projected in the perspective two-dimensional space (S 6 ).
- the RSL expression of the road picture is matched with the projected road horizontal shape model, and the evaluation value is obtained (S 7 ).
- Determination is made whether or not the obtained evaluation value is the maximum value (S 8 ). If determining that the evaluation value is the maximum value (Y of S 8 ), the posture vector then is outputted (S 9 ), and the outputted value is feed backed as the search origin for the next estimation processing (S 10 ). If determining that the evaluation value is not the maximum value (N of S 8 ), the posture vector is updated by Hooke&Jeeves method (S 11 ), and reevaluation is made (S 11 to S 6 through S 8 ). This procedural loop is repeated until the maximum value is obtained (until Y of S 8 ).
- the image data creating section 405 creates the data for specifically displaying each piece of the navigation information. More specifically, creating such data is made based on the 9 types of the three-dimensional modeling rules 1 to 9 described above. In the result, the images shown as an example in FIG. 8 to FIG. 12 are displayed on the screen of the picture display section 5 .
- the landmark and the facility information are displayed with the three-dimensional icon expressing them schematically.
- a three-dimensional icon 801 indicating existence of a “hospital” is composited and displayed in the vicinity of the hospital in the real picture.
- the three-dimensional icon 801 indicating the hospital has, for example, the design consisting of a white cross on a red background.
- the size (outer dimensions) and the tint thereof are set to the degree with which the visibility of the picture of the buildings and the road of the real picture as a background is not inhibited.
- the three-dimensional icon 801 is displayed as if the icon 801 rotates at a given rotation speed centering on a vertical axis 802 .
- the three-dimensional icon 801 is oriented to the front face ( FIG. 9A )
- a user can clearly recognize that the icon 801 indicates the “hospital.”
- the icon 801 is turned 90 deg from the state oriented to the front face, as shown in the enlarged view of FIG. 9B , only an extremely thin area of a thickness thereof 803 is displayed. Therefore, the icon does not inhibit visibility of the buildings, pedestrians, obstructions and the like in the real picture existing (or being displayed) behind the three-dimensional icon 801 , and the user can clearly recognize the existence thereof.
- Indicative information such as the traveling direction is displayed with a three-dimensional arrow.
- a virtual road painting 804 indicating the limiting speed
- a three-dimensional arrow 805 indicating the direction to travel
- virtual road paintings 806 , 807 indicating the lane are composited and displayed in the real picture of the road.
- character information or the like for displaying the destination is not displayed to overlap on the region under a virtual vanishing point or the virtual horizon line in the real picture, that is, more specifically, on the road real picture.
- a destination display 808 mainly composed of the character information is displayed in the region of the real picture in which almost only the sky is shown that is located in the vicinity of the upper end of an entire real picture 809 (entire screen in the case of FIG. 8 ).
- the image mainly composed of the character information such as the destination display 808 is not displayed in the region under the virtual vanishing point or the virtual horizon line in the real picture or the real landscape in which roads, pedestrians, and buildings exist with high probability. Thereby, it is possible to inhibit the real picture in which roads, pedestrians, and buildings exist or the real landscape from being hidden and invisible.
- the navigation information is composited and displayed in the determined position in the captured real picture as the three-dimensional icon that rotates centering on the vertical axis (or the horizontal axis).
- each of the plurality of three-dimensional icons may be rotated at the phase different from e one another or at the rotational speed different from one another.
- a first three-dimensional icon 121 and a second three-dimensional icon 122 are displayed so that the icons are rotated in the same direction and at the same rotational speed but at the phase different from each other by about 90 deg.
- the phase different from each other By setting the phase different from each other by about 90 deg, when the first three-dimensional icon 121 is oriented to the front face, the second three-dimensional icon 122 is oriented to the side face, and only the first three-dimensional icon 121 is substantially viewed by the user. Meanwhile, when the second three-dimensional icon 122 is oriented to the front face, the first three-dimensional icon 121 is oriented to the side face, and only the second three-dimensional icon 122 is substantially viewed by the user.
- road name information 811 and limiting speed information 812 set to be displayed as the character information is displayed not in the real picture but in a character information exclusive display region 810 previously provided on the periphery of the real picture 809 .
- each piece of the navigation information can be composited and displayed in a relevant position in the real picture.
- warning information that is, navigation information with the urgency lower than that of the information such as the warning information
- display is made on and after when the current position of the movable body to the warning target in the real picture (or real landscape in the case of HUD) enters within a given distance.
- a description thereof will be given more specifically with reference to an example shown in FIG. 11 .
- 3 three-dimensional icons, 821 (representing a “hotel”), 822 (representing a gas station), 823 (representing a convenience store) are displayed.
- each real substance starts to be displayed when each real substance enters within a predetermined distance such as 100 m to the vehicle of the user.
- Each icon is not displayed when the real substance is located apart from the vehicle farther than the predetermined distance.
- the farthest three-dimensional icon 822 just enters within the given distance and starts to be displayed.
- simple road map information 824 displayed in the lower left of the real picture (screen) of FIG. 11 the image of the navigation information is displayed as a translucent icon so that the real picture as a background can be viewed translucently. It is needless to say that not only the simple road map information 824 but also other information can be displayed as a translucent icon.
- the image of the navigation information can be accurately composited in the appropriate position in the real picture of the road ahead of the movable body or in the real landscape seen thorough the windshield.
- the picture can be displayed so that the driver can intuitively and accurately recognize the relation between the navigation information and the real picture or the real landscape.
Abstract
A movable-body navigation information display unit is provided. In the movable-body navigation information display unit, a driver can intuitively and accurately recognize a relation between navigation information and a real picture or a real landscape. In addition, it is possible to avoid a state that visibility of a caution-needed picture such as a pedestrian in the real picture and a real picture of a road construction site is inhibited by an image of the navigation information. An image data creating section (405) matches road shape data with a road shape model to estimate posture data. In addition, the image data creating section creates picture (image) data for accurately compositing and displaying the image of the navigation information in an appropriate position in a real picture (or in a real landscape) of a road ahead of a movable body, and displays the navigation information as a three-dimensional icon or the like. A picture display section (5) performs display based on the picture data.
Description
The present invention relates to a movable-body navigation information display method and movable-body navigation information display unit for, for example, collecting, processing, and displaying information on a current position of a movable body and information for navigation related thereto, such as a so-called car navigation system and a display method of car navigation information thereby.
movable-body navigation system such as a so-called car navigation system has been developed. In the movable-body navigation system, a current position of a movable body such as a motor vehicle is detected, the detected current position is displayed on a display screen together with a road map of surroundings, various guide information such as a destination and a route is displayed, and voice output is made.
In such an existing car navigation system, the current position of the movable body is checked as follows. That is, an autonomous-navigation controlling section performs tracing on a map previously stored as data in a storing means such as a CD-ROM (Compact Disc Read Only Memory) based on the speed pulses from a speed sensor and a direction obtained by earth magnetism from a geomagnetic sensor. In addition, a GPS signal transmitted from a GPS satellite is used. In the car navigation system in the early stage of the development in this field, there was an apparent malfunction (detection error) or the like in the past. That is, due to a measurement error or the like, an actual traveling position is significantly deviated from the trace position on the map. For example, when a vehicle is traveling along the coast actually, the traveling position of the vehicle detected by the car navigation system is indicated in the sea. However, for addressing such a significant position deviation, a technique of position correction (map matching) and the like have been developed. Therefore, in these days, the precision of detecting the traveling position of the vehicle is sufficient.
By superimposing information of the current position of the vehicle obtained as above on map information obtained from a large-volume data storing means such as a CD-ROM, an image showing the current position and the road map in a given range in the vicinity thereof can be displayed as a plane map-like two-dimensional image on a screen of an image display unit such as a liquid crystal display unit. In addition, it is possible to display so-called navigation information such as a vehicle position, a traveling direction, speed, map information, a course guide, the shortest route, and road information by being superimposed on the two-dimensional image map, or by being inserted in an appropriate position in the two-dimensional image map.
In some existing navigation display units, an image of information, for example, a traveling position and a traveling direction indicated by an icon of an arrow or the like, a recommended route (optimal route) from the current position to the destination is displayed by being superimposed on a two-dimensional image road planar map or on a road map that shows the surrounding landscape in a state of a three-dimensional CG. Further, in some existing navigation display units, traffic jam information or the like obtained by a VICS sensor or the like is displayed with a text label on the screen.
However, such an existing technique can only provide indirect navigation information using the plane (two dimensional) simplified map and the image expressed three dimensionally but abstractly and simply expressed by CG. Therefore, for a driver in the motor vehicle, there are problems, for example, as follows. That is, it is not possible to intuitively recognize a relation between the information such as the current position and the course guide and the road in the landscape actually seen through the windshield.
To solve the foregoing problems, for example, in Patent document 1, Patent document 2 and the like, a navigation system using a real picture in addition to the map information is proposed.
In the technique disclosed in Patent document 1, a current position and a traveling direction of a movable body are detected, and navigation information to be displayed according to the detected current position and the detected traveling direction is selected. An image of the navigation information read according to the current position of the vehicle is superimposed on a real picture captured by an image pickup device from a camera angle approximately similar to a landscape ahead of the vehicle in the traveling direction seen through the windshield of the vehicle, and the superimposed result is displayed on a screen or the like of a liquid crystal display panel. It is stated in Patent document 1 that a relation between the picture of the real road, the surrounding landscape thereof and the navigation information can be displayed easily recognizable visually. Further, in a general car navigation system, the navigation information is mainly picture information displayed on the picture display section. Therefore, there is a possibility that a driver gets distracted easily while driving.
Therefore, in Patent document 2, the following technique is proposed. That is, an image pickup device such as a CCD camera that captures a landscape ahead of a body for motor vehicle is provided, for example, in a position near the ceiling of the windshield of the vehicle or in the vicinity of the dashboard. A picture (image) of the landscape including a road ahead of the body for motor vehicle captured by the image pickup device is displayed by being inserted in a given position of the screen displaying map information as a sub-screen. According to such a technique, even when the driver of the vehicle views navigation information such as a map displayed on the screen of the image display unit while driving, the driver views the real picture of the landscape ahead of the vehicle displayed as the sub-image in the given position on the screen. It is stated in Patent document 2 that the driver can thereby comprehend a state in front of the vehicle without returning her/his attention to the landscape ahead of the vehicle.
Further, the following technique or the like has been proposed. In such a technique, setting is made so that, for example, when it is determined that an obstruction shown in the real picture of the sub-screen is large or when an object suddenly darts out from the side, the outer dimensions of the sub-screen (screen size) are enlarged. Thereby high-risk state in front of the vehicle is informed to the driver promptly and visually. In addition, such a high-risk state can be shown to the driver with high visibility. Consequently, safer driving can be secured.
Further, to determine the leading vehicle and operate automatic steering, it is necessary to more accurately comprehend the landscape including the road and the like ahead of the movable body as data. Therefore, for example, in Patent document 3, the following technique is proposed. In such a technique, a three-dimensional road shape is estimated by using image road information and map road information. In the technique disclosed in Patent document 3, the image road shape extracted from image data of the landscape ahead of the vehicle and map data around the vehicle are projected in one two-dimensional or three-dimensional logical space. Based on an overlapping state in the logical space of both of the projected road shapes, estimation is made for the road shape, a posture of the vehicle to the road surface, an absolute position of the vehicle and the like. In this technique, based on a picture captured by a monocular CCD camera, a sufficiently accurate road shape is estimated, and accurate determination of the leading vehicle and automatic steering are realized.
- Patent document 1: Japanese Unexamined Patent Application Publication No. 10-132598
- Patent document 2: Japanese Unexamined Patent Application Publication No. 11-304499
- Patent document 3: Japanese Unexamined Patent Application Publication No. 2001-331787
However, in the technique disclosed in the Patent document 1, there are problems as follows. That is, first, the image information is only used as a background picture. For example, regarding an image of an arrow indicating the destination, the image is firstly displayed in the central portion of the screen, and then is displayed to move linearly from the original position to the backward landscape (in general, from the upper side to the lower side of the display screen) in the central section of the screen as the vehicle travels.
Therefore, in the case of displaying the navigation information on a curved road or in the case of a road provided with multi-lanes such as two lanes on each side, there are the problems as follows. That is, it is not possible to determine what position in the landscape the navigation information display accurately indicates. In some cases, the navigation information display indicates a totally deviated position.
Secondly, the posture parameter of the camera is previously fixed as the angle of the camera optical axis to the ground surface (or horizontal direction). Therefore, for example, when the posture of the camera is changed due to vibration of the running vehicle, rolling and pitching of motor vehicle by steering, or tilt of the vehicle on the uphill slope and the downhill slope, a navigation information image of route such as right turning position and indicated by an arrow or the like of a traveling direction is largely deviated to the picture obtained by capturing the real landscape. Consequently, the arrow indicates a wrong direction substantially, and the right turning position is displayed unclearly.
For example, in an example shown in FIG. 13 , an arrow 901 as a navigation information image appears to indicate the left turning position on the road in the landscape ahead of the vehicle clearly. However, in the case of a general passenger car, or even in the case of a vehicle such as a bus and a track with the driver seat located in a high position, there is no possibility that the road landscape actually seen through the windshield from the driver seat of the vehicle becomes a bird's-eye view down from a high position as shown in FIG. 13 . In other words, except for a driver seat located in an extremely high position being 10 m or more high from the ground such as a cockpit of a jumbo jet that is unlikely to exist as a driver seat of a motor vehicle, it is not possible actually to capture the road picture in the landscape in which buildings and houses are crammed side by side as the bird's-eye view with which even turning points and back roads of the buildings can be seen as shown in FIG. 13 . In reality, in the case of a general motor vehicle in which the driver's visual line is located only at about 1 m high, or in the case of a general vehicle such as a large truck and a bus in which the driver's visual line is located only at about 2 to 3 m high at the maximum, as an example in FIG. 14 , the road picture often results in a landscape (and real picture) in which anteroposterior density is particularly high and a road that crosses the road in the traveling direction is not easily seen since such a road is hidden by the buildings and the like along the road in the traveling direction.
More specifically, in the technique as Patent document 1 in which the navigation information such as the arrow is located in a fixed position such as on the central line on the screen of the real picture of the landscape, even if the posture of the vehicle is changed slightly, the deviation between the real picture of the landscape and the image of the navigation information becomes large. In addition, it is often the case that the landscape located near the vehicle along the road such as buildings and trees blocks the landscape located farther than the foregoing landscape. Therefore, to address such a problem, it is necessary to display the image of the navigation information on a more accurate position. However, it is not possible to take such a measure in Patent document 1.
Further, in general, in a movable body such as a motor vehicle, the posture thereof is in reality frequently changed due to rolling, pitching and the like of the vehicle while moving. Furthermore, for the foregoing reason, in the landscape viewed from only about 1 to 3 m high of a general vehicle, even slight posture change results in large position deviation. Therefore, in the technology of Patent document 1, there is a possibility that the following state is frequently caused. That is, the display position of the navigation information such as an arrow indicating the route is largely deviated from the appropriate display position in the real picture.
Furthermore, such posture change often varies according to each vehicle type, for example, according to difference of the barycentric position dominantly determined by the structure of a vehicle driving method or the like and the arrangement position of the engine. Therefore, it is necessary to address such posture change differently according to the vehicle type, the degree and the direction of the posture change. However, it is not possible to take such a measure in Patent document 1. Further, in the case of a two-wheeled vehicle and a scooter that pass a curved line by largely tilting the body for motor vehicle and the driver's body, the posture change of the vehicle becomes more significant. Therefore, in the case of the navigation system for such a two-wheeled vehicle, deviation of the display position resulting from the posture change of the vehicle becomes more significant.
Thirdly, it is desirable that a place as a promising eyemark for navigation such as a road name, a landmark, and a route guide of a hospital or the like that are attached to road map data is displayed three dimensionally in real time based on the current position of the vehicle. However, such an aspect is not totally considered or proposed in Patent document 1. Furthermore, as an example shown in FIG. 14 , it is often the case that the building as an eyemark along the road is hardly seen since, such a building is hidden by the building and the landscape near the vehicle.
In the technique of Patent document 2, there is the following problem in addition to the problems similar to those of the Patent document 1 described above. That is, in Patent document 2, the real picture of the road landscape ahead of the vehicle is only displayed in the sub-screen. Therefore, matching of the road map information and the real picture should be made by the driver in his/her mind. Therefore, for example, when the driver drives a road with many crossings and forks in an unfamiliar (or firstly driving) place, it is difficult to intuitively comprehend the navigation information. In the result, there is a possibility that the user takes a wrong route, falsely recognizes the content indicated by the navigation information, and is not able to understand the content.
The technique of Patent document 3 has the following problems. That is, in such a technique, there is no specific road shape model. Therefore, in the case of a multilane road, there is a possibility that large deviation is caused between the central line of the driving lane of the vehicle extracted from the image data and the road centerline estimated from the road map data. Further, it is not possible to estimate information of the driving lane of the vehicle. Thus, accurate navigation information is not able to be provided in changing the lane and in turning left or right.
Further, such a technique does not consider the road structure that the one-way grade (cant) of the traveling road surface is set to be changed according to the horizontal curvature. Therefore, in the curved road, the estimation result of the posture data to the road surface of the lane adjacent to the driving lane of the vehicle is largely changed. In the result, there is a possibility that the road shape is not estimated accurately.
Further, in extracting the road characteristics from the image, the estimation result of the road shape is not feed backed, and the section with large luminance change is simply extracted as a white line by a differential filter for every frame. However, the estimation result obtained by the foregoing method is largely subject to influences of various environmental factors such as weather change and shadow or dirt on the road surface. Therefore, the road shape model expressed by the data obtained by extracting the road characteristics may become an incorrect model that is significantly deviated from the road shape of the real object.
Further, when the image of the navigation information is composited and displayed in the real picture in front of the vehicle, there is a possibility as follows. For example, the image of the navigation information is superimposed on a caution-needed picture such as a pedestrian and other obstruction in the real picture. Thereby, the visibility of the caution-needed picture may be inhibited.
In view of the foregoing, it is an object of the invention to provide a navigation information display method and a movable-body navigation information display unit. In the navigation information display method and the movable-body navigation information display unit, navigation information such as a course guide, a vehicle position, and map information is accurately projected and displayed in an appropriate position in a real picture or in a real landscape of the road ahead of the movable body. Thereby, a driver can intuitively and accurately recognize a relation between the navigation information and the real picture or the real landscape. In addition, visibility of a caution-needed picture such as a pedestrian in the real picture and a real picture of a road construction site is not blocked by the image of the navigation information.
A first movable-body navigation information display method according to the invention includes the steps of detecting a current position of a movable body, and capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data; creating a road shape model of a road assumed to be captured from the current position, based on the current position of the movable body and the road map data; extracting road shape data as image data of the road included in the landscape from the real picture; matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, and determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data; and displaying an image in which the read navigation. information is composited with display dimensions and tone set by a user in the determined position in the captured real picture.
A first movable-body navigation information display unit according to the invention includes: a current position detecting means for detecting a current position of a movable body; a capturing means for capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; a data processing means for reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data, creating a road shape model of a road assumed to be captured from the current position based on the current position of the movable body and the road map data, extracting road shape data as image data of the road included in the landscape from the real picture, matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data, and outputting data for displaying an image in which the read navigation information is composited as an image displayed with display dimensions and tone set by a user in the determined position in the captured real picture; and an image display means for compositing and displaying the read navigation information as the image displayed with the display dimensions and the tone set by the user in the determined position of the captured real picture based on the data outputted from the data processing means.
In the first movable-body navigation information display method or in the first movable-body navigation information display unit according to the invention, the road shape model of the road assumed to be captured from the current position is created based on the current position of the movable body such as a motor vehicle and the road map data including the current position. In addition, the road shape data as the image data of the road included in the landscape ahead of the vehicle or in the traveling direction is extracted from the real picture. The data of the road shape model is matched with the road shape data to estimate the posture data of the in-vehicle camera or the movable body to the road in the landscape as the object. Then, based on the posture data, the appropriate display position in the captured real picture of the navigation information read according to the current position of the movable body is determined. In the result, the image in which the read navigation information is composited with the display dimensions and the tone set by the user is displayed in the determined appropriate position of the captured real picture.
As mentioned above, the appropriate composite position of the navigation information in the real picture is determined by matching the data of the road shape model with the road shape data. Thereby, for example, display is enabled so that the navigation information such as a course guide, a position of a vehicle, and map information is projected accurately in an appropriate position in a real picture of a road ahead of the movable body or a real landscape. In the result, a driver can intuitively and accurately recognize a relation between the navigation information and the real picture or the real landscape. In addition, the display dimensions and the tone of the image (generically referred to as image, including character information and the like) of the navigation information composited in the real picture can be set as appropriate according to the user's option. Therefore, deterioration of visibility of the entire image in which the image of the navigation information is composited in the real picture is avoided. In addition, the user her/himself can customize the image of the navigation information to obtain the image with higher visibility for the user.
A second movable-body navigation information display method according to the invention includes the steps of: detecting a current position of a movable body, and capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data; creating a road shape model of a road assumed to be captured from the current position, based on the current position of the movable body and the road map data; extracting road shape data as image data of the road included in the landscape from the real picture; matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, and determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data; and previously defining three classifications of warning information, guidance information, and additional information depending on urgency of display necessity for the navigation information, previously determining priority for each classification, determining to which classification the navigation information corresponds, and compositing and displaying the navigation information in the determined position in the captured real picture according to an order corresponding to the classifications.
A second movable-body navigation information display unit according to the invention includes: a current position detecting means for detecting a current position of a movable body; a capturing means for capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; a data processing means for reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data, creating a road shape model of a road assumed to be captured from the current position based on the current position of the movable body and the road map data, extracting road shape data as image data of the road included in the landscape from the real picture, matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data, previously defining three classifications of warning information, guidance information, and additional information depending on urgency of display necessity for the navigation information, previously determining priority for each classification, determining to which classification the navigation information corresponds, and outputting data for compositing and displaying the navigation information as an image in the determined position in the captured real picture according to an order corresponding to the classifications and; an image display means for compositing and displaying the image of the read navigation information in the determined position of the captured real picture according to the order corresponding to the classifications, based on the data outputted from the data processing means.
In the second movable-body navigation information display method or in the second movable-body navigation information display unit according to the invention, as in the first movable-body navigation information display method or in the first movable-body navigation information display unit according to the invention, the appropriate composite position of the navigation information in the real picture is determined by matching the data of the road shape model with the road shape data. Thereby, for example, display is enabled so that the navigation information such as a course guide, a position of a vehicle, and map information is projected accurately in an appropriate position in a real picture of a road ahead of the movable body or a real landscape. In the result, a driver can intuitively and accurately recognize a relation between the navigation information and the real picture or the real landscape. In addition, the image of the read navigation information is composited and displayed according to the order corresponding to the classification based on the priority described above. Therefore, many navigation information images are inhibited from being displayed in one real picture. Consequently, deterioration of visibility of the entire image in which the image of the navigation information is composited in the real picture is avoided.
The navigation information other than the foregoing warning information may be displayed on and after when the current position of the movable body to a warning target in the real picture enters within a given distance. Thereby, the data processing amount for creating the image of the navigation information is saved. In addition, a navigation information image necessary for the user can be displayed timely when such navigation information becomes really necessary for the user.
A third movable-body navigation information display method according to the invention includes the steps of: detecting a current position of a movable body, and capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data; creating a road shape model of a road assumed to be captured from the current position, based on the current position of the movable body and the road map data; extracting road shape data as image data of the road included in the landscape from the real picture; matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, and determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data; and displaying an image in which the read navigation information is composited in the determined position in the captured real picture, and displaying the navigation information other than graphic display for guiding the traveling direction not in a region under a virtual vanishing point or a virtual horizon line in the real picture but in a region above the virtual vanishing point or the virtual horizon line.
A third movable-body navigation information display unit according to the invention includes: a current position detecting means for detecting a current position of a movable body; a capturing means for capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; a data processing means for reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data, creating a road shape model of a road assumed to be captured from the current position based on the current position of the movable body and the road map data, extracting road shape data as image data of the road included in the landscape from the real picture, matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data, and outputting data for compositing and displaying the navigation information other than graphic display for guiding the traveling direction not in a region under a virtual vanishing point or a virtual horizon line in the real picture but in a region above the virtual vanishing point or the virtual horizon line; and an image display means for compositing and displaying the read navigation information in the determined position of the captured real picture based on the data outputted from the data processing means.
In the third movable-body navigation information display method or in the third movable-body navigation information display unit according to the invention, as in the first movable-body navigation information display method or in the first movable-body navigation information display unit according to the invention, the appropriate composite position of the navigation information in the real picture is determined by matching the data of the road shape model with the road shape data. Thereby, for example, display is enabled so that the navigation information such as a course guide, a position of a vehicle, and map information is projected accurately in an appropriate position in a real picture of a road ahead of the movable body or a real landscape. In the result, a driver can intuitively and accurately recognize a relation between the navigation information and the real picture or the real landscape. In addition, the navigation information other than the graphic display for guiding the traveling direction is not displayed in the region under the virtual vanishing point or the virtual horizon line in the real picture but in the region above the virtual vanishing point or the virtual horizon line. Consequently, visibility of the real picture of a road and a surface ground is not inhibited by display of the navigation information.
A fourth movable-body navigation information display method according to the invention includes the steps of detecting a current position of a movable body, and capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data; creating a road shape model of a road assumed to be captured from the current position, based on the current position of the movable body and the road map data; extracting road shape data as image data of the road included in the landscape from the real picture; matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, and determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data; and compositing and displaying the read navigation information as a three-dimensional icon that rotates centering on a vertical axis or a horizontal axis in the determined position in the captured real picture.
A fourth movable-body navigation information display unit according to the invention includes: a current position detecting means for detecting a current position of a movable body; a capturing means for capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; a data processing means for reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data, creating a road shape model of a road assumed to be captured from the current position based on the current position of the movable body and the road map data, extracting road shape data as image data of the road included in the landscape from the real picture, matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data, and outputting data for displaying an image in which the read navigation information is composited as a three-dimensional icon that rotates centering on a vertical axis or a horizontal axis in the determined position in the captured real picture; and an image display means for compositing and displaying the read navigation information as the three-dimensional icon that rotates centering on the vertical axis or the horizontal axis in the determined position in the captured real picture based on the data outputted from the data processing means.
In the fourth movable-body navigation information display method or in the fourth movable-body navigation information display unit according to the invention, as in the first movable-body navigation information display method or in the first movable-body navigation information display unit according to the invention, the appropriate composite position of the navigation information in the real picture is determined by matching the data of the road shape model with the road shape data. Thereby, for example, display is enabled so that the navigation information such as a course guide, a position of a vehicle, and map information is projected accurately in an appropriate position in a real picture of a road ahead of the movable body or a real landscape. In the result, a driver can intuitively and accurately recognize a relation between the navigation information and the real picture or the real landscape. In addition, the image of the navigation information is composited and displayed in the real picture as the rotatable three-dimensional icon that rotates centering on the vertical axis or the horizontal axis. Consequently, visibility of the real picture of a road and a surface ground is not inhibited by display of the navigation information.
The foregoing navigation information may be, for example, at least one of a course guide of a route to reach a destination of the movable body, a position of a vehicle, a driving lane of the vehicle, and a building as an eyemark for a driver of the movable body to check the course guide or the position of the vehicle.
When the navigation information is information of a character, a symbol, or a number, the information may be iconified, and the icon image may be composited and displayed in the captured real picture.
Further, in the foregoing step of performing the data processing or the foregoing data processing means, the navigation information may be expressed as a virtual object in the three-dimensional augmented reality space, and a corresponding position in the road shape data converted into the two-dimensional characteristics space may be assigned to the expressed navigation information based on already obtained posture data. Thereby, the image of the navigation information can be composited in the real picture as a virtual object. In the result, the navigation information composed of a character, a symbol, or a number for a building as an eyemark of the course guide can be expressed visually and intuitively, even when the building as the eyemark is hidden by the building or the like in front of the building as the eyemark or is hidden by a curved road.
Further, in the foregoing step of performing the data processing or the foregoing data processing means, it is possible that the road shape data is converted to image data in a perspective two-dimensional characteristics space and the data of the road shape model is converted to image data in the perspective two-dimensional characteristics space, and the two-dimensional road shape data is matched with the two-dimensional data of the road shape model in the two-dimensional characteristics space to estimate the posture data of the in-vehicle camera or the movable body to the road surface as the object. Thereby, matching the road shape data with the data of the road shape model performed for estimating the posture data can be made with the use of 2 pieces of the two-dimensional data in the pseudo-three-dimensional two-dimensional characteristics space, without use of a method assumed to lead to the case that the information amount is enormously large and high speed processing of data becomes difficult such as a method using 2 pieces of three-dimensional data in the three-dimensional logical space. Therefore, the matching process can be simplified and the speed thereof can be accelerated.
Further, in the foregoing step of performing the data processing or the foregoing data processing means, it is possible that in creating the road shape model, the road shape model of a multilane road may be created by performing modeling while considering a road structure in which one-way grade of a traveling road surface in a curved road is set to change according to a horizontal curvature of the curved road. Thereby, even when a movable body such as a motor vehicle is traveling on the multilane road, the road shape of the multilane road can be accurately comprehended. In the result, it is possible to synthesize and display the navigation information in an appropriate position accurately corresponding to such a road shape of the multilane road.
Further, in the foregoing step of performing the data processing or the foregoing data processing means, it is possible that an existence probability of a road white line included in the landscape is obtained from the real picture and an RSL value is calculated by using a road shaped look-up table (RSL), the posture data is obtained so that an evaluation value based on the RSL value becomes the maximum, and thereby the data of the road shape model is matched with the road shape data. Thereby, it is possible to always extract accurate road shape data without being adversely influenced by various environmental factors such as weather change and shadow or dirt of the road surface. In the result, accurate posture data can be estimated by using the extracted data.
Further, in the foregoing step of displaying the image or the foregoing image display means, it is possible to display the image in which the read navigation information is composited in the determined appropriate position in the captured real picture on a given display screen of the display unit such as a liquid crystal display panel for car navigation set in the almost central portion of the dashboard or the like.
Otherwise, it is also possible that the image in which the read navigation information is composited in the determined position in the captured real picture is projected and displayed on the internal surface of a transparent window in front of the driver seat by a display unit such as a so-called HUD (Head Up Display) projection device.
Further, it is also adopt the following aspect or the like. That is, in the foregoing step of performing the data processing or the foregoing data processing means, the navigation information corresponding to the detected current position of the movable body is read from the navigation information previously stored in association with the road map data, the road shape model of the road assumed to be captured from the current position is created based on the current position of the movable body and the road map data, the road shape data as the image data of the road included in the landscape is extracted from the real picture, the data of the road shape model is matched with the road shape data to estimate the posture data of the in-vehicle camera or the movable body to the road as the object, the display position in the captured real picture of the navigation information read according to the current position of the movable body is determined based on the posture data, and the data for displaying the image of the read navigation information in the determined position is outputted. In addition, in the foregoing image display means or in the foregoing step of displaying the image, the image of the navigation information is projected and displayed on the internal surface of the transparent window in front of the driver seat of the movable body, and thereby the image of the navigation information is composited and displayed in the landscape seen through the transparent window in front of the driver seat.
As described above, according to the movable-body navigation information display method or the movable-body navigation information display unit of the invention, the road shape model of the road assumed to be captured from the current position is created based on the current position of the movable body such as a motor vehicle and the road map data including the current position, the road shape data as the image data of the road included in the landscape ahead of the vehicle or in the traveling direction is extracted from the real picture, the data of the road shape model is matched with the road shape data to estimate the posture data of the in-vehicle camera or the movable body to the road in the landscape as the object, the appropriate display position in the captured real picture of the navigation information read according to the current position of the movable body is determined based on the posture data, and the image in which the read navigation information is composited in the determined appropriate position in the captured real image is displayed. Therefore, for example, the navigation information such as a course guide, a vehicle position, and map information can be accurately projected and displayed in an appropriate position in the real picture or in the real landscape of the road ahead of the movable body. Thereby, the driver can intuitively and accurately recognize a relation between the navigation information and the real picture or the real landscape. In addition, visibility of a caution-needed picture such as a pedestrian in the real picture and a real picture of a road construction site is not inhibited by the image of the navigation information. Consequently, the visibility can be favorable over the entire display image.
An embodiment of the invention will be hereinafter described in detail with reference to the drawings.
The movable-body navigation information display unit includes a sensor input section 1, an operating section 2, a map managing section 3, a central processing section 4, a picture display section 5, and a controlling section 6 as the main sections.
More specifically, the sensor input section 1 includes a CCD (solid-state image sensing device) camera 101, a GPS sensor 102, an INS sensor 103, and a VICS 104. The CCD camera 101 is set on the dashboard of the driver seat or in the vicinity of the ceiling (not shown) in a movable body such as a motor vehicle (hereinafter referred to as vehicle or the movable body) in which the movable-body navigation information display unit is mounted, so that the CCD camera 101 shoots (captures) a landscape ahead of the vehicle at the camera angle almost similar to the visual line of a driver seeing the landscape through the windshield. The CCD camera 101 is, for example, a fixed focal length monocled type camera. The CCD camera 101 captures an image of the landscape ahead of the vehicle including a road, and takes in an image signal thereof. The taken image signal is transmitted as data to an image memory (not shown) in the central processing section 4. Traveling direction data and speed data of the movable body acquired by the GPS sensor 102 and the INS sensor 103 is transmitted to the central processing section 4, while being synchronized with the image data acquired by the CCD camera 101. Data received by the VICS 104 as a road traffic information receiver is also transmitted to the central processing section 4.
The operating section 2 transmits a command such as system setting and mode change to the central processing section 4 in response to an operation instruction inputted by a user through a button operation, a remote controller input apparatus (not shown) or the like.
In accordance with a reading command by the central processing section 4, the map managing section 3 reads various information of a road position designated by a command input through the operating section 2 from map data CD 301 that previously records map data in a given geographic region, and transmits read data to the central processing section 4.
The central processing section 4 includes four modules composed of an image processing module 401, a positioning processing module 402, a picture output processing module 403, and a control output processing module 404 and an image data creating section 405 as main sections thereof.
The image processing module 401 performs, for example, posture estimation of the mounted CCD camera 101, tracing of a driving lane, detection of an obstruction, and calculation of the distance between two vehicles.
The positioning processing module 402 matches the direction and the speed sent from the sensor input section 1 with the road map data of the map managing section 3, calculates accurate road position information, and outputs data thereof.
The picture output processing module 403 expresses a course guide, a vehicle position, and map information in the picture display section 5 as a virtual object in a three-dimensional augmented reality space, projects the virtual object on a two-dimensional road picture with the use of the posture parameter of the CCD camera 101 obtained by the after-mentioned estimation method, and integrates (composites) the virtual object in a real picture of the landscape ahead of the movable body. Further, the picture output processing module 403 creates data for, for example, emphatically displaying road sectional lines in the bad weather and displaying dangerous substances such as an obstruction. Furthermore, the picture output processing module 403 iconifies information of objects to be able to become an eyemark in route guide such as a landmark, a railroad station, a hospital, and a gas station as information to be added to the road map, and projects the iconified information on the real picture of the road by using the camera posture parameter.
The control output processing module 404. comprehensively determines each analysis result, and gives an alarm output instruction for outputting an alarm or the like corresponding to the degree of danger for the vehicle to the controlling section 6.
Based on the data mainly outputted from the image processing module and the positioning processing module and the map data read by the map managing section 3, for example, the image data creating section 405 recognizes the driving lane of the vehicle, a road shape, and an obstruction of currently driving vehicle, recognizes a vehicle's absolute on-road position, and estimates the camera posture. The image data creating section 405 thereby creates data for compositing and displaying the navigation information on an appropriate position in the real picture. More specifically, such display data is created based on the following three-dimensional modeling rules in the virtual space:
- 1. The landmark and facility information are displayed with a three-dimensional icon schematically expressing the landmark and the facility information.
- 2. Indicative information such as a traveling direction is displayed with a three-dimensional arrow.
- 3. Fundamentally, all navigation information can be changed by a user. However, it is needless to say that for each navigation information, the default display dimensions and the default color are previously specified. For example, the landmark indicating a hospital is set by default to be displayed with the use of a plate-like three-dimensional icon having a given area and a given thickness with the design consisting of a white cross on a red background. However, it is possible for the user to change the foregoing setting to a plate-like three-dimensional icon with the design consisting of a red and white cross on a white background. Such setting data is inputted by the user with the use of the
operating section 2. The inputted such setting data is further inputted to the imagedata creating section 405 via theimage processing module 401. Then, the imagedata creating section 405 creates data for displaying various navigation information with the dimensions and the color corresponding to each piece of the navigation information. - 4. For the navigation information, three classifications of warning information, guidance information, and additional information are previously defined depending on urgency of display. Information of the definition is previously stored in the image
data creating section 405. The warning information is information for warning the user of pullout of the lane, approximation of an obstruction (or possibility of crash) and the like. The guidance information is information or the like on a traveling direction to the destination, a park availability guide, and the landmark or a large facility along the road. The additional information is information on a medium and small sized facility, a shop and the like along the road. For the respective classification, priorities are previously determined. Information on the priorities is also previously stored in the imagedata creating section 405. Determination is made to which classification of the foregoing classifications the navigation information corresponds. In accordance with the order corresponding to the classification, the data for compositing and displaying each piece of the navigation information in a corresponding position in the real picture is created by the imagedata creating section 405. - 5. The navigation information other than the warning information is displayed on and after when the current position of the movable body to a warning target in the real picture (or the real landscape in the case of HUD) enters within a given distance.
- 6. Of the navigation information, the navigation information other than the three-dimensional arrow for guiding the traveling direction and graphic display such as virtual road painting is not displayed in the region under a virtual vanishing point or a virtual horizon line in the real picture, but is displayed in the region above the virtual vanishing point or the virtual horizon line.
- 7. The navigation information is composited and displayed in a determined position in the captured real picture as a three-dimensional icon that rotates centering on a vertical axis or a horizontal axis. When multiple rotatable three-dimensional icons are displayed in one screen, each of the multiple rotatable three-dimensional icons is displayed while being rotated at the phase different from one another or at the rotational speed different from one another.
- 8. The navigation information is displayed as a translucent icon through which the real picture can be transparently viewed.
- 9. Of the navigation information, character information (information displayed as a character) is not displayed in the real picture, but displayed in a display region preset on the periphery of the real picture.
Based on the data created by the image data creating section 405, the picture display section 5 displays the picture (image) in which the navigation information is composited in the appropriate position of the real picture on the screen such as a liquid crystal display panel.
The controlling section 6, for example, controls outputting the alarm or the like corresponding to the foregoing alarm output instruction, controls voice output corresponding to the analysis result by the control output module, controls a brake, and controls steering by, for example, controlling operations of each servo motor system provided for adjusting the control amount thereof.
Next, a description will be given of operations of the movable-body navigation information display unit.
In the movable-body navigation information display unit, regarding the road map data of the geography in the vicinity of the vehicle including the current position of the vehicle, the image data creating section 405 creates a road shape model for a road assumed to be captured from the current position, based on current position data detected by the GPS sensor 102, the INS sensor 103, and the positioning processing module. In addition, road shape data as image data of a road included in a landscape in the traveling direction is extracted from the real picture, based on, for example, white line image data of a road traveling sectional line and the like.
Subsequently, the image data creating section 405 matches the road shape data with data of the road shape model data. Then, the image data creating section 405 estimates posture data of the CCD camera 101 (or posture data of the vehicle) to the road in the landscape as an object captured by the CCD camera 101.
Next, based on the posture data, an appropriate display position in the captured real picture of the navigation information read according to the current position of the movable body is determined. Then, image data that enables displaying an image in which the read navigation information is composited in the determined appropriate position in the captured real picture is created.
As above, the image data creating section 405 determines what position in the real picture the navigation information is appropriately composited by matching the data of the road shape model with the road shape data. Based on the image data created as above, the picture display section 5 can display the image in which the navigation information such as the route guide, the vehicle position, and the map information is appropriately composited in the appropriate position in the real picture of the road ahead of the movable body or in the real landscape seen through the windshield. In the result, according to the movable-body navigation information display unit of this embodiment, it is possible to display the picture with which the relation between the navigation information and the real picture/real landscape can be intuitively and accurately recognized by the driver.
The foregoing navigation information presents at least one of a course guide for a route to arrive at the destination of the movable body, the vehicle position, the driving lane of the vehicle, and the building as an eyemark for the driver of the movable body to check the course guide or the vehicle position. When the navigation information is information of a character, a symbol, or a number, it is desirable that the information is iconified, and the icon image is composited and displayed in the captured real picture.
Further, the image data creating section 405 expresses the navigation information as a virtual object in the three-dimensional augmented reality space, and assigns a corresponding position in the road shape data converted to the two-dimensional characteristics space to the expressed navigation information based on already obtained posture data and the like. Thereby, the image data creating section 405 composites the image of the navigation information as a virtual object in an appropriate position of the real picture. For example, the navigation information composed of a character, a symbol, or a number for a building as an eyemark of the course guide can be expressed visually and intuitively, even when the building as the eyemark is hidden by the building or the like in front of the building as the eyemark or is hidden by a curved road.
Further, the image data creating section 405 converts the road shape data to image data of the perspective two-dimensional characteristics space, and converts the data of the road shape model to image data of the perspective two-dimensional characteristics space. The image data creating section 405 matches the foregoing 2 pieces of two-dimensional data each other in the two-dimensional characteristics space, and estimates the posture data of the in-vehicle camera or the movable body to the road surface as an object. Since matching the road shape data with the data of the road shape model performed for estimating the posture data is made with the use of 2 pieces of the two-dimensional data in the pseudo-three-dimensional two-dimensional characteristics space, the matching process is simplified and the speed thereof is accelerated.
Further, when the road shape model is created, modeling is performed by considering the road structure in which the one-way grade of the traveling road surface in the curved road is set to change according to the horizontal curvature of the curved road. Thereby, even when the movable body such as a motor vehicle is traveling on a multilane road, the road shape of the multi-lane road can be accurately comprehended. Consequently, it is possible to composite and display the navigation information in an appropriate position accurately corresponding to the road shape of such a multilane road.
Further, in matching the data of the road shape model with the road shape data, the posture data of the movable body may be obtained as follows. That is, a road look-up table is used (RSL, reference: “Research on extracting and simultaneous tracing multiple movable bodies by visual motion interpretation of an in-vehicle camera,” H U Zhencheng, doctoral thesis, Graduate school of Science and Technology, National University Corporation Kumamoto University). The existence probability of a road white line included in the landscape is obtained from the real picture, and thereby the RSL value is calculated. The posture data of the movable body is obtained so that the evaluation value based on the RSL value becomes the maximum. Thereby, it is possible to always extract accurate road shape data without being adversely influenced by various environmental factors such as weather change and shadow or dirt of the road surface. In the result, an accurate posture data can be estimated by using the extracted data.
Further, it is also possible to display the image in which the navigation information is composited in the determined appropriate position in the captured real picture on a given display screen of the display unit such as a liquid crystal display panel for car navigation set in the almost central section of the dashboard or the like.
Further, it is also possible that the image in which the read navigation information is composited in the appropriate position in the captured real picture determined by the foregoing matching is projected and displayed on the internal surface of a transparent window in front of the driver seat by a display unit such as a so-called HUD projection device.
Further, instead of compositing the image of the navigation information in the captured real picture, it is possible to take the following method. That is, by using data of the position determined as the appropriate position to display the image of the navigation information based on the foregoing matching, the image of the navigation information is HUD-projected and displayed on the internal surface of the windshield in front of the driver seat corresponding to the foregoing position. Thereby, the image of the navigation information is composited and displayed in the landscape seen through the transparent window in front of the driver seat.
Next, a description will be hereinafter given of more specific examples of the movable-body navigation information display unit and the movable-body navigation information display method according to this embodiment.
The conversion relation from the camera coordinate system CCS to the image coordinate system ICS is orthogonal projection. Therefore, the relation can be described as the relational expression based on a matrix as shown in Formula 1 of FIG. 7 . P is the coordinates [Xc, Yc, Zc, 1] in the CCS coordinate system, and p is the coordinates [xi, yi, 1] in the ICS coordinate system.
A is a 3×4 projection matrix, and can be generally decomposed as shown in Formula 2 of FIG. 7 . K is called the parameter matrix inside the camera, and is determined by the image horizontal and vertical direction transformation ratio (Sx, Sy), the image central point (uo, vo), and the rotation transformation ratio Sθ. K is expressed as Formula 3 in FIG. 7 .
The camera posture matrix M is called the parameter matrix outside the camera, and represents a conversion relation from a view point to the target model coordinate system. In general, the camera posture matrix M can be expressed as Formula 4 of FIG. 7 by three-dimensional translation and rotational conversion of a rigid body. R11 to R33 (all elements of R) represent the rotation parameters, and Tx, Ty, and Tz, (all elements of T) represent the translation parameters.
In general, the camera magnification can be approximated with 1. Therefore, based on formula 1 to formula 4, the restraint formula as shown in Formula 5 of FIG. 7 is established.
When 6 parameters of the rotation and the translation representing the camera posture are displayed by a posture vector, the projection relation between the image coordinate system and the vehicle coordinate system is expressed by the formula shown in Formula 6 of FIG. 7 . That is, according to Formula 6, one pair of corresponding points in the 2D-3D space (p, P) determines one restriction formula as shown in Formula 6 of FIG. 7 to the camera posture data. Theoretically, such 6 pairs of corresponding points are enough to estimate the camera posture.
However, it is theoretically very difficult or not possible to accurately and surely estimate the depth of the three-dimensional space only by monochrome real image data of the landscape ahead of the vehicle that is captured by the simple monocular CCD camera. Therefore, in this example, a multilane road shape model is estimated from road map information by avoiding the matching in 2D-3D (matching of the two-dimensional space with the three-dimensional space). The estimated multilane road shape model is matched with multilane road shape data extracted from real picture data in the 2D-2D characteristics space. Then, the camera posture data is estimated by such matching in the two-dimensional space to the two-dimensional space. However, it is needless to say that the matching is not limited to the matching in the 2D-2D characteristics space. It is needless to say that, for example, it is also possible to perform estimation and matching as follows. That is, the depth data in the three-dimensional space is accurately and surely estimated from an information source other than the real image data of the landscape ahead of the vehicle, and matching of the two-dimensional space to the three-dimensional space is made by using such data. However, in that case, it is needless to say that in general, the data amount to be processed tends to become larger than that of the case of matching in the 2D-2D characteristics space.
In general, in the country adopting the left-hand traffic such as Japan, the traveling position of vehicles is not on the road centerline, but is often deviated leftward from the road centerline. By using information on the road position of the vehicle (the displacement amount from the origin of the vehicle coordinate system to the center of the road) and the direction (the displacement angle made by the Z axis direction of the vehicle coordinate system and the road horizontal tangent line), the road horizontal shape model of the road centerline corresponding to the fact that the actual traveling position is deviated to the one lane side can be converted to a new model based on the vehicle coordinate system VCS.
Assuming that the road surface within the visible distance (region assumed to be captured as the real picture) is flat, the projection formula of the road horizontal shape model as shown in Formula 8 of FIG. 7 is estimated from Formula 6 of FIG. 7 as the projection conversion formula. In Formula 8 of FIG. 7 , the three-dimensional road shape is projected perspectively and two dimensionally as it were. Therefore, the 2D-3D matching having a possibility to result in tangled data processing as a matching method to estimate the camera posture can be simplified to the 2D-2D matching.
As above, in this example, the road shape model estimated from the road map data and the road shape data extracted from the real picture are 2D-2D matched in the two-dimensional characteristics space, and the optimal posture vector is estimated. FIG. 5 shows an example of the road horizontal shape model used for the 2D-2D matching.
In the actual road environment, it is often the case that a light irradiation state due to rays of the sun, artificial lighting and the like, a weather state and the like lead to large change of the brightness, the tint and the like of the road white line indicating the section of the lane. Therefore, in some cases, it is not possible to successfully and directly match the road shape model with the road shape data extracted from the real picture. Therefore, in this example, instead of the brightness value of the road white line, the existence probability thereof is picturized by using the concept of the road white line look-up table (RSL). Thereby, the high-speed and robust road shape matching can be realized.
As closer to the road white line, the higher the RSL value is. As a specific calculation method, first, the road white line, the sectional line, and a borderline prospect of the road region are extracted as a characteristics region, and the image is binarized (a pixel belonging to the characteristics region is 1, and the other pixels are 0). Then, by using Formula 9 shown in FIG. 7 , the RSL value of each pixel is calculated. λ, x, and y are binarized pixel values. χ, i, and j are the kernel coefficient for the RSL. To inhibit noise, the kernel size is generally set to 5 or 7. Each coefficient is determined by the Gaussin distribution formula. The final evaluation formula of the camera posture estimation is shown in Formula 10 of FIG. 7 . ησ is a set of two-dimensional projection points of the road horizontal shape model. By using Formula 10, it is possible to obtain the best RSL evaluation value for totally matching the road shape model created based on the road map data with the road shape data extracted from the real picture.
To obtain the optimal value of the posture vector σ of the 6 parameters, various local extreme value search methods can be used. Of those methods, in particular, the direct search algorithm by Hooke and Jeeves can be suitably adopted (R. Hooke and T. Jeeves, “Direct search solution of numerical and statistical problems,” Journal of the Association for Computing Machinery (ACM), pp. 212 to 229 (1961)).
The camera posture data obtained as above is used for data matching for determining the position in which the navigation information should be displayed. The camera posture data is also used for the next estimation as the feedback amount.
First, the real picture data of the landscape including the road ahead of the vehicle took by the CCD camera is captured (S1).
Concurrently, in sync with captured real picture data, the data acquired by the GPS sensor and the INS sensor is captured (S2).
Next, a white line of the road traveling sectional line or the like and a sectional line region of a lane of a paved face or the like are extracted from the real picture data, and the RSL value is calculated (S3).
Further, the current position (absolute position) of the vehicle is obtained (detected) by so-called map matching, and the related map data corresponding to the current position is read out from information stored in the map data CD (S4).
The road horizontal shape model for the road in the landscape captured as the real picture is constructed (S5).
Based on the updated camera posture vector, the road horizontal shape model is projected in the perspective two-dimensional space (S6).
The RSL expression of the road picture is matched with the projected road horizontal shape model, and the evaluation value is obtained (S7).
Determination is made whether or not the obtained evaluation value is the maximum value (S8). If determining that the evaluation value is the maximum value (Y of S8), the posture vector then is outputted (S9), and the outputted value is feed backed as the search origin for the next estimation processing (S10). If determining that the evaluation value is not the maximum value (N of S8), the posture vector is updated by Hooke&Jeeves method (S11), and reevaluation is made (S11 to S6 through S8). This procedural loop is repeated until the maximum value is obtained (until Y of S8).
When it comes to the time to start processing of taking in the next data (Y of S12), the foregoing series of processing is executed again from the first step of capturing the data in the order similar to the foregoing order.
The matching is performed as above, and determination is made that what navigation is displayed in what position. Next, the image data creating section 405 creates the data for specifically displaying each piece of the navigation information. More specifically, creating such data is made based on the 9 types of the three-dimensional modeling rules 1 to 9 described above. In the result, the images shown as an example in FIG. 8 to FIG. 12 are displayed on the screen of the picture display section 5.
The landmark and the facility information are displayed with the three-dimensional icon expressing them schematically. In an example of FIG. 8 , a three-dimensional icon 801 indicating existence of a “hospital” is composited and displayed in the vicinity of the hospital in the real picture. The three-dimensional icon 801 indicating the hospital has, for example, the design consisting of a white cross on a red background. The size (outer dimensions) and the tint thereof are set to the degree with which the visibility of the picture of the buildings and the road of the real picture as a background is not inhibited.
As shown in the enlarged view of FIGS. 9A and 9B , the three-dimensional icon 801 is displayed as if the icon 801 rotates at a given rotation speed centering on a vertical axis 802. When the three-dimensional icon 801 is oriented to the front face (FIG. 9A ), a user can clearly recognize that the icon 801 indicates the “hospital.” When the icon 801 is turned 90 deg from the state oriented to the front face, as shown in the enlarged view of FIG. 9B , only an extremely thin area of a thickness thereof 803 is displayed. Therefore, the icon does not inhibit visibility of the buildings, pedestrians, obstructions and the like in the real picture existing (or being displayed) behind the three-dimensional icon 801, and the user can clearly recognize the existence thereof.
As described above, fundamentally, it is possible for the user to change the display dimensions and the display color (use of color and tint) for all navigation information such as the three-dimensional icon 801.
Indicative information such as the traveling direction is displayed with a three-dimensional arrow. In the example of FIG. 8 , for example, on the road lane on which the user's vehicle currently goes straight ahead, a virtual road painting 804 indicating the limiting speed, a three-dimensional arrow 805 indicating the direction to travel, and virtual road paintings 806, 807 indicating the lane are composited and displayed in the real picture of the road. However, character information or the like for displaying the destination is not displayed to overlap on the region under a virtual vanishing point or the virtual horizon line in the real picture, that is, more specifically, on the road real picture.
In the example of FIG. 8 , a destination display 808 mainly composed of the character information is displayed in the region of the real picture in which almost only the sky is shown that is located in the vicinity of the upper end of an entire real picture 809 (entire screen in the case of FIG. 8 ). As above, the image mainly composed of the character information such as the destination display 808 is not displayed in the region under the virtual vanishing point or the virtual horizon line in the real picture or the real landscape in which roads, pedestrians, and buildings exist with high probability. Thereby, it is possible to inhibit the real picture in which roads, pedestrians, and buildings exist or the real landscape from being hidden and invisible.
Further, as described above, the navigation information is composited and displayed in the determined position in the captured real picture as the three-dimensional icon that rotates centering on the vertical axis (or the horizontal axis). In the case that a plurality of three-dimensional icons are displayed on one screen, each of the plurality of three-dimensional icons may be rotated at the phase different from e one another or at the rotational speed different from one another. In an example shown in FIG. 12 , a first three-dimensional icon 121 and a second three-dimensional icon 122 are displayed so that the icons are rotated in the same direction and at the same rotational speed but at the phase different from each other by about 90 deg.
By setting the phase different from each other by about 90 deg, when the first three-dimensional icon 121 is oriented to the front face, the second three-dimensional icon 122 is oriented to the side face, and only the first three-dimensional icon 121 is substantially viewed by the user. Meanwhile, when the second three-dimensional icon 122 is oriented to the front face, the first three-dimensional icon 121 is oriented to the side face, and only the second three-dimensional icon 122 is substantially viewed by the user. By limiting the number of the three-dimensional icons that can be viewed by the user at once as above, it is possible to avoid tangle and visibility deterioration of the display screen resulting from a state that the plurality of three-dimensional icons are seen in parallel at once. In addition to setting different phases, for example, it is possible to set the rotational speed of the plurality of three-dimensional icons different from each other.
Otherwise, it is possible to adopt the method as shown in an example of FIG. 10 . In the method, of the navigation information, road name information 811 and limiting speed information 812 set to be displayed as the character information is displayed not in the real picture but in a character information exclusive display region 810 previously provided on the periphery of the real picture 809.
Otherwise, it is possible to classify the navigation information into three types of the warning information, the guidance information, and the additional information depending on urgency of display. In the order corresponding to the classification, each piece of the navigation information can be composited and displayed in a relevant position in the real picture.
That is, for general navigation information such as a land mark and small to medium facilities other than the information necessary for urgently issuing a warning to the user such as warning information (that is, navigation information with the urgency lower than that of the information such as the warning information), display is made on and after when the current position of the movable body to the warning target in the real picture (or real landscape in the case of HUD) enters within a given distance. A description thereof will be given more specifically with reference to an example shown in FIG. 11 . At this point, 3 three-dimensional icons, 821 (representing a “hotel”), 822 (representing a gas station), 823 (representing a convenience store) are displayed. All thereof starts to be displayed when each real substance enters within a predetermined distance such as 100 m to the vehicle of the user. Each icon is not displayed when the real substance is located apart from the vehicle farther than the predetermined distance. In the example of FIG. 11 , the farthest three-dimensional icon 822 just enters within the given distance and starts to be displayed.
Further, as simple road map information 824 displayed in the lower left of the real picture (screen) of FIG. 11 , the image of the navigation information is displayed as a translucent icon so that the real picture as a background can be viewed translucently. It is needless to say that not only the simple road map information 824 but also other information can be displayed as a translucent icon.
As hereinbefore described in details, according to the movable-body navigation information display unit and the movable-body navigation information display means of this embodiment, the image of the navigation information can be accurately composited in the appropriate position in the real picture of the road ahead of the movable body or in the real landscape seen thorough the windshield. In the result, the picture can be displayed so that the driver can intuitively and accurately recognize the relation between the navigation information and the real picture or the real landscape. In addition, it is possible to avoid the state that visibility of the real picture (or real landscape) is inhibited by the image of the navigation information due to the composition of the navigation information by the various methods described above. Consequently, the visibility can be favorable over the entire display image.
Claims (6)
1. A movable-body navigation information display method comprising the steps of:
detecting a current position of a movable body, and capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body;
reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data;
creating a road shape model of a road assumed to be captured from the current position, based on the current position of the movable body and the road map data;
extracting road shape data as image data of the road included in the landscape from the real picture;
converting the road shape data to image data in a perspective two-dimensional characteristics space and the data of the road shape model to image data in the perspective two-dimensional characteristics space;
obtaining an existence probability of a road white line included in the landscape from the real picture and calculating an RSL value by using a road look-up table (RSL);
matching the two-dimensional road shape data with the two-dimensional data of the road shape model in the two-dimensional characteristics space so that an evaluation value based on the RSL value becomes the maximum, and thereby estimating the posture data of the in-vehicle camera or the movable body to the road as the object;
determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data; and
displaying an image in which the read navigation information is composited with display dimensions and tone set by a user in the determined position in the captured real picture.
2. The movable-body navigation information display method according to claim 1 , wherein in creating the road shape model, the road shape model of a multilane road is created by performing modeling while considering a road structure in which one-way grade of a traveling road surface in a curved road is set to change according to a horizontal curvature of the curved road.
3. The movable-body navigation information display method according to claim 1 , wherein the navigation information is at least one of a course guide of a route to reach a destination of the movable body, a position of a vehicle, a driving lane of the vehicle, and a building as an eyemark for a driver of the movable body to check the course guide or the position of the vehicle.
4. A movable-body navigation information display unit comprising:
a current position detecting means for detecting a current position of a movable body;
a capturing means for capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body;
a data processing means for reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data, creating a road shape model of a road assumed to be captured from the current position based on the current position of the movable body and the road map data, extracting road shape data as image data of the road included in the landscape from the real picture, converting the road shape data to image data in a perspective two-dimensional characteristics space and the data of the road shape model to image data in the perspective two-dimensional characteristics space, obtaining an existence probability of a road white line included in the landscape from the real picture and calculating an RSL value by using a road look-up table (RSL), matching the two-dimensional road shape data with the two dimensional data of the road shape model in the two-dimensional characteristics space so that an evaluation value based on the RSL value becomes the maximum and thereby estimating the posture data of the in-vehicle camera or the movable body to the road as the object, determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data, and outputting data for displaying an image in which the read navigation information is composited as an image displayed with display dimensions and tone set by a user in the determined position in the captured real picture; and
an image display means for compositing and displaying the read navigation information as the image displayed with the display dimensions and the tone set by the user in the determined position of the captured real picture based on the data outputted from the data processing means.
5. The movable-body navigation information display unit according to claim 4 , wherein in creating the road shape model, the data processing means creates the road shape model of a multilane road by performing modeling while considering a road structure in which one-way grade of a traveling road surface in a curved road is set to change according to a horizontal curvature of the curved road.
6. The movable-body navigation information display unit according to claim 4 , wherein the navigation information is at least one of a course guide of a route to reach a destination of the movable body, a position of a vehicle, a driving lane of the vehicle, a building as an eyemark for a driver of the movable body to check the course guide or the position of the vehicle.
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2004282190 | 2004-09-28 | ||
JP2004-282190 | 2004-09-28 | ||
PCT/JP2005/017715 WO2006035755A1 (en) | 2004-09-28 | 2005-09-27 | Method for displaying movable-body navigation information and device for displaying movable-body navigation information |
Publications (2)
Publication Number | Publication Date |
---|---|
US20080195315A1 US20080195315A1 (en) | 2008-08-14 |
US8195386B2 true US8195386B2 (en) | 2012-06-05 |
Family
ID=36118900
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/663,909 Expired - Fee Related US8195386B2 (en) | 2004-09-28 | 2005-09-27 | Movable-body navigation information display method and movable-body navigation information display unit |
Country Status (3)
Country | Link |
---|---|
US (1) | US8195386B2 (en) |
JP (1) | JP4696248B2 (en) |
WO (1) | WO2006035755A1 (en) |
Cited By (35)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100030462A1 (en) * | 2007-02-15 | 2010-02-04 | Pioneer Corporation | Display control apparatus, display control method, display control program, and recording medium |
US20100280747A1 (en) * | 2008-05-02 | 2010-11-04 | Olaf Achthoven | Navigation device and method for displaying map information |
US20110054783A1 (en) * | 2008-01-28 | 2011-03-03 | Geo Technical Laboratory Co., Ltd. | Data structure of route guidance database |
US20120083923A1 (en) * | 2009-06-01 | 2012-04-05 | Kosei Matsumoto | Robot control system, robot control terminal, and robot control method |
US20120194554A1 (en) * | 2011-01-28 | 2012-08-02 | Akihiko Kaino | Information processing device, alarm method, and program |
US20120296561A1 (en) * | 2011-05-16 | 2012-11-22 | Samsung Electronics Co., Ltd. | User interface method for terminal for vehicle and apparatus thereof |
US20120307002A1 (en) * | 2007-02-06 | 2012-12-06 | Meadow William D | Methods and apparatus for generating a continuum of image data |
US20130124084A1 (en) * | 2011-11-15 | 2013-05-16 | Jaehong SEO | Mobile terminal and method of controlling the same |
US20130158849A1 (en) * | 2011-12-15 | 2013-06-20 | Masao MAURA | Evaluation indication system, evaluation indication method and computer-readable storage medium |
US20130304383A1 (en) * | 2012-05-11 | 2013-11-14 | Honeywell International Inc. | Systems and methods for landmark selection for navigation |
US20140210822A1 (en) * | 2013-01-31 | 2014-07-31 | Samsung Electronics Co. Ltd. | Apparatus and method for compass intelligent lighting for user interfaces |
US8890866B2 (en) | 2004-08-31 | 2014-11-18 | Visual Real Estate, Inc. | Method and apparatus of providing street view data of a comparable real estate property |
US20140372020A1 (en) * | 2013-06-13 | 2014-12-18 | Gideon Stein | Vision augmented navigation |
US20150019128A1 (en) * | 2003-06-19 | 2015-01-15 | Here Global B.V. | Method and System for Representing Traffic Signals in a Road Network Database |
US20150073663A1 (en) * | 2013-09-12 | 2015-03-12 | Volvo Car Corporation | Manoeuver generation for automated driving |
KR20150062418A (en) * | 2013-11-29 | 2015-06-08 | (주) 유라시아디지털문화유산연구소 | System for displaying virtual image matched with real street figure |
US20150226568A1 (en) * | 2014-02-11 | 2015-08-13 | Hyundai Motor Company | Apparatus and method of providing road guidance based on augmented reality head-up display |
US9304003B1 (en) | 2015-03-18 | 2016-04-05 | Microsoft Technology Licensing, Llc | Augmented reality navigation |
US9341485B1 (en) | 2003-06-19 | 2016-05-17 | Here Global B.V. | Method and apparatus for representing road intersections |
US9395192B1 (en) * | 2013-12-20 | 2016-07-19 | Google Inc. | Methods and systems for road and lane boundary tracing |
US9435648B2 (en) | 2012-12-17 | 2016-09-06 | Industrial Technology Research Institute | Map matching device, system and method |
US9652980B2 (en) | 2008-04-24 | 2017-05-16 | GM Global Technology Operations LLC | Enhanced clear path detection in the presence of traffic infrastructure indicator |
US9852357B2 (en) | 2008-04-24 | 2017-12-26 | GM Global Technology Operations LLC | Clear path detection using an example-based approach |
US9909894B2 (en) | 2016-01-07 | 2018-03-06 | Here Global B.V. | Componentized junction models |
US20180341822A1 (en) * | 2017-05-26 | 2018-11-29 | Dura Operating, Llc | Method and system for classifying objects in a perception scene graph by using a scene-detection-schema |
US10234294B2 (en) | 2016-04-01 | 2019-03-19 | Here Global B.V. | Road geometry matching with componentized junction models |
US20190138261A1 (en) * | 2016-06-22 | 2019-05-09 | Sony Corporation | Information processing device, information processing method, and program |
US10488215B1 (en) | 2018-10-26 | 2019-11-26 | Phiar Technologies, Inc. | Augmented reality interface for navigation assistance |
US10495476B1 (en) | 2018-09-27 | 2019-12-03 | Phiar Technologies, Inc. | Augmented reality navigation systems and methods |
US20190384294A1 (en) * | 2015-02-10 | 2019-12-19 | Mobileye Vision Technologies Ltd. | Crowd sourcing data for autonomous vehicle navigation |
US10573183B1 (en) | 2018-09-27 | 2020-02-25 | Phiar Technologies, Inc. | Mobile real-time driving safety systems and methods |
US10663315B2 (en) * | 2015-02-09 | 2020-05-26 | Denso Corporation | Vehicle display control device and vehicle display control method |
US20220111840A1 (en) * | 2019-01-10 | 2022-04-14 | Jaguar Land Rover Limited | A control system, system and method for providing assistance to an occupant of a vehicle |
US11448518B2 (en) | 2018-09-27 | 2022-09-20 | Phiar Technologies, Inc. | Augmented reality navigational overlay |
US11499834B2 (en) * | 2019-03-07 | 2022-11-15 | Mobileye Vision Technologies Ltd. | Aligning road information for navigation |
Families Citing this family (102)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8108142B2 (en) * | 2005-01-26 | 2012-01-31 | Volkswagen Ag | 3D navigation system for motor vehicles |
DE102005018082A1 (en) * | 2005-04-19 | 2006-10-26 | Robert Bosch Gmbh | Method for the three-dimensional representation of a digital road map |
US8423292B2 (en) * | 2008-08-19 | 2013-04-16 | Tomtom International B.V. | Navigation device with camera-info |
NZ564319A (en) | 2005-06-06 | 2009-05-31 | Tomtom Int Bv | Navigation device with camera-info |
US7728869B2 (en) * | 2005-06-14 | 2010-06-01 | Lg Electronics Inc. | Matching camera-photographed image with map data in portable terminal and travel route guidance method |
KR100657962B1 (en) * | 2005-06-21 | 2006-12-14 | 삼성전자주식회사 | Apparatus and method for displaying 3-dimension graphics |
JP4246195B2 (en) * | 2005-11-01 | 2009-04-02 | パナソニック株式会社 | Car navigation system |
JP4935145B2 (en) * | 2006-03-29 | 2012-05-23 | 株式会社デンソー | Car navigation system |
JP2008039596A (en) * | 2006-08-07 | 2008-02-21 | Pioneer Electronic Corp | System, method, program for providing information and memory medium |
JP4231884B2 (en) * | 2006-09-27 | 2009-03-04 | 株式会社デンソーアイティーラボラトリ | Gaze object detection device and gaze object detection method |
US7990394B2 (en) | 2007-05-25 | 2011-08-02 | Google Inc. | Viewing and navigating within panoramic images, and applications thereof |
JP2009036726A (en) * | 2007-08-03 | 2009-02-19 | Yahoo Japan Corp | Method and system for displaying search path |
US20090089705A1 (en) * | 2007-09-27 | 2009-04-02 | Microsoft Corporation | Virtual object navigation |
JP2009163504A (en) * | 2008-01-07 | 2009-07-23 | Panasonic Corp | Image deformation method and the like |
JP4986883B2 (en) * | 2008-02-27 | 2012-07-25 | 三菱電機株式会社 | Orientation device, orientation method and orientation program |
US8452053B2 (en) * | 2008-04-24 | 2013-05-28 | GM Global Technology Operations LLC | Pixel-based texture-rich clear path detection |
US8634593B2 (en) * | 2008-04-24 | 2014-01-21 | GM Global Technology Operations LLC | Pixel-based texture-less clear path detection |
CN102016929A (en) * | 2008-05-29 | 2011-04-13 | 通腾科技股份有限公司 | Generating a display image |
JP5223502B2 (en) * | 2008-07-03 | 2013-06-26 | ヤマハ株式会社 | Direction tracking display device, direction tracking display method, and direction tracking display program |
DE102008045994A1 (en) * | 2008-09-05 | 2010-03-11 | Volkswagen Ag | Method and device for displaying information in a vehicle |
JP5372157B2 (en) * | 2008-09-17 | 2013-12-18 | ノキア コーポレイション | User interface for augmented reality |
US20100085350A1 (en) * | 2008-10-02 | 2010-04-08 | Microsoft Corporation | Oblique display with additional detail |
JP2010151658A (en) * | 2008-12-25 | 2010-07-08 | Zhencheng Hu | Apparatus and method for measuring position of mobile object |
JP5161760B2 (en) * | 2008-12-26 | 2013-03-13 | 株式会社東芝 | In-vehicle display system and display method |
DE102009006471A1 (en) * | 2009-01-28 | 2010-09-02 | Audi Ag | Method for operating a navigation device of a motor vehicle and motor vehicle therefor |
US20100198506A1 (en) * | 2009-02-03 | 2010-08-05 | Robert Steven Neilhouse | Street and landmark name(s) and/or turning indicators superimposed on user's field of vision with dynamic moving capabilities |
JP5402160B2 (en) * | 2009-03-31 | 2014-01-29 | 沖電気工業株式会社 | Map display device, map display method and program |
US9477368B1 (en) | 2009-03-31 | 2016-10-25 | Google Inc. | System and method of indicating the distance or the surface of an image of a geographical object |
US20110098910A1 (en) * | 2009-10-22 | 2011-04-28 | Nokia Corporation | Method and apparatus for intelligent guidance using markers |
JP2011128838A (en) * | 2009-12-17 | 2011-06-30 | Panasonic Corp | Image display device |
KR101001842B1 (en) * | 2010-01-28 | 2010-12-17 | 팅크웨어(주) | Navigation device for a vehicle and method for inducing right position of blackbox device of a navigation system |
CN102713948A (en) * | 2010-02-01 | 2012-10-03 | 日本电气株式会社 | Portable terminal, action history depiction method, and action history depiction system |
JP5136671B2 (en) * | 2010-05-13 | 2013-02-06 | 株式会社デンソー | Map display device |
JP5413516B2 (en) * | 2010-08-19 | 2014-02-12 | 日産自動車株式会社 | Three-dimensional object detection apparatus and three-dimensional object detection method |
JP2012068481A (en) * | 2010-09-24 | 2012-04-05 | Asia Air Survey Co Ltd | Augmented reality expression system and method |
US10331864B1 (en) | 2010-12-27 | 2019-06-25 | Mlb Advanced Media, L.P. | Watermarking systems and methods |
KR101708207B1 (en) | 2011-01-10 | 2017-02-20 | 삼성전자주식회사 | Apparatus and method for providing user's route information in mobile communication system |
DE102011006347B4 (en) * | 2011-03-29 | 2023-02-09 | Bayerische Motoren Werke Aktiengesellschaft | Process for outputting graphical driving instructions |
JP5521134B2 (en) * | 2011-08-12 | 2014-06-11 | 新日鉄住金ソリューションズ株式会社 | Information processing system, information processing method, and program |
KR101407670B1 (en) * | 2011-09-15 | 2014-06-16 | 주식회사 팬택 | Mobile terminal, server and method for forming communication channel using augmented reality |
US9177204B1 (en) * | 2011-09-28 | 2015-11-03 | Rockwell Collins, Inc. | Spectrally enhanced vision system for low visibility operations |
JP5708449B2 (en) * | 2011-11-08 | 2015-04-30 | アイシン・エィ・ダブリュ株式会社 | Lane guidance display system, method and program |
JP5735658B2 (en) * | 2011-12-15 | 2015-06-17 | パイオニア株式会社 | Display device and display method |
JP5910180B2 (en) * | 2012-03-06 | 2016-04-27 | 日産自動車株式会社 | Moving object position and orientation estimation apparatus and method |
JP6000780B2 (en) * | 2012-09-21 | 2016-10-05 | オリンパス株式会社 | Imaging device |
US20140095294A1 (en) * | 2012-09-28 | 2014-04-03 | Cornelius V. Vick | Mechanism for facilitating context-aware broadcast and virtual visualization of advertisements |
CN103868517A (en) * | 2012-12-10 | 2014-06-18 | 深圳富泰宏精密工业有限公司 | Indoor positioning system and method |
US20140184643A1 (en) * | 2012-12-27 | 2014-07-03 | Caterpillar Inc. | Augmented Reality Worksite |
US20140232746A1 (en) * | 2013-02-21 | 2014-08-21 | Hyundai Motor Company | Three dimensional augmented reality display apparatus and method using eye tracking |
JP2014220604A (en) * | 2013-05-07 | 2014-11-20 | 三菱電機株式会社 | Photographing position information display device |
JP6236954B2 (en) * | 2013-07-23 | 2017-11-29 | アイシン・エィ・ダブリュ株式会社 | Driving support system, method and program |
FR3011923B1 (en) * | 2013-10-14 | 2017-05-26 | Renault Sas | METHOD FOR GUIDING A USER OF A MOTOR VEHICLE, CORRESPONDING SYSTEM AND MOTOR VEHICLE |
KR101519277B1 (en) * | 2013-12-24 | 2015-05-11 | 현대자동차주식회사 | Apparatus and Method for Recognize of Drive Way of Vehicle |
JP6376807B2 (en) * | 2014-04-02 | 2018-08-22 | キヤノン株式会社 | Display device, display control method, and program |
US9858720B2 (en) | 2014-07-25 | 2018-01-02 | Microsoft Technology Licensing, Llc | Three-dimensional mixed-reality viewport |
US10451875B2 (en) | 2014-07-25 | 2019-10-22 | Microsoft Technology Licensing, Llc | Smart transparency for virtual objects |
US10311638B2 (en) | 2014-07-25 | 2019-06-04 | Microsoft Technology Licensing, Llc | Anti-trip when immersed in a virtual reality environment |
US10416760B2 (en) | 2014-07-25 | 2019-09-17 | Microsoft Technology Licensing, Llc | Gaze-based object placement within a virtual reality environment |
US9904055B2 (en) | 2014-07-25 | 2018-02-27 | Microsoft Technology Licensing, Llc | Smart placement of virtual objects to stay in the field of view of a head mounted display |
US9766460B2 (en) * | 2014-07-25 | 2017-09-19 | Microsoft Technology Licensing, Llc | Ground plane adjustment in a virtual reality environment |
US9865089B2 (en) | 2014-07-25 | 2018-01-09 | Microsoft Technology Licensing, Llc | Virtual reality environment with real world objects |
US11370422B2 (en) * | 2015-02-12 | 2022-06-28 | Honda Research Institute Europe Gmbh | Method and system in a vehicle for improving prediction results of an advantageous driver assistant system |
CN104833367A (en) * | 2015-05-11 | 2015-08-12 | 京东方科技集团股份有限公司 | Vehicle-mounted projection system |
JP6696149B2 (en) * | 2015-10-29 | 2020-05-20 | 富士通株式会社 | Image generation method, image generation program, information processing device, and display control method |
US10217283B2 (en) | 2015-12-17 | 2019-02-26 | Google Llc | Navigation through multidimensional images spaces |
KR101916993B1 (en) * | 2015-12-24 | 2018-11-08 | 엘지전자 주식회사 | Display apparatus for vehicle and control method thereof |
US10935789B2 (en) * | 2016-03-31 | 2021-03-02 | Honda Motor Co., Ltd. | Image display apparatus and image display method |
KR20180070198A (en) * | 2016-12-16 | 2018-06-26 | 현대자동차주식회사 | Vehicle and controlling method of vehicle |
US10839203B1 (en) | 2016-12-27 | 2020-11-17 | Amazon Technologies, Inc. | Recognizing and tracking poses using digital imagery captured from multiple fields of view |
JP6930120B2 (en) * | 2017-02-02 | 2021-09-01 | 株式会社リコー | Display device, mobile device and display method. |
US9992461B1 (en) * | 2017-02-08 | 2018-06-05 | Hyundai Motor Company | Projection orientation correction system for vehicle |
US10977953B2 (en) * | 2017-02-17 | 2021-04-13 | The Charles Stark Draper Laboratory, Inc. | Probabilistic landmark navigation (PLN) system |
US10339812B2 (en) * | 2017-03-02 | 2019-07-02 | Denso International America, Inc. | Surrounding view camera blockage detection |
JP6759132B2 (en) * | 2017-03-10 | 2020-09-23 | アルパイン株式会社 | Head-up display device and display control method |
JP7151073B2 (en) * | 2017-03-31 | 2022-10-12 | 株式会社アイシン | Display device and computer program |
KR102427854B1 (en) | 2017-09-25 | 2022-08-01 | 삼성전자주식회사 | Method and apparatus for rendering image |
US11232294B1 (en) | 2017-09-27 | 2022-01-25 | Amazon Technologies, Inc. | Generating tracklets from digital imagery |
KR102434580B1 (en) * | 2017-11-09 | 2022-08-22 | 삼성전자주식회사 | Method and apparatus of dispalying virtual route |
US11511627B2 (en) | 2017-11-17 | 2022-11-29 | Aisin Corporation | Display device and computer program |
US20190340797A1 (en) * | 2018-05-03 | 2019-11-07 | Visteon Global Technologies, Inc. | System and method for a dynamic map display with actual images |
CN108665556B (en) * | 2018-05-20 | 2024-03-01 | 福州市极化律网络科技有限公司 | Road indication display method based on mixed reality and storage medium |
US11482045B1 (en) | 2018-06-28 | 2022-10-25 | Amazon Technologies, Inc. | Associating events with actors using digital imagery and machine learning |
US11468698B1 (en) | 2018-06-28 | 2022-10-11 | Amazon Technologies, Inc. | Associating events with actors using digital imagery and machine learning |
US11468681B1 (en) | 2018-06-28 | 2022-10-11 | Amazon Technologies, Inc. | Associating events with actors using digital imagery and machine learning |
KR101957896B1 (en) * | 2018-06-29 | 2019-07-04 | 한화시스템(주) | integrated image and situation display system for remote control and method of displaying the same |
JP1632766S (en) * | 2018-09-10 | 2019-06-03 | ||
KR20200029785A (en) * | 2018-09-11 | 2020-03-19 | 삼성전자주식회사 | Localization method and apparatus of displaying virtual object in augmented reality |
KR102559203B1 (en) | 2018-10-01 | 2023-07-25 | 삼성전자주식회사 | Method and apparatus of outputting pose information |
GB2580400B (en) * | 2019-01-10 | 2021-11-03 | Jaguar Land Rover Ltd | A control system, system and method for providing assistance to an occupant of a vehicle |
CN110779541B (en) * | 2019-04-10 | 2021-11-23 | 北京嘀嘀无限科技发展有限公司 | Display method and system of steering arrow |
US20220335698A1 (en) * | 2019-12-17 | 2022-10-20 | Ashley SinHee Kim | System and method for transforming mapping information to an illustrated map |
CN113137970B (en) * | 2020-01-20 | 2022-11-15 | 北京智慧图科技有限责任公司 | Small program AR navigation system |
US11443516B1 (en) | 2020-04-06 | 2022-09-13 | Amazon Technologies, Inc. | Locally and globally locating actors by digital cameras and machine learning |
US11398094B1 (en) * | 2020-04-06 | 2022-07-26 | Amazon Technologies, Inc. | Locally and globally locating actors by digital cameras and machine learning |
KR20220022340A (en) * | 2020-08-18 | 2022-02-25 | 삼성전자주식회사 | Device and method to visualize content |
KR20220036456A (en) * | 2020-09-15 | 2022-03-23 | 현대자동차주식회사 | Apparatus for displaying information based on augmented reality |
JP6976537B1 (en) * | 2020-10-08 | 2021-12-08 | 株式会社Fronteo | Information retrieval device, information retrieval method and information retrieval program |
CN112562375A (en) * | 2020-12-11 | 2021-03-26 | 奇瑞汽车股份有限公司 | Augmented reality road name display method and system, mobile terminal and vehicle-mounted terminal |
CN113516770A (en) * | 2021-04-14 | 2021-10-19 | 孙继雄 | Method for synthesizing road design drawing and aerial photography real-scene special effect |
US20220388507A1 (en) * | 2021-06-04 | 2022-12-08 | Telenav, Inc. | Vehicle system with mechanism for determining clear path and method of operation thereof |
CN113554712A (en) * | 2021-06-29 | 2021-10-26 | 北京百度网讯科技有限公司 | Registration method and device of automatic driving vehicle, electronic equipment and vehicle |
CN115171412A (en) * | 2022-08-09 | 2022-10-11 | 阿波罗智联(北京)科技有限公司 | Method, system and device for displaying vehicle running state |
Citations (47)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH10132598A (en) | 1996-10-31 | 1998-05-22 | Sony Corp | Navigating method, navigation device and automobile |
JPH10281794A (en) | 1997-04-03 | 1998-10-23 | Toyota Motor Corp | Guidance display device for vehicle |
JPH11281387A (en) | 1998-03-31 | 1999-10-15 | Sony Corp | Navigation apparatus |
JPH11304499A (en) | 1998-04-22 | 1999-11-05 | Matsushita Electric Ind Co Ltd | Car navigation system |
US6018697A (en) * | 1995-12-26 | 2000-01-25 | Aisin Aw Co., Ltd. | Navigation system for vehicles |
US6181302B1 (en) * | 1996-04-24 | 2001-01-30 | C. Macgill Lynde | Marine navigation binoculars with virtual display superimposing real world image |
US6182010B1 (en) * | 1999-01-28 | 2001-01-30 | International Business Machines Corporation | Method and apparatus for displaying real-time visual information on an automobile pervasive computing client |
US6199014B1 (en) * | 1997-12-23 | 2001-03-06 | Walker Digital, Llc | System for providing driving directions with visual cues |
US6208933B1 (en) * | 1998-12-04 | 2001-03-27 | Northrop Grumman Corporation | Cartographic overlay on sensor video |
US6285317B1 (en) * | 1998-05-01 | 2001-09-04 | Lucent Technologies Inc. | Navigation system with three-dimensional display |
JP2001331787A (en) | 2000-05-19 | 2001-11-30 | Toyota Central Res & Dev Lab Inc | Road shape estimating device |
US6356840B2 (en) * | 1998-06-12 | 2002-03-12 | Mitsubishi Denki Kabushiki Kaisha | Navigation device with a three dimensional display |
US6360168B1 (en) * | 1999-09-14 | 2002-03-19 | Alpine Electronics, Inc. | Navigation apparatus |
JP2002236027A (en) | 2001-02-08 | 2002-08-23 | Denso Corp | Car navigation apparatus |
US6470265B1 (en) * | 1997-12-25 | 2002-10-22 | Toyota Jidosha Kabushiki Kaisha | Method and apparatus for processing digital map data |
JP2003047050A (en) | 2001-07-26 | 2003-02-14 | Komatsu Ltd | Communication apparatus of moving body |
JP2003121167A (en) | 2001-10-12 | 2003-04-23 | Denso Corp | Guidance image-generating apparatus, guidance image- displaying apparatus, navigation apparatus, and program |
US20030080978A1 (en) * | 2001-10-04 | 2003-05-01 | Nassir Navab | Augmented reality system |
US20030160867A1 (en) * | 2002-01-17 | 2003-08-28 | Yasunori Ohto | Information providing apparatus, information providing method, storage medium, and computer program |
JP2003256997A (en) | 2002-02-28 | 2003-09-12 | Honda Motor Co Ltd | Device for supporting enhancement of vehicle driving ability |
US20030210228A1 (en) * | 2000-02-25 | 2003-11-13 | Ebersole John Franklin | Augmented reality situational awareness system and method |
US20040066376A1 (en) * | 2000-07-18 | 2004-04-08 | Max Donath | Mobility assist device |
JP2004150918A (en) | 2002-10-30 | 2004-05-27 | Shinichi Yamanaka | Map displaying method |
WO2004048895A1 (en) | 2002-11-22 | 2004-06-10 | Kumamoto Technology & Industry Foundation | Moving body navigate information display method and moving body navigate information display device |
US20040179104A1 (en) * | 2003-03-10 | 2004-09-16 | Charles Benton | Augmented reality navigation system |
US6885939B2 (en) * | 2002-12-31 | 2005-04-26 | Robert Bosch Gmbh | System and method for advanced 3D visualization for mobile navigation units |
US20050116964A1 (en) * | 2003-11-19 | 2005-06-02 | Canon Kabushiki Kaisha | Image reproducing method and apparatus for displaying annotations on a real image in virtual space |
US20050278111A1 (en) * | 2004-05-20 | 2005-12-15 | Koji Ujino | Image processing system, method and apparatus for correlating position data with image data |
US20060004512A1 (en) * | 2004-06-30 | 2006-01-05 | Herbst James M | Method of operating a navigation system using images |
US20060155466A1 (en) * | 2005-01-12 | 2006-07-13 | Sanyo Electric Co., Ltd. | Mobile terminal with navigation function |
US20060155467A1 (en) * | 2002-08-07 | 2006-07-13 | Horst Hortner | Method and device for displaying navigational information for a vehicle |
US20060190812A1 (en) * | 2005-02-22 | 2006-08-24 | Geovector Corporation | Imaging systems including hyperlink associations |
US20060238383A1 (en) * | 2005-04-21 | 2006-10-26 | Microsoft Corporation | Virtual earth rooftop overlay and bounding |
US20060262140A1 (en) * | 2005-05-18 | 2006-11-23 | Kujawa Gregory A | Method and apparatus to facilitate visual augmentation of perceived reality |
US20080147325A1 (en) * | 2006-12-18 | 2008-06-19 | Maassel Paul W | Method and system for providing augmented reality |
US20090005961A1 (en) * | 2004-06-03 | 2009-01-01 | Making Virtual Solid, L.L.C. | En-Route Navigation Display Method and Apparatus Using Head-Up Display |
US20090222203A1 (en) * | 2006-03-07 | 2009-09-03 | Robrt Bosch Gmbh | Method and system for displaying navigation instructions |
US7728869B2 (en) * | 2005-06-14 | 2010-06-01 | Lg Electronics Inc. | Matching camera-photographed image with map data in portable terminal and travel route guidance method |
US7737965B2 (en) * | 2005-06-09 | 2010-06-15 | Honeywell International Inc. | Handheld synthetic vision device |
US7751970B2 (en) * | 2006-08-07 | 2010-07-06 | Pioneer Corporation | Information providing apparatus, information providing method, and computer product |
US7783422B2 (en) * | 2006-03-29 | 2010-08-24 | Denso Corporation | Navigation device and method of navigating vehicle |
US20100268451A1 (en) * | 2009-04-17 | 2010-10-21 | Lg Electronics Inc. | Method and apparatus for displaying image of mobile communication terminal |
US7852355B2 (en) * | 2003-11-10 | 2010-12-14 | Siemens Aktiengesellschaft | System and method for carrying out and visually displaying simulations in an augmented reality |
US7966128B2 (en) * | 2006-09-27 | 2011-06-21 | Denso It Laboratory, Inc. | Observed-object detection apparatus and observed-object detection method |
US20110153198A1 (en) * | 2009-12-21 | 2011-06-23 | Navisus LLC | Method for the display of navigation instructions using an augmented-reality concept |
US7990394B2 (en) * | 2007-05-25 | 2011-08-02 | Google Inc. | Viewing and navigating within panoramic images, and applications thereof |
US20110228078A1 (en) * | 2010-03-22 | 2011-09-22 | Institute For Information Industry | Real-time augmented reality device, real-time augmented reality method and computer storage medium thereof |
Family Cites Families (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH07114692A (en) * | 1993-10-15 | 1995-05-02 | Fujitsu Ten Ltd | Route guiding method |
JP2001034160A (en) * | 1999-05-14 | 2001-02-09 | Denso Corp | Map display device |
JP3717045B2 (en) * | 2000-01-19 | 2005-11-16 | 松下電器産業株式会社 | Navigation device |
JP4014447B2 (en) * | 2002-05-23 | 2007-11-28 | 富士通テン株式会社 | Navigation device |
-
2005
- 2005-09-27 US US11/663,909 patent/US8195386B2/en not_active Expired - Fee Related
- 2005-09-27 WO PCT/JP2005/017715 patent/WO2006035755A1/en active Application Filing
- 2005-09-27 JP JP2006537744A patent/JP4696248B2/en active Active
Patent Citations (48)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6018697A (en) * | 1995-12-26 | 2000-01-25 | Aisin Aw Co., Ltd. | Navigation system for vehicles |
US6181302B1 (en) * | 1996-04-24 | 2001-01-30 | C. Macgill Lynde | Marine navigation binoculars with virtual display superimposing real world image |
JPH10132598A (en) | 1996-10-31 | 1998-05-22 | Sony Corp | Navigating method, navigation device and automobile |
JPH10281794A (en) | 1997-04-03 | 1998-10-23 | Toyota Motor Corp | Guidance display device for vehicle |
US6199014B1 (en) * | 1997-12-23 | 2001-03-06 | Walker Digital, Llc | System for providing driving directions with visual cues |
US6470265B1 (en) * | 1997-12-25 | 2002-10-22 | Toyota Jidosha Kabushiki Kaisha | Method and apparatus for processing digital map data |
JPH11281387A (en) | 1998-03-31 | 1999-10-15 | Sony Corp | Navigation apparatus |
JPH11304499A (en) | 1998-04-22 | 1999-11-05 | Matsushita Electric Ind Co Ltd | Car navigation system |
US6285317B1 (en) * | 1998-05-01 | 2001-09-04 | Lucent Technologies Inc. | Navigation system with three-dimensional display |
US6356840B2 (en) * | 1998-06-12 | 2002-03-12 | Mitsubishi Denki Kabushiki Kaisha | Navigation device with a three dimensional display |
US6208933B1 (en) * | 1998-12-04 | 2001-03-27 | Northrop Grumman Corporation | Cartographic overlay on sensor video |
US6182010B1 (en) * | 1999-01-28 | 2001-01-30 | International Business Machines Corporation | Method and apparatus for displaying real-time visual information on an automobile pervasive computing client |
US6360168B1 (en) * | 1999-09-14 | 2002-03-19 | Alpine Electronics, Inc. | Navigation apparatus |
US20030210228A1 (en) * | 2000-02-25 | 2003-11-13 | Ebersole John Franklin | Augmented reality situational awareness system and method |
JP2001331787A (en) | 2000-05-19 | 2001-11-30 | Toyota Central Res & Dev Lab Inc | Road shape estimating device |
US6977630B1 (en) * | 2000-07-18 | 2005-12-20 | University Of Minnesota | Mobility assist device |
US20040066376A1 (en) * | 2000-07-18 | 2004-04-08 | Max Donath | Mobility assist device |
JP2002236027A (en) | 2001-02-08 | 2002-08-23 | Denso Corp | Car navigation apparatus |
JP2003047050A (en) | 2001-07-26 | 2003-02-14 | Komatsu Ltd | Communication apparatus of moving body |
US20030080978A1 (en) * | 2001-10-04 | 2003-05-01 | Nassir Navab | Augmented reality system |
JP2003121167A (en) | 2001-10-12 | 2003-04-23 | Denso Corp | Guidance image-generating apparatus, guidance image- displaying apparatus, navigation apparatus, and program |
US20030160867A1 (en) * | 2002-01-17 | 2003-08-28 | Yasunori Ohto | Information providing apparatus, information providing method, storage medium, and computer program |
JP2003256997A (en) | 2002-02-28 | 2003-09-12 | Honda Motor Co Ltd | Device for supporting enhancement of vehicle driving ability |
US20060155467A1 (en) * | 2002-08-07 | 2006-07-13 | Horst Hortner | Method and device for displaying navigational information for a vehicle |
JP2004150918A (en) | 2002-10-30 | 2004-05-27 | Shinichi Yamanaka | Map displaying method |
WO2004048895A1 (en) | 2002-11-22 | 2004-06-10 | Kumamoto Technology & Industry Foundation | Moving body navigate information display method and moving body navigate information display device |
US6885939B2 (en) * | 2002-12-31 | 2005-04-26 | Robert Bosch Gmbh | System and method for advanced 3D visualization for mobile navigation units |
US20040179104A1 (en) * | 2003-03-10 | 2004-09-16 | Charles Benton | Augmented reality navigation system |
US7852355B2 (en) * | 2003-11-10 | 2010-12-14 | Siemens Aktiengesellschaft | System and method for carrying out and visually displaying simulations in an augmented reality |
US20050116964A1 (en) * | 2003-11-19 | 2005-06-02 | Canon Kabushiki Kaisha | Image reproducing method and apparatus for displaying annotations on a real image in virtual space |
US20050278111A1 (en) * | 2004-05-20 | 2005-12-15 | Koji Ujino | Image processing system, method and apparatus for correlating position data with image data |
US20090005961A1 (en) * | 2004-06-03 | 2009-01-01 | Making Virtual Solid, L.L.C. | En-Route Navigation Display Method and Apparatus Using Head-Up Display |
US20060004512A1 (en) * | 2004-06-30 | 2006-01-05 | Herbst James M | Method of operating a navigation system using images |
US20060155466A1 (en) * | 2005-01-12 | 2006-07-13 | Sanyo Electric Co., Ltd. | Mobile terminal with navigation function |
US20060190812A1 (en) * | 2005-02-22 | 2006-08-24 | Geovector Corporation | Imaging systems including hyperlink associations |
US20060238383A1 (en) * | 2005-04-21 | 2006-10-26 | Microsoft Corporation | Virtual earth rooftop overlay and bounding |
US20060262140A1 (en) * | 2005-05-18 | 2006-11-23 | Kujawa Gregory A | Method and apparatus to facilitate visual augmentation of perceived reality |
US7737965B2 (en) * | 2005-06-09 | 2010-06-15 | Honeywell International Inc. | Handheld synthetic vision device |
US7728869B2 (en) * | 2005-06-14 | 2010-06-01 | Lg Electronics Inc. | Matching camera-photographed image with map data in portable terminal and travel route guidance method |
US20090222203A1 (en) * | 2006-03-07 | 2009-09-03 | Robrt Bosch Gmbh | Method and system for displaying navigation instructions |
US7783422B2 (en) * | 2006-03-29 | 2010-08-24 | Denso Corporation | Navigation device and method of navigating vehicle |
US7751970B2 (en) * | 2006-08-07 | 2010-07-06 | Pioneer Corporation | Information providing apparatus, information providing method, and computer product |
US7966128B2 (en) * | 2006-09-27 | 2011-06-21 | Denso It Laboratory, Inc. | Observed-object detection apparatus and observed-object detection method |
US20080147325A1 (en) * | 2006-12-18 | 2008-06-19 | Maassel Paul W | Method and system for providing augmented reality |
US7990394B2 (en) * | 2007-05-25 | 2011-08-02 | Google Inc. | Viewing and navigating within panoramic images, and applications thereof |
US20100268451A1 (en) * | 2009-04-17 | 2010-10-21 | Lg Electronics Inc. | Method and apparatus for displaying image of mobile communication terminal |
US20110153198A1 (en) * | 2009-12-21 | 2011-06-23 | Navisus LLC | Method for the display of navigation instructions using an augmented-reality concept |
US20110228078A1 (en) * | 2010-03-22 | 2011-09-22 | Institute For Information Industry | Real-time augmented reality device, real-time augmented reality method and computer storage medium thereof |
Cited By (72)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20150019128A1 (en) * | 2003-06-19 | 2015-01-15 | Here Global B.V. | Method and System for Representing Traffic Signals in a Road Network Database |
US10309796B2 (en) | 2003-06-19 | 2019-06-04 | Here Global B.V. | Method of representing road lanes |
US9494434B2 (en) | 2003-06-19 | 2016-11-15 | Here Global B.V. | Method of representing road lanes |
US9341485B1 (en) | 2003-06-19 | 2016-05-17 | Here Global B.V. | Method and apparatus for representing road intersections |
US9285237B2 (en) * | 2003-06-19 | 2016-03-15 | Here Global B.V. | Method and system for representing traffic signals in a road network database |
US9384277B2 (en) | 2004-08-31 | 2016-07-05 | Visual Real Estate, Inc. | Three dimensional image data models |
US8890866B2 (en) | 2004-08-31 | 2014-11-18 | Visual Real Estate, Inc. | Method and apparatus of providing street view data of a comparable real estate property |
US9311396B2 (en) | 2004-08-31 | 2016-04-12 | Visual Real Estate, Inc. | Method of providing street view data of a real estate property |
USRE45264E1 (en) | 2004-08-31 | 2014-12-02 | Visual Real Estate, Inc. | Methods and apparatus for generating three-dimensional image data models |
US9311397B2 (en) | 2004-08-31 | 2016-04-12 | Visual Real Estates, Inc. | Method and apparatus of providing street view data of a real estate property |
US8902226B2 (en) | 2004-08-31 | 2014-12-02 | Visual Real Estate, Inc. | Method for using drive-by image data to generate a valuation report of a selected real estate property |
US9098870B2 (en) | 2007-02-06 | 2015-08-04 | Visual Real Estate, Inc. | Internet-accessible real estate marketing street view system and method |
US8554015B2 (en) * | 2007-02-06 | 2013-10-08 | William D. Meadow | Methods and apparatus for generating a continuum of image data |
US20120307002A1 (en) * | 2007-02-06 | 2012-12-06 | Meadow William D | Methods and apparatus for generating a continuum of image data |
US20100030462A1 (en) * | 2007-02-15 | 2010-02-04 | Pioneer Corporation | Display control apparatus, display control method, display control program, and recording medium |
US20110054783A1 (en) * | 2008-01-28 | 2011-03-03 | Geo Technical Laboratory Co., Ltd. | Data structure of route guidance database |
US8600654B2 (en) * | 2008-01-28 | 2013-12-03 | Geo Technical Laboratory Co., Ltd. | Data structure of route guidance database |
US9852357B2 (en) | 2008-04-24 | 2017-12-26 | GM Global Technology Operations LLC | Clear path detection using an example-based approach |
US9652980B2 (en) | 2008-04-24 | 2017-05-16 | GM Global Technology Operations LLC | Enhanced clear path detection in the presence of traffic infrastructure indicator |
US20100280747A1 (en) * | 2008-05-02 | 2010-11-04 | Olaf Achthoven | Navigation device and method for displaying map information |
US8775071B2 (en) * | 2008-05-02 | 2014-07-08 | Tomtom International B.V. | Navigation device and method for displaying map information |
US9242378B2 (en) * | 2009-06-01 | 2016-01-26 | Hitachi, Ltd. | System and method for determing necessity of map data recreation in robot operation |
US20120083923A1 (en) * | 2009-06-01 | 2012-04-05 | Kosei Matsumoto | Robot control system, robot control terminal, and robot control method |
US20120194554A1 (en) * | 2011-01-28 | 2012-08-02 | Akihiko Kaino | Information processing device, alarm method, and program |
US20140107911A1 (en) * | 2011-05-16 | 2014-04-17 | Samsung Electronics Co., Ltd. | User interface method for terminal for vehicle and apparatus thereof |
US8626427B2 (en) * | 2011-05-16 | 2014-01-07 | Samsung Electronics Co., Ltd. | User interface method for terminal for vehicle and apparatus thereof |
US20120296561A1 (en) * | 2011-05-16 | 2012-11-22 | Samsung Electronics Co., Ltd. | User interface method for terminal for vehicle and apparatus thereof |
US9318018B2 (en) * | 2011-05-16 | 2016-04-19 | Samsung Electronics Co., Ltd. | User interface method for terminal for vehicle and apparatus thereof |
US9723243B2 (en) | 2011-05-16 | 2017-08-01 | Samsung Electronics Co., Ltd. | User interface method for terminal for vehicle and apparatus thereof |
US9080892B2 (en) * | 2011-11-15 | 2015-07-14 | Lg Electronics Inc. | Mobile terminal and method of controlling the same |
US20130124084A1 (en) * | 2011-11-15 | 2013-05-16 | Jaehong SEO | Mobile terminal and method of controlling the same |
US20130158849A1 (en) * | 2011-12-15 | 2013-06-20 | Masao MAURA | Evaluation indication system, evaluation indication method and computer-readable storage medium |
US8862374B2 (en) * | 2011-12-15 | 2014-10-14 | Aisin Aw Co., Ltd. | Evaluation indication system, evaluation indication method and computer-readable storage medium |
US9037411B2 (en) * | 2012-05-11 | 2015-05-19 | Honeywell International Inc. | Systems and methods for landmark selection for navigation |
US20130304383A1 (en) * | 2012-05-11 | 2013-11-14 | Honeywell International Inc. | Systems and methods for landmark selection for navigation |
US9435648B2 (en) | 2012-12-17 | 2016-09-06 | Industrial Technology Research Institute | Map matching device, system and method |
US9041714B2 (en) * | 2013-01-31 | 2015-05-26 | Samsung Electronics Co., Ltd. | Apparatus and method for compass intelligent lighting for user interfaces |
US20140210822A1 (en) * | 2013-01-31 | 2014-07-31 | Samsung Electronics Co. Ltd. | Apparatus and method for compass intelligent lighting for user interfaces |
US9671243B2 (en) * | 2013-06-13 | 2017-06-06 | Mobileye Vision Technologies Ltd. | Vision augmented navigation |
US10533869B2 (en) * | 2013-06-13 | 2020-01-14 | Mobileye Vision Technologies Ltd. | Vision augmented navigation |
US11604076B2 (en) * | 2013-06-13 | 2023-03-14 | Mobileye Vision Technologies Ltd. | Vision augmented navigation |
US20200173803A1 (en) * | 2013-06-13 | 2020-06-04 | Mobileye Vision Technologies Ltd. | Vision augmented navigation |
US20140372020A1 (en) * | 2013-06-13 | 2014-12-18 | Gideon Stein | Vision augmented navigation |
US20150073663A1 (en) * | 2013-09-12 | 2015-03-12 | Volvo Car Corporation | Manoeuver generation for automated driving |
US9469296B2 (en) * | 2013-09-12 | 2016-10-18 | Volvo Car Corporation | Manoeuver generation for automated driving |
KR20150062418A (en) * | 2013-11-29 | 2015-06-08 | (주) 유라시아디지털문화유산연구소 | System for displaying virtual image matched with real street figure |
US9395192B1 (en) * | 2013-12-20 | 2016-07-19 | Google Inc. | Methods and systems for road and lane boundary tracing |
US20150226568A1 (en) * | 2014-02-11 | 2015-08-13 | Hyundai Motor Company | Apparatus and method of providing road guidance based on augmented reality head-up display |
US9151634B2 (en) * | 2014-02-11 | 2015-10-06 | Hyundai Motor Company | Apparatus and method of providing road guidance based on augmented reality head-up display |
US10663315B2 (en) * | 2015-02-09 | 2020-05-26 | Denso Corporation | Vehicle display control device and vehicle display control method |
US20190384295A1 (en) * | 2015-02-10 | 2019-12-19 | Mobileye Vision Technologies Ltd. | Systems and methods for identifying landmarks |
US11774251B2 (en) * | 2015-02-10 | 2023-10-03 | Mobileye Vision Technologies Ltd. | Systems and methods for identifying landmarks |
US11599113B2 (en) * | 2015-02-10 | 2023-03-07 | Mobileye Vision Technologies Ltd. | Crowd sourcing data for autonomous vehicle navigation |
US20190384294A1 (en) * | 2015-02-10 | 2019-12-19 | Mobileye Vision Technologies Ltd. | Crowd sourcing data for autonomous vehicle navigation |
US9304003B1 (en) | 2015-03-18 | 2016-04-05 | Microsoft Technology Licensing, Llc | Augmented reality navigation |
US9909894B2 (en) | 2016-01-07 | 2018-03-06 | Here Global B.V. | Componentized junction models |
US10234294B2 (en) | 2016-04-01 | 2019-03-19 | Here Global B.V. | Road geometry matching with componentized junction models |
US10921134B2 (en) | 2016-04-01 | 2021-02-16 | Here Global B.V. | Road geometry matching with componentized junction models |
US20190138261A1 (en) * | 2016-06-22 | 2019-05-09 | Sony Corporation | Information processing device, information processing method, and program |
US10922043B2 (en) * | 2016-06-22 | 2021-02-16 | Sony Corporation | Information processing device and information processing method for acquiring information associated with a target |
US20180341822A1 (en) * | 2017-05-26 | 2018-11-29 | Dura Operating, Llc | Method and system for classifying objects in a perception scene graph by using a scene-detection-schema |
US10495476B1 (en) | 2018-09-27 | 2019-12-03 | Phiar Technologies, Inc. | Augmented reality navigation systems and methods |
US11313695B2 (en) | 2018-09-27 | 2022-04-26 | Phiar Technologies, Inc. | Augmented reality navigational indicator |
US11448518B2 (en) | 2018-09-27 | 2022-09-20 | Phiar Technologies, Inc. | Augmented reality navigational overlay |
US11545036B2 (en) | 2018-09-27 | 2023-01-03 | Google Llc | Real-time driving behavior and safety monitoring |
US10573183B1 (en) | 2018-09-27 | 2020-02-25 | Phiar Technologies, Inc. | Mobile real-time driving safety systems and methods |
US11156472B2 (en) | 2018-10-26 | 2021-10-26 | Phiar Technologies, Inc. | User interface for augmented reality navigation |
US11085787B2 (en) * | 2018-10-26 | 2021-08-10 | Phiar Technologies, Inc. | Augmented reality interface for navigation assistance |
US10488215B1 (en) | 2018-10-26 | 2019-11-26 | Phiar Technologies, Inc. | Augmented reality interface for navigation assistance |
US20220111840A1 (en) * | 2019-01-10 | 2022-04-14 | Jaguar Land Rover Limited | A control system, system and method for providing assistance to an occupant of a vehicle |
US11919513B2 (en) * | 2019-01-10 | 2024-03-05 | Jaguar Land Rover Limited | Control system, system and method for providing assistance to an occupant of a vehicle |
US11499834B2 (en) * | 2019-03-07 | 2022-11-15 | Mobileye Vision Technologies Ltd. | Aligning road information for navigation |
Also Published As
Publication number | Publication date |
---|---|
WO2006035755A1 (en) | 2006-04-06 |
JPWO2006035755A1 (en) | 2008-05-15 |
US20080195315A1 (en) | 2008-08-14 |
JP4696248B2 (en) | 2011-06-08 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8195386B2 (en) | Movable-body navigation information display method and movable-body navigation information display unit | |
US8352180B2 (en) | Device with camera-info | |
US7733244B2 (en) | Navigation system | |
US9171214B2 (en) | Projecting location based elements over a heads up display | |
US8423292B2 (en) | Navigation device with camera-info | |
US10891859B1 (en) | Displaying sensor data and supplemental data as a mask for autonomous vehicles | |
US20110103651A1 (en) | Computer arrangement and method for displaying navigation data in 3d | |
CN106564432A (en) | Apparatus and method for controlling viewing angle for vehicle, and vehicle including the apparatus | |
US20100245561A1 (en) | Navigation device | |
JP2009020089A (en) | System, method, and program for navigation | |
KR20080020956A (en) | Method of operating a night-view system in a vehicle and corresponding night-view system | |
KR20110114114A (en) | Real 3d navigation implementing method | |
JP4787196B2 (en) | Car navigation system | |
JPWO2004048895A1 (en) | MOBILE NAVIGATION INFORMATION DISPLAY METHOD AND MOBILE NAVIGATION INFORMATION DISPLAY DEVICE | |
EP3859390A1 (en) | Method and system for rendering a representation of an evinronment of a vehicle | |
KR20080019690A (en) | Navigation device with camera-info | |
JP2003302235A (en) | Navigation display device | |
JP2021071885A (en) | Area cut-out method and area cut-out program |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: NATIONAL UNIVERSITY CORPORATION KUMAMOTO UNIVERSIT Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HU, ZHENCHENG;UCHIMURA, KEIICHI;REEL/FRAME:019124/0258 Effective date: 20070326 |
|
REMI | Maintenance fee reminder mailed | ||
LAPS | Lapse for failure to pay maintenance fees | ||
STCH | Information on status: patent discontinuation |
Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362 |
|
FP | Expired due to failure to pay maintenance fee |
Effective date: 20160605 |