US8195386B2 - Movable-body navigation information display method and movable-body navigation information display unit - Google Patents

Movable-body navigation information display method and movable-body navigation information display unit Download PDF

Info

Publication number
US8195386B2
US8195386B2 US11/663,909 US66390905A US8195386B2 US 8195386 B2 US8195386 B2 US 8195386B2 US 66390905 A US66390905 A US 66390905A US 8195386 B2 US8195386 B2 US 8195386B2
Authority
US
United States
Prior art keywords
road
data
navigation information
movable body
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related, expires
Application number
US11/663,909
Other versions
US20080195315A1 (en
Inventor
Zhencheng Hu
Keiichi Uchimura
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Kumamoto University NUC
Original Assignee
Kumamoto University NUC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Kumamoto University NUC filed Critical Kumamoto University NUC
Assigned to NATIONAL UNIVERSITY CORPORATION KUMAMOTO UNIVERSITY reassignment NATIONAL UNIVERSITY CORPORATION KUMAMOTO UNIVERSITY ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: HU, ZHENCHENG, UCHIMURA, KEIICHI
Publication of US20080195315A1 publication Critical patent/US20080195315A1/en
Application granted granted Critical
Publication of US8195386B2 publication Critical patent/US8195386B2/en
Expired - Fee Related legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G08SIGNALLING
    • G08GTRAFFIC CONTROL SYSTEMS
    • G08G1/00Traffic control systems for road vehicles
    • G08G1/09Arrangements for giving variable traffic instructions
    • G08G1/0962Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages
    • G08G1/0968Systems involving transmission of navigation instructions to the vehicle
    • G08G1/0969Systems involving transmission of navigation instructions to the vehicle having a display in the form of a map
    • GPHYSICS
    • G01MEASURING; TESTING
    • G01CMEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/26Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
    • G01C21/34Route searching; Route guidance
    • G01C21/36Input/output arrangements for on-board computers
    • G01C21/3626Details of the output of route guidance instructions
    • G01C21/3647Guidance involving output of stored or live camera images or video streams
    • GPHYSICS
    • G01MEASURING; TESTING
    • G01CMEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/26Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
    • G01C21/34Route searching; Route guidance
    • G01C21/36Input/output arrangements for on-board computers
    • G01C21/3626Details of the output of route guidance instructions
    • G01C21/365Guidance using head up displays or projectors, e.g. virtual vehicles or arrows projected on the windscreen or on the road itself
    • GPHYSICS
    • G08SIGNALLING
    • G08GTRAFFIC CONTROL SYSTEMS
    • G08G1/00Traffic control systems for road vehicles
    • G08G1/09Arrangements for giving variable traffic instructions
    • G08G1/0962Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages
    • G08G1/0968Systems involving transmission of navigation instructions to the vehicle
    • G08G1/096855Systems involving transmission of navigation instructions to the vehicle where the output is provided in a suitable form to the driver
    • G08G1/096861Systems involving transmission of navigation instructions to the vehicle where the output is provided in a suitable form to the driver where the immediate route instructions are output to the driver, e.g. arrow signs for next turn

Definitions

  • the present invention relates to a movable-body navigation information display method and movable-body navigation information display unit for, for example, collecting, processing, and displaying information on a current position of a movable body and information for navigation related thereto, such as a so-called car navigation system and a display method of car navigation information thereby.
  • movable-body navigation system such as a so-called car navigation system has been developed.
  • a current position of a movable body such as a motor vehicle is detected, the detected current position is displayed on a display screen together with a road map of surroundings, various guide information such as a destination and a route is displayed, and voice output is made.
  • an autonomous-navigation controlling section performs tracing on a map previously stored as data in a storing means such as a CD-ROM (Compact Disc Read Only Memory) based on the speed pulses from a speed sensor and a direction obtained by earth magnetism from a geomagnetic sensor.
  • a GPS signal transmitted from a GPS satellite is used.
  • an image showing the current position and the road map in a given range in the vicinity thereof can be displayed as a plane map-like two-dimensional image on a screen of an image display unit such as a liquid crystal display unit.
  • navigation information such as a vehicle position, a traveling direction, speed, map information, a course guide, the shortest route, and road information by being superimposed on the two-dimensional image map, or by being inserted in an appropriate position in the two-dimensional image map.
  • an image of information for example, a traveling position and a traveling direction indicated by an icon of an arrow or the like
  • a recommended route (optimal route) from the current position to the destination is displayed by being superimposed on a two-dimensional image road planar map or on a road map that shows the surrounding landscape in a state of a three-dimensional CG.
  • traffic jam information or the like obtained by a VICS sensor or the like is displayed with a text label on the screen.
  • Patent document 1 Patent document 2 and the like, a navigation system using a real picture in addition to the map information is proposed.
  • Patent document 1 a current position and a traveling direction of a movable body are detected, and navigation information to be displayed according to the detected current position and the detected traveling direction is selected.
  • An image of the navigation information read according to the current position of the vehicle is superimposed on a real picture captured by an image pickup device from a camera angle approximately similar to a landscape ahead of the vehicle in the traveling direction seen through the windshield of the vehicle, and the superimposed result is displayed on a screen or the like of a liquid crystal display panel.
  • a relation between the picture of the real road, the surrounding landscape thereof and the navigation information can be displayed easily recognizable visually.
  • the navigation information is mainly picture information displayed on the picture display section. Therefore, there is a possibility that a driver gets distracted easily while driving.
  • Patent document 2 the following technique is proposed. That is, an image pickup device such as a CCD camera that captures a landscape ahead of a body for motor vehicle is provided, for example, in a position near the ceiling of the windshield of the vehicle or in the vicinity of the dashboard. A picture (image) of the landscape including a road ahead of the body for motor vehicle captured by the image pickup device is displayed by being inserted in a given position of the screen displaying map information as a sub-screen. According to such a technique, even when the driver of the vehicle views navigation information such as a map displayed on the screen of the image display unit while driving, the driver views the real picture of the landscape ahead of the vehicle displayed as the sub-image in the given position on the screen. It is stated in Patent document 2 that the driver can thereby comprehend a state in front of the vehicle without returning her/his attention to the landscape ahead of the vehicle.
  • an image pickup device such as a CCD camera that captures a landscape ahead of a body for motor vehicle is provided, for example, in a position near the ceiling of the windshield
  • a three-dimensional road shape is estimated by using image road information and map road information.
  • the image road shape extracted from image data of the landscape ahead of the vehicle and map data around the vehicle are projected in one two-dimensional or three-dimensional logical space. Based on an overlapping state in the logical space of both of the projected road shapes, estimation is made for the road shape, a posture of the vehicle to the road surface, an absolute position of the vehicle and the like.
  • a sufficiently accurate road shape is estimated, and accurate determination of the leading vehicle and automatic steering are realized.
  • the image information is only used as a background picture.
  • the image is firstly displayed in the central portion of the screen, and then is displayed to move linearly from the original position to the backward landscape (in general, from the upper side to the lower side of the display screen) in the central section of the screen as the vehicle travels.
  • the posture parameter of the camera is previously fixed as the angle of the camera optical axis to the ground surface (or horizontal direction). Therefore, for example, when the posture of the camera is changed due to vibration of the running vehicle, rolling and pitching of motor vehicle by steering, or tilt of the vehicle on the uphill slope and the downhill slope, a navigation information image of route such as right turning position and indicated by an arrow or the like of a traveling direction is largely deviated to the picture obtained by capturing the real landscape. Consequently, the arrow indicates a wrong direction substantially, and the right turning position is displayed unclearly.
  • an arrow 901 as a navigation information image appears to indicate the left turning position on the road in the landscape ahead of the vehicle clearly.
  • a general passenger car or even in the case of a vehicle such as a bus and a track with the driver seat located in a high position, there is no possibility that the road landscape actually seen through the windshield from the driver seat of the vehicle becomes a bird's-eye view down from a high position as shown in FIG. 13 .
  • the road picture often results in a landscape (and real picture) in which anteroposterior density is particularly high and a road that crosses the road in the traveling direction is not easily seen since such a road is hidden by the buildings and the like along the road in the traveling direction.
  • FIGS. 15A and 15B show comparative schematic views.
  • FIG. 15A shows the degree of change ( ⁇ LA) of a projection image onto a road surface with respect to the unit angle change (posture to the road surface) ⁇ e in the visual line at about 1 m high from a driver seat of a general vehicle.
  • FIG. 15B shows the degree of change ( ⁇ LB) of a projection image onto a road surface with respect to the unit angle change (posture to the road surface) ⁇ e in the visual line at about 10 m high, which is higher than the height in FIG. 15A .
  • ⁇ LA degree of change
  • ⁇ LB degree of change
  • Patent document 1 in which the navigation information such as the arrow is located in a fixed position such as on the central line on the screen of the real picture of the landscape, even if the posture of the vehicle is changed slightly, the deviation between the real picture of the landscape and the image of the navigation information becomes large.
  • the landscape located near the vehicle along the road such as buildings and trees blocks the landscape located farther than the foregoing landscape. Therefore, to address such a problem, it is necessary to display the image of the navigation information on a more accurate position.
  • Patent document 2 there is the following problem in addition to the problems similar to those of the Patent document 1 described above. That is, in Patent document 2, the real picture of the road landscape ahead of the vehicle is only displayed in the sub-screen. Therefore, matching of the road map information and the real picture should be made by the driver in his/her mind. Therefore, for example, when the driver drives a road with many crossings and forks in an unfamiliar (or firstly driving) place, it is difficult to intuitively comprehend the navigation information. In the result, there is a possibility that the user takes a wrong route, falsely recognizes the content indicated by the navigation information, and is not able to understand the content.
  • Patent document 3 has the following problems. That is, in such a technique, there is no specific road shape model. Therefore, in the case of a multilane road, there is a possibility that large deviation is caused between the central line of the driving lane of the vehicle extracted from the image data and the road centerline estimated from the road map data. Further, it is not possible to estimate information of the driving lane of the vehicle. Thus, accurate navigation information is not able to be provided in changing the lane and in turning left or right.
  • the estimation result of the road shape is not feed backed, and the section with large luminance change is simply extracted as a white line by a differential filter for every frame.
  • the estimation result obtained by the foregoing method is largely subject to influences of various environmental factors such as weather change and shadow or dirt on the road surface. Therefore, the road shape model expressed by the data obtained by extracting the road characteristics may become an incorrect model that is significantly deviated from the road shape of the real object.
  • the image of the navigation information is composited and displayed in the real picture in front of the vehicle, there is a possibility as follows.
  • the image of the navigation information is superimposed on a caution-needed picture such as a pedestrian and other obstruction in the real picture. Thereby, the visibility of the caution-needed picture may be inhibited.
  • navigation information display method and a movable-body navigation information display unit In the navigation information display method and the movable-body navigation information display unit, navigation information such as a course guide, a vehicle position, and map information is accurately projected and displayed in an appropriate position in a real picture or in a real landscape of the road ahead of the movable body. Thereby, a driver can intuitively and accurately recognize a relation between the navigation information and the real picture or the real landscape. In addition, visibility of a caution-needed picture such as a pedestrian in the real picture and a real picture of a road construction site is not blocked by the image of the navigation information.
  • a caution-needed picture such as a pedestrian in the real picture and a real picture of a road construction site is not blocked by the image of the navigation information.
  • a first movable-body navigation information display method includes the steps of detecting a current position of a movable body, and capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data; creating a road shape model of a road assumed to be captured from the current position, based on the current position of the movable body and the road map data; extracting road shape data as image data of the road included in the landscape from the real picture; matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, and determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data; and displaying an image in which the read navigation
  • a first movable-body navigation information display unit includes: a current position detecting means for detecting a current position of a movable body; a capturing means for capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; a data processing means for reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data, creating a road shape model of a road assumed to be captured from the current position based on the current position of the movable body and the road map data, extracting road shape data as image data of the road included in the landscape from the real picture, matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based
  • the road shape model of the road assumed to be captured from the current position is created based on the current position of the movable body such as a motor vehicle and the road map data including the current position.
  • the road shape data as the image data of the road included in the landscape ahead of the vehicle or in the traveling direction is extracted from the real picture.
  • the data of the road shape model is matched with the road shape data to estimate the posture data of the in-vehicle camera or the movable body to the road in the landscape as the object.
  • the appropriate display position in the captured real picture of the navigation information read according to the current position of the movable body is determined. In the result, the image in which the read navigation information is composited with the display dimensions and the tone set by the user is displayed in the determined appropriate position of the captured real picture.
  • the appropriate composite position of the navigation information in the real picture is determined by matching the data of the road shape model with the road shape data.
  • display is enabled so that the navigation information such as a course guide, a position of a vehicle, and map information is projected accurately in an appropriate position in a real picture of a road ahead of the movable body or a real landscape.
  • the display dimensions and the tone of the image (generically referred to as image, including character information and the like) of the navigation information composited in the real picture can be set as appropriate according to the user's option. Therefore, deterioration of visibility of the entire image in which the image of the navigation information is composited in the real picture is avoided.
  • the user her/himself can customize the image of the navigation information to obtain the image with higher visibility for the user.
  • a second movable-body navigation information display method includes the steps of: detecting a current position of a movable body, and capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data; creating a road shape model of a road assumed to be captured from the current position, based on the current position of the movable body and the road map data; extracting road shape data as image data of the road included in the landscape from the real picture; matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, and determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data; and previously defining three classifications of warning
  • a second movable-body navigation information display unit includes: a current position detecting means for detecting a current position of a movable body; a capturing means for capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; a data processing means for reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data, creating a road shape model of a road assumed to be captured from the current position based on the current position of the movable body and the road map data, extracting road shape data as image data of the road included in the landscape from the real picture, matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based
  • the appropriate composite position of the navigation information in the real picture is determined by matching the data of the road shape model with the road shape data.
  • display is enabled so that the navigation information such as a course guide, a position of a vehicle, and map information is projected accurately in an appropriate position in a real picture of a road ahead of the movable body or a real landscape.
  • a driver can intuitively and accurately recognize a relation between the navigation information and the real picture or the real landscape.
  • the image of the read navigation information is composited and displayed according to the order corresponding to the classification based on the priority described above. Therefore, many navigation information images are inhibited from being displayed in one real picture. Consequently, deterioration of visibility of the entire image in which the image of the navigation information is composited in the real picture is avoided.
  • the navigation information other than the foregoing warning information may be displayed on and after when the current position of the movable body to a warning target in the real picture enters within a given distance. Thereby, the data processing amount for creating the image of the navigation information is saved. In addition, a navigation information image necessary for the user can be displayed timely when such navigation information becomes really necessary for the user.
  • a third movable-body navigation information display method includes the steps of: detecting a current position of a movable body, and capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data; creating a road shape model of a road assumed to be captured from the current position, based on the current position of the movable body and the road map data; extracting road shape data as image data of the road included in the landscape from the real picture; matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, and determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data; and displaying an image in which the read
  • a third movable-body navigation information display unit includes: a current position detecting means for detecting a current position of a movable body; a capturing means for capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; a data processing means for reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data, creating a road shape model of a road assumed to be captured from the current position based on the current position of the movable body and the road map data, extracting road shape data as image data of the road included in the landscape from the real picture, matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based
  • the appropriate composite position of the navigation information in the real picture is determined by matching the data of the road shape model with the road shape data.
  • display is enabled so that the navigation information such as a course guide, a position of a vehicle, and map information is projected accurately in an appropriate position in a real picture of a road ahead of the movable body or a real landscape.
  • a driver can intuitively and accurately recognize a relation between the navigation information and the real picture or the real landscape.
  • the navigation information other than the graphic display for guiding the traveling direction is not displayed in the region under the virtual vanishing point or the virtual horizon line in the real picture but in the region above the virtual vanishing point or the virtual horizon line. Consequently, visibility of the real picture of a road and a surface ground is not inhibited by display of the navigation information.
  • a fourth movable-body navigation information display method includes the steps of detecting a current position of a movable body, and capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data; creating a road shape model of a road assumed to be captured from the current position, based on the current position of the movable body and the road map data; extracting road shape data as image data of the road included in the landscape from the real picture; matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, and determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data; and compositing and displaying the read
  • a fourth movable-body navigation information display unit includes: a current position detecting means for detecting a current position of a movable body; a capturing means for capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; a data processing means for reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data, creating a road shape model of a road assumed to be captured from the current position based on the current position of the movable body and the road map data, extracting road shape data as image data of the road included in the landscape from the real picture, matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based
  • the appropriate composite position of the navigation information in the real picture is determined by matching the data of the road shape model with the road shape data.
  • display is enabled so that the navigation information such as a course guide, a position of a vehicle, and map information is projected accurately in an appropriate position in a real picture of a road ahead of the movable body or a real landscape.
  • a driver can intuitively and accurately recognize a relation between the navigation information and the real picture or the real landscape.
  • the image of the navigation information is composited and displayed in the real picture as the rotatable three-dimensional icon that rotates centering on the vertical axis or the horizontal axis. Consequently, visibility of the real picture of a road and a surface ground is not inhibited by display of the navigation information.
  • the foregoing navigation information may be, for example, at least one of a course guide of a route to reach a destination of the movable body, a position of a vehicle, a driving lane of the vehicle, and a building as an eyemark for a driver of the movable body to check the course guide or the position of the vehicle.
  • the navigation information is information of a character, a symbol, or a number
  • the information may be iconified, and the icon image may be composited and displayed in the captured real picture.
  • the navigation information may be expressed as a virtual object in the three-dimensional augmented reality space, and a corresponding position in the road shape data converted into the two-dimensional characteristics space may be assigned to the expressed navigation information based on already obtained posture data.
  • the image of the navigation information can be composited in the real picture as a virtual object.
  • the navigation information composed of a character, a symbol, or a number for a building as an eyemark of the course guide can be expressed visually and intuitively, even when the building as the eyemark is hidden by the building or the like in front of the building as the eyemark or is hidden by a curved road.
  • the road shape data is converted to image data in a perspective two-dimensional characteristics space and the data of the road shape model is converted to image data in the perspective two-dimensional characteristics space, and the two-dimensional road shape data is matched with the two-dimensional data of the road shape model in the two-dimensional characteristics space to estimate the posture data of the in-vehicle camera or the movable body to the road surface as the object.
  • matching the road shape data with the data of the road shape model performed for estimating the posture data can be made with the use of 2 pieces of the two-dimensional data in the pseudo-three-dimensional two-dimensional characteristics space, without use of a method assumed to lead to the case that the information amount is enormously large and high speed processing of data becomes difficult such as a method using 2 pieces of three-dimensional data in the three-dimensional logical space. Therefore, the matching process can be simplified and the speed thereof can be accelerated.
  • the road shape model of a multilane road may be created by performing modeling while considering a road structure in which one-way grade of a traveling road surface in a curved road is set to change according to a horizontal curvature of the curved road.
  • an existence probability of a road white line included in the landscape is obtained from the real picture and an RSL value is calculated by using a road shaped look-up table (RSL), the posture data is obtained so that an evaluation value based on the RSL value becomes the maximum, and thereby the data of the road shape model is matched with the road shape data.
  • RSL road shaped look-up table
  • the foregoing step of displaying the image or the foregoing image display means it is possible to display the image in which the read navigation information is composited in the determined appropriate position in the captured real picture on a given display screen of the display unit such as a liquid crystal display panel for car navigation set in the almost central portion of the dashboard or the like.
  • the image in which the read navigation information is composited in the determined position in the captured real picture is projected and displayed on the internal surface of a transparent window in front of the driver seat by a display unit such as a so-called HUD (Head Up Display) projection device.
  • a display unit such as a so-called HUD (Head Up Display) projection device.
  • the navigation information corresponding to the detected current position of the movable body is read from the navigation information previously stored in association with the road map data
  • the road shape model of the road assumed to be captured from the current position is created based on the current position of the movable body and the road map data
  • the road shape data as the image data of the road included in the landscape is extracted from the real picture
  • the data of the road shape model is matched with the road shape data to estimate the posture data of the in-vehicle camera or the movable body to the road as the object
  • the display position in the captured real picture of the navigation information read according to the current position of the movable body is determined based on the posture data
  • the data for displaying the image of the read navigation information in the determined position is outputted.
  • the image of the navigation information is projected and displayed on the internal surface of the transparent window in front of the driver seat of the movable body, and thereby the image of the navigation information is composited and displayed in the landscape seen through the transparent window in front of the driver seat.
  • the road shape model of the road assumed to be captured from the current position is created based on the current position of the movable body such as a motor vehicle and the road map data including the current position
  • the road shape data as the image data of the road included in the landscape ahead of the vehicle or in the traveling direction is extracted from the real picture
  • the data of the road shape model is matched with the road shape data to estimate the posture data of the in-vehicle camera or the movable body to the road in the landscape as the object
  • the appropriate display position in the captured real picture of the navigation information read according to the current position of the movable body is determined based on the posture data
  • the image in which the read navigation information is composited in the determined appropriate position in the captured real image is displayed.
  • the navigation information such as a course guide, a vehicle position, and map information can be accurately projected and displayed in an appropriate position in the real picture or in the real landscape of the road ahead of the movable body.
  • the driver can intuitively and accurately recognize a relation between the navigation information and the real picture or the real landscape.
  • visibility of a caution-needed picture such as a pedestrian in the real picture and a real picture of a road construction site is not inhibited by the image of the navigation information. Consequently, the visibility can be favorable over the entire display image.
  • FIG. 1 is a diagram showing an outline structure of a movable-body navigation information display unit according to an embodiment of the invention
  • FIGS. 2A to 2D are views showing a relative position relation among three-dimensional vehicle coordinate system VCS, three-dimensional camera coordinate system CCS, and two-dimensional projection image coordinate system ICS;
  • FIG. 3 is a view showing an example of mapping composed of dots and lines expressed by road map data
  • FIG. 4 is a diagram showing a road segment horizontal shape model obtained by approximation with the clothoid curved line
  • FIG. 5 is a view showing an example of a road horizontal shape model used in 2D-2D matching
  • FIG. 6 is a flowchart showing a flow of a series of main processing including extracting road shape data, creating a road shape model, estimating a camera posture parameter and the like in a central processing section;
  • FIG. 7 is a diagram showing various mathematical formulas together that are used for various calculations performed in the central processing section
  • FIG. 8 is a view showing an example of an image finally displayed
  • FIGS. 9A and 9B are schematic views showing rotation of a three-dimensional icon displayed in a composite image
  • FIG. 10 is a view showing an example of a screen structure in the case that a character information exclusive display region is previously provided on the periphery of a real picture;
  • FIG. 11 is a view showing another example (different from the view of FIG. 8 ) of the image finally displayed;
  • FIG. 12 is a schematic view showing a state that 2 three-dimensional icons are displayed while being rotated at a phase different from each other;
  • FIG. 13 is a view showing an example of a picture on which an image of navigation information is superimposed, which is stated displayable in Patent document 1;
  • FIG. 14 is a view showing an example of a landscape ahead of a real motor vehicle that is seen through a windshield from a driver seat of the real motor vehicle and;
  • FIGS. 15A and 15B are schematic views to compare the degree of change of a projection image onto a road surface with respect to unit angle change in the visual line from a driver seat at about 1 m high of a general vehicle ( FIG. 15A ) to the degree of change of a projection image onto a road surface with respect to unit angle change in the visual line at about 10 m high, which is higher than the height in FIG. 15A ( FIG. 15B ).
  • FIG. 1 shows an outline structure of a movable-body navigation information display unit of the invention.
  • a movable-body navigation information display method according to an embodiment of the invention is embodied by operations or functions of the movable-body navigation information display unit. Therefore, a description will be given of both the movable-body navigation information display unit and the movable-body navigation information display method together.
  • the movable-body navigation information display unit includes a sensor input section 1 , an operating section 2 , a map managing section 3 , a central processing section 4 , a picture display section 5 , and a controlling section 6 as the main sections.
  • the sensor input section 1 includes a CCD (solid-state image sensing device) camera 101 , a GPS sensor 102 , an INS sensor 103 , and a VICS 104 .
  • the CCD camera 101 is set on the dashboard of the driver seat or in the vicinity of the ceiling (not shown) in a movable body such as a motor vehicle (hereinafter referred to as vehicle or the movable body) in which the movable-body navigation information display unit is mounted, so that the CCD camera 101 shoots (captures) a landscape ahead of the vehicle at the camera angle almost similar to the visual line of a driver seeing the landscape through the windshield.
  • vehicle or the movable body a motor vehicle
  • the CCD camera 101 is, for example, a fixed focal length monocled type camera.
  • the CCD camera 101 captures an image of the landscape ahead of the vehicle including a road, and takes in an image signal thereof.
  • the taken image signal is transmitted as data to an image memory (not shown) in the central processing section 4 .
  • Traveling direction data and speed data of the movable body acquired by the GPS sensor 102 and the INS sensor 103 is transmitted to the central processing section 4 , while being synchronized with the image data acquired by the CCD camera 101 .
  • Data received by the VICS 104 as a road traffic information receiver is also transmitted to the central processing section 4 .
  • the operating section 2 transmits a command such as system setting and mode change to the central processing section 4 in response to an operation instruction inputted by a user through a button operation, a remote controller input apparatus (not shown) or the like.
  • the map managing section 3 reads various information of a road position designated by a command input through the operating section 2 from map data CD 301 that previously records map data in a given geographic region, and transmits read data to the central processing section 4 .
  • the central processing section 4 includes four modules composed of an image processing module 401 , a positioning processing module 402 , a picture output processing module 403 , and a control output processing module 404 and an image data creating section 405 as main sections thereof.
  • the image processing module 401 performs, for example, posture estimation of the mounted CCD camera 101 , tracing of a driving lane, detection of an obstruction, and calculation of the distance between two vehicles.
  • the positioning processing module 402 matches the direction and the speed sent from the sensor input section 1 with the road map data of the map managing section 3 , calculates accurate road position information, and outputs data thereof.
  • the picture output processing module 403 expresses a course guide, a vehicle position, and map information in the picture display section 5 as a virtual object in a three-dimensional augmented reality space, projects the virtual object on a two-dimensional road picture with the use of the posture parameter of the CCD camera 101 obtained by the after-mentioned estimation method, and integrates (composites) the virtual object in a real picture of the landscape ahead of the movable body. Further, the picture output processing module 403 creates data for, for example, emphatically displaying road sectional lines in the bad weather and displaying dangerous substances such as an obstruction.
  • the picture output processing module 403 iconifies information of objects to be able to become an eyemark in route guide such as a landmark, a railroad station, a hospital, and a gas station as information to be added to the road map, and projects the iconified information on the real picture of the road by using the camera posture parameter.
  • the control output processing module 404 comprehensively determines each analysis result, and gives an alarm output instruction for outputting an alarm or the like corresponding to the degree of danger for the vehicle to the controlling section 6 .
  • the image data creating section 405 Based on the data mainly outputted from the image processing module and the positioning processing module and the map data read by the map managing section 3 , for example, the image data creating section 405 recognizes the driving lane of the vehicle, a road shape, and an obstruction of currently driving vehicle, recognizes a vehicle's absolute on-road position, and estimates the camera posture. The image data creating section 405 thereby creates data for compositing and displaying the navigation information on an appropriate position in the real picture. More specifically, such display data is created based on the following three-dimensional modeling rules in the virtual space:
  • the picture display section 5 displays the picture (image) in which the navigation information is composited in the appropriate position of the real picture on the screen such as a liquid crystal display panel.
  • the controlling section 6 controls outputting the alarm or the like corresponding to the foregoing alarm output instruction, controls voice output corresponding to the analysis result by the control output module, controls a brake, and controls steering by, for example, controlling operations of each servo motor system provided for adjusting the control amount thereof.
  • the image data creating section 405 creates a road shape model for a road assumed to be captured from the current position, based on current position data detected by the GPS sensor 102 , the INS sensor 103 , and the positioning processing module.
  • road shape data as image data of a road included in a landscape in the traveling direction is extracted from the real picture, based on, for example, white line image data of a road traveling sectional line and the like.
  • the image data creating section 405 matches the road shape data with data of the road shape model data. Then, the image data creating section 405 estimates posture data of the CCD camera 101 (or posture data of the vehicle) to the road in the landscape as an object captured by the CCD camera 101 .
  • an appropriate display position in the captured real picture of the navigation information read according to the current position of the movable body is determined.
  • image data that enables displaying an image in which the read navigation information is composited in the determined appropriate position in the captured real picture is created.
  • the image data creating section 405 determines what position in the real picture the navigation information is appropriately composited by matching the data of the road shape model with the road shape data.
  • the picture display section 5 can display the image in which the navigation information such as the route guide, the vehicle position, and the map information is appropriately composited in the appropriate position in the real picture of the road ahead of the movable body or in the real landscape seen through the windshield.
  • the movable-body navigation information display unit of this embodiment it is possible to display the picture with which the relation between the navigation information and the real picture/real landscape can be intuitively and accurately recognized by the driver.
  • the foregoing navigation information presents at least one of a course guide for a route to arrive at the destination of the movable body, the vehicle position, the driving lane of the vehicle, and the building as an eyemark for the driver of the movable body to check the course guide or the vehicle position.
  • the navigation information is information of a character, a symbol, or a number, it is desirable that the information is iconified, and the icon image is composited and displayed in the captured real picture.
  • the image data creating section 405 expresses the navigation information as a virtual object in the three-dimensional augmented reality space, and assigns a corresponding position in the road shape data converted to the two-dimensional characteristics space to the expressed navigation information based on already obtained posture data and the like. Thereby, the image data creating section 405 composites the image of the navigation information as a virtual object in an appropriate position of the real picture.
  • the navigation information composed of a character, a symbol, or a number for a building as an eyemark of the course guide can be expressed visually and intuitively, even when the building as the eyemark is hidden by the building or the like in front of the building as the eyemark or is hidden by a curved road.
  • the image data creating section 405 converts the road shape data to image data of the perspective two-dimensional characteristics space, and converts the data of the road shape model to image data of the perspective two-dimensional characteristics space.
  • the image data creating section 405 matches the foregoing 2 pieces of two-dimensional data each other in the two-dimensional characteristics space, and estimates the posture data of the in-vehicle camera or the movable body to the road surface as an object. Since matching the road shape data with the data of the road shape model performed for estimating the posture data is made with the use of 2 pieces of the two-dimensional data in the pseudo-three-dimensional two-dimensional characteristics space, the matching process is simplified and the speed thereof is accelerated.
  • the road shape model is created, modeling is performed by considering the road structure in which the one-way grade of the traveling road surface in the curved road is set to change according to the horizontal curvature of the curved road.
  • the posture data of the movable body may be obtained as follows. That is, a road look-up table is used (RSL, reference: “Research on extracting and simultaneous tracing multiple movable bodies by visual motion interpretation of an in-vehicle camera,” H U Zhencheng, doctoral thesis, graduate school of Science and Technology, National University Corporation Kumamoto University). The existence probability of a road white line included in the landscape is obtained from the real picture, and thereby the RSL value is calculated. The posture data of the movable body is obtained so that the evaluation value based on the RSL value becomes the maximum. Thereby, it is possible to always extract accurate road shape data without being adversely influenced by various environmental factors such as weather change and shadow or dirt of the road surface. In the result, an accurate posture data can be estimated by using the extracted data.
  • RSL road look-up table
  • the image in which the navigation information is composited in the determined appropriate position in the captured real picture on a given display screen of the display unit such as a liquid crystal display panel for car navigation set in the almost central section of the dashboard or the like.
  • the image in which the read navigation information is composited in the appropriate position in the captured real picture determined by the foregoing matching is projected and displayed on the internal surface of a transparent window in front of the driver seat by a display unit such as a so-called HUD projection device.
  • the image of the navigation information is HUD-projected and displayed on the internal surface of the windshield in front of the driver seat corresponding to the foregoing position. Thereby, the image of the navigation information is composited and displayed in the landscape seen through the transparent window in front of the driver seat.
  • FIG. 2 shows a relative position relation among three-dimensional vehicle coordinate system VCS (X v , Y v , Z v ), three-dimensional camera coordinate system CCS (X c , Y c , Z c ), and two-dimensional projected image coordinate system ICS (x i , y i ).
  • the origin of the three-dimensional vehicle coordinate system VCS is located at the midpoint of the central line of the vehicle rear wheels.
  • the axis Z v is set on the central line, the axis X v is set to be oriented leftward, and the axis Y v is set to be oriented upward, respectively.
  • the origin of the three-dimensional camera coordinate system CCS is located at the central point of the lens of the CCD camera.
  • the axis Z c is set to be overlapped on the optical axis of the camera.
  • the conversion relation from the camera coordinate system CCS to the image coordinate system ICS is orthogonal projection. Therefore, the relation can be described as the relational expression based on a matrix as shown in Formula 1 of FIG. 7 .
  • P is the coordinates [X c , Y c , Z c , 1] in the CCS coordinate system
  • p is the coordinates [x i , y i , 1] in the ICS coordinate system.
  • A is a 3 ⁇ 4 projection matrix, and can be generally decomposed as shown in Formula 2 of FIG. 7 .
  • K is called the parameter matrix inside the camera, and is determined by the image horizontal and vertical direction transformation ratio (S x , Sy), the image central point (uo, vo), and the rotation transformation ratio S ⁇ . K is expressed as Formula 3 in FIG. 7 .
  • the camera posture matrix M is called the parameter matrix outside the camera, and represents a conversion relation from a view point to the target model coordinate system.
  • the camera posture matrix M can be expressed as Formula 4 of FIG. 7 by three-dimensional translation and rotational conversion of a rigid body.
  • R11 to R33 (all elements of R) represent the rotation parameters
  • T x , T y , and T z , (all elements of T) represent the translation parameters.
  • the camera magnification can be approximated with 1. Therefore, based on formula 1 to formula 4, the restraint formula as shown in Formula 5 of FIG. 7 is established.
  • the projection relation between the image coordinate system and the vehicle coordinate system is expressed by the formula shown in Formula 6 of FIG. 7 . That is, according to Formula 6, one pair of corresponding points in the 2D-3D space (p, P) determines one restriction formula as shown in Formula 6 of FIG. 7 to the camera posture data. Theoretically, such 6 pairs of corresponding points are enough to estimate the camera posture.
  • a multilane road shape model is estimated from road map information by avoiding the matching in 2D-3D (matching of the two-dimensional space with the three-dimensional space).
  • the estimated multilane road shape model is matched with multilane road shape data extracted from real picture data in the 2D-2D characteristics space.
  • the camera posture data is estimated by such matching in the two-dimensional space to the two-dimensional space.
  • the matching is not limited to the matching in the 2D-2D characteristics space.
  • the depth data in the three-dimensional space is accurately and surely estimated from an information source other than the real image data of the landscape ahead of the vehicle, and matching of the two-dimensional space to the three-dimensional space is made by using such data.
  • the data amount to be processed tends to become larger than that of the case of matching in the 2D-2D characteristics space.
  • FIG. 3 shows an example of a mapping with the use of dots and lines expressed by the road map data.
  • the road map data three-dimensional position information such as the latitude, the longitude, and the altitude of a road segment generally called a node, a road name, a road class, the number of lanes, a crossing state and the like are recorded.
  • the road width can be estimated based on the road class.
  • a node position described in the map data is located on the road centerline.
  • the structure of the road is composed of a complex curved surface using the horizontal curvature and the vertical curvature.
  • FIG. 4 shows a road segment horizontal shape model obtained by approximation with the clothoid curved line.
  • a road segment horizontal shape model can be modeled by using the mathematical formula as shown in Formula 7 of FIG. 7 .
  • c 0 and c 1 in Formula 7 respectively represent the change parameters of the initial curvature and the curvature of the horizontal curved line.
  • nli represents the number of up lanes
  • nri represents the number of down lanes
  • wi represents the average road width between segments.
  • Li represents the segment length.
  • the traveling position of vehicles is not on the road centerline, but is often deviated leftward from the road centerline.
  • the road horizontal shape model of the road centerline corresponding to the fact that the actual traveling position is deviated to the one lane side can be converted to a new model based on the vehicle coordinate system VCS.
  • the projection formula of the road horizontal shape model as shown in Formula 8 of FIG. 7 is estimated from Formula 6 of FIG. 7 as the projection conversion formula.
  • the three-dimensional road shape is projected perspectively and two dimensionally as it were. Therefore, the 2D-3D matching having a possibility to result in tangled data processing as a matching method to estimate the camera posture can be simplified to the 2D-2D matching.
  • the road shape model estimated from the road map data and the road shape data extracted from the real picture are 2D-2D matched in the two-dimensional characteristics space, and the optimal posture vector is estimated.
  • FIG. 5 shows an example of the road horizontal shape model used for the 2D-2D matching.
  • the road white line, the sectional line, and a borderline prospect of the road region are extracted as a characteristics region, and the image is binarized (a pixel belonging to the characteristics region is 1, and the other pixels are 0). Then, by using Formula 9 shown in FIG. 7 , the RSL value of each pixel is calculated. ⁇ , x, and y are binarized pixel values. ⁇ , i, and j are the kernel coefficient for the RSL. To inhibit noise, the kernel size is generally set to 5 or 7. Each coefficient is determined by the Gaussin distribution formula. The final evaluation formula of the camera posture estimation is shown in Formula 10 of FIG. 7 . ⁇ is a set of two-dimensional projection points of the road horizontal shape model. By using Formula 10, it is possible to obtain the best RSL evaluation value for totally matching the road shape model created based on the road map data with the road shape data extracted from the real picture.
  • the camera posture data obtained as above is used for data matching for determining the position in which the navigation information should be displayed.
  • the camera posture data is also used for the next estimation as the feedback amount.
  • FIG. 6 is a flowchart showing a flow of a series of main processing including extracting the road shape date, creating the road shape model, estimating the camera posture parameter and the like in the central processing section.
  • the real picture data of the landscape including the road ahead of the vehicle took by the CCD camera is captured (S 1 ).
  • the data acquired by the GPS sensor and the INS sensor is captured (S 2 ).
  • a white line of the road traveling sectional line or the like and a sectional line region of a lane of a paved face or the like are extracted from the real picture data, and the RSL value is calculated (S 3 ).
  • the current position (absolute position) of the vehicle is obtained (detected) by so-called map matching, and the related map data corresponding to the current position is read out from information stored in the map data CD (S 4 ).
  • the road horizontal shape model for the road in the landscape captured as the real picture is constructed (S 5 ).
  • the road horizontal shape model is projected in the perspective two-dimensional space (S 6 ).
  • the RSL expression of the road picture is matched with the projected road horizontal shape model, and the evaluation value is obtained (S 7 ).
  • Determination is made whether or not the obtained evaluation value is the maximum value (S 8 ). If determining that the evaluation value is the maximum value (Y of S 8 ), the posture vector then is outputted (S 9 ), and the outputted value is feed backed as the search origin for the next estimation processing (S 10 ). If determining that the evaluation value is not the maximum value (N of S 8 ), the posture vector is updated by Hooke&Jeeves method (S 11 ), and reevaluation is made (S 11 to S 6 through S 8 ). This procedural loop is repeated until the maximum value is obtained (until Y of S 8 ).
  • the image data creating section 405 creates the data for specifically displaying each piece of the navigation information. More specifically, creating such data is made based on the 9 types of the three-dimensional modeling rules 1 to 9 described above. In the result, the images shown as an example in FIG. 8 to FIG. 12 are displayed on the screen of the picture display section 5 .
  • the landmark and the facility information are displayed with the three-dimensional icon expressing them schematically.
  • a three-dimensional icon 801 indicating existence of a “hospital” is composited and displayed in the vicinity of the hospital in the real picture.
  • the three-dimensional icon 801 indicating the hospital has, for example, the design consisting of a white cross on a red background.
  • the size (outer dimensions) and the tint thereof are set to the degree with which the visibility of the picture of the buildings and the road of the real picture as a background is not inhibited.
  • the three-dimensional icon 801 is displayed as if the icon 801 rotates at a given rotation speed centering on a vertical axis 802 .
  • the three-dimensional icon 801 is oriented to the front face ( FIG. 9A )
  • a user can clearly recognize that the icon 801 indicates the “hospital.”
  • the icon 801 is turned 90 deg from the state oriented to the front face, as shown in the enlarged view of FIG. 9B , only an extremely thin area of a thickness thereof 803 is displayed. Therefore, the icon does not inhibit visibility of the buildings, pedestrians, obstructions and the like in the real picture existing (or being displayed) behind the three-dimensional icon 801 , and the user can clearly recognize the existence thereof.
  • Indicative information such as the traveling direction is displayed with a three-dimensional arrow.
  • a virtual road painting 804 indicating the limiting speed
  • a three-dimensional arrow 805 indicating the direction to travel
  • virtual road paintings 806 , 807 indicating the lane are composited and displayed in the real picture of the road.
  • character information or the like for displaying the destination is not displayed to overlap on the region under a virtual vanishing point or the virtual horizon line in the real picture, that is, more specifically, on the road real picture.
  • a destination display 808 mainly composed of the character information is displayed in the region of the real picture in which almost only the sky is shown that is located in the vicinity of the upper end of an entire real picture 809 (entire screen in the case of FIG. 8 ).
  • the image mainly composed of the character information such as the destination display 808 is not displayed in the region under the virtual vanishing point or the virtual horizon line in the real picture or the real landscape in which roads, pedestrians, and buildings exist with high probability. Thereby, it is possible to inhibit the real picture in which roads, pedestrians, and buildings exist or the real landscape from being hidden and invisible.
  • the navigation information is composited and displayed in the determined position in the captured real picture as the three-dimensional icon that rotates centering on the vertical axis (or the horizontal axis).
  • each of the plurality of three-dimensional icons may be rotated at the phase different from e one another or at the rotational speed different from one another.
  • a first three-dimensional icon 121 and a second three-dimensional icon 122 are displayed so that the icons are rotated in the same direction and at the same rotational speed but at the phase different from each other by about 90 deg.
  • the phase different from each other By setting the phase different from each other by about 90 deg, when the first three-dimensional icon 121 is oriented to the front face, the second three-dimensional icon 122 is oriented to the side face, and only the first three-dimensional icon 121 is substantially viewed by the user. Meanwhile, when the second three-dimensional icon 122 is oriented to the front face, the first three-dimensional icon 121 is oriented to the side face, and only the second three-dimensional icon 122 is substantially viewed by the user.
  • road name information 811 and limiting speed information 812 set to be displayed as the character information is displayed not in the real picture but in a character information exclusive display region 810 previously provided on the periphery of the real picture 809 .
  • each piece of the navigation information can be composited and displayed in a relevant position in the real picture.
  • warning information that is, navigation information with the urgency lower than that of the information such as the warning information
  • display is made on and after when the current position of the movable body to the warning target in the real picture (or real landscape in the case of HUD) enters within a given distance.
  • a description thereof will be given more specifically with reference to an example shown in FIG. 11 .
  • 3 three-dimensional icons, 821 (representing a “hotel”), 822 (representing a gas station), 823 (representing a convenience store) are displayed.
  • each real substance starts to be displayed when each real substance enters within a predetermined distance such as 100 m to the vehicle of the user.
  • Each icon is not displayed when the real substance is located apart from the vehicle farther than the predetermined distance.
  • the farthest three-dimensional icon 822 just enters within the given distance and starts to be displayed.
  • simple road map information 824 displayed in the lower left of the real picture (screen) of FIG. 11 the image of the navigation information is displayed as a translucent icon so that the real picture as a background can be viewed translucently. It is needless to say that not only the simple road map information 824 but also other information can be displayed as a translucent icon.
  • the image of the navigation information can be accurately composited in the appropriate position in the real picture of the road ahead of the movable body or in the real landscape seen thorough the windshield.
  • the picture can be displayed so that the driver can intuitively and accurately recognize the relation between the navigation information and the real picture or the real landscape.

Abstract

A movable-body navigation information display unit is provided. In the movable-body navigation information display unit, a driver can intuitively and accurately recognize a relation between navigation information and a real picture or a real landscape. In addition, it is possible to avoid a state that visibility of a caution-needed picture such as a pedestrian in the real picture and a real picture of a road construction site is inhibited by an image of the navigation information. An image data creating section (405) matches road shape data with a road shape model to estimate posture data. In addition, the image data creating section creates picture (image) data for accurately compositing and displaying the image of the navigation information in an appropriate position in a real picture (or in a real landscape) of a road ahead of a movable body, and displays the navigation information as a three-dimensional icon or the like. A picture display section (5) performs display based on the picture data.

Description

TECHNICAL FIELD
The present invention relates to a movable-body navigation information display method and movable-body navigation information display unit for, for example, collecting, processing, and displaying information on a current position of a movable body and information for navigation related thereto, such as a so-called car navigation system and a display method of car navigation information thereby.
BACKGROUND ART
movable-body navigation system such as a so-called car navigation system has been developed. In the movable-body navigation system, a current position of a movable body such as a motor vehicle is detected, the detected current position is displayed on a display screen together with a road map of surroundings, various guide information such as a destination and a route is displayed, and voice output is made.
In such an existing car navigation system, the current position of the movable body is checked as follows. That is, an autonomous-navigation controlling section performs tracing on a map previously stored as data in a storing means such as a CD-ROM (Compact Disc Read Only Memory) based on the speed pulses from a speed sensor and a direction obtained by earth magnetism from a geomagnetic sensor. In addition, a GPS signal transmitted from a GPS satellite is used. In the car navigation system in the early stage of the development in this field, there was an apparent malfunction (detection error) or the like in the past. That is, due to a measurement error or the like, an actual traveling position is significantly deviated from the trace position on the map. For example, when a vehicle is traveling along the coast actually, the traveling position of the vehicle detected by the car navigation system is indicated in the sea. However, for addressing such a significant position deviation, a technique of position correction (map matching) and the like have been developed. Therefore, in these days, the precision of detecting the traveling position of the vehicle is sufficient.
By superimposing information of the current position of the vehicle obtained as above on map information obtained from a large-volume data storing means such as a CD-ROM, an image showing the current position and the road map in a given range in the vicinity thereof can be displayed as a plane map-like two-dimensional image on a screen of an image display unit such as a liquid crystal display unit. In addition, it is possible to display so-called navigation information such as a vehicle position, a traveling direction, speed, map information, a course guide, the shortest route, and road information by being superimposed on the two-dimensional image map, or by being inserted in an appropriate position in the two-dimensional image map.
In some existing navigation display units, an image of information, for example, a traveling position and a traveling direction indicated by an icon of an arrow or the like, a recommended route (optimal route) from the current position to the destination is displayed by being superimposed on a two-dimensional image road planar map or on a road map that shows the surrounding landscape in a state of a three-dimensional CG. Further, in some existing navigation display units, traffic jam information or the like obtained by a VICS sensor or the like is displayed with a text label on the screen.
However, such an existing technique can only provide indirect navigation information using the plane (two dimensional) simplified map and the image expressed three dimensionally but abstractly and simply expressed by CG. Therefore, for a driver in the motor vehicle, there are problems, for example, as follows. That is, it is not possible to intuitively recognize a relation between the information such as the current position and the course guide and the road in the landscape actually seen through the windshield.
To solve the foregoing problems, for example, in Patent document 1, Patent document 2 and the like, a navigation system using a real picture in addition to the map information is proposed.
In the technique disclosed in Patent document 1, a current position and a traveling direction of a movable body are detected, and navigation information to be displayed according to the detected current position and the detected traveling direction is selected. An image of the navigation information read according to the current position of the vehicle is superimposed on a real picture captured by an image pickup device from a camera angle approximately similar to a landscape ahead of the vehicle in the traveling direction seen through the windshield of the vehicle, and the superimposed result is displayed on a screen or the like of a liquid crystal display panel. It is stated in Patent document 1 that a relation between the picture of the real road, the surrounding landscape thereof and the navigation information can be displayed easily recognizable visually. Further, in a general car navigation system, the navigation information is mainly picture information displayed on the picture display section. Therefore, there is a possibility that a driver gets distracted easily while driving.
Therefore, in Patent document 2, the following technique is proposed. That is, an image pickup device such as a CCD camera that captures a landscape ahead of a body for motor vehicle is provided, for example, in a position near the ceiling of the windshield of the vehicle or in the vicinity of the dashboard. A picture (image) of the landscape including a road ahead of the body for motor vehicle captured by the image pickup device is displayed by being inserted in a given position of the screen displaying map information as a sub-screen. According to such a technique, even when the driver of the vehicle views navigation information such as a map displayed on the screen of the image display unit while driving, the driver views the real picture of the landscape ahead of the vehicle displayed as the sub-image in the given position on the screen. It is stated in Patent document 2 that the driver can thereby comprehend a state in front of the vehicle without returning her/his attention to the landscape ahead of the vehicle.
Further, the following technique or the like has been proposed. In such a technique, setting is made so that, for example, when it is determined that an obstruction shown in the real picture of the sub-screen is large or when an object suddenly darts out from the side, the outer dimensions of the sub-screen (screen size) are enlarged. Thereby high-risk state in front of the vehicle is informed to the driver promptly and visually. In addition, such a high-risk state can be shown to the driver with high visibility. Consequently, safer driving can be secured.
Further, to determine the leading vehicle and operate automatic steering, it is necessary to more accurately comprehend the landscape including the road and the like ahead of the movable body as data. Therefore, for example, in Patent document 3, the following technique is proposed. In such a technique, a three-dimensional road shape is estimated by using image road information and map road information. In the technique disclosed in Patent document 3, the image road shape extracted from image data of the landscape ahead of the vehicle and map data around the vehicle are projected in one two-dimensional or three-dimensional logical space. Based on an overlapping state in the logical space of both of the projected road shapes, estimation is made for the road shape, a posture of the vehicle to the road surface, an absolute position of the vehicle and the like. In this technique, based on a picture captured by a monocular CCD camera, a sufficiently accurate road shape is estimated, and accurate determination of the leading vehicle and automatic steering are realized.
  • Patent document 1: Japanese Unexamined Patent Application Publication No. 10-132598
  • Patent document 2: Japanese Unexamined Patent Application Publication No. 11-304499
  • Patent document 3: Japanese Unexamined Patent Application Publication No. 2001-331787
DISCLOSURE OF THE INVENTION
However, in the technique disclosed in the Patent document 1, there are problems as follows. That is, first, the image information is only used as a background picture. For example, regarding an image of an arrow indicating the destination, the image is firstly displayed in the central portion of the screen, and then is displayed to move linearly from the original position to the backward landscape (in general, from the upper side to the lower side of the display screen) in the central section of the screen as the vehicle travels.
Therefore, in the case of displaying the navigation information on a curved road or in the case of a road provided with multi-lanes such as two lanes on each side, there are the problems as follows. That is, it is not possible to determine what position in the landscape the navigation information display accurately indicates. In some cases, the navigation information display indicates a totally deviated position.
Secondly, the posture parameter of the camera is previously fixed as the angle of the camera optical axis to the ground surface (or horizontal direction). Therefore, for example, when the posture of the camera is changed due to vibration of the running vehicle, rolling and pitching of motor vehicle by steering, or tilt of the vehicle on the uphill slope and the downhill slope, a navigation information image of route such as right turning position and indicated by an arrow or the like of a traveling direction is largely deviated to the picture obtained by capturing the real landscape. Consequently, the arrow indicates a wrong direction substantially, and the right turning position is displayed unclearly.
For example, in an example shown in FIG. 13, an arrow 901 as a navigation information image appears to indicate the left turning position on the road in the landscape ahead of the vehicle clearly. However, in the case of a general passenger car, or even in the case of a vehicle such as a bus and a track with the driver seat located in a high position, there is no possibility that the road landscape actually seen through the windshield from the driver seat of the vehicle becomes a bird's-eye view down from a high position as shown in FIG. 13. In other words, except for a driver seat located in an extremely high position being 10 m or more high from the ground such as a cockpit of a jumbo jet that is unlikely to exist as a driver seat of a motor vehicle, it is not possible actually to capture the road picture in the landscape in which buildings and houses are crammed side by side as the bird's-eye view with which even turning points and back roads of the buildings can be seen as shown in FIG. 13. In reality, in the case of a general motor vehicle in which the driver's visual line is located only at about 1 m high, or in the case of a general vehicle such as a large truck and a bus in which the driver's visual line is located only at about 2 to 3 m high at the maximum, as an example in FIG. 14, the road picture often results in a landscape (and real picture) in which anteroposterior density is particularly high and a road that crosses the road in the traveling direction is not easily seen since such a road is hidden by the buildings and the like along the road in the traveling direction.
FIGS. 15A and 15B show comparative schematic views. FIG. 15A shows the degree of change (ΔLA) of a projection image onto a road surface with respect to the unit angle change (posture to the road surface) Δθe in the visual line at about 1 m high from a driver seat of a general vehicle. FIG. 15B shows the degree of change (ΔLB) of a projection image onto a road surface with respect to the unit angle change (posture to the road surface) Δθe in the visual line at about 10 m high, which is higher than the height in FIG. 15A. As shown in FIGS. 15A and 15B, comparing the degree of change of the projection image from the lower position (corresponding to the position deviation degree to the posture change Δθe in the case of capturing the image from the lower position) ΔLA to the degree of change of the projection image from the higher position (corresponding to the position deviation degree to the posture change Δθe in the case of capturing the image from the higher position) ΔLB, it is apparent that the relation thereof is expressed as ΔLA>>LB. Therefore, when the navigation information is superimposed on the real picture in a fixed position as shown in Patent document 1, due to posture change of the vehicle and posture change of the vehicle relative to the road resulting from a three-dimensional shape or the like of the road such as one-way grade (cant), it is often the case that the display position of the navigation information is largely deviated from the appropriate display position of the navigation information in the real picture. In the result, it is difficult or not possible that a user (driver) intuitively and accurately comprehends what position in the landscape the navigation information indicates.
More specifically, in the technique as Patent document 1 in which the navigation information such as the arrow is located in a fixed position such as on the central line on the screen of the real picture of the landscape, even if the posture of the vehicle is changed slightly, the deviation between the real picture of the landscape and the image of the navigation information becomes large. In addition, it is often the case that the landscape located near the vehicle along the road such as buildings and trees blocks the landscape located farther than the foregoing landscape. Therefore, to address such a problem, it is necessary to display the image of the navigation information on a more accurate position. However, it is not possible to take such a measure in Patent document 1.
Further, in general, in a movable body such as a motor vehicle, the posture thereof is in reality frequently changed due to rolling, pitching and the like of the vehicle while moving. Furthermore, for the foregoing reason, in the landscape viewed from only about 1 to 3 m high of a general vehicle, even slight posture change results in large position deviation. Therefore, in the technology of Patent document 1, there is a possibility that the following state is frequently caused. That is, the display position of the navigation information such as an arrow indicating the route is largely deviated from the appropriate display position in the real picture.
Furthermore, such posture change often varies according to each vehicle type, for example, according to difference of the barycentric position dominantly determined by the structure of a vehicle driving method or the like and the arrangement position of the engine. Therefore, it is necessary to address such posture change differently according to the vehicle type, the degree and the direction of the posture change. However, it is not possible to take such a measure in Patent document 1. Further, in the case of a two-wheeled vehicle and a scooter that pass a curved line by largely tilting the body for motor vehicle and the driver's body, the posture change of the vehicle becomes more significant. Therefore, in the case of the navigation system for such a two-wheeled vehicle, deviation of the display position resulting from the posture change of the vehicle becomes more significant.
Thirdly, it is desirable that a place as a promising eyemark for navigation such as a road name, a landmark, and a route guide of a hospital or the like that are attached to road map data is displayed three dimensionally in real time based on the current position of the vehicle. However, such an aspect is not totally considered or proposed in Patent document 1. Furthermore, as an example shown in FIG. 14, it is often the case that the building as an eyemark along the road is hardly seen since, such a building is hidden by the building and the landscape near the vehicle.
In the technique of Patent document 2, there is the following problem in addition to the problems similar to those of the Patent document 1 described above. That is, in Patent document 2, the real picture of the road landscape ahead of the vehicle is only displayed in the sub-screen. Therefore, matching of the road map information and the real picture should be made by the driver in his/her mind. Therefore, for example, when the driver drives a road with many crossings and forks in an unfamiliar (or firstly driving) place, it is difficult to intuitively comprehend the navigation information. In the result, there is a possibility that the user takes a wrong route, falsely recognizes the content indicated by the navigation information, and is not able to understand the content.
The technique of Patent document 3 has the following problems. That is, in such a technique, there is no specific road shape model. Therefore, in the case of a multilane road, there is a possibility that large deviation is caused between the central line of the driving lane of the vehicle extracted from the image data and the road centerline estimated from the road map data. Further, it is not possible to estimate information of the driving lane of the vehicle. Thus, accurate navigation information is not able to be provided in changing the lane and in turning left or right.
Further, such a technique does not consider the road structure that the one-way grade (cant) of the traveling road surface is set to be changed according to the horizontal curvature. Therefore, in the curved road, the estimation result of the posture data to the road surface of the lane adjacent to the driving lane of the vehicle is largely changed. In the result, there is a possibility that the road shape is not estimated accurately.
Further, in extracting the road characteristics from the image, the estimation result of the road shape is not feed backed, and the section with large luminance change is simply extracted as a white line by a differential filter for every frame. However, the estimation result obtained by the foregoing method is largely subject to influences of various environmental factors such as weather change and shadow or dirt on the road surface. Therefore, the road shape model expressed by the data obtained by extracting the road characteristics may become an incorrect model that is significantly deviated from the road shape of the real object.
Further, when the image of the navigation information is composited and displayed in the real picture in front of the vehicle, there is a possibility as follows. For example, the image of the navigation information is superimposed on a caution-needed picture such as a pedestrian and other obstruction in the real picture. Thereby, the visibility of the caution-needed picture may be inhibited.
In view of the foregoing, it is an object of the invention to provide a navigation information display method and a movable-body navigation information display unit. In the navigation information display method and the movable-body navigation information display unit, navigation information such as a course guide, a vehicle position, and map information is accurately projected and displayed in an appropriate position in a real picture or in a real landscape of the road ahead of the movable body. Thereby, a driver can intuitively and accurately recognize a relation between the navigation information and the real picture or the real landscape. In addition, visibility of a caution-needed picture such as a pedestrian in the real picture and a real picture of a road construction site is not blocked by the image of the navigation information.
A first movable-body navigation information display method according to the invention includes the steps of detecting a current position of a movable body, and capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data; creating a road shape model of a road assumed to be captured from the current position, based on the current position of the movable body and the road map data; extracting road shape data as image data of the road included in the landscape from the real picture; matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, and determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data; and displaying an image in which the read navigation. information is composited with display dimensions and tone set by a user in the determined position in the captured real picture.
A first movable-body navigation information display unit according to the invention includes: a current position detecting means for detecting a current position of a movable body; a capturing means for capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; a data processing means for reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data, creating a road shape model of a road assumed to be captured from the current position based on the current position of the movable body and the road map data, extracting road shape data as image data of the road included in the landscape from the real picture, matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data, and outputting data for displaying an image in which the read navigation information is composited as an image displayed with display dimensions and tone set by a user in the determined position in the captured real picture; and an image display means for compositing and displaying the read navigation information as the image displayed with the display dimensions and the tone set by the user in the determined position of the captured real picture based on the data outputted from the data processing means.
In the first movable-body navigation information display method or in the first movable-body navigation information display unit according to the invention, the road shape model of the road assumed to be captured from the current position is created based on the current position of the movable body such as a motor vehicle and the road map data including the current position. In addition, the road shape data as the image data of the road included in the landscape ahead of the vehicle or in the traveling direction is extracted from the real picture. The data of the road shape model is matched with the road shape data to estimate the posture data of the in-vehicle camera or the movable body to the road in the landscape as the object. Then, based on the posture data, the appropriate display position in the captured real picture of the navigation information read according to the current position of the movable body is determined. In the result, the image in which the read navigation information is composited with the display dimensions and the tone set by the user is displayed in the determined appropriate position of the captured real picture.
As mentioned above, the appropriate composite position of the navigation information in the real picture is determined by matching the data of the road shape model with the road shape data. Thereby, for example, display is enabled so that the navigation information such as a course guide, a position of a vehicle, and map information is projected accurately in an appropriate position in a real picture of a road ahead of the movable body or a real landscape. In the result, a driver can intuitively and accurately recognize a relation between the navigation information and the real picture or the real landscape. In addition, the display dimensions and the tone of the image (generically referred to as image, including character information and the like) of the navigation information composited in the real picture can be set as appropriate according to the user's option. Therefore, deterioration of visibility of the entire image in which the image of the navigation information is composited in the real picture is avoided. In addition, the user her/himself can customize the image of the navigation information to obtain the image with higher visibility for the user.
A second movable-body navigation information display method according to the invention includes the steps of: detecting a current position of a movable body, and capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data; creating a road shape model of a road assumed to be captured from the current position, based on the current position of the movable body and the road map data; extracting road shape data as image data of the road included in the landscape from the real picture; matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, and determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data; and previously defining three classifications of warning information, guidance information, and additional information depending on urgency of display necessity for the navigation information, previously determining priority for each classification, determining to which classification the navigation information corresponds, and compositing and displaying the navigation information in the determined position in the captured real picture according to an order corresponding to the classifications.
A second movable-body navigation information display unit according to the invention includes: a current position detecting means for detecting a current position of a movable body; a capturing means for capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; a data processing means for reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data, creating a road shape model of a road assumed to be captured from the current position based on the current position of the movable body and the road map data, extracting road shape data as image data of the road included in the landscape from the real picture, matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data, previously defining three classifications of warning information, guidance information, and additional information depending on urgency of display necessity for the navigation information, previously determining priority for each classification, determining to which classification the navigation information corresponds, and outputting data for compositing and displaying the navigation information as an image in the determined position in the captured real picture according to an order corresponding to the classifications and; an image display means for compositing and displaying the image of the read navigation information in the determined position of the captured real picture according to the order corresponding to the classifications, based on the data outputted from the data processing means.
In the second movable-body navigation information display method or in the second movable-body navigation information display unit according to the invention, as in the first movable-body navigation information display method or in the first movable-body navigation information display unit according to the invention, the appropriate composite position of the navigation information in the real picture is determined by matching the data of the road shape model with the road shape data. Thereby, for example, display is enabled so that the navigation information such as a course guide, a position of a vehicle, and map information is projected accurately in an appropriate position in a real picture of a road ahead of the movable body or a real landscape. In the result, a driver can intuitively and accurately recognize a relation between the navigation information and the real picture or the real landscape. In addition, the image of the read navigation information is composited and displayed according to the order corresponding to the classification based on the priority described above. Therefore, many navigation information images are inhibited from being displayed in one real picture. Consequently, deterioration of visibility of the entire image in which the image of the navigation information is composited in the real picture is avoided.
The navigation information other than the foregoing warning information may be displayed on and after when the current position of the movable body to a warning target in the real picture enters within a given distance. Thereby, the data processing amount for creating the image of the navigation information is saved. In addition, a navigation information image necessary for the user can be displayed timely when such navigation information becomes really necessary for the user.
A third movable-body navigation information display method according to the invention includes the steps of: detecting a current position of a movable body, and capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data; creating a road shape model of a road assumed to be captured from the current position, based on the current position of the movable body and the road map data; extracting road shape data as image data of the road included in the landscape from the real picture; matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, and determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data; and displaying an image in which the read navigation information is composited in the determined position in the captured real picture, and displaying the navigation information other than graphic display for guiding the traveling direction not in a region under a virtual vanishing point or a virtual horizon line in the real picture but in a region above the virtual vanishing point or the virtual horizon line.
A third movable-body navigation information display unit according to the invention includes: a current position detecting means for detecting a current position of a movable body; a capturing means for capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; a data processing means for reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data, creating a road shape model of a road assumed to be captured from the current position based on the current position of the movable body and the road map data, extracting road shape data as image data of the road included in the landscape from the real picture, matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data, and outputting data for compositing and displaying the navigation information other than graphic display for guiding the traveling direction not in a region under a virtual vanishing point or a virtual horizon line in the real picture but in a region above the virtual vanishing point or the virtual horizon line; and an image display means for compositing and displaying the read navigation information in the determined position of the captured real picture based on the data outputted from the data processing means.
In the third movable-body navigation information display method or in the third movable-body navigation information display unit according to the invention, as in the first movable-body navigation information display method or in the first movable-body navigation information display unit according to the invention, the appropriate composite position of the navigation information in the real picture is determined by matching the data of the road shape model with the road shape data. Thereby, for example, display is enabled so that the navigation information such as a course guide, a position of a vehicle, and map information is projected accurately in an appropriate position in a real picture of a road ahead of the movable body or a real landscape. In the result, a driver can intuitively and accurately recognize a relation between the navigation information and the real picture or the real landscape. In addition, the navigation information other than the graphic display for guiding the traveling direction is not displayed in the region under the virtual vanishing point or the virtual horizon line in the real picture but in the region above the virtual vanishing point or the virtual horizon line. Consequently, visibility of the real picture of a road and a surface ground is not inhibited by display of the navigation information.
A fourth movable-body navigation information display method according to the invention includes the steps of detecting a current position of a movable body, and capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data; creating a road shape model of a road assumed to be captured from the current position, based on the current position of the movable body and the road map data; extracting road shape data as image data of the road included in the landscape from the real picture; matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, and determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data; and compositing and displaying the read navigation information as a three-dimensional icon that rotates centering on a vertical axis or a horizontal axis in the determined position in the captured real picture.
A fourth movable-body navigation information display unit according to the invention includes: a current position detecting means for detecting a current position of a movable body; a capturing means for capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body; a data processing means for reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data, creating a road shape model of a road assumed to be captured from the current position based on the current position of the movable body and the road map data, extracting road shape data as image data of the road included in the landscape from the real picture, matching data of the road shape model with the road shape data to estimate posture data of the in-vehicle camera or the movable body to the road as the object, determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data, and outputting data for displaying an image in which the read navigation information is composited as a three-dimensional icon that rotates centering on a vertical axis or a horizontal axis in the determined position in the captured real picture; and an image display means for compositing and displaying the read navigation information as the three-dimensional icon that rotates centering on the vertical axis or the horizontal axis in the determined position in the captured real picture based on the data outputted from the data processing means.
In the fourth movable-body navigation information display method or in the fourth movable-body navigation information display unit according to the invention, as in the first movable-body navigation information display method or in the first movable-body navigation information display unit according to the invention, the appropriate composite position of the navigation information in the real picture is determined by matching the data of the road shape model with the road shape data. Thereby, for example, display is enabled so that the navigation information such as a course guide, a position of a vehicle, and map information is projected accurately in an appropriate position in a real picture of a road ahead of the movable body or a real landscape. In the result, a driver can intuitively and accurately recognize a relation between the navigation information and the real picture or the real landscape. In addition, the image of the navigation information is composited and displayed in the real picture as the rotatable three-dimensional icon that rotates centering on the vertical axis or the horizontal axis. Consequently, visibility of the real picture of a road and a surface ground is not inhibited by display of the navigation information.
The foregoing navigation information may be, for example, at least one of a course guide of a route to reach a destination of the movable body, a position of a vehicle, a driving lane of the vehicle, and a building as an eyemark for a driver of the movable body to check the course guide or the position of the vehicle.
When the navigation information is information of a character, a symbol, or a number, the information may be iconified, and the icon image may be composited and displayed in the captured real picture.
Further, in the foregoing step of performing the data processing or the foregoing data processing means, the navigation information may be expressed as a virtual object in the three-dimensional augmented reality space, and a corresponding position in the road shape data converted into the two-dimensional characteristics space may be assigned to the expressed navigation information based on already obtained posture data. Thereby, the image of the navigation information can be composited in the real picture as a virtual object. In the result, the navigation information composed of a character, a symbol, or a number for a building as an eyemark of the course guide can be expressed visually and intuitively, even when the building as the eyemark is hidden by the building or the like in front of the building as the eyemark or is hidden by a curved road.
Further, in the foregoing step of performing the data processing or the foregoing data processing means, it is possible that the road shape data is converted to image data in a perspective two-dimensional characteristics space and the data of the road shape model is converted to image data in the perspective two-dimensional characteristics space, and the two-dimensional road shape data is matched with the two-dimensional data of the road shape model in the two-dimensional characteristics space to estimate the posture data of the in-vehicle camera or the movable body to the road surface as the object. Thereby, matching the road shape data with the data of the road shape model performed for estimating the posture data can be made with the use of 2 pieces of the two-dimensional data in the pseudo-three-dimensional two-dimensional characteristics space, without use of a method assumed to lead to the case that the information amount is enormously large and high speed processing of data becomes difficult such as a method using 2 pieces of three-dimensional data in the three-dimensional logical space. Therefore, the matching process can be simplified and the speed thereof can be accelerated.
Further, in the foregoing step of performing the data processing or the foregoing data processing means, it is possible that in creating the road shape model, the road shape model of a multilane road may be created by performing modeling while considering a road structure in which one-way grade of a traveling road surface in a curved road is set to change according to a horizontal curvature of the curved road. Thereby, even when a movable body such as a motor vehicle is traveling on the multilane road, the road shape of the multilane road can be accurately comprehended. In the result, it is possible to synthesize and display the navigation information in an appropriate position accurately corresponding to such a road shape of the multilane road.
Further, in the foregoing step of performing the data processing or the foregoing data processing means, it is possible that an existence probability of a road white line included in the landscape is obtained from the real picture and an RSL value is calculated by using a road shaped look-up table (RSL), the posture data is obtained so that an evaluation value based on the RSL value becomes the maximum, and thereby the data of the road shape model is matched with the road shape data. Thereby, it is possible to always extract accurate road shape data without being adversely influenced by various environmental factors such as weather change and shadow or dirt of the road surface. In the result, accurate posture data can be estimated by using the extracted data.
Further, in the foregoing step of displaying the image or the foregoing image display means, it is possible to display the image in which the read navigation information is composited in the determined appropriate position in the captured real picture on a given display screen of the display unit such as a liquid crystal display panel for car navigation set in the almost central portion of the dashboard or the like.
Otherwise, it is also possible that the image in which the read navigation information is composited in the determined position in the captured real picture is projected and displayed on the internal surface of a transparent window in front of the driver seat by a display unit such as a so-called HUD (Head Up Display) projection device.
Further, it is also adopt the following aspect or the like. That is, in the foregoing step of performing the data processing or the foregoing data processing means, the navigation information corresponding to the detected current position of the movable body is read from the navigation information previously stored in association with the road map data, the road shape model of the road assumed to be captured from the current position is created based on the current position of the movable body and the road map data, the road shape data as the image data of the road included in the landscape is extracted from the real picture, the data of the road shape model is matched with the road shape data to estimate the posture data of the in-vehicle camera or the movable body to the road as the object, the display position in the captured real picture of the navigation information read according to the current position of the movable body is determined based on the posture data, and the data for displaying the image of the read navigation information in the determined position is outputted. In addition, in the foregoing image display means or in the foregoing step of displaying the image, the image of the navigation information is projected and displayed on the internal surface of the transparent window in front of the driver seat of the movable body, and thereby the image of the navigation information is composited and displayed in the landscape seen through the transparent window in front of the driver seat.
As described above, according to the movable-body navigation information display method or the movable-body navigation information display unit of the invention, the road shape model of the road assumed to be captured from the current position is created based on the current position of the movable body such as a motor vehicle and the road map data including the current position, the road shape data as the image data of the road included in the landscape ahead of the vehicle or in the traveling direction is extracted from the real picture, the data of the road shape model is matched with the road shape data to estimate the posture data of the in-vehicle camera or the movable body to the road in the landscape as the object, the appropriate display position in the captured real picture of the navigation information read according to the current position of the movable body is determined based on the posture data, and the image in which the read navigation information is composited in the determined appropriate position in the captured real image is displayed. Therefore, for example, the navigation information such as a course guide, a vehicle position, and map information can be accurately projected and displayed in an appropriate position in the real picture or in the real landscape of the road ahead of the movable body. Thereby, the driver can intuitively and accurately recognize a relation between the navigation information and the real picture or the real landscape. In addition, visibility of a caution-needed picture such as a pedestrian in the real picture and a real picture of a road construction site is not inhibited by the image of the navigation information. Consequently, the visibility can be favorable over the entire display image.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a diagram showing an outline structure of a movable-body navigation information display unit according to an embodiment of the invention;
FIGS. 2A to 2D are views showing a relative position relation among three-dimensional vehicle coordinate system VCS, three-dimensional camera coordinate system CCS, and two-dimensional projection image coordinate system ICS;
FIG. 3 is a view showing an example of mapping composed of dots and lines expressed by road map data;
FIG. 4 is a diagram showing a road segment horizontal shape model obtained by approximation with the clothoid curved line;
FIG. 5 is a view showing an example of a road horizontal shape model used in 2D-2D matching;
FIG. 6 is a flowchart showing a flow of a series of main processing including extracting road shape data, creating a road shape model, estimating a camera posture parameter and the like in a central processing section;
FIG. 7 is a diagram showing various mathematical formulas together that are used for various calculations performed in the central processing section;
FIG. 8 is a view showing an example of an image finally displayed;
FIGS. 9A and 9B are schematic views showing rotation of a three-dimensional icon displayed in a composite image;
FIG. 10 is a view showing an example of a screen structure in the case that a character information exclusive display region is previously provided on the periphery of a real picture;
FIG. 11 is a view showing another example (different from the view of FIG. 8) of the image finally displayed;
FIG. 12 is a schematic view showing a state that 2 three-dimensional icons are displayed while being rotated at a phase different from each other;
FIG. 13 is a view showing an example of a picture on which an image of navigation information is superimposed, which is stated displayable in Patent document 1;
FIG. 14 is a view showing an example of a landscape ahead of a real motor vehicle that is seen through a windshield from a driver seat of the real motor vehicle and;
FIGS. 15A and 15B are schematic views to compare the degree of change of a projection image onto a road surface with respect to unit angle change in the visual line from a driver seat at about 1 m high of a general vehicle (FIG. 15A) to the degree of change of a projection image onto a road surface with respect to unit angle change in the visual line at about 10 m high, which is higher than the height in FIG. 15A (FIG. 15B).
BEST MODE FOR CARRYING OUT THE INVENTION
An embodiment of the invention will be hereinafter described in detail with reference to the drawings.
FIG. 1 shows an outline structure of a movable-body navigation information display unit of the invention. A movable-body navigation information display method according to an embodiment of the invention is embodied by operations or functions of the movable-body navigation information display unit. Therefore, a description will be given of both the movable-body navigation information display unit and the movable-body navigation information display method together.
The movable-body navigation information display unit includes a sensor input section 1, an operating section 2, a map managing section 3, a central processing section 4, a picture display section 5, and a controlling section 6 as the main sections.
More specifically, the sensor input section 1 includes a CCD (solid-state image sensing device) camera 101, a GPS sensor 102, an INS sensor 103, and a VICS 104. The CCD camera 101 is set on the dashboard of the driver seat or in the vicinity of the ceiling (not shown) in a movable body such as a motor vehicle (hereinafter referred to as vehicle or the movable body) in which the movable-body navigation information display unit is mounted, so that the CCD camera 101 shoots (captures) a landscape ahead of the vehicle at the camera angle almost similar to the visual line of a driver seeing the landscape through the windshield. The CCD camera 101 is, for example, a fixed focal length monocled type camera. The CCD camera 101 captures an image of the landscape ahead of the vehicle including a road, and takes in an image signal thereof. The taken image signal is transmitted as data to an image memory (not shown) in the central processing section 4. Traveling direction data and speed data of the movable body acquired by the GPS sensor 102 and the INS sensor 103 is transmitted to the central processing section 4, while being synchronized with the image data acquired by the CCD camera 101. Data received by the VICS 104 as a road traffic information receiver is also transmitted to the central processing section 4.
The operating section 2 transmits a command such as system setting and mode change to the central processing section 4 in response to an operation instruction inputted by a user through a button operation, a remote controller input apparatus (not shown) or the like.
In accordance with a reading command by the central processing section 4, the map managing section 3 reads various information of a road position designated by a command input through the operating section 2 from map data CD 301 that previously records map data in a given geographic region, and transmits read data to the central processing section 4.
The central processing section 4 includes four modules composed of an image processing module 401, a positioning processing module 402, a picture output processing module 403, and a control output processing module 404 and an image data creating section 405 as main sections thereof.
The image processing module 401 performs, for example, posture estimation of the mounted CCD camera 101, tracing of a driving lane, detection of an obstruction, and calculation of the distance between two vehicles.
The positioning processing module 402 matches the direction and the speed sent from the sensor input section 1 with the road map data of the map managing section 3, calculates accurate road position information, and outputs data thereof.
The picture output processing module 403 expresses a course guide, a vehicle position, and map information in the picture display section 5 as a virtual object in a three-dimensional augmented reality space, projects the virtual object on a two-dimensional road picture with the use of the posture parameter of the CCD camera 101 obtained by the after-mentioned estimation method, and integrates (composites) the virtual object in a real picture of the landscape ahead of the movable body. Further, the picture output processing module 403 creates data for, for example, emphatically displaying road sectional lines in the bad weather and displaying dangerous substances such as an obstruction. Furthermore, the picture output processing module 403 iconifies information of objects to be able to become an eyemark in route guide such as a landmark, a railroad station, a hospital, and a gas station as information to be added to the road map, and projects the iconified information on the real picture of the road by using the camera posture parameter.
The control output processing module 404. comprehensively determines each analysis result, and gives an alarm output instruction for outputting an alarm or the like corresponding to the degree of danger for the vehicle to the controlling section 6.
Based on the data mainly outputted from the image processing module and the positioning processing module and the map data read by the map managing section 3, for example, the image data creating section 405 recognizes the driving lane of the vehicle, a road shape, and an obstruction of currently driving vehicle, recognizes a vehicle's absolute on-road position, and estimates the camera posture. The image data creating section 405 thereby creates data for compositing and displaying the navigation information on an appropriate position in the real picture. More specifically, such display data is created based on the following three-dimensional modeling rules in the virtual space:
  • 1. The landmark and facility information are displayed with a three-dimensional icon schematically expressing the landmark and the facility information.
  • 2. Indicative information such as a traveling direction is displayed with a three-dimensional arrow.
  • 3. Fundamentally, all navigation information can be changed by a user. However, it is needless to say that for each navigation information, the default display dimensions and the default color are previously specified. For example, the landmark indicating a hospital is set by default to be displayed with the use of a plate-like three-dimensional icon having a given area and a given thickness with the design consisting of a white cross on a red background. However, it is possible for the user to change the foregoing setting to a plate-like three-dimensional icon with the design consisting of a red and white cross on a white background. Such setting data is inputted by the user with the use of the operating section 2. The inputted such setting data is further inputted to the image data creating section 405 via the image processing module 401. Then, the image data creating section 405 creates data for displaying various navigation information with the dimensions and the color corresponding to each piece of the navigation information.
  • 4. For the navigation information, three classifications of warning information, guidance information, and additional information are previously defined depending on urgency of display. Information of the definition is previously stored in the image data creating section 405. The warning information is information for warning the user of pullout of the lane, approximation of an obstruction (or possibility of crash) and the like. The guidance information is information or the like on a traveling direction to the destination, a park availability guide, and the landmark or a large facility along the road. The additional information is information on a medium and small sized facility, a shop and the like along the road. For the respective classification, priorities are previously determined. Information on the priorities is also previously stored in the image data creating section 405. Determination is made to which classification of the foregoing classifications the navigation information corresponds. In accordance with the order corresponding to the classification, the data for compositing and displaying each piece of the navigation information in a corresponding position in the real picture is created by the image data creating section 405.
  • 5. The navigation information other than the warning information is displayed on and after when the current position of the movable body to a warning target in the real picture (or the real landscape in the case of HUD) enters within a given distance.
  • 6. Of the navigation information, the navigation information other than the three-dimensional arrow for guiding the traveling direction and graphic display such as virtual road painting is not displayed in the region under a virtual vanishing point or a virtual horizon line in the real picture, but is displayed in the region above the virtual vanishing point or the virtual horizon line.
  • 7. The navigation information is composited and displayed in a determined position in the captured real picture as a three-dimensional icon that rotates centering on a vertical axis or a horizontal axis. When multiple rotatable three-dimensional icons are displayed in one screen, each of the multiple rotatable three-dimensional icons is displayed while being rotated at the phase different from one another or at the rotational speed different from one another.
  • 8. The navigation information is displayed as a translucent icon through which the real picture can be transparently viewed.
  • 9. Of the navigation information, character information (information displayed as a character) is not displayed in the real picture, but displayed in a display region preset on the periphery of the real picture.
Based on the data created by the image data creating section 405, the picture display section 5 displays the picture (image) in which the navigation information is composited in the appropriate position of the real picture on the screen such as a liquid crystal display panel.
The controlling section 6, for example, controls outputting the alarm or the like corresponding to the foregoing alarm output instruction, controls voice output corresponding to the analysis result by the control output module, controls a brake, and controls steering by, for example, controlling operations of each servo motor system provided for adjusting the control amount thereof.
Next, a description will be given of operations of the movable-body navigation information display unit.
In the movable-body navigation information display unit, regarding the road map data of the geography in the vicinity of the vehicle including the current position of the vehicle, the image data creating section 405 creates a road shape model for a road assumed to be captured from the current position, based on current position data detected by the GPS sensor 102, the INS sensor 103, and the positioning processing module. In addition, road shape data as image data of a road included in a landscape in the traveling direction is extracted from the real picture, based on, for example, white line image data of a road traveling sectional line and the like.
Subsequently, the image data creating section 405 matches the road shape data with data of the road shape model data. Then, the image data creating section 405 estimates posture data of the CCD camera 101 (or posture data of the vehicle) to the road in the landscape as an object captured by the CCD camera 101.
Next, based on the posture data, an appropriate display position in the captured real picture of the navigation information read according to the current position of the movable body is determined. Then, image data that enables displaying an image in which the read navigation information is composited in the determined appropriate position in the captured real picture is created.
As above, the image data creating section 405 determines what position in the real picture the navigation information is appropriately composited by matching the data of the road shape model with the road shape data. Based on the image data created as above, the picture display section 5 can display the image in which the navigation information such as the route guide, the vehicle position, and the map information is appropriately composited in the appropriate position in the real picture of the road ahead of the movable body or in the real landscape seen through the windshield. In the result, according to the movable-body navigation information display unit of this embodiment, it is possible to display the picture with which the relation between the navigation information and the real picture/real landscape can be intuitively and accurately recognized by the driver.
The foregoing navigation information presents at least one of a course guide for a route to arrive at the destination of the movable body, the vehicle position, the driving lane of the vehicle, and the building as an eyemark for the driver of the movable body to check the course guide or the vehicle position. When the navigation information is information of a character, a symbol, or a number, it is desirable that the information is iconified, and the icon image is composited and displayed in the captured real picture.
Further, the image data creating section 405 expresses the navigation information as a virtual object in the three-dimensional augmented reality space, and assigns a corresponding position in the road shape data converted to the two-dimensional characteristics space to the expressed navigation information based on already obtained posture data and the like. Thereby, the image data creating section 405 composites the image of the navigation information as a virtual object in an appropriate position of the real picture. For example, the navigation information composed of a character, a symbol, or a number for a building as an eyemark of the course guide can be expressed visually and intuitively, even when the building as the eyemark is hidden by the building or the like in front of the building as the eyemark or is hidden by a curved road.
Further, the image data creating section 405 converts the road shape data to image data of the perspective two-dimensional characteristics space, and converts the data of the road shape model to image data of the perspective two-dimensional characteristics space. The image data creating section 405 matches the foregoing 2 pieces of two-dimensional data each other in the two-dimensional characteristics space, and estimates the posture data of the in-vehicle camera or the movable body to the road surface as an object. Since matching the road shape data with the data of the road shape model performed for estimating the posture data is made with the use of 2 pieces of the two-dimensional data in the pseudo-three-dimensional two-dimensional characteristics space, the matching process is simplified and the speed thereof is accelerated.
Further, when the road shape model is created, modeling is performed by considering the road structure in which the one-way grade of the traveling road surface in the curved road is set to change according to the horizontal curvature of the curved road. Thereby, even when the movable body such as a motor vehicle is traveling on a multilane road, the road shape of the multi-lane road can be accurately comprehended. Consequently, it is possible to composite and display the navigation information in an appropriate position accurately corresponding to the road shape of such a multilane road.
Further, in matching the data of the road shape model with the road shape data, the posture data of the movable body may be obtained as follows. That is, a road look-up table is used (RSL, reference: “Research on extracting and simultaneous tracing multiple movable bodies by visual motion interpretation of an in-vehicle camera,” H U Zhencheng, doctoral thesis, Graduate school of Science and Technology, National University Corporation Kumamoto University). The existence probability of a road white line included in the landscape is obtained from the real picture, and thereby the RSL value is calculated. The posture data of the movable body is obtained so that the evaluation value based on the RSL value becomes the maximum. Thereby, it is possible to always extract accurate road shape data without being adversely influenced by various environmental factors such as weather change and shadow or dirt of the road surface. In the result, an accurate posture data can be estimated by using the extracted data.
Further, it is also possible to display the image in which the navigation information is composited in the determined appropriate position in the captured real picture on a given display screen of the display unit such as a liquid crystal display panel for car navigation set in the almost central section of the dashboard or the like.
Further, it is also possible that the image in which the read navigation information is composited in the appropriate position in the captured real picture determined by the foregoing matching is projected and displayed on the internal surface of a transparent window in front of the driver seat by a display unit such as a so-called HUD projection device.
Further, instead of compositing the image of the navigation information in the captured real picture, it is possible to take the following method. That is, by using data of the position determined as the appropriate position to display the image of the navigation information based on the foregoing matching, the image of the navigation information is HUD-projected and displayed on the internal surface of the windshield in front of the driver seat corresponding to the foregoing position. Thereby, the image of the navigation information is composited and displayed in the landscape seen through the transparent window in front of the driver seat.
Next, a description will be hereinafter given of more specific examples of the movable-body navigation information display unit and the movable-body navigation information display method according to this embodiment.
FIG. 2 shows a relative position relation among three-dimensional vehicle coordinate system VCS (Xv, Yv, Zv), three-dimensional camera coordinate system CCS (Xc, Yc, Zc), and two-dimensional projected image coordinate system ICS (xi, yi). The origin of the three-dimensional vehicle coordinate system VCS is located at the midpoint of the central line of the vehicle rear wheels. The axis Zv is set on the central line, the axis Xv is set to be oriented leftward, and the axis Yv is set to be oriented upward, respectively. The origin of the three-dimensional camera coordinate system CCS is located at the central point of the lens of the CCD camera. The axis Zc is set to be overlapped on the optical axis of the camera. The two-dimensional projected image coordinate system ICS is located on the plane where Zc=fconst (plane where Zc=f).
The conversion relation from the camera coordinate system CCS to the image coordinate system ICS is orthogonal projection. Therefore, the relation can be described as the relational expression based on a matrix as shown in Formula 1 of FIG. 7. P is the coordinates [Xc, Yc, Zc, 1] in the CCS coordinate system, and p is the coordinates [xi, yi, 1] in the ICS coordinate system.
A is a 3×4 projection matrix, and can be generally decomposed as shown in Formula 2 of FIG. 7. K is called the parameter matrix inside the camera, and is determined by the image horizontal and vertical direction transformation ratio (Sx, Sy), the image central point (uo, vo), and the rotation transformation ratio Sθ. K is expressed as Formula 3 in FIG. 7.
The camera posture matrix M is called the parameter matrix outside the camera, and represents a conversion relation from a view point to the target model coordinate system. In general, the camera posture matrix M can be expressed as Formula 4 of FIG. 7 by three-dimensional translation and rotational conversion of a rigid body. R11 to R33 (all elements of R) represent the rotation parameters, and Tx, Ty, and Tz, (all elements of T) represent the translation parameters.
In general, the camera magnification can be approximated with 1. Therefore, based on formula 1 to formula 4, the restraint formula as shown in Formula 5 of FIG. 7 is established.
When 6 parameters of the rotation and the translation representing the camera posture are displayed by a posture vector, the projection relation between the image coordinate system and the vehicle coordinate system is expressed by the formula shown in Formula 6 of FIG. 7. That is, according to Formula 6, one pair of corresponding points in the 2D-3D space (p, P) determines one restriction formula as shown in Formula 6 of FIG. 7 to the camera posture data. Theoretically, such 6 pairs of corresponding points are enough to estimate the camera posture.
However, it is theoretically very difficult or not possible to accurately and surely estimate the depth of the three-dimensional space only by monochrome real image data of the landscape ahead of the vehicle that is captured by the simple monocular CCD camera. Therefore, in this example, a multilane road shape model is estimated from road map information by avoiding the matching in 2D-3D (matching of the two-dimensional space with the three-dimensional space). The estimated multilane road shape model is matched with multilane road shape data extracted from real picture data in the 2D-2D characteristics space. Then, the camera posture data is estimated by such matching in the two-dimensional space to the two-dimensional space. However, it is needless to say that the matching is not limited to the matching in the 2D-2D characteristics space. It is needless to say that, for example, it is also possible to perform estimation and matching as follows. That is, the depth data in the three-dimensional space is accurately and surely estimated from an information source other than the real image data of the landscape ahead of the vehicle, and matching of the two-dimensional space to the three-dimensional space is made by using such data. However, in that case, it is needless to say that in general, the data amount to be processed tends to become larger than that of the case of matching in the 2D-2D characteristics space.
FIG. 3 shows an example of a mapping with the use of dots and lines expressed by the road map data. In the road map data, three-dimensional position information such as the latitude, the longitude, and the altitude of a road segment generally called a node, a road name, a road class, the number of lanes, a crossing state and the like are recorded. The road width can be estimated based on the road class. Further, in general, a node position described in the map data is located on the road centerline. In general, the structure of the road is composed of a complex curved surface using the horizontal curvature and the vertical curvature.
FIG. 4 shows a road segment horizontal shape model obtained by approximation with the clothoid curved line. Such a road segment horizontal shape model can be modeled by using the mathematical formula as shown in Formula 7 of FIG. 7. c0 and c1 in Formula 7 respectively represent the change parameters of the initial curvature and the curvature of the horizontal curved line. nli represents the number of up lanes, nri represents the number of down lanes, and wi represents the average road width between segments. Li represents the segment length. By using the model, the road shape model in a given road position can be structured simply and in a very short time based on the map data.
In general, in the country adopting the left-hand traffic such as Japan, the traveling position of vehicles is not on the road centerline, but is often deviated leftward from the road centerline. By using information on the road position of the vehicle (the displacement amount from the origin of the vehicle coordinate system to the center of the road) and the direction (the displacement angle made by the Z axis direction of the vehicle coordinate system and the road horizontal tangent line), the road horizontal shape model of the road centerline corresponding to the fact that the actual traveling position is deviated to the one lane side can be converted to a new model based on the vehicle coordinate system VCS.
Assuming that the road surface within the visible distance (region assumed to be captured as the real picture) is flat, the projection formula of the road horizontal shape model as shown in Formula 8 of FIG. 7 is estimated from Formula 6 of FIG. 7 as the projection conversion formula. In Formula 8 of FIG. 7, the three-dimensional road shape is projected perspectively and two dimensionally as it were. Therefore, the 2D-3D matching having a possibility to result in tangled data processing as a matching method to estimate the camera posture can be simplified to the 2D-2D matching.
As above, in this example, the road shape model estimated from the road map data and the road shape data extracted from the real picture are 2D-2D matched in the two-dimensional characteristics space, and the optimal posture vector is estimated. FIG. 5 shows an example of the road horizontal shape model used for the 2D-2D matching.
In the actual road environment, it is often the case that a light irradiation state due to rays of the sun, artificial lighting and the like, a weather state and the like lead to large change of the brightness, the tint and the like of the road white line indicating the section of the lane. Therefore, in some cases, it is not possible to successfully and directly match the road shape model with the road shape data extracted from the real picture. Therefore, in this example, instead of the brightness value of the road white line, the existence probability thereof is picturized by using the concept of the road white line look-up table (RSL). Thereby, the high-speed and robust road shape matching can be realized.
As closer to the road white line, the higher the RSL value is. As a specific calculation method, first, the road white line, the sectional line, and a borderline prospect of the road region are extracted as a characteristics region, and the image is binarized (a pixel belonging to the characteristics region is 1, and the other pixels are 0). Then, by using Formula 9 shown in FIG. 7, the RSL value of each pixel is calculated. λ, x, and y are binarized pixel values. χ, i, and j are the kernel coefficient for the RSL. To inhibit noise, the kernel size is generally set to 5 or 7. Each coefficient is determined by the Gaussin distribution formula. The final evaluation formula of the camera posture estimation is shown in Formula 10 of FIG. 7. ησ is a set of two-dimensional projection points of the road horizontal shape model. By using Formula 10, it is possible to obtain the best RSL evaluation value for totally matching the road shape model created based on the road map data with the road shape data extracted from the real picture.
To obtain the optimal value of the posture vector σ of the 6 parameters, various local extreme value search methods can be used. Of those methods, in particular, the direct search algorithm by Hooke and Jeeves can be suitably adopted (R. Hooke and T. Jeeves, “Direct search solution of numerical and statistical problems,” Journal of the Association for Computing Machinery (ACM), pp. 212 to 229 (1961)).
The camera posture data obtained as above is used for data matching for determining the position in which the navigation information should be displayed. The camera posture data is also used for the next estimation as the feedback amount.
FIG. 6 is a flowchart showing a flow of a series of main processing including extracting the road shape date, creating the road shape model, estimating the camera posture parameter and the like in the central processing section.
First, the real picture data of the landscape including the road ahead of the vehicle took by the CCD camera is captured (S1).
Concurrently, in sync with captured real picture data, the data acquired by the GPS sensor and the INS sensor is captured (S2).
Next, a white line of the road traveling sectional line or the like and a sectional line region of a lane of a paved face or the like are extracted from the real picture data, and the RSL value is calculated (S3).
Further, the current position (absolute position) of the vehicle is obtained (detected) by so-called map matching, and the related map data corresponding to the current position is read out from information stored in the map data CD (S4).
The road horizontal shape model for the road in the landscape captured as the real picture is constructed (S5).
Based on the updated camera posture vector, the road horizontal shape model is projected in the perspective two-dimensional space (S6).
The RSL expression of the road picture is matched with the projected road horizontal shape model, and the evaluation value is obtained (S7).
Determination is made whether or not the obtained evaluation value is the maximum value (S8). If determining that the evaluation value is the maximum value (Y of S8), the posture vector then is outputted (S9), and the outputted value is feed backed as the search origin for the next estimation processing (S10). If determining that the evaluation value is not the maximum value (N of S8), the posture vector is updated by Hooke&Jeeves method (S11), and reevaluation is made (S11 to S6 through S8). This procedural loop is repeated until the maximum value is obtained (until Y of S8).
When it comes to the time to start processing of taking in the next data (Y of S12), the foregoing series of processing is executed again from the first step of capturing the data in the order similar to the foregoing order.
The matching is performed as above, and determination is made that what navigation is displayed in what position. Next, the image data creating section 405 creates the data for specifically displaying each piece of the navigation information. More specifically, creating such data is made based on the 9 types of the three-dimensional modeling rules 1 to 9 described above. In the result, the images shown as an example in FIG. 8 to FIG. 12 are displayed on the screen of the picture display section 5.
The landmark and the facility information are displayed with the three-dimensional icon expressing them schematically. In an example of FIG. 8, a three-dimensional icon 801 indicating existence of a “hospital” is composited and displayed in the vicinity of the hospital in the real picture. The three-dimensional icon 801 indicating the hospital has, for example, the design consisting of a white cross on a red background. The size (outer dimensions) and the tint thereof are set to the degree with which the visibility of the picture of the buildings and the road of the real picture as a background is not inhibited.
As shown in the enlarged view of FIGS. 9A and 9B, the three-dimensional icon 801 is displayed as if the icon 801 rotates at a given rotation speed centering on a vertical axis 802. When the three-dimensional icon 801 is oriented to the front face (FIG. 9A), a user can clearly recognize that the icon 801 indicates the “hospital.” When the icon 801 is turned 90 deg from the state oriented to the front face, as shown in the enlarged view of FIG. 9B, only an extremely thin area of a thickness thereof 803 is displayed. Therefore, the icon does not inhibit visibility of the buildings, pedestrians, obstructions and the like in the real picture existing (or being displayed) behind the three-dimensional icon 801, and the user can clearly recognize the existence thereof.
As described above, fundamentally, it is possible for the user to change the display dimensions and the display color (use of color and tint) for all navigation information such as the three-dimensional icon 801.
Indicative information such as the traveling direction is displayed with a three-dimensional arrow. In the example of FIG. 8, for example, on the road lane on which the user's vehicle currently goes straight ahead, a virtual road painting 804 indicating the limiting speed, a three-dimensional arrow 805 indicating the direction to travel, and virtual road paintings 806, 807 indicating the lane are composited and displayed in the real picture of the road. However, character information or the like for displaying the destination is not displayed to overlap on the region under a virtual vanishing point or the virtual horizon line in the real picture, that is, more specifically, on the road real picture.
In the example of FIG. 8, a destination display 808 mainly composed of the character information is displayed in the region of the real picture in which almost only the sky is shown that is located in the vicinity of the upper end of an entire real picture 809 (entire screen in the case of FIG. 8). As above, the image mainly composed of the character information such as the destination display 808 is not displayed in the region under the virtual vanishing point or the virtual horizon line in the real picture or the real landscape in which roads, pedestrians, and buildings exist with high probability. Thereby, it is possible to inhibit the real picture in which roads, pedestrians, and buildings exist or the real landscape from being hidden and invisible.
Further, as described above, the navigation information is composited and displayed in the determined position in the captured real picture as the three-dimensional icon that rotates centering on the vertical axis (or the horizontal axis). In the case that a plurality of three-dimensional icons are displayed on one screen, each of the plurality of three-dimensional icons may be rotated at the phase different from e one another or at the rotational speed different from one another. In an example shown in FIG. 12, a first three-dimensional icon 121 and a second three-dimensional icon 122 are displayed so that the icons are rotated in the same direction and at the same rotational speed but at the phase different from each other by about 90 deg.
By setting the phase different from each other by about 90 deg, when the first three-dimensional icon 121 is oriented to the front face, the second three-dimensional icon 122 is oriented to the side face, and only the first three-dimensional icon 121 is substantially viewed by the user. Meanwhile, when the second three-dimensional icon 122 is oriented to the front face, the first three-dimensional icon 121 is oriented to the side face, and only the second three-dimensional icon 122 is substantially viewed by the user. By limiting the number of the three-dimensional icons that can be viewed by the user at once as above, it is possible to avoid tangle and visibility deterioration of the display screen resulting from a state that the plurality of three-dimensional icons are seen in parallel at once. In addition to setting different phases, for example, it is possible to set the rotational speed of the plurality of three-dimensional icons different from each other.
Otherwise, it is possible to adopt the method as shown in an example of FIG. 10. In the method, of the navigation information, road name information 811 and limiting speed information 812 set to be displayed as the character information is displayed not in the real picture but in a character information exclusive display region 810 previously provided on the periphery of the real picture 809.
Otherwise, it is possible to classify the navigation information into three types of the warning information, the guidance information, and the additional information depending on urgency of display. In the order corresponding to the classification, each piece of the navigation information can be composited and displayed in a relevant position in the real picture.
That is, for general navigation information such as a land mark and small to medium facilities other than the information necessary for urgently issuing a warning to the user such as warning information (that is, navigation information with the urgency lower than that of the information such as the warning information), display is made on and after when the current position of the movable body to the warning target in the real picture (or real landscape in the case of HUD) enters within a given distance. A description thereof will be given more specifically with reference to an example shown in FIG. 11. At this point, 3 three-dimensional icons, 821 (representing a “hotel”), 822 (representing a gas station), 823 (representing a convenience store) are displayed. All thereof starts to be displayed when each real substance enters within a predetermined distance such as 100 m to the vehicle of the user. Each icon is not displayed when the real substance is located apart from the vehicle farther than the predetermined distance. In the example of FIG. 11, the farthest three-dimensional icon 822 just enters within the given distance and starts to be displayed.
Further, as simple road map information 824 displayed in the lower left of the real picture (screen) of FIG. 11, the image of the navigation information is displayed as a translucent icon so that the real picture as a background can be viewed translucently. It is needless to say that not only the simple road map information 824 but also other information can be displayed as a translucent icon.
As hereinbefore described in details, according to the movable-body navigation information display unit and the movable-body navigation information display means of this embodiment, the image of the navigation information can be accurately composited in the appropriate position in the real picture of the road ahead of the movable body or in the real landscape seen thorough the windshield. In the result, the picture can be displayed so that the driver can intuitively and accurately recognize the relation between the navigation information and the real picture or the real landscape. In addition, it is possible to avoid the state that visibility of the real picture (or real landscape) is inhibited by the image of the navigation information due to the composition of the navigation information by the various methods described above. Consequently, the visibility can be favorable over the entire display image.

Claims (6)

1. A movable-body navigation information display method comprising the steps of:
detecting a current position of a movable body, and capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body;
reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data;
creating a road shape model of a road assumed to be captured from the current position, based on the current position of the movable body and the road map data;
extracting road shape data as image data of the road included in the landscape from the real picture;
converting the road shape data to image data in a perspective two-dimensional characteristics space and the data of the road shape model to image data in the perspective two-dimensional characteristics space;
obtaining an existence probability of a road white line included in the landscape from the real picture and calculating an RSL value by using a road look-up table (RSL);
matching the two-dimensional road shape data with the two-dimensional data of the road shape model in the two-dimensional characteristics space so that an evaluation value based on the RSL value becomes the maximum, and thereby estimating the posture data of the in-vehicle camera or the movable body to the road as the object;
determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data; and
displaying an image in which the read navigation information is composited with display dimensions and tone set by a user in the determined position in the captured real picture.
2. The movable-body navigation information display method according to claim 1, wherein in creating the road shape model, the road shape model of a multilane road is created by performing modeling while considering a road structure in which one-way grade of a traveling road surface in a curved road is set to change according to a horizontal curvature of the curved road.
3. The movable-body navigation information display method according to claim 1, wherein the navigation information is at least one of a course guide of a route to reach a destination of the movable body, a position of a vehicle, a driving lane of the vehicle, and a building as an eyemark for a driver of the movable body to check the course guide or the position of the vehicle.
4. A movable-body navigation information display unit comprising:
a current position detecting means for detecting a current position of a movable body;
a capturing means for capturing a real picture with an object of a landscape including a road in a traveling direction of the movable body by an in-vehicle camera set in the movable body;
a data processing means for reading navigation information on travel of the movable body corresponding to the detected current position of the movable body from navigation information previously stored in association with road map data, creating a road shape model of a road assumed to be captured from the current position based on the current position of the movable body and the road map data, extracting road shape data as image data of the road included in the landscape from the real picture, converting the road shape data to image data in a perspective two-dimensional characteristics space and the data of the road shape model to image data in the perspective two-dimensional characteristics space, obtaining an existence probability of a road white line included in the landscape from the real picture and calculating an RSL value by using a road look-up table (RSL), matching the two-dimensional road shape data with the two dimensional data of the road shape model in the two-dimensional characteristics space so that an evaluation value based on the RSL value becomes the maximum and thereby estimating the posture data of the in-vehicle camera or the movable body to the road as the object, determining a display position in the captured real picture of the navigation information read according to the current position of the movable body based on the posture data, and outputting data for displaying an image in which the read navigation information is composited as an image displayed with display dimensions and tone set by a user in the determined position in the captured real picture; and
an image display means for compositing and displaying the read navigation information as the image displayed with the display dimensions and the tone set by the user in the determined position of the captured real picture based on the data outputted from the data processing means.
5. The movable-body navigation information display unit according to claim 4, wherein in creating the road shape model, the data processing means creates the road shape model of a multilane road by performing modeling while considering a road structure in which one-way grade of a traveling road surface in a curved road is set to change according to a horizontal curvature of the curved road.
6. The movable-body navigation information display unit according to claim 4, wherein the navigation information is at least one of a course guide of a route to reach a destination of the movable body, a position of a vehicle, a driving lane of the vehicle, a building as an eyemark for a driver of the movable body to check the course guide or the position of the vehicle.
US11/663,909 2004-09-28 2005-09-27 Movable-body navigation information display method and movable-body navigation information display unit Expired - Fee Related US8195386B2 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
JP2004282190 2004-09-28
JP2004-282190 2004-09-28
PCT/JP2005/017715 WO2006035755A1 (en) 2004-09-28 2005-09-27 Method for displaying movable-body navigation information and device for displaying movable-body navigation information

Publications (2)

Publication Number Publication Date
US20080195315A1 US20080195315A1 (en) 2008-08-14
US8195386B2 true US8195386B2 (en) 2012-06-05

Family

ID=36118900

Family Applications (1)

Application Number Title Priority Date Filing Date
US11/663,909 Expired - Fee Related US8195386B2 (en) 2004-09-28 2005-09-27 Movable-body navigation information display method and movable-body navigation information display unit

Country Status (3)

Country Link
US (1) US8195386B2 (en)
JP (1) JP4696248B2 (en)
WO (1) WO2006035755A1 (en)

Cited By (35)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100030462A1 (en) * 2007-02-15 2010-02-04 Pioneer Corporation Display control apparatus, display control method, display control program, and recording medium
US20100280747A1 (en) * 2008-05-02 2010-11-04 Olaf Achthoven Navigation device and method for displaying map information
US20110054783A1 (en) * 2008-01-28 2011-03-03 Geo Technical Laboratory Co., Ltd. Data structure of route guidance database
US20120083923A1 (en) * 2009-06-01 2012-04-05 Kosei Matsumoto Robot control system, robot control terminal, and robot control method
US20120194554A1 (en) * 2011-01-28 2012-08-02 Akihiko Kaino Information processing device, alarm method, and program
US20120296561A1 (en) * 2011-05-16 2012-11-22 Samsung Electronics Co., Ltd. User interface method for terminal for vehicle and apparatus thereof
US20120307002A1 (en) * 2007-02-06 2012-12-06 Meadow William D Methods and apparatus for generating a continuum of image data
US20130124084A1 (en) * 2011-11-15 2013-05-16 Jaehong SEO Mobile terminal and method of controlling the same
US20130158849A1 (en) * 2011-12-15 2013-06-20 Masao MAURA Evaluation indication system, evaluation indication method and computer-readable storage medium
US20130304383A1 (en) * 2012-05-11 2013-11-14 Honeywell International Inc. Systems and methods for landmark selection for navigation
US20140210822A1 (en) * 2013-01-31 2014-07-31 Samsung Electronics Co. Ltd. Apparatus and method for compass intelligent lighting for user interfaces
US8890866B2 (en) 2004-08-31 2014-11-18 Visual Real Estate, Inc. Method and apparatus of providing street view data of a comparable real estate property
US20140372020A1 (en) * 2013-06-13 2014-12-18 Gideon Stein Vision augmented navigation
US20150019128A1 (en) * 2003-06-19 2015-01-15 Here Global B.V. Method and System for Representing Traffic Signals in a Road Network Database
US20150073663A1 (en) * 2013-09-12 2015-03-12 Volvo Car Corporation Manoeuver generation for automated driving
KR20150062418A (en) * 2013-11-29 2015-06-08 (주) 유라시아디지털문화유산연구소 System for displaying virtual image matched with real street figure
US20150226568A1 (en) * 2014-02-11 2015-08-13 Hyundai Motor Company Apparatus and method of providing road guidance based on augmented reality head-up display
US9304003B1 (en) 2015-03-18 2016-04-05 Microsoft Technology Licensing, Llc Augmented reality navigation
US9341485B1 (en) 2003-06-19 2016-05-17 Here Global B.V. Method and apparatus for representing road intersections
US9395192B1 (en) * 2013-12-20 2016-07-19 Google Inc. Methods and systems for road and lane boundary tracing
US9435648B2 (en) 2012-12-17 2016-09-06 Industrial Technology Research Institute Map matching device, system and method
US9652980B2 (en) 2008-04-24 2017-05-16 GM Global Technology Operations LLC Enhanced clear path detection in the presence of traffic infrastructure indicator
US9852357B2 (en) 2008-04-24 2017-12-26 GM Global Technology Operations LLC Clear path detection using an example-based approach
US9909894B2 (en) 2016-01-07 2018-03-06 Here Global B.V. Componentized junction models
US20180341822A1 (en) * 2017-05-26 2018-11-29 Dura Operating, Llc Method and system for classifying objects in a perception scene graph by using a scene-detection-schema
US10234294B2 (en) 2016-04-01 2019-03-19 Here Global B.V. Road geometry matching with componentized junction models
US20190138261A1 (en) * 2016-06-22 2019-05-09 Sony Corporation Information processing device, information processing method, and program
US10488215B1 (en) 2018-10-26 2019-11-26 Phiar Technologies, Inc. Augmented reality interface for navigation assistance
US10495476B1 (en) 2018-09-27 2019-12-03 Phiar Technologies, Inc. Augmented reality navigation systems and methods
US20190384294A1 (en) * 2015-02-10 2019-12-19 Mobileye Vision Technologies Ltd. Crowd sourcing data for autonomous vehicle navigation
US10573183B1 (en) 2018-09-27 2020-02-25 Phiar Technologies, Inc. Mobile real-time driving safety systems and methods
US10663315B2 (en) * 2015-02-09 2020-05-26 Denso Corporation Vehicle display control device and vehicle display control method
US20220111840A1 (en) * 2019-01-10 2022-04-14 Jaguar Land Rover Limited A control system, system and method for providing assistance to an occupant of a vehicle
US11448518B2 (en) 2018-09-27 2022-09-20 Phiar Technologies, Inc. Augmented reality navigational overlay
US11499834B2 (en) * 2019-03-07 2022-11-15 Mobileye Vision Technologies Ltd. Aligning road information for navigation

Families Citing this family (102)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8108142B2 (en) * 2005-01-26 2012-01-31 Volkswagen Ag 3D navigation system for motor vehicles
DE102005018082A1 (en) * 2005-04-19 2006-10-26 Robert Bosch Gmbh Method for the three-dimensional representation of a digital road map
US8423292B2 (en) * 2008-08-19 2013-04-16 Tomtom International B.V. Navigation device with camera-info
NZ564319A (en) 2005-06-06 2009-05-31 Tomtom Int Bv Navigation device with camera-info
US7728869B2 (en) * 2005-06-14 2010-06-01 Lg Electronics Inc. Matching camera-photographed image with map data in portable terminal and travel route guidance method
KR100657962B1 (en) * 2005-06-21 2006-12-14 삼성전자주식회사 Apparatus and method for displaying 3-dimension graphics
JP4246195B2 (en) * 2005-11-01 2009-04-02 パナソニック株式会社 Car navigation system
JP4935145B2 (en) * 2006-03-29 2012-05-23 株式会社デンソー Car navigation system
JP2008039596A (en) * 2006-08-07 2008-02-21 Pioneer Electronic Corp System, method, program for providing information and memory medium
JP4231884B2 (en) * 2006-09-27 2009-03-04 株式会社デンソーアイティーラボラトリ Gaze object detection device and gaze object detection method
US7990394B2 (en) 2007-05-25 2011-08-02 Google Inc. Viewing and navigating within panoramic images, and applications thereof
JP2009036726A (en) * 2007-08-03 2009-02-19 Yahoo Japan Corp Method and system for displaying search path
US20090089705A1 (en) * 2007-09-27 2009-04-02 Microsoft Corporation Virtual object navigation
JP2009163504A (en) * 2008-01-07 2009-07-23 Panasonic Corp Image deformation method and the like
JP4986883B2 (en) * 2008-02-27 2012-07-25 三菱電機株式会社 Orientation device, orientation method and orientation program
US8452053B2 (en) * 2008-04-24 2013-05-28 GM Global Technology Operations LLC Pixel-based texture-rich clear path detection
US8634593B2 (en) * 2008-04-24 2014-01-21 GM Global Technology Operations LLC Pixel-based texture-less clear path detection
CN102016929A (en) * 2008-05-29 2011-04-13 通腾科技股份有限公司 Generating a display image
JP5223502B2 (en) * 2008-07-03 2013-06-26 ヤマハ株式会社 Direction tracking display device, direction tracking display method, and direction tracking display program
DE102008045994A1 (en) * 2008-09-05 2010-03-11 Volkswagen Ag Method and device for displaying information in a vehicle
JP5372157B2 (en) * 2008-09-17 2013-12-18 ノキア コーポレイション User interface for augmented reality
US20100085350A1 (en) * 2008-10-02 2010-04-08 Microsoft Corporation Oblique display with additional detail
JP2010151658A (en) * 2008-12-25 2010-07-08 Zhencheng Hu Apparatus and method for measuring position of mobile object
JP5161760B2 (en) * 2008-12-26 2013-03-13 株式会社東芝 In-vehicle display system and display method
DE102009006471A1 (en) * 2009-01-28 2010-09-02 Audi Ag Method for operating a navigation device of a motor vehicle and motor vehicle therefor
US20100198506A1 (en) * 2009-02-03 2010-08-05 Robert Steven Neilhouse Street and landmark name(s) and/or turning indicators superimposed on user's field of vision with dynamic moving capabilities
JP5402160B2 (en) * 2009-03-31 2014-01-29 沖電気工業株式会社 Map display device, map display method and program
US9477368B1 (en) 2009-03-31 2016-10-25 Google Inc. System and method of indicating the distance or the surface of an image of a geographical object
US20110098910A1 (en) * 2009-10-22 2011-04-28 Nokia Corporation Method and apparatus for intelligent guidance using markers
JP2011128838A (en) * 2009-12-17 2011-06-30 Panasonic Corp Image display device
KR101001842B1 (en) * 2010-01-28 2010-12-17 팅크웨어(주) Navigation device for a vehicle and method for inducing right position of blackbox device of a navigation system
CN102713948A (en) * 2010-02-01 2012-10-03 日本电气株式会社 Portable terminal, action history depiction method, and action history depiction system
JP5136671B2 (en) * 2010-05-13 2013-02-06 株式会社デンソー Map display device
JP5413516B2 (en) * 2010-08-19 2014-02-12 日産自動車株式会社 Three-dimensional object detection apparatus and three-dimensional object detection method
JP2012068481A (en) * 2010-09-24 2012-04-05 Asia Air Survey Co Ltd Augmented reality expression system and method
US10331864B1 (en) 2010-12-27 2019-06-25 Mlb Advanced Media, L.P. Watermarking systems and methods
KR101708207B1 (en) 2011-01-10 2017-02-20 삼성전자주식회사 Apparatus and method for providing user's route information in mobile communication system
DE102011006347B4 (en) * 2011-03-29 2023-02-09 Bayerische Motoren Werke Aktiengesellschaft Process for outputting graphical driving instructions
JP5521134B2 (en) * 2011-08-12 2014-06-11 新日鉄住金ソリューションズ株式会社 Information processing system, information processing method, and program
KR101407670B1 (en) * 2011-09-15 2014-06-16 주식회사 팬택 Mobile terminal, server and method for forming communication channel using augmented reality
US9177204B1 (en) * 2011-09-28 2015-11-03 Rockwell Collins, Inc. Spectrally enhanced vision system for low visibility operations
JP5708449B2 (en) * 2011-11-08 2015-04-30 アイシン・エィ・ダブリュ株式会社 Lane guidance display system, method and program
JP5735658B2 (en) * 2011-12-15 2015-06-17 パイオニア株式会社 Display device and display method
JP5910180B2 (en) * 2012-03-06 2016-04-27 日産自動車株式会社 Moving object position and orientation estimation apparatus and method
JP6000780B2 (en) * 2012-09-21 2016-10-05 オリンパス株式会社 Imaging device
US20140095294A1 (en) * 2012-09-28 2014-04-03 Cornelius V. Vick Mechanism for facilitating context-aware broadcast and virtual visualization of advertisements
CN103868517A (en) * 2012-12-10 2014-06-18 深圳富泰宏精密工业有限公司 Indoor positioning system and method
US20140184643A1 (en) * 2012-12-27 2014-07-03 Caterpillar Inc. Augmented Reality Worksite
US20140232746A1 (en) * 2013-02-21 2014-08-21 Hyundai Motor Company Three dimensional augmented reality display apparatus and method using eye tracking
JP2014220604A (en) * 2013-05-07 2014-11-20 三菱電機株式会社 Photographing position information display device
JP6236954B2 (en) * 2013-07-23 2017-11-29 アイシン・エィ・ダブリュ株式会社 Driving support system, method and program
FR3011923B1 (en) * 2013-10-14 2017-05-26 Renault Sas METHOD FOR GUIDING A USER OF A MOTOR VEHICLE, CORRESPONDING SYSTEM AND MOTOR VEHICLE
KR101519277B1 (en) * 2013-12-24 2015-05-11 현대자동차주식회사 Apparatus and Method for Recognize of Drive Way of Vehicle
JP6376807B2 (en) * 2014-04-02 2018-08-22 キヤノン株式会社 Display device, display control method, and program
US9858720B2 (en) 2014-07-25 2018-01-02 Microsoft Technology Licensing, Llc Three-dimensional mixed-reality viewport
US10451875B2 (en) 2014-07-25 2019-10-22 Microsoft Technology Licensing, Llc Smart transparency for virtual objects
US10311638B2 (en) 2014-07-25 2019-06-04 Microsoft Technology Licensing, Llc Anti-trip when immersed in a virtual reality environment
US10416760B2 (en) 2014-07-25 2019-09-17 Microsoft Technology Licensing, Llc Gaze-based object placement within a virtual reality environment
US9904055B2 (en) 2014-07-25 2018-02-27 Microsoft Technology Licensing, Llc Smart placement of virtual objects to stay in the field of view of a head mounted display
US9766460B2 (en) * 2014-07-25 2017-09-19 Microsoft Technology Licensing, Llc Ground plane adjustment in a virtual reality environment
US9865089B2 (en) 2014-07-25 2018-01-09 Microsoft Technology Licensing, Llc Virtual reality environment with real world objects
US11370422B2 (en) * 2015-02-12 2022-06-28 Honda Research Institute Europe Gmbh Method and system in a vehicle for improving prediction results of an advantageous driver assistant system
CN104833367A (en) * 2015-05-11 2015-08-12 京东方科技集团股份有限公司 Vehicle-mounted projection system
JP6696149B2 (en) * 2015-10-29 2020-05-20 富士通株式会社 Image generation method, image generation program, information processing device, and display control method
US10217283B2 (en) 2015-12-17 2019-02-26 Google Llc Navigation through multidimensional images spaces
KR101916993B1 (en) * 2015-12-24 2018-11-08 엘지전자 주식회사 Display apparatus for vehicle and control method thereof
US10935789B2 (en) * 2016-03-31 2021-03-02 Honda Motor Co., Ltd. Image display apparatus and image display method
KR20180070198A (en) * 2016-12-16 2018-06-26 현대자동차주식회사 Vehicle and controlling method of vehicle
US10839203B1 (en) 2016-12-27 2020-11-17 Amazon Technologies, Inc. Recognizing and tracking poses using digital imagery captured from multiple fields of view
JP6930120B2 (en) * 2017-02-02 2021-09-01 株式会社リコー Display device, mobile device and display method.
US9992461B1 (en) * 2017-02-08 2018-06-05 Hyundai Motor Company Projection orientation correction system for vehicle
US10977953B2 (en) * 2017-02-17 2021-04-13 The Charles Stark Draper Laboratory, Inc. Probabilistic landmark navigation (PLN) system
US10339812B2 (en) * 2017-03-02 2019-07-02 Denso International America, Inc. Surrounding view camera blockage detection
JP6759132B2 (en) * 2017-03-10 2020-09-23 アルパイン株式会社 Head-up display device and display control method
JP7151073B2 (en) * 2017-03-31 2022-10-12 株式会社アイシン Display device and computer program
KR102427854B1 (en) 2017-09-25 2022-08-01 삼성전자주식회사 Method and apparatus for rendering image
US11232294B1 (en) 2017-09-27 2022-01-25 Amazon Technologies, Inc. Generating tracklets from digital imagery
KR102434580B1 (en) * 2017-11-09 2022-08-22 삼성전자주식회사 Method and apparatus of dispalying virtual route
US11511627B2 (en) 2017-11-17 2022-11-29 Aisin Corporation Display device and computer program
US20190340797A1 (en) * 2018-05-03 2019-11-07 Visteon Global Technologies, Inc. System and method for a dynamic map display with actual images
CN108665556B (en) * 2018-05-20 2024-03-01 福州市极化律网络科技有限公司 Road indication display method based on mixed reality and storage medium
US11482045B1 (en) 2018-06-28 2022-10-25 Amazon Technologies, Inc. Associating events with actors using digital imagery and machine learning
US11468698B1 (en) 2018-06-28 2022-10-11 Amazon Technologies, Inc. Associating events with actors using digital imagery and machine learning
US11468681B1 (en) 2018-06-28 2022-10-11 Amazon Technologies, Inc. Associating events with actors using digital imagery and machine learning
KR101957896B1 (en) * 2018-06-29 2019-07-04 한화시스템(주) integrated image and situation display system for remote control and method of displaying the same
JP1632766S (en) * 2018-09-10 2019-06-03
KR20200029785A (en) * 2018-09-11 2020-03-19 삼성전자주식회사 Localization method and apparatus of displaying virtual object in augmented reality
KR102559203B1 (en) 2018-10-01 2023-07-25 삼성전자주식회사 Method and apparatus of outputting pose information
GB2580400B (en) * 2019-01-10 2021-11-03 Jaguar Land Rover Ltd A control system, system and method for providing assistance to an occupant of a vehicle
CN110779541B (en) * 2019-04-10 2021-11-23 北京嘀嘀无限科技发展有限公司 Display method and system of steering arrow
US20220335698A1 (en) * 2019-12-17 2022-10-20 Ashley SinHee Kim System and method for transforming mapping information to an illustrated map
CN113137970B (en) * 2020-01-20 2022-11-15 北京智慧图科技有限责任公司 Small program AR navigation system
US11443516B1 (en) 2020-04-06 2022-09-13 Amazon Technologies, Inc. Locally and globally locating actors by digital cameras and machine learning
US11398094B1 (en) * 2020-04-06 2022-07-26 Amazon Technologies, Inc. Locally and globally locating actors by digital cameras and machine learning
KR20220022340A (en) * 2020-08-18 2022-02-25 삼성전자주식회사 Device and method to visualize content
KR20220036456A (en) * 2020-09-15 2022-03-23 현대자동차주식회사 Apparatus for displaying information based on augmented reality
JP6976537B1 (en) * 2020-10-08 2021-12-08 株式会社Fronteo Information retrieval device, information retrieval method and information retrieval program
CN112562375A (en) * 2020-12-11 2021-03-26 奇瑞汽车股份有限公司 Augmented reality road name display method and system, mobile terminal and vehicle-mounted terminal
CN113516770A (en) * 2021-04-14 2021-10-19 孙继雄 Method for synthesizing road design drawing and aerial photography real-scene special effect
US20220388507A1 (en) * 2021-06-04 2022-12-08 Telenav, Inc. Vehicle system with mechanism for determining clear path and method of operation thereof
CN113554712A (en) * 2021-06-29 2021-10-26 北京百度网讯科技有限公司 Registration method and device of automatic driving vehicle, electronic equipment and vehicle
CN115171412A (en) * 2022-08-09 2022-10-11 阿波罗智联(北京)科技有限公司 Method, system and device for displaying vehicle running state

Citations (47)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH10132598A (en) 1996-10-31 1998-05-22 Sony Corp Navigating method, navigation device and automobile
JPH10281794A (en) 1997-04-03 1998-10-23 Toyota Motor Corp Guidance display device for vehicle
JPH11281387A (en) 1998-03-31 1999-10-15 Sony Corp Navigation apparatus
JPH11304499A (en) 1998-04-22 1999-11-05 Matsushita Electric Ind Co Ltd Car navigation system
US6018697A (en) * 1995-12-26 2000-01-25 Aisin Aw Co., Ltd. Navigation system for vehicles
US6181302B1 (en) * 1996-04-24 2001-01-30 C. Macgill Lynde Marine navigation binoculars with virtual display superimposing real world image
US6182010B1 (en) * 1999-01-28 2001-01-30 International Business Machines Corporation Method and apparatus for displaying real-time visual information on an automobile pervasive computing client
US6199014B1 (en) * 1997-12-23 2001-03-06 Walker Digital, Llc System for providing driving directions with visual cues
US6208933B1 (en) * 1998-12-04 2001-03-27 Northrop Grumman Corporation Cartographic overlay on sensor video
US6285317B1 (en) * 1998-05-01 2001-09-04 Lucent Technologies Inc. Navigation system with three-dimensional display
JP2001331787A (en) 2000-05-19 2001-11-30 Toyota Central Res & Dev Lab Inc Road shape estimating device
US6356840B2 (en) * 1998-06-12 2002-03-12 Mitsubishi Denki Kabushiki Kaisha Navigation device with a three dimensional display
US6360168B1 (en) * 1999-09-14 2002-03-19 Alpine Electronics, Inc. Navigation apparatus
JP2002236027A (en) 2001-02-08 2002-08-23 Denso Corp Car navigation apparatus
US6470265B1 (en) * 1997-12-25 2002-10-22 Toyota Jidosha Kabushiki Kaisha Method and apparatus for processing digital map data
JP2003047050A (en) 2001-07-26 2003-02-14 Komatsu Ltd Communication apparatus of moving body
JP2003121167A (en) 2001-10-12 2003-04-23 Denso Corp Guidance image-generating apparatus, guidance image- displaying apparatus, navigation apparatus, and program
US20030080978A1 (en) * 2001-10-04 2003-05-01 Nassir Navab Augmented reality system
US20030160867A1 (en) * 2002-01-17 2003-08-28 Yasunori Ohto Information providing apparatus, information providing method, storage medium, and computer program
JP2003256997A (en) 2002-02-28 2003-09-12 Honda Motor Co Ltd Device for supporting enhancement of vehicle driving ability
US20030210228A1 (en) * 2000-02-25 2003-11-13 Ebersole John Franklin Augmented reality situational awareness system and method
US20040066376A1 (en) * 2000-07-18 2004-04-08 Max Donath Mobility assist device
JP2004150918A (en) 2002-10-30 2004-05-27 Shinichi Yamanaka Map displaying method
WO2004048895A1 (en) 2002-11-22 2004-06-10 Kumamoto Technology & Industry Foundation Moving body navigate information display method and moving body navigate information display device
US20040179104A1 (en) * 2003-03-10 2004-09-16 Charles Benton Augmented reality navigation system
US6885939B2 (en) * 2002-12-31 2005-04-26 Robert Bosch Gmbh System and method for advanced 3D visualization for mobile navigation units
US20050116964A1 (en) * 2003-11-19 2005-06-02 Canon Kabushiki Kaisha Image reproducing method and apparatus for displaying annotations on a real image in virtual space
US20050278111A1 (en) * 2004-05-20 2005-12-15 Koji Ujino Image processing system, method and apparatus for correlating position data with image data
US20060004512A1 (en) * 2004-06-30 2006-01-05 Herbst James M Method of operating a navigation system using images
US20060155466A1 (en) * 2005-01-12 2006-07-13 Sanyo Electric Co., Ltd. Mobile terminal with navigation function
US20060155467A1 (en) * 2002-08-07 2006-07-13 Horst Hortner Method and device for displaying navigational information for a vehicle
US20060190812A1 (en) * 2005-02-22 2006-08-24 Geovector Corporation Imaging systems including hyperlink associations
US20060238383A1 (en) * 2005-04-21 2006-10-26 Microsoft Corporation Virtual earth rooftop overlay and bounding
US20060262140A1 (en) * 2005-05-18 2006-11-23 Kujawa Gregory A Method and apparatus to facilitate visual augmentation of perceived reality
US20080147325A1 (en) * 2006-12-18 2008-06-19 Maassel Paul W Method and system for providing augmented reality
US20090005961A1 (en) * 2004-06-03 2009-01-01 Making Virtual Solid, L.L.C. En-Route Navigation Display Method and Apparatus Using Head-Up Display
US20090222203A1 (en) * 2006-03-07 2009-09-03 Robrt Bosch Gmbh Method and system for displaying navigation instructions
US7728869B2 (en) * 2005-06-14 2010-06-01 Lg Electronics Inc. Matching camera-photographed image with map data in portable terminal and travel route guidance method
US7737965B2 (en) * 2005-06-09 2010-06-15 Honeywell International Inc. Handheld synthetic vision device
US7751970B2 (en) * 2006-08-07 2010-07-06 Pioneer Corporation Information providing apparatus, information providing method, and computer product
US7783422B2 (en) * 2006-03-29 2010-08-24 Denso Corporation Navigation device and method of navigating vehicle
US20100268451A1 (en) * 2009-04-17 2010-10-21 Lg Electronics Inc. Method and apparatus for displaying image of mobile communication terminal
US7852355B2 (en) * 2003-11-10 2010-12-14 Siemens Aktiengesellschaft System and method for carrying out and visually displaying simulations in an augmented reality
US7966128B2 (en) * 2006-09-27 2011-06-21 Denso It Laboratory, Inc. Observed-object detection apparatus and observed-object detection method
US20110153198A1 (en) * 2009-12-21 2011-06-23 Navisus LLC Method for the display of navigation instructions using an augmented-reality concept
US7990394B2 (en) * 2007-05-25 2011-08-02 Google Inc. Viewing and navigating within panoramic images, and applications thereof
US20110228078A1 (en) * 2010-03-22 2011-09-22 Institute For Information Industry Real-time augmented reality device, real-time augmented reality method and computer storage medium thereof

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH07114692A (en) * 1993-10-15 1995-05-02 Fujitsu Ten Ltd Route guiding method
JP2001034160A (en) * 1999-05-14 2001-02-09 Denso Corp Map display device
JP3717045B2 (en) * 2000-01-19 2005-11-16 松下電器産業株式会社 Navigation device
JP4014447B2 (en) * 2002-05-23 2007-11-28 富士通テン株式会社 Navigation device

Patent Citations (48)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6018697A (en) * 1995-12-26 2000-01-25 Aisin Aw Co., Ltd. Navigation system for vehicles
US6181302B1 (en) * 1996-04-24 2001-01-30 C. Macgill Lynde Marine navigation binoculars with virtual display superimposing real world image
JPH10132598A (en) 1996-10-31 1998-05-22 Sony Corp Navigating method, navigation device and automobile
JPH10281794A (en) 1997-04-03 1998-10-23 Toyota Motor Corp Guidance display device for vehicle
US6199014B1 (en) * 1997-12-23 2001-03-06 Walker Digital, Llc System for providing driving directions with visual cues
US6470265B1 (en) * 1997-12-25 2002-10-22 Toyota Jidosha Kabushiki Kaisha Method and apparatus for processing digital map data
JPH11281387A (en) 1998-03-31 1999-10-15 Sony Corp Navigation apparatus
JPH11304499A (en) 1998-04-22 1999-11-05 Matsushita Electric Ind Co Ltd Car navigation system
US6285317B1 (en) * 1998-05-01 2001-09-04 Lucent Technologies Inc. Navigation system with three-dimensional display
US6356840B2 (en) * 1998-06-12 2002-03-12 Mitsubishi Denki Kabushiki Kaisha Navigation device with a three dimensional display
US6208933B1 (en) * 1998-12-04 2001-03-27 Northrop Grumman Corporation Cartographic overlay on sensor video
US6182010B1 (en) * 1999-01-28 2001-01-30 International Business Machines Corporation Method and apparatus for displaying real-time visual information on an automobile pervasive computing client
US6360168B1 (en) * 1999-09-14 2002-03-19 Alpine Electronics, Inc. Navigation apparatus
US20030210228A1 (en) * 2000-02-25 2003-11-13 Ebersole John Franklin Augmented reality situational awareness system and method
JP2001331787A (en) 2000-05-19 2001-11-30 Toyota Central Res & Dev Lab Inc Road shape estimating device
US6977630B1 (en) * 2000-07-18 2005-12-20 University Of Minnesota Mobility assist device
US20040066376A1 (en) * 2000-07-18 2004-04-08 Max Donath Mobility assist device
JP2002236027A (en) 2001-02-08 2002-08-23 Denso Corp Car navigation apparatus
JP2003047050A (en) 2001-07-26 2003-02-14 Komatsu Ltd Communication apparatus of moving body
US20030080978A1 (en) * 2001-10-04 2003-05-01 Nassir Navab Augmented reality system
JP2003121167A (en) 2001-10-12 2003-04-23 Denso Corp Guidance image-generating apparatus, guidance image- displaying apparatus, navigation apparatus, and program
US20030160867A1 (en) * 2002-01-17 2003-08-28 Yasunori Ohto Information providing apparatus, information providing method, storage medium, and computer program
JP2003256997A (en) 2002-02-28 2003-09-12 Honda Motor Co Ltd Device for supporting enhancement of vehicle driving ability
US20060155467A1 (en) * 2002-08-07 2006-07-13 Horst Hortner Method and device for displaying navigational information for a vehicle
JP2004150918A (en) 2002-10-30 2004-05-27 Shinichi Yamanaka Map displaying method
WO2004048895A1 (en) 2002-11-22 2004-06-10 Kumamoto Technology & Industry Foundation Moving body navigate information display method and moving body navigate information display device
US6885939B2 (en) * 2002-12-31 2005-04-26 Robert Bosch Gmbh System and method for advanced 3D visualization for mobile navigation units
US20040179104A1 (en) * 2003-03-10 2004-09-16 Charles Benton Augmented reality navigation system
US7852355B2 (en) * 2003-11-10 2010-12-14 Siemens Aktiengesellschaft System and method for carrying out and visually displaying simulations in an augmented reality
US20050116964A1 (en) * 2003-11-19 2005-06-02 Canon Kabushiki Kaisha Image reproducing method and apparatus for displaying annotations on a real image in virtual space
US20050278111A1 (en) * 2004-05-20 2005-12-15 Koji Ujino Image processing system, method and apparatus for correlating position data with image data
US20090005961A1 (en) * 2004-06-03 2009-01-01 Making Virtual Solid, L.L.C. En-Route Navigation Display Method and Apparatus Using Head-Up Display
US20060004512A1 (en) * 2004-06-30 2006-01-05 Herbst James M Method of operating a navigation system using images
US20060155466A1 (en) * 2005-01-12 2006-07-13 Sanyo Electric Co., Ltd. Mobile terminal with navigation function
US20060190812A1 (en) * 2005-02-22 2006-08-24 Geovector Corporation Imaging systems including hyperlink associations
US20060238383A1 (en) * 2005-04-21 2006-10-26 Microsoft Corporation Virtual earth rooftop overlay and bounding
US20060262140A1 (en) * 2005-05-18 2006-11-23 Kujawa Gregory A Method and apparatus to facilitate visual augmentation of perceived reality
US7737965B2 (en) * 2005-06-09 2010-06-15 Honeywell International Inc. Handheld synthetic vision device
US7728869B2 (en) * 2005-06-14 2010-06-01 Lg Electronics Inc. Matching camera-photographed image with map data in portable terminal and travel route guidance method
US20090222203A1 (en) * 2006-03-07 2009-09-03 Robrt Bosch Gmbh Method and system for displaying navigation instructions
US7783422B2 (en) * 2006-03-29 2010-08-24 Denso Corporation Navigation device and method of navigating vehicle
US7751970B2 (en) * 2006-08-07 2010-07-06 Pioneer Corporation Information providing apparatus, information providing method, and computer product
US7966128B2 (en) * 2006-09-27 2011-06-21 Denso It Laboratory, Inc. Observed-object detection apparatus and observed-object detection method
US20080147325A1 (en) * 2006-12-18 2008-06-19 Maassel Paul W Method and system for providing augmented reality
US7990394B2 (en) * 2007-05-25 2011-08-02 Google Inc. Viewing and navigating within panoramic images, and applications thereof
US20100268451A1 (en) * 2009-04-17 2010-10-21 Lg Electronics Inc. Method and apparatus for displaying image of mobile communication terminal
US20110153198A1 (en) * 2009-12-21 2011-06-23 Navisus LLC Method for the display of navigation instructions using an augmented-reality concept
US20110228078A1 (en) * 2010-03-22 2011-09-22 Institute For Information Industry Real-time augmented reality device, real-time augmented reality method and computer storage medium thereof

Cited By (72)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150019128A1 (en) * 2003-06-19 2015-01-15 Here Global B.V. Method and System for Representing Traffic Signals in a Road Network Database
US10309796B2 (en) 2003-06-19 2019-06-04 Here Global B.V. Method of representing road lanes
US9494434B2 (en) 2003-06-19 2016-11-15 Here Global B.V. Method of representing road lanes
US9341485B1 (en) 2003-06-19 2016-05-17 Here Global B.V. Method and apparatus for representing road intersections
US9285237B2 (en) * 2003-06-19 2016-03-15 Here Global B.V. Method and system for representing traffic signals in a road network database
US9384277B2 (en) 2004-08-31 2016-07-05 Visual Real Estate, Inc. Three dimensional image data models
US8890866B2 (en) 2004-08-31 2014-11-18 Visual Real Estate, Inc. Method and apparatus of providing street view data of a comparable real estate property
US9311396B2 (en) 2004-08-31 2016-04-12 Visual Real Estate, Inc. Method of providing street view data of a real estate property
USRE45264E1 (en) 2004-08-31 2014-12-02 Visual Real Estate, Inc. Methods and apparatus for generating three-dimensional image data models
US9311397B2 (en) 2004-08-31 2016-04-12 Visual Real Estates, Inc. Method and apparatus of providing street view data of a real estate property
US8902226B2 (en) 2004-08-31 2014-12-02 Visual Real Estate, Inc. Method for using drive-by image data to generate a valuation report of a selected real estate property
US9098870B2 (en) 2007-02-06 2015-08-04 Visual Real Estate, Inc. Internet-accessible real estate marketing street view system and method
US8554015B2 (en) * 2007-02-06 2013-10-08 William D. Meadow Methods and apparatus for generating a continuum of image data
US20120307002A1 (en) * 2007-02-06 2012-12-06 Meadow William D Methods and apparatus for generating a continuum of image data
US20100030462A1 (en) * 2007-02-15 2010-02-04 Pioneer Corporation Display control apparatus, display control method, display control program, and recording medium
US20110054783A1 (en) * 2008-01-28 2011-03-03 Geo Technical Laboratory Co., Ltd. Data structure of route guidance database
US8600654B2 (en) * 2008-01-28 2013-12-03 Geo Technical Laboratory Co., Ltd. Data structure of route guidance database
US9852357B2 (en) 2008-04-24 2017-12-26 GM Global Technology Operations LLC Clear path detection using an example-based approach
US9652980B2 (en) 2008-04-24 2017-05-16 GM Global Technology Operations LLC Enhanced clear path detection in the presence of traffic infrastructure indicator
US20100280747A1 (en) * 2008-05-02 2010-11-04 Olaf Achthoven Navigation device and method for displaying map information
US8775071B2 (en) * 2008-05-02 2014-07-08 Tomtom International B.V. Navigation device and method for displaying map information
US9242378B2 (en) * 2009-06-01 2016-01-26 Hitachi, Ltd. System and method for determing necessity of map data recreation in robot operation
US20120083923A1 (en) * 2009-06-01 2012-04-05 Kosei Matsumoto Robot control system, robot control terminal, and robot control method
US20120194554A1 (en) * 2011-01-28 2012-08-02 Akihiko Kaino Information processing device, alarm method, and program
US20140107911A1 (en) * 2011-05-16 2014-04-17 Samsung Electronics Co., Ltd. User interface method for terminal for vehicle and apparatus thereof
US8626427B2 (en) * 2011-05-16 2014-01-07 Samsung Electronics Co., Ltd. User interface method for terminal for vehicle and apparatus thereof
US20120296561A1 (en) * 2011-05-16 2012-11-22 Samsung Electronics Co., Ltd. User interface method for terminal for vehicle and apparatus thereof
US9318018B2 (en) * 2011-05-16 2016-04-19 Samsung Electronics Co., Ltd. User interface method for terminal for vehicle and apparatus thereof
US9723243B2 (en) 2011-05-16 2017-08-01 Samsung Electronics Co., Ltd. User interface method for terminal for vehicle and apparatus thereof
US9080892B2 (en) * 2011-11-15 2015-07-14 Lg Electronics Inc. Mobile terminal and method of controlling the same
US20130124084A1 (en) * 2011-11-15 2013-05-16 Jaehong SEO Mobile terminal and method of controlling the same
US20130158849A1 (en) * 2011-12-15 2013-06-20 Masao MAURA Evaluation indication system, evaluation indication method and computer-readable storage medium
US8862374B2 (en) * 2011-12-15 2014-10-14 Aisin Aw Co., Ltd. Evaluation indication system, evaluation indication method and computer-readable storage medium
US9037411B2 (en) * 2012-05-11 2015-05-19 Honeywell International Inc. Systems and methods for landmark selection for navigation
US20130304383A1 (en) * 2012-05-11 2013-11-14 Honeywell International Inc. Systems and methods for landmark selection for navigation
US9435648B2 (en) 2012-12-17 2016-09-06 Industrial Technology Research Institute Map matching device, system and method
US9041714B2 (en) * 2013-01-31 2015-05-26 Samsung Electronics Co., Ltd. Apparatus and method for compass intelligent lighting for user interfaces
US20140210822A1 (en) * 2013-01-31 2014-07-31 Samsung Electronics Co. Ltd. Apparatus and method for compass intelligent lighting for user interfaces
US9671243B2 (en) * 2013-06-13 2017-06-06 Mobileye Vision Technologies Ltd. Vision augmented navigation
US10533869B2 (en) * 2013-06-13 2020-01-14 Mobileye Vision Technologies Ltd. Vision augmented navigation
US11604076B2 (en) * 2013-06-13 2023-03-14 Mobileye Vision Technologies Ltd. Vision augmented navigation
US20200173803A1 (en) * 2013-06-13 2020-06-04 Mobileye Vision Technologies Ltd. Vision augmented navigation
US20140372020A1 (en) * 2013-06-13 2014-12-18 Gideon Stein Vision augmented navigation
US20150073663A1 (en) * 2013-09-12 2015-03-12 Volvo Car Corporation Manoeuver generation for automated driving
US9469296B2 (en) * 2013-09-12 2016-10-18 Volvo Car Corporation Manoeuver generation for automated driving
KR20150062418A (en) * 2013-11-29 2015-06-08 (주) 유라시아디지털문화유산연구소 System for displaying virtual image matched with real street figure
US9395192B1 (en) * 2013-12-20 2016-07-19 Google Inc. Methods and systems for road and lane boundary tracing
US20150226568A1 (en) * 2014-02-11 2015-08-13 Hyundai Motor Company Apparatus and method of providing road guidance based on augmented reality head-up display
US9151634B2 (en) * 2014-02-11 2015-10-06 Hyundai Motor Company Apparatus and method of providing road guidance based on augmented reality head-up display
US10663315B2 (en) * 2015-02-09 2020-05-26 Denso Corporation Vehicle display control device and vehicle display control method
US20190384295A1 (en) * 2015-02-10 2019-12-19 Mobileye Vision Technologies Ltd. Systems and methods for identifying landmarks
US11774251B2 (en) * 2015-02-10 2023-10-03 Mobileye Vision Technologies Ltd. Systems and methods for identifying landmarks
US11599113B2 (en) * 2015-02-10 2023-03-07 Mobileye Vision Technologies Ltd. Crowd sourcing data for autonomous vehicle navigation
US20190384294A1 (en) * 2015-02-10 2019-12-19 Mobileye Vision Technologies Ltd. Crowd sourcing data for autonomous vehicle navigation
US9304003B1 (en) 2015-03-18 2016-04-05 Microsoft Technology Licensing, Llc Augmented reality navigation
US9909894B2 (en) 2016-01-07 2018-03-06 Here Global B.V. Componentized junction models
US10234294B2 (en) 2016-04-01 2019-03-19 Here Global B.V. Road geometry matching with componentized junction models
US10921134B2 (en) 2016-04-01 2021-02-16 Here Global B.V. Road geometry matching with componentized junction models
US20190138261A1 (en) * 2016-06-22 2019-05-09 Sony Corporation Information processing device, information processing method, and program
US10922043B2 (en) * 2016-06-22 2021-02-16 Sony Corporation Information processing device and information processing method for acquiring information associated with a target
US20180341822A1 (en) * 2017-05-26 2018-11-29 Dura Operating, Llc Method and system for classifying objects in a perception scene graph by using a scene-detection-schema
US10495476B1 (en) 2018-09-27 2019-12-03 Phiar Technologies, Inc. Augmented reality navigation systems and methods
US11313695B2 (en) 2018-09-27 2022-04-26 Phiar Technologies, Inc. Augmented reality navigational indicator
US11448518B2 (en) 2018-09-27 2022-09-20 Phiar Technologies, Inc. Augmented reality navigational overlay
US11545036B2 (en) 2018-09-27 2023-01-03 Google Llc Real-time driving behavior and safety monitoring
US10573183B1 (en) 2018-09-27 2020-02-25 Phiar Technologies, Inc. Mobile real-time driving safety systems and methods
US11156472B2 (en) 2018-10-26 2021-10-26 Phiar Technologies, Inc. User interface for augmented reality navigation
US11085787B2 (en) * 2018-10-26 2021-08-10 Phiar Technologies, Inc. Augmented reality interface for navigation assistance
US10488215B1 (en) 2018-10-26 2019-11-26 Phiar Technologies, Inc. Augmented reality interface for navigation assistance
US20220111840A1 (en) * 2019-01-10 2022-04-14 Jaguar Land Rover Limited A control system, system and method for providing assistance to an occupant of a vehicle
US11919513B2 (en) * 2019-01-10 2024-03-05 Jaguar Land Rover Limited Control system, system and method for providing assistance to an occupant of a vehicle
US11499834B2 (en) * 2019-03-07 2022-11-15 Mobileye Vision Technologies Ltd. Aligning road information for navigation

Also Published As

Publication number Publication date
WO2006035755A1 (en) 2006-04-06
JPWO2006035755A1 (en) 2008-05-15
US20080195315A1 (en) 2008-08-14
JP4696248B2 (en) 2011-06-08

Similar Documents

Publication Publication Date Title
US8195386B2 (en) Movable-body navigation information display method and movable-body navigation information display unit
US8352180B2 (en) Device with camera-info
US7733244B2 (en) Navigation system
US9171214B2 (en) Projecting location based elements over a heads up display
US8423292B2 (en) Navigation device with camera-info
US10891859B1 (en) Displaying sensor data and supplemental data as a mask for autonomous vehicles
US20110103651A1 (en) Computer arrangement and method for displaying navigation data in 3d
CN106564432A (en) Apparatus and method for controlling viewing angle for vehicle, and vehicle including the apparatus
US20100245561A1 (en) Navigation device
JP2009020089A (en) System, method, and program for navigation
KR20080020956A (en) Method of operating a night-view system in a vehicle and corresponding night-view system
KR20110114114A (en) Real 3d navigation implementing method
JP4787196B2 (en) Car navigation system
JPWO2004048895A1 (en) MOBILE NAVIGATION INFORMATION DISPLAY METHOD AND MOBILE NAVIGATION INFORMATION DISPLAY DEVICE
EP3859390A1 (en) Method and system for rendering a representation of an evinronment of a vehicle
KR20080019690A (en) Navigation device with camera-info
JP2003302235A (en) Navigation display device
JP2021071885A (en) Area cut-out method and area cut-out program

Legal Events

Date Code Title Description
AS Assignment

Owner name: NATIONAL UNIVERSITY CORPORATION KUMAMOTO UNIVERSIT

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HU, ZHENCHENG;UCHIMURA, KEIICHI;REEL/FRAME:019124/0258

Effective date: 20070326

REMI Maintenance fee reminder mailed
LAPS Lapse for failure to pay maintenance fees
STCH Information on status: patent discontinuation

Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362

FP Expired due to failure to pay maintenance fee

Effective date: 20160605