EP2470986A2 - Portable telepresence apparatus - Google Patents

Portable telepresence apparatus

Info

Publication number
EP2470986A2
EP2470986A2 EP10814289A EP10814289A EP2470986A2 EP 2470986 A2 EP2470986 A2 EP 2470986A2 EP 10814289 A EP10814289 A EP 10814289A EP 10814289 A EP10814289 A EP 10814289A EP 2470986 A2 EP2470986 A2 EP 2470986A2
Authority
EP
European Patent Office
Prior art keywords
telepresence apparatus
housing
station
coupled
camera
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP10814289A
Other languages
German (de)
French (fr)
Other versions
EP2470986A4 (en
Inventor
Kelton Temby
Charles S. Jordan
Daniel S. Sanchez
Steven E. Butner
Derek J. Walters
Fuji Lai
Kevin P. Hanrahan
Joshua A. Bouganim
James Rosenthal
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
InTouch Technologies Inc
Original Assignee
InTouch Technologies Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US12/548,122 external-priority patent/US8384755B2/en
Application filed by InTouch Technologies Inc filed Critical InTouch Technologies Inc
Publication of EP2470986A2 publication Critical patent/EP2470986A2/en
Publication of EP2470986A4 publication Critical patent/EP2470986A4/en
Withdrawn legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/1454Digital output to display device ; Cooperation and interconnection of the display device with other functional units involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50Constructional details
    • H04N23/53Constructional details of electronic viewfinders, e.g. rotatable or detachable
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00Television systems
    • H04N7/14Systems for two-way working
    • H04N7/141Systems for two-way working between two video terminals, e.g. videophone
    • H04N7/142Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/147Digital output to display device ; Cooperation and interconnection of the display device with other functional units using display panels
    • GPHYSICS
    • G16INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16HHEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H40/00ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
    • G16H40/60ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
    • G16H40/67ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for remote operation
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60Control of cameras or camera modules
    • H04N23/63Control of cameras or camera modules by using electronic viewfinders
    • H04N23/631Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60Control of cameras or camera modules
    • H04N23/66Remote control of cameras or camera parts, e.g. by remote control devices
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60Control of cameras or camera modules
    • H04N23/66Remote control of cameras or camera parts, e.g. by remote control devices
    • H04N23/661Transmitting camera control signals through networks, e.g. control via the Internet
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09GARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2354/00Aspects of interface with display user
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09GARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2370/00Aspects of data communication
    • G09G2370/16Use of wireless transmission of display information
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09GARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2380/00Specific applications
    • G09G2380/08Biomedical applications
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50Constructional details
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60Control of cameras or camera modules
    • H04N23/68Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/90Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Definitions

  • the present disclosure relates generally to the field of telepresence systems. More specifically, the present disclosure relates to portable telepresence systems that include an apparatus remotely controlled through a remote station, where the system includes cameras, monitors, microphones, and speakers to allow for two-way audio-visual communication.
  • FIG. 1 is an illustration of a telepresence system that includes a remote station coupled to a portable telepresence apparatus located within an ambulance;
  • FIG. 2 is an illustration showing the portable telepresence apparatus within the ambulance;
  • FIG. 3 is an illustration showing the portable telepresence apparatus detached from a platform mounted to the ambulance ceiling;
  • FIG. 4 is an illustration showing the portable telepresence apparatus attached to a patient gurney
  • FIG. 5 is an illustration showing the portable telepresence apparatus attached to a stand
  • FIG. 6 is an illustration showing a patient within a healthcare facility that has a telepresence apparatus attached to a boom;
  • FIGS. 7A and 7B are illustrations of an alternate embodiment of the telepresence apparatus
  • FIG. 8 is an illustration of a rear view of the telepresence apparatus shown in Fig. 7;
  • FIG. 9 is an illustration showing a user holding the portable telepresence apparatus while viewing an image captured by the camera through a viewfinder screen;
  • FIG. 10 is an illustration showing a user interface that allows a user to vary speaker and microphone volume
  • FIG. 1 1 is an illustration showing a user interface that allows a user to perform certain functions through graphical icons
  • FIG. 12 is an illustration showing a picture in picture display
  • FIG. 13 is an illustration showing the pictures in swapped positions.
  • FIG. 14 is an illustration showing a graphical interface showing communication links in the system. Detailed Description of Preferred Embodiments
  • a telepresence system that includes a portable telepresence apparatus coupled to a remote station.
  • the telepresence apparatus includes a monitor, a camera, a speaker and a microphone.
  • the remote station includes a station monitor, a station camera, a station speaker and a station microphone.
  • the portable telepresence apparatus can be attached to a platform mounted to the ceiling of an ambulance.
  • the portable telepresence apparatus can be used by a physician or healthcare worker at the remote station to provide remote medical consultation. When the patient is moved from the ambulance the portable telepresence apparatus can be detached from the platform and moved with the patient.
  • a portable telepresence apparatus with a monitor, a camera, a speaker, a microphone and a viewfinder screen all coupled to a housing.
  • the viewfinder screen allows the user to view the image being captured by camera.
  • the portable telepresence apparatus is a hand held device that can be moved by a holder of the device in response to audio commands from the remote station.
  • the telepresence apparatus can be used by medical personnel to remotely view a patient in a fast and efficient manner.
  • FIG. 1 shows a telepresence system 10.
  • the system 10 includes a portable telepresence apparatus 12 that is coupled to a remote control station 14 through a wireless network 18.
  • the wireless network may be a cellular broadband network, satellite, WiMAX, and/or a WiFi network.
  • the portable telepresence apparatus 12 may be located within an ambulance 20.
  • the remote control station 14 may include a computer 22 that has a monitor 24, a camera 26, a microphone 28 and a speaker 30.
  • the computer 22 may also contain an input device 32 such as a joystick or a mouse.
  • the control station 14 is typically located in a place that is remote from the telepresence apparatus 12.
  • the system 10 may include a plurality of remote stations 14.
  • any number of telepresence apparatus 12 may be coupled to any number of remote stations 14 or other telepresence apparatus 12.
  • one remote station 14 may be coupled to a plurality of telepresence apparatus 12, or one telepresence apparatus 12 may be coupled to a plurality of remote stations 14, or a plurality of telepresence apparatus 12.
  • the system may include an arbitrator (not shown) that controls access between the telepresence apparatus 12 and the remote stations 14.
  • the portable telepresence apparatus 12 may be attached to a platform 34.
  • the platform 34 may extend from the ceiling (not shown) of the ambulance 20.
  • the platform 34 may include articulate joints 36 and 38 that provide at least two degrees of freedom and allow a user to move the telepresence apparatus 12 to different positions to view a patient and an EMT within an ambulance.
  • Each telepresence apparatus 12 includes a camera(s) 50, a monitor 52, a microphone(s) 54 and a speaker(s) 56 that are all attached to a housing 58.
  • the camera 50 is coupled to the remote monitor 24, so that a user at the remote station 14 can view the patient and/or EMT.
  • the monitor 52 is coupled to the remote camera 26 so the patient and EMT may view the user of the remote station 14.
  • the microphones 28 and 54, and speakers 30 and 56 allow for audible communication between the system operator and the patient and/or EMT.
  • the system 10 allows a system user such as a physician to view a patient in the ambulance and provide remote medical consultation through the remote station 14 and the telepresence apparatus 12.
  • Personnel such as the EMT can transmit questions and responses through the system back to the physician.
  • the camera 50 allows the physician to view the patient and enhance the medical consultation.
  • the monitor 52 can display the physician to provide a feeling of presence, such as in an ambulance.
  • the platform 34 allows the physician to pan and tilt the telepresence apparatus 12.
  • the telepresence apparatus 12 may include a wireless transceiver 60 that is coupled to the wireless network.
  • the portable telepresence apparatus 12 also includes a battery 62.
  • the system 10 may have certain components and software that are the same or similar to a robotic system provided by InTouch Technologies, Inc. of Goleta, California and embodies a system described in U.S. Pat. 6,925,357, which is hereby incorporated by reference.
  • the portable telepresence apparatus 12 can be detached from the platform 34.
  • the telepresence apparatus 12 and platform 34 may have mechanical connectors 64 that allow the telepresence apparatus 12 to be readily attached and detached from the platform 34.
  • the telepresence apparatus 12 and platform 34 may include electrical connectors 66.
  • the location where the system 10 is utilized, such as in an ambulance, may include a wireless transceiver (not shown) that can provide wireless communication to the remote control station 14.
  • the electrical connectors 66 provide an electrical connection between the telepresence apparatus 12 and a wireless transceiver.
  • the connectors 66 may also provide power to the telepresence apparatus 12.
  • the wireless transceiver 60 of the telepresence apparatus 12 may be coupled to the remote control station 14 through a wireless transceiver in the vicinity in which the system 10 is utilized, such as in an ambulance.
  • the telepresence apparatus may include an actuator system 68 that can move the camera 50 in two or more degrees of freedom. This allows the operator to move the camera field of view even when the telepresence apparatus 12 is detached from the platform 34.
  • the portable telepresence apparatus 12 can be detached from the platform (not shown) and attached to the patient gurney 70.
  • the telepresence apparatus 12 may be attached to a platform 72 with two degrees of freedom that allow the remote station user to move the telepresence apparatus 12.
  • the platform 72 may include a clamp 74 that allows for attachment to the gurney 70.
  • the telepresence apparatus 12 and patient can be moved out of the ambulance on the gurney 70.
  • the portable aspect of the telepresence apparatus 12 allows it to be moved with the patient.
  • the telepresence apparatus 12 should be of a size and weight so that an individual can lift it with ease.
  • the portable telepresence apparatus 12 can be detached from the ambulance platform (not shown) and attached to a stand 80 at a remote location.
  • the portable nature of the telepresence apparatus 12 allows it to be taken to any location to allow for remote telepresence of the operator of the remote station. If the operator is a physician, the portable telepresence apparatus 12 allows for remote medical consultation at any site.
  • FIG. 6 shows the patient and gurney moved into a healthcare facility with a telepresence apparatus 90 attached to a boom 92.
  • the telepresence apparatus wireless transceiver may be coupled to the remote station through the healthcare facility local wireless network such as a WiFi network.
  • the portable telepresence apparatus 90 can be connected to an electrical power outlet and a network for Ethernet connection.
  • An electronic ID device 94 may be attached to the patient. The ID device 94 may transmit a wireless signal to the telepresence apparatus 90 attached to the boom 92.
  • Receipt of the signal by the telepresence apparatus 90 may cause the remote station to be coupled to the telepresence apparatus 90 attached to the boom 92 instead of the portable telepresence apparatus 90.
  • the telepresence apparatus 90 may be coupled to the remote station by other means. For example, a nurse may type in information into the healthcare facility network system that identifies the new location of the patient. Such an entry may cause the system to switch the remote control station to the telepresence apparatus 90. Additionally, there may be other methodologies for inducing the system to automatically transfer the remote station from one telepresence apparatus to another.
  • FIGS. 7A, 7B and 8 show another embodiment of a portable telepresence apparatus 100.
  • the telepresence apparatus 100 includes a monitor 102, a first camera 104, a microphone 106 and a speaker 108 all attached to a first face 1 10 of a housing 1 12.
  • the camera 104 may include a fish eye lens with a 180 degree field of view and a zoom feature. More specifically, the camera is a 10 megapixel CCD.
  • the remote user When the remote user is "zoomed out", the system re-samples the 10MP image to 1440x768 before compressing and sending it over the Internet. The remote user may select a portion of the image, for instance the upper left quarter, by drawing a box around it.
  • the coordinates of this box are transmitted to the telepresence device, which in turn will restrict the re-sampling area to the upper-left 2.5MP of camera input.
  • the face 100 is constructed to have a size and weight so that it can be carried by a single human being.
  • the telepresence apparatus 100 may have a handle 1 14 to facilitate carrying and moving it. By way of example, the telepresence apparatus may weigh less than 5 pounds.
  • the housing 1 12 may be constructed so that the telepresence apparatus 100 can stand in an upright position on a surface, possibly with a lean-back angle of 5 degrees to facilitate viewing.
  • the telepresence apparatus 100 may include a viewfinder screen 1 16 and a second camera 1 18 attached to a second face 120 of the housing 1 12.
  • the second camera 1 18 can capture images of a person holding the telepresence apparatus that are transmitted to the remote station, such that the remote user may switch to a view of the holder in order to give them instructions via 2-way audio video communication.
  • Located within the housing 1 12 are electronic circuits and devices, including a processor(s), memory and hard disk drive (not shown) that can perform the various functions of the telepresence apparatus 100.
  • One side of the telepresence apparatus 100 may include various ports 122, 124, 126, 128 and 130.
  • Port 122 may provide a USB and/or Bluetooth, or other connection.
  • the USB port can be used to attach a medical instrument, such as a stethoscope or a blood pulse oximeter to the telepresence apparatus 100.
  • Port 124 may provide C video, S video auxiliary inputs.
  • a battery of the telepresence apparatus may be charged through connector 126.
  • a cell phone connection may be established through a transceiver 128 within the housing 1 12.
  • Connector 130 may provide 801 .1 1 WiFi connectivity.
  • the other side of the telepresence apparatus 100 may include different input buttons 132 that can establish videoconferencing controls, such as audio volume adjustment.
  • the monitor 102 may display the various ports and pluggable devices that can be used with the telepresence apparatus 100 through touch screens operated by the user.
  • the system 100 may have certain components and software that are the same or similar to a robotic system provided by InTouch Technologies, Inc. of Goleta, California, such as the system described in U.S. Pat. 6,925,357, which is hereby incorporated by reference.
  • FIG. 9 shows a user holding the portable telepresence apparatus 100 to allow a remote operator to view a patient through the first camera 104 located on the opposite side of the telepresence apparatus 100.
  • the viewfinder screen 1 16 allows the holder to view the image being captured by the first camera 104 and move the telepresence apparatus 100, accordingly.
  • the operator at the remote station can provide oral instructions to the holder to move the portable telepresence apparatus 100 to obtain a desired view of the patient.
  • the user performs the functions of the actuators shown in Figs. 2 and 3, as described above.
  • the telepresence apparatus 100 may include a motion sensing device 134 such as an accelerometer, gyro and/or magnetometer.
  • the motion sensing device 134 can be utilized, so that the person displayed by the monitor is right sized even if the user is holding the telepresence apparatus 100 in a tilted manner. In other words, the top of the head of the person displayed by the monitor will always be pointing upward and away from gravity, such that if the device is held at a 45 degree angle, the top of the head will appear in the upper-left corner of the screen.
  • the motion sensing device 134 can be used to provide a right sized version of an image captured on camera 104 to the remote station.
  • the viewfinder screen 1 16 may include touch features that allow the holder of the telepresence apparatus 100 to change the image being captured. For example, movement of the holder's fingers from an inward location in an outward manner may cause the captured image to be zoomed in. An opposite movement of the user's fingers may cause the image to zoom out. Alternatively, the holder may draw on the small touch screen which shows the patient's image, and the resultant telestration may be shared to the remote user. The remote user retains shared feature control, and may disable certain of these features from being available to the holder.
  • FIG. 10 shows a user interface 140 displayed by the viewfinder 1 16.
  • the interface 140 includes graphical icons 142 that can be touched by the user to change the volume of the speaker and microphone.
  • the viewfinder 1 16 may also display the interface 144 shown in FIG. 1 1 .
  • the interface 144 includes graphical icons 146 that can be touched to control such functions as power, audio modes, connect/disconnect and a hold button. For example, the user may change the audio source from local microphone to Bluetooth-based headset.
  • FIG. 12 shows the monitor 102 displaying an image 148 of the remote operator and the image 150 captured by the camera in a picture in picture format.
  • the images 148 and 150 can be swapped as shown in FIG. 13.
  • the images can be swapped by a touch screen toggle (not shown) displayed by the viewfinder screen, or by a graphical switch at the remote station.
  • Figures 12 and 13 also show an alternative embodiment of the cameras, whereby a single camera can be physically rotated 180 degrees to point either forward or backward.
  • the telepresence apparatus and/or the remote station may provide a graphical interface 160 that shows the connectivity between the telepresence apparatus and the remote station.
  • Graphical icons 162, 164, 166 and 168 may represent the telepresence apparatus, a server, the network and remote station, respectively.
  • a solid line between two devices indicates an established link.
  • a broken line indicates a broken communication link between two devices.
  • Fig. 14 depicts a broken communication between the network and remote station.
  • the system can perform diagnostic and corrective action functions for broken links. The corrective actions may be automatic, or include prompt messages to the user to perform certain tasks such as plugging in their Ethernet cable, or provide instructions to configure a firewall.
  • Graphical icon 162 may change depending on the type of telepresence endpoint to which connection is attempted. For example, if the user is attempting a connection to a mobile robot, the icon would depict a mobile robot, whereas if the user is connecting to a head-only device, the icon would depict that device.
  • the telepresence apparatus 100 may provide simultaneous access to wireless cellular carriers, WiFi and WiMAX local wireless and satellite connectivity using a number of onboard modems.
  • One carrier may operate in accordance with the 2G standard, another on 3G, another on 4G, and so forth.
  • These network connections may exist onboard the telepresence apparatus 100, or alternatively the telepresence apparatus may have a single local wireless connection to a nearby base communications unit, which in turn controls outgoing connectivity.
  • the base communications unit may take the form of a mobile system, which may be mounted in a vehicle, or travel with the telepresence apparatus.
  • the system may also be enhanced with additional amplification stages installed in a vehicle.
  • the system may aggregate available connections and improve Quality of Service metrics in one of a variety of modes.
  • the modes may include: a Redundancy mode that sends the same signal over multiple connections; a Balancing mode that sends parts of a signal over different connections; a Switching mode that sends all traffic over the best available connection; and a VPN switching mode which switches between the Redundancy, Balancing and Switching modes during a session.
  • the system may evaluate reliability and dynamic bandwidth on each of its wireless connections. Based on current networking statistics, the networking algorithm may choose to split the communication stream among the different connections, for example control commands on one connection and video/audio on another connection, or video on one and control on another connection. When a certain connection degrades significantly, the algorithm may choose to move its data stream to another connection. At times, this may mean giving up video (on a 3G connection) to switch to an audio-only (2G) connection. Further, the algorithm may choose to use a wireless connection which is on an older, slower but more reliable network (e.g. 2G) for audio, and simultaneously use a connection on a newer/faster but less reliable network (e.g. 3G) for video.
  • 2G older, slower but more reliable network
  • 3G newer/faster but less reliable network
  • This approach may ensure that audio is always robustly transported, while the video may have interruptions but will maintain good visual quality and frame rate.
  • the system may have video and audio on the newer/faster connection, with simultaneous backup of audio on the older/slower connection. If the newer connection becomes less reliable, the system may cross-fade the played audio to use the data from the older/slower connection.
  • the portable telepresence apparatus 100 can be used in various applications.
  • the telepresence apparatus 100 can be used to allow for remote examination of a patient.
  • the telepresence apparatus 100 can remain in an active setup-and-recording mode, even when there is no session with a remote operator in progress. This allows for offline recording of patient status, as well as pre-session "setup".
  • Pre-session setup allows a user to position the telepresence apparatus and use the digital box-zoom controls to ensure optimal viewing of the patient prior to the remote physician's entry. This is to be contrasted with prior art telepresence systems, wherein at the start of a new session, the camera pan/tilt/zoom settings are either at default, or previous settings.
  • the telepresence apparatus allows a local user can set up the optimal view field for the remote doctor prior to his/her session initiation; and further can update the view field when the remote doctor becomes temporarily busy or requests local assistance.
  • the telepresence apparatus 100 may have an "aircraft mode" that inhibits outbound transmission during take-off and landing when the telepresence apparatus is located in an aircraft, for example. Additionally, the system may be switched to a "capture-then-send" modality during periods of limited wireless connectivity. In this modality, a user can make a video recording of a patient exam intended for a physician. The video recording may be supplemented by telemetry data from attached medical devices. Exam reports are then automatically forwarded to the physician upon the system regaining adequate connectivity, and placed in a queue at the physician's remote station.
  • the telepresence apparatus may also be equipped with a GPS (not shown). This allows for real-time tracking of the geographic location of each telepresence apparatus, and geo-tagging of session statistics. This serves a variety of functions, including: analysis of wireless connectivity based on geographic location; tracking of video clips and patient data based on proximity to a hospital and ambulance speed; and hospital and billing auditing.
  • the GPS feature may also provide for unique fleet monitoring, anti-theft, etc.
  • the portable telepresence apparatus can be used for various applications in the medical field.
  • One application is specialty transport, in particular pediatric transport.
  • An ambulance and team can be deployed from Hospital A to Hospital B for patient transport.
  • Hospital B Upon arrival at Hospital B, a patient may be found to be in need of stabilization prior to transport.
  • An expert consultation can occur in Hospital B, or during transport on the trip back to Hospital A.
  • a call may be placed for a transport of a patient from a spoke Hospital B which does not have expertise that Hospital A has (e.g., pediatric intensivist specialist care).
  • a transport team from Hospital A is deployed to Hospital B.
  • the team brings the telepresence apparatus 100, mounts it on a gurney and places the gurney in an ambulance.
  • the team arrives at Hospital B and views the patient.
  • the remote physician from Hospital A establishes a link with the telepresence apparatus located on the gurney.
  • the remote physician can pan-tilt-zoom the image to obtain a desired view.
  • someone at the telepresence apparatus side can assist by repositioning the telepresence apparatus 100 and using the viewfinder to help position the front camera on the patient/desired view.
  • the telepresence apparatus side team is able to communicate with the remote physician via the main speaker/mic on the unit.
  • the remote physician may speak with various members of the team and patient/family at Hospital B to make a recommendation.
  • a Bluetooth or wireless headset can be used as an alternative.
  • the remote physician is able to help with decisions regarding care/transport of the patient.
  • Care can be advanced either through decision to continue transport, to not continue transport, or to administer certain care as determined by the remote physician in collaboration with the onsite team.
  • the consult can also occur during transport if there are situations where the patient starts to decompensate.
  • the link would be between a remote station and a telepresence apparatus located in the ambulance during transport of the patient.
  • the telepresence apparatus would be mounted on a gurney, the remote physician can view the patient and communicate with the transport team to help make a care decision.
  • Another application may include a nurse conducting a scheduled visit to a chronically ill patient in their home.
  • the nurse views the patient.
  • the touch screen of the telepresence apparatus can be used to document various symptoms.
  • the data is stored in the telepresence apparatus.
  • the data and video of certain patient interactions can be forwarded to a server.
  • the telepresence apparatus may receive requested information from the server.
  • the nurse may observe a troubling symptom and request a physician consult.
  • the nurse may call the physician, who establishes a link with the telepresence apparatus and initiates a telehealth session with the patient, facilitated by the nurse.
  • the physician may request that the nurse attach a digital stethoscope to the telepresence apparatus and apply it to the patient.
  • the physician may then request that the nurse attach a portable ultrasound device to the auxiliary video port of the telepresence apparatus. Finally, the physician may decide that the patient should be taken immediately to a medical facility. The nurse may call the ambulance. The nurse stays by the patient's side, with the remote physician logged into the telepresence apparatus, as the patient is transported to the facility.

Abstract

A telepresence system that includes a portable telepresence apparatus coupled to a remote control station. The telepresence apparatus comprises a monitor, a camera, a speaker, a microphone and a viewfinder screen coupled to a housing. The view finder screen allows the user to view the image being captured by the camera. The portable telepresence apparatus is a hand held device that can be moved by a holder of the device in response to audio commands from the remote station. The telepresence apparatus can be used by medical personnel to remotely view a patient in a fast and efficient manner.

Description

PORTABLE TELEPRESENCE APPARATUS
Cross-Reference to Related Applications
[0001] This application claims priority to Application No. 61/399,637, titled TELE- ROBOTIC SYSTEM WITH A HAND HELD ROBOTIC FACE, filed on Jul 14, 2010 and is a continuation-in-part of Application 12/548,122, titled PORTABLE REMOTE PRESENCE ROBOT, filed on August 26, 2009, the entire contents of each of which are hereby incorporated by reference herein.
Technical Field
[0002] The present disclosure relates generally to the field of telepresence systems. More specifically, the present disclosure relates to portable telepresence systems that include an apparatus remotely controlled through a remote station, where the system includes cameras, monitors, microphones, and speakers to allow for two-way audio-visual communication.
Brief Description of the Drawings
[0003] The present embodiments will become more fully apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. Understanding that the accompanying drawings depict only typical embodiments of the invention and are, therefore, not meant to limit the scope of the invention, the embodiments will be described and explained with specificity and detail through use of the accompanying drawings as listed below.
[0004] FIG. 1 is an illustration of a telepresence system that includes a remote station coupled to a portable telepresence apparatus located within an ambulance; [0005] FIG. 2 is an illustration showing the portable telepresence apparatus within the ambulance;
[0006] FIG. 3 is an illustration showing the portable telepresence apparatus detached from a platform mounted to the ambulance ceiling;
[0007] FIG. 4 is an illustration showing the portable telepresence apparatus attached to a patient gurney;
[0008] FIG. 5 is an illustration showing the portable telepresence apparatus attached to a stand;
[0009] FIG. 6 is an illustration showing a patient within a healthcare facility that has a telepresence apparatus attached to a boom;
[0010] FIGS. 7A and 7B are illustrations of an alternate embodiment of the telepresence apparatus;
[0011] FIG. 8 is an illustration of a rear view of the telepresence apparatus shown in Fig. 7;
[0012] FIG. 9 is an illustration showing a user holding the portable telepresence apparatus while viewing an image captured by the camera through a viewfinder screen;
[0013] FIG. 10 is an illustration showing a user interface that allows a user to vary speaker and microphone volume;
[0014] FIG. 1 1 is an illustration showing a user interface that allows a user to perform certain functions through graphical icons;
[0015] FIG. 12 is an illustration showing a picture in picture display;
[0016] FIG. 13 is an illustration showing the pictures in swapped positions; and
[0017] FIG. 14 is an illustration showing a graphical interface showing communication links in the system. Detailed Description of Preferred Embodiments
[0018] It will be readily understood that the components of the embodiments as generally described and illustrated in the Figures herein could be arranged and designed in a wide variety of different configurations. Thus the following more detailed description of various embodiments, as represented in the Figures, is not intended to limit the scope of the invention as claimed, but is merely representative of various embodiments of the invention. In addition, in some cases, well-known structures, materials, or operations are not shown or described in detail in order to avoid obscuring aspects of the invention. Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0019] The word "exemplary" and the term "for example" are used herein to mean "serving as an example, for instance, or illustration." Any embodiment described herein as "exemplary" or "for example" is not necessarily to be construed as preferred or advantageous over other embodiments. Although various aspects of the embodiments are presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.
[0020] Disclosed is a telepresence system that includes a portable telepresence apparatus coupled to a remote station. The telepresence apparatus includes a monitor, a camera, a speaker and a microphone. The remote station includes a station monitor, a station camera, a station speaker and a station microphone. The portable telepresence apparatus can be attached to a platform mounted to the ceiling of an ambulance. The portable telepresence apparatus can be used by a physician or healthcare worker at the remote station to provide remote medical consultation. When the patient is moved from the ambulance the portable telepresence apparatus can be detached from the platform and moved with the patient.
[0021] Also disclosed is a portable telepresence apparatus with a monitor, a camera, a speaker, a microphone and a viewfinder screen all coupled to a housing. The viewfinder screen allows the user to view the image being captured by camera. The portable telepresence apparatus is a hand held device that can be moved by a holder of the device in response to audio commands from the remote station. The telepresence apparatus can be used by medical personnel to remotely view a patient in a fast and efficient manner.
[0022] Referring to the drawings more particularly by reference numbers, FIG. 1 shows a telepresence system 10. The system 10 includes a portable telepresence apparatus 12 that is coupled to a remote control station 14 through a wireless network 18. The wireless network may be a cellular broadband network, satellite, WiMAX, and/or a WiFi network. The portable telepresence apparatus 12 may be located within an ambulance 20.
[0023] The remote control station 14 may include a computer 22 that has a monitor 24, a camera 26, a microphone 28 and a speaker 30. The computer 22 may also contain an input device 32 such as a joystick or a mouse. The control station 14 is typically located in a place that is remote from the telepresence apparatus 12. Although only one remote control station 14 is shown, the system 10 may include a plurality of remote stations 14. In general any number of telepresence apparatus 12 may be coupled to any number of remote stations 14 or other telepresence apparatus 12. For example, one remote station 14 may be coupled to a plurality of telepresence apparatus 12, or one telepresence apparatus 12 may be coupled to a plurality of remote stations 14, or a plurality of telepresence apparatus 12. The system may include an arbitrator (not shown) that controls access between the telepresence apparatus 12 and the remote stations 14.
[0024] As shown in FIGS. 2 and 3, the portable telepresence apparatus 12 may be attached to a platform 34. The platform 34 may extend from the ceiling (not shown) of the ambulance 20. The platform 34 may include articulate joints 36 and 38 that provide at least two degrees of freedom and allow a user to move the telepresence apparatus 12 to different positions to view a patient and an EMT within an ambulance.
[0025] Each telepresence apparatus 12 includes a camera(s) 50, a monitor 52, a microphone(s) 54 and a speaker(s) 56 that are all attached to a housing 58. The camera 50 is coupled to the remote monitor 24, so that a user at the remote station 14 can view the patient and/or EMT. Likewise, the monitor 52 is coupled to the remote camera 26 so the patient and EMT may view the user of the remote station 14. The microphones 28 and 54, and speakers 30 and 56, allow for audible communication between the system operator and the patient and/or EMT.
[0026] The system 10 allows a system user such as a physician to view a patient in the ambulance and provide remote medical consultation through the remote station 14 and the telepresence apparatus 12. Personnel such as the EMT can transmit questions and responses through the system back to the physician. The camera 50 allows the physician to view the patient and enhance the medical consultation. The monitor 52 can display the physician to provide a feeling of presence, such as in an ambulance. The platform 34 allows the physician to pan and tilt the telepresence apparatus 12. [0027] The telepresence apparatus 12 may include a wireless transceiver 60 that is coupled to the wireless network. The portable telepresence apparatus 12 also includes a battery 62.
[0028] The system 10 may have certain components and software that are the same or similar to a robotic system provided by InTouch Technologies, Inc. of Goleta, California and embodies a system described in U.S. Pat. 6,925,357, which is hereby incorporated by reference.
[0029] As shown in FIG. 3, the portable telepresence apparatus 12 can be detached from the platform 34. The telepresence apparatus 12 and platform 34 may have mechanical connectors 64 that allow the telepresence apparatus 12 to be readily attached and detached from the platform 34. Likewise, the telepresence apparatus 12 and platform 34 may include electrical connectors 66. The location where the system 10 is utilized, such as in an ambulance, may include a wireless transceiver (not shown) that can provide wireless communication to the remote control station 14. The electrical connectors 66 provide an electrical connection between the telepresence apparatus 12 and a wireless transceiver. The connectors 66 may also provide power to the telepresence apparatus 12. Alternatively, the wireless transceiver 60 of the telepresence apparatus 12 may be coupled to the remote control station 14 through a wireless transceiver in the vicinity in which the system 10 is utilized, such as in an ambulance. The telepresence apparatus may include an actuator system 68 that can move the camera 50 in two or more degrees of freedom. This allows the operator to move the camera field of view even when the telepresence apparatus 12 is detached from the platform 34.
[0030] As shown in FIG. 4, the portable telepresence apparatus 12 can be detached from the platform (not shown) and attached to the patient gurney 70. The telepresence apparatus 12 may be attached to a platform 72 with two degrees of freedom that allow the remote station user to move the telepresence apparatus 12. The platform 72 may include a clamp 74 that allows for attachment to the gurney 70. The telepresence apparatus 12 and patient can be moved out of the ambulance on the gurney 70. The portable aspect of the telepresence apparatus 12 allows it to be moved with the patient. The telepresence apparatus 12 should be of a size and weight so that an individual can lift it with ease.
[0031] As shown in FIG. 5, the portable telepresence apparatus 12 can be detached from the ambulance platform (not shown) and attached to a stand 80 at a remote location. The portable nature of the telepresence apparatus 12 allows it to be taken to any location to allow for remote telepresence of the operator of the remote station. If the operator is a physician, the portable telepresence apparatus 12 allows for remote medical consultation at any site.
[0032] FIG. 6 shows the patient and gurney moved into a healthcare facility with a telepresence apparatus 90 attached to a boom 92. When the gurney 70 is moved into close proximity with the healthcare facility, the telepresence apparatus wireless transceiver may be coupled to the remote station through the healthcare facility local wireless network such as a WiFi network. Once inside the facility, the portable telepresence apparatus 90 can be connected to an electrical power outlet and a network for Ethernet connection. An electronic ID device 94 may be attached to the patient. The ID device 94 may transmit a wireless signal to the telepresence apparatus 90 attached to the boom 92. Receipt of the signal by the telepresence apparatus 90 may cause the remote station to be coupled to the telepresence apparatus 90 attached to the boom 92 instead of the portable telepresence apparatus 90. The telepresence apparatus 90 may be coupled to the remote station by other means. For example, a nurse may type in information into the healthcare facility network system that identifies the new location of the patient. Such an entry may cause the system to switch the remote control station to the telepresence apparatus 90. Additionally, there may be other methodologies for inducing the system to automatically transfer the remote station from one telepresence apparatus to another.
[0033] FIGS. 7A, 7B and 8 show another embodiment of a portable telepresence apparatus 100. The telepresence apparatus 100 includes a monitor 102, a first camera 104, a microphone 106 and a speaker 108 all attached to a first face 1 10 of a housing 1 12. The camera 104 may include a fish eye lens with a 180 degree field of view and a zoom feature. More specifically, the camera is a 10 megapixel CCD. When the remote user is "zoomed out", the system re-samples the 10MP image to 1440x768 before compressing and sending it over the Internet. The remote user may select a portion of the image, for instance the upper left quarter, by drawing a box around it. The coordinates of this box are transmitted to the telepresence device, which in turn will restrict the re-sampling area to the upper-left 2.5MP of camera input. The face 100 is constructed to have a size and weight so that it can be carried by a single human being. The telepresence apparatus 100 may have a handle 1 14 to facilitate carrying and moving it. By way of example, the telepresence apparatus may weigh less than 5 pounds. The housing 1 12 may be constructed so that the telepresence apparatus 100 can stand in an upright position on a surface, possibly with a lean-back angle of 5 degrees to facilitate viewing.
[0034] The telepresence apparatus 100 may include a viewfinder screen 1 16 and a second camera 1 18 attached to a second face 120 of the housing 1 12. The second camera 1 18 can capture images of a person holding the telepresence apparatus that are transmitted to the remote station, such that the remote user may switch to a view of the holder in order to give them instructions via 2-way audio video communication. Located within the housing 1 12 are electronic circuits and devices, including a processor(s), memory and hard disk drive (not shown) that can perform the various functions of the telepresence apparatus 100. One side of the telepresence apparatus 100 may include various ports 122, 124, 126, 128 and 130. Port 122 may provide a USB and/or Bluetooth, or other connection. The USB port can be used to attach a medical instrument, such as a stethoscope or a blood pulse oximeter to the telepresence apparatus 100. Port 124 may provide C video, S video auxiliary inputs. A battery of the telepresence apparatus may be charged through connector 126. A cell phone connection may be established through a transceiver 128 within the housing 1 12. Connector 130 may provide 801 .1 1 WiFi connectivity. As shown in fig. 7B, the other side of the telepresence apparatus 100 may include different input buttons 132 that can establish videoconferencing controls, such as audio volume adjustment. The monitor 102 may display the various ports and pluggable devices that can be used with the telepresence apparatus 100 through touch screens operated by the user. The system 100 may have certain components and software that are the same or similar to a robotic system provided by InTouch Technologies, Inc. of Goleta, California, such as the system described in U.S. Pat. 6,925,357, which is hereby incorporated by reference.
[0035] FIG. 9 shows a user holding the portable telepresence apparatus 100 to allow a remote operator to view a patient through the first camera 104 located on the opposite side of the telepresence apparatus 100. The viewfinder screen 1 16 allows the holder to view the image being captured by the first camera 104 and move the telepresence apparatus 100, accordingly. By way of example, the operator at the remote station can provide oral instructions to the holder to move the portable telepresence apparatus 100 to obtain a desired view of the patient. To this extent the user performs the functions of the actuators shown in Figs. 2 and 3, as described above.
[0036] The telepresence apparatus 100 may include a motion sensing device 134 such as an accelerometer, gyro and/or magnetometer. The motion sensing device 134 can be utilized, so that the person displayed by the monitor is right sized even if the user is holding the telepresence apparatus 100 in a tilted manner. In other words, the top of the head of the person displayed by the monitor will always be pointing upward and away from gravity, such that if the device is held at a 45 degree angle, the top of the head will appear in the upper-left corner of the screen. Likewise, the motion sensing device 134 can be used to provide a right sized version of an image captured on camera 104 to the remote station.
[0037] The viewfinder screen 1 16 may include touch features that allow the holder of the telepresence apparatus 100 to change the image being captured. For example, movement of the holder's fingers from an inward location in an outward manner may cause the captured image to be zoomed in. An opposite movement of the user's fingers may cause the image to zoom out. Alternatively, the holder may draw on the small touch screen which shows the patient's image, and the resultant telestration may be shared to the remote user. The remote user retains shared feature control, and may disable certain of these features from being available to the holder.
[0038] FIG. 10 shows a user interface 140 displayed by the viewfinder 1 16. The interface 140 includes graphical icons 142 that can be touched by the user to change the volume of the speaker and microphone. The viewfinder 1 16 may also display the interface 144 shown in FIG. 1 1 . The interface 144 includes graphical icons 146 that can be touched to control such functions as power, audio modes, connect/disconnect and a hold button. For example, the user may change the audio source from local microphone to Bluetooth-based headset.
[0039] FIG. 12 shows the monitor 102 displaying an image 148 of the remote operator and the image 150 captured by the camera in a picture in picture format. The images 148 and 150 can be swapped as shown in FIG. 13. The images can be swapped by a touch screen toggle (not shown) displayed by the viewfinder screen, or by a graphical switch at the remote station. Figures 12 and 13 also show an alternative embodiment of the cameras, whereby a single camera can be physically rotated 180 degrees to point either forward or backward.
[0040] As shown in FIG. 14, the telepresence apparatus and/or the remote station may provide a graphical interface 160 that shows the connectivity between the telepresence apparatus and the remote station. Graphical icons 162, 164, 166 and 168 may represent the telepresence apparatus, a server, the network and remote station, respectively. A solid line between two devices indicates an established link. A broken line indicates a broken communication link between two devices. For example, Fig. 14 depicts a broken communication between the network and remote station. The system can perform diagnostic and corrective action functions for broken links. The corrective actions may be automatic, or include prompt messages to the user to perform certain tasks such as plugging in their Ethernet cable, or provide instructions to configure a firewall. Graphical icon 162 may change depending on the type of telepresence endpoint to which connection is attempted. For example, if the user is attempting a connection to a mobile robot, the icon would depict a mobile robot, whereas if the user is connecting to a head-only device, the icon would depict that device.
[0041] The telepresence apparatus 100 may provide simultaneous access to wireless cellular carriers, WiFi and WiMAX local wireless and satellite connectivity using a number of onboard modems. One carrier may operate in accordance with the 2G standard, another on 3G, another on 4G, and so forth. These network connections may exist onboard the telepresence apparatus 100, or alternatively the telepresence apparatus may have a single local wireless connection to a nearby base communications unit, which in turn controls outgoing connectivity. The base communications unit may take the form of a mobile system, which may be mounted in a vehicle, or travel with the telepresence apparatus. The system may also be enhanced with additional amplification stages installed in a vehicle. The system may aggregate available connections and improve Quality of Service metrics in one of a variety of modes. The modes may include: a Redundancy mode that sends the same signal over multiple connections; a Balancing mode that sends parts of a signal over different connections; a Switching mode that sends all traffic over the best available connection; and a VPN switching mode which switches between the Redundancy, Balancing and Switching modes during a session.
[0042] The system may evaluate reliability and dynamic bandwidth on each of its wireless connections. Based on current networking statistics, the networking algorithm may choose to split the communication stream among the different connections, for example control commands on one connection and video/audio on another connection, or video on one and control on another connection. When a certain connection degrades significantly, the algorithm may choose to move its data stream to another connection. At times, this may mean giving up video (on a 3G connection) to switch to an audio-only (2G) connection. Further, the algorithm may choose to use a wireless connection which is on an older, slower but more reliable network (e.g. 2G) for audio, and simultaneously use a connection on a newer/faster but less reliable network (e.g. 3G) for video. This approach may ensure that audio is always robustly transported, while the video may have interruptions but will maintain good visual quality and frame rate. Alternatively, the system may have video and audio on the newer/faster connection, with simultaneous backup of audio on the older/slower connection. If the newer connection becomes less reliable, the system may cross-fade the played audio to use the data from the older/slower connection.
[0043] The portable telepresence apparatus 100 can be used in various applications. For example, the telepresence apparatus 100 can be used to allow for remote examination of a patient. The telepresence apparatus 100 can remain in an active setup-and-recording mode, even when there is no session with a remote operator in progress. This allows for offline recording of patient status, as well as pre-session "setup". Pre-session setup allows a user to position the telepresence apparatus and use the digital box-zoom controls to ensure optimal viewing of the patient prior to the remote physician's entry. This is to be contrasted with prior art telepresence systems, wherein at the start of a new session, the camera pan/tilt/zoom settings are either at default, or previous settings. The telepresence apparatus allows a local user can set up the optimal view field for the remote doctor prior to his/her session initiation; and further can update the view field when the remote doctor becomes temporarily busy or requests local assistance.
[0044] The telepresence apparatus 100 may have an "aircraft mode" that inhibits outbound transmission during take-off and landing when the telepresence apparatus is located in an aircraft, for example. Additionally, the system may be switched to a "capture-then-send" modality during periods of limited wireless connectivity. In this modality, a user can make a video recording of a patient exam intended for a physician. The video recording may be supplemented by telemetry data from attached medical devices. Exam reports are then automatically forwarded to the physician upon the system regaining adequate connectivity, and placed in a queue at the physician's remote station.
[0045] The telepresence apparatus may also be equipped with a GPS (not shown). This allows for real-time tracking of the geographic location of each telepresence apparatus, and geo-tagging of session statistics. This serves a variety of functions, including: analysis of wireless connectivity based on geographic location; tracking of video clips and patient data based on proximity to a hospital and ambulance speed; and hospital and billing auditing. The GPS feature may also provide for unique fleet monitoring, anti-theft, etc.
[0046] The portable telepresence apparatus can be used for various applications in the medical field. One application is specialty transport, in particular pediatric transport. An ambulance and team can be deployed from Hospital A to Hospital B for patient transport. Upon arrival at Hospital B, a patient may be found to be in need of stabilization prior to transport. An expert consultation can occur in Hospital B, or during transport on the trip back to Hospital A.
[0047] For example, a call may be placed for a transport of a patient from a spoke Hospital B which does not have expertise that Hospital A has (e.g., pediatric intensivist specialist care). A transport team from Hospital A is deployed to Hospital B. The team brings the telepresence apparatus 100, mounts it on a gurney and places the gurney in an ambulance. The team arrives at Hospital B and views the patient. If at any point the transport team would like to request a consult, the remote physician from Hospital A establishes a link with the telepresence apparatus located on the gurney. The remote physician can pan-tilt-zoom the image to obtain a desired view. If still unable to access the desired views, someone at the telepresence apparatus side can assist by repositioning the telepresence apparatus 100 and using the viewfinder to help position the front camera on the patient/desired view. The telepresence apparatus side team is able to communicate with the remote physician via the main speaker/mic on the unit. The remote physician may speak with various members of the team and patient/family at Hospital B to make a recommendation. In the event of noisy environment, or privacy situation, a Bluetooth or wireless headset can be used as an alternative. The remote physician is able to help with decisions regarding care/transport of the patient.
[0048] Care can be advanced either through decision to continue transport, to not continue transport, or to administer certain care as determined by the remote physician in collaboration with the onsite team. The consult can also occur during transport if there are situations where the patient starts to decompensate. In this case the link would be between a remote station and a telepresence apparatus located in the ambulance during transport of the patient. The telepresence apparatus would be mounted on a gurney, the remote physician can view the patient and communicate with the transport team to help make a care decision.
[0049] Another application may include a nurse conducting a scheduled visit to a chronically ill patient in their home. The nurse views the patient. The touch screen of the telepresence apparatus can be used to document various symptoms. The data is stored in the telepresence apparatus. The data and video of certain patient interactions can be forwarded to a server. The telepresence apparatus may receive requested information from the server. The nurse may observe a troubling symptom and request a physician consult. The nurse may call the physician, who establishes a link with the telepresence apparatus and initiates a telehealth session with the patient, facilitated by the nurse. The physician may request that the nurse attach a digital stethoscope to the telepresence apparatus and apply it to the patient. The physician may then request that the nurse attach a portable ultrasound device to the auxiliary video port of the telepresence apparatus. Finally, the physician may decide that the patient should be taken immediately to a medical facility. The nurse may call the ambulance. The nurse stays by the patient's side, with the remote physician logged into the telepresence apparatus, as the patient is transported to the facility.
[0050] It will be apparent to those having skill in the art that changes may be made to the details of the above-described embodiments without departing from the underlying principles of the invention. While certain exemplary embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, and that this invention not be limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those ordinarily skilled in the art. In the claims, the conjunction "and" is inclusive, the conjunction "or" is exclusive and the conjunction "and/or" is either inclusive or exclusive. Embodiments of the invention in which an exclusive property or privilege is claimed are defined as follows.

Claims

Claims
1 . A portable telepresence apparatus that is adapted to be coupled to a remote station that has a station monitor, a station camera, a station speaker and a station microphone, comprising:
a housing;
a first camera coupled to said housing;
a monitor that is coupled to said housing and is adapted to display images captured by the station camera;
a speaker that is coupled to said housing and is adapted to generate a sound provided through the station microphone;
a microphone coupled to said housing;
a battery coupled to said housing;
a wireless transceiver coupled to said housing; and
a viewfinder screen coupled to said housing.
2. The portable telepresence apparatus of claim 1 , further comprising a second camera coupled to said housing, said first camera being located on a first face of said housing that includes said monitor and said second camera being location on a second face of said housing that includes said viewfinder screen.
3. The portable telepresence apparatus of claim 1 , wherein said viewfinder screen includes at least one touch screen function that can vary an image captured by said first camera.
4. The portable telepresence apparatus of claim 1 , further comprising a motion sensing device attached to said housing.
5. The portable telepresence apparatus of claim 4, wherein said motion sensing device is utilized to correct an image displayed by said monitor.
6. The portable telepresence apparatus of claim 4, wherein said motion sensing device is utilized to correct an image provided to the remote station.
7. The portable telepresence apparatus of claim 1 , wherein said monitor includes a graphical user interface that allows a user to vary an audio characteristic.
8. The portable telepresence apparatus of claim 1 , wherein said housing is configured to be placed on a surface in an upright position.
9. The portable telepresence apparatus of claim 1 , wherein said remote station monitor displays hardware icons and depicts a break in a communication link between hardware devices.
10. The portable telepresence apparatus of claim 1 , further comprising an actuator system that can move the first camera and is controlled by the remote station.
1 1 . The portable telepresence apparatus of claim 1 , further comprising a GPS apparatus.
12. A portable telepresence apparatus that is adapted to be coupled to a remote station that has a station monitor, a station camera, a station speaker and a station microphone, comprising:
a housing;
a camera coupled to said housing;
a monitor that is coupled to said housing and is adapted to display images captured by the station camera;
a speaker that is coupled to said housing and is adapted to generate a sound provided through the station microphone;
a microphone coupled to said housing;
a battery coupled to said housing;
a wireless transceiver coupled to said housing; and
a memory device for capturing at least one image prior to establishing communication that is transmitted to the remote station after the remote station establishes a communication with the portable telepresence apparatus.
13. The portable telepresence apparatus of claim 12, further comprising at least one input that allows a user to vary an input characteristic of the portable telepresence apparatus before the remote station establishes the communication with the portable telepresence apparatus.
14. A portable telepresence apparatus that is adapted to be coupled to a remote station that has a station monitor, a station camera, a station speaker and a station microphone, comprising:
a housing;
a first camera coupled to said housing;
a monitor that is coupled to said housing and is adapted to display images captured by the station camera;
a speaker that is coupled to said housing and is adapted to generate a sound provided through the station microphone;
a microphone coupled to said housing;
a battery coupled to said housing;
a wireless transceiver coupled to said housing; and
a motion sensing device coupled to said housing.
15. The portable telepresence apparatus of claim 14, wherein said motion sensing device is utilized to correct an image displayed by said monitor.
16. The portable telepresence apparatus of claim 14, wherein said motion sensing device is utilized to correct an image provided to the remote station.
17. A portable telepresence apparatus that is adapted to be coupled to a remote station that has a station monitor, a station camera, a station speaker and a station microphone, comprising:
a housing;
a first camera coupled to said housing;
a monitor that is coupled to said housing and is adapted to display images captured by the station camera;
a speaker that is coupled to said housing and is adapted to generate a sound provided through the station microphone;
a microphone coupled to said housing;
a battery coupled to said housing; and
a wireless transceiver coupled to said housing, wherein said wireless transceiver transmits one type of data on a lower bandwidth and transmits a second type of data on a higher bandwidth network.
18. A method for providing a remote medical consultation, comprising: setting up a portable telepresence apparatus to view a patient, the portable telepresence apparatus includes a camera, a monitor, a speaker and a microphone; capturing an image of the patient with the portable telepresence apparatus; linking the portable telepresence apparatus to a remote station that includes a remote station camera that can capture an image of a healthcare worker operating the remote station, a remote station monitor, a remote station speaker and a remote station microphone;
transmitting the image of the healthcare worker to the portable telepresence apparatus;
displaying the healthcare worker image on the monitor;
transmitting the patient image to the remote station;
displaying the patient image on the remote station monitor; and
transmitting an audio command from the remote station to the portable telepresence apparatus.
19. The method of claim 18, further comprising storing the patient image and transmitting the stored patient image after the remote station is linked with the portable telepresence apparatus.
20. The method of claim 18, wherein the patient is being transported while the camera captures the patient image.
21 . The method of claim 20, further comprising terminating the link between the remote station and the portable telepresence apparatus and continuing to capture and store an image of the patient with the portable telepresence apparatus.
22. The method of claim 20, further comprising moving the portable telepresence apparatus to a first hospital.
23. The method of claim 22, further comprising moving the patient to a second hospital.
24. The method of claim 23, wherein the portable telepresence apparatus is moved with the patient to the second hospital.
25. The method of claim 24, wherein the portable telepresence apparatus transmits an image of the patient to the remote station while the patient is being moved to the second hospital.
26. The method of claim 25, wherein the remote station is located at the second hospital.
27. The method of claim 18, further comprising moving the portable telepresence apparatus to a home.
28. The method of claim 27, further comprising moving the patient to a medical facility.
29. The method of claim 28, further comprising storing data inputted into the portable telepresence apparatus and transmitting the data to the remote station.
30. The method of claim 28, further comprising attaching a medical instrument to the portable telepresence apparatus and obtaining patient data through the medical instrument.
31 . A method for obtaining a remotely captured image, comprising:
linking a portable telepresence apparatus to a remote station, the portable telepresence apparatus includes a first camera, a monitor, a speaker and a microphone, the remote station includes a remote station camera, a remote station monitor, a remote station speaker and a remote station microphone;
transmitting an image that is captured by the first camera to the remote station;
transmitting an audio instruction from the remote station to the portable telepresence apparatus to a user holding the portable telepresence apparatus; and moving the portable telepresence apparatus by the user holding the portable telepresence apparatus.
32. The method of claim 31 , further comprising viewing the image captured by the camera through a viewfinder screen on the portable telepresence apparatus.
33. The method of claim 32, further comprising capturing an image of the user holding the portable telepresence apparatus with a second camera and transmitting the image of the user to the remote station.
EP10814289.4A 2009-08-26 2010-08-26 Portable telepresence apparatus Withdrawn EP2470986A4 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US12/548,122 US8384755B2 (en) 2009-08-26 2009-08-26 Portable remote presence robot
US39963710P 2010-07-14 2010-07-14
PCT/US2010/046740 WO2011028589A2 (en) 2009-08-26 2010-08-26 Portable telepresence apparatus

Publications (2)

Publication Number Publication Date
EP2470986A2 true EP2470986A2 (en) 2012-07-04
EP2470986A4 EP2470986A4 (en) 2013-06-26

Family

ID=43649908

Family Applications (1)

Application Number Title Priority Date Filing Date
EP10814289.4A Withdrawn EP2470986A4 (en) 2009-08-26 2010-08-26 Portable telepresence apparatus

Country Status (6)

Country Link
EP (1) EP2470986A4 (en)
JP (1) JP2013503571A (en)
KR (1) KR20120068879A (en)
CN (2) CN102473081B (en)
IN (1) IN2012DN02197A (en)
WO (1) WO2011028589A2 (en)

Families Citing this family (48)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040162637A1 (en) 2002-07-25 2004-08-19 Yulun Wang Medical tele-robotic system with a master remote station with an arbitrator
US6925357B2 (en) 2002-07-25 2005-08-02 Intouch Health, Inc. Medical tele-robotic system
US7813836B2 (en) 2003-12-09 2010-10-12 Intouch Technologies, Inc. Protocol for a remotely controlled videoconferencing robot
US20050204438A1 (en) 2004-02-26 2005-09-15 Yulun Wang Graphical interface for a remote presence system
US8077963B2 (en) 2004-07-13 2011-12-13 Yulun Wang Mobile robot with a head-based movement mapping scheme
US9198728B2 (en) 2005-09-30 2015-12-01 Intouch Technologies, Inc. Multi-camera mobile teleconferencing platform
US8849679B2 (en) * 2006-06-15 2014-09-30 Intouch Technologies, Inc. Remote controlled robot system that provides medical images
US8265793B2 (en) 2007-03-20 2012-09-11 Irobot Corporation Mobile robot for telecommunication
US9160783B2 (en) 2007-05-09 2015-10-13 Intouch Technologies, Inc. Robot system that operates through a network firewall
US10875182B2 (en) 2008-03-20 2020-12-29 Teladoc Health, Inc. Remote presence system mounted to operating room hardware
US8179418B2 (en) 2008-04-14 2012-05-15 Intouch Technologies, Inc. Robotic based health care system
US8170241B2 (en) 2008-04-17 2012-05-01 Intouch Technologies, Inc. Mobile tele-presence system with a microphone system
US9193065B2 (en) 2008-07-10 2015-11-24 Intouch Technologies, Inc. Docking system for a tele-presence robot
US9842192B2 (en) 2008-07-11 2017-12-12 Intouch Technologies, Inc. Tele-presence robot system with multi-cast features
US8340819B2 (en) 2008-09-18 2012-12-25 Intouch Technologies, Inc. Mobile videoconferencing robot system with network adaptive driving
US8996165B2 (en) 2008-10-21 2015-03-31 Intouch Technologies, Inc. Telepresence robot with a camera boom
US8463435B2 (en) 2008-11-25 2013-06-11 Intouch Technologies, Inc. Server connectivity control for tele-presence robot
US9138891B2 (en) 2008-11-25 2015-09-22 Intouch Technologies, Inc. Server connectivity control for tele-presence robot
US8849680B2 (en) 2009-01-29 2014-09-30 Intouch Technologies, Inc. Documentation through a remote presence robot
US8897920B2 (en) 2009-04-17 2014-11-25 Intouch Technologies, Inc. Tele-presence robot system with software modularity, projector and laser pointer
US11399153B2 (en) 2009-08-26 2022-07-26 Teladoc Health, Inc. Portable telepresence apparatus
US8384755B2 (en) 2009-08-26 2013-02-26 Intouch Technologies, Inc. Portable remote presence robot
US11154981B2 (en) 2010-02-04 2021-10-26 Teladoc Health, Inc. Robot user interface for telepresence robot system
US8670017B2 (en) 2010-03-04 2014-03-11 Intouch Technologies, Inc. Remote presence system including a cart that supports a robot face and an overhead camera
US8918213B2 (en) 2010-05-20 2014-12-23 Irobot Corporation Mobile human interface robot
US8935005B2 (en) 2010-05-20 2015-01-13 Irobot Corporation Operating a mobile robot
US9014848B2 (en) 2010-05-20 2015-04-21 Irobot Corporation Mobile robot system
US10343283B2 (en) 2010-05-24 2019-07-09 Intouch Technologies, Inc. Telepresence robot system that can be accessed by a cellular phone
US10808882B2 (en) 2010-05-26 2020-10-20 Intouch Technologies, Inc. Tele-robotic system with a robot face placed on a chair
US9264664B2 (en) 2010-12-03 2016-02-16 Intouch Technologies, Inc. Systems and methods for dynamic bandwidth allocation
US8930019B2 (en) 2010-12-30 2015-01-06 Irobot Corporation Mobile human interface robot
JP5905031B2 (en) 2011-01-28 2016-04-20 インタッチ テクノロジーズ インコーポレイテッド Interfacing with mobile telepresence robot
US9323250B2 (en) 2011-01-28 2016-04-26 Intouch Technologies, Inc. Time-dependent navigation of telepresence robots
US10769739B2 (en) 2011-04-25 2020-09-08 Intouch Technologies, Inc. Systems and methods for management of information among medical providers and facilities
US20140139616A1 (en) 2012-01-27 2014-05-22 Intouch Technologies, Inc. Enhanced Diagnostics for a Telepresence Robot
US9098611B2 (en) 2012-11-26 2015-08-04 Intouch Technologies, Inc. Enhanced video interaction for a user interface of a telepresence network
US8836751B2 (en) 2011-11-08 2014-09-16 Intouch Technologies, Inc. Tele-presence system with a user interface that displays different communication links
US9251313B2 (en) 2012-04-11 2016-02-02 Intouch Technologies, Inc. Systems and methods for visualizing and managing telepresence devices in healthcare networks
US8902278B2 (en) 2012-04-11 2014-12-02 Intouch Technologies, Inc. Systems and methods for visualizing and managing telepresence devices in healthcare networks
US9361021B2 (en) 2012-05-22 2016-06-07 Irobot Corporation Graphical user interfaces including touchpad driving interfaces for telemedicine devices
EP2852475A4 (en) 2012-05-22 2016-01-20 Intouch Technologies Inc Social behavior rules for a medical telepresence robot
CN103051924A (en) * 2012-12-27 2013-04-17 宁波中研瑞华数字技术有限公司 Dual-mode third-generation (3G) video server
CN104063192B (en) * 2013-03-19 2017-05-24 联想(北京)有限公司 Display method and electronic equipment
US11862302B2 (en) 2017-04-24 2024-01-02 Teladoc Health, Inc. Automated transcription and documentation of tele-health encounters
US10483007B2 (en) 2017-07-25 2019-11-19 Intouch Technologies, Inc. Modular telehealth cart with thermal imaging and touch screen user interface
US11636944B2 (en) 2017-08-25 2023-04-25 Teladoc Health, Inc. Connectivity infrastructure for a telehealth platform
US10617299B2 (en) 2018-04-27 2020-04-14 Intouch Technologies, Inc. Telehealth cart that supports a removable tablet with seamless audio/video switching
WO2023150575A2 (en) * 2022-02-01 2023-08-10 The George Washington University Cyber-physical system to enhance usability and quality of telehealth consultation

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1304872A1 (en) * 2001-10-19 2003-04-23 Nokia Corporation An image stabiliser for a microcamera module of a handheld device and method for stabilising a microcamera module of a hand-held device
US20070025711A1 (en) * 2005-07-26 2007-02-01 Marcus Brian I Remote view and controller for a camera
EP1763243A2 (en) * 2005-09-09 2007-03-14 LG Electronics Inc. Image capturing and displaying method and system

Family Cites Families (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6369847B1 (en) * 2000-03-17 2002-04-09 Emtel, Inc. Emergency facility video-conferencing system
JP3513084B2 (en) * 2000-06-14 2004-03-31 株式会社東芝 Information processing system, information equipment and information processing method
US7539504B2 (en) * 2001-12-05 2009-05-26 Espre Solutions, Inc. Wireless telepresence collaboration system
JP3910112B2 (en) * 2002-06-21 2007-04-25 シャープ株式会社 Camera phone
US7593030B2 (en) * 2002-07-25 2009-09-22 Intouch Technologies, Inc. Tele-robotic videoconferencing in a corporate environment
US6925357B2 (en) * 2002-07-25 2005-08-02 Intouch Health, Inc. Medical tele-robotic system
US6753899B2 (en) * 2002-09-03 2004-06-22 Audisoft Method and apparatus for telepresence
US20060052676A1 (en) * 2004-09-07 2006-03-09 Yulun Wang Tele-presence system that allows for remote monitoring/observation and review of a patient and their medical records
CN1315351C (en) * 2004-09-20 2007-05-09 华为技术有限公司 Mobile communication terminal, mobile communication system and its method
KR100672338B1 (en) * 2005-09-09 2007-01-24 엘지전자 주식회사 Mobile communication terminal having dual display equipment and method of taking picture using same
US7643051B2 (en) * 2005-09-09 2010-01-05 Roy Benjamin Sandberg Mobile video teleconferencing system and control method
US20070291128A1 (en) * 2006-06-15 2007-12-20 Yulun Wang Mobile teleconferencing system that projects an image provided by a mobile robot
US7761185B2 (en) * 2006-10-03 2010-07-20 Intouch Technologies, Inc. Remote presence display through remotely controlled robot
US7557758B2 (en) * 2007-03-26 2009-07-07 Broadcom Corporation Very high frequency dielectric substrate wave guide

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1304872A1 (en) * 2001-10-19 2003-04-23 Nokia Corporation An image stabiliser for a microcamera module of a handheld device and method for stabilising a microcamera module of a hand-held device
US20070025711A1 (en) * 2005-07-26 2007-02-01 Marcus Brian I Remote view and controller for a camera
EP1763243A2 (en) * 2005-09-09 2007-03-14 LG Electronics Inc. Image capturing and displaying method and system

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Anonymous: "IPhone", Wikipedia, the free encyclopedia , XP002696350, Retrieved from the Internet: URL:http://en.wikipedia.org/wiki/Iphone [retrieved on 2013-04-30] *
See also references of WO2011028589A2 *

Also Published As

Publication number Publication date
IN2012DN02197A (en) 2015-08-07
CN102473081B (en) 2016-05-25
KR20120068879A (en) 2012-06-27
EP2470986A4 (en) 2013-06-26
JP2013503571A (en) 2013-01-31
CN102473081A (en) 2012-05-23
WO2011028589A3 (en) 2011-04-28
CN106406801A (en) 2017-02-15
WO2011028589A2 (en) 2011-03-10

Similar Documents

Publication Publication Date Title
US11399153B2 (en) Portable telepresence apparatus
US10871889B2 (en) Tele-presence system with a user interface that displays different communication links
WO2011028589A2 (en) Portable telepresence apparatus
US11389962B2 (en) Telepresence robot system that can be accessed by a cellular phone
US20210260749A1 (en) Graphical interface for a remote presence system
US20210366605A1 (en) Tele-presence robot system with multi-cast features
US20190088364A1 (en) Telepresence robot with a camera boom
US20060259193A1 (en) Telerobotic system with a dual application screen presentation
US20120072024A1 (en) Telerobotic system with dual application screen presentation
US20060052676A1 (en) Tele-presence system that allows for remote monitoring/observation and review of a patient and their medical records
WO2010065257A1 (en) A remote controlled robot system that provides medical images
JP2009540759A (en) Mobile electronic conferencing system for projecting images provided by mobile robots
JP6123961B1 (en) Terminal device of monitored person monitoring system, terminal side monitoring processing method of the terminal device, and monitored person monitoring system

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20120228

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

DAX Request for extension of the european patent (deleted)
RIN1 Information on inventor provided before grant (corrected)

Inventor name: SANCHEZ, DANIEL S.

Inventor name: LAI, FUJI

Inventor name: ROSENTHAL, JAMES

Inventor name: BUTNER, STEVEN E.

Inventor name: TEMBY, KELTON

Inventor name: WALTERS, DEREK J.

Inventor name: BOUGANIM, JOSHUA A.

Inventor name: HANRAHAN, KEVIN P.

Inventor name: JORDAN, CHARLES S.

REG Reference to a national code

Ref country code: HK

Ref legal event code: DE

Ref document number: 1172407

Country of ref document: HK

RIC1 Information provided on ipc code assigned before grant

Ipc: H04N 7/15 20060101ALI20130502BHEP

Ipc: G06F 3/14 20060101ALI20130502BHEP

Ipc: H04N 5/225 20060101ALI20130502BHEP

Ipc: H04N 5/232 20060101AFI20130502BHEP

A4 Supplementary search report drawn up and despatched

Effective date: 20130528

RIC1 Information provided on ipc code assigned before grant

Ipc: H04N 7/15 20060101ALI20130521BHEP

Ipc: G06F 3/14 20060101ALI20130521BHEP

Ipc: H04N 5/225 20060101ALI20130521BHEP

Ipc: H04N 5/232 20060101AFI20130521BHEP

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20140103