US7619607B2 - Embedding a pattern design onto a liquid crystal display - Google Patents

Embedding a pattern design onto a liquid crystal display Download PDF

Info

Publication number
US7619607B2
US7619607B2 US11/169,889 US16988905A US7619607B2 US 7619607 B2 US7619607 B2 US 7619607B2 US 16988905 A US16988905 A US 16988905A US 7619607 B2 US7619607 B2 US 7619607B2
Authority
US
United States
Prior art keywords
lcd
pixel
pattern
pixels
sub
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related, expires
Application number
US11/169,889
Other versions
US20070001950A1 (en
Inventor
Chunhui Zhang
Jian Wang
Qiang Wang
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Microsoft Technology Licensing LLC
Original Assignee
Microsoft Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Corp filed Critical Microsoft Corp
Priority to US11/169,889 priority Critical patent/US7619607B2/en
Assigned to MICROSOFT CORPORATION reassignment MICROSOFT CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: WANG, JIAN, WANG, QIANG, ZHANG, CHUNHUI
Publication of US20070001950A1 publication Critical patent/US20070001950A1/en
Application granted granted Critical
Publication of US7619607B2 publication Critical patent/US7619607B2/en
Assigned to MICROSOFT TECHNOLOGY LICENSING, LLC reassignment MICROSOFT TECHNOLOGY LICENSING, LLC ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: MICROSOFT CORPORATION
Expired - Fee Related legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/037Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor using the raster scan of a cathode-ray tube [CRT] for detecting the position of the member, e.g. light pens cooperating with CRT monitors

Definitions

  • a user will scan in the annotations and the original text, thereby creating a new document. These multiple steps make the interaction between the printed document and the electronic version of the document difficult to handle on a repeated basis. Further, scanned-in images are frequently non-modifiable. There may be no way to separate the annotations from the original text. This makes using the annotations difficult.
  • One technique of capturing handwritten information is by using a pen whose location may be determined during writing.
  • One type of pen functions by using a camera to capture an image of paper encoded with a predefined pattern. This pattern is used by the pen to determine a location of a pen on a piece of paper. However, the pattern is encoded onto a paper document. A user must still maintain a paper document for use with the pen to be uploaded to a personal computer.
  • aspects of the present invention are directed generally to presenting data on a medium for capture by an input device. More particularly, aspects of the present invention are directed to a method and system for embedding a symbol design, such as an embedded interaction code (EIC), in an image on a display screen, such as a liquid crystal display (LCD).
  • a grid with a plurality of pixels defines a size of an EIC pattern on an LCD.
  • An area of the grid is used to embed position data and/or metadata information. The area may be defined by a sub-pixel of the plurality of pixels and/or the edges of the grid itself.
  • the sub-pixel with the EIC pattern may be configured to have a grey value based upon a color of an underlying image associated with an application program being displayed on the LCD.
  • Another aspect of the present invention provides for decoding of the position data and/or metadata information in the EIC pattern on the LCD.
  • FIG. 1A illustrates a schematic diagram of a general-purpose digital computing environment in which certain aspects of the present invention may be implemented
  • FIGS. 1B through 1M show a general-purpose computer environment supporting one or more aspects of the present invention
  • FIG. 2 illustrates an example of an embedded interaction code (EIC) symbol composed of liquid crystal display (LCD) sub-pixels in accordance with at least one aspect of the present invention
  • FIG. 3 illustrates an example of liquid crystal cells and sub-pixels arranged in an LCD
  • FIGS. 4A and 4B illustrate examples of a symbol of 4 ⁇ 4 LCD pixels, where each sub-pixel may be embedded with 2 bits in accordance with at least one aspect of the present invention
  • FIG. 5 illustrates an example EIC pattern tiled with the 4 ⁇ 4 symbol of FIG. 4A when magnified in accordance with at least one aspect of the present invention
  • FIG. 6 illustrates another example of a symbol of 3 ⁇ 3 LCD pixels, where each sub-pixel may be embedded with 2 bits in accordance with at least one aspect of the present invention
  • FIG. 7 illustrates another example of a symbol of 4 ⁇ 5 LCD pixels, where each sub-pixel may be embedded with 3 bits in accordance with at least one aspect of the present invention
  • FIG. 8 illustrates another examples of a symbol of 4 ⁇ 4 LCD pixels, where the black matrix may be embedded with positioning data bits in accordance with at least one aspect of the present invention
  • FIGS. 9A-9C illustrate example EIC pattern images captured by a camera in accordance with at least one aspect of the present invention
  • FIGS. 10A and 10B illustrate a portion of an LCD screen with an EIC pattern watermarked on the screen in accordance with at least one aspect of the present invention
  • FIG. 11 illustrates an example array of embedded interaction codes in accordance with at least one aspect of the present invention
  • FIG. 12 illustrates example liquid crystal cells and sub-pixels arranged in an LCD, with four possible positions, in the red channel (R) of a sub-pixel for embedding an EIC pattern in accordance with at least one aspect of the present invention
  • FIG. 13 illustrates an example of a captured image by a camera in accordance with at least one aspect of the present invention
  • FIG. 14 illustrates an example of a normalized image of FIG. 13 with an effective LCD grid mask in accordance with at least one aspect of the present invention
  • FIG. 15 illustrates an example pixel index for determining orientation and position data in accordance with at least one aspect of the present invention
  • FIG. 16 illustrates an example of a mask M 1 with pixels that lie on the edge and vertex of LCD grids in accordance with at least one aspect of the present invention
  • FIG. 17 illustrates an example block diagram for defining pixels that neighbor a pixel (x, y) in accordance with at least one aspect of the present invention
  • FIG. 18 illustrates an example of a mask M 2 eroded from mask M 1 in accordance with at least one aspect of the present invention
  • FIG. 19 illustrates an example of a mask M 3 in accordance with at least one aspect of the present invention.
  • FIG. 20 illustrates an example of a mask M 4 in accordance with at least one aspect of the present invention
  • FIG. 21 illustrates an example of a mask M 5 in accordance with at least one aspect of the present invention
  • FIG. 22 illustrates an example of a mask M 6 in accordance with at least one aspect of the present invention
  • FIG. 23 illustrates an example of LCD-pixel grid analysis when applied to mask M 6 in accordance with at least one aspect of the present invention
  • FIG. 24 illustrates two coordinate systems in accordance with at least one aspect of the present invention.
  • FIG. 25 illustrates effective embedded interaction coding pixels in accordance with at least one aspect of the present invention
  • FIG. 26 illustrates an example LCD pixel grid showing 8 sampling points taken within the grid in accordance with at least one aspect of the present invention
  • FIGS. 27 and 28 illustrate an example of how a reference frame for the LCD pixel grids is redefined so that an axis is consistent with an ideal reference frame in accordance with at least one aspect of the present invention
  • FIG. 29 illustrates an example LCD pixel with the smallest average grey-level difference of four orientations for the LCD pixel as determined in accordance with at least one aspect of the present invention
  • FIGS. 30A-30D illustrate four interpretation bits in an EIC symbol in accordance with at least one aspect of the present invention
  • FIG. 31 illustrates example bit arrays that are constructed from retrieved bits in accordance with at least one aspect of the present invention
  • FIG. 32 is an illustrative block diagram of a system for creating an EIC pattern on an LCD in accordance with at least one aspect of the present invention.
  • FIG. 33 is a flowchart of an illustrative method for creating an EIC pattern on an LCD in accordance with at least one aspect of the present invention.
  • FIG. 34 is an illustrative block diagram of a system for decoding an EIC pattern on an LCD in accordance with at least one aspect of the present invention.
  • FIGS. 35-37 are flowcharts of illustrative methods for decoding an EIC pattern on an LCD in accordance with at least one aspect of the present invention.
  • FIG. 1 illustrates an example of a suitable computing system environment 100 on which the invention may be implemented.
  • the computing system environment 100 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the invention. Neither should the computing system environment 100 be interpreted as having any dependency nor requirement relating to any one or combination of components illustrated in the exemplary computing system environment 100 .
  • the invention is operational with numerous other general purpose or special purpose computing system environments or configurations.
  • Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
  • the invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer.
  • program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types.
  • the invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.
  • program modules may be located in both local and remote computer storage media including memory storage devices.
  • an exemplary system for implementing the invention includes a general-purpose computing device in the form of a computer 110 .
  • Components of computer 110 may include, but are not limited to, a processing unit 120 , a system memory 130 , and a system bus 121 that couples various system components including the system memory to the processing unit 120 .
  • the system bus 121 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures.
  • such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus also known as Mezzanine bus.
  • ISA Industry Standard Architecture
  • MCA Micro Channel Architecture
  • EISA Enhanced ISA
  • VESA Video Electronics Standards Association
  • PCI Peripheral Component Interconnect
  • Computer 110 typically includes a variety of computer readable media.
  • Computer readable media can be any available media that can be accessed by computer 110 and includes both volatile and nonvolatile media, removable and non-removable media.
  • Computer readable media may comprise computer storage media and communication media.
  • Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.
  • Computer storage media includes, but is not limited to, random access memory (RAM), read only memory (ROM), electronically erasable programmable read only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by computer 110 .
  • Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.
  • modulated data signal means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.
  • communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.
  • the system memory 130 includes computer storage media in the form of volatile and/or nonvolatile memory such as ROM 131 and RAM 132 .
  • RAM 132 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 120 .
  • FIG. 1A illustrates operating system 134 , application programs 135 , other program modules 136 , and program data 137 .
  • the computer 110 may also include other removable/non-removable, volatile/nonvolatile computer storage media.
  • FIG. 1A illustrates a hard disk drive 141 that reads from or writes to non-removable, nonvolatile magnetic media, a magnetic disk drive 151 that reads from or writes to a removable, nonvolatile magnetic disk 152 , and an optical disc drive 155 that reads from or writes to a removable, nonvolatile optical disc 156 such as a CD ROM or other optical media.
  • removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like.
  • the hard disk drive 141 is typically connected to the system bus 121 through a non-removable memory interface such as interface 140
  • magnetic disk drive 151 and optical disc drive 155 are typically connected to the system bus 121 by a removable memory interface, such as interface 150 .
  • hard disk drive 141 is illustrated as storing operating system 144 , application programs 145 , other program modules 146 , and program data 147 . Note that these components can either be the same as or different from operating system 134 , application programs 135 , other program modules 136 , and program data 137 . Operating system 144 , application programs 145 , other program modules 146 , and program data 147 are given different numbers here to illustrate that, at a minimum, they are different copies.
  • a user may enter commands and information into the computer 110 through input devices such as a digital camera (not shown), a keyboard 162 , and pointing device 161 , commonly referred to as a mouse, trackball or touch pad.
  • Other input devices may include a pen, stylus and tablet, microphone, joystick, game pad, satellite dish, scanner, or the like.
  • These and other input devices are often connected to the processing unit 120 through a user input interface 160 that is coupled to the system bus 121 , but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB).
  • a monitor 191 or other type of display device is also connected to the system bus 121 via an interface, such as a video interface 190 .
  • computers may also include other peripheral output devices such as speakers 197 and printer 196 , which may be connected through an output peripheral interface 195 .
  • a digitizer 165 and accompanying pen or stylus 166 are provided in order to digitally capture freehand input. Although a direct connection between the pen digitizer 165 and the user input interface 160 is shown, in practice, the digitizer 165 may be coupled to the processing unit 120 directly, via a parallel port or other interface and the system bus 121 as known in the art. Furthermore, although the digitizer 165 is shown apart from the monitor 191 , the usable input area of the digitizer 165 may be co-extensive with the display area of the monitor 191 . Further still, the digitizer 165 may be integrated in the monitor 191 , or may exist as a separate device overlaying or otherwise appended to the monitor 191 .
  • the computer 110 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 180 .
  • the remote computer 180 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 110 , although only a memory storage device 181 has been illustrated in FIG. 1A .
  • the logical connections depicted in FIG. 1A include a local area network (LAN) 171 and a wide area network (WAN) 173 , but may also include other networks.
  • LAN local area network
  • WAN wide area network
  • Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.
  • the computer 110 When used in a LAN networking environment, the computer 110 is connected to the LAN 171 through a network interface or adapter 170 .
  • the computer 110 When used in a WAN networking environment, the computer 110 typically includes a modem 172 or other means for establishing communications over the WAN 173 , such as the Internet.
  • the modem 172 which may be internal or external, may be connected to the system bus 121 via the user input interface 160 , or other appropriate mechanism.
  • program modules depicted relative to the computer 110 may be stored in the remote memory storage device.
  • FIG. 1A illustrates remote application programs 185 as residing on memory device 181 . It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.
  • network connections shown are exemplary and other means of establishing a communications link between the computers can be used.
  • the existence of any of various well-known protocols such as TCP/IP, Ethernet, FTP, HTTP and the like is presumed, and the system can be operated in a client-server configuration to permit a user to retrieve web pages from a web-based server.
  • Any of various conventional web browsers can be used to display and manipulate data on web pages.
  • a programming interface may be viewed as any mechanism, process, protocol for enabling one or more segment(s) of code to communicate with or access the functionality provided by one or more other segment(s) of code.
  • a programming interface may be viewed as one or more mechanism(s), method(s), function call(s), module(s), object(s), etc. of a component of a system capable of communicative coupling to one or more mechanism(s), method(s), function call(s), module(s), etc. of other component(s).
  • segment of code in the preceding sentence is intended to include one or more instructions or lines of code, and includes, e.g., code modules, objects, subroutines, functions, and so on, regardless of the terminology applied or whether the code segments are separately compiled, or whether the code segments are provided as source, intermediate, or object code, whether the code segments are utilized in a runtime system or process, or whether they are located on the same or different machines or distributed across multiple machines, or whether the functionality represented by the segments of code are implemented wholly in software, wholly in hardware, or a combination of hardware and software.
  • FIG. 1B illustrates an interface Interface 1 as a conduit through which first and second code segments communicate.
  • FIG. 1C illustrates an interface as comprising interface objects I 1 and I 2 (which may or may not be part of the first and second code segments), which enable first and second code segments of a system to communicate via medium M.
  • interface objects I 1 and I 2 are separate interfaces of the same system and one may also consider that objects I 1 and I 2 plus medium M comprise the interface.
  • API application programming interface
  • COM component object model
  • aspects of such a programming interface may include the method whereby the first code segment transmits information (where “information” is used in its broadest sense and includes data, commands, requests, etc.) to the second code segment; the method whereby the second code segment receives the information; and the structure, sequence, syntax, organization, schema, timing and content of the information.
  • the underlying transport medium itself may be unimportant to the operation of the interface, whether the medium be wired or wireless, or a combination of both, as long as the information is transported in the manner defined by the interface.
  • information may not be passed in one or both directions in the conventional sense, as the information transfer may be either via another mechanism (e.g. information placed in a buffer, file, etc.
  • a communication from one code segment to another may be accomplished indirectly by breaking the communication into multiple discrete communications.
  • FIGS. 1D and 1E depicted schematically in FIGS. 1D and 1E .
  • some interfaces can be described in terms of divisible sets of functionality.
  • the interface functionality of FIGS. 1B and 1C may be factored to achieve the same result, just as one may mathematically provide 24, or 2 times 2 times 3 times 2.
  • the function provided by interface Interface 1 may be subdivided to convert the communications of the interface into multiple interfaces Interface 1 A, Interface 1 B, Interface 1 C, etc. while achieving the same result.
  • FIG. 1D the function provided by interface Interface 1 may be subdivided to convert the communications of the interface into multiple interfaces Interface 1 A, Interface 1 B, Interface 1 C, etc. while achieving the same result.
  • interface I 1 may be subdivided into multiple interfaces I 1 a , I 1 b , I 1 c , etc. while achieving the same result.
  • interface I 2 of the second code segment which receives information from the first code segment may be factored into multiple interfaces I 2 a , I 2 b , I 2 c , etc.
  • the number of interfaces included with the 1st code segment need not match the number of interfaces included with the 2nd code segment.
  • FIGS. 1D and 1E the functional spirit of interfaces Interface 1 and I 1 remain the same as with FIGS. 1B and 1C , respectively.
  • the factoring of interfaces may also follow associative, commutative, and other mathematical properties such that the factoring may be difficult to recognize. For instance, ordering of operations may be unimportant, and consequently, a function carried out by an interface may be carried out well in advance of reaching the interface, by another piece of code or interface, or performed by a separate component of the system. Moreover, one of ordinary skill in the programming arts can appreciate that there are a variety of ways of making different function calls that achieve the same result.
  • FIGS. 1F and 1G it may be possible to ignore, add or redefine certain aspects (e.g., parameters) of a programming interface while still accomplishing the intended result.
  • interface Interface 1 of FIG. 1B includes a function call Square (input, precision, output), a call that includes three parameters, input, precision and output, and which is issued from the 1st Code Segment to the 2nd Code Segment. If the middle parameter precision is of no concern in a given scenario, as shown in FIG. 1F , it could just as well be ignored or even replaced with a meaningless (in this situation) parameter. One may also add an additional parameter of no concern.
  • the functionality of square can be achieved, so long as output is returned after input is squared by the second code segment.
  • Precision may very well be a meaningful parameter to some downstream or other portion of the computing system; however, once it is recognized that precision is not necessary for the narrow purpose of calculating the square, it may be replaced or ignored. For example, instead of passing a valid precision value, a meaningless value such as a birth date could be passed without adversely affecting the result.
  • interface I 1 is replaced by interface I 1 ′, redefined to ignore or add parameters to the interface.
  • Interface I 2 may similarly be redefined as interface I 2 ′, redefined to ignore unnecessary parameters, or parameters that may be processed elsewhere.
  • a programming interface may include aspects, such as parameters, which are not needed for some purpose, and so they may be ignored or redefined, or processed elsewhere for other purposes.
  • FIGS. 1B and 1C may be converted to the functionality of FIGS. 1H and 1I , respectively.
  • FIG. 1H the previous 1st and 2nd Code Segments of FIG. 1B are merged into a module containing both of them.
  • the code segments may still be communicating with each other but the interface may be adapted to a form which is more suitable to the single module.
  • formal Call and Return statements may no longer be necessary, but similar processing or response(s) pursuant to interface Interface 1 may still be in effect.
  • FIG. 11 part (or all) of interface 12 from FIG.
  • interface I 1C may be written inline into interface I 1 to form interface I 1 ′′.
  • interface I 2 is divided into I 2 a and I 2 b , and interface portion I 2 a has been coded in-line with interface I 1 to form interface I 1 ′′.
  • interface I 1 from FIG. 1C performs a function call square (input, output), which is received by interface I 2 , which after processing the value passed with input (to square it) by the second code segment, passes back the squared result with output.
  • the processing performed by the second code segment (squaring input) can be performed by the first code segment without a call to the interface.
  • a communication from one code segment to another may be accomplished indirectly by breaking the communication into multiple discrete communications. This is depicted schematically in FIGS. 1J and 1K .
  • one or more piece(s) of middleware (Divorce Interface(s), since they divorce functionality and/or interface functions from the original interface) are provided to convert the communications on the first interface, Interface 1 , to conform them to a different interface, in this case interfaces Interface 2 A, Interface 2 B and Interface 2 C.
  • a third code segment can be introduced with divorce interface DI 1 to receive the communications from interface I 1 and with divorce interface D 12 to transmit the interface functionality to, for example, interfaces I 2 a and I 2 b , redesigned to work with DI 2 , but to provide the same functional result.
  • DI 1 and DI 2 may work together to translate the functionality of interfaces I 1 and I 2 of FIG. 1C to a new operating system, while providing the same or similar functional result.
  • Yet another possible variant is to dynamically rewrite the code to replace the interface functionality with something else but which achieves the same overall result.
  • a code segment presented in an intermediate language e.g. Microsoft IL, Java ByteCode, etc.
  • JIT Just-in-Time
  • the JIT compiler may be written so as to dynamically convert the communications from the 1st Code Segment to the 2nd Code Segment, i.e., to conform them to a different interface as may be required by the 2nd Code Segment (either the original or a different 2nd Code Segment).
  • FIGS. 1L and 1M This is depicted in FIGS. 1L and 1M .
  • this approach is similar to the Divorce scenario described above. It might be done, e.g., where an installed base of applications are designed to communicate with an operating system in accordance with an Interface 1 protocol, but then the operating system is changed to use a different interface.
  • the JIT Compiler could be used to conform the communications on the fly from the installed-base applications to the new interface of the operating system.
  • this approach of dynamically rewriting the interface(s) may be applied to dynamically factor, or otherwise alter the interface(s) as well.
  • aspects of the present invention include placing an encoded data pattern in a liquid crystal display (LCD) form.
  • the encoded data stream may be represented as an encoded image overlying the displayed image or may be a physical encoded image on a display screen (so any image portion captured by a pen is locatable on the display screen).
  • the design pattern hides position information ad/or metadata information.
  • the pattern is displayed on an LCD screen, so when a camera samples pictures from the LCD screen, the camera or computing device associated with the camera may determine its position on the LCD screen accordingly and/or any metadata stored within the pattern.
  • aspects of the present invention provide an embedded interaction code (EIC) pattern that is embedded into sub pixels of an LCD, showing up temporally.
  • EIC embedded interaction code
  • An embedded interaction code (EIC) symbol is the smallest unit of visual representation of an EIC array, which is a single or multiple binary array that are used to represent x-y position and metadata.
  • An EIC symbol includes: one or more bits encoded in one symbol, visual representation of the bits, and physical size of the one symbol.
  • the EIC pattern design may be represented by LCD pixels or sub-pixels.
  • FIG. 2 is a sketch figure of an EIC symbol 200 composed of LCD sub-pixels, which represents the design of an EF-diamond-8 bit-a-16.
  • FIG. 2 illustrates a symbol of 12 ⁇ 4 LCD pixels, where each sub-pixel may be embedded with 8 bits.
  • An EF-diamond-8 bit-a-16 was originally designed for operation on printed documents.
  • dots 201 may be dots which are shown as clock dots
  • dots 203 may be dot positions which are left blank to help detect orientation of the EIC pattern
  • dots 205 may be dot positions which are used to embed a bit for x-y position and metadata.
  • An LCD screen utilizes the open and shutter operation of liquid crystal cells arranged uniformly in the screen to show images.
  • the operation of the lighting and non-lighting of pixels and sub-pixels are well known by those skilled in the art. Behind the crystal cells are many lamps that are lit when the displayer is electrified. If all the liquid crystal cells are open, back light comes into an individual's eyes fully, so the screen is white.
  • each pixel's state of being open or close may be determined through the analog to digital conversion of the display signals, so the picture may be shown by controlling the state of each LCD pixel. Because the lighting rate of the back lamp in an LCD is higher than the refresh rate of a cathode ray tube (CRT) device, the pictures on an LCD are very steady.
  • a displayer emits three kinds of light, namely red, green, and blue.
  • An LCD pixel 300 may be composed of an arrangement of 4 ⁇ 4 sub-pixels 310 .
  • One sub-pixel 310 is shown by a dashed line.
  • the red 311 , green 313 , and/or blue 315 channels may be used to embed code, which decreases the stimulus to eyes in spatial distribution as compared to the application of an EIC pattern onto a paper surface.
  • the LCD grid 320 may be used to detect the grid information.
  • a black dot used as clock dot in an EIC pattern on a paper surface may be neglected for application on an LCD screen. Sparser distribution of coded dots also contributes to a decreased affect on a user during normal reading of an LCD screen as described below.
  • an EIC pattern may be configured and displayed with the detection functionalities of a standard EIC paper based design pattern while being less sensitive to the human eye.
  • aspects of the present invention describe an EIC symbol pattern to embed into an EIC array.
  • Features of LCD technology may be used to decrease the algorithm complexity and affect on the human eye.
  • FIG. 4A illustrates a symbol of 4 ⁇ 4 LCD pixels, where each sub-pixel may be embedded with 2 bits.
  • FIG. 5 is an illustrative EIC pattern tiled with the 4 ⁇ 4 symbol when magnified 800%.
  • the red channel (R) influences intensity more than the green channel (G) or the blue channel (B), and is more sensitive to a camera.
  • the color red is more sensitive to the human eye compared to the color green or the color blue.
  • red channel R
  • green channel G
  • blue channel B
  • red channel (R) sub-pixel position when embedding code at a red channel (R) sub-pixel position, the gray value of red channel (R) equals zero and the grey value of green channel (G) and blue channel (B) equals 255 respectively.
  • the red channel (R) sub-pixel is dark with no light irradiating, whereas, the green channel (G) and blue channel (B) sub-pixels are fully open, and are shown as white.
  • the grey value of each of the red channel (R), green channel (G), and blue channel (B) may be changed based upon a color of an underlying image displayed on the LCD. Moreover, as shown in FIG.
  • each configuration such as the 4 ⁇ 4 configuration shown in FIG. 4A
  • the right-most column C Right and bottom-most row R Bottom may remain blank.
  • Such a design with the right most column and bottom most row remaining blank ensures that two embedded codes are not connected, potentially causing an error when attempting to read information.
  • the configuration to leave a column blank and a row blank of code may be different, such as shown in FIG. 4B where the left most column remains blank and the top most row remains blank of possible code positions. It should be understood by those skilled in the art that other configurations may be implemented in accordance with aspects of the present invention, including the use of all row and/or columns for positioning of code data.
  • FIGS. 6 and 7 illustrate other symbols of LCD pixels.
  • FIG. 6 illustrates a symbol of 3 ⁇ 3 LCD pixels, where each sub-pixel may be embedded with two (2) bits.
  • FIG. 7 illustrates a symbol of 4 ⁇ 5 LCD pixels, where each sub-pixel may be embedded with three (3) bits.
  • the size of the EIC pattern may be defined by a certain number of columns 1 to M and a certain number of rows 1 to N.
  • FIG. 6 illustrates an example of three columns and three rows
  • FIG. 7 illustrates an example of 5 columns and 4 rows.
  • the orientation of the EIC pattern is known if the LCD grid is known.
  • the grid of an LCD may be captured by a camera as shown in FIG. 9A . Because the left side of an LCD pixel value is dark and the right side is white, determination of the orientation of the pattern may be done. If the orientation of the pattern has been determined, then the black dots may be decoded in the symbol. Consequently, the position information and/or metadata information may be decoded as is done in paper-type EIC pattern applications.
  • an EIC symbol pattern may also be embedded into other areas for the sub-pixels.
  • an EIC symbol pattern may also be embedded into a black matrix, e.g., the LCD grid.
  • a black matrix e.g., the LCD grid.
  • an edge 831 of a sub-pixel may be darkened.
  • the code may be configured to include some of the region of the sub-pixel designated for a particular color channel.
  • one point may be to decrease the appearance of the pattern in temporal and spatial distribution, so as to decrease the stimulus to the human eye.
  • an overlay technology such as Microsoft® DirectX, by Microsoft® Corporation of Redmond, Wash., may be used to show the EIC pattern, periodically switching to the LCD screen surface.
  • the pattern is flashing.
  • the interchanging frequency depends on the refresh time of the particular LCD. For example, if the refresh time of an LCD screen is 12 ms, and the ratio to show the pattern versus screen content is set to 1:1, then the EIC pattern presentation frequency is approximately 40 Hz.
  • FIGS. 9A-9C illustrate three EIC pattern images captured by a camera.
  • FIG. 9A illustrates a captured EIC pattern image of symbols of 4 ⁇ 4 LCD pixels, where each sub-pixel may be embedded with 2 bits. Such a captured EIC pattern correlates to the EIC pattern symbol as shown in FIG. 4A .
  • FIG. 9B illustrates a captured EIC pattern image of symbols of 3 ⁇ 3 LCD pixels, where each sub-pixel may be embedded with 2 bits. Such a captured EIC pattern correlates to the EIC pattern symbol as shown in FIG. 6 .
  • FIG. 9C illustrates a captured EIC pattern image of symbols of 4 ⁇ 5 LCD pixels, where each sub-pixel may be embedded with 3 bits. Such a captured EIC pattern correlates to the EIC pattern symbol as shown in FIG. 7 . Image processing and decoding for aspects of the present invention are described more fully in detail below.
  • an image on an LCD screen may be watermarked with an EIC pattern.
  • the EIC pattern may not flash. Instead, the EIC pattern is updated when content refresh is refreshed on the LCD screen.
  • the process of watermarking may be performed in a memory of the LCD with graphics processing units (GPU) or by other means and methods.
  • FIGS. 10A and 10B a portion of an LCD screen is shown with an EIC pattern watermarked on the screen image.
  • the grey value of the sub-pixels used to embed code may be changed in accordance with the background color of the underlying image displayed on the LCD.
  • FIG. 10A illustrates a portion in which sub-pixels 1040 are shown with EIC code embedded therein.
  • FIG. 10B illustrates the portion in a magnified form to illustrate how one sub-pixel 1041 with an embedded EIC code has a different grey value from another sub-pixel 1043 with an embedded EIC code.
  • the grey value of sub-pixels used to embed the code is changed in accordance with the background color of the screen image displayed on the LCD.
  • the sub-pixel of that portion of the LCD screen for display of an image and embedded EIC code is changed as well in accordance with the background color of the underlying image.
  • Methods and systems for changing the color may include alpha-blending with a standard implemented color for the EIC code on a white background, varying the grey of the sub-pixel by a certain percentage with respect to the background image grey value, or other methods and systems.
  • FIG. 11 shows an array 1101 of EIC elements arranged into symbols 1103 that have the EIC elements located on gridlines 1104 .
  • An image 1102 is captured by a camera.
  • the EIC symbols 1103 and images of EIC dots can be analyzed to find the corresponding grid lines 1104 in image 1102 . If the grid lines 1104 in the image 1102 are found, the corresponding EIC symbols 1103 can be identified and the image 1102 located within array 1101 .
  • array 1101 is shown with a height of 2 14 ⁇ 1 and a width of 2 14 +1.
  • the size of the array can vary, of course, as well as the resulting information provided in each EIC symbol 1103 (where the smaller the array, the less information needed in EIC symbols to locate the image 1102 in the array 1101 ).
  • An m-array may be used to represent the X, Y position in an array and multiple m-arrays may be used to represent metadata. These multiple m-arrays may be encoded in EIC symbols. In each EIC symbol, one bit from each m-array may be encoded. EIC symbols in place of the bits of the m-array representing X, Y position and/or metadata information form an EIC symbol array (visually, the tiling of EIC symbols forms the EIC pattern).
  • FIG. 11 shows an EIC symbol array 1101 (of EIC symbol 8-a-16, in an m-array with the order of 28).
  • EIC pattern analysis includes two main steps. First, images may be processed to improve contrast or undergo other preprocessing. Next, features of an EIC pattern in the image are analyzed.
  • a pen-type input device may be used to analyze the image. The input device is assumed to capture images from a plane. The spatial transform converts the plane to a resulting image (which is also assumed to be a plane). This transformation may be a perspective transform.
  • an EIC pattern in image is expected to lie on grid lines 1104 that are a perspective transform of the grid lines in the EIC symbol array.
  • the perspective transform is first assumed to be an affine transform (i.e. evenly spaced parallel lines are kept evenly spaced and parallel, but perpendicular lines may not be perpendicular anymore). Rotation, scale and translation of the affine transform are estimated from analyzing the EIC pattern in image. The perspective transform can then be obtained by fitting an effective EIC pattern to the affine transformed grid lines.
  • an x-y position and metadata information may be embedded on an LCD screen and a pen-type device with a camera assembly may track the position by reading the embedded data.
  • position data and metadata information may be decoded from the images captured by a camera, such as a pen-type input device with a camera, when the camera moves over an LCD screen.
  • the position data and metadata information in an EIC pattern in an image may be decoded by a camera and/or a computer associated with a camera that captures the EIC pattern.
  • the algorithms described herein illustrate methods and systems for decoding 4 ⁇ 4 LCD pixels, where each sub-pixel may be embedded with 2 bits.
  • FIG. 12 illustrates liquid crystal cells and sub-pixels arranged in an LCD, with four possible positions, 1221 , 1223 , 1225 , and 1227 in the red channel (R) of a sub-pixel for embedding an EIC pattern.
  • the first step is vision processing of a captured image, where the input is the captured image and the output is an extracted bit stream.
  • the second step is decoding the x-y position data and/or metadata from the extracted bit stream.
  • FIG. 13 illustrates a sample of a captured image. Certain aspects of the present invention with respect to decoding data from a captured image will utilize the image captured in FIG. 13 .
  • the vision processing of a captured image may contain three steps. The steps are pre-processing, LCD-pixel grid analysis, and EIC symbol recognition.
  • One aspect of the step of pre-processing is to obtain effective pixels that form the LCD grids, i.e., the LCD grid mask. All pixels on the LCD grids are not needed for further data processing, just the pixels near the grid vertex for the algorithm within the next step to determine the perspective transform of the LCD grids from the LCD screen plane to the sensor plane.
  • the input of the pre-processing step is a captured image, such as the image shown in FIG. 13 .
  • the output is an effective LCD grid mask.
  • FIG. 14 illustrates a normalized image of FIG. 13 with an effective LCD grid mask indicated by the white dotted pixels 1401 .
  • FIG. 15 illustrates an example pixel index utilized in the examples described below.
  • a morphological approach is used to separate the LCD grid mask.
  • pixels that lie on the edge and vertex of the LCD grids are separated.
  • the size of an LCD grid cell in a captured image is known roughly. Therefore, a rough ratio is determined so that the number of pixels that lie on the edge or vertex of the LCD grids in a small block may be estimated.
  • the ratio may be determined to be 2/3.
  • the grey level value of pixels that lie on the edge or vertex of the LCD grids is lower than the grey level values of the pixels that lie on the center of grids.
  • the captured image may be divided into small blocks, e.g., the block size may be 6 ⁇ 6, and the darkest 24 pixels (2/3*6 ⁇ 6) may be selected as the estimated pixels that form the LCD grids.
  • M 1 is a binary array the same size as the captured image; 1 means valid and 0 means invalid. If a pixel (x, y) lies on an edge or vertex of the LCD grids, then M 1 (x, y) is equal to 1, else M 1 (x, y) is equal to 0.
  • the next step is the binary erosion of the mask M 1 .
  • a binary erosion operator may be used in this step.
  • the input of the operator is the mask M 1 obtained in the last step and the output is a new mask M 2 .
  • M 2 (x, y) is equal to 1 if M 1 (x, y) is equal to 1 and there are 3 or more neighboring pixels of (x, y), which equal to 1, else M 2 (x, y) is equal to 0.
  • FIG. 17 shows neighboring pixels in accordance with aspects of the present invention.
  • Pixel 1701 is a given pixel (x, y).
  • Pixels 1703 , 1705 , 1707 , and 1709 are neighboring pixels of pixel (x, y) 1701 .
  • Some neighboring pixels are shown in the cardinal directions (up, down, left, right) of pixel 1701 .
  • Others, pixels 1702 , 1704 , 1706 , and 1708 are on the diagonals from pixel 1701 .
  • By using a binary erosion operator many pixels that lie on the edge of grids are thrown off, as shown by mask M 2 in FIG. 18 .
  • the third step of pre-processing includes separating large connected regions of mask M 2 .
  • Two new masks M 3 and M 4 are obtained by judging the size of connected regions of the mask M 2 .
  • Mask M 3 (x, y) is equal to 1 if pixel (x, y) belongs to the small size, e.g., size ⁇ 15, connected regions of mask M 2 .
  • Mask M 4 (x, y) is equal to 1 if (x, y) belongs to the large size, e.g., size ⁇ 15, connected regions of mask M 2 .
  • R t ⁇ (x i , y i )
  • M 2 (x i , y i ) 1, (x i , y i ) are neighbors ⁇ , the size of a connected region means the count of pixels in the region.
  • mask M 3 (x, y) is equal to 1
  • the pixel (x, y) is considered to lie on the vertex of the LCD grids, as shown by the pixels in FIG. 19 .
  • the pixels that lie on the large connected regions, as represented by mask M 3 and as the pixels shown in FIG. 20 are further processed in the next step.
  • the fourth step of pre-processing includes thinning the mask M 4 .
  • Mask M 4 is processed by another binary erosion operator and a new mask M 5 is obtained.
  • Mask M 5 (x, y) is equal to 1 if M 4 (x, y) is equal to 1 and the value of 3 or 4 neighbor pixels of (x, y) in mask M 4 are equal to 1.
  • the pixels that belong to the small connected regions of M 5 form another mask M 6 .
  • Mask M 5 and mask M 6 are illustrated in FIGS. 21 and 22 respectively.
  • the LCD grid mask is obtained by merging the mask M 3 and mask M 6 , as shown in FIG. 14 .
  • LCD-pixel grid analysis The second step of vision processing, LCD-pixel grid analysis, is described below.
  • FIG. 23 shows, using the result of pre-processing, LCD-pixel grid analysis may be applied to retrieve the accurate perspective LCD pixel grid. It is understood by those skilled in the art that various methods may be used for LCD-pixel grid analysis. One example method for LCD-pixel grid analysis is described below.
  • EIC pattern analysis obtains a homography matrix describing the perspective transform from a grid to an image.
  • a perspective transform is a planar transform that is a perspective projection. With a perspective transform, a line is still a line. Parallel lines may not be parallel anymore. Instead, they may appear to converge to a vanishing point.
  • the affine transform is first obtained. Then, the affine transform is used as an initial estimate of perspective transform, and a perspective transform is determined. An affine transform keeps evenly spaced parallel lines evenly spaced and parallel, but perpendicular lines may not remain perpendicular. This step obtains the rotation, scale (distance between parallel lines) and translation (distance between the origins) of the affine transform. The output of this step is a homography matrix that describes the affine transform.
  • the system finds two main directions of EIC symbols. This step looks at all the directions formed by pairs of connected effective EIC pattern regions and finds two directions that occur the most often.
  • EIC Pattern Mask ( x i , y i ) 1, ( x i , y i ) are neighbors ⁇ .
  • gray-level centroid ( x t , y t ) is:
  • (x i , y i ) is a pixel in region R t
  • G(x i , y i ) is the gray-level of the pixel
  • N t is the total number of pixels in region R t .
  • FIG. 24 shows an example of ⁇ .
  • is a rotation about point 2401 from horizontal line 2402 to a line 2403 connecting centroids 2401 with another centroid 2404 .
  • the determination of angle ⁇ may be performed for all centroids from all centroids in the capture image.
  • the X axis is ⁇ .
  • the Y axis is the frequency count of ⁇ .
  • the X value that has the highest Y value is set as Candidate 1.
  • the reason for the suppression of the regions is that these angles are where EIC symbols are not expected to lie.
  • centroid of a small area near x i and x j is calculated:
  • the system determines the scale and translation for the EIC symbols.
  • Scale is the distance between two adjacent parallel lines.
  • Translation is the distance between the image center and the origin of the coordinate system formed by the grid lines. Both scale and translation are measured in pixels.
  • FIG. 25 shows the affine transformed grid lines.
  • the two main directions are: ⁇ h , ⁇ v .
  • the two scales along the two directions are: S h , S v .
  • the two translations are: ⁇ h , ⁇ v .
  • the H, V coordinate system shown in FIG. 25 is the coordinate system of the grid. Grid lines are not seen in image. But as explained before, if one assumes an affine transform is from a paper plane to an image plane, the EIC pattern in the image may appear to lie on grid lines that are an affine transform of the grid lines in the EIC symbol array. Therefore, one can draw grid lines in image (the gridlines parallel to lines 2501 and 2502 ) and the H, V coordinate system based on an affine transform of the grid lines in EIC symbol array.
  • the X, Y coordinate system shown in FIG. 25 is the coordinate system of the image. Facing the image, the X ⁇ is always horizontal and Y is always vertical. The image center is the origin of the X, Y coordinate system.
  • the image may be rotated counterclockwise with ⁇ h , ⁇ v respectively.
  • the perspective transform may be obtained from the affine transform.
  • the affine transform obtained from the affine transform.
  • three algorithms apply the grid and image to recognize the embedded EIC information: EIC symbol orientation; symbol analysis; and EIC bits extraction.
  • EIC symbol orientation in the image, steps may be utilized as described below.
  • each LCD pixel grid (i, j) eight (8) points of each LCD pixel are sampled.
  • the positions (h i,j,p , v i,j,p ) of the 8 sampling points G i,j,p (0 ⁇ p ⁇ 7) in each LCD pixel grid (i, j) are:
  • the LCD pixel grid is valid only if 8 sampling points are all valid.
  • the sampling algorithm may include bilinear sampling.
  • the threshold 25 is the average visible EIC symbol on an LCD in a captured image by a camera. It should be understood by those skilled in the art that the threshold value of 25 is one illustrative value, but other smaller or larger values may be utilized.
  • most possible correct orientation q which has the smallest D q (0 ⁇ q ⁇ 3), is determined. As shown in FIGS. 27 and 28 , the reference frame for the LCD pixel grids is redefined so that an axis is consistent with an ideal reference frame. Finally, the average grey-level difference D i,j,0 of all valid LCD pixel grids is recalculated in the new reference frame.
  • the second algorithm to recognize the embedded EIC information is symbol analysis.
  • EIC symbol orientation has been determined above; however, the EIC symbol on the LCD reference alignment remains unknown.
  • two algorithms may be used to retrieve horizontal and vertical alignment respectively.
  • the threshold 25 is the average visible EIC symbol on an LCD in a captured image by a camera. It should be understood by those skilled in the art that the threshold value of 25 is but one value and other smaller or larger values may be utilized.
  • EIC symbol horizontal alignment is calculated.
  • a histogram count H p (0 ⁇ p ⁇ 3) of the selected D i,j,0 is obtained by Mod(j,4).
  • the correct horizontal alignment position is obtained as the histogram index P with the biggest value H p (0 ⁇ p ⁇ 3).
  • EIC symbol vertical alignment is calculated.
  • a histogram count V p (0 ⁇ p ⁇ 3) of the selected D i,j,0 is obtained by Mod(i,4).
  • the correct vertical alignment position is obtained as the histogram index P with the biggest value V p (0 ⁇ p ⁇ 3).
  • the third algorithm to recognize the embedded EIC information is the EIC bits extraction algorithm, where EIC bits for each EIC symbol in the image are calculated.
  • EIC bits for each EIC symbol in the image are calculated.
  • the smallest D i,j,0 2601 of four possible information LCD pixels for each EIC symbol is identified.
  • two EIC bits (B 1 B 0 ) are retrieved from the EIC symbol as shown in FIGS. 30A-30D .
  • bit arrays are constructed from the retrieved bits such as shown in FIG. 31 .
  • the extracted EIC bits are a piece of single m-array bits that are used to decode x-y position data and/or metadata information. It should be understood by those skilled in the art that various methods may be used for EIC decoding of the EIC bit stream. One example method for EIC decoding of the EIC bit stream is described below.
  • x s /P n (x) and r(x)/P n (x) will generate the same m-sequence.
  • Q(x) 1
  • m can be assumed to be the m-sequence generated by 1/P n (x).
  • R also corresponds to r(x)/P n (x).
  • m (m 0 m 1 . . . m i . . . m 2 n ⁇ 3 m 2 n ⁇ 2 ) t (where the superscript t stands for vector or matrix transpose)
  • ⁇ i (m t ) (m 2 n ⁇ 1 ⁇ i m 2 n ⁇ i . . . m 0 . . . m 2 n ⁇ 3 ⁇ i m 2 n ⁇ 2 ⁇ i )
  • r(x)/P n (x) and 1/P n (x) will have the following relationship:
  • R t r t ⁇ circumflex over (M) ⁇
  • R is the m-sequence that starts from the s′-th bit of m
  • r (r 0 r 1 r 2 . . . r n ⁇ 1 ) t are the coefficients of r(x)
  • M ⁇ ( m t ⁇ ⁇ ( m t ) ⁇ ⁇ n - 1 ⁇ ( m t ) ) .
  • addition and multiplication operations are binary operations, i.e., addition is XOR and multiplication is AND.
  • an image cannot capture a set of bits b that does not contain error bits.
  • improper illumination, document content, dust and creases can all obscure the visual representation of bits in an image, preventing these bits from being recognized or causing the value of these bits to be improperly recognized.
  • the solution of r becomes difficult when there are error bits in b.
  • decoding becomes even more difficult because the coefficient matrix M is not fixed when the pen moves, changing the image from frame to frame. Moreover, the structure of M is irregular. Therefore, traditional decoding algorithms cannot effectively be applied to solve r under practical circumstances.
  • independent n bits i.e., the sub-matrix consisting of the corresponding columns of M is non-degenerate
  • n bits are randomly selected from the group of b that are captured in an image of a document.
  • b (0) are the n bits chosen
  • n bits chosen from b to make up b (0) can be moved to the beginning of b, with the remaining bits making up b moved to the end of b.
  • This leads to the relationship ([ b (0) ] t ,[ b (0) ] t ) [ r (0) ] t ( M (0) , M (0) )+(0 n t ,[e (0) ] t )
  • b (0) are the chosen n bits
  • b (0) are the remaining bits from the set b
  • M (0) is the corresponding columns of M for the chosen bits
  • M (0) is the corresponding columns of M for the remaining bits
  • 0 n t (0 0 . . .
  • the value (0 n t ,[e (0) ] t ) refers to the “difference vector” between ([b (0) ] t ,[ b (0) ] t ) and [r (0) ] t (M (0) , M (0) ), or simply the different vector of r (0) , and the number of 1's in (0 n t ,[e (0) ] t ) is called the number of different bits.
  • the vector containing different bits between ([b (0) ] t .[ b (0) ] t ) and [r (0) ] t (M (0) , M (0) ) alternately can be identified as D (0) .
  • D (0) (0 n t ,[e (0) ] t )
  • bits n from the set b are switched with some of the remaining bits from the set b.
  • J bit pairs (k j ,l j ) are switched between the original chosen bits n and the remaining bits from the set of bits b, where k 1 ⁇ k 2 ⁇ . . . ⁇ k J ⁇ n, n ⁇ l 1 ⁇ l 2 ⁇ . . . ⁇ l J ⁇ K.
  • bit order is redefined in ([b (0) ] t ,[ b (0) ] t ), and these bits are not maintained in their original order.
  • the relationship between the bits before and after switching is:
  • l J is equivalent to switching a smaller number of bit pairs, and therefore does not conform to the goal of switching J bits. It should be noted that, as long as the rank of E k t P (0) E p ⁇ n is J, the choice of l 1 , l 2 , . . . , l J will result in the identical location vector. Therefore, choosing one combination is sufficient. Moreover, as long as P R J (0) is invertible, the newly selected n bits are also independent.
  • E k t P (0) E l ⁇ n actually means choosing the k 1 , . . . , k J -th rows and l 1 ⁇ n, . . . l J ⁇ n-th columns of P (0)
  • 1 J t E k t P (0) actually means summing the k 1 , . . . , k J -th rows of P (0) . No matrix computation is needed.
  • an independent n-bit combination is generated from the group of bits b captured in an image. It should be noted that, with various embodiments of the invention, the selection of the n-bits can be combined with bit recognition confidence techniques, to help ensure that the most accurately recognized bits are selected for the n-bit combination.
  • the above process has to be repeated several times in order to ensure a high enough probability of successful decoding.
  • the number r of the error bits in b is first predicted to be d. If r is changed, then
  • s 2 is computed, such that 1 ⁇ (1 ⁇ P 2 ) s 2 ⁇ P e , where P e is the expected probability of successful decoding. If the times S of chosen b (0) is equal to or larger than s 2 , then the process is stopped and the results are output. Otherwise, the process is repeated with a new independent n-bit combination b (0) generated from the group of bits b captured in an image. Using this process, as long as the chosen n bits contain less than J+1 error bits, the correct solution can be found.
  • System 3200 is shown to include a design component 3201 , a processor component 3203 , an embedding component 3205 , an LCD 3207 , and a decoder component 3209 .
  • Design component 3201 may be configured to define a size 3221 of the EIC pattern for display on the LCD 3207 .
  • the size 3221 may be defined by a grid with pixels arranged into columns and rows.
  • Processor component 3203 may be configured to determine an area 3223 of the grid for embedding position data associated with the dimensions of the LCD 3207 .
  • Embedding component 3205 may be configured to embed the position data 3225 in the determined area of the grid.
  • LCD 3207 may be configured to display the position data in the EIC pattern on a screen.
  • LCD 3207 may be configured to intermittently display 3227 the EIC pattern and/or display the EIC pattern as a watermark 3237 on an underlying image associated with an application program.
  • Decoder component 3209 may be configured to decode the embedded position data and/or metadata information. Decoder component 3209 may be configured to include a camera 3229 configured to capture an image of the EIC pattern and an extractor 3239 configured to extract the embedded position data and/or metadata information from the captured image of the EIC pattern.
  • FIG. 33 is an illustrative method for creating an EIC pattern on an LCD in accordance with at least one aspect of the present invention.
  • the process starts at step 3301 where a size of the EIC pattern on an LCD is defined by a grid and pixels.
  • an area of the grid for embedding position data and/or metadata information is determined.
  • a determination is made at step 3305 as to whether the determined area is a pixel. If the determined area is a pixel, the process moves to step 3307 where a specific sub-pixel of the determined pixel is determined to embed the EIC pattern.
  • the position data and/or metadata information is embedded in the specific sub-pixel of the determined pixel.
  • Such a sub-pixel may be the red channel sub-pixel of a pixel. Proceeding to step 3311 , the grey value of the sub-pixel of the determined pixel may be changed based upon a background color of an underlying image currently displayed on the LCD screen.
  • the process then moves to step 3317 .
  • step 3313 the position data and/or metadata information is embedded in a portion of the black matrix of the grid itself. Such a portion may be an edge that separates two sub-pixels of one pixel or two sub-pixels of two different pixels. Proceeding to step 3315 , the grey value of the portion of the grid may be changed based upon a background color of an underlying image currently displayed on the LCD screen. The process then moves to step 3317 . At step 3317 , an orientation of the EIC pattern is determined. Moving to step 3319 , the position data and/or metadata information is decoded and the process ends.
  • System 3400 is shown to include an input component 3401 , a processor component 3403 , and a decoding component 3405 .
  • Input component 3401 may be configured to receive captured images.
  • Decoding component 3405 may be configured to decode position data and/or metadata information from an extracted bit stream to determine a position of the capture images with respect to a screen of the LCD.
  • Processor component 3403 may be configured to process the captured images and output an extracted bit stream.
  • Processor component may be configured to include a preprocessing component 3411 , a definition component 3413 , a recognition component 3415 , and an extracting component 3417 .
  • Preprocessing component 3411 may be configured to determine pixels corresponding to LCD pixel grids associated with the EIC pattern.
  • Definition component 3413 may be configured to define the LCD pixel grids based upon the determined pixels.
  • Recognition component 3415 may be configured to recognize the EIC pattern from the LCD pixel grids, and extracting component 3417 may be configured to extract a bit stream of EIC bits based upon the recognized EIC pattern.
  • Preprocessing component 3411 may further be configured to include a pixel separator, a binary erosion component, a separation component, a mask definer, and a mask merger.
  • the pixel separator may be configured to separate pixels that lie on edges and vertexes of the LCD pixel grids as a first mask.
  • the binary erosion component may be configured to erode the first mask to obtain a second mask, the second mask including fewer pixels than the first mask.
  • the separation component may be configured to separate connected regions of the second mask in accordance with a first threshold to obtain a third mask and a fourth mask.
  • the mask definer may be configured to define a sixth mask as pixels that belong to regions of the fifth mask, and the mask merger may be configured to merge the third and the sixth masks to define the LCD pixel grids.
  • the binary erosion component may be further configured to erode the fourth mask to obtain a fifth mask, and a second threshold may be defined to correspond to a region of the fifth mask.
  • Recognition component 3415 may further be configured to include an orientation component, an alignment retriever, and a calculation component.
  • the orientation component may be configured to determine an orientation of the EIC pattern in the captured image.
  • the alignment retriever may be configured to retrieve horizontal and vertical alignment information for the orientation, and the calculation component may be configured to calculate EIC bits for the EIC pattern.
  • FIG. 35 is an illustrative method for decoding an EIC pattern on an LCD in accordance with at least one aspect of the present invention.
  • the process starts at step 3501 where a captured image is received by the system.
  • the captured image is processed to output an extracted bit stream.
  • position data and/or metadata information is decoded from the extracted bit stream. Proceeding to step 3507 , a position of the captured image with respect to a screen of an LCD is determined and the process ends.
  • FIG. 36 provides an illustrative method for step 3503 shown in FIG. 35 in accordance with at least one aspect of the present invention.
  • the process starts at step 3601 where the captured image is preprocessed to determine pixels corresponding to LCD pixel grids associated with the EIC pattern.
  • the LCD pixel grids are defined based upon the pixels.
  • the EIC pattern is recognized from the LCD pixel grids.
  • a bit stream of EIC bits based upon the recognized EIC pattern is extracted at step 3607 and the process ends.
  • FIG. 37 provides an illustrative method for step 3601 shown in FIG. 36 in accordance with at least one aspect of the present invention.
  • the process starts at step 3701 where pixels that lie on edges or vertexes of the LCD grids are separated as a first mask.
  • a binary erosion operation is performed on the first mask to obtain a second mask. In this step, fewer pixels remain in the second mask compared to the first mask.
  • connected regions of the second mask are separated in accordance with a first threshold to obtain a third mask and a fourth mask.
  • Another binary erosion operation is performed at step 3707 on the fourth mask to obtain a fifth mask.
  • a second threshold corresponding to a region of the fifth mask is defined.
  • a sixth mask is defined as pixels that belong to the regions of the fifth mask.
  • the third and the sixth masks are merged to define the LCD pixel grids and the process ends.
  • a software architecture for processing data representative of positioning information on an LCD screen and/or metadata information may include a component configured to recognize the position of an input device with respect to an area of an LCD screen and an application program interface to access the component.
  • An API may receive a request to recognize the position of an input device, access the necessary function(s) to perform the operation, and then send the results back to an operating system.
  • the operating system may use the data provided from the API to perform the various features of the present invention.
  • the component may be configured to process a captured image to output an extracted bit stream, to decode position data and/or metadata information from the extracted bit stream, and to determine a position of the captured image with respect to a screen of an LCD.
  • the computer system may include at least one computer such as a microprocessor, digital signal processor, and associated peripheral electronic circuitry.

Abstract

A method and system for presenting data on a medium for capture by an input device is described. The method embeds a symbol design, such as an embedded interaction code (EIC), in an image on a display screen, such as a liquid crystal display (LCD). A grid with a plurality of pixels defines a size of an EIC pattern on an LCD. An area of the grid is used to embed position data and/or metadata information. The area may be defined by a sub-pixel of the plurality of pixels and/or the edges of the grid itself. The sub-pixel with the EIC pattern may be configured to have a grey value based upon a color of an underlying image associated with an application program being displayed on the LCD. The method also allows for decoding of the position data and/or metadata information in the EIC pattern on the LCD.

Description

BACKGROUND
The advent of the computer has boomed to a point of prolific use in a variety of different areas. Computers are used throughout the world for everything from controlling speed and direction in a vehicle, to making an online order for a new product, to allowing a young child to learn how to count or read. Within the business industry, the use of the computer has greatly increased the efficiency for performing tasks.
Computer users are accustomed to using a mouse and keyboard as a way of interacting with a personal computer. Despite the advances in technology, most computer users tend to use documents printed on paper as their primary editing tool. Some advantages of printed paper include its readability and portability. Others include the ability to share annotated paper documents and the ease at which one can archive printed paper. In the case of annotations, the printed document assumes a greater significance because of the annotations placed on it by the user. One of the difficulties, however, with having a printed document with annotations is the later need to have the annotations entered back into the electronic form of the document. This requires the original user or another user to wade through the annotations and enter them into a personal computer. In some cases, a user will scan in the annotations and the original text, thereby creating a new document. These multiple steps make the interaction between the printed document and the electronic version of the document difficult to handle on a repeated basis. Further, scanned-in images are frequently non-modifiable. There may be no way to separate the annotations from the original text. This makes using the annotations difficult.
Computer technology has advanced in some areas in order to attempt to address these problems. With the advent of the tablet style personal computer, a computer user can input handwritten strokes onto the display regions of the monitor. As such, annotations may be entered directly into a document. However, many problems exist with respect to tracking the movement of an input device for the handwritten annotations.
One technique of capturing handwritten information is by using a pen whose location may be determined during writing. One type of pen functions by using a camera to capture an image of paper encoded with a predefined pattern. This pattern is used by the pen to determine a location of a pen on a piece of paper. However, the pattern is encoded onto a paper document. A user must still maintain a paper document for use with the pen to be uploaded to a personal computer.
SUMMARY
Aspects of the present invention are directed generally to presenting data on a medium for capture by an input device. More particularly, aspects of the present invention are directed to a method and system for embedding a symbol design, such as an embedded interaction code (EIC), in an image on a display screen, such as a liquid crystal display (LCD). A grid with a plurality of pixels defines a size of an EIC pattern on an LCD. An area of the grid is used to embed position data and/or metadata information. The area may be defined by a sub-pixel of the plurality of pixels and/or the edges of the grid itself. The sub-pixel with the EIC pattern may be configured to have a grey value based upon a color of an underlying image associated with an application program being displayed on the LCD. Another aspect of the present invention provides for decoding of the position data and/or metadata information in the EIC pattern on the LCD.
BRIEF DESCRIPTION OF THE DRAWINGS
The foregoing summary of the invention, as well as the following detailed description of illustrative embodiments, is better understood when read in conjunction with the accompanying drawings, which are included by way of example, and not by way of limitation with regard to the claimed invention.
FIG. 1A illustrates a schematic diagram of a general-purpose digital computing environment in which certain aspects of the present invention may be implemented;
FIGS. 1B through 1M show a general-purpose computer environment supporting one or more aspects of the present invention;
FIG. 2 illustrates an example of an embedded interaction code (EIC) symbol composed of liquid crystal display (LCD) sub-pixels in accordance with at least one aspect of the present invention;
FIG. 3 illustrates an example of liquid crystal cells and sub-pixels arranged in an LCD;
FIGS. 4A and 4B illustrate examples of a symbol of 4×4 LCD pixels, where each sub-pixel may be embedded with 2 bits in accordance with at least one aspect of the present invention;
FIG. 5 illustrates an example EIC pattern tiled with the 4×4 symbol of FIG. 4A when magnified in accordance with at least one aspect of the present invention;
FIG. 6 illustrates another example of a symbol of 3×3 LCD pixels, where each sub-pixel may be embedded with 2 bits in accordance with at least one aspect of the present invention;
FIG. 7 illustrates another example of a symbol of 4×5 LCD pixels, where each sub-pixel may be embedded with 3 bits in accordance with at least one aspect of the present invention;
FIG. 8 illustrates another examples of a symbol of 4×4 LCD pixels, where the black matrix may be embedded with positioning data bits in accordance with at least one aspect of the present invention;
FIGS. 9A-9C illustrate example EIC pattern images captured by a camera in accordance with at least one aspect of the present invention;
FIGS. 10A and 10B illustrate a portion of an LCD screen with an EIC pattern watermarked on the screen in accordance with at least one aspect of the present invention;
FIG. 11 illustrates an example array of embedded interaction codes in accordance with at least one aspect of the present invention;
FIG. 12 illustrates example liquid crystal cells and sub-pixels arranged in an LCD, with four possible positions, in the red channel (R) of a sub-pixel for embedding an EIC pattern in accordance with at least one aspect of the present invention;
FIG. 13 illustrates an example of a captured image by a camera in accordance with at least one aspect of the present invention;
FIG. 14 illustrates an example of a normalized image of FIG. 13 with an effective LCD grid mask in accordance with at least one aspect of the present invention;
FIG. 15 illustrates an example pixel index for determining orientation and position data in accordance with at least one aspect of the present invention;
FIG. 16 illustrates an example of a mask M1 with pixels that lie on the edge and vertex of LCD grids in accordance with at least one aspect of the present invention;
FIG. 17 illustrates an example block diagram for defining pixels that neighbor a pixel (x, y) in accordance with at least one aspect of the present invention;
FIG. 18 illustrates an example of a mask M2 eroded from mask M1 in accordance with at least one aspect of the present invention;
FIG. 19 illustrates an example of a mask M3 in accordance with at least one aspect of the present invention;
FIG. 20 illustrates an example of a mask M4 in accordance with at least one aspect of the present invention;
FIG. 21 illustrates an example of a mask M5 in accordance with at least one aspect of the present invention;
FIG. 22 illustrates an example of a mask M6 in accordance with at least one aspect of the present invention;
FIG. 23 illustrates an example of LCD-pixel grid analysis when applied to mask M6 in accordance with at least one aspect of the present invention;
FIG. 24 illustrates two coordinate systems in accordance with at least one aspect of the present invention;
FIG. 25 illustrates effective embedded interaction coding pixels in accordance with at least one aspect of the present invention;
FIG. 26 illustrates an example LCD pixel grid showing 8 sampling points taken within the grid in accordance with at least one aspect of the present invention;
FIGS. 27 and 28 illustrate an example of how a reference frame for the LCD pixel grids is redefined so that an axis is consistent with an ideal reference frame in accordance with at least one aspect of the present invention;
FIG. 29 illustrates an example LCD pixel with the smallest average grey-level difference of four orientations for the LCD pixel as determined in accordance with at least one aspect of the present invention;
FIGS. 30A-30D illustrate four interpretation bits in an EIC symbol in accordance with at least one aspect of the present invention;
FIG. 31 illustrates example bit arrays that are constructed from retrieved bits in accordance with at least one aspect of the present invention;
FIG. 32 is an illustrative block diagram of a system for creating an EIC pattern on an LCD in accordance with at least one aspect of the present invention;
FIG. 33 is a flowchart of an illustrative method for creating an EIC pattern on an LCD in accordance with at least one aspect of the present invention;
FIG. 34 is an illustrative block diagram of a system for decoding an EIC pattern on an LCD in accordance with at least one aspect of the present invention; and
FIGS. 35-37 are flowcharts of illustrative methods for decoding an EIC pattern on an LCD in accordance with at least one aspect of the present invention.
DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS
In the following description of various illustrative embodiments, reference is made to the accompanying drawings, which form a part hereof, and in which are shown, by way of illustration, various embodiments in which the invention may be practiced. It is to be understood that other embodiments may be utilized and structural and functional modifications may be made without departing from the scope of the present invention.
Terms
  • Pen Any writing implement that may or may not include the ability to store ink. In some examples, a stylus with no ink capability may be used as a pen in accordance with embodiments of the present invention.
  • Camera An image capture system that may capture an image from paper, a computer display, or any other medium.
  • EIC Symbol The EIC symbol array is formed by EIC symbols in place of the bits of
  • Array the m-array representing X-Y position and metadata. Represents the physical space covered by the m-array representing X-Y position and metadata. EIC symbol array can be allocated to pages of a document.
  • Stroke An action made by the pen on a document or surface contain the EIC symbol array, where the array is captured by the camera during the performance of the action.
  • Grid The grid is a set of lines formed by EIC dots in EIC symbol array.
FIG. 1 illustrates an example of a suitable computing system environment 100 on which the invention may be implemented. The computing system environment 100 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the invention. Neither should the computing system environment 100 be interpreted as having any dependency nor requirement relating to any one or combination of components illustrated in the exemplary computing system environment 100.
The invention is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
With reference to FIG. 1A, an exemplary system for implementing the invention includes a general-purpose computing device in the form of a computer 110. Components of computer 110 may include, but are not limited to, a processing unit 120, a system memory 130, and a system bus 121 that couples various system components including the system memory to the processing unit 120. The system bus 121 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus also known as Mezzanine bus.
Computer 110 typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by computer 110 and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read only memory (ROM), electronically erasable programmable read only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by computer 110. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.
The system memory 130 includes computer storage media in the form of volatile and/or nonvolatile memory such as ROM 131 and RAM 132. A basic input/output system 133 (BIOS), containing the basic routines that help to transfer information between elements within computer 110, such as during start-up, is typically stored in ROM 131. RAM 132 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 120. By way of example, and not limitation, FIG. 1A illustrates operating system 134, application programs 135, other program modules 136, and program data 137.
The computer 110 may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only, FIG. 1A illustrates a hard disk drive 141 that reads from or writes to non-removable, nonvolatile magnetic media, a magnetic disk drive 151 that reads from or writes to a removable, nonvolatile magnetic disk 152, and an optical disc drive 155 that reads from or writes to a removable, nonvolatile optical disc 156 such as a CD ROM or other optical media. Other removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The hard disk drive 141 is typically connected to the system bus 121 through a non-removable memory interface such as interface 140, and magnetic disk drive 151 and optical disc drive 155 are typically connected to the system bus 121 by a removable memory interface, such as interface 150.
The drives and their associated computer storage media discussed above and illustrated in FIG. 1A, provide storage of computer readable instructions, data structures, program modules and other data for the computer 110. In FIG. 1A, for example, hard disk drive 141 is illustrated as storing operating system 144, application programs 145, other program modules 146, and program data 147. Note that these components can either be the same as or different from operating system 134, application programs 135, other program modules 136, and program data 137. Operating system 144, application programs 145, other program modules 146, and program data 147 are given different numbers here to illustrate that, at a minimum, they are different copies. A user may enter commands and information into the computer 110 through input devices such as a digital camera (not shown), a keyboard 162, and pointing device 161, commonly referred to as a mouse, trackball or touch pad. Other input devices (not shown) may include a pen, stylus and tablet, microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 120 through a user input interface 160 that is coupled to the system bus 121, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB). A monitor 191 or other type of display device is also connected to the system bus 121 via an interface, such as a video interface 190. In addition to the monitor, computers may also include other peripheral output devices such as speakers 197 and printer 196, which may be connected through an output peripheral interface 195. A digitizer 165 and accompanying pen or stylus 166 are provided in order to digitally capture freehand input. Although a direct connection between the pen digitizer 165 and the user input interface 160 is shown, in practice, the digitizer 165 may be coupled to the processing unit 120 directly, via a parallel port or other interface and the system bus 121 as known in the art. Furthermore, although the digitizer 165 is shown apart from the monitor 191, the usable input area of the digitizer 165 may be co-extensive with the display area of the monitor 191. Further still, the digitizer 165 may be integrated in the monitor 191, or may exist as a separate device overlaying or otherwise appended to the monitor 191.
The computer 110 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 180. The remote computer 180 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 110, although only a memory storage device 181 has been illustrated in FIG. 1A. The logical connections depicted in FIG. 1A include a local area network (LAN) 171 and a wide area network (WAN) 173, but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.
When used in a LAN networking environment, the computer 110 is connected to the LAN 171 through a network interface or adapter 170. When used in a WAN networking environment, the computer 110 typically includes a modem 172 or other means for establishing communications over the WAN 173, such as the Internet. The modem 172, which may be internal or external, may be connected to the system bus 121 via the user input interface 160, or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 110, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation, FIG. 1A illustrates remote application programs 185 as residing on memory device 181. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.
It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers can be used. The existence of any of various well-known protocols such as TCP/IP, Ethernet, FTP, HTTP and the like is presumed, and the system can be operated in a client-server configuration to permit a user to retrieve web pages from a web-based server. Any of various conventional web browsers can be used to display and manipulate data on web pages.
A programming interface (or more simply, interface) may be viewed as any mechanism, process, protocol for enabling one or more segment(s) of code to communicate with or access the functionality provided by one or more other segment(s) of code. Alternatively, a programming interface may be viewed as one or more mechanism(s), method(s), function call(s), module(s), object(s), etc. of a component of a system capable of communicative coupling to one or more mechanism(s), method(s), function call(s), module(s), etc. of other component(s). The term “segment of code” in the preceding sentence is intended to include one or more instructions or lines of code, and includes, e.g., code modules, objects, subroutines, functions, and so on, regardless of the terminology applied or whether the code segments are separately compiled, or whether the code segments are provided as source, intermediate, or object code, whether the code segments are utilized in a runtime system or process, or whether they are located on the same or different machines or distributed across multiple machines, or whether the functionality represented by the segments of code are implemented wholly in software, wholly in hardware, or a combination of hardware and software.
Notionally, a programming interface may be viewed generically, as shown in FIG. 1B or FIG. 1C. FIG. 1B illustrates an interface Interface1 as a conduit through which first and second code segments communicate. FIG. 1C illustrates an interface as comprising interface objects I1 and I2 (which may or may not be part of the first and second code segments), which enable first and second code segments of a system to communicate via medium M. In the view of FIG. 1C, one may consider interface objects I1 and I2 as separate interfaces of the same system and one may also consider that objects I1 and I2 plus medium M comprise the interface. Although FIGS. 1B and 1C show bi-directional flow and interfaces on each side of the flow, certain implementations may only have information flow in one direction (or no information flow as described below) or may only have an interface object on one side. By way of example, and not limitation, terms such as application programming interface (API), entry point, method, function, subroutine, remote procedure call, and component object model (COM) interface, are encompassed within the definition of programming interface.
Aspects of such a programming interface may include the method whereby the first code segment transmits information (where “information” is used in its broadest sense and includes data, commands, requests, etc.) to the second code segment; the method whereby the second code segment receives the information; and the structure, sequence, syntax, organization, schema, timing and content of the information. In this regard, the underlying transport medium itself may be unimportant to the operation of the interface, whether the medium be wired or wireless, or a combination of both, as long as the information is transported in the manner defined by the interface. In certain situations, information may not be passed in one or both directions in the conventional sense, as the information transfer may be either via another mechanism (e.g. information placed in a buffer, file, etc. separate from information flow between the code segments) or non-existent, as when one code segment simply accesses functionality performed by a second code segment. Any or all of these aspects may be important in a given situation, e.g., depending on whether the code segments are part of a system in a loosely coupled or tightly coupled configuration, and so this list should be considered illustrative and non-limiting.
This notion of a programming interface is known to those skilled in the art and is clear from the foregoing detailed description of the invention. There are, however, other ways to implement a programming interface, and, unless expressly excluded, these too are intended to be encompassed by the claims set forth at the end of this specification. Such other ways may appear to be more sophisticated or complex than the simplistic view of FIGS. 1B and 1C, but they nonetheless perform a similar function to accomplish the same overall result. We will now briefly describe some illustrative alternative implementations of a programming interface.
A. Factoring
A communication from one code segment to another may be accomplished indirectly by breaking the communication into multiple discrete communications. This is depicted schematically in FIGS. 1D and 1E. As shown, some interfaces can be described in terms of divisible sets of functionality. Thus, the interface functionality of FIGS. 1B and 1C may be factored to achieve the same result, just as one may mathematically provide 24, or 2 times 2 times 3 times 2. Accordingly, as illustrated in FIG. 1D, the function provided by interface Interface1 may be subdivided to convert the communications of the interface into multiple interfaces Interface1A, Interface1B, Interface1C, etc. while achieving the same result. As illustrated in FIG. 1E, the function provided by interface I1 may be subdivided into multiple interfaces I1 a, I1 b, I1 c, etc. while achieving the same result. Similarly, interface I2 of the second code segment which receives information from the first code segment may be factored into multiple interfaces I2 a, I2 b, I2 c, etc. When factoring, the number of interfaces included with the 1st code segment need not match the number of interfaces included with the 2nd code segment. In either of the cases of FIGS. 1D and 1E, the functional spirit of interfaces Interface1 and I1 remain the same as with FIGS. 1B and 1C, respectively. The factoring of interfaces may also follow associative, commutative, and other mathematical properties such that the factoring may be difficult to recognize. For instance, ordering of operations may be unimportant, and consequently, a function carried out by an interface may be carried out well in advance of reaching the interface, by another piece of code or interface, or performed by a separate component of the system. Moreover, one of ordinary skill in the programming arts can appreciate that there are a variety of ways of making different function calls that achieve the same result.
B. Redefinition
In some cases, it may be possible to ignore, add or redefine certain aspects (e.g., parameters) of a programming interface while still accomplishing the intended result. This is illustrated in FIGS. 1F and 1G. For example, assume interface Interface1 of FIG. 1B includes a function call Square (input, precision, output), a call that includes three parameters, input, precision and output, and which is issued from the 1st Code Segment to the 2nd Code Segment. If the middle parameter precision is of no concern in a given scenario, as shown in FIG. 1F, it could just as well be ignored or even replaced with a meaningless (in this situation) parameter. One may also add an additional parameter of no concern. In either event, the functionality of square can be achieved, so long as output is returned after input is squared by the second code segment. Precision may very well be a meaningful parameter to some downstream or other portion of the computing system; however, once it is recognized that precision is not necessary for the narrow purpose of calculating the square, it may be replaced or ignored. For example, instead of passing a valid precision value, a meaningless value such as a birth date could be passed without adversely affecting the result. Similarly, as shown in FIG. 1G, interface I1 is replaced by interface I1′, redefined to ignore or add parameters to the interface. Interface I2 may similarly be redefined as interface I2′, redefined to ignore unnecessary parameters, or parameters that may be processed elsewhere. The point here is that in some cases a programming interface may include aspects, such as parameters, which are not needed for some purpose, and so they may be ignored or redefined, or processed elsewhere for other purposes.
C. Inline Coding
It may also be feasible to merge some or all of the functionality of two separate code modules such that the “interface” between them changes form. For example, the functionality of FIGS. 1B and 1C may be converted to the functionality of FIGS. 1H and 1I, respectively. In FIG. 1H, the previous 1st and 2nd Code Segments of FIG. 1B are merged into a module containing both of them. In this case, the code segments may still be communicating with each other but the interface may be adapted to a form which is more suitable to the single module. Thus, for example, formal Call and Return statements may no longer be necessary, but similar processing or response(s) pursuant to interface Interface1 may still be in effect. Similarly, shown in FIG. 11, part (or all) of interface 12 from FIG. 1C may be written inline into interface I1 to form interface I1″. As illustrated, interface I2 is divided into I2 a and I2 b, and interface portion I2 a has been coded in-line with interface I1 to form interface I1″. For a concrete example, consider that the interface I1 from FIG. 1C performs a function call square (input, output), which is received by interface I2, which after processing the value passed with input (to square it) by the second code segment, passes back the squared result with output. In such a case, the processing performed by the second code segment (squaring input) can be performed by the first code segment without a call to the interface.
D. Divorce
A communication from one code segment to another may be accomplished indirectly by breaking the communication into multiple discrete communications. This is depicted schematically in FIGS. 1J and 1K. As shown in FIG. 1J, one or more piece(s) of middleware (Divorce Interface(s), since they divorce functionality and/or interface functions from the original interface) are provided to convert the communications on the first interface, Interface1, to conform them to a different interface, in this case interfaces Interface2A, Interface2B and Interface2C. This might be done, e.g., where there is an installed base of applications designed to communicate with, say, an operating system in accordance with an Interface1 protocol, but then the operating system is changed to use a different interface, in this case interfaces Interface2A, Interface2B and Interface2C. The point is that the original interface used by the 2nd Code Segment is changed such that it is no longer compatible with the interface used by the 1st Code Segment, and so an intermediary is used to make the old and new interfaces compatible. Similarly, as shown in FIG. 1K, a third code segment can be introduced with divorce interface DI1 to receive the communications from interface I1 and with divorce interface D12 to transmit the interface functionality to, for example, interfaces I2 a and I2 b, redesigned to work with DI2, but to provide the same functional result. Similarly, DI1 and DI2 may work together to translate the functionality of interfaces I1 and I2 of FIG. 1C to a new operating system, while providing the same or similar functional result.
E. Rewriting
Yet another possible variant is to dynamically rewrite the code to replace the interface functionality with something else but which achieves the same overall result. For example, there may be a system in which a code segment presented in an intermediate language (e.g. Microsoft IL, Java ByteCode, etc.) is provided to a Just-in-Time (JIT) compiler or interpreter in an execution environment (such as that provided by the .Net framework, the Java runtime environment, or other similar runtime type environments). The JIT compiler may be written so as to dynamically convert the communications from the 1st Code Segment to the 2nd Code Segment, i.e., to conform them to a different interface as may be required by the 2nd Code Segment (either the original or a different 2nd Code Segment). This is depicted in FIGS. 1L and 1M. As can be seen in FIG. 1L, this approach is similar to the Divorce scenario described above. It might be done, e.g., where an installed base of applications are designed to communicate with an operating system in accordance with an Interface1 protocol, but then the operating system is changed to use a different interface. The JIT Compiler could be used to conform the communications on the fly from the installed-base applications to the new interface of the operating system. As depicted in FIG. 1M, this approach of dynamically rewriting the interface(s) may be applied to dynamically factor, or otherwise alter the interface(s) as well.
It is also noted that the above-described scenarios for achieving the same or similar result as an interface via alternative embodiments may also be combined in various ways, serially and/or in parallel, or with other intervening code. Thus, the alternative embodiments presented above are not mutually exclusive and may be mixed, matched and combined to produce the same or equivalent scenarios to the generic scenarios presented in FIGS. 1B and 1C. It is also noted that, as with most programming constructs, there are other similar ways of achieving the same or similar functionality of an interface which may not be described herein, but nonetheless are represented by the spirit and scope of the invention, i.e., it is noted that it is at least partly the functionality represented by, and the advantageous results enabled by, an interface that underlie the value of an interface.
Aspects of the present invention include placing an encoded data pattern in a liquid crystal display (LCD) form. For example, the encoded data stream may be represented as an encoded image overlying the displayed image or may be a physical encoded image on a display screen (so any image portion captured by a pen is locatable on the display screen). In accordance with one aspect of the invention, the design pattern hides position information ad/or metadata information. The pattern is displayed on an LCD screen, so when a camera samples pictures from the LCD screen, the camera or computing device associated with the camera may determine its position on the LCD screen accordingly and/or any metadata stored within the pattern. Aspects of the present invention provide an embedded interaction code (EIC) pattern that is embedded into sub pixels of an LCD, showing up temporally.
An embedded interaction code (EIC) symbol is the smallest unit of visual representation of an EIC array, which is a single or multiple binary array that are used to represent x-y position and metadata. An EIC symbol includes: one or more bits encoded in one symbol, visual representation of the bits, and physical size of the one symbol. For LCD embodiments, the EIC pattern design may be represented by LCD pixels or sub-pixels. FIG. 2 is a sketch figure of an EIC symbol 200 composed of LCD sub-pixels, which represents the design of an EF-diamond-8 bit-a-16. FIG. 2 illustrates a symbol of 12×4 LCD pixels, where each sub-pixel may be embedded with 8 bits. An EF-diamond-8 bit-a-16 was originally designed for operation on printed documents. In accordance with at least one aspect of the present invention, such a symbol may also operate on LCD screen. As shown, dots 201 may be dots which are shown as clock dots, dots 203 may be dot positions which are left blank to help detect orientation of the EIC pattern, and dots 205 may be dot positions which are used to embed a bit for x-y position and metadata.
Although utilizing an EIC pattern on paper is known and used, there are issues when one attempts to directly use an EIC pattern for paper on an LCD. Some of these issues include the symbol size, the visual effect, and the algorithm complexity. As such, an EIC symbol for an LCD are redesigned and optimized in accordance with features of an LCD.
An LCD screen utilizes the open and shutter operation of liquid crystal cells arranged uniformly in the screen to show images. The operation of the lighting and non-lighting of pixels and sub-pixels are well known by those skilled in the art. Behind the crystal cells are many lamps that are lit when the displayer is electrified. If all the liquid crystal cells are open, back light comes into an individual's eyes fully, so the screen is white. When displaying, each pixel's state of being open or close may be determined through the analog to digital conversion of the display signals, so the picture may be shown by controlling the state of each LCD pixel. Because the lighting rate of the back lamp in an LCD is higher than the refresh rate of a cathode ray tube (CRT) device, the pictures on an LCD are very steady. A displayer emits three kinds of light, namely red, green, and blue.
The liquid crystal cells and sub-pixels arranged in the LCD are formed as shown in FIG. 3. An LCD pixel 300 may be composed of an arrangement of 4×4 sub-pixels 310. One sub-pixel 310 is shown by a dashed line. Within a sub-pixel 310, the red 311, green 313, and/or blue 315 channels may be used to embed code, which decreases the stimulus to eyes in spatial distribution as compared to the application of an EIC pattern onto a paper surface. The LCD grid 320 may be used to detect the grid information. A black dot used as clock dot in an EIC pattern on a paper surface may be neglected for application on an LCD screen. Sparser distribution of coded dots also contributes to a decreased affect on a user during normal reading of an LCD screen as described below.
For an LCD screen, an EIC pattern may be configured and displayed with the detection functionalities of a standard EIC paper based design pattern while being less sensitive to the human eye. Aspects of the present invention describe an EIC symbol pattern to embed into an EIC array. Features of LCD technology may be used to decrease the algorithm complexity and affect on the human eye.
FIG. 4A illustrates a symbol of 4×4 LCD pixels, where each sub-pixel may be embedded with 2 bits. FIG. 5 is an illustrative EIC pattern tiled with the 4×4 symbol when magnified 800%. Normally in computer graphics, the intensity formula I=0.59*R+0.3*G+0.11*B is used. The red channel (R) influences intensity more than the green channel (G) or the blue channel (B), and is more sensitive to a camera. Moreover, the color red is more sensitive to the human eye compared to the color green or the color blue. As such, implementations of the present invention are illustrated with the red channel (R) being used to embed code; however, it should be understood by those skilled in the art that the green channel, (G), blue channel (B), or combinations of the red channel, green channel and/or blue channel may be used and that the present invention is not so limited to the illustrations provided herein.
In one embodiment, when embedding code at a red channel (R) sub-pixel position, the gray value of red channel (R) equals zero and the grey value of green channel (G) and blue channel (B) equals 255 respectively. The red channel (R) sub-pixel is dark with no light irradiating, whereas, the green channel (G) and blue channel (B) sub-pixels are fully open, and are shown as white. In accordance with another embodiment, the grey value of each of the red channel (R), green channel (G), and blue channel (B) may be changed based upon a color of an underlying image displayed on the LCD. Moreover, as shown in FIG. 4A, in order to make the codes sparser, there is no black dot in the symbol design, and only one dot is shown of four possible positions, 421, 423, 425, and 427, in each symbol. For example, if the code is 00, then a dot is displayed at red channel (0, 0) sub-pixel position 421. If the code is 01, then a dot is displayed at red channel (0, 1) sub-pixel position 423.
In addition, in each configuration, such as the 4×4 configuration shown in FIG. 4A, the right-most column CRight and bottom-most row RBottom may remain blank. Such a design with the right most column and bottom most row remaining blank ensures that two embedded codes are not connected, potentially causing an error when attempting to read information. The configuration to leave a column blank and a row blank of code may be different, such as shown in FIG. 4B where the left most column remains blank and the top most row remains blank of possible code positions. It should be understood by those skilled in the art that other configurations may be implemented in accordance with aspects of the present invention, including the use of all row and/or columns for positioning of code data.
FIGS. 6 and 7 illustrate other symbols of LCD pixels. FIG. 6 illustrates a symbol of 3×3 LCD pixels, where each sub-pixel may be embedded with two (2) bits. FIG. 7 illustrates a symbol of 4×5 LCD pixels, where each sub-pixel may be embedded with three (3) bits. Although these are but some examples, it should be understood that one or more aspects of the present invention may be implemented within other configurations. For example, the size of the EIC pattern may be defined by a certain number of columns 1 to M and a certain number of rows 1 to N. FIG. 6 illustrates an example of three columns and three rows, while FIG. 7 illustrates an example of 5 columns and 4 rows.
Because the code is embedded into a sub-pixel, e.g., the red channel as shown in the illustrated Figures, the orientation of the EIC pattern is known if the LCD grid is known. In implementation, the grid of an LCD may be captured by a camera as shown in FIG. 9A. Because the left side of an LCD pixel value is dark and the right side is white, determination of the orientation of the pattern may be done. If the orientation of the pattern has been determined, then the black dots may be decoded in the symbol. Consequently, the position information and/or metadata information may be decoded as is done in paper-type EIC pattern applications.
In accordance with another aspect of the present invention, an EIC symbol pattern may also be embedded into other areas for the sub-pixels. For example, as shown in FIG. 8, an EIC symbol pattern may also be embedded into a black matrix, e.g., the LCD grid. As shown in FIG. 8, as opposed to positioning code within a color channel sub-pixel, such as the red channel (R) position in a sub-pixel, an edge 831 of a sub-pixel may be darkened. The code may be configured to include some of the region of the sub-pixel designated for a particular color channel.
With respect to actually embedding an EIC pattern on an LCD screen, one point may be to decrease the appearance of the pattern in temporal and spatial distribution, so as to decrease the stimulus to the human eye. In accordance with one implementation, an overlay technology, such as Microsoft® DirectX, by Microsoft® Corporation of Redmond, Wash., may be used to show the EIC pattern, periodically switching to the LCD screen surface. In such an implementation, the pattern is flashing. The interchanging frequency depends on the refresh time of the particular LCD. For example, if the refresh time of an LCD screen is 12 ms, and the ratio to show the pattern versus screen content is set to 1:1, then the EIC pattern presentation frequency is approximately 40 Hz.
FIGS. 9A-9C illustrate three EIC pattern images captured by a camera. FIG. 9A illustrates a captured EIC pattern image of symbols of 4×4 LCD pixels, where each sub-pixel may be embedded with 2 bits. Such a captured EIC pattern correlates to the EIC pattern symbol as shown in FIG. 4A. FIG. 9B illustrates a captured EIC pattern image of symbols of 3×3 LCD pixels, where each sub-pixel may be embedded with 2 bits. Such a captured EIC pattern correlates to the EIC pattern symbol as shown in FIG. 6. FIG. 9C illustrates a captured EIC pattern image of symbols of 4×5 LCD pixels, where each sub-pixel may be embedded with 3 bits. Such a captured EIC pattern correlates to the EIC pattern symbol as shown in FIG. 7. Image processing and decoding for aspects of the present invention are described more fully in detail below.
In accordance with another implementation of aspects of the present invention, an image on an LCD screen may be watermarked with an EIC pattern. In such an implementation, the EIC pattern may not flash. Instead, the EIC pattern is updated when content refresh is refreshed on the LCD screen. The process of watermarking may be performed in a memory of the LCD with graphics processing units (GPU) or by other means and methods.
As shown in FIGS. 10A and 10B, a portion of an LCD screen is shown with an EIC pattern watermarked on the screen image. The grey value of the sub-pixels used to embed code may be changed in accordance with the background color of the underlying image displayed on the LCD. For example, FIG. 10A illustrates a portion in which sub-pixels 1040 are shown with EIC code embedded therein. FIG. 10B illustrates the portion in a magnified form to illustrate how one sub-pixel 1041 with an embedded EIC code has a different grey value from another sub-pixel 1043 with an embedded EIC code. In the example shown, the grey value of sub-pixels used to embed the code is changed in accordance with the background color of the screen image displayed on the LCD. As such, if a user changes a general location of an image on the LCD display screen, the sub-pixel of that portion of the LCD screen for display of an image and embedded EIC code is changed as well in accordance with the background color of the underlying image. Methods and systems for changing the color may include alpha-blending with a standard implemented color for the EIC code on a white background, varying the grey of the sub-pixel by a certain percentage with respect to the background image grey value, or other methods and systems.
FIG. 11 shows an array 1101 of EIC elements arranged into symbols 1103 that have the EIC elements located on gridlines 1104. An image 1102 is captured by a camera. The EIC symbols 1103 and images of EIC dots can be analyzed to find the corresponding grid lines 1104 in image 1102. If the grid lines 1104 in the image 1102 are found, the corresponding EIC symbols 1103 can be identified and the image 1102 located within array 1101. Here, array 1101 is shown with a height of 214−1 and a width of 214+1. The size of the array can vary, of course, as well as the resulting information provided in each EIC symbol 1103 (where the smaller the array, the less information needed in EIC symbols to locate the image 1102 in the array 1101).
An m-array may be used to represent the X, Y position in an array and multiple m-arrays may be used to represent metadata. These multiple m-arrays may be encoded in EIC symbols. In each EIC symbol, one bit from each m-array may be encoded. EIC symbols in place of the bits of the m-array representing X, Y position and/or metadata information form an EIC symbol array (visually, the tiling of EIC symbols forms the EIC pattern). FIG. 11 shows an EIC symbol array 1101 (of EIC symbol 8-a-16, in an m-array with the order of 28).
EIC pattern analysis includes two main steps. First, images may be processed to improve contrast or undergo other preprocessing. Next, features of an EIC pattern in the image are analyzed. A pen-type input device may be used to analyze the image. The input device is assumed to capture images from a plane. The spatial transform converts the plane to a resulting image (which is also assumed to be a plane). This transformation may be a perspective transform. Here, an EIC pattern in image is expected to lie on grid lines 1104 that are a perspective transform of the grid lines in the EIC symbol array. The perspective transform is first assumed to be an affine transform (i.e. evenly spaced parallel lines are kept evenly spaced and parallel, but perpendicular lines may not be perpendicular anymore). Rotation, scale and translation of the affine transform are estimated from analyzing the EIC pattern in image. The perspective transform can then be obtained by fitting an effective EIC pattern to the affine transformed grid lines.
As described above, an x-y position and metadata information may be embedded on an LCD screen and a pen-type device with a camera assembly may track the position by reading the embedded data. In accordance with further aspects of the present invention, position data and metadata information may be decoded from the images captured by a camera, such as a pen-type input device with a camera, when the camera moves over an LCD screen. As such, the position data and metadata information in an EIC pattern in an image may be decoded by a camera and/or a computer associated with a camera that captures the EIC pattern. The algorithms described herein illustrate methods and systems for decoding 4×4 LCD pixels, where each sub-pixel may be embedded with 2 bits. FIG. 12 illustrates liquid crystal cells and sub-pixels arranged in an LCD, with four possible positions, 1221, 1223, 1225, and 1227 in the red channel (R) of a sub-pixel for embedding an EIC pattern.
In accordance with aspects of the present invention, there are two steps for decoding x-y position data and/or metadata information from captured images. The first step is vision processing of a captured image, where the input is the captured image and the output is an extracted bit stream. The second step is decoding the x-y position data and/or metadata from the extracted bit stream.
FIG. 13 illustrates a sample of a captured image. Certain aspects of the present invention with respect to decoding data from a captured image will utilize the image captured in FIG. 13. In accordance with these aspects of the present invention, the vision processing of a captured image may contain three steps. The steps are pre-processing, LCD-pixel grid analysis, and EIC symbol recognition.
One aspect of the step of pre-processing is to obtain effective pixels that form the LCD grids, i.e., the LCD grid mask. All pixels on the LCD grids are not needed for further data processing, just the pixels near the grid vertex for the algorithm within the next step to determine the perspective transform of the LCD grids from the LCD screen plane to the sensor plane.
As described above, the input of the pre-processing step is a captured image, such as the image shown in FIG. 13. The output is an effective LCD grid mask. FIG. 14 illustrates a normalized image of FIG. 13 with an effective LCD grid mask indicated by the white dotted pixels 1401.
In the descriptions below, an image sensor size of 128×100 pixels is utilized. Pixels are indexed from the top left corner. FIG. 15 illustrates an example pixel index utilized in the examples described below. In accordance with aspects of the present invention, a morphological approach is used to separate the LCD grid mask.
First, pixels that lie on the edge and vertex of the LCD grids are separated. The size of an LCD grid cell in a captured image is known roughly. Therefore, a rough ratio is determined so that the number of pixels that lie on the edge or vertex of the LCD grids in a small block may be estimated. For example, the ratio may be determined to be 2/3. The grey level value of pixels that lie on the edge or vertex of the LCD grids is lower than the grey level values of the pixels that lie on the center of grids. From this information, the captured image may be divided into small blocks, e.g., the block size may be 6×6, and the darkest 24 pixels (2/3*6×6) may be selected as the estimated pixels that form the LCD grids. FIG. 16 illustrates pixels that lie on the edge and vertex of the LCD grids. By dividing the image into small blocks, the pixels that lie on both edge and vertex of the LCD grids are separated as a mask M1. A mask such as M1 is a binary array the same size as the captured image; 1 means valid and 0 means invalid. If a pixel (x, y) lies on an edge or vertex of the LCD grids, then M1 (x, y) is equal to 1, else M1 (x, y) is equal to 0.
The next step is the binary erosion of the mask M1. A binary erosion operator may be used in this step. The input of the operator is the mask M1 obtained in the last step and the output is a new mask M2. For each pixel (x, y), M2(x, y) is equal to 1 if M1(x, y) is equal to 1 and there are 3 or more neighboring pixels of (x, y), which equal to 1, else M2(x, y) is equal to 0. The concept and associated algorithms of binary erosion is readily understood by those skilled in the art.
Two pixels are neighbors to each other if they are directly below, above, or next to each other. FIG. 17 shows neighboring pixels in accordance with aspects of the present invention. Pixel 1701 is a given pixel (x, y). Pixels 1703, 1705, 1707, and 1709 are neighboring pixels of pixel (x, y) 1701. Some neighboring pixels are shown in the cardinal directions (up, down, left, right) of pixel 1701. Others, pixels 1702, 1704, 1706, and 1708, are on the diagonals from pixel 1701. By using a binary erosion operator, many pixels that lie on the edge of grids are thrown off, as shown by mask M2 in FIG. 18.
The third step of pre-processing includes separating large connected regions of mask M2. Two new masks M3 and M4 are obtained by judging the size of connected regions of the mask M2. Mask M3(x, y) is equal to 1 if pixel (x, y) belongs to the small size, e.g., size <15, connected regions of mask M2. Mask M4(x, y) is equal to 1 if (x, y) belongs to the large size, e.g., size ≧15, connected regions of mask M2. Connected regions are identified as Rt, where t=1, 2, . . . , T,
Rt={(xi, yi)|M2(xi, yi)=1, (xi, yi) are neighbors}, the size of a connected region means the count of pixels in the region.
If mask M3(x, y) is equal to 1, the pixel (x, y) is considered to lie on the vertex of the LCD grids, as shown by the pixels in FIG. 19. The pixels that lie on the large connected regions, as represented by mask M3 and as the pixels shown in FIG. 20, are further processed in the next step.
The fourth step of pre-processing includes thinning the mask M4. Mask M4 is processed by another binary erosion operator and a new mask M5 is obtained. Mask M5(x, y) is equal to 1 if M4(x, y) is equal to 1 and the value of 3 or 4 neighbor pixels of (x, y) in mask M4 are equal to 1. The pixels that belong to the small connected regions of M5 form another mask M6. Mask M5 and mask M6 are illustrated in FIGS. 21 and 22 respectively. The LCD grid mask is obtained by merging the mask M3 and mask M6, as shown in FIG. 14.
The second step of vision processing, LCD-pixel grid analysis, is described below. As FIG. 23 shows, using the result of pre-processing, LCD-pixel grid analysis may be applied to retrieve the accurate perspective LCD pixel grid. It is understood by those skilled in the art that various methods may be used for LCD-pixel grid analysis. One example method for LCD-pixel grid analysis is described below.
EIC pattern analysis obtains a homography matrix describing the perspective transform from a grid to an image. A perspective transform is a planar transform that is a perspective projection. With a perspective transform, a line is still a line. Parallel lines may not be parallel anymore. Instead, they may appear to converge to a vanishing point. To obtain the perspective transform, the affine transform is first obtained. Then, the affine transform is used as an initial estimate of perspective transform, and a perspective transform is determined. An affine transform keeps evenly spaced parallel lines evenly spaced and parallel, but perpendicular lines may not remain perpendicular. This step obtains the rotation, scale (distance between parallel lines) and translation (distance between the origins) of the affine transform. The output of this step is a homography matrix that describes the affine transform.
First, the system finds two main directions of EIC symbols. This step looks at all the directions formed by pairs of connected effective EIC pattern regions and finds two directions that occur the most often.
As described above, given effective EIC pattern mask, regions Rt, where t=1, 2, . . . , T, of pixels (xi, yi) are identified:
R t={(x i , y i)|EIC Pattern Mask (x i , y i)=1, (x i , y i) are neighbors}.
Next, gray-level centroids of the regions are identified. For each region Rt, where t=1, 2, . . . T, gray-level centroid ( x t, y t) is:
x _ t = 1 i = 0 N t G ( x i , y i ) i = 0 N t ( x i · G ( x i , y i ) ) y _ t = 1 i = 0 N t G ( x i , y i ) i = 0 N t ( y i · G ( x i , y i ) )
where (xi, yi) is a pixel in region Rt, G(xi, yi) is the gray-level of the pixel, and Nt is the total number of pixels in region Rt.
Third, for each pair of regions, Ru and Rv, a direction of the pair is obtained:
θ u , v = arc tan ( y _ u - y _ v x _ u - x _ v )
where 0≦θu,v<180. FIG. 24 shows an example of θ. Here, θ is a rotation about point 2401 from horizontal line 2402 to a line 2403 connecting centroids 2401 with another centroid 2404. The determination of angle θ may be performed for all centroids from all centroids in the capture image.
Once all the directions are obtained, a histogram of directions can be created. The X axis is θ. The Y axis is the frequency count of θ.
Next, the histogram is first normalized using template [−0.1, −0.1, −0.1, −0.1, −0.1, 0, 0, 0, 1, 0, 0, 0, −0.1, −0.1, −0.1, −0.1, −0.1]. That is, for every x, where x=0, 1, . . . , 179,
Y ( x ) = i = 0 16 ( template [ i ] · Y ( mod ( x + i - 8 , 180 ) ) )
mod(x, y) is a function that returns the positive remainder of x divided by y. For example, mod(3,2)=1, mod(−3,2)=1.
Next, four candidates of the main directions are identified. The X value that has the highest Y value is set as Candidate 1. Y values of adjacent X are suppressed to 0, i.e. Y(x)=0, where x is in the range of mod(Candidate 1±30, 180). Find the next X value with the highest Y value and set as Candidate 2. Y values of adjacent X are suppressed to 0, i.e. Y(x)=0, where x is in the range of mod(Candidate 2±30, 180). Repeat the process to find Candidate 3 and Candidate 4. The reason for the suppression of the regions is that these angles are where EIC symbols are not expected to lie.
From the 4 candidates, 2 pairs of near perpendicular directions are identified. That is, for a candidate xi, select another candidate xj, such that abs(90−abs(xi−xj)) is minimized. abs(x) is a function that returns the absolute value of x. For example, abs(1.8)=1.8, abs(−1.8)=1.8.
Now, select (xi, xj) such that Y(xi)+Y(xj) is maximized.
Given the pair selected, (xi, xj), centroid of a small area near xi and xj is calculated:
x _ i = k = - 5 5 ( mod ( x i + k , 180 ) · Y ( mod ( x i + k , 180 ) ) ) k = - 5 5 Y ( mod ( x i + k , 180 ) ) x _ j = k = - 5 5 ( mod ( x j + k , 180 ) · Y ( mod ( x j + k , 180 ) ) ) k = - 5 5 Y ( mod ( x j + k , 180 ) )
The two centroids are the two main directions. That is, suppose x i< x j, θh= x i, and θv= x j.
Next, the system determines the scale and translation for the EIC symbols.
In the step, one looks for the scale and translation of the affine transform. Scale is the distance between two adjacent parallel lines. Translation is the distance between the image center and the origin of the coordinate system formed by the grid lines. Both scale and translation are measured in pixels.
FIG. 25 shows the affine transformed grid lines. The two main directions are: θh, θv. The two scales along the two directions are: Sh, Sv. The two translations are: Δh, Δv.
Note that the H, V coordinate system shown in FIG. 25 is the coordinate system of the grid. Grid lines are not seen in image. But as explained before, if one assumes an affine transform is from a paper plane to an image plane, the EIC pattern in the image may appear to lie on grid lines that are an affine transform of the grid lines in the EIC symbol array. Therefore, one can draw grid lines in image (the gridlines parallel to lines 2501 and 2502) and the H, V coordinate system based on an affine transform of the grid lines in EIC symbol array.
The X, Y coordinate system shown in FIG. 25 is the coordinate system of the image. Facing the image, the X− is always horizontal and Y is always vertical. The image center is the origin of the X, Y coordinate system.
To obtain the two scales Sh, Sv, the image may be rotated counterclockwise with θh, θv respectively.
From here, with the affine transform obtained, the perspective transform may be obtained from the affine transform. It should be understood by those skilled in the art that there are a number of different way in which to retrieve the LCD grid and that the examples provided herein are but illustrative examples. After successfully retrieving the LCD grid, three algorithms apply the grid and image to recognize the embedded EIC information: EIC symbol orientation; symbol analysis; and EIC bits extraction. First, to determine the EIC symbol orientation in the image, steps may be utilized as described below.
As shown in FIG. 26, eight (8) points of each LCD pixel are sampled. The positions (hi,j,p, vi,j,p) of the 8 sampling points Gi,j,p(0≦p≦7) in each LCD pixel grid (i, j) are:
( 1 3 , 1 3 ) , ( 1 2 , 1 3 ) , ( 2 3 , 1 3 ) , ( 1 3 , 1 2 ) , ( 2 3 , 1 2 ) , ( 1 3 , 2 3 ) , ( 1 2 , 2 3 ) and ( 2 3 , 2 3 ) .
In accordance with aspects of the present invention, the LCD pixel grid is valid only if 8 sampling points are all valid. The sampling algorithm may include bilinear sampling.
Next, the average grey-level difference Di,j,q(0≦q≦3) of four orientations for each valid LCD pixel grid is calculated. One method for calculating the grey-level difference follows the equation:
{ D i , j , 0 = ( G i , j , 0 + G i , j , 3 + G i , j , 5 ) - ( G i , j , 2 + G i , j , 4 + G i , j , 7 ) 3 D i , j , 1 = ( G i , j , 0 + G i , j , 1 + G i , j , 2 ) - ( G i , j , 5 + G i , j , 6 + G i , j , 7 ) 3 D i , j , 2 = ( G i , j , 2 + G i , j , 4 + G i , j , 7 ) - ( G i , j , 0 + G i , j , 3 + G i , j , 5 ) 3 D i , j , 3 = ( G i , j , 5 + G i , j , 6 + G i , j , 7 ) - ( G i , j , 0 + G i , j , 1 + G i , j , 2 ) 3
Then, smallest 25 differences Di,j,q for each orientation q from all valid LCD pixel grids are selected and their average grey-level difference Dq(0≦q≦3) of each orientation is averaged. The threshold 25 is the average visible EIC symbol on an LCD in a captured image by a camera. It should be understood by those skilled in the art that the threshold value of 25 is one illustrative value, but other smaller or larger values may be utilized. Next, most possible correct orientation q, which has the smallest Dq(0≦q≦3), is determined. As shown in FIGS. 27 and 28, the reference frame for the LCD pixel grids is redefined so that an axis is consistent with an ideal reference frame. Finally, the average grey-level difference Di,j,0 of all valid LCD pixel grids is recalculated in the new reference frame.
The second algorithm to recognize the embedded EIC information is symbol analysis. EIC symbol orientation has been determined above; however, the EIC symbol on the LCD reference alignment remains unknown. For symbol analysis, two algorithms may be used to retrieve horizontal and vertical alignment respectively. First, the smallest 25 differences Di,j,0 from all valid LCD pixel grids are selected. The threshold 25 is the average visible EIC symbol on an LCD in a captured image by a camera. It should be understood by those skilled in the art that the threshold value of 25 is but one value and other smaller or larger values may be utilized.
Then, EIC symbol horizontal alignment is calculated. First, a histogram count Hp(0≦p≦3) of the selected Di,j,0 is obtained by Mod(j,4). Second, the correct horizontal alignment position is obtained as the histogram index P with the biggest value Hp(0≦p≦3).
Next, EIC symbol vertical alignment is calculated. First, a histogram count Vp(0≦p≦3) of the selected Di,j,0 is obtained by Mod(i,4). Second, the correct vertical alignment position is obtained as the histogram index P with the biggest value Vp(0≦p≦3).
Finally, the third algorithm to recognize the embedded EIC information is the EIC bits extraction algorithm, where EIC bits for each EIC symbol in the image are calculated. First, as shown in FIG. 26, the smallest Di,j,0 2601 of four possible information LCD pixels for each EIC symbol is identified. Second, two EIC bits (B1B0) are retrieved from the EIC symbol as shown in FIGS. 30A-30D. Finally, bit arrays are constructed from the retrieved bits such as shown in FIG. 31.
The extracted EIC bits are a piece of single m-array bits that are used to decode x-y position data and/or metadata information. It should be understood by those skilled in the art that various methods may be used for EIC decoding of the EIC bit stream. One example method for EIC decoding of the EIC bit stream is described below.
In order to determine the position of an image relative to an LCD using an m-array, it is necessary to determine the position of a bit captured in the bit relative to the m-array. That is, it is necessary to determine if the bit is the first bit, second bit, etc. in the m-sequence to determine the position of the bit in the m-array.
For any number s, where 0≦s≦2n−1, there exists a unique polynomial r(x), where
r ( x ) = i = 0 n - 1 r i x i
whose order is less than n, such that xs≡r(x)(mod Pn(x)), and vice versa. In other words, there is a one-to-one relationship between s and r(x). Thus, xs/Pn(x) and r(x)/Pn(x) will generate the same m-sequence. For convenience, setting Q(x)=1, m can be assumed to be the m-sequence generated by 1/Pn(x). If a bit is the s′-th bit of m, where 0≦s′<2n−1, the m-sequence that starts from that bit is R=σ−s′(m)=σ2 n −1−s′(m)=σs(m), where s=2n−1−s′. R corresponds to division xs/Pn(x).
As previously noted, there exists
r ( x ) = i = 0 n - 1 r i x i ,
that satisfies r(x)≡xs(mod Pn(x)). R also corresponds to r(x)/Pn(x). Letting m=(m0 m1 . . . mi . . . m2 n −3 m2 n −2)t (where the superscript t stands for vector or matrix transpose), and σi(mt)=(m2 n −1−i m2 n −i . . . m0 . . . m2 n −3−i m2 n −2−i), r(x)/Pn(x) and 1/Pn(x) will have the following relationship:
r ( x ) / P n ( x ) = ( i = 0 n - 1 r i x i ) / P n ( x ) = i = 0 n - 1 [ r i x i / P n ( x ) ] = i = 0 n - 1 r i [ x i · 1 / P n ( x ) ] .
With R corresponding to r(x)/Pn(x), and σi(m) corresponding to xi·1/Pn(x), then,
Rt=rt{circumflex over (M)}
where R is the m-sequence that starts from the s′-th bit of m, r=(r0 r1 r2 . . . rn−1)t are the coefficients of r(x), and
M ^ = ( m t σ ( m t ) σ n - 1 ( m t ) ) .
Again, the addition and multiplication operations are binary operations, i.e., addition is XOR and multiplication is AND.
If an image captures K bits b=(b0 b1 b2 . . . bK−1)t of m (K≧n), and the relative distances between the positions of the bits in the m-sequence are: si=d(bi, b0)), where i=0, 1, . . . , K−1 and s0=0, selecting the si+1-th bits of R and the si+1-th columns of {circumflex over (M)} will result in:
bt=rtM
where bt is the transpose of b, M is a sub-matrix of {circumflex over (M)} and consists of the si+1-th columns of {circumflex over (M)}, where i=0, 1, 2, . . . , K−1.
If M is a non-degenerate matrix and b does not contain error bits, then r can be solved by selecting n bits from b by solving for:
rt={tilde over (b)}t{tilde over (M)}−1
where {tilde over (M)} is any non-degenerate n×n sub-matrix of M, and {tilde over (b)} is the corresponding sub-vector of b consisting of the selected n bits.
In most cases, however, an image cannot capture a set of bits b that does not contain error bits. For example, improper illumination, document content, dust and creases can all obscure the visual representation of bits in an image, preventing these bits from being recognized or causing the value of these bits to be improperly recognized. The solution of r becomes difficult when there are error bits in b. Further, decoding becomes even more difficult because the coefficient matrix M is not fixed when the pen moves, changing the image from frame to frame. Moreover, the structure of M is irregular. Therefore, traditional decoding algorithms cannot effectively be applied to solve r under practical circumstances.
To address these difficulties, stochastic solution techniques provide a high decoding accuracy under practical conditions. As will be described in more detail, these techniques solve the equation bt=rtM incrementally so that many solution candidates are readily available without having to solve this equation exactly.
According to various examples of the invention, independent n bits (i.e., the sub-matrix consisting of the corresponding columns of M is non-degenerate) are randomly selected from the group of b that are captured in an image of a document. Supposing that b(0) are the n bits chosen, a solution for r can then be obtained as:
[r(0)]t=[b(0)]t[M(0)]−1
where M(0) contains the corresponding columns of the array M for the chosen bits.
For simplicity, the n bits chosen from b to make up b(0) can be moved to the beginning of b, with the remaining bits making up b moved to the end of b. This leads to the relationship
([b (0)]t ,[ b (0)]t)=[r (0)]t(M (0) , M (0))+(0n t ,[e (0)]t)
where b(0) are the chosen n bits, b (0) are the remaining bits from the set b, M(0) is the corresponding columns of M for the chosen bits, M (0) is the corresponding columns of M for the remaining bits, 0n t=(0 0 . . . 0)1×n, [r(0)]t=[b(0)]t[M(0)]−1, and [e(0)]t=[ b (0)]t+[r(0)]t M (0).
The value (0n t,[e(0)]t) refers to the “difference vector” between ([b(0)]t,[ b (0)]t) and [r(0)]t(M(0), M (0)), or simply the different vector of r(0), and the number of 1's in (0n t,[e(0)]t) is called the number of different bits. The vector containing different bits between ([b(0)]t.[ b (0)]t) and [r(0)]t(M(0), M (0)) alternately can be identified as D(0). If D(0)=(0n t,[e(0)]t), then the number d(0) of 1's in D(0) is d(0)=HammingWeight(D(0))=HammingWeight(e(0)). That is, d(0) is the number of different bits between ([b(0)]t,[ b (0)]t) and [r(0)]t(M(0), M (0)).
Next, some of the chosen bits n from the set b are switched with some of the remaining bits from the set b. In particular, J bit pairs (kj,lj) are switched between the original chosen bits n and the remaining bits from the set of bits b, where k1≠k2≠ . . . ≠kJ≦n, n<l1≠l2≠ . . . ≠lJ≦K. It should be noted that the bit order is redefined in ([b(0)]t,[ b (0)]t), and these bits are not maintained in their original order. The relationship between the bits before and after switching is:
[ ( 1 ) ] t = [ ( 0 ) ] t + [ ( 0 ) ] t E l - n [ P R j ( 0 ) ] - 1 ( E k t P ( 0 ) + E l - n t ) , [ r ( 1 ) ] t = [ r ( 0 ) ] t + [ ( 0 ) ] t E l - n [ P R j ( 0 ) ] - 1 E k t [ M ( 0 ) ] - 1 , P ( 1 ) = P ( 0 ) + ( E k + P ( 0 ) E l - n ) [ P R j ( 0 ) ] - 1 ( E k t P ( 0 ) + E l - n t ) , [ M ( 1 ) ] - 1 = [ M ( 0 ) ] - 1 + ( E k + P ( 0 ) E l - n ) [ P R j ( 0 ) ] - 1 E k t [ M ( 0 ) ] - 1 , where E k = ( e k 1 e k 2 e k J ) n × J , E l - n = ( e l 1 - n e l 2 - n e l J - n ) ( K - n ) × J , P R j ( 0 ) = E k t P ( 0 ) E l - n e i t = ( 0 0 1 i 0 0 ) l × n or l × ( K - n ) , and P ( i ) = [ M ( i ) ] - 1 M _ ( i ) , i = 0 , 1.
If the choice of (kj,lj) is to make:
[e(0)]tEl−n[PR J (0)]−1=1J t,
where 1J t=(1 1 . . . 1)1×J, then
[e (1)]t =[e (0)]t+1J t(E k t P (0) +E l−n t)
[r(1)]t=[r(0)]t+1J tEk t[M(0)]−1.
In view of [e(0)]tEl−n[PR J (0)]−1=1J t given k1≠k2≠ . . . ≠kJ≦n, the choice of n<l1≠l2≠ . . . ≠lJ≦K is as follows: {l1, l2, . . . , lJ}⊂{p1, . . . , pm}, where {p1, . . . , pm.} are the indices of the O-bits of [e(0)]t+1J tEk tP(0), and PR J (0) is invertible. Therefore, if the rank of Ek tP(0)Ep−n is less than J, then such l1, l2, . . . , lJ cannot be chosen, where Ep−n=(ep 1 −n ep 2 −n . . . ep m −n)(K−n)×m. Choosing other l1, l2, . . . , lJ is equivalent to switching a smaller number of bit pairs, and therefore does not conform to the goal of switching J bits. It should be noted that, as long as the rank of Ek tP(0)Ep−n is J, the choice of l1, l2, . . . , lJ will result in the identical location vector. Therefore, choosing one combination is sufficient. Moreover, as long as PR J (0) is invertible, the newly selected n bits are also independent.
With the above choice of l1, l2, . . . , lJ, the number of different bits in e(i+1) is:
The number of 1's in ([e(0)]t+1J tEk tP(0))+J
It should be noted that Ek tP(0)El−n actually means choosing the k1, . . . , kJ-th rows and l1−n, . . . lJ−n-th columns of P(0), while 1J tEk tP(0) actually means summing the k1, . . . , kJ-th rows of P(0). No matrix computation is needed.
Thus, the decoding steps can be summarized as follows. First, an independent n-bit combination is generated from the group of bits b captured in an image. It should be noted that, with various embodiments of the invention, the selection of the n-bits can be combined with bit recognition confidence techniques, to help ensure that the most accurately recognized bits are selected for the n-bit combination.
Next, the relationship ([b(0)]t,[ b (0)]t)=[r(0)]t(M(0), M (0))+(0n t,[e(0)]t) is solved to determine d(0)=HammingWeight(D(0))=HammingWeight(e(0)). If the number of different bits d(0) is 0, then the process is stopped and the solution r(0) is output. Otherwise, all J (=1 and 2) bit pairs are switched, and the number of different bits d is again determined using the relationship ([e(0)]t+1J tEk tP(0))+J. It should be noted, however, that this relationship can only be evaluated when the rank of Ek tP(0)Ep−n is J. In this case there is no need to specify l1, l2, . . . , lJ. Next, the minimal number d of different bits is determined.
The above process has to be repeated several times in order to ensure a high enough probability of successful decoding. To estimate the times of selecting the n-bit b(0) from b, the number r of the error bits in b is first predicted to be d. If r is changed, then
p s = C r s C K - r n - s C K n ,
is computed, which is the probability that the chosen n bits contain s error bits, where
C a b = a ! b ! ( a - b ) !
is the combinatory number, and
P s = i = 0 s p i ,
is the probability that the chosen n bits contain less than s+1 error bits. In practice, s=2 in order to minimize the computation load. Next, s2 is computed, such that 1−(1−P2)s 2 ≧Pe, where Pe is the expected probability of successful decoding. If the times S of chosen b(0) is equal to or larger than s2, then the process is stopped and the results are output. Otherwise, the process is repeated with a new independent n-bit combination b(0) generated from the group of bits b captured in an image. Using this process, as long as the chosen n bits contain less than J+1 error bits, the correct solution can be found.
Illustrative components for a system 3200 for creating an EIC pattern on an LCD is shown in FIG. 32. System 3200 is shown to include a design component 3201, a processor component 3203, an embedding component 3205, an LCD 3207, and a decoder component 3209. Design component 3201 may be configured to define a size 3221 of the EIC pattern for display on the LCD 3207. The size 3221 may be defined by a grid with pixels arranged into columns and rows. Processor component 3203 may be configured to determine an area 3223 of the grid for embedding position data associated with the dimensions of the LCD 3207.
Embedding component 3205 may be configured to embed the position data 3225 in the determined area of the grid. LCD 3207 may be configured to display the position data in the EIC pattern on a screen. LCD 3207 may be configured to intermittently display 3227 the EIC pattern and/or display the EIC pattern as a watermark 3237 on an underlying image associated with an application program. Decoder component 3209 may be configured to decode the embedded position data and/or metadata information. Decoder component 3209 may be configured to include a camera 3229 configured to capture an image of the EIC pattern and an extractor 3239 configured to extract the embedded position data and/or metadata information from the captured image of the EIC pattern.
FIG. 33 is an illustrative method for creating an EIC pattern on an LCD in accordance with at least one aspect of the present invention. The process starts at step 3301 where a size of the EIC pattern on an LCD is defined by a grid and pixels. At step 3303, an area of the grid for embedding position data and/or metadata information is determined. A determination is made at step 3305 as to whether the determined area is a pixel. If the determined area is a pixel, the process moves to step 3307 where a specific sub-pixel of the determined pixel is determined to embed the EIC pattern. At step 3309, the position data and/or metadata information is embedded in the specific sub-pixel of the determined pixel. Such a sub-pixel may be the red channel sub-pixel of a pixel. Proceeding to step 3311, the grey value of the sub-pixel of the determined pixel may be changed based upon a background color of an underlying image currently displayed on the LCD screen. The process then moves to step 3317.
If the determined area is not a pixel in step 3305, the process moves to step 3313 where the position data and/or metadata information is embedded in a portion of the black matrix of the grid itself. Such a portion may be an edge that separates two sub-pixels of one pixel or two sub-pixels of two different pixels. Proceeding to step 3315, the grey value of the portion of the grid may be changed based upon a background color of an underlying image currently displayed on the LCD screen. The process then moves to step 3317. At step 3317, an orientation of the EIC pattern is determined. Moving to step 3319, the position data and/or metadata information is decoded and the process ends.
Illustrative components for a system 3400 for decoding an EIC pattern on an LCD is shown in FIG. 34. System 3400 is shown to include an input component 3401, a processor component 3403, and a decoding component 3405. Input component 3401 may be configured to receive captured images. Decoding component 3405 may be configured to decode position data and/or metadata information from an extracted bit stream to determine a position of the capture images with respect to a screen of the LCD. Processor component 3403 may be configured to process the captured images and output an extracted bit stream.
Processor component may be configured to include a preprocessing component 3411, a definition component 3413, a recognition component 3415, and an extracting component 3417. Preprocessing component 3411 may be configured to determine pixels corresponding to LCD pixel grids associated with the EIC pattern. Definition component 3413 may be configured to define the LCD pixel grids based upon the determined pixels. Recognition component 3415 may be configured to recognize the EIC pattern from the LCD pixel grids, and extracting component 3417 may be configured to extract a bit stream of EIC bits based upon the recognized EIC pattern.
Preprocessing component 3411 may further be configured to include a pixel separator, a binary erosion component, a separation component, a mask definer, and a mask merger. The pixel separator may be configured to separate pixels that lie on edges and vertexes of the LCD pixel grids as a first mask. The binary erosion component may be configured to erode the first mask to obtain a second mask, the second mask including fewer pixels than the first mask. The separation component may be configured to separate connected regions of the second mask in accordance with a first threshold to obtain a third mask and a fourth mask. The mask definer may be configured to define a sixth mask as pixels that belong to regions of the fifth mask, and the mask merger may be configured to merge the third and the sixth masks to define the LCD pixel grids. The binary erosion component may be further configured to erode the fourth mask to obtain a fifth mask, and a second threshold may be defined to correspond to a region of the fifth mask.
Recognition component 3415 may further be configured to include an orientation component, an alignment retriever, and a calculation component. The orientation component may be configured to determine an orientation of the EIC pattern in the captured image. The alignment retriever may be configured to retrieve horizontal and vertical alignment information for the orientation, and the calculation component may be configured to calculate EIC bits for the EIC pattern.
FIG. 35 is an illustrative method for decoding an EIC pattern on an LCD in accordance with at least one aspect of the present invention. The process starts at step 3501 where a captured image is received by the system. At step 3503, the captured image is processed to output an extracted bit stream. At step 3505, position data and/or metadata information is decoded from the extracted bit stream. Proceeding to step 3507, a position of the captured image with respect to a screen of an LCD is determined and the process ends.
FIG. 36 provides an illustrative method for step 3503 shown in FIG. 35 in accordance with at least one aspect of the present invention. The process starts at step 3601 where the captured image is preprocessed to determine pixels corresponding to LCD pixel grids associated with the EIC pattern. At step 3603, the LCD pixel grids are defined based upon the pixels. Moving to step 3605, the EIC pattern is recognized from the LCD pixel grids. A bit stream of EIC bits based upon the recognized EIC pattern is extracted at step 3607 and the process ends.
FIG. 37 provides an illustrative method for step 3601 shown in FIG. 36 in accordance with at least one aspect of the present invention. The process starts at step 3701 where pixels that lie on edges or vertexes of the LCD grids are separated as a first mask. At step 3703, a binary erosion operation is performed on the first mask to obtain a second mask. In this step, fewer pixels remain in the second mask compared to the first mask. Proceeding to step 3705, connected regions of the second mask are separated in accordance with a first threshold to obtain a third mask and a fourth mask. Another binary erosion operation is performed at step 3707 on the fourth mask to obtain a fifth mask. Moving to step 3709, a second threshold corresponding to a region of the fifth mask is defined. At step 3711, a sixth mask is defined as pixels that belong to the regions of the fifth mask. Finally, at step 3713, the third and the sixth masks are merged to define the LCD pixel grids and the process ends.
With respect to an application programming interface (API), various aspects of the present invention may be performed by an API. For example, public APIs may interface with an operating system to allow the operating system to provide the various features of the present invention. In one illustrative embodiment, a software architecture for processing data representative of positioning information on an LCD screen and/or metadata information may include a component configured to recognize the position of an input device with respect to an area of an LCD screen and an application program interface to access the component. An API may receive a request to recognize the position of an input device, access the necessary function(s) to perform the operation, and then send the results back to an operating system. The operating system may use the data provided from the API to perform the various features of the present invention. In another embodiment, the component may be configured to process a captured image to output an extracted bit stream, to decode position data and/or metadata information from the extracted bit stream, and to determine a position of the captured image with respect to a screen of an LCD.
As can be appreciated by one skilled in the art, a computer system with an associated computer-readable medium containing instructions for controlling the computer system can be utilized to implement the illustrative embodiments that are disclosed herein. The computer system may include at least one computer such as a microprocessor, digital signal processor, and associated peripheral electronic circuitry.
While illustrative systems and methods as described herein embodying various aspects of the present invention are shown, it will be understood by those skilled in the art, that the invention is not limited to these embodiments. Modifications may be made by those skilled in the art, particularly in light of the foregoing teachings. For example, each of the elements of the aforementioned embodiments may be utilized alone or in combination or subcombination with elements of the other embodiments. It will also be appreciated and understood that modifications may be made without departing from the true spirit and scope of the present invention. The description is thus to be regarded as illustrative instead of restrictive of the present invention.

Claims (18)

1. A method for creating a pattern on a liquid crystal display (LCD) and decoding position data from the pattern, the method comprising steps of:
defining a size of the pattern on an LCD by a grid with a plurality of pixels, each pixel having sub-pixels with an adjustable intensity value;
determining an area of the grid for embedding position data, wherein the position data provides an indication of position relative to the LCD and wherein the position data is comprised of a plurality of dots; and
embedding the position data in the determined area of the grid by:
for each dot associated with the position data,
selecting one of the plurality of pixels of the grid corresponding to the position of the dot of the pattern,
selecting a sub-pixel of the selected pixel, and
adjusting the intensity value of the selected sub-pixel to indicate its correspondence with the dot wherein the adjusted intensity value of the selected sub-pixel is based on an underlying image displayed on the LCD;
receiving a captured image:
decoding position data from the pattern, wherein decoding position data includes,
processing the captured image to output an extracted bit stream by,
preprocessing the captured image to determine pixels corresponding to LCD pixel grids associated with the embedded interaction code pattern, wherein preprocessing includes:
separating pixels that lie on edges and vertexes of the LCD pixel grids as a first mask,
binary eroding the first mask to obtain a second mask, the second mask including fewer pixels than the first mask,
separating connected regions of the second mask in accordance with a first threshold to obtain a third mask and a fourth mask,
binary eroding the fourth mask to obtain a fifth mask,
defining a second threshold corresponding to a region of the fifth mask,
defining a sixth mask as pixels that belong to the regions of the fifth mask, and
merging the third and the sixth masks to define the LCD pixel grids,
defining the LCD pixel grids based upon the pixels,
recognizing the embedded interaction code pattern from the LCD pixel grids,
extracting a bit stream of embedded interaction code bits based upon the recognized embedded interaction code pattern to obtain the extracted bit stream;
decoding position data from the extracted bit stream, and
determining a position of the captured image relative to the LCD based upon the decoded position data
so that the position of an image captured from the LCD can be determined relative to the LCD based on the adjusted intensity values of the sub-pixels.
2. The method of claim 1, wherein the determined area of the grid includes a pixel of the plurality of pixels and the step of embedding includes embedding the position data in a sub-pixel of the pixel.
3. The method of claim 2, wherein a pixel includes three sub-pixels including a red channel sub-pixel, a green channel sub-pixel, and a blue channel sub-pixel.
4. The method of claim 3, wherein the step of embedding the position data in the sub-pixel includes embedding the position data in the red channel sub-pixel.
5. The method of claim 4 wherein the grey value of the red channel sub-pixel equals zero and the grey value of the green channel sub-pixel and the blue channel sub-pixel equals 255.
6. The method of claim 4, wherein the intensity value of the red channel sub-pixel is based upon a color of an underlying image displayed on the LCD.
7. The method of claim 1, wherein the step of decoding includes a step of determining the orientation of the pattern.
8. The method of claim 1, wherein the size of the pattern is defined by columns 1 to M and rows 1 to N, M and N being integers.
9. The method of claim 8, wherein M and N are equal in value.
10. The method of claim 8, wherein the step of embedding includes embedding the position data in one of columns 1 to M−1 and in one of rows 1 to N−1.
11. The method of claim 1, wherein the position data includes two bits of data to define a position along an x axis and a y axis.
12. The method of claim 1, further comprising a step of displaying the EIC pattern on the LCD by intermittently flashing the EIC pattern.
13. The method of claim 1, further comprising a step of displaying the pattern on the LCD by watermarking an image on the LCD with the pattern.
14. The method of claim 1, wherein the determined area of the grid includes first and second pixels of the plurality of pixels and the step of embedding includes embedding the position data in a sub-pixel of the first pixel and a sub-pixel of the second pixel, wherein the intensity value of the first sub-pixel is different from the intensity value of the second sub-pixel and is based upon a color of an underlying image displayed on the LCD.
15. A system for creating a pattern on a liquid crystal display (LCD), the system comprising:
a design component configured to define a size of the pattern for display on the LCD, the size defined by a grid with a matrix formed by a plurality of pixels arranged in a plurality of columns and a plurality of rows, each pixel having sub-pixels with adjustable intensity values;
a processor component configured to determine an area of the grid for embedding position data relative to the LCD;
an embedding component configured to, when the determined area is a pixel, embed the position data relative to the LCD in the determined area of the grid by adjusting the intensity value of a sub-pixel of at least one of the plurality of pixels;
an embedding component configured to, when the determined area is not a pixel, embed the position data relative to the LCD in the determined area of the grid by adjusting the intensity value of a portion of the matrix, wherein the position data is not based on an underlying image displayed on the LCD; and
an LCD configured to display the position data in the pattern wherein the location of the position data is not based on an underlying image displayed on the LCD.
16. The system of claim 15, further comprising a decoder component configured to decode the embedded position data.
17. The system of claim 16, wherein the decoder component includes a camera configured to capture an image of the pattern and an extracting component configured to extract the embedded position data from the captured image of the pattern.
18. The system of claim 15, wherein the LCD is further configured to display the position data in the EIC pattern by intermittently flashing the EIC pattern on the LCD.
US11/169,889 2005-06-30 2005-06-30 Embedding a pattern design onto a liquid crystal display Expired - Fee Related US7619607B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US11/169,889 US7619607B2 (en) 2005-06-30 2005-06-30 Embedding a pattern design onto a liquid crystal display

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US11/169,889 US7619607B2 (en) 2005-06-30 2005-06-30 Embedding a pattern design onto a liquid crystal display

Publications (2)

Publication Number Publication Date
US20070001950A1 US20070001950A1 (en) 2007-01-04
US7619607B2 true US7619607B2 (en) 2009-11-17

Family

ID=37588822

Family Applications (1)

Application Number Title Priority Date Filing Date
US11/169,889 Expired - Fee Related US7619607B2 (en) 2005-06-30 2005-06-30 Embedding a pattern design onto a liquid crystal display

Country Status (1)

Country Link
US (1) US7619607B2 (en)

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060215183A1 (en) * 2005-03-28 2006-09-28 Fuji Xerox Co., Ltd. Image processing apparatus and method
US7817816B2 (en) 2005-08-17 2010-10-19 Microsoft Corporation Embedded interaction code enabled surface type identification
US20130321357A1 (en) * 2012-06-01 2013-12-05 Panasonic Corporation Display panel and display device
US8692212B1 (en) 2012-10-29 2014-04-08 3M Innovative Properties Company Optical digitizer system with position-unique photoluminescent indicia
US9068845B2 (en) 2011-12-16 2015-06-30 3M Innovative Properties Company Optical digitizer system with position-unique photoluminescent indicia
US9946947B2 (en) 2012-10-31 2018-04-17 Cognex Corporation System and method for finding saddle point-like structures in an image and determining information from the same
US9958954B2 (en) 2012-12-13 2018-05-01 3M Innovative Properties Company System and methods for calibrating a digitizer system
US10429950B2 (en) * 2014-07-16 2019-10-01 Hewlett-Packard Development Company, L.P. Diffusive emissive display
US10753746B2 (en) 2012-11-29 2020-08-25 3M Innovative Properties, Inc. Multi-mode stylus and digitizer system
US10762405B2 (en) 2017-10-26 2020-09-01 Datalogic Ip Tech S.R.L. System and method for extracting bitstream data in two-dimensional optical codes
US11010651B1 (en) 2018-11-26 2021-05-18 National Technology & Engineering Solutions Of Sandia, Llc Optically configurable charge-transfer materials and methods thereof

Families Citing this family (37)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7133563B2 (en) * 2002-10-31 2006-11-07 Microsoft Corporation Passive embedded interaction code
US7583842B2 (en) * 2004-01-06 2009-09-01 Microsoft Corporation Enhanced approach of m-array decoding and error correction
US7263224B2 (en) * 2004-01-16 2007-08-28 Microsoft Corporation Strokes localization by m-array decoding and fast image matching
US7505982B2 (en) 2004-12-03 2009-03-17 Microsoft Corporation Local metadata embedding solution
US7536051B2 (en) * 2005-02-17 2009-05-19 Microsoft Corporation Digital pen calibration by local linearization
US7607076B2 (en) * 2005-02-18 2009-10-20 Microsoft Corporation Embedded interaction code document
US7826074B1 (en) 2005-02-25 2010-11-02 Microsoft Corporation Fast embedded interaction code printing with custom postscript commands
US7532366B1 (en) 2005-02-25 2009-05-12 Microsoft Corporation Embedded interaction code printing with Microsoft Office documents
US20060215913A1 (en) * 2005-03-24 2006-09-28 Microsoft Corporation Maze pattern analysis with image matching
US20060242562A1 (en) * 2005-04-22 2006-10-26 Microsoft Corporation Embedded method for embedded interaction code array
US7421439B2 (en) * 2005-04-22 2008-09-02 Microsoft Corporation Global metadata embedding and decoding
US7400777B2 (en) * 2005-05-25 2008-07-15 Microsoft Corporation Preprocessing for information pattern analysis
US7729539B2 (en) * 2005-05-31 2010-06-01 Microsoft Corporation Fast error-correcting of embedded interaction codes
US7580576B2 (en) * 2005-06-02 2009-08-25 Microsoft Corporation Stroke localization and binding to electronic document
US7528848B2 (en) * 2005-06-30 2009-05-05 Microsoft Corporation Embedded interaction code decoding for a liquid crystal display
CN101115170B (en) * 2006-07-28 2012-02-22 深圳Tcl新技术有限公司 System and method for providing touch screen function in digital optical process video unit
GB2451482B (en) * 2007-07-31 2012-03-14 Hewlett Packard Development Co Method and apparatus for reading a surface coded pattern
KR101058726B1 (en) * 2009-11-11 2011-08-22 삼성전자주식회사 Image correction device and method for removing lighting components
US8892118B2 (en) 2010-07-23 2014-11-18 Qualcomm Incorporated Methods and apparatuses for use in providing position assistance data to mobile stations
US8818401B2 (en) 2010-07-30 2014-08-26 Qualcomm Incorporated Methods and apparatuses for use in determining that a mobile station is at one or more particular indoor regions
US9148763B2 (en) 2010-07-30 2015-09-29 Qualcomm Incorporated Methods and apparatuses for mobile station centric determination of positioning assistance data
KR20130108614A (en) 2010-10-28 2013-10-04 사이프레스 세미컨덕터 코포레이션 Capacitive stylus with palm rejection
KR20120110817A (en) * 2011-03-30 2012-10-10 삼성전자주식회사 Handwriting input apparatus and input method using the handwriting input apparatus, display divice for inputting of the handwriting input apparatus
US8797301B2 (en) 2012-02-15 2014-08-05 Cypress Semiconductor Corporation Active stylus to host data transmitting method
JPWO2013035319A1 (en) * 2011-09-06 2015-03-23 パナソニック株式会社 Display device and display control system
KR20130038081A (en) * 2011-10-07 2013-04-17 삼성전자주식회사 Apparatus and method for detecting input position by deter using displaying pattern determination
DE102011086318A1 (en) * 2011-11-14 2013-05-16 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Position determination of an object by detection of a position pattern by optical sensor
JPWO2013161246A1 (en) * 2012-04-26 2015-12-21 パナソニック株式会社 Display control system and display device
KR101373203B1 (en) * 2012-06-05 2014-03-12 (주)펜제너레이션스 E-paper display and electroni pen system using the same
US9373302B2 (en) * 2012-08-10 2016-06-21 Blackberry Limited Stacked device position identification
US9111073B1 (en) 2012-11-19 2015-08-18 Trend Micro Inc. Password protection using pattern
TWI557608B (en) * 2013-04-03 2016-11-11 宏碁股份有限公司 Input apparatus of electronic apparatus and setting method thereof
US9117068B1 (en) * 2013-09-25 2015-08-25 Trend Micro Inc. Password protection using pattern
US20150378168A1 (en) * 2014-05-19 2015-12-31 Smartershade, Inc. Patterned image device
US9832338B2 (en) * 2015-03-06 2017-11-28 Intel Corporation Conveyance of hidden image data between output panel and digital camera
FR3099611B1 (en) * 2019-08-02 2022-05-27 Continental Automotive Gmbh Method and system for verifying a message display on a digital display
WO2023043928A1 (en) * 2021-09-17 2023-03-23 Google Llc Encoding and recognizing positions of a display

Citations (103)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4686329A (en) 1985-06-21 1987-08-11 Advanced Robotic Technology, Inc. Absolute position mouse
US4742558A (en) 1984-02-14 1988-05-03 Nippon Telegraph & Telephone Public Corporation Image information retrieval/display apparatus
US4745269A (en) 1985-05-22 1988-05-17 U.S. Philips Corporation Method of identifying objects provided with a code field containing a dot code, a device for identifying such a dot code, and a product provided with such a dot code
US4829583A (en) 1985-06-03 1989-05-09 Sino Business Machines, Inc. Method and apparatus for processing ideographic characters
US4941124A (en) 1982-01-25 1990-07-10 Skinner Jr James T Text comparator with counter shift register
US5032924A (en) 1989-04-10 1991-07-16 Nilford Laboratories, Inc. System for producing an image from a sequence of pixels
US5051736A (en) 1989-06-28 1991-09-24 International Business Machines Corporation Optical stylus and passive digitizing tablet data input system
US5073966A (en) 1983-05-10 1991-12-17 Canon Kabushiki Kaisha Image data processing apparatus
US5146552A (en) 1990-02-28 1992-09-08 International Business Machines Corporation Method for associating annotation with electronically published material
US5153928A (en) 1989-06-09 1992-10-06 Casio Computer Co., Ltd. Method and apparatus for recording/reproducing mesh pattern data
US5181257A (en) 1990-04-20 1993-01-19 Man Roland Druckmaschinen Ag Method and apparatus for determining register differences from a multi-color printed image
US5196875A (en) 1988-08-03 1993-03-23 RoyoCad Gesellschaft fur Hard-und Software mbH Projection head
US5235654A (en) 1992-04-30 1993-08-10 International Business Machines Corporation Advanced data capture architecture data processing system and method for scanned images of document forms
US5243149A (en) 1992-04-10 1993-09-07 International Business Machines Corp. Method and apparatus for improving the paper interface to computing systems
US5247137A (en) 1991-10-25 1993-09-21 Mark Epperson Autonomous computer input device and marking instrument
US5253336A (en) 1989-04-20 1993-10-12 Kabushiki Kaisha Toshiba Method for fitting third-degree Bezier curves to series of data points
US5288986A (en) 1992-09-17 1994-02-22 Motorola, Inc. Binary code matrix having data and parity bits
US5294792A (en) 1991-12-31 1994-03-15 Texas Instruments Incorporated Writing tip position sensing and processing apparatus
US5335150A (en) 1993-08-31 1994-08-02 Huang Chao C Laser pointer with a constant power output control
US5365598A (en) 1986-07-25 1994-11-15 Ast Research, Inc. Handwritten keyboardless entry computer system
US5394487A (en) 1993-10-27 1995-02-28 International Business Machines Corporation Forms recognition management system and method
US5398082A (en) 1993-05-20 1995-03-14 Hughes-Jvc Technology Corporation Scanned illumination for light valve video projectors
US5414227A (en) 1993-04-29 1995-05-09 International Business Machines Corporation Stylus tilt detection apparatus for communication with a remote digitizing display
US5442147A (en) 1991-04-03 1995-08-15 Hewlett-Packard Company Position-sensing apparatus
US5448372A (en) 1991-12-18 1995-09-05 Eastman Kodak Company Storage and retrieval of digitized photographic images
US5450603A (en) 1992-12-18 1995-09-12 Xerox Corporation SIMD architecture with transfer register or value source circuitry connected to bus
US5471533A (en) 1990-01-05 1995-11-28 Symbol Technologies, Inc. Record with encoded data
US5477012A (en) 1992-04-03 1995-12-19 Sekendur; Oral F. Optical position determination
US5511156A (en) 1990-04-05 1996-04-23 Seiko Epson Corporation Interpreter for executing rasterize processing to obtain printing picture element information
US5546515A (en) 1992-07-08 1996-08-13 Matsushita Electric Industrial Co., Ltd. Image processing apparatus
US5581637A (en) 1994-12-09 1996-12-03 Xerox Corporation System for registering component image tiles in a camera-based scanner device transcribing scene images
US5581682A (en) 1991-06-28 1996-12-03 International Business Machines Corporation Method for storing and retrieving annotations and redactions in final form documents
US5587558A (en) 1992-01-24 1996-12-24 Seiko Instruments Inc. Coordinate detecting apparatus having acceleration detectors
US5612524A (en) 1987-11-25 1997-03-18 Veritec Inc. Identification symbol system and method with orientation mechanism
US5626620A (en) 1995-02-21 1997-05-06 Medtronic, Inc. Dual chamber pacing system and method with continual adjustment of the AV escape interval so as to maintain optimized ventricular pacing for treating cardiomyopathy
US5629499A (en) 1993-11-30 1997-05-13 Hewlett-Packard Company Electronic board to store and transfer information
US5635697A (en) 1989-03-01 1997-06-03 Symbol Technologies, Inc. Method and apparatus for decoding two-dimensional bar code
US5644652A (en) 1993-11-23 1997-07-01 International Business Machines Corporation System and method for automatic handwriting recognition with a writer-independent chirographic label alphabet
US5652412A (en) 1994-07-11 1997-07-29 Sia Technology Corp. Pen and paper information recording system
US5661291A (en) 1995-06-07 1997-08-26 Hand Held Products, Inc. Audio proof of delivery system and method
US5661506A (en) 1994-11-10 1997-08-26 Sia Technology Corporation Pen and paper information recording system using an imaging pen
US5670897A (en) 1992-11-05 1997-09-23 Xilinx, Inc. High speed mask register for a configurable cellular array
US5686718A (en) 1995-03-15 1997-11-11 Sharp Kabushiki Kaisha Recording method, decoding method, and decoding apparatus for digital information
US5692073A (en) 1996-05-03 1997-11-25 Xerox Corporation Formless forms and paper web using a reference-based mark extraction technique
US5719884A (en) 1995-07-27 1998-02-17 Hewlett-Packard Company Error correction method and apparatus based on two-dimensional code array with reduced redundancy
US5721940A (en) 1993-11-24 1998-02-24 Canon Information Systems, Inc. Form identification and processing system using hierarchical form profiles
US5726435A (en) 1994-03-14 1998-03-10 Nippondenso Co., Ltd. Optically readable two-dimensional code and method and apparatus using the same
US5727098A (en) 1994-09-07 1998-03-10 Jacobson; Joseph M. Oscillating fiber optic display and imager
US5754280A (en) 1995-05-23 1998-05-19 Olympus Optical Co., Ltd. Two-dimensional rangefinding sensor
US5756981A (en) 1992-02-27 1998-05-26 Symbol Technologies, Inc. Optical scanner for reading and decoding one- and-two-dimensional symbologies at variable depths of field including memory efficient high speed image processing means and high accuracy image analysis means
US5765176A (en) 1996-09-06 1998-06-09 Xerox Corporation Performing document image management tasks using an iconic image having embedded encoded information
US5774602A (en) 1994-07-13 1998-06-30 Yashima Electric Co., Ltd. Writing device for storing handwriting
US5818436A (en) 1993-03-15 1998-10-06 Kabushiki Kaisha Toshiba Apparatus and method for playing back continuous data
US5817992A (en) 1997-03-05 1998-10-06 Rpg Diffusor Systems, Inc. Planar binary amplitude diffusor
US5822465A (en) 1992-09-01 1998-10-13 Apple Computer, Inc. Image encoding by vector quantization of regions of an image and codebook updates
US5822436A (en) 1996-04-25 1998-10-13 Digimarc Corporation Photographic products and methods employing embedded information
US5825892A (en) * 1996-10-28 1998-10-20 International Business Machines Corporation Protecting images with an image watermark
US5825015A (en) 1994-04-29 1998-10-20 Electronic Automation Limited Machine readable binary codes
US5850058A (en) 1995-11-17 1998-12-15 Hitachi, Ltd. Information processor
US5852434A (en) 1992-04-03 1998-12-22 Sekendur; Oral F. Absolute optical position determination
US5855483A (en) 1994-11-21 1999-01-05 Compaq Computer Corp. Interactive play with a computer
US5855594A (en) 1997-08-08 1999-01-05 Cardiac Pacemakers, Inc. Self-calibration system for capture verification in pacing devices
US5875264A (en) 1993-12-03 1999-02-23 Kaman Sciences Corporation Pixel hashing image recognition system
US5890177A (en) 1996-04-24 1999-03-30 International Business Machines Corporation Method and apparatus for consolidating edits made by multiple editors working on multiple document copies
US5897648A (en) 1994-06-27 1999-04-27 Numonics Corporation Apparatus and method for editing electronic documents
US5898166A (en) 1995-05-23 1999-04-27 Olympus Optical Co., Ltd. Information reproduction system which utilizes physical information on an optically-readable code and which optically reads the code to reproduce multimedia information
US5902968A (en) 1996-02-20 1999-05-11 Ricoh Company, Ltd. Pen-shaped handwriting input apparatus using accelerometers and gyroscopes and an associated operational device for determining pen movement
US5937110A (en) 1996-12-20 1999-08-10 Xerox Corporation Parallel propagating embedded binary sequences for characterizing objects in N-dimensional address space
US5939703A (en) 1995-01-03 1999-08-17 Xerox Corporation Distributed dimensional labeling for dimensional characterization of embedded data blocks
US5960124A (en) 1994-07-13 1999-09-28 Yashima Electric Co., Ltd. Image reproducing method for reproducing handwriting
US5961571A (en) 1994-12-27 1999-10-05 Siemens Corporated Research, Inc Method and apparatus for automatically tracking the location of vehicles
US5995084A (en) 1997-01-17 1999-11-30 Tritech Microelectronics, Ltd. Touchpad pen-input and mouse controller
US6000614A (en) 1996-12-20 1999-12-14 Denso Corporation Two-dimensional code reading apparatus
US6000946A (en) 1998-03-09 1999-12-14 Hewlett-Packard Company Collaborative drawing device
US6000621A (en) 1995-12-21 1999-12-14 Xerox Corporation Tilings of mono-code and dual-code embedded data pattern strips for robust asynchronous capture
US6005973A (en) 1993-12-01 1999-12-21 Motorola, Inc. Combined dictionary based and likely character string method of handwriting recognition
US6041335A (en) 1997-02-10 2000-03-21 Merritt; Charles R. Method of annotating a primary image with an image and for transmitting the annotated primary image
US6044301A (en) 1998-04-29 2000-03-28 Medtronic, Inc. Audible sound confirmation of programming change in an implantable medical device
US6044165A (en) 1995-06-15 2000-03-28 California Institute Of Technology Apparatus and method for tracking handwriting from visual input
US6052481A (en) 1994-09-02 2000-04-18 Apple Computers, Inc. Automatic method for scoring and clustering prototypes of handwritten stroke-based data
US6054990A (en) 1996-07-05 2000-04-25 Tran; Bao Q. Computer system with handwriting annotation
US6076734A (en) 1997-10-07 2000-06-20 Interval Research Corporation Methods and systems for providing human/computer interfaces
US6081261A (en) 1995-11-01 2000-06-27 Ricoh Corporation Manual entry interactive paper and electronic document handling and processing system
US6108453A (en) 1998-09-16 2000-08-22 Intel Corporation General image enhancement framework
US6141014A (en) 1995-04-20 2000-10-31 Hitachi, Ltd. Bird's-eye view forming method, map display apparatus and navigation system
US6151424A (en) 1994-04-28 2000-11-21 Hsu; Shin-Yi System for identifying objects and features in an image
US6157935A (en) 1996-12-17 2000-12-05 Tran; Bao Q. Remote data access and management system
US6181329B1 (en) 1997-12-23 2001-01-30 Ricoh Company, Ltd. Method and apparatus for tracking a hand-held writing instrument with multiple sensors that are calibrated by placing the writing instrument in predetermined positions with respect to the writing surface
US6188392B1 (en) 1997-06-30 2001-02-13 Intel Corporation Electronic pen device
US6186405B1 (en) 1997-03-24 2001-02-13 Olympus Optical Co., Ltd. Dot code and code reading apparatus
US6192380B1 (en) 1998-03-31 2001-02-20 Intel Corporation Automatic web based form fill-in
US6202060B1 (en) 1996-10-29 2001-03-13 Bao Q. Tran Data management system
US6208894B1 (en) 1997-02-26 2001-03-27 Alfred E. Mann Foundation For Scientific Research And Advanced Bionics System of implantable devices for monitoring and/or affecting body parameters
US6219149B1 (en) 1997-04-01 2001-04-17 Fuji Xerox Co., Ltd. Print processing apparatus
US6226636B1 (en) 1998-11-20 2001-05-01 Philips Electronics North America Corp. System for retrieving images using a database
US6230304B1 (en) 1997-12-24 2001-05-08 Magma Design Automation, Inc. Method of designing a constraint-driven integrated circuit layout
US6243071B1 (en) 1993-11-03 2001-06-05 Apple Computer, Inc. Tool set for navigating through an electronic book
US6249614B1 (en) 1998-03-06 2001-06-19 Alaris, Inc. Video compression and decompression using dynamic quantization and/or encoding
US6256398B1 (en) * 1998-08-22 2001-07-03 Kenneth H. P. Chang Encoding and decoding a message within an image
US20030095098A1 (en) * 1999-05-25 2003-05-22 Lapstun Paul Computer system interface surface with reference points and processing sensor
US20030193515A1 (en) * 1998-10-07 2003-10-16 William Hill High resolution display of image data using pixel sub-components
US20040056191A1 (en) * 2002-07-31 2004-03-25 General Electric Company Ion mobility spectrometers with improved resolution
US20040086181A1 (en) * 2002-10-31 2004-05-06 Microsoft Corporation Active embedded interaction code

Family Cites Families (32)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6573887B1 (en) * 1996-04-22 2003-06-03 O'donnell, Jr. Francis E. Combined writing instrument and digital documentor
US6330976B1 (en) * 1998-04-01 2001-12-18 Xerox Corporation Marking medium area with encoded identifier for producing action through network
JPH11355547A (en) * 1998-05-22 1999-12-24 Internatl Business Mach Corp <Ibm> Geometrical transformation specification system
US20020048404A1 (en) * 2000-03-21 2002-04-25 Christer Fahraeus Apparatus and method for determining spatial orientation
US20010038383A1 (en) * 2000-04-05 2001-11-08 Petter Ericson Method and apparatus for information management
US20020050982A1 (en) * 2000-04-05 2002-05-02 Petter Ericson Data form having a position-coding pattern detectable by an optical sensor
US20030050803A1 (en) * 2000-07-20 2003-03-13 Marchosky J. Alexander Record system
US20030030638A1 (en) * 2001-06-07 2003-02-13 Karl Astrom Method and apparatus for extracting information from a target area within a two-dimensional graphical object in an image
US6732927B2 (en) * 2001-06-26 2004-05-11 Anoto Ab Method and device for data decoding
SE520682C2 (en) * 2001-12-06 2003-08-12 Anoto Ab Reconstruction of a virtual screen
US7123742B2 (en) * 2002-04-06 2006-10-17 Chang Kenneth H P Print user interface system and its applications
US7116840B2 (en) * 2002-10-31 2006-10-03 Microsoft Corporation Decoding and error correction in 2-D arrays
US7430497B2 (en) * 2002-10-31 2008-09-30 Microsoft Corporation Statistical model for global localization
US7009594B2 (en) * 2002-10-31 2006-03-07 Microsoft Corporation Universal computing device
US7036938B2 (en) * 2002-10-31 2006-05-02 Microsoft Corporation Pen projection display
US7133563B2 (en) * 2002-10-31 2006-11-07 Microsoft Corporation Passive embedded interaction code
US7136054B2 (en) * 2004-01-06 2006-11-14 Microsoft Corporation Camera-pen-tip mapping and calibration
US7583842B2 (en) * 2004-01-06 2009-09-01 Microsoft Corporation Enhanced approach of m-array decoding and error correction
US7263224B2 (en) * 2004-01-16 2007-08-28 Microsoft Corporation Strokes localization by m-array decoding and fast image matching
US7505982B2 (en) * 2004-12-03 2009-03-17 Microsoft Corporation Local metadata embedding solution
US7536051B2 (en) * 2005-02-17 2009-05-19 Microsoft Corporation Digital pen calibration by local linearization
US7607076B2 (en) * 2005-02-18 2009-10-20 Microsoft Corporation Embedded interaction code document
US7477784B2 (en) * 2005-03-01 2009-01-13 Microsoft Corporation Spatial transforms from displayed codes
US7599560B2 (en) * 2005-04-22 2009-10-06 Microsoft Corporation Embedded interaction code recognition
US7421439B2 (en) * 2005-04-22 2008-09-02 Microsoft Corporation Global metadata embedding and decoding
US20060242562A1 (en) * 2005-04-22 2006-10-26 Microsoft Corporation Embedded method for embedded interaction code array
US7542976B2 (en) * 2005-04-22 2009-06-02 Microsoft Corporation Local metadata embedding and decoding
US7400777B2 (en) * 2005-05-25 2008-07-15 Microsoft Corporation Preprocessing for information pattern analysis
US7580576B2 (en) * 2005-06-02 2009-08-25 Microsoft Corporation Stroke localization and binding to electronic document
US7528848B2 (en) * 2005-06-30 2009-05-05 Microsoft Corporation Embedded interaction code decoding for a liquid crystal display
US7817816B2 (en) * 2005-08-17 2010-10-19 Microsoft Corporation Embedded interaction code enabled surface type identification
US7622182B2 (en) * 2005-08-17 2009-11-24 Microsoft Corporation Embedded interaction code enabled display

Patent Citations (106)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4941124A (en) 1982-01-25 1990-07-10 Skinner Jr James T Text comparator with counter shift register
US5073966A (en) 1983-05-10 1991-12-17 Canon Kabushiki Kaisha Image data processing apparatus
US4742558A (en) 1984-02-14 1988-05-03 Nippon Telegraph & Telephone Public Corporation Image information retrieval/display apparatus
US4745269A (en) 1985-05-22 1988-05-17 U.S. Philips Corporation Method of identifying objects provided with a code field containing a dot code, a device for identifying such a dot code, and a product provided with such a dot code
US4829583A (en) 1985-06-03 1989-05-09 Sino Business Machines, Inc. Method and apparatus for processing ideographic characters
US4686329A (en) 1985-06-21 1987-08-11 Advanced Robotic Technology, Inc. Absolute position mouse
US5365598A (en) 1986-07-25 1994-11-15 Ast Research, Inc. Handwritten keyboardless entry computer system
US5612524A (en) 1987-11-25 1997-03-18 Veritec Inc. Identification symbol system and method with orientation mechanism
US5196875A (en) 1988-08-03 1993-03-23 RoyoCad Gesellschaft fur Hard-und Software mbH Projection head
US5635697A (en) 1989-03-01 1997-06-03 Symbol Technologies, Inc. Method and apparatus for decoding two-dimensional bar code
US5032924A (en) 1989-04-10 1991-07-16 Nilford Laboratories, Inc. System for producing an image from a sequence of pixels
US5253336A (en) 1989-04-20 1993-10-12 Kabushiki Kaisha Toshiba Method for fitting third-degree Bezier curves to series of data points
US5153928A (en) 1989-06-09 1992-10-06 Casio Computer Co., Ltd. Method and apparatus for recording/reproducing mesh pattern data
US5454054A (en) 1989-06-09 1995-09-26 Casio Computer Co., Ltd. Method and apparatus for recording/reproducing mesh pattern data
US5051736A (en) 1989-06-28 1991-09-24 International Business Machines Corporation Optical stylus and passive digitizing tablet data input system
US5471533B1 (en) 1990-01-05 2000-04-18 Symbol Technologies Inc Record with encoded data
US5471533A (en) 1990-01-05 1995-11-28 Symbol Technologies, Inc. Record with encoded data
US5146552A (en) 1990-02-28 1992-09-08 International Business Machines Corporation Method for associating annotation with electronically published material
US5511156A (en) 1990-04-05 1996-04-23 Seiko Epson Corporation Interpreter for executing rasterize processing to obtain printing picture element information
US5181257A (en) 1990-04-20 1993-01-19 Man Roland Druckmaschinen Ag Method and apparatus for determining register differences from a multi-color printed image
US5442147A (en) 1991-04-03 1995-08-15 Hewlett-Packard Company Position-sensing apparatus
US5581682A (en) 1991-06-28 1996-12-03 International Business Machines Corporation Method for storing and retrieving annotations and redactions in final form documents
US5247137A (en) 1991-10-25 1993-09-21 Mark Epperson Autonomous computer input device and marking instrument
US5448372A (en) 1991-12-18 1995-09-05 Eastman Kodak Company Storage and retrieval of digitized photographic images
US5294792A (en) 1991-12-31 1994-03-15 Texas Instruments Incorporated Writing tip position sensing and processing apparatus
US5587558A (en) 1992-01-24 1996-12-24 Seiko Instruments Inc. Coordinate detecting apparatus having acceleration detectors
US5756981A (en) 1992-02-27 1998-05-26 Symbol Technologies, Inc. Optical scanner for reading and decoding one- and-two-dimensional symbologies at variable depths of field including memory efficient high speed image processing means and high accuracy image analysis means
US5852434A (en) 1992-04-03 1998-12-22 Sekendur; Oral F. Absolute optical position determination
US5477012A (en) 1992-04-03 1995-12-19 Sekendur; Oral F. Optical position determination
US5243149A (en) 1992-04-10 1993-09-07 International Business Machines Corp. Method and apparatus for improving the paper interface to computing systems
US5235654A (en) 1992-04-30 1993-08-10 International Business Machines Corporation Advanced data capture architecture data processing system and method for scanned images of document forms
US5546515A (en) 1992-07-08 1996-08-13 Matsushita Electric Industrial Co., Ltd. Image processing apparatus
US5822465A (en) 1992-09-01 1998-10-13 Apple Computer, Inc. Image encoding by vector quantization of regions of an image and codebook updates
US5288986A (en) 1992-09-17 1994-02-22 Motorola, Inc. Binary code matrix having data and parity bits
US5670897A (en) 1992-11-05 1997-09-23 Xilinx, Inc. High speed mask register for a configurable cellular array
US5450603A (en) 1992-12-18 1995-09-12 Xerox Corporation SIMD architecture with transfer register or value source circuitry connected to bus
US5818436A (en) 1993-03-15 1998-10-06 Kabushiki Kaisha Toshiba Apparatus and method for playing back continuous data
US5414227A (en) 1993-04-29 1995-05-09 International Business Machines Corporation Stylus tilt detection apparatus for communication with a remote digitizing display
US5398082A (en) 1993-05-20 1995-03-14 Hughes-Jvc Technology Corporation Scanned illumination for light valve video projectors
US5335150A (en) 1993-08-31 1994-08-02 Huang Chao C Laser pointer with a constant power output control
US5394487A (en) 1993-10-27 1995-02-28 International Business Machines Corporation Forms recognition management system and method
US6243071B1 (en) 1993-11-03 2001-06-05 Apple Computer, Inc. Tool set for navigating through an electronic book
US5644652A (en) 1993-11-23 1997-07-01 International Business Machines Corporation System and method for automatic handwriting recognition with a writer-independent chirographic label alphabet
US5721940A (en) 1993-11-24 1998-02-24 Canon Information Systems, Inc. Form identification and processing system using hierarchical form profiles
US5629499A (en) 1993-11-30 1997-05-13 Hewlett-Packard Company Electronic board to store and transfer information
US6005973A (en) 1993-12-01 1999-12-21 Motorola, Inc. Combined dictionary based and likely character string method of handwriting recognition
US5875264A (en) 1993-12-03 1999-02-23 Kaman Sciences Corporation Pixel hashing image recognition system
US5726435A (en) 1994-03-14 1998-03-10 Nippondenso Co., Ltd. Optically readable two-dimensional code and method and apparatus using the same
US6151424A (en) 1994-04-28 2000-11-21 Hsu; Shin-Yi System for identifying objects and features in an image
US5825015A (en) 1994-04-29 1998-10-20 Electronic Automation Limited Machine readable binary codes
US5897648A (en) 1994-06-27 1999-04-27 Numonics Corporation Apparatus and method for editing electronic documents
US5652412A (en) 1994-07-11 1997-07-29 Sia Technology Corp. Pen and paper information recording system
US5774602A (en) 1994-07-13 1998-06-30 Yashima Electric Co., Ltd. Writing device for storing handwriting
US5960124A (en) 1994-07-13 1999-09-28 Yashima Electric Co., Ltd. Image reproducing method for reproducing handwriting
US6052481A (en) 1994-09-02 2000-04-18 Apple Computers, Inc. Automatic method for scoring and clustering prototypes of handwritten stroke-based data
US5727098A (en) 1994-09-07 1998-03-10 Jacobson; Joseph M. Oscillating fiber optic display and imager
US5661506A (en) 1994-11-10 1997-08-26 Sia Technology Corporation Pen and paper information recording system using an imaging pen
US5855483A (en) 1994-11-21 1999-01-05 Compaq Computer Corp. Interactive play with a computer
US5581637A (en) 1994-12-09 1996-12-03 Xerox Corporation System for registering component image tiles in a camera-based scanner device transcribing scene images
US5961571A (en) 1994-12-27 1999-10-05 Siemens Corporated Research, Inc Method and apparatus for automatically tracking the location of vehicles
US5939703A (en) 1995-01-03 1999-08-17 Xerox Corporation Distributed dimensional labeling for dimensional characterization of embedded data blocks
US5626620A (en) 1995-02-21 1997-05-06 Medtronic, Inc. Dual chamber pacing system and method with continual adjustment of the AV escape interval so as to maintain optimized ventricular pacing for treating cardiomyopathy
US5686718A (en) 1995-03-15 1997-11-11 Sharp Kabushiki Kaisha Recording method, decoding method, and decoding apparatus for digital information
US6141014A (en) 1995-04-20 2000-10-31 Hitachi, Ltd. Bird's-eye view forming method, map display apparatus and navigation system
US5754280A (en) 1995-05-23 1998-05-19 Olympus Optical Co., Ltd. Two-dimensional rangefinding sensor
US5898166A (en) 1995-05-23 1999-04-27 Olympus Optical Co., Ltd. Information reproduction system which utilizes physical information on an optically-readable code and which optically reads the code to reproduce multimedia information
US5661291A (en) 1995-06-07 1997-08-26 Hand Held Products, Inc. Audio proof of delivery system and method
US6044165A (en) 1995-06-15 2000-03-28 California Institute Of Technology Apparatus and method for tracking handwriting from visual input
US5719884A (en) 1995-07-27 1998-02-17 Hewlett-Packard Company Error correction method and apparatus based on two-dimensional code array with reduced redundancy
US6081261A (en) 1995-11-01 2000-06-27 Ricoh Corporation Manual entry interactive paper and electronic document handling and processing system
US5850058A (en) 1995-11-17 1998-12-15 Hitachi, Ltd. Information processor
US6000621A (en) 1995-12-21 1999-12-14 Xerox Corporation Tilings of mono-code and dual-code embedded data pattern strips for robust asynchronous capture
US5902968A (en) 1996-02-20 1999-05-11 Ricoh Company, Ltd. Pen-shaped handwriting input apparatus using accelerometers and gyroscopes and an associated operational device for determining pen movement
US5890177A (en) 1996-04-24 1999-03-30 International Business Machines Corporation Method and apparatus for consolidating edits made by multiple editors working on multiple document copies
US5822436A (en) 1996-04-25 1998-10-13 Digimarc Corporation Photographic products and methods employing embedded information
US5692073A (en) 1996-05-03 1997-11-25 Xerox Corporation Formless forms and paper web using a reference-based mark extraction technique
US6054990A (en) 1996-07-05 2000-04-25 Tran; Bao Q. Computer system with handwriting annotation
US5765176A (en) 1996-09-06 1998-06-09 Xerox Corporation Performing document image management tasks using an iconic image having embedded encoded information
US5825892A (en) * 1996-10-28 1998-10-20 International Business Machines Corporation Protecting images with an image watermark
US6202060B1 (en) 1996-10-29 2001-03-13 Bao Q. Tran Data management system
US6157935A (en) 1996-12-17 2000-12-05 Tran; Bao Q. Remote data access and management system
US6208771B1 (en) 1996-12-20 2001-03-27 Xerox Parc Methods and apparatus for robust decoding of glyph address carpets
US6000614A (en) 1996-12-20 1999-12-14 Denso Corporation Two-dimensional code reading apparatus
US5937110A (en) 1996-12-20 1999-08-10 Xerox Corporation Parallel propagating embedded binary sequences for characterizing objects in N-dimensional address space
US5995084A (en) 1997-01-17 1999-11-30 Tritech Microelectronics, Ltd. Touchpad pen-input and mouse controller
US6041335A (en) 1997-02-10 2000-03-21 Merritt; Charles R. Method of annotating a primary image with an image and for transmitting the annotated primary image
US6208894B1 (en) 1997-02-26 2001-03-27 Alfred E. Mann Foundation For Scientific Research And Advanced Bionics System of implantable devices for monitoring and/or affecting body parameters
US5817992A (en) 1997-03-05 1998-10-06 Rpg Diffusor Systems, Inc. Planar binary amplitude diffusor
US6186405B1 (en) 1997-03-24 2001-02-13 Olympus Optical Co., Ltd. Dot code and code reading apparatus
US6219149B1 (en) 1997-04-01 2001-04-17 Fuji Xerox Co., Ltd. Print processing apparatus
US6188392B1 (en) 1997-06-30 2001-02-13 Intel Corporation Electronic pen device
US5855594A (en) 1997-08-08 1999-01-05 Cardiac Pacemakers, Inc. Self-calibration system for capture verification in pacing devices
US6076734A (en) 1997-10-07 2000-06-20 Interval Research Corporation Methods and systems for providing human/computer interfaces
US6181329B1 (en) 1997-12-23 2001-01-30 Ricoh Company, Ltd. Method and apparatus for tracking a hand-held writing instrument with multiple sensors that are calibrated by placing the writing instrument in predetermined positions with respect to the writing surface
US6230304B1 (en) 1997-12-24 2001-05-08 Magma Design Automation, Inc. Method of designing a constraint-driven integrated circuit layout
US6249614B1 (en) 1998-03-06 2001-06-19 Alaris, Inc. Video compression and decompression using dynamic quantization and/or encoding
US6000946A (en) 1998-03-09 1999-12-14 Hewlett-Packard Company Collaborative drawing device
US6192380B1 (en) 1998-03-31 2001-02-20 Intel Corporation Automatic web based form fill-in
US6044301A (en) 1998-04-29 2000-03-28 Medtronic, Inc. Audible sound confirmation of programming change in an implantable medical device
US6256398B1 (en) * 1998-08-22 2001-07-03 Kenneth H. P. Chang Encoding and decoding a message within an image
US6108453A (en) 1998-09-16 2000-08-22 Intel Corporation General image enhancement framework
US20030193515A1 (en) * 1998-10-07 2003-10-16 William Hill High resolution display of image data using pixel sub-components
US6226636B1 (en) 1998-11-20 2001-05-01 Philips Electronics North America Corp. System for retrieving images using a database
US20030095098A1 (en) * 1999-05-25 2003-05-22 Lapstun Paul Computer system interface surface with reference points and processing sensor
US20040056191A1 (en) * 2002-07-31 2004-03-25 General Electric Company Ion mobility spectrometers with improved resolution
US20040086181A1 (en) * 2002-10-31 2004-05-06 Microsoft Corporation Active embedded interaction code

Non-Patent Citations (99)

* Cited by examiner, † Cited by third party
Title
"VPEN, Revolutionizing Human Interaction With The Digital World(TM)," OTM Technologies, pp. 1-6, Jul. 15, 2002.
Brush, A.J. et al., "Robust Annotation Positioning in Digital Documents," SIGCHI '01, Mar. 31-Apr. 4, 2001, ACM, Seattle, Washington, USA, pp. 285-292.
Cai, Z., "A New Decode Algorithm for Binary Bar Codes," Pattern Recognition Letters 15 (Dec. 1994), pp. 1191-1199.
Champaneria, "PADCAM: A Real-Time, Human-Centric Notetaking System," MIT Laboratory for Computer Science, Mar. 2002.
Clark et al., "Maximal and Near-Maximal Shift Register Sequences: Efficient Event Counters and Easy Discrete Logarithms," IEEE Transactions on Computers vol. 43, No. 5, pp. 560-568, May 1994.
Cotting, D. et al., "Embedding Imperceptible Patterns into Projected Images for Simultaneous Acquisition and Display," Proceedings of the Third IEEE and ACM International Symposium on Mixed and Augmented Reality, Nov. 2-5, 2004, IEEE Computer Society, Washington, DC, pp. 100-109.
Crowley et al., "ThingssThat See," Communications of the A.C.M., vol. 43, No. 3, pp. 54-64, Mar. 2000.
Decurtins, C. et al., "Digital Annotation of Printed Documents," Proceedings of the Twelfth International Conference on Information and Knowledge Management Nov. 3-8, New Orleans, Louisiana, United States, CIKM'03. ACM 2003, pp. 552-555.
Dey, et al., "A Fast Algorithm for Computing the Euler Number of an Image and its VLSI Implementation," IEEE; 13th International Conference on VLSI Design, Jan. 2003.
Dumer et al., "Hardness of Approximating the Minimum Distance of a Linear Code," IEEE Symposium of Foundations of Computer Science, pp. 475-485, 1999.
European Search Report for Application No. 03021224.5; Applicant: Microsoft Corporation; Date of Mailing: Jun. 1, 2005 (3 pages).
European Search Report for Application No. 03021236.9; Applicant: Microsoft Corporation; Date of Mailing: Sep. 16, 2005 (5 pages).
European Search Report for Application No. 03021237.7-1527, Applicant: Microsoft Corporation; Date of Mailing: Jan. 6, 2005 (4 pages).
European Search Report for Application No. EP 03021235; Applicant: Microsoft Corporation; Date of Mailing: Jun. 1, 2005 (2 pages).
European Search Report for Application No. EP 03021852; Applicant: Microsoft Corporation; Date of Mailing: Mar. 2, 2004 (3 pages).
European Search Report for Application No. EP 05000170.0-1527; Applicant: Microsoft Corporation; Date of Mailing: Jan. 6, 2005 (7 pages).
European Search Report for Application No. EP050000749; Applicant: Microsoft Corporation; Date of Mailing: Apr. 26, 2007 (2 pages).
European Search Report, Application No. 03021238.5-1527; Applicant: Microsoft Corporation; Date of Mailing: Jun. 1, 2005 (6 pages).
Fujieda et al., "Development Of Pen-Shaped Scanners," Nec, vol. 51, No. 10, 1998.
Golovchinsky, G. and Denoue, L., "Moving Markup: Repositioning Freeform Annotations," UIST '02, Oct. 27-30, 2002, Paris, France, vol. 4, Issue 2, pp. 21-24.
Gonzalez, Rafael et al., "Digital Image Processing," 2002, Table of Contents and Preface, Second Edition, Prentice Hall, Upper Saddle Rive, New Jersey.
Grasso et al., "Augmenting Recommender Systems by Embedding Interfaces into Practices," pp. 267-275, 1999.
Guerrero, J.J. and Sagues, C. "From Lines to Homographies Between Uncalibrated Images," IX Symposium on Pattern Recognition and Image Analysis, VO4, 233-240, 2001.
Haynes, "Wacom PL-500," www.wacom.co.uk, Jul. 15, 2002.
Hecht, D.L., "Printed embedded data graphical user interfaces," Computer vol. 34, Issue 3, Mar. 2001, pp. 47-55.
IEEExplore # Search Session History, May 7, 2008, http://ieee.org/search/history.jsp, 1 page.
International Search Report for Application No. PCT/US2006/032230; Applicant: Microsoft Corporation; Date of Mailing: Jan. 9, 2007 (3 pages).
Internet Print Out "Products-Get Everyone On The Same Page," Mimio, dated Sep. 5, 2003.
Internet Print Out: "(Electro-Magnetic Resonance) send and position sensing technology," Wacom, Technology, http://www.wacom-components.com/tech.asp, pp. 1-6, Jul 15, 2002.
Internet Print Out: "Anoto functionality," News, dated Jul. 15, 2002.
Internet Print Out: "Anoto functionality," Showroom, dated Jul. 15, 2002.
Internet Print Out: "Anoto Pen Bluetooth," Tabletpccorner, http://www.tabletpccorner.net, dated Sep. 5, 2003.
Internet Print Out: "ChatPen CHA-30," Digital Pens, Anoto Functionality, dated Jul. 15, 2002.
Internet Print Out: "Cintiq18SX-A Powerful New Way To Work Directly On The Screen," Wacom Technology, Cintiq-Interactive Pen Display, dated Sep. 5, 2003.
Internet Print Out: "Communicate Digitally With Ordinary Pen and Paper," Anoto Functionality, dated Jul. 15, 2002.
Internet Print Out: "Competitive Technologies' Investee Introduces N-Scribe Pen-Digital Ink Presents Wireless Pen At Demo 2001," Competitive Technologies, http://www.competitivetech, dated Sep. 5, 2003.
Internet Print Out: "Cordless Batteryless Pen," Wacom Penabled, Components, http://www.wacom.com/components/index.cfm, dated Jul. 15, 2002.
Internet Print Out: "Creating A Global De Facto Standard," Anoto Functionality, dated Jul. 15, 2002.
Internet Print Out: "Daily News," dated Aug. 15, 2002.
Internet Print Out: "DataGlyphs(R): Embedding Digital Data," Parc Solutions, http://www.parc.com/solutions/dataglyphs/, dated Jul. 15, 2002.
Internet Print Out: "Digital Pens and Technical Data," Anoto Functionality, dated Jul. 15, 2002.
Internet Print Out: "Digital Pens," http://www.anotofunctionality.com/navigate.asp?PageID=96, pp. 1-70, Sep. 26, 2002.
Internet Print Out: "Downloads," Anoto Functionality, dated Jul. 15, 2002.
Internet Print Out: "Jot This-Nscribe Pen," PC Magazine, http://ww.pcmag.com/article2/0,4149,316500,00.asp, dated Jul. 15, 2002.
Internet Print Out: "Jot This-Nscribe Pen," PC Magazine, http://ww.pcmag.com/article2/0,4149,316500,00.asp, dated Sep. 5, 2003.
Internet Print Out: "Maxell Digital Pen To Use Anoto System," Gizmo, http://www.gizmo.com.au/, dated Sep. 5, 2003.
Internet Print Out: "Mimio-Products," Mimio, http://www.mimio.com, pp. 1-8, Sep. 5, 2003.
Internet Print Out: "Navilite-Optical Opportunities-Bluetooth-enabled optical transition measurement technology paves the way for an untethered stylus that can write on any surface," vol. 8, Issue No. 34, Jul. 5-11, 2002. www.computerworld.com, dated Aug. 15, 2002.
Internet Print Out: "N-Scribe For Digital Writing," Flash Commerce News, http://flashcommerce.com/articles/, dated Sep. 5, 2003.
Internet Print Out: "N-Scribe For Digital Writing," Mobileinfo.com, News issue #2001-15 (Apr. 2001), http://www.mobileinfo.com/News-2001/Issue15/Digital-nscribe.htm, dated Jul. 15, 2002.
Internet Print Out: "Optical Translation Measurement (OTM(TM))," Technologies, dated Jul. 15, 2002.
Internet Print Out: "OTM Technologies-V Pen," searched http://www.otmtech.com/vpen3.asp, pp. 1-7, Jul. 15, 2002.
Internet Print Out: "Partners Supporting Anoto Functionality," Anoto Functionality, dated 15, 2002.
Internet Print Out: "PL-500-15.1 inch Screen Digital LCD Pen Tablet System," Wacom Graphics Tablet Solution, http://ap.wacm.co.jp/products/pl/contents/pl500.html, pp. 1-13, 2002.
Internet Print Out: "Possibilities," Anoto Functionality, dated Jul. 15, 2002.
Internet Print Out: "Preprocessing in the Npen++ System," http://www.is.cs.cmu.edu/mie/multimodal-npen-preproc.html, dated Aug. 8, 2002.
Internet Print Out: "Product VPen(TM)," OTM Technologies, dated Jul. 15, 2002.
Internet Print Out: "RF Pen Sends Your Scribbles," Appliance Manufacturing Magazine, http://www.ammagazine.com. Dated Sep. 26, 2002.
Internet Print Out: "Sensor Board and Pen," Wacom, Product, dated Jul. 15, 2002.
Internet Print Out: "Smart Board Interactive Whiteboard," Smarttech, http://www.smarttech.com, pp. 1-26, Sep. 5, 2003.
Internet Print Out: "The Hot New Medium: Paper-How The Oldest Interface In The Book is Redrawing The Map Of The Networked World," http://www.wired.com/wired/, dated Sep. 5, 2003.
Internet Print Out: "The Solution," Anoto Functionality, dated Jul. 15, 2002.
Internet Print Out: "Vision and Mission," Anoto Functionality, dated Jul. 15, 2002.
Internet Print Out: "Wacom Displays Pressure-Sensitive Pen Sensor for Tablet PCs," Wacom, News, dated Jul. 15, 2002.
Internet Print Out: "Welcome To www.anoto.com," Anoto, dated Jul. 15, 2002.
Internet Printout-http://www.anoto.com: Construction, Sep. 5, 2003.
Internet Printout-http://www.anoto.com: Page template, Sep. 5, 2003.
Internet Printout-http://www.anoto.com: Paper and Printing, Sep. 5, 2003.
Internet Printout-http://www.anoto.com: Paper space, Sep. 5, 2003.
Internet Printout-http://www.anoto.com: Pattern, Sep. 5, 2003.
Internet Printout-http://www.anoto.com: Printers supporting Anoto functionality, Sep. 5, 2003.
Internet Printout-http://www.anotofunctionality.com: Anoto Functionality-Anoto pattern & digital paper, Sep. 26, 2002.
Internet Printout-http://www.anotofunctionality.com: Anoto Functionality-Anoto pattern & digital paper, Sep. 5, 2003.
Internet Printout-http://www.anotofunctionality.com: Anoto Functionality-Applications, Sep. 5, 2003.
Internet Printout-http://www.anotofunctionality.com: Anoto Functionality-Corporate applications, Sep. 5, 2003.
Internet Printout-http://www.anotofunctionality.com: Anoto Functionality-Digital notes, Sep. 5, 2003.
Internet Printout-http://www.anotofunctionality.com: Anoto Functionality-Digital paper, Sep. 26, 2002.
Internet Printout-http://www.anotofunctionality.com: Anoto Functionality-Digital paper, Sep. 5, 2003.
Internet Printout-http://www.anotofunctionality.com: Anoto Functionality-Digital pens Use with mobile phones, Sep. 5, 2003.
Internet Printout-http://www.anotofunctionality.com: Anoto Functionality-Digital Pens Use with personal computers, Sep. 26, 2002.
Internet Printout-http://www.anotofunctionality.com: Anoto Functionality-Digital Pens, Sep. 26, 2002.
Internet Printout-http://www.anotofunctionality.com: Anoto Functionality-Digital pens, Sep. 5, 2003.
Internet Printout-http://www.anotofunctionality.com: Anoto Functionality-Digital service, Sep. 26, 2002.
Internet Printout-http://www.anotofunctionality.com: Anoto Functionality-Digital service, Sep. 5, 2003.
Internet Printout-http://www.anotofunctionality.com: Anoto Functionality-E-mail, Sep. 5, 2003.
Internet Printout-http://www.anotofunctionality.com: Anoto Functionality-Fax, Sep. 5, 2003.
Internet Printout-http://www.anotofunctionality.com: Anoto Functionality-Freedom of expression, Sep. 26, 2002.
Internet Printout-http://www.anotofunctionality.com: Anoto Functionality-Graphical SMS, Sep. 5, 2003.
Internet Printout-http://www.anotofunctionality.com: Anoto Functionality-Hints & tips Using your digital paper, Sep. 5, 2003.
Internet Printout-http://www.anotofunctionality.com: Anoto Functionality-Hints & tips Using your digital pen, Sep. 5, 2003.
Internet Printout-http://www.anotofunctionality.com: Anoto Functionality-Hints & tips Using Your Digital Service, Sep. 5, 2003.
Internet Printout-http://www.anotofunctionality.com: Anoto Functionality-Hints & tips, Sep. 5, 2003.
Internet Printout-http://www.anotofunctionality.com: Anoto Functionality-How does it work?, Sep. 26, 2002.
Internet Printout-http://www.anotofunctionality.com: Anoto Functionality-Security, Sep. 5, 2003.
U.S. Appl. No. 11/066,800, filed Feb. 25, 2005, Wang, et al.
U.S. Appl. No. 11/067,193, filed Feb. 25, 2005, Yang, et al.
U.S. Appl. No. 12/131,810, filed Jun. 2, 2008, Wang.
U.S. Appl. No. 12/138,339, filed Jun. 12, 2008, Wang et al.
U.S. Appl. No. 12/180,484, filed Jul. 25, 2008, Wang.

Cited By (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7889884B2 (en) * 2005-03-28 2011-02-15 Fuji Xerox Co., Ltd. Image processing apparatus and method
US20060215183A1 (en) * 2005-03-28 2006-09-28 Fuji Xerox Co., Ltd. Image processing apparatus and method
US7817816B2 (en) 2005-08-17 2010-10-19 Microsoft Corporation Embedded interaction code enabled surface type identification
US9557827B2 (en) 2011-12-16 2017-01-31 3M Innovative Properties Company Optical digitizer system with position-unique photoluminescent indicia
US9068845B2 (en) 2011-12-16 2015-06-30 3M Innovative Properties Company Optical digitizer system with position-unique photoluminescent indicia
US20130321357A1 (en) * 2012-06-01 2013-12-05 Panasonic Corporation Display panel and display device
US9836164B2 (en) 2012-10-29 2017-12-05 3M Innovative Properties Company Optical digitizer system with position-unique photoluminescent indicia
US9075452B2 (en) 2012-10-29 2015-07-07 3M Innovative Properties Company Optical digitizer system with position-unique photoluminescent indicia
US8692212B1 (en) 2012-10-29 2014-04-08 3M Innovative Properties Company Optical digitizer system with position-unique photoluminescent indicia
US9946947B2 (en) 2012-10-31 2018-04-17 Cognex Corporation System and method for finding saddle point-like structures in an image and determining information from the same
US10753746B2 (en) 2012-11-29 2020-08-25 3M Innovative Properties, Inc. Multi-mode stylus and digitizer system
US9958954B2 (en) 2012-12-13 2018-05-01 3M Innovative Properties Company System and methods for calibrating a digitizer system
US10429950B2 (en) * 2014-07-16 2019-10-01 Hewlett-Packard Development Company, L.P. Diffusive emissive display
US10762405B2 (en) 2017-10-26 2020-09-01 Datalogic Ip Tech S.R.L. System and method for extracting bitstream data in two-dimensional optical codes
US11010651B1 (en) 2018-11-26 2021-05-18 National Technology & Engineering Solutions Of Sandia, Llc Optically configurable charge-transfer materials and methods thereof
US11222249B2 (en) 2018-11-26 2022-01-11 National Technology & Engineering Solutions Of Sandia, Llc Optically configurable charge-transfer materials and methods thereof

Also Published As

Publication number Publication date
US20070001950A1 (en) 2007-01-04

Similar Documents

Publication Publication Date Title
US7619607B2 (en) Embedding a pattern design onto a liquid crystal display
US7528848B2 (en) Embedded interaction code decoding for a liquid crystal display
US20240078646A1 (en) Image processing method, image processing apparatus, and non-transitory storage medium
US7580576B2 (en) Stroke localization and binding to electronic document
US7477784B2 (en) Spatial transforms from displayed codes
US7048198B2 (en) Coded pattern for an optical device and a prepared surface
RU2360303C2 (en) Positionally encoded document image analysis and labelling
KR101026580B1 (en) Active embedded interaction coding
US6935562B2 (en) Operations on images having glyph carpets
KR101312830B1 (en) Embedded interaction code enabled surface type identification
US7403658B2 (en) Direct homography computation by local linearization
US6678425B1 (en) Method and apparatus for decoding angular orientation of lattice codes
RU2375740C2 (en) DETERMINATION OF LOCATION OF LINES BY m-ARRAY DECODING AND FAST COMPARISON OF IMAGES
US6880755B2 (en) Method and apparatus for display of spatially registered information using embedded data
WO2017193886A1 (en) Method and apparatus for generating two-dimensional code picture having dynamic effect
KR102423710B1 (en) Translucent image watermark detection
RU2347271C2 (en) Global localisation by prompt coordination of images
EP2591441B1 (en) Dot code pattern for absolute position and other information using an optical pen, process of printing the dot code, process of reading the dot code
US20120308138A1 (en) Multi-resolution spatial feature extraction for automatic handwriting recognition
CN116746155A (en) End-to-end watermarking system
EP1557793B1 (en) Method and apparatus for registering, displaying and decoding of information for spatial registration

Legal Events

Date Code Title Description
AS Assignment

Owner name: MICROSOFT CORPORATION, WASHINGTON

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ZHANG, CHUNHUI;WANG, JIAN;WANG, QIANG;REEL/FRAME:016268/0313

Effective date: 20050627

STCF Information on status: patent grant

Free format text: PATENTED CASE

CC Certificate of correction
CC Certificate of correction
FPAY Fee payment

Year of fee payment: 4

AS Assignment

Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC, WASHINGTON

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MICROSOFT CORPORATION;REEL/FRAME:034543/0001

Effective date: 20141014

FPAY Fee payment

Year of fee payment: 8

FEPP Fee payment procedure

Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

LAPS Lapse for failure to pay maintenance fees

Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STCH Information on status: patent discontinuation

Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362

FP Lapsed due to failure to pay maintenance fee

Effective date: 20211117