WO1999050736A1

WO1999050736A1 - Paper indexing of recordings

Info

Publication number: WO1999050736A1
Application number: PCT/US1998/020593
Authority: WO
Inventors: Marc Dymetman; Max Copperman
Original assignee: Xerox Corporation
Priority date: 1998-04-01
Filing date: 1998-09-30
Publication date: 1999-10-07
Also published as: WO1999050787A1; WO1999050751A1

Abstract

Access to a stored item of data is provided by using a manually positionable scanner (506) to scan one or more regions of a sheet and to provide a scan signal. The sheet bears machine-readable markings that define two or more zones of the sheet (108); the markings (204) within each zone indicating a position of the zone within the sheet. The scan signal is used to obtain position data indicating one or more positions indicated by the machine readable markings (606). One or more positions indicated by the position data are encoded with time information to obtain encoded position-time data. Time data indicating the time information are associated with the stored item of data to provide access to the stored item of data using the encoded position-time data.

Description

1

Paper Indexing Of Recordings

The present invention relates to information capture and processing, and more particularly to paper-based indexing of recordings.

EP-A-495612 discloses a notetaking system based on a notepad computer with an integrated audio/video recorder, whereby a document is created or retrieved.

As the user types on the computer's keyboard or writes with a stylus on a touchscreen, each character or stroke that is input by the user is invisibly time-stamped by the computer. The audio/video stream is continuously time-stamped during recording. To play a section of the recording back, the user selects part of the note and invokes a "playback selection" command. The computer then examines the time-stamp and winds the records toe the corresponding place in the audio/video recording, where it starts playing, so that the user hears and/or sees what was being recorded at the instant the selected text or strokes were input. (A similar system is Hewlett Packard's Filochart system). However, the abovementioned systems rely on the use of a dedicated computer with a touchscreen which electronically both captures the user's notes (or 'indicia') at the time of recording and displays the indicia for selection by the user in a 'playback' mode. The systems require a file to be created or retrieved before a user can start making notes in it or continue making notes in an already existing electronic document. By considering an example — a user being handed a set of papers accompanying a talk or in a meeting — it can be seen that the user cannot, using the aforementioned system, mark his notes or indicia on the paper he has just been given, but must break off from the meeting, create a new file or electronic document and set this up displayed on the screen of his computer so that his notes can be captured and recorded.

Indexing audio recordings via notes taken on paper using a pressure-sensitive clipboard has been reported (Lisa Stifelman's Audio Notebook, MIT Media Lab ).

The present invention provides a method carried out in an information processing system, for providing access to a data set, comprising: recording the data set and, concurrently therewith, recording user data, the user data being dependent upon user actions carried out during the recording of said data set. 2

The invention preferably provides a method carried out in an information processing system, for providing access to a data set, comprising: (a) providing a manually operable an positionable input device, the input device including a marking device and an image capture device intergral therewith for capturing images at or near a marking tip of the marking device, (b) providing a recording device for recording said data set, and means for automatically time-stamping said data set, (c) in response to the user, with respect to a substrate, the substrate comprising a sheet medium and coded machine-readable markings formed on the sheet medium, said markings being formed so as to define a plurality of zones on the substrate, each zone containing machine readable first markings defining a page identification code, the page identification code being unique to the substrate, (A) positioning the input device at a location on the substrate, or (B) causing the input device to perform a movement across the substrate, capturing images of the substrate within the field of view of said image capture device while the input device is positioned at said location or during said movement, (d) from the image captured in step (c), determining the page location codes within the or each zone of the substrate within said images, and automatically time-stamping said page-location codes upon their generation, said data set and page location codes being time-stamped with a common time base.

The invention further provides a method carried out in an information processing system for outputting randomly accessible parts of a recorded data set, the data set having been recorded according to the above methods, comprising: storing association data defining a mapping between user data and respective parts of said data set, and in response to a user input defining a user action selection, said user action selection defining a selected one of the user actions, using the user data corresponding to the selected user action, outputting the part(s) of the data set corresponding to said user action.

This invention preferably provides a method carried out in an information processing system for outputting randomly accessible parts of a recorded data set, the data set having been recorded according to the above methods, comprising: (k) providing a manually operable an positionable input device, the input device including an image capture device intergral therewith for capturing images at or near a tip of the input device, and providing an output device for outputting said data set, (1) in response to the user, with respect to a substrate, the substrate comprising a sheet 3 medium and coded machine-readable markings formed on the sheet medium, said markings being formed so as to define a plurality of zones on the substrate, each zone containing machine readable first markings defining a page identification code, the page identification code being unique to the substrate, (A) positioning the input device at a location on the substrate, or (B) causing the input device to perform a movement across the substrate, capturing images of the substrate within the field of view of said image capture device while the input device is positioned at said location or during said movement, (m) from the image captured in step (1), determining the page location codes within the or each zone of the substrate within said images, (n) from the page location code(s) determined in step (m), deriving the time-stamp(s) associated therewith, (o) from the time-stamp(s) derived in step (n), outputting corresponding part(s) of said data said using said output device.

The invention further provides a programmable information processing system when suitably programmed for carrying out the method of any of the preceding claims, the system including a processor, memory, a manually operable an positionable input device, the input device including a marking device and an image capture device intergral therewith for capturing images at or near a marking tip of the marking device, a recording device for recording said data set, means for automatically time-stamping said data set, and an output device for outputting said data set, the processor being operable in conjunction with the memory, input device, recording device, automatic time-stamping means, and output device, to execute instructions corresponding to said methods.

The invention is concerned with ways of indexing recordings with notes written on paper, in a number of embodiments. A preferred embodiment is via coded substrates (see GB application 98 (applicants' ref R/98003/JDR), which provides functionalities not provided by other embodiments (because they are standard functionalities of the coded substrates).

Thus, the invention may be embodied in a system for taking notes on paper during a recording (typically audio or video) and using them as an index to control playback of the recording. Extensions may allow the user to take notes either on paper or with an electronic device, and to control playback either with a paper copy or an electronic version of the notes. 4

The user makes a note on paper with a combined pen/camera or with pen or pencil; if with pen or pencil, either subsequently scanning in the marks or writing under the eye of a camera. An identifying feature of the note is computed. The feature is stored with the current time-stamp of the recording. This process may be repeated. A note may be subsequently selected. Its identifying feature is computed and matched against the set of stored features and the time-stamp associated with the matching element is recovered, and used to initiate replay of the associated portion of the recording.

A preferred embodiment uses substrates, e.g. paper, or documents produced therefrom, which include visible or invisible coded markings identifiying the substrate and preferably locations or zones within it. This marking scheme in turn preferably uses Xerox DataGlyphs.

An advantage is that the invention allows handwritten notes on 'standard' paper or other conventional media to index and control playback of recordings. In an alternative, handwritten paper notes may be timestamped without respect to an explicit recording (examples: ships log, news reporting). Handwritten paper notes may be time-correlated with instrumentation logs (examples: laboratory experiments, telescope parameters, surgery-room parameters, meteorological conditions). A variant would not use time, but directly associate parameter values (such as the telescope's position) with the note. In this case a specialized application would be needed to interface to the instrument.

In implementing the techniques according to the invention, the identifier routing techniques described in GB application 98 (applicants' ref

R/98005/JDR) may be employed. Embodiments of the invention will now be described, by way of example, with reference to the accompanying drawings, in which:

Figure 1 illustrates the components of a printed document as printed on a coded substrate;

Figure 2 shows a sample of zones, and the disposition of machine readable data, on a coded substrate;

Figures 3 and 4 show how digital data is encoded in the zones illustrated in Fig. 2; 5

Figure 5 schematically illustrates an embodiment of a pointer which may be used in implementing the invention;

Figure 6 shows a configuration for passing page identification codes and/or page location codes from the pointer of Fig. 5 to a network computer, in accordance with an embodiment of the invention;

Figure 7 illustrates an overview of a system suitable for implementing an embodiment of the invention;

Figure 8 is a schematic illustration of a first processing operation, in recording according to one embodiment of the invention, performed in the processor device of Fig. 7;

Figure 9 shows a schematic illustration of a second processing operation, in recording according to one embodiment of the invention, performed in the processor device of Fig. 7;

Figure 10 is a schematic illustration of a third processing operation, in recording according to one embodiment of the invention, performed in the processor device of Fig. 7;

Figure 11 is a schematic illustration of a fourth processing operation, in recording according to one embodiment of the invention, performed in the processor device of Fig. 7; Figure 12 shows a schematic illustration of a first processing operation, in retrieval according to one embodiment of the invention, performed in the processor device of Fig. 7;

Figure 13 is a schematic illustration of a second processing operation, in retrieval according to one embodiment of the invention, performed in the processor device of Fig. 7;

Figure 14 illustrates an overview of a system suitable for implementing an embodiment of the invention permitting paper annotaion for electronic retrieval;

Figure 15 is a schematic illustration of a first processing operation, in recording according to one embodiment of the invention, performed in the processor device of FigJ4;

Figure 16 shows a schematic illustration of a second processing operation, in recording according to one embodiment of the invention, performed in the processor device of Fig. 14; 1. General features

The invention concerns means of taking notes on paper during a recording

(or during playback of a recording), and using them as indices into the recording for controlling playback. Two means are described, one in which the position of the annotation is used to identify it, and one in which the shape of the annotation identifies it.

An alternative form involves a facility to take notes using an electronic device (keyboard, palmtop, ...), but to print them and control playback using the printed copy, or to take notes on paper but control playback through an electronic device.

2. Paper Annotation and Retrieval

Elements of embodiments of the invention are: Annotation

During a recording session, or while a recording is being played: the user takes notes on paper; for each note, the system stores some identifying feature of the note with the time-stamp of the recording (as of the moment the note was taken).

Selection and retrieval In order to control playback of a recording: the user selects a note from the paper; the system determines the identifying feature of the note, looks it up among those it has stored, and retrieves the associated time-stamp (which is subsequently used to play the recording starting at the position identified by the time-stamp, in accordance with prior art).

An overview of the system is shown in Fig. 7. The components are: A sheet of paper, on which a person takes notes during a recording session or playback of a recording in order to indexing the recording, or selects a note in order to control playback of an indexed recording;

2. A camera, which delivers an image of the paper to 7

3. A processing system, which produces from that image

4. an Annotation ID;

5. A recording or playback device, which produces the current Timestamp during a recording session or playback of a recording when indexing the recording, or consumes a timestamp and plays the recording beginning at that timestamp, when playback is being controlled;

6. a Timestamp,

7. a table, in which pairs consisting of an Annotation ID and a Timestamp are stored during indexing or looked up during control of playback. For simplicity of exposition, assume that there is a button available to the user. She clicks it to signal that she's starting an annotation. The time-stamp of the recording as of when she clicks is selected to be associated with the annotation. Presumably she clicks in reaction to an interesting event in the recording session. There will be a lag due to her reaction time, so when she views the material associated with the annotation, she probably wants to start playback of the video at a point prior to the selected time-stamp. The system should provide a parameter to let the user offset actual playback from the time-stamp point. When she's done, she signals the fact by clicking again.

There is a natural place for this button in some variants and not in others; in addition, in some variants the beginning of an annotation may be automatically detectable. These are not the most essential elements of the invention, so we don't discuss them further.

2.1 Identifying Annotations by Position In this embodiment, the identifying feature of an annotation is its position on the page. The position of the marks the user makes on the page are used to define a region that bounds the entire annotation. This can be done in many ways, and the exact method used is not important. A representation of the region identifies the annotation and is stored with the time-stamp. You select a note on the page by pointing at some part of it. The pointed-at position is checked against the defined regions, and the time-stamp of the one it lies within is retrieved. 8

Coded substrates provide one mechanism for recovering positional information, and an externally mounted camera provides another.

2. LI Coded substrates Figure 1 illustrates the components of a pointed document as printed on a coded substrate. The printed document 102 comprises a layer 104 of printed visible (human-readable) information printed on a coded substrate 106. The coded substrate 106 in turn comprises a layer 108 of visible or invisible machine readable markings printed on a sheet medium 110 (e.g. paper). Figure 2 shows a sample of zones, and the disposition of machine readable data, on a coded substrate. Each zone or cell 202 includes a border 204 and an orientation marker 206. A first set of markings 208 over part of the interior of the cell 202 are encoded representations of the page-id, while a second set of markings 210 over a (smaller) part of the interior of the cell 202 are encoded representations of the localisation (page-loc) - uniquely defining the position of the cell 202 within the page.

Figures 3 and 4 show how digital data is encoded in the zones illustrated in

Fig. 2. Fig. 3 shows the binary data, i.e. 47 bits of page-id in the upper section 302

(the bit stream wraps at the cell border 204), and 16 its of page localisation data (loc) in the lower section 304). The page-id code denotes 108429159095492 = 629DA182DCC4 (hexadecimal)

11000101001110110100001 1000001011011100110001000001000000010101 (binary; to make the wrapping explicit). In the 16 bit loc code in section 304, there are 8 bits for the X co-ordinate and 8 bits for the Y co-ordinate. Thus, for the cell (zone) shown, its position is 16,21 on the substrate. Fig. 4 shows the same data as in Fig. 3, but represented by Data Glyph markings. Encoding using data glyphs and the retrieval of data therefrom is discussed further in US-A-5,486,686, EP-A-469864, and the abovementioned GB application (ref.R/98003/JDR). Here, there is a first set of glyphs (markings) in upper section 402 and a second set in lower section 404, the two sets of glyphs being encoded representations of page-id and loc codes.

Figure 5 schematically illustrates an embodiment of a pointer which may be used in implementing the invention. The pointer 502 comprises a marking device 504 (which may be a pen or any other marking device suitable for making marks which 9 are visible to a user), and an image capture device 506. In use, whether or not the user is making marks using the marking device 504, the image capture device 506 is able to capture images of an area A of a document 508. (For the sake of illustration, the sizes of these elements are exaggerated - e.g. in practice, the area A may be much closer to the tip 505 of the marking device 504 than appears). In certain embodiments, the marking device 504 may be omitted.

The document 508 may be a 'blank' coded substrate, or such a substrate having human-readable information printed thereon.

Figure 6 shows a configuration for passing page identification codes and/or page location codes from the pointer of Fig. 5 to a network computer, in accordance with an embodiment of the invention. The image capture device (e.g. CCD camera) 506 is coupled by wired or wireless (e.g. IR or RF) link to processing device 602 and in use provides image data defining capture images to the processing device 602. The operative elements of the processing device 602 are a grame grabber circuit 604, image decoding software 606, and a CPU 608, which are known in the art. (In certain embodiments, the camera 506 and processing device 602 may be combined into an integral handheld unit). In use, the processing device 602 extracts from the image data the corresponding page-id and page-location data (<pid, loc>) and communicates them in a wired or wireless fashion to a local device (here, a network computer 610, which is linked to the network (intranet, internet) in a known manner). The computer

610 has its own unique network address, but need not have any information output device (e.g. display screen, printer).

The user writes on a coded substrate with the pointer (a hand-held pen/camera combination). As she writes, the positions of the pointer while it is in contact with the page are recovered. (This is one of the functionalities of the coded substrates; it is enabled by a substrate of glyphs on the page.)

The user selects an annotation by clicking the pointer on some part of the annotation; the position of the pointer is recovered (again, as a natural product of the use of Intelligent Paper).

The use of coded substrates also — for free — automatically launches the annotation/playback application when you click on the page, and automatically 10 locates the set of note identifiers and time-stamps for the page in use, wherever they're stored.

Figure 8 shows the operations performed by the processor of Fig. 7.

Here, a series of images are recovered by the camera. Processing these in the standard manner for coded substrates produces a series of positions. A bounding region (for example, a bounding rectangle) is computed and serves as the annotation identifier.

2.1.2 Under the Camera With coded substrates, the camera lies in a hand-held device. In the systems of EP-A-622J22 and EP-A-840J99, the camera is in a fixed position over the desk. This invention does not use the full set of concepts of those European patent applications, but employs the mounted camera concept from those works.

The user annotates a sheet of paper with pen or pencil. The page is within the field of view of the camera. It has registration marks on it which allow distinct images of it to be registered properly with each other in the case that the page is moved. It may also have identification marks to distinguish it from other sheets of paper.

For simplicity's sake, assume that the entire page is unobscured immediately before and after the annotation is made. The last image of the page taken before the annotation is made is subtracted from the first image taken after, isolating the marks comprising the annotation. The positions of the marks relative to the page frame are recovered.

The user selects an annotation by pointing at some part of the annotation. Pointing is recognized by the image processing system. (Selection techniques are discussed in EP-A-622,722 and EP-A-840,199).

Figure 9 shows the operations performed by the processor of Fig. 7 in this embodiment.

Here, two images are recovered by the camera and calibrated using the identifying marks in the corners. Subtracting the first from the second produces a third image containing only the current annotation.. A region bounding the annotation is computed and serves as the annotation identifier. 11

2.2 Identifying Annotations on Special-Purpose Pages

In this embodiment, the identifying feature of the annotation is a number that is encoded in marks on the paper ahead of time. For annotation, the user scans one of the marks, then writes on the paper next to it. The scanned mark is decoded, and the number is stored with the time-stamp of the recording.

For selection and retrieval, the user selects a note by scanning the mark. It is decoded into a number, and the time-stamp associated with that number is retrieved.

Glyphs are a natural choice for encoding the numbers.The additional functionalities of coded sybstrates can be provided by using such substrates without localisation codes pages as the special-purpose pages.

Figure 10 shows the operations performed by the processor of Fig. 7 in this embodiment.

Here, an image is recovered by the camera. Processing the image and decoding the glyphs in the standard manner (using one of Xerox's glyph toolkits, either SmartPaper, DAE, or DataGlyphs) produces a number, which serves as the annotation identifier.

2.3 Identifying Annotations by Shape Processing

In this embodiment, the annotations are written on the page with pen or pencil and subsequently scanned. The shape of the annotation identifies the annotation. This is similar to a method used in PaperLink to implement paper hyperlinks. An annotation is selected to control playback by scanning it.

In slightly more detail: when something interesting happens in the recording, the user clicks to select the current point, and the system picks up the time-stamp. She writes a note. When she's done writing, she scans what she has written with a hand-held scanner. When she's using the system to control playback of the recording, she scans a note, the system retrieves the associated time-stamp and initiates playback of the recording at that point.

The annotation identifier (a set of features extracted from the scanned image) and time-stamp are stored for each note. When the user selects an annotation, it is compared to the stored images (or features are extracted and compared to the stored feature sets). The one that matches best is identified, and its time-stamp is retrieved. 12

Matching should be tractable and robust because there is a limited set of annotation identifiers to match against.

The user need not scan the entire note, either when taking notes or selecting them; of course, she does have to make sure that what she scans for selection is pretty much what she scanned at the point of making the annotation.

Figure 11 shows the operations performed by the processor of Fig. 7 in which embodiment.

Here, an image is recovered by the camera. Standard image processing of it produces a set of image features, which serves as the annotation identifier.

2.4 Multiple Pages

A given page of notes is associated with a particular recording. Under the assumption that multiple pages of notes are taken, referring to multiple recordings, storing this association with the note/time-stamp mapping, and identifying this aggregate structure via a page identifier becomes useful. Intelligent Paper incorporates page identification, and as mentioned, the mounted-camera embodiment provides for page identification. Pre-marked pages can also provide page identification by keeping the numbers distinct across distinct pages. For the shape-processing embodiment, a set of image features can serve to identify a page. The user can make an arbitrary mark on the page to identify it, then scan the mark. Collisions are possible but can be detected.

2.5 Other Input Devices

Audio and video recordings are in widespread use, but this technique can index any data stream whose values change over time. For example, if the user had a

GPS device whose values were accessible to the system, notes could be associated with locations.

3. Electronic Annotation and Paper Retrieval Here the user takes notes using an electronic device, and the notes are printed so that playback can be controlled from a paper copy of the notes. As in the previous case, the system stores for each note an identifying feature of the note along with the 13 time-stamp. The choice of feature and process of computing the feature depends on the nature of the input.

3J ASCII and OCR If the annotations are input via a keyboard we have them in the form of ascii text. They can be printed in any fashion. An annotation is selected from the printed page by scanning it with a hand-held scanner. The scanned text is recovered using OCR. The annotation containing the most closely matching text is identified, and its associated time-stamp is retrieved. OCR accuracy can be improved by storing font information at the point of printing the annotations.

Figure 12 shows the operations performed by the processor of Fig. 7 in this embodiment.

Here, an image is recovered by the camera. Optical character recognition is performed, producing a text. This text is matched against the set of stored annotation identifiers (which are the text of the annotation, in this embodiment) and the closest match serves as the annotation identifier.

3.2 Annotation ID Marks When the annotations are printed, an identifying mark can be printed with each note (in the margin, for instance). The mark is used as the key in time-stamp retrieval, or if the mark represents a number, the number is used as the key. The user selects the note by hand-scanning the mark, which is recognized using techniques appropriate to the kind of mark. Possible marks are glyphs, bar codes, numbers or symbols in OCR fonts, icons, or text.

Figure 13 shows the operations performed by the processor of Fig. 7 in this embodiment. This is similar to Fig. 10.

3.3 Compatibility In a system designed for taking notes electronically, one of the two above- described methods is appropriate for selecting them from paper to control playback. But a system designed for taking notes on paper could be extended to allow notes to be taken electronically and printed. In this case, selection from paper should use the 14 same mechanism as if the notes had been taken on paper; the annotation identifiers are defined by the system for taking notes on paper.

4. Paper Annotation and Electronic Retrieval The above-described systems for paper annotation can be augmented to allow electronic retrieval.

An overview of the system is shown in Fig. 14. The elements are the same as in Fig. 7, with one addition: the images are stored in the table with the Annotation EDs and Timestamps. This allows a composite image (a) identical to the paper that the notes were written on to be constructed and displayed for the user on a screen.

In the positional case, i.e. with localisation code (loc), when the annotations are made, the image of the page must be stored. The final stored image is displayed to the user. The mouse position (relative to the page image frame) is used just as the position on the paper would be for retrieval purposes. In the hand-scanning case, a display can be built from the various scanned notes. When the display is built, the time-stamps must be associated with regions of the display, so that the mouse can be used for retrieval.

Figure 15 shows the operations performed by the processor of Fig. 14 for positional-based retrieval. This Figure shows the elements as used in retrieval (using annotations to control playback of an indexed recording) in the positional cases (coded substrates and Over-the-desk Camera). The composite image (a) described above is displayed on a screen to the user. The user selects an annotation with the mouse by placing it on some part of the annotation. Because regions in image (a) corresponds to regions on the original paper sheet the notes were written on, the mouse coordinates identify one of the regions serving as annotation identifiers. The timestamp stored with the annotation identifiers is retrieved and used for playback.

Figure 16 shows the elements as used in retrieval in the hand-scanning case.

Timestamps are associated directly with regions of the display when the display is built, so the mouse position when the user selects an annotation identifies a timestamp directly. This timestamp is retrieved and used for playback.

Claims

15CLAIMS

1. A method for providing access to a stored item of data comprising:

using a manually positionable scanner to scan one or more regions of a sheet and to provide a scan signal; the sheet bearing machine-readable markings; the markings defining two or more zones of the sheet; the markings within each zone indicating a position of the zone within the sheet;

using the scan signal to obtain position data indicating one or more positions indicated by the machine readable markings;

encoding one or more positions indicated by the position data with time information to obtain encoded position-time data; and

associating time data indicating the time information with the stored item of data to provide access to the stored item of data using the encoded position-time data.