US20060002635A1 - Computing a higher resolution image from multiple lower resolution images using model-based, robust bayesian estimation - Google Patents
Computing a higher resolution image from multiple lower resolution images using model-based, robust bayesian estimation Download PDFInfo
- Publication number
- US20060002635A1 US20060002635A1 US10/882,723 US88272304A US2006002635A1 US 20060002635 A1 US20060002635 A1 US 20060002635A1 US 88272304 A US88272304 A US 88272304A US 2006002635 A1 US2006002635 A1 US 2006002635A1
- Authority
- US
- United States
- Prior art keywords
- image
- images
- function
- methodology
- robust
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
Definitions
- An embodiment of the invention is directed to signal processing techniques to obtain a higher resolution, HR, image (or sequence of images) from multiple observed lower resolution images. Other embodiments are also described.
- HR images have many applications, including medical imaging, satellite imaging, and computer vision.
- An HR image may be obtained by simply increasing the number and/or density of pixel sensor elements in the electronic image sensor chip that is used to capture the image. This, however, may increase the size of the chip so much that capacitance effects will hamper the rapid transfer of pixel signal values, thereby causing difficulty for obtaining high-speed captures and video. Another possibility is to reduce the physical size of each pixel sensor element; however, doing so may increase the noise level in the resulting pixel signal value. Additionally, increasing the number of pixel sensor elements increases the cost of the device, which in many situations is undesirable (e.g., cameras mounted on mobile devices whose primary function is not image acquisition, like personal digital assistants (PDA) and cellular phones), and in others is prohibitive (e.g., infrared sensors). Therefore, another approach to obtaining HR images (that need not modify the lower resolution sensor) is to perform digital signal processing upon multiple lower resolution (LR) images captured by the sensor, to enhance resolution (also referred to as super resolution, SR, image reconstruction).
- LR lower resolution
- SR super
- SR image reconstruction multiple observed LR images or frames of a scene have been obtained that in effect are different “looks” of the same scene. These may be obtained using the same camera, for example, while introducing small, so-called sub-pixel shifts in the camera location from frame to frame, or capturing a small amount of motion in the scene. Alternatively, the LR images may be captured using different cameras aimed at the same scene. A “result” HR image is then reconstructed by aligning and combining properly the LR images, so that additional information, e.g. an increase in resolution or de-aliasing, is obtained for the result HR image. The process may also include image restoration, where de-blurring and de-noising operations are performed as well, to yield an even higher quality result HR image.
- the reconstruction of the result HR image is a difficult problem because it belongs to the class of inverse, ill-posed mathematical problems.
- the needed signal processing may be interpreted as being the reverse of a so-called observation model, which is a mathematically deterministic way to describe the formation of LR images of a scene (based upon known camera parameters). Since the scene is approximated by an acceptable quality HR image of it, the observation model is usually defined as relating an HR discrete image of the scene (with a given resolution and pixel grid) to its corresponding LR images. This relationship (which may apply to the formation of both still images and video) may be given as the concatenation of a geometric transform, a blur operator, and a down-sampling operator, plus an additive noise term.
- Examples of the geometric transform include, global or local translation and rotation, while the blur operator attempts to duplicate camera non-idealities, such as out of focus, diffraction limits, aberration, slow motion blur, and image sensor integration on a spatial region (sometimes combined all together in a point spread function).
- the down-sampling operator down samples the HR image into aliased, lower resolution images.
- This observation model may be expressed by the mathematical relationship Y ⁇ W*f+n, (1) where Y is the set of observed LR images and W represents the linear transformation of HR pixels in an HR image f to the LR pixels in Y (including the effect of down-sampling, geometric transform and blur).
- SR image reconstruction estimates the HR image f that corresponds to a given set of LR images Y.
- a Bayesian estimation process (also referred to as stochastic or probabilistic SR image reconstruction) may be used to estimate f, to get the “result” HR image mentioned above.
- an “a posteriori” probability function (typically, a probability density function) is mathematically defined as p(f
- Y) is mathematically defined as p(f
- Bayes Law the optimization problem, which is finding a suitable HR image f, e.g.
- p(f) is called the “Prior” probability density function that gives the probabilities of a particular HR image prior to any observation.
- the Prior indicates what HR images are more probable to occur based on, for example, a statistical characterization of an ensemble of different HR images.
- the Prior probability may be a joint probability, defined over all of the pixels in an HR image, and should be based on statistical data from a large number of images. However, estimating and describing the Prior probability as a joint distribution over all pixels may not be computationally feasible.
- the Prior may be based on a probabilistic construct called Markov Random Fields (MRFs). Rather than take the position that all HR images are equally likely, the MRF is tailored to indicate for example that certain pixel patterns (e.g., piece-wise continuous; text images) are more likely than others.
- MRFs Markov Random Fields
- An image may be assumed to be globally smooth in a mathematical sense, so the MRF typically used to define the Prior has a normal (Gaussian) probability distribution.
- f) that is called the “Likelihood” function; it is a probability density function that defines the probabilities of observing LR images that would correspond to a particular HR image.
- the Likelihood may be determined based on the observation model described above by the mathematical relationship in Equation (1), where the noise term is typically assumed to have a Gaussian probability distribution.
- the estimation process becomes one of iteratively determining trial HR images and stopping when there is convergence, which may signify that a maximum of the a posteriori probability function has been reached.
- FIG. 1 is a graph of robust and normal probability densities.
- FIG. 2 is a graph of Likelihood and Prior probability functions for a trial HR image.
- FIG. 3 is a flow diagram of some of the operations in a super resolution image reconstruction process.
- FIG. 4 is a flow diagram of some of the operations in a super resolution image reconstruction method operating on color images.
- FIGS. 5 and 6 shows two images that illustrate the results of applying the super resolution method to webcam images.
- FIGS. 7-11 shows images that illustrate the results of applying the super resolution method to images from a scanning beam nano-imaging device.
- An embodiment of the invention is a method for image processing in which a Bayesian estimation image reconstruction methodology computes a result HR image of a scene given multiple observed LR images.
- the result HR image is based on a Likelihood probability function that implements an observation model for the formation of LR images in the presence of noise.
- the methodology models the noise by a probabilistic, non-Gaussian, robust function.
- Such robust functions are defined in the statistical estimation literature and are characterized by long tails in the probability density function, as shown in FIG. 1 .
- the robust distribution acknowledges the occurrence of a few points that are affected by an unusually high amount of noise, also referred to as outliers (which are at the tail ends of the density graphs shown in FIG. 1 ).
- FIG. 2 a graph of probability density for a trial HR image is shown in which the example Likelihood and Prior function have been drawn.
- the maximum a posteriori (MAP) is proportional to the Prior and the Likelihood as given by Equation (2) above.
- MAP maximum a posteriori
- R assumed noise distributions
- G normal or Gaussian
- the graph illustrates the effect of an outlier in a given LR image (not shown) that translates into a dip in the Likelihood (G) for certain areas of a trial HR image.
- FIG. 3 illustrates a flow diagram of some of the operations in a SR method.
- the method contains a main loop that is repeatedly performed as part of an iterative process to determine the result (or final) HR image 104 .
- This process may attempt to find an optimum value, here a minimum, for an error function E. More specifically, this error function may be defined as the negative logarithm of the posterior probability in Equation (2). This error function may be minimized using any standard minimization techniques.
- FIG. 3 shows the use of the conjugate gradient method which is an iterative method that provides an acceptable balance between complexity and speed of convergence.
- ⁇ E ⁇ T The criteria for convergence is ⁇ E ⁇ T, which tests whether the error or difference in the posterior probability of Equation (2), between two successive trial HR images, is less than a predefined threshold, T (block 106 ).
- T a predefined threshold
- An alternative test is to define ⁇ E as a difference between consecutive trial HR images.
- the conjugate gradient method computes the gradient of the error function which has two terms in this embodiment, one corresponding to the Likelihood and the other to the Prior.
- the computation of the Likelihood gradient involves the application of standard image processing operations including geometric warping, linear filtering, and subsampling/upsampling, for example, that model both the forward and the reverse of the LR image formation process.
- an initial, trial HR image is needed. This may be, for example, a combination of one or more of an input (observed) LR image sequence (block 110 ) that have been aligned (block 114 ) to yield an HR image with an initial alignment (block 116 ).
- the results of this initial alignment are then used to compute the Likelihood gradient (block 108 ).
- the SR method assumes that the input LR images are the result of resampling an HR image, and the goal is to find the HR image which, when resampled in the grid of the input LR images according to the imaging observation model, predicts well the input (observed) LR images.
- the other half of the main computation loop in FIG. 3 is concerned with the Prior gradient (block 120 ).
- Different types of probability functions may be used for the Prior, but in the case of a robust MRF, the Prior gradient is equivalent to one update of a corresponding robust anisotropic diffusion filter, as described in Michael J. Black, et al., “Robust Anisotropic Diffusion”, Institute of Electrical and Electronics Engineers, IEEE Transactions on Image Processing, Vol. 7, No. 3, March 1998.
- Other implementations of the Prior function and its corresponding gradient may alternatively be used.
- the gradients computed in blocks 108 and 120 indicate to the iterative process the direction in which to move so as to come closer to a peak or trough in the combination of the Likelihood and Prior functions (see FIG. 2 ).
- This movement along the plots of the Likelihood and Prior functions results in a change or update (block 124 ) to the next HR image, which generates the current, trial HR image 126 .
- This current trial HR image 126 is then inserted into Equation (2) and a ⁇ E, which is the difference between the current value of Equation (2) and a previous value of Equation (2) is compared to a threshold T (block 106 ). If the ⁇ E is still too high, then the gradient computation loop is repeated.
- An additional decision may be made as to whether or not a refinement of the LR image initial alignment (block 116 ) is needed, in block 128 .
- This alignment may be evaluated using any one of conventional techniques. Operation may then proceed with an alignment of the LR images to a new HR image (block 130 ) resulting in a refined alignment (block 134 ).
- the next gradient computation for the Likelihood may use an HR image that has this refined alignment 134 .
- the HR image update (block 124 ) may cause the next trial HR image 126 to be changed too much, due to an outlier in the input LR image sequence 110 , thereby causing the methodology to select a less optimal final HR image 104 .
- a methodology for using the robust functions to model the noise in the observation model may be as follows.
- the probability distribution of the noise should be learned given a set of training examples consisting of HR images and their corresponding LR images. This set can be difficult to obtain, and even if it is available, it might not contain the noise attributed to errors in the alignment. For this reason, in most cases it may be better to use a generic robust function from the statistics literature.
- the choice of the robust function to use might depend on the knowledge available about the current images. For example, the process may use one of two different robust functions depending on the available knowledge about the presence of outliers.
- the robust function used to model the additive noise may be the well known Huber function. Note that such outliers may be caused by alignment errors, inaccurate modeling of blur, random noise, moving objects, motion blur, as well as other sources. Thus, if a process is expected to have, for example, relatively accurate image alignment, the Huber function may be used to model the additive noise.
- the Huber function although not being extremely robust, has the advantage of being convex, thus essentially guaranteeing a unique optimum (maximum or minimum) in the Likelihood function.
- the robust function may be set to a Tukey function which is considered very robust, thereby essentially eliminating any effect of the outliers in the solution.
- a shape of the robust function may be estimated and altered according to the availability of training data.
- the shape of the robust function may be adjusted by a scale factor, where if there is sufficient training data in the form of one or more ground truth HR images and their corresponding LR images, the scale factor is estimated from samples obtained in computing an error between the observed LR images of the scene and their projections from the ground truth HR images.
- the scale factor may be estimated by taking a current, trial HR image 126 ( FIG. 3 ) as a ground truth HR image, and applying a robust estimator as the scale factor.
- This robust estimator may be, for example, the median of residuals with respect to the median value. Other types of robust estimators may alternatively be used here.
- the Prior function may be as follows. If there is specific or statistical information concerning the expected HR images, such as computer aided design (CAD) models for structures captured in the observed LR images, then procedures similar to those described in U.S. patent application Ser. No. 10/685,867 entitled “Model Based De-Noising of Images and Image Sequences”, assigned to the same Assignee as that of this patent application, may be used. Those procedures may be particularly beneficial in applications such as microscopic imaging of silicon structures using scanning methods (e.g., focused ion beam; scanning electron microscope). That is because the structures being imaged in that case have corresponding, underlying CAD models.
- scanning methods e.g., focused ion beam; scanning electron microscope
- these geometrical transforms may be estimated as follows.
- an initial estimate of the geometric transforms between the observed or input LR images is obtained.
- Different options may be used here, depending on the characteristics of the motion of the image acquisition device relative to the scene being imaged. For generic sequences, with small changes in perspective, a global affine transformation model is used. For images with large changes in perspective, the affine model may be no longer appropriate so that higher order models (e.g., projective) should be used. Finally, if there is relative motion between the objects in the scene or perspective changes together with discontinuities in depth, global models may generally not be appropriate, such that either a dense local motion model (optical flow) or a layered model should be used.
- the initial alignment 116 may be refined (block 134 ) using the current version of the trial HR image 126 .
- the latter is expected to provide more accurate results than the LR to LR image alignment 114 , because the LR images are affected by aliasing.
- This technique may be compared to a combined Bayesian estimation for both the HR image and the geometrical transform.
- SR methods may be assumed to operate with gray-level images.
- These SR methods may also be applied to color images, which are usually presented as three components for each pixel, corresponding to Red (R), Green (G) and Blue (B) colors bands.
- the method can be applied to each color band independently to obtain a final HR image in RGB.
- applying the method to the three RGB bands is very computationally demanding. For this reason an alternative method is described in the flow diagram shown in FIG. 4 , which is less computationally intensive, and produces results that are perceptually equivalent to applying the method to all three color bands.
- operation begins with converting the input LR color image sequence 404 from the RGB color space into a color space that is consistent with the human perception of color, in this case CIELab (Commite Internationale de l'Eclairage) (block 408 ).
- CIELab Commission Internationale de l'Eclairage
- the three components are luminance (L) and two opponent color components (a, b).
- the SR methodology described above is applied only to the L component sequence 412 , rather than the a, b components 416 , because the human visual system detects high spatial frequencies mostly on luminance, and not in the opponent color components.
- the reconstruction to obtain HR a, b images 422 may be simply taking the average of aligned LR images (block 417 ), where this operation helps reduce noise in the component images, and then interpolating to match the needed HR image resolution using standard interpolation methods, such as bilinear interpolation (block 418 ).
- This methodology is much faster than applying the SR method 414 to all three color channels, and it is expected to be perceptually the same, in most cases.
- a conversion back to RGB color components (block 430 ) is performed to obtain the result HR color image 432 in the conventional RGB space.
- FIG. 4 The methodology of FIG. 4 has been implemented and applied to a color image sequence acquired with a relatively inexpensive digital camera of the consumer product variety used in Web interactive applications (also known as a webcam).
- the LR color image sequence 404 was recorded while a person held the camera in his hand for about one second (resulting in a sequence of frames being captured).
- the natural shaking of the user's hand provided the necessary motion for obtaining different sampling grids in the LR images.
- the image is a linear interpolation (by a factor of ⁇ 3) of the three color channels (to match the higher resolution) from a single LR frame
- the image in FIG. 6 is the HR reconstruction obtained by the SR method for color images described above, where in this case a generic Huber function was used for the Likelihoods and Priors. It is evident that the resulting HR image contains much more detail than the interpolated image.
- PSF point spread function
- an image acquisition system also referred to as an image acquisition system.
- a PSF may be theoretically computed based on the specifications of the image acquisition system. For example, in a video charge coupled device (CCD) camera, the lens and the CCD sensor specification may be used to compute the PSF. However, that information is not always available, in which case the PSF is estimated by calibration.
- CCD video charge coupled device
- An existing method to estimate the PSF is to obtain an image that corresponds to a punctual source (e.g., a white point on a black background).
- the image may correspond to an equivalent punctual source, such as an expanded laser beam.
- the image thus projected in the image plane (focal plane) of the camera sensor corresponds to the PSF.
- This optical image is sampled by the sensor, to obtain a digital version. If the sampling frequency is higher than twice the highest frequency of the PSF, then the digital version may be considered a complete representation of the underlying, continuous PSF.
- the sampling frequency for the LR images
- the sampling frequency is clearly lower than the one needed to avoid aliasing. Therefore, a single, LR image of a punctual source is a noisy and potentially aliased version of the underlying PSF.
- a higher resolution, aliasing free version of the PSF is recovered using an LR image sequence of a moving punctual source, instead of a single image.
- This method may be essentially the same as the ones described above for obtaining an HR image from an LR image sequence, except that in this case the process has the knowledge that the result HR image is that of a punctual source, and also that the PSF is not known. Since there is a linear relation between a punctual source and a PSF, it is possible to interchange the roles of the scene being imaged and the PSF.
- the PSF it may be sufficient to apply the same SR method described above to an image sequence obtained using the punctual source, with the PSF as a point (or, more generally, the known images used as a test for calibrating the PSF).
- the recovered HR image should be a higher resolution version of the underlying PSF.
- This resulting, calibrated PSF may then be used in the observation model, for determining the Likelihood function in the SR methods described earlier.
- the SR methods described above may be used in a variety of different system applications, provided there is enough computational power to produce a solution to the estimation process in a reasonable time.
- the SR methods may be implemented using LR images captured by such devices, to provide enhanced digital images from limited image acquisition hardware capability. Specific examples include resolution improvement in images acquired with solid state digital cameras attached to cellular/mobile telephones, personal digital assistants, and other small electronic devices whose main purpose is not to acquire images. In such applications, a sequence of LR images are captured while the camera is being held by the user, where the natural motion of the user's hand will produce the motion needed to generate the needed LR images.
- the LR image sequence could instead be transmitted to either a dedicated server that provides computing services (such as a Web based service business model) for this particular application, or to a personal computer in which the HR image or image sequence may be reconstructed.
- a dedicated server that provides computing services (such as a Web based service business model) for this particular application, or to a personal computer in which the HR image or image sequence may be reconstructed.
- the SR methods will convert this relatively inexpensive, low resolution device into a high resolution camera.
- the increase in resolution may allow a webcam with a standard video graphics resolution of 640 ⁇ 480 to scan a letter sized document at a resolution of 200 dots per inch, suitable for printing and fax transmission at reasonable quality.
- This inexpensive and relatively common device may then be used as an occasional document scanner, by simply placing the document to be scanned on the user's desk and aiming the webcam at the document, taking a sequence of images while the user is holding the webcam above the document in her hand. No additional equipment is needed to hold the camera, because the natural shaking of the user's hand provides the motion needed for differences between the LR images so that the super resolution method will work to yield a high resolution image.
- resolution improvement may be achieved for conversion of standard video to high definition video.
- N frames may be collected from time t to time t+N (in frames), where these frames become the LR images used to generate the high resolution frame corresponding to time t+N.
- the resolution improvement may be limited to the part of a scene that is visible during the interval in which the low resolution frames are collected.
- This resulting HR frame will be a clear perceptual improvement with respect to a simple interpolation of the standard video to high definition video.
- This embodiment may be used to generate, for example, high definition television, HDTV, video from standard video sequences, or to generate HR images that are suitable for high definition printing from standard (lower resolution) video sequences.
- the SR methods may also be applied to obtain image enhancement, including de-noising, de-blurring, and resolution improvement, in images that have been acquired with scanning imaging devices (e.g., scanning electron microscope, focused ion beam, and laser voltage probe).
- scanning imaging devices e.g., scanning electron microscope, focused ion beam, and laser voltage probe.
- these scanning imaging devices allow the scanning pattern to be varied, thus producing different sampling grids with sub-pixel shifts needed for the SR method.
- Such devices may be part of tools used in microelectronic test and manufacturing, to image and/or repair semiconductor structures and lithography masks. In some cases, such tools need to be operated at a lower resolution than the maximum possible, to increase throughput or because the parameters of the tool are optimized for nano-machining rather than optimal imaging. With such images, specific Prior models may be available that can be adapted to render the SR methods more effective.
- FIGS. 7-9 and 10 - 11 show two examples, respectively of applying the SR method to reconstruct a high resolution scanning imaging device image.
- a high resolution focused ion beam image is to be reconstructed, from a simulated noisy low resolution milling sequence.
- FIG. 7 an original HR image acquired with a focused ion beam tool is shown.
- FIG. 7 an original HR image acquired with a focused ion beam tool is shown.
- FIG. 9 shows the SR reconstruction. Note the clear improvement in detail between the SR reconstruction ( FIG. 9 ) and the LR image ( FIG. 8 ). The improvement in detail is also apparent in the second example, corresponding to a real milling sequence with displaced millboxes. Compare one of the initial LR images ( FIG. 10 ), magnified ⁇ 8 using nearest neighbor interpolation, and the result HR image after applying SR reconstruction, magnified ⁇ 8 ( FIG. 11 ).
- the SR methods described above may be implemented using a programmed computer.
- a computer program product or software may include a machine or computer-readable medium having stored thereon instructions which may be used to program a computer (or other electronic devices) to perform a process according to an embodiment of the invention.
- operations might be performed by specific hardware components that contain microcode, hardwired logic, or by any combination of programmed computer components and custom hardware components.
- a machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), but is not limited to, floppy diskettes, optical disks, Compact Disc, Read-Only Memory (CD-ROMs), and magneto-optical disks, Read-Only Memory (ROMs), Random Access Memory (RAM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), magnetic or optical cards, flash memory, a transmission over the Internet, electrical, optical, acoustical or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.) or the like.
- a machine e.g., a computer
- ROMs Read-Only Memory
- RAM Random Access Memory
- EPROM Erasable Programmable Read-Only Memory
- EEPROM Electrically Erasable Programmable Read-Only Memory
- the noise n in the observation model of Equation (1) which is modeled as a non-Gaussian robust function, may alternatively be any noise distribution previously learned from pairs of HR images and LR image sequences. Accordingly, other embodiments are within the scope of the claims.
Abstract
Description
- An embodiment of the invention is directed to signal processing techniques to obtain a higher resolution, HR, image (or sequence of images) from multiple observed lower resolution images. Other embodiments are also described.
- In most electronic imaging applications, images with higher resolution are generally more desirable. These are images that have greater pixel density and hence show greater detail than lower resolution images of the same scene. HR images have many applications, including medical imaging, satellite imaging, and computer vision.
- An HR image may be obtained by simply increasing the number and/or density of pixel sensor elements in the electronic image sensor chip that is used to capture the image. This, however, may increase the size of the chip so much that capacitance effects will hamper the rapid transfer of pixel signal values, thereby causing difficulty for obtaining high-speed captures and video. Another possibility is to reduce the physical size of each pixel sensor element; however, doing so may increase the noise level in the resulting pixel signal value. Additionally, increasing the number of pixel sensor elements increases the cost of the device, which in many situations is undesirable (e.g., cameras mounted on mobile devices whose primary function is not image acquisition, like personal digital assistants (PDA) and cellular phones), and in others is prohibitive (e.g., infrared sensors). Therefore, another approach to obtaining HR images (that need not modify the lower resolution sensor) is to perform digital signal processing upon multiple lower resolution (LR) images captured by the sensor, to enhance resolution (also referred to as super resolution, SR, image reconstruction).
- With SR image reconstruction, multiple observed LR images or frames of a scene have been obtained that in effect are different “looks” of the same scene. These may be obtained using the same camera, for example, while introducing small, so-called sub-pixel shifts in the camera location from frame to frame, or capturing a small amount of motion in the scene. Alternatively, the LR images may be captured using different cameras aimed at the same scene. A “result” HR image is then reconstructed by aligning and combining properly the LR images, so that additional information, e.g. an increase in resolution or de-aliasing, is obtained for the result HR image. The process may also include image restoration, where de-blurring and de-noising operations are performed as well, to yield an even higher quality result HR image.
- The reconstruction of the result HR image, however, is a difficult problem because it belongs to the class of inverse, ill-posed mathematical problems. The needed signal processing may be interpreted as being the reverse of a so-called observation model, which is a mathematically deterministic way to describe the formation of LR images of a scene (based upon known camera parameters). Since the scene is approximated by an acceptable quality HR image of it, the observation model is usually defined as relating an HR discrete image of the scene (with a given resolution and pixel grid) to its corresponding LR images. This relationship (which may apply to the formation of both still images and video) may be given as the concatenation of a geometric transform, a blur operator, and a down-sampling operator, plus an additive noise term. Examples of the geometric transform include, global or local translation and rotation, while the blur operator attempts to duplicate camera non-idealities, such as out of focus, diffraction limits, aberration, slow motion blur, and image sensor integration on a spatial region (sometimes combined all together in a point spread function). The down-sampling operator down samples the HR image into aliased, lower resolution images. This observation model may be expressed by the mathematical relationship
Y═W*f+n, (1)
where Y is the set of observed LR images and W represents the linear transformation of HR pixels in an HR image f to the LR pixels in Y (including the effect of down-sampling, geometric transform and blur). The n represents additive noise having random characteristics, which may represent, for example, the variation (or error) between LR images that have been captured by the same camera without any changes in the scene and without any changes to camera or lighting settings. Based on the observation model in Equation (1), SR image reconstruction estimates the HR image f that corresponds to a given set of LR images Y. - A Bayesian estimation process (also referred to as stochastic or probabilistic SR image reconstruction) may be used to estimate f, to get the “result” HR image mentioned above. In that case, an “a posteriori” probability function (typically, a probability density function) is mathematically defined as p(f|Y), which is the probability of a particular HR image f given the set of observed LR images Y. Applying a mathematical manipulation, known as Bayes Law, the optimization problem, which is finding a suitable HR image f, e.g. one that has the highest probability given a set of LR images or that maximizes p(f|Y), may be re-written as
P(f|Y)=p(Y|f)*p(f), (2)
where p(f) is called the “Prior” probability density function that gives the probabilities of a particular HR image prior to any observation. The Prior indicates what HR images are more probable to occur based on, for example, a statistical characterization of an ensemble of different HR images. The Prior probability may be a joint probability, defined over all of the pixels in an HR image, and should be based on statistical data from a large number of images. However, estimating and describing the Prior probability as a joint distribution over all pixels may not be computationally feasible. Accordingly existing methods use approximate models, based on the fact that in many types of images, correlations among pixels decay relatively quickly with pixel distance. For example, the Prior may be based on a probabilistic construct called Markov Random Fields (MRFs). Rather than take the position that all HR images are equally likely, the MRF is tailored to indicate for example that certain pixel patterns (e.g., piece-wise continuous; text images) are more likely than others. An image may be assumed to be globally smooth in a mathematical sense, so the MRF typically used to define the Prior has a normal (Gaussian) probability distribution. - As to p(Y|f), that is called the “Likelihood” function; it is a probability density function that defines the probabilities of observing LR images that would correspond to a particular HR image. The Likelihood may be determined based on the observation model described above by the mathematical relationship in Equation (1), where the noise term is typically assumed to have a Gaussian probability distribution. The estimation process becomes one of iteratively determining trial HR images and stopping when there is convergence, which may signify that a maximum of the a posteriori probability function has been reached.
- The embodiments of the invention are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings in which like references indicate similar elements. It should be noted that references to “an” embodiment of the invention in this disclosure are not necessarily to the same embodiment, and they mean at least one.
-
FIG. 1 is a graph of robust and normal probability densities. -
FIG. 2 is a graph of Likelihood and Prior probability functions for a trial HR image. -
FIG. 3 is a flow diagram of some of the operations in a super resolution image reconstruction process. -
FIG. 4 is a flow diagram of some of the operations in a super resolution image reconstruction method operating on color images. -
FIGS. 5 and 6 shows two images that illustrate the results of applying the super resolution method to webcam images. -
FIGS. 7-11 shows images that illustrate the results of applying the super resolution method to images from a scanning beam nano-imaging device. - An embodiment of the invention is a method for image processing in which a Bayesian estimation image reconstruction methodology computes a result HR image of a scene given multiple observed LR images. The result HR image is based on a Likelihood probability function that implements an observation model for the formation of LR images in the presence of noise. The methodology models the noise by a probabilistic, non-Gaussian, robust function. Such robust functions are defined in the statistical estimation literature and are characterized by long tails in the probability density function, as shown in
FIG. 1 . In contrast to the normal or Gaussian distribution, the robust distribution acknowledges the occurrence of a few points that are affected by an unusually high amount of noise, also referred to as outliers (which are at the tail ends of the density graphs shown inFIG. 1 ). This change to the modeling of noise better models the formation of LR images from the HR image, so that the method produces a more accurate solution. Thus, although implementing the SR process is made easier when the noise is modeled by a Gaussian probability function, such an assumption does not adequately handle images that contain different levels of outliers, which are common in SR reconstruction, due especially to inaccuracies in the image alignment. - Referring now to
FIG. 2 , a graph of probability density for a trial HR image is shown in which the example Likelihood and Prior function have been drawn. The maximum a posteriori (MAP) is proportional to the Prior and the Likelihood as given by Equation (2) above. In this case Likelihoods for two different assumed noise distributions (R) and (G) are shown, corresponding respectively to a robust probability function to model the noise (R), and another using a normal or Gaussian (G). The graph illustrates the effect of an outlier in a given LR image (not shown) that translates into a dip in the Likelihood (G) for certain areas of a trial HR image. This strong dip in the Likelihood (G) is due to the outlier dominating the Likelihood function, indicating a relatively low probability for the set of observed LR images, given this particular trial HR image. However, in actuality, it may be that the trial HR image is a good one, and that the only reason why the Likelihood value is low is due to the outlier (in one or more of the observed LR images). This domination of the Likelihood function by an outlier is negated by the use of a robust function which downplays the role of outlier pixels in observed LR images. Accordingly, the computed robust Likelihood (R) for this particular set of observed LR images (given that trial HR image) is higher than if the noise was modeled by a Gaussian function. - The various embodiments of the invention described here may prove the robustness of the SR process such that it can be used in different types of real world applications to be described below.
FIG. 3 illustrates a flow diagram of some of the operations in a SR method. The method contains a main loop that is repeatedly performed as part of an iterative process to determine the result (or final)HR image 104. This process may attempt to find an optimum value, here a minimum, for an error function E. More specifically, this error function may be defined as the negative logarithm of the posterior probability in Equation (2). This error function may be minimized using any standard minimization techniques. For example,FIG. 3 shows the use of the conjugate gradient method which is an iterative method that provides an acceptable balance between complexity and speed of convergence. The criteria for convergence is ΔE<T, which tests whether the error or difference in the posterior probability of Equation (2), between two successive trial HR images, is less than a predefined threshold, T (block 106). An alternative test is to define ΔE as a difference between consecutive trial HR images. - The conjugate gradient method computes the gradient of the error function which has two terms in this embodiment, one corresponding to the Likelihood and the other to the Prior. The computation of the Likelihood gradient (block 108) involves the application of standard image processing operations including geometric warping, linear filtering, and subsampling/upsampling, for example, that model both the forward and the reverse of the LR image formation process. To compute the Likelihood gradient, an initial, trial HR image is needed. This may be, for example, a combination of one or more of an input (observed) LR image sequence (block 110) that have been aligned (block 114) to yield an HR image with an initial alignment (block 116). The results of this initial alignment are then used to compute the Likelihood gradient (block 108). Recall once again that the SR method assumes that the input LR images are the result of resampling an HR image, and the goal is to find the HR image which, when resampled in the grid of the input LR images according to the imaging observation model, predicts well the input (observed) LR images.
- The other half of the main computation loop in
FIG. 3 is concerned with the Prior gradient (block 120). Different types of probability functions may be used for the Prior, but in the case of a robust MRF, the Prior gradient is equivalent to one update of a corresponding robust anisotropic diffusion filter, as described in Michael J. Black, et al., “Robust Anisotropic Diffusion”, Institute of Electrical and Electronics Engineers, IEEE Transactions on Image Processing, Vol. 7, No. 3, March 1998. Other implementations of the Prior function and its corresponding gradient may alternatively be used. - The gradients computed in
blocks FIG. 2 ). This movement along the plots of the Likelihood and Prior functions results in a change or update (block 124) to the next HR image, which generates the current,trial HR image 126. This currenttrial HR image 126 is then inserted into Equation (2) and a ΔE, which is the difference between the current value of Equation (2) and a previous value of Equation (2) is compared to a threshold T (block 106). If the ΔE is still too high, then the gradient computation loop is repeated. An additional decision may be made as to whether or not a refinement of the LR image initial alignment (block 116) is needed, inblock 128. This alignment may be evaluated using any one of conventional techniques. Operation may then proceed with an alignment of the LR images to a new HR image (block 130) resulting in a refined alignment (block 134). The next gradient computation for the Likelihood may use an HR image that has this refinedalignment 134. - Note that if a normal or Gaussian function is assigned to model the additive noise for computing the Likelihood (and its gradient), then the HR image update (block 124) may cause the next
trial HR image 126 to be changed too much, due to an outlier in the inputLR image sequence 110, thereby causing the methodology to select a less optimalfinal HR image 104. - A methodology for using the robust functions to model the noise in the observation model, which functions are able to “down weight” or in some cases essentially ignore outliers in the SR process, may be as follows. Ideally, the probability distribution of the noise should be learned given a set of training examples consisting of HR images and their corresponding LR images. This set can be difficult to obtain, and even if it is available, it might not contain the noise attributed to errors in the alignment. For this reason, in most cases it may be better to use a generic robust function from the statistics literature. The choice of the robust function to use might depend on the knowledge available about the current images. For example, the process may use one of two different robust functions depending on the available knowledge about the presence of outliers. If it is expected that the observed LR images will have relatively few outliers, then the robust function used to model the additive noise may be the well known Huber function. Note that such outliers may be caused by alignment errors, inaccurate modeling of blur, random noise, moving objects, motion blur, as well as other sources. Thus, if a process is expected to have, for example, relatively accurate image alignment, the Huber function may be used to model the additive noise. The Huber function, although not being extremely robust, has the advantage of being convex, thus essentially guaranteeing a unique optimum (maximum or minimum) in the Likelihood function.
- On the other hand, if it is expected that the observed LR images will have relatively many outliers (e.g., salt and pepper noise, and/or regions in the aligned image that have inaccurate alignment), the robust function may be set to a Tukey function which is considered very robust, thereby essentially eliminating any effect of the outliers in the solution.
- In addition to the option of setting the robust function to be a different one depending on whether relatively few or many outliers are expected, a shape of the robust function may be estimated and altered according to the availability of training data. For example, the shape of the robust function may be adjusted by a scale factor, where if there is sufficient training data in the form of one or more ground truth HR images and their corresponding LR images, the scale factor is estimated from samples obtained in computing an error between the observed LR images of the scene and their projections from the ground truth HR images.
- On the other hand, if there is no such training data, the scale factor may be estimated by taking a current, trial HR image 126 (
FIG. 3 ) as a ground truth HR image, and applying a robust estimator as the scale factor. This robust estimator may be, for example, the median of residuals with respect to the median value. Other types of robust estimators may alternatively be used here. - According to another embodiment of the invention, the Prior function may be as follows. If there is specific or statistical information concerning the expected HR images, such as computer aided design (CAD) models for structures captured in the observed LR images, then procedures similar to those described in U.S. patent application Ser. No. 10/685,867 entitled “Model Based De-Noising of Images and Image Sequences”, assigned to the same Assignee as that of this patent application, may be used. Those procedures may be particularly beneficial in applications such as microscopic imaging of silicon structures using scanning methods (e.g., focused ion beam; scanning electron microscope). That is because the structures being imaged in that case have corresponding, underlying CAD models.
- On the other hand, if no such model-based knowledge of the expected HR images exists, then a generic Prior function in the form of, for example, a robust MRF may be used. The portion of the gradient that corresponds to such a Prior is equivalent to one update of an anisotropic diffusion methodology. For this reason, any one of several different anisotropic diffusion methods that best adapts to the type of images that are to be expected may be used. For generic images, however, a good option for preserving edges in detail in the image is the Tukey function on a 4-neighbor MRF, as described by Black, et al., the article identified above. Other options include neighbor schemes (e.g., 8-neighbor) with cost functions that are adapted to the type of filter being used, that can be generic or learned from a training set of images. See also H. Scharr, et al. “Image Statistics and Anisotropic Diffusion”, IEEE Conference on Computer Vision and Pattern Recognition, Pages 840-847, Oct. 13-16, 2003. Use of either of the above options in the SR methods described here is expected to provide improved performance relative to the use of a Gaussian MRF as the generic Prior.
- Image Alignment
- In the previous discussion, it may be assumed that the geometric transformations that align the sampling grids of the observed or input
LR image sequence 110 with the sampling grid of theHR image 126 were known. However, in most cases, this information is not known a priori, unless the LR image sequence has been obtained under explicit controlled motion of the image acquisition device relative to the objects in the scene. Therefore, an estimate of these geometrical transforms are often needed. According to another embodiment of the invention, these geometrical transforms may be estimated as follows. - First, an initial estimate of the geometric transforms between the observed or input LR images is obtained. Different options may be used here, depending on the characteristics of the motion of the image acquisition device relative to the scene being imaged. For generic sequences, with small changes in perspective, a global affine transformation model is used. For images with large changes in perspective, the affine model may be no longer appropriate so that higher order models (e.g., projective) should be used. Finally, if there is relative motion between the objects in the scene or perspective changes together with discontinuities in depth, global models may generally not be appropriate, such that either a dense local motion model (optical flow) or a layered model should be used.
- Once a reasonable estimate of the HR image has been obtained (for example after 4-6 iterations), the initial alignment 116 (
FIG. 3 ) may be refined (block 134) using the current version of thetrial HR image 126. The latter is expected to provide more accurate results than the LR toLR image alignment 114, because the LR images are affected by aliasing. This technique may be compared to a combined Bayesian estimation for both the HR image and the geometrical transform. - Regardless of the motion model used for the alignment, as well as the type of alignment (that is LR to LR, or HR to HR), state of the art gradient based, multi-resolution, robust image motion estimation methods should be used to determine the alignment that will be input into the Likelihood gradient computation block 108 (
FIG. 3 ). - Color Images
- The embodiments of the invention described above may be assumed to operate with gray-level images. These SR methods, however, may also be applied to color images, which are usually presented as three components for each pixel, corresponding to Red (R), Green (G) and Blue (B) colors bands. The method can be applied to each color band independently to obtain a final HR image in RGB. However, applying the method to the three RGB bands is very computationally demanding. For this reason an alternative method is described in the flow diagram shown in
FIG. 4 , which is less computationally intensive, and produces results that are perceptually equivalent to applying the method to all three color bands. In this embodiment, operation begins with converting the input LRcolor image sequence 404 from the RGB color space into a color space that is consistent with the human perception of color, in this case CIELab (Commite Internationale de l'Eclairage) (block 408). In the CIELab color space, the three components are luminance (L) and two opponent color components (a, b). The SR methodology described above is applied only to theL component sequence 412, rather than the a,b components 416, because the human visual system detects high spatial frequencies mostly on luminance, and not in the opponent color components. Therefore, for the a, bopponent color components 416, the reconstruction to obtain HR a,b images 422 may be simply taking the average of aligned LR images (block 417), where this operation helps reduce noise in the component images, and then interpolating to match the needed HR image resolution using standard interpolation methods, such as bilinear interpolation (block 418). This methodology is much faster than applying theSR method 414 to all three color channels, and it is expected to be perceptually the same, in most cases. A conversion back to RGB color components (block 430) is performed to obtain the resultHR color image 432 in the conventional RGB space. - The methodology of
FIG. 4 has been implemented and applied to a color image sequence acquired with a relatively inexpensive digital camera of the consumer product variety used in Web interactive applications (also known as a webcam). In that case, the LRcolor image sequence 404 was recorded while a person held the camera in his hand for about one second (resulting in a sequence of frames being captured). The natural shaking of the user's hand provided the necessary motion for obtaining different sampling grids in the LR images. As can be seen inFIG. 5 , the image is a linear interpolation (by a factor of ×3) of the three color channels (to match the higher resolution) from a single LR frame, whereas the image inFIG. 6 is the HR reconstruction obtained by the SR method for color images described above, where in this case a generic Huber function was used for the Likelihoods and Priors. It is evident that the resulting HR image contains much more detail than the interpolated image. - Point Spread Function Calibration
- Recall that the point spread function (PSF) models the non-ideality of the camera (also referred to as an image acquisition system). Although a precise knowledge of the PSF of an image acquisition system may not be critical for SR methods to work, the quality of the result HR image may be further improved if such knowledge is incorporated into the SR method. A PSF may be theoretically computed based on the specifications of the image acquisition system. For example, in a video charge coupled device (CCD) camera, the lens and the CCD sensor specification may be used to compute the PSF. However, that information is not always available, in which case the PSF is estimated by calibration.
- An existing method to estimate the PSF is to obtain an image that corresponds to a punctual source (e.g., a white point on a black background). Alternatively, the image may correspond to an equivalent punctual source, such as an expanded laser beam. The image thus projected in the image plane (focal plane) of the camera sensor corresponds to the PSF. This optical image is sampled by the sensor, to obtain a digital version. If the sampling frequency is higher than twice the highest frequency of the PSF, then the digital version may be considered a complete representation of the underlying, continuous PSF. However, in the case of super resolution reconstruction, the sampling frequency (for the LR images) is clearly lower than the one needed to avoid aliasing. Therefore, a single, LR image of a punctual source is a noisy and potentially aliased version of the underlying PSF.
- According to an embodiment of the invention, a higher resolution, aliasing free version of the PSF is recovered using an LR image sequence of a moving punctual source, instead of a single image. This method may be essentially the same as the ones described above for obtaining an HR image from an LR image sequence, except that in this case the process has the knowledge that the result HR image is that of a punctual source, and also that the PSF is not known. Since there is a linear relation between a punctual source and a PSF, it is possible to interchange the roles of the scene being imaged and the PSF. Thus, to recover the PSF, it may be sufficient to apply the same SR method described above to an image sequence obtained using the punctual source, with the PSF as a point (or, more generally, the known images used as a test for calibrating the PSF). The recovered HR image should be a higher resolution version of the underlying PSF. This resulting, calibrated PSF may then be used in the observation model, for determining the Likelihood function in the SR methods described earlier.
- System Applications
- The SR methods described above may be used in a variety of different system applications, provided there is enough computational power to produce a solution to the estimation process in a reasonable time. As small and inexpensive digital image acquisition devices are becoming common place, such as consumer grade digital cameras and webcams, the SR methods may be implemented using LR images captured by such devices, to provide enhanced digital images from limited image acquisition hardware capability. Specific examples include resolution improvement in images acquired with solid state digital cameras attached to cellular/mobile telephones, personal digital assistants, and other small electronic devices whose main purpose is not to acquire images. In such applications, a sequence of LR images are captured while the camera is being held by the user, where the natural motion of the user's hand will produce the motion needed to generate the needed LR images. Such portable devices may, however, lack the computational power to execute the operations required by SR methods in a reasonable time. The LR image sequence could instead be transmitted to either a dedicated server that provides computing services (such as a Web based service business model) for this particular application, or to a personal computer in which the HR image or image sequence may be reconstructed.
- With respect to webcams, again their primary purpose may not be to take high resolution images. Accordingly, the SR methods will convert this relatively inexpensive, low resolution device into a high resolution camera. For example, the increase in resolution may allow a webcam with a standard video graphics resolution of 640×480 to scan a letter sized document at a resolution of 200 dots per inch, suitable for printing and fax transmission at reasonable quality. This inexpensive and relatively common device may then be used as an occasional document scanner, by simply placing the document to be scanned on the user's desk and aiming the webcam at the document, taking a sequence of images while the user is holding the webcam above the document in her hand. No additional equipment is needed to hold the camera, because the natural shaking of the user's hand provides the motion needed for differences between the LR images so that the super resolution method will work to yield a high resolution image.
- In yet another application, resolution improvement may be achieved for conversion of standard video to high definition video. In that case, N frames may be collected from time t to time t+N (in frames), where these frames become the LR images used to generate the high resolution frame corresponding to time t+N. In this case, the resolution improvement may be limited to the part of a scene that is visible during the interval in which the low resolution frames are collected. This resulting HR frame will be a clear perceptual improvement with respect to a simple interpolation of the standard video to high definition video. This embodiment may be used to generate, for example, high definition television, HDTV, video from standard video sequences, or to generate HR images that are suitable for high definition printing from standard (lower resolution) video sequences.
- The SR methods may also be applied to obtain image enhancement, including de-noising, de-blurring, and resolution improvement, in images that have been acquired with scanning imaging devices (e.g., scanning electron microscope, focused ion beam, and laser voltage probe). To obtain the different LR images needed for the SR method, these scanning imaging devices allow the scanning pattern to be varied, thus producing different sampling grids with sub-pixel shifts needed for the SR method. Such devices may be part of tools used in microelectronic test and manufacturing, to image and/or repair semiconductor structures and lithography masks. In some cases, such tools need to be operated at a lower resolution than the maximum possible, to increase throughput or because the parameters of the tool are optimized for nano-machining rather than optimal imaging. With such images, specific Prior models may be available that can be adapted to render the SR methods more effective.
- Also, as microelectronic manufacturing advances, the features of the structures being inspected are becoming smaller and smaller, such that lower quality images may be produced in the future when using current scanning imaging devices. By enhancing images from older generation scanning imaging devices, the life span of such tools will be extended in the future, without having to upgrade or replace the tools, thereby translating into significant savings in tooling costs.
FIGS. 7-9 and 10-11 show two examples, respectively of applying the SR method to reconstruct a high resolution scanning imaging device image. In the first example (FIGS. 7-9 ), a high resolution focused ion beam image is to be reconstructed, from a simulated noisy low resolution milling sequence. InFIG. 7 , an original HR image acquired with a focused ion beam tool is shown. InFIG. 8 , one LR image out of a sequence of 4× subsampled images after low pass filtering, with additive noise is shown.FIG. 9 shows the SR reconstruction. Note the clear improvement in detail between the SR reconstruction (FIG. 9 ) and the LR image (FIG. 8 ). The improvement in detail is also apparent in the second example, corresponding to a real milling sequence with displaced millboxes. Compare one of the initial LR images (FIG. 10 ), magnified ×8 using nearest neighbor interpolation, and the result HR image after applying SR reconstruction, magnified ×8 (FIG. 11 ). - The SR methods described above may be implemented using a programmed computer. A computer program product or software may include a machine or computer-readable medium having stored thereon instructions which may be used to program a computer (or other electronic devices) to perform a process according to an embodiment of the invention. In other embodiments, operations might be performed by specific hardware components that contain microcode, hardwired logic, or by any combination of programmed computer components and custom hardware components.
- A machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), but is not limited to, floppy diskettes, optical disks, Compact Disc, Read-Only Memory (CD-ROMs), and magneto-optical disks, Read-Only Memory (ROMs), Random Access Memory (RAM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), magnetic or optical cards, flash memory, a transmission over the Internet, electrical, optical, acoustical or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.) or the like.
- The invention is not limited to the specific embodiments described above. For example, the noise n in the observation model of Equation (1), which is modeled as a non-Gaussian robust function, may alternatively be any noise distribution previously learned from pairs of HR images and LR image sequences. Accordingly, other embodiments are within the scope of the claims.
Claims (21)
Priority Applications (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US10/882,723 US7447382B2 (en) | 2004-06-30 | 2004-06-30 | Computing a higher resolution image from multiple lower resolution images using model-based, robust Bayesian estimation |
PCT/US2005/021961 WO2006012126A1 (en) | 2004-06-30 | 2005-06-20 | Computing a higher resolution image from multiple lower resolution images using model-based, robust bayesian estimation |
TW094121590A TWI298466B (en) | 2004-06-30 | 2005-06-28 | Computing a higher resolution image from multiple lower resolution images using model-based, robust bayesian estimation |
CN200510098055.1A CN1734500B (en) | 2004-06-30 | 2005-06-30 | Image processing method and system by using robust bayesian estimation based on mode |
US11/321,580 US20060104540A1 (en) | 2004-06-30 | 2005-12-28 | Computing a higher resoultion image from multiple lower resolution images using model-based, robust bayesian estimation |
US11/479,999 US7809155B2 (en) | 2004-06-30 | 2006-06-29 | Computing a higher resolution image from multiple lower resolution images using model-base, robust Bayesian estimation |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US10/882,723 US7447382B2 (en) | 2004-06-30 | 2004-06-30 | Computing a higher resolution image from multiple lower resolution images using model-based, robust Bayesian estimation |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/321,580 Continuation-In-Part US20060104540A1 (en) | 2004-06-30 | 2005-12-28 | Computing a higher resoultion image from multiple lower resolution images using model-based, robust bayesian estimation |
Publications (2)
Publication Number | Publication Date |
---|---|
US20060002635A1 true US20060002635A1 (en) | 2006-01-05 |
US7447382B2 US7447382B2 (en) | 2008-11-04 |
Family
ID=34972559
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US10/882,723 Active 2026-08-26 US7447382B2 (en) | 2004-06-30 | 2004-06-30 | Computing a higher resolution image from multiple lower resolution images using model-based, robust Bayesian estimation |
US11/321,580 Abandoned US20060104540A1 (en) | 2004-06-30 | 2005-12-28 | Computing a higher resoultion image from multiple lower resolution images using model-based, robust bayesian estimation |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/321,580 Abandoned US20060104540A1 (en) | 2004-06-30 | 2005-12-28 | Computing a higher resoultion image from multiple lower resolution images using model-based, robust bayesian estimation |
Country Status (4)
Country | Link |
---|---|
US (2) | US7447382B2 (en) |
CN (1) | CN1734500B (en) |
TW (1) | TWI298466B (en) |
WO (1) | WO2006012126A1 (en) |
Cited By (74)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060165314A1 (en) * | 2005-01-21 | 2006-07-27 | Microsoft Corporation | System and process for increasing the apparent resolution of a display |
US20070083114A1 (en) * | 2005-08-26 | 2007-04-12 | The University Of Connecticut | Systems and methods for image resolution enhancement |
US20070263119A1 (en) * | 2006-05-15 | 2007-11-15 | Microsoft Corporation | Object matting using flash and no-flash images |
US20070296829A1 (en) * | 2004-11-15 | 2007-12-27 | Olympus Corporation | Imaging Device, Imaging System and Photography Method of Image |
US20080240525A1 (en) * | 2007-03-29 | 2008-10-02 | Martti Kalke | Method and system for reconstructing a medical image of an object |
US20090110285A1 (en) * | 2007-10-26 | 2009-04-30 | Technion Research And Development Foundation Ltd | Apparatus and method for improving image resolution using fuzzy motion estimation |
US20090324118A1 (en) * | 2008-06-30 | 2009-12-31 | Oleg Maslov | Computing higher resolution images from multiple lower resolution images |
US20100074549A1 (en) * | 2008-09-22 | 2010-03-25 | Microsoft Corporation | Image upsampling with training images |
US20100188424A1 (en) * | 2009-01-26 | 2010-07-29 | Hamamatsu Photonics K.K. | Image outputting system, image outputting method, and image outputting program |
CN101794440A (en) * | 2010-03-12 | 2010-08-04 | 东南大学 | Weighted adaptive super-resolution reconstructing method for image sequence |
US20100303385A1 (en) * | 2009-05-29 | 2010-12-02 | Putman Matthew C | Unique digital imaging method employing known background |
US20110069189A1 (en) * | 2008-05-20 | 2011-03-24 | Pelican Imaging Corporation | Capturing and processing of images using monolithic camera array with heterogeneous imagers |
US20110074824A1 (en) * | 2009-09-30 | 2011-03-31 | Microsoft Corporation | Dynamic image presentation |
US20110080487A1 (en) * | 2008-05-20 | 2011-04-07 | Pelican Imaging Corporation | Capturing and processing of images using monolithic camera array with heterogeneous imagers |
US20120147205A1 (en) * | 2010-12-14 | 2012-06-14 | Pelican Imaging Corporation | Systems and methods for synthesizing high resolution images using super-resolution processes |
US20130135496A1 (en) * | 2011-11-29 | 2013-05-30 | Sony Corporation | Image processing device, image processing method, and program |
US8831367B2 (en) | 2011-09-28 | 2014-09-09 | Pelican Imaging Corporation | Systems and methods for decoding light field image files |
US8861089B2 (en) | 2009-11-20 | 2014-10-14 | Pelican Imaging Corporation | Capturing and processing of images using monolithic camera array with heterogeneous imagers |
US8928793B2 (en) | 2010-05-12 | 2015-01-06 | Pelican Imaging Corporation | Imager array interfaces |
US9100635B2 (en) | 2012-06-28 | 2015-08-04 | Pelican Imaging Corporation | Systems and methods for detecting defective camera arrays and optic arrays |
US9100586B2 (en) | 2013-03-14 | 2015-08-04 | Pelican Imaging Corporation | Systems and methods for photometric normalization in array cameras |
US9106784B2 (en) | 2013-03-13 | 2015-08-11 | Pelican Imaging Corporation | Systems and methods for controlling aliasing in images captured by an array camera for use in super-resolution processing |
US9123117B2 (en) | 2012-08-21 | 2015-09-01 | Pelican Imaging Corporation | Systems and methods for generating depth maps and corresponding confidence maps indicating depth estimation reliability |
US9124864B2 (en) | 2013-03-10 | 2015-09-01 | Pelican Imaging Corporation | System and methods for calibration of an array camera |
US9128228B2 (en) | 2011-06-28 | 2015-09-08 | Pelican Imaging Corporation | Optical arrangements for use with an array camera |
US9143711B2 (en) | 2012-11-13 | 2015-09-22 | Pelican Imaging Corporation | Systems and methods for array camera focal plane control |
US9185276B2 (en) | 2013-11-07 | 2015-11-10 | Pelican Imaging Corporation | Methods of manufacturing array camera modules incorporating independently aligned lens stacks |
US9197821B2 (en) | 2011-05-11 | 2015-11-24 | Pelican Imaging Corporation | Systems and methods for transmitting and receiving array camera image data |
US9210392B2 (en) | 2012-05-01 | 2015-12-08 | Pelican Imaging Coporation | Camera modules patterned with pi filter groups |
US9214013B2 (en) | 2012-09-14 | 2015-12-15 | Pelican Imaging Corporation | Systems and methods for correcting user identified artifacts in light field images |
US9247117B2 (en) | 2014-04-07 | 2016-01-26 | Pelican Imaging Corporation | Systems and methods for correcting for warpage of a sensor array in an array camera module by introducing warpage into a focal plane of a lens stack array |
US9253380B2 (en) | 2013-02-24 | 2016-02-02 | Pelican Imaging Corporation | Thin form factor computational array cameras and modular array cameras |
US9412206B2 (en) | 2012-02-21 | 2016-08-09 | Pelican Imaging Corporation | Systems and methods for the manipulation of captured light field image data |
US9426361B2 (en) | 2013-11-26 | 2016-08-23 | Pelican Imaging Corporation | Array camera configurations incorporating multiple constituent array cameras |
US9438888B2 (en) | 2013-03-15 | 2016-09-06 | Pelican Imaging Corporation | Systems and methods for stereo imaging with camera arrays |
US9445003B1 (en) | 2013-03-15 | 2016-09-13 | Pelican Imaging Corporation | Systems and methods for synthesizing high resolution images using image deconvolution based on motion and depth information |
US9462164B2 (en) | 2013-02-21 | 2016-10-04 | Pelican Imaging Corporation | Systems and methods for generating compressed light field representation data using captured light fields, array geometry, and parallax information |
US9478010B2 (en) | 2013-12-12 | 2016-10-25 | Google Technology Holdings LLC | Generating an enhanced image of a predetermined scene from a plurality of images of the predetermined |
US9497370B2 (en) | 2013-03-15 | 2016-11-15 | Pelican Imaging Corporation | Array camera architecture implementing quantum dot color filters |
US9497429B2 (en) | 2013-03-15 | 2016-11-15 | Pelican Imaging Corporation | Extended color processing on pelican array cameras |
US9516222B2 (en) | 2011-06-28 | 2016-12-06 | Kip Peli P1 Lp | Array cameras incorporating monolithic array camera modules with high MTF lens stacks for capture of images used in super-resolution processing |
US9519972B2 (en) | 2013-03-13 | 2016-12-13 | Kip Peli P1 Lp | Systems and methods for synthesizing images from image data captured by an array camera using restricted depth of field depth maps in which depth estimation precision varies |
US9521319B2 (en) | 2014-06-18 | 2016-12-13 | Pelican Imaging Corporation | Array cameras and array camera modules including spectral filters disposed outside of a constituent image sensor |
US9521416B1 (en) | 2013-03-11 | 2016-12-13 | Kip Peli P1 Lp | Systems and methods for image data compression |
US9578259B2 (en) | 2013-03-14 | 2017-02-21 | Fotonation Cayman Limited | Systems and methods for reducing motion blur in images or video in ultra low light with array cameras |
US9633442B2 (en) | 2013-03-15 | 2017-04-25 | Fotonation Cayman Limited | Array cameras including an array camera module augmented with a separate camera |
US9638883B1 (en) | 2013-03-04 | 2017-05-02 | Fotonation Cayman Limited | Passive alignment of array camera modules constructed from lens stack arrays and sensors based upon alignment information obtained during manufacture of array camera modules using an active alignment process |
US9741118B2 (en) | 2013-03-13 | 2017-08-22 | Fotonation Cayman Limited | System and methods for calibration of an array camera |
RU2629432C2 (en) * | 2011-11-23 | 2017-08-29 | Конинклейке Филипс Н.В. | Removal of noise in image area |
US9766380B2 (en) | 2012-06-30 | 2017-09-19 | Fotonation Cayman Limited | Systems and methods for manufacturing camera modules using active alignment of lens stack arrays and sensors |
US9774789B2 (en) | 2013-03-08 | 2017-09-26 | Fotonation Cayman Limited | Systems and methods for high dynamic range imaging using array cameras |
US9794476B2 (en) | 2011-09-19 | 2017-10-17 | Fotonation Cayman Limited | Systems and methods for controlling aliasing in images captured by an array camera for use in super resolution processing using pixel apertures |
US9813616B2 (en) | 2012-08-23 | 2017-11-07 | Fotonation Cayman Limited | Feature based high resolution motion estimation from low resolution images captured using an array source |
US9888194B2 (en) | 2013-03-13 | 2018-02-06 | Fotonation Cayman Limited | Array camera architecture implementing quantum film image sensors |
US9898856B2 (en) | 2013-09-27 | 2018-02-20 | Fotonation Cayman Limited | Systems and methods for depth-assisted perspective distortion correction |
US9942474B2 (en) | 2015-04-17 | 2018-04-10 | Fotonation Cayman Limited | Systems and methods for performing high speed video capture and depth estimation using array cameras |
US10089740B2 (en) | 2014-03-07 | 2018-10-02 | Fotonation Limited | System and methods for depth regularization and semiautomatic interactive matting using RGB-D images |
US10119808B2 (en) | 2013-11-18 | 2018-11-06 | Fotonation Limited | Systems and methods for estimating depth from projected texture using camera arrays |
US10122993B2 (en) | 2013-03-15 | 2018-11-06 | Fotonation Limited | Autofocus system for a conventional camera that uses depth information from an array camera |
US10250871B2 (en) | 2014-09-29 | 2019-04-02 | Fotonation Limited | Systems and methods for dynamic calibration of array cameras |
US10390005B2 (en) | 2012-09-28 | 2019-08-20 | Fotonation Limited | Generating images from light fields utilizing virtual viewpoints |
US10438322B2 (en) | 2017-05-26 | 2019-10-08 | Microsoft Technology Licensing, Llc | Image resolution enhancement |
US10482618B2 (en) | 2017-08-21 | 2019-11-19 | Fotonation Limited | Systems and methods for hybrid depth regularization |
CN113099146A (en) * | 2019-12-19 | 2021-07-09 | 华为技术有限公司 | Video generation method and device and related equipment |
US11270110B2 (en) | 2019-09-17 | 2022-03-08 | Boston Polarimetrics, Inc. | Systems and methods for surface modeling using polarization cues |
US11290658B1 (en) | 2021-04-15 | 2022-03-29 | Boston Polarimetrics, Inc. | Systems and methods for camera exposure control |
US11302012B2 (en) | 2019-11-30 | 2022-04-12 | Boston Polarimetrics, Inc. | Systems and methods for transparent object segmentation using polarization cues |
US11525906B2 (en) | 2019-10-07 | 2022-12-13 | Intrinsic Innovation Llc | Systems and methods for augmentation of sensor systems and imaging systems with polarization |
US11580667B2 (en) | 2020-01-29 | 2023-02-14 | Intrinsic Innovation Llc | Systems and methods for characterizing object pose detection and measurement systems |
US11689813B2 (en) | 2021-07-01 | 2023-06-27 | Intrinsic Innovation Llc | Systems and methods for high dynamic range imaging using crossed polarizers |
US11792538B2 (en) | 2008-05-20 | 2023-10-17 | Adeia Imaging Llc | Capturing and processing of images including occlusions focused on an image sensor by a lens stack array |
US11797863B2 (en) | 2020-01-30 | 2023-10-24 | Intrinsic Innovation Llc | Systems and methods for synthesizing data for training statistical models on different imaging modalities including polarized images |
WO2023231138A1 (en) * | 2022-05-30 | 2023-12-07 | 元潼(北京)技术有限公司 | Multi-angle-of-view image super-resolution reconstruction method and apparatus based on meta-imaging |
US11953700B2 (en) | 2021-05-27 | 2024-04-09 | Intrinsic Innovation Llc | Multi-aperture polarization optical systems using beam splitters |
Families Citing this family (42)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7106914B2 (en) * | 2003-02-27 | 2006-09-12 | Microsoft Corporation | Bayesian image super resolution |
US7447382B2 (en) * | 2004-06-30 | 2008-11-04 | Intel Corporation | Computing a higher resolution image from multiple lower resolution images using model-based, robust Bayesian estimation |
US7809155B2 (en) * | 2004-06-30 | 2010-10-05 | Intel Corporation | Computing a higher resolution image from multiple lower resolution images using model-base, robust Bayesian estimation |
DE102005010119A1 (en) * | 2005-03-02 | 2006-11-23 | Siemens Ag | Image-enhancing operation of diagnostic x-ray apparatus, takes low resolution images at various distances and combines image data to form super-resolution image |
US7929774B2 (en) * | 2006-06-28 | 2011-04-19 | Intel Corporation | Method of inferential analysis of low resolution images |
US8031940B2 (en) * | 2006-06-29 | 2011-10-04 | Google Inc. | Recognizing text in images using ranging data |
US8098934B2 (en) | 2006-06-29 | 2012-01-17 | Google Inc. | Using extracted image text |
US7953295B2 (en) * | 2006-06-29 | 2011-05-31 | Google Inc. | Enhancing text in images |
US7881511B2 (en) * | 2007-01-19 | 2011-02-01 | Korea Advanced Institute Of Science And Technology | Method for super-resolution reconstruction using focal underdetermined system solver algorithm |
US8279341B1 (en) * | 2007-02-26 | 2012-10-02 | MotionDSP, Inc. | Enhancing the resolution and quality of sequential digital images |
US8494234B1 (en) | 2007-03-07 | 2013-07-23 | MotionDSP, Inc. | Video hashing system and method |
JP5023805B2 (en) * | 2007-05-16 | 2012-09-12 | ソニー株式会社 | Image processing apparatus, image processing method, and program |
CN101903885A (en) * | 2007-12-18 | 2010-12-01 | 皇家飞利浦电子股份有限公司 | Consistency metric based image registration |
US7941004B2 (en) * | 2008-04-30 | 2011-05-10 | Nec Laboratories America, Inc. | Super resolution using gaussian regression |
JP2010010810A (en) * | 2008-06-24 | 2010-01-14 | Canon Inc | Image forming apparatus, and image forming method |
JP4513906B2 (en) * | 2008-06-27 | 2010-07-28 | ソニー株式会社 | Image processing apparatus, image processing method, program, and recording medium |
JP2010074732A (en) * | 2008-09-22 | 2010-04-02 | Canon Inc | Image processor, image processing method, and program for performing the image processing method |
US8903191B2 (en) | 2008-12-30 | 2014-12-02 | Intel Corporation | Method and apparatus for noise reduction in video |
US7952355B2 (en) * | 2009-01-30 | 2011-05-31 | General Electric Company | Apparatus and method for reconstructing an MR image |
JP5448981B2 (en) | 2009-04-08 | 2014-03-19 | 株式会社半導体エネルギー研究所 | Driving method of liquid crystal display device |
US8897515B2 (en) * | 2009-09-08 | 2014-11-25 | Mayo Foundation For Medical Education And Research | Method for compressed sensing image reconstruction using a priori knowledge of spatial support |
JP5407737B2 (en) | 2009-10-16 | 2014-02-05 | 富士通セミコンダクター株式会社 | Model generation program, model generation apparatus, and model generation method |
US8330827B2 (en) * | 2009-11-19 | 2012-12-11 | Eastman Kodak Company | Increasing image resolution using combined differential image |
KR20110065997A (en) * | 2009-12-10 | 2011-06-16 | 삼성전자주식회사 | Image processing apparatus and method of processing image |
KR20130001213A (en) * | 2010-01-28 | 2013-01-03 | 이섬 리서치 디벨러프먼트 컴파니 오브 더 히브루 유니버시티 오브 예루살렘 엘티디. | Method and system for generating an output image of increased pixel resolution from an input image |
US8995719B2 (en) | 2012-12-10 | 2015-03-31 | Intel Corporation | Techniques for improved image disparity estimation |
US10586162B2 (en) | 2013-03-15 | 2020-03-10 | Ppg Industries Ohio, Inc. | Systems and methods for determining a coating formulation |
US10147043B2 (en) | 2013-03-15 | 2018-12-04 | Ppg Industries Ohio, Inc. | Systems and methods for texture assessment of a coating formulation |
NZ631047A (en) | 2013-11-08 | 2015-10-30 | Ppg Ind Ohio Inc | Texture analysis of a coated surface using kepler’s planetary motion laws |
NZ631063A (en) | 2013-11-08 | 2015-10-30 | Ppg Ind Ohio Inc | Texture analysis of a coated surface using cross-normalization |
NZ631068A (en) | 2013-11-08 | 2015-10-30 | Ppg Ind Ohio Inc | Texture analysis of a coated surface using electrostatics calculations |
CN105392009B (en) * | 2015-11-27 | 2019-04-16 | 四川大学 | Low bit rate image sequence coding method based on block adaptive sampling and super-resolution rebuilding |
US9818205B2 (en) | 2016-02-19 | 2017-11-14 | Ppg Industries Ohio, Inc. | Simplified texture comparison engine |
US10613727B2 (en) | 2016-02-19 | 2020-04-07 | Ppg Industries Ohio, Inc. | Color and texture match ratings for optimal match selection |
US10970879B2 (en) | 2018-04-26 | 2021-04-06 | Ppg Industries Ohio, Inc. | Formulation systems and methods employing target coating data results |
US11119035B2 (en) | 2018-04-26 | 2021-09-14 | Ppg Industries Ohio, Inc. | Systems and methods for rapid coating composition determinations |
US11874220B2 (en) | 2018-04-26 | 2024-01-16 | Ppg Industries Ohio, Inc. | Formulation systems and methods employing target coating data results |
US10871888B2 (en) | 2018-04-26 | 2020-12-22 | Ppg Industries Ohio, Inc. | Systems, methods, and interfaces for rapid coating generation |
US10169852B1 (en) * | 2018-07-03 | 2019-01-01 | Nanotronics Imaging, Inc. | Systems, devices, and methods for providing feedback on and improving the accuracy of super-resolution imaging |
WO2021140620A1 (en) * | 2020-01-09 | 2021-07-15 | 株式会社日立ハイテク | System for generating image, and non-transitory computer-readable medium |
US11669943B2 (en) | 2020-10-16 | 2023-06-06 | Microsoft Technology Licensing, Llc | Dual-stage system for computational photography, and technique for training same |
KR20220121533A (en) | 2021-02-25 | 2022-09-01 | 삼성전자주식회사 | Method and device for restoring image obtained from array camera |
Citations (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5781666A (en) * | 1990-04-17 | 1998-07-14 | Canon Kabushiki Kaisha | Image processing method and apparatus suitable for both high-resolution and low-resolution image data |
US5875268A (en) * | 1993-09-27 | 1999-02-23 | Canon Kabushiki Kaisha | Image processing with low-resolution to high-resolution conversion |
US6038257A (en) * | 1997-03-12 | 2000-03-14 | Telefonaktiebolaget L M Ericsson | Motion and still video picture transmission and display |
US6044375A (en) * | 1998-04-30 | 2000-03-28 | Hewlett-Packard Company | Automatic extraction of metadata using a neural network |
US6198467B1 (en) * | 1998-02-11 | 2001-03-06 | Unipac Octoelectronics Corp. | Method of displaying a high-resolution digital color image on a low-resolution dot-matrix display with high fidelity |
US6442555B1 (en) * | 1999-10-26 | 2002-08-27 | Hewlett-Packard Company | Automatic categorization of documents using document signatures |
US20040213443A1 (en) * | 2002-10-17 | 2004-10-28 | Horst Haussecker | Model-based fusion of scanning probe microscopic images for detection and identificaton of molecular structures |
US7006576B1 (en) * | 1999-07-19 | 2006-02-28 | Nokia Mobile Phones Limited | Video coding |
US20060104540A1 (en) * | 2004-06-30 | 2006-05-18 | Intel Corporation | Computing a higher resoultion image from multiple lower resolution images using model-based, robust bayesian estimation |
US7151801B2 (en) * | 2002-03-25 | 2006-12-19 | The Trustees Of Columbia University In The City Of New York | Method and system for enhancing data quality |
US20070019887A1 (en) * | 2004-06-30 | 2007-01-25 | Oscar Nestares | Computing a higher resolution image from multiple lower resolution images using model-base, robust bayesian estimation |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
AU1048399A (en) * | 1997-11-10 | 1999-05-31 | Gentech Corporation | System and method for generating super-resolution-enhanced mosaic images |
US7019778B1 (en) * | 1999-06-02 | 2006-03-28 | Eastman Kodak Company | Customizing a digital camera |
DE60009159T2 (en) | 1999-12-22 | 2004-11-25 | General Instrument Corporation | VIDEO COMPRESSION FOR MULTIPLE TRANSMITTER DISTRIBUTIONS USING SPATIAL SCALABILITY AND SAME-WAVE RADIO CODING |
GB0005337D0 (en) * | 2000-03-07 | 2000-04-26 | Hewlett Packard Co | Image transfer over mobile radio network |
TW490978B (en) | 2001-01-12 | 2002-06-11 | Chroma Ate Inc | Shooting method for high resolution display with low resolution camera |
US7239428B2 (en) * | 2001-06-11 | 2007-07-03 | Solectronics, Llc | Method of super image resolution |
GB0226294D0 (en) * | 2002-11-12 | 2002-12-18 | Autodesk Canada Inc | Image processing |
US7382937B2 (en) * | 2003-03-07 | 2008-06-03 | Hewlett-Packard Development Company, L.P. | Method and apparatus for re-constructing high-resolution images |
-
2004
- 2004-06-30 US US10/882,723 patent/US7447382B2/en active Active
-
2005
- 2005-06-20 WO PCT/US2005/021961 patent/WO2006012126A1/en active Application Filing
- 2005-06-28 TW TW094121590A patent/TWI298466B/en not_active IP Right Cessation
- 2005-06-30 CN CN200510098055.1A patent/CN1734500B/en not_active Expired - Fee Related
- 2005-12-28 US US11/321,580 patent/US20060104540A1/en not_active Abandoned
Patent Citations (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5781666A (en) * | 1990-04-17 | 1998-07-14 | Canon Kabushiki Kaisha | Image processing method and apparatus suitable for both high-resolution and low-resolution image data |
US5875268A (en) * | 1993-09-27 | 1999-02-23 | Canon Kabushiki Kaisha | Image processing with low-resolution to high-resolution conversion |
US6038257A (en) * | 1997-03-12 | 2000-03-14 | Telefonaktiebolaget L M Ericsson | Motion and still video picture transmission and display |
US6198467B1 (en) * | 1998-02-11 | 2001-03-06 | Unipac Octoelectronics Corp. | Method of displaying a high-resolution digital color image on a low-resolution dot-matrix display with high fidelity |
US6044375A (en) * | 1998-04-30 | 2000-03-28 | Hewlett-Packard Company | Automatic extraction of metadata using a neural network |
US7006576B1 (en) * | 1999-07-19 | 2006-02-28 | Nokia Mobile Phones Limited | Video coding |
US6442555B1 (en) * | 1999-10-26 | 2002-08-27 | Hewlett-Packard Company | Automatic categorization of documents using document signatures |
US7151801B2 (en) * | 2002-03-25 | 2006-12-19 | The Trustees Of Columbia University In The City Of New York | Method and system for enhancing data quality |
US20040213443A1 (en) * | 2002-10-17 | 2004-10-28 | Horst Haussecker | Model-based fusion of scanning probe microscopic images for detection and identificaton of molecular structures |
US20060104540A1 (en) * | 2004-06-30 | 2006-05-18 | Intel Corporation | Computing a higher resoultion image from multiple lower resolution images using model-based, robust bayesian estimation |
US20070019887A1 (en) * | 2004-06-30 | 2007-01-25 | Oscar Nestares | Computing a higher resolution image from multiple lower resolution images using model-base, robust bayesian estimation |
Cited By (209)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070296829A1 (en) * | 2004-11-15 | 2007-12-27 | Olympus Corporation | Imaging Device, Imaging System and Photography Method of Image |
US7548662B2 (en) * | 2005-01-21 | 2009-06-16 | Microsoft Corporation | System and process for increasing the apparent resolution of a display |
US20060165314A1 (en) * | 2005-01-21 | 2006-07-27 | Microsoft Corporation | System and process for increasing the apparent resolution of a display |
US20070083114A1 (en) * | 2005-08-26 | 2007-04-12 | The University Of Connecticut | Systems and methods for image resolution enhancement |
US7724952B2 (en) | 2006-05-15 | 2010-05-25 | Microsoft Corporation | Object matting using flash and no-flash images |
US20070263119A1 (en) * | 2006-05-15 | 2007-11-15 | Microsoft Corporation | Object matting using flash and no-flash images |
US20080240525A1 (en) * | 2007-03-29 | 2008-10-02 | Martti Kalke | Method and system for reconstructing a medical image of an object |
US8335358B2 (en) * | 2007-03-29 | 2012-12-18 | Palodex Group Oy | Method and system for reconstructing a medical image of an object |
US20090110285A1 (en) * | 2007-10-26 | 2009-04-30 | Technion Research And Development Foundation Ltd | Apparatus and method for improving image resolution using fuzzy motion estimation |
WO2009053978A2 (en) * | 2007-10-26 | 2009-04-30 | Technion Research And Development Foundation Ltd | Apparatus and method for improving image resolution using fuzzy motion estimation |
WO2009053978A3 (en) * | 2007-10-26 | 2010-03-11 | Technion Research And Development Foundation Ltd | Apparatus and method for improving image resolution using fuzzy motion estimation |
US9060120B2 (en) | 2008-05-20 | 2015-06-16 | Pelican Imaging Corporation | Systems and methods for generating depth maps using images captured by camera arrays |
US9060124B2 (en) | 2008-05-20 | 2015-06-16 | Pelican Imaging Corporation | Capturing and processing of images using non-monolithic camera arrays |
US9188765B2 (en) | 2008-05-20 | 2015-11-17 | Pelican Imaging Corporation | Capturing and processing of images including occlusions focused on an image sensor by a lens stack array |
US9191580B2 (en) | 2008-05-20 | 2015-11-17 | Pelican Imaging Corporation | Capturing and processing of images including occlusions captured by camera arrays |
US20110069189A1 (en) * | 2008-05-20 | 2011-03-24 | Pelican Imaging Corporation | Capturing and processing of images using monolithic camera array with heterogeneous imagers |
US9235898B2 (en) | 2008-05-20 | 2016-01-12 | Pelican Imaging Corporation | Systems and methods for generating depth maps using light focused on an image sensor by a lens element array |
US20110080487A1 (en) * | 2008-05-20 | 2011-04-07 | Pelican Imaging Corporation | Capturing and processing of images using monolithic camera array with heterogeneous imagers |
US9485496B2 (en) | 2008-05-20 | 2016-11-01 | Pelican Imaging Corporation | Systems and methods for measuring depth using images captured by a camera array including cameras surrounding a central camera |
US9049390B2 (en) | 2008-05-20 | 2015-06-02 | Pelican Imaging Corporation | Capturing and processing of images captured by arrays including polychromatic cameras |
US9124815B2 (en) | 2008-05-20 | 2015-09-01 | Pelican Imaging Corporation | Capturing and processing of images including occlusions captured by arrays of luma and chroma cameras |
US9049381B2 (en) | 2008-05-20 | 2015-06-02 | Pelican Imaging Corporation | Systems and methods for normalizing image data captured by camera arrays |
US9049391B2 (en) | 2008-05-20 | 2015-06-02 | Pelican Imaging Corporation | Capturing and processing of near-IR images including occlusions using camera arrays incorporating near-IR light sources |
US9049367B2 (en) | 2008-05-20 | 2015-06-02 | Pelican Imaging Corporation | Systems and methods for synthesizing higher resolution images using images captured by camera arrays |
US9576369B2 (en) | 2008-05-20 | 2017-02-21 | Fotonation Cayman Limited | Systems and methods for generating depth maps using images captured by camera arrays incorporating cameras having different fields of view |
US9712759B2 (en) | 2008-05-20 | 2017-07-18 | Fotonation Cayman Limited | Systems and methods for generating depth maps using a camera arrays incorporating monochrome and color cameras |
US11792538B2 (en) | 2008-05-20 | 2023-10-17 | Adeia Imaging Llc | Capturing and processing of images including occlusions focused on an image sensor by a lens stack array |
US9749547B2 (en) | 2008-05-20 | 2017-08-29 | Fotonation Cayman Limited | Capturing and processing of images using camera array incorperating Bayer cameras having different fields of view |
US8866920B2 (en) | 2008-05-20 | 2014-10-21 | Pelican Imaging Corporation | Capturing and processing of images using monolithic camera array with heterogeneous imagers |
US10027901B2 (en) | 2008-05-20 | 2018-07-17 | Fotonation Cayman Limited | Systems and methods for generating depth maps using a camera arrays incorporating monochrome and color cameras |
US8885059B1 (en) | 2008-05-20 | 2014-11-11 | Pelican Imaging Corporation | Systems and methods for measuring depth using images captured by camera arrays |
US8896719B1 (en) | 2008-05-20 | 2014-11-25 | Pelican Imaging Corporation | Systems and methods for parallax measurement using camera arrays incorporating 3 x 3 camera configurations |
US8902321B2 (en) | 2008-05-20 | 2014-12-02 | Pelican Imaging Corporation | Capturing and processing of images using monolithic camera array with heterogeneous imagers |
US9094661B2 (en) | 2008-05-20 | 2015-07-28 | Pelican Imaging Corporation | Systems and methods for generating depth maps using a set of images containing a baseline image |
US9077893B2 (en) | 2008-05-20 | 2015-07-07 | Pelican Imaging Corporation | Capturing and processing of images captured by non-grid camera arrays |
US11412158B2 (en) | 2008-05-20 | 2022-08-09 | Fotonation Limited | Capturing and processing of images including occlusions focused on an image sensor by a lens stack array |
US9049411B2 (en) | 2008-05-20 | 2015-06-02 | Pelican Imaging Corporation | Camera arrays incorporating 3×3 imager configurations |
US9060121B2 (en) | 2008-05-20 | 2015-06-16 | Pelican Imaging Corporation | Capturing and processing of images captured by camera arrays including cameras dedicated to sampling luma and cameras dedicated to sampling chroma |
US9060142B2 (en) | 2008-05-20 | 2015-06-16 | Pelican Imaging Corporation | Capturing and processing of images captured by camera arrays including heterogeneous optics |
US10142560B2 (en) | 2008-05-20 | 2018-11-27 | Fotonation Limited | Capturing and processing of images including occlusions focused on an image sensor by a lens stack array |
US9055213B2 (en) | 2008-05-20 | 2015-06-09 | Pelican Imaging Corporation | Systems and methods for measuring depth using images captured by monolithic camera arrays including at least one bayer camera |
US9055233B2 (en) | 2008-05-20 | 2015-06-09 | Pelican Imaging Corporation | Systems and methods for synthesizing higher resolution images using a set of images containing a baseline image |
US9041823B2 (en) | 2008-05-20 | 2015-05-26 | Pelican Imaging Corporation | Systems and methods for performing post capture refocus using images captured by camera arrays |
US9041829B2 (en) | 2008-05-20 | 2015-05-26 | Pelican Imaging Corporation | Capturing and processing of high dynamic range images using camera arrays |
US20090324118A1 (en) * | 2008-06-30 | 2009-12-31 | Oleg Maslov | Computing higher resolution images from multiple lower resolution images |
US8326069B2 (en) | 2008-06-30 | 2012-12-04 | Intel Corporation | Computing higher resolution images from multiple lower resolution images |
US8233734B2 (en) | 2008-09-22 | 2012-07-31 | Microsoft Corporation | Image upsampling with training images |
US20100074549A1 (en) * | 2008-09-22 | 2010-03-25 | Microsoft Corporation | Image upsampling with training images |
US20100188424A1 (en) * | 2009-01-26 | 2010-07-29 | Hamamatsu Photonics K.K. | Image outputting system, image outputting method, and image outputting program |
US7991245B2 (en) * | 2009-05-29 | 2011-08-02 | Putman Matthew C | Increasing image resolution method employing known background and specimen |
KR101689177B1 (en) * | 2009-05-29 | 2016-12-23 | 나노트로닉스 이미징, 엘엘씨. | Unique digital imaging method employing known background |
KR20120037379A (en) * | 2009-05-29 | 2012-04-19 | 나노트로닉스 이미징, 엘엘씨. | Unique digital imaging method employing known background |
US20100303385A1 (en) * | 2009-05-29 | 2010-12-02 | Putman Matthew C | Unique digital imaging method employing known background |
US20110074824A1 (en) * | 2009-09-30 | 2011-03-31 | Microsoft Corporation | Dynamic image presentation |
AU2010300971B2 (en) * | 2009-09-30 | 2014-06-05 | Microsoft Technology Licensing, Llc | Dynamic image presentation |
US9264610B2 (en) | 2009-11-20 | 2016-02-16 | Pelican Imaging Corporation | Capturing and processing of images including occlusions captured by heterogeneous camera arrays |
US10306120B2 (en) | 2009-11-20 | 2019-05-28 | Fotonation Limited | Capturing and processing of images captured by camera arrays incorporating cameras with telephoto and conventional lenses to generate depth maps |
US8861089B2 (en) | 2009-11-20 | 2014-10-14 | Pelican Imaging Corporation | Capturing and processing of images using monolithic camera array with heterogeneous imagers |
CN101794440A (en) * | 2010-03-12 | 2010-08-04 | 东南大学 | Weighted adaptive super-resolution reconstructing method for image sequence |
US9936148B2 (en) | 2010-05-12 | 2018-04-03 | Fotonation Cayman Limited | Imager array interfaces |
US8928793B2 (en) | 2010-05-12 | 2015-01-06 | Pelican Imaging Corporation | Imager array interfaces |
US10455168B2 (en) | 2010-05-12 | 2019-10-22 | Fotonation Limited | Imager array interfaces |
US8878950B2 (en) * | 2010-12-14 | 2014-11-04 | Pelican Imaging Corporation | Systems and methods for synthesizing high resolution images using super-resolution processes |
US11423513B2 (en) | 2010-12-14 | 2022-08-23 | Fotonation Limited | Systems and methods for synthesizing high resolution images using images captured by an array of independently controllable imagers |
US9047684B2 (en) | 2010-12-14 | 2015-06-02 | Pelican Imaging Corporation | Systems and methods for synthesizing high resolution images using a set of geometrically registered images |
US11875475B2 (en) | 2010-12-14 | 2024-01-16 | Adeia Imaging Llc | Systems and methods for synthesizing high resolution images using images captured by an array of independently controllable imagers |
US20120147205A1 (en) * | 2010-12-14 | 2012-06-14 | Pelican Imaging Corporation | Systems and methods for synthesizing high resolution images using super-resolution processes |
US10366472B2 (en) | 2010-12-14 | 2019-07-30 | Fotonation Limited | Systems and methods for synthesizing high resolution images using images captured by an array of independently controllable imagers |
US9041824B2 (en) | 2010-12-14 | 2015-05-26 | Pelican Imaging Corporation | Systems and methods for dynamic refocusing of high resolution images generated using images captured by a plurality of imagers |
US9361662B2 (en) | 2010-12-14 | 2016-06-07 | Pelican Imaging Corporation | Systems and methods for synthesizing high resolution images using images captured by an array of independently controllable imagers |
US10742861B2 (en) | 2011-05-11 | 2020-08-11 | Fotonation Limited | Systems and methods for transmitting and receiving array camera image data |
US9197821B2 (en) | 2011-05-11 | 2015-11-24 | Pelican Imaging Corporation | Systems and methods for transmitting and receiving array camera image data |
US9866739B2 (en) | 2011-05-11 | 2018-01-09 | Fotonation Cayman Limited | Systems and methods for transmitting and receiving array camera image data |
US10218889B2 (en) | 2011-05-11 | 2019-02-26 | Fotonation Limited | Systems and methods for transmitting and receiving array camera image data |
US9578237B2 (en) | 2011-06-28 | 2017-02-21 | Fotonation Cayman Limited | Array cameras incorporating optics with modulation transfer functions greater than sensor Nyquist frequency for capture of images used in super-resolution processing |
US9128228B2 (en) | 2011-06-28 | 2015-09-08 | Pelican Imaging Corporation | Optical arrangements for use with an array camera |
US9516222B2 (en) | 2011-06-28 | 2016-12-06 | Kip Peli P1 Lp | Array cameras incorporating monolithic array camera modules with high MTF lens stacks for capture of images used in super-resolution processing |
US9794476B2 (en) | 2011-09-19 | 2017-10-17 | Fotonation Cayman Limited | Systems and methods for controlling aliasing in images captured by an array camera for use in super resolution processing using pixel apertures |
US10375302B2 (en) | 2011-09-19 | 2019-08-06 | Fotonation Limited | Systems and methods for controlling aliasing in images captured by an array camera for use in super resolution processing using pixel apertures |
US9129183B2 (en) | 2011-09-28 | 2015-09-08 | Pelican Imaging Corporation | Systems and methods for encoding light field image files |
US9042667B2 (en) | 2011-09-28 | 2015-05-26 | Pelican Imaging Corporation | Systems and methods for decoding light field image files using a depth map |
US9864921B2 (en) | 2011-09-28 | 2018-01-09 | Fotonation Cayman Limited | Systems and methods for encoding image files containing depth maps stored as metadata |
US9036931B2 (en) | 2011-09-28 | 2015-05-19 | Pelican Imaging Corporation | Systems and methods for decoding structured light field image files |
US9031343B2 (en) | 2011-09-28 | 2015-05-12 | Pelican Imaging Corporation | Systems and methods for encoding light field image files having a depth map |
US10019816B2 (en) | 2011-09-28 | 2018-07-10 | Fotonation Cayman Limited | Systems and methods for decoding image files containing depth maps stored as metadata |
US11729365B2 (en) | 2011-09-28 | 2023-08-15 | Adela Imaging LLC | Systems and methods for encoding image files containing depth maps stored as metadata |
US9811753B2 (en) | 2011-09-28 | 2017-11-07 | Fotonation Cayman Limited | Systems and methods for encoding light field image files |
US9025895B2 (en) | 2011-09-28 | 2015-05-05 | Pelican Imaging Corporation | Systems and methods for decoding refocusable light field image files |
US9025894B2 (en) | 2011-09-28 | 2015-05-05 | Pelican Imaging Corporation | Systems and methods for decoding light field image files having depth and confidence maps |
US9536166B2 (en) | 2011-09-28 | 2017-01-03 | Kip Peli P1 Lp | Systems and methods for decoding image files containing depth maps stored as metadata |
US10984276B2 (en) | 2011-09-28 | 2021-04-20 | Fotonation Limited | Systems and methods for encoding image files containing depth maps stored as metadata |
US9031335B2 (en) | 2011-09-28 | 2015-05-12 | Pelican Imaging Corporation | Systems and methods for encoding light field image files having depth and confidence maps |
US20180197035A1 (en) | 2011-09-28 | 2018-07-12 | Fotonation Cayman Limited | Systems and Methods for Encoding Image Files Containing Depth Maps Stored as Metadata |
US8831367B2 (en) | 2011-09-28 | 2014-09-09 | Pelican Imaging Corporation | Systems and methods for decoding light field image files |
US10275676B2 (en) | 2011-09-28 | 2019-04-30 | Fotonation Limited | Systems and methods for encoding image files containing depth maps stored as metadata |
US9031342B2 (en) | 2011-09-28 | 2015-05-12 | Pelican Imaging Corporation | Systems and methods for encoding refocusable light field image files |
US10430682B2 (en) | 2011-09-28 | 2019-10-01 | Fotonation Limited | Systems and methods for decoding image files containing depth maps stored as metadata |
US9036928B2 (en) | 2011-09-28 | 2015-05-19 | Pelican Imaging Corporation | Systems and methods for encoding structured light field image files |
RU2629432C2 (en) * | 2011-11-23 | 2017-08-29 | Конинклейке Филипс Н.В. | Removal of noise in image area |
US20130135496A1 (en) * | 2011-11-29 | 2013-05-30 | Sony Corporation | Image processing device, image processing method, and program |
US10311649B2 (en) | 2012-02-21 | 2019-06-04 | Fotonation Limited | Systems and method for performing depth based image editing |
US9754422B2 (en) | 2012-02-21 | 2017-09-05 | Fotonation Cayman Limited | Systems and method for performing depth based image editing |
US9412206B2 (en) | 2012-02-21 | 2016-08-09 | Pelican Imaging Corporation | Systems and methods for the manipulation of captured light field image data |
US9706132B2 (en) | 2012-05-01 | 2017-07-11 | Fotonation Cayman Limited | Camera modules patterned with pi filter groups |
US9210392B2 (en) | 2012-05-01 | 2015-12-08 | Pelican Imaging Coporation | Camera modules patterned with pi filter groups |
US9100635B2 (en) | 2012-06-28 | 2015-08-04 | Pelican Imaging Corporation | Systems and methods for detecting defective camera arrays and optic arrays |
US9807382B2 (en) | 2012-06-28 | 2017-10-31 | Fotonation Cayman Limited | Systems and methods for detecting defective camera arrays and optic arrays |
US10334241B2 (en) | 2012-06-28 | 2019-06-25 | Fotonation Limited | Systems and methods for detecting defective camera arrays and optic arrays |
US11022725B2 (en) | 2012-06-30 | 2021-06-01 | Fotonation Limited | Systems and methods for manufacturing camera modules using active alignment of lens stack arrays and sensors |
US9766380B2 (en) | 2012-06-30 | 2017-09-19 | Fotonation Cayman Limited | Systems and methods for manufacturing camera modules using active alignment of lens stack arrays and sensors |
US10261219B2 (en) | 2012-06-30 | 2019-04-16 | Fotonation Limited | Systems and methods for manufacturing camera modules using active alignment of lens stack arrays and sensors |
US9123117B2 (en) | 2012-08-21 | 2015-09-01 | Pelican Imaging Corporation | Systems and methods for generating depth maps and corresponding confidence maps indicating depth estimation reliability |
US9123118B2 (en) | 2012-08-21 | 2015-09-01 | Pelican Imaging Corporation | System and methods for measuring depth using an array camera employing a bayer filter |
US9147254B2 (en) | 2012-08-21 | 2015-09-29 | Pelican Imaging Corporation | Systems and methods for measuring depth in the presence of occlusions using a subset of images |
US9129377B2 (en) | 2012-08-21 | 2015-09-08 | Pelican Imaging Corporation | Systems and methods for measuring depth based upon occlusion patterns in images |
US9858673B2 (en) | 2012-08-21 | 2018-01-02 | Fotonation Cayman Limited | Systems and methods for estimating depth and visibility from a reference viewpoint for pixels in a set of images captured from different viewpoints |
US10380752B2 (en) | 2012-08-21 | 2019-08-13 | Fotonation Limited | Systems and methods for estimating depth and visibility from a reference viewpoint for pixels in a set of images captured from different viewpoints |
US9235900B2 (en) | 2012-08-21 | 2016-01-12 | Pelican Imaging Corporation | Systems and methods for estimating depth and visibility from a reference viewpoint for pixels in a set of images captured from different viewpoints |
US9240049B2 (en) | 2012-08-21 | 2016-01-19 | Pelican Imaging Corporation | Systems and methods for measuring depth using an array of independently controllable cameras |
US10462362B2 (en) | 2012-08-23 | 2019-10-29 | Fotonation Limited | Feature based high resolution motion estimation from low resolution images captured using an array source |
US9813616B2 (en) | 2012-08-23 | 2017-11-07 | Fotonation Cayman Limited | Feature based high resolution motion estimation from low resolution images captured using an array source |
US9214013B2 (en) | 2012-09-14 | 2015-12-15 | Pelican Imaging Corporation | Systems and methods for correcting user identified artifacts in light field images |
US10390005B2 (en) | 2012-09-28 | 2019-08-20 | Fotonation Limited | Generating images from light fields utilizing virtual viewpoints |
US9749568B2 (en) | 2012-11-13 | 2017-08-29 | Fotonation Cayman Limited | Systems and methods for array camera focal plane control |
US9143711B2 (en) | 2012-11-13 | 2015-09-22 | Pelican Imaging Corporation | Systems and methods for array camera focal plane control |
US9462164B2 (en) | 2013-02-21 | 2016-10-04 | Pelican Imaging Corporation | Systems and methods for generating compressed light field representation data using captured light fields, array geometry, and parallax information |
US10009538B2 (en) | 2013-02-21 | 2018-06-26 | Fotonation Cayman Limited | Systems and methods for generating compressed light field representation data using captured light fields, array geometry, and parallax information |
US9374512B2 (en) | 2013-02-24 | 2016-06-21 | Pelican Imaging Corporation | Thin form factor computational array cameras and modular array cameras |
US9774831B2 (en) | 2013-02-24 | 2017-09-26 | Fotonation Cayman Limited | Thin form factor computational array cameras and modular array cameras |
US9253380B2 (en) | 2013-02-24 | 2016-02-02 | Pelican Imaging Corporation | Thin form factor computational array cameras and modular array cameras |
US9743051B2 (en) | 2013-02-24 | 2017-08-22 | Fotonation Cayman Limited | Thin form factor computational array cameras and modular array cameras |
US9638883B1 (en) | 2013-03-04 | 2017-05-02 | Fotonation Cayman Limited | Passive alignment of array camera modules constructed from lens stack arrays and sensors based upon alignment information obtained during manufacture of array camera modules using an active alignment process |
US9917998B2 (en) | 2013-03-08 | 2018-03-13 | Fotonation Cayman Limited | Systems and methods for measuring scene information while capturing images using array cameras |
US9774789B2 (en) | 2013-03-08 | 2017-09-26 | Fotonation Cayman Limited | Systems and methods for high dynamic range imaging using array cameras |
US10958892B2 (en) | 2013-03-10 | 2021-03-23 | Fotonation Limited | System and methods for calibration of an array camera |
US11272161B2 (en) | 2013-03-10 | 2022-03-08 | Fotonation Limited | System and methods for calibration of an array camera |
US11570423B2 (en) | 2013-03-10 | 2023-01-31 | Adeia Imaging Llc | System and methods for calibration of an array camera |
US9124864B2 (en) | 2013-03-10 | 2015-09-01 | Pelican Imaging Corporation | System and methods for calibration of an array camera |
US9986224B2 (en) | 2013-03-10 | 2018-05-29 | Fotonation Cayman Limited | System and methods for calibration of an array camera |
US10225543B2 (en) | 2013-03-10 | 2019-03-05 | Fotonation Limited | System and methods for calibration of an array camera |
US9521416B1 (en) | 2013-03-11 | 2016-12-13 | Kip Peli P1 Lp | Systems and methods for image data compression |
US9519972B2 (en) | 2013-03-13 | 2016-12-13 | Kip Peli P1 Lp | Systems and methods for synthesizing images from image data captured by an array camera using restricted depth of field depth maps in which depth estimation precision varies |
US9741118B2 (en) | 2013-03-13 | 2017-08-22 | Fotonation Cayman Limited | System and methods for calibration of an array camera |
US9800856B2 (en) | 2013-03-13 | 2017-10-24 | Fotonation Cayman Limited | Systems and methods for synthesizing images from image data captured by an array camera using restricted depth of field depth maps in which depth estimation precision varies |
US9733486B2 (en) | 2013-03-13 | 2017-08-15 | Fotonation Cayman Limited | Systems and methods for controlling aliasing in images captured by an array camera for use in super-resolution processing |
US9106784B2 (en) | 2013-03-13 | 2015-08-11 | Pelican Imaging Corporation | Systems and methods for controlling aliasing in images captured by an array camera for use in super-resolution processing |
US10127682B2 (en) | 2013-03-13 | 2018-11-13 | Fotonation Limited | System and methods for calibration of an array camera |
US9888194B2 (en) | 2013-03-13 | 2018-02-06 | Fotonation Cayman Limited | Array camera architecture implementing quantum film image sensors |
US10091405B2 (en) | 2013-03-14 | 2018-10-02 | Fotonation Cayman Limited | Systems and methods for reducing motion blur in images or video in ultra low light with array cameras |
US9578259B2 (en) | 2013-03-14 | 2017-02-21 | Fotonation Cayman Limited | Systems and methods for reducing motion blur in images or video in ultra low light with array cameras |
US9787911B2 (en) | 2013-03-14 | 2017-10-10 | Fotonation Cayman Limited | Systems and methods for photometric normalization in array cameras |
US10547772B2 (en) | 2013-03-14 | 2020-01-28 | Fotonation Limited | Systems and methods for reducing motion blur in images or video in ultra low light with array cameras |
US10412314B2 (en) | 2013-03-14 | 2019-09-10 | Fotonation Limited | Systems and methods for photometric normalization in array cameras |
US9100586B2 (en) | 2013-03-14 | 2015-08-04 | Pelican Imaging Corporation | Systems and methods for photometric normalization in array cameras |
US9633442B2 (en) | 2013-03-15 | 2017-04-25 | Fotonation Cayman Limited | Array cameras including an array camera module augmented with a separate camera |
US10182216B2 (en) | 2013-03-15 | 2019-01-15 | Fotonation Limited | Extended color processing on pelican array cameras |
US9445003B1 (en) | 2013-03-15 | 2016-09-13 | Pelican Imaging Corporation | Systems and methods for synthesizing high resolution images using image deconvolution based on motion and depth information |
US10122993B2 (en) | 2013-03-15 | 2018-11-06 | Fotonation Limited | Autofocus system for a conventional camera that uses depth information from an array camera |
US9602805B2 (en) | 2013-03-15 | 2017-03-21 | Fotonation Cayman Limited | Systems and methods for estimating depth using ad hoc stereo array cameras |
US9955070B2 (en) | 2013-03-15 | 2018-04-24 | Fotonation Cayman Limited | Systems and methods for synthesizing high resolution images using image deconvolution based on motion and depth information |
US10674138B2 (en) | 2013-03-15 | 2020-06-02 | Fotonation Limited | Autofocus system for a conventional camera that uses depth information from an array camera |
US10638099B2 (en) | 2013-03-15 | 2020-04-28 | Fotonation Limited | Extended color processing on pelican array cameras |
US9438888B2 (en) | 2013-03-15 | 2016-09-06 | Pelican Imaging Corporation | Systems and methods for stereo imaging with camera arrays |
US9800859B2 (en) | 2013-03-15 | 2017-10-24 | Fotonation Cayman Limited | Systems and methods for estimating depth using stereo array cameras |
US10542208B2 (en) | 2013-03-15 | 2020-01-21 | Fotonation Limited | Systems and methods for synthesizing high resolution images using image deconvolution based on motion and depth information |
US9497429B2 (en) | 2013-03-15 | 2016-11-15 | Pelican Imaging Corporation | Extended color processing on pelican array cameras |
US10455218B2 (en) | 2013-03-15 | 2019-10-22 | Fotonation Limited | Systems and methods for estimating depth using stereo array cameras |
US9497370B2 (en) | 2013-03-15 | 2016-11-15 | Pelican Imaging Corporation | Array camera architecture implementing quantum dot color filters |
US9898856B2 (en) | 2013-09-27 | 2018-02-20 | Fotonation Cayman Limited | Systems and methods for depth-assisted perspective distortion correction |
US10540806B2 (en) | 2013-09-27 | 2020-01-21 | Fotonation Limited | Systems and methods for depth-assisted perspective distortion correction |
US9426343B2 (en) | 2013-11-07 | 2016-08-23 | Pelican Imaging Corporation | Array cameras incorporating independently aligned lens stacks |
US9924092B2 (en) | 2013-11-07 | 2018-03-20 | Fotonation Cayman Limited | Array cameras incorporating independently aligned lens stacks |
US9264592B2 (en) | 2013-11-07 | 2016-02-16 | Pelican Imaging Corporation | Array camera modules incorporating independently aligned lens stacks |
US9185276B2 (en) | 2013-11-07 | 2015-11-10 | Pelican Imaging Corporation | Methods of manufacturing array camera modules incorporating independently aligned lens stacks |
US10119808B2 (en) | 2013-11-18 | 2018-11-06 | Fotonation Limited | Systems and methods for estimating depth from projected texture using camera arrays |
US11486698B2 (en) | 2013-11-18 | 2022-11-01 | Fotonation Limited | Systems and methods for estimating depth from projected texture using camera arrays |
US10767981B2 (en) | 2013-11-18 | 2020-09-08 | Fotonation Limited | Systems and methods for estimating depth from projected texture using camera arrays |
US9426361B2 (en) | 2013-11-26 | 2016-08-23 | Pelican Imaging Corporation | Array camera configurations incorporating multiple constituent array cameras |
US10708492B2 (en) | 2013-11-26 | 2020-07-07 | Fotonation Limited | Array camera configurations incorporating constituent array cameras and constituent cameras |
US9456134B2 (en) | 2013-11-26 | 2016-09-27 | Pelican Imaging Corporation | Array camera configurations incorporating constituent array cameras and constituent cameras |
US9813617B2 (en) | 2013-11-26 | 2017-11-07 | Fotonation Cayman Limited | Array camera configurations incorporating constituent array cameras and constituent cameras |
US9478010B2 (en) | 2013-12-12 | 2016-10-25 | Google Technology Holdings LLC | Generating an enhanced image of a predetermined scene from a plurality of images of the predetermined |
US10789680B2 (en) | 2013-12-12 | 2020-09-29 | Google Technology Holdings LLC | Generating an enhanced image of a predetermined scene from a plurality of images of the predetermined scene |
US10134111B2 (en) | 2013-12-12 | 2018-11-20 | Google Technology Holdings LLC | Generating an enhanced image of a predetermined scene from a plurality of images of the predetermined scene |
US10574905B2 (en) | 2014-03-07 | 2020-02-25 | Fotonation Limited | System and methods for depth regularization and semiautomatic interactive matting using RGB-D images |
US10089740B2 (en) | 2014-03-07 | 2018-10-02 | Fotonation Limited | System and methods for depth regularization and semiautomatic interactive matting using RGB-D images |
US9247117B2 (en) | 2014-04-07 | 2016-01-26 | Pelican Imaging Corporation | Systems and methods for correcting for warpage of a sensor array in an array camera module by introducing warpage into a focal plane of a lens stack array |
US9521319B2 (en) | 2014-06-18 | 2016-12-13 | Pelican Imaging Corporation | Array cameras and array camera modules including spectral filters disposed outside of a constituent image sensor |
US10250871B2 (en) | 2014-09-29 | 2019-04-02 | Fotonation Limited | Systems and methods for dynamic calibration of array cameras |
US11546576B2 (en) | 2014-09-29 | 2023-01-03 | Adeia Imaging Llc | Systems and methods for dynamic calibration of array cameras |
US9942474B2 (en) | 2015-04-17 | 2018-04-10 | Fotonation Cayman Limited | Systems and methods for performing high speed video capture and depth estimation using array cameras |
US10438322B2 (en) | 2017-05-26 | 2019-10-08 | Microsoft Technology Licensing, Llc | Image resolution enhancement |
US10482618B2 (en) | 2017-08-21 | 2019-11-19 | Fotonation Limited | Systems and methods for hybrid depth regularization |
US10818026B2 (en) | 2017-08-21 | 2020-10-27 | Fotonation Limited | Systems and methods for hybrid depth regularization |
US11562498B2 (en) | 2017-08-21 | 2023-01-24 | Adela Imaging LLC | Systems and methods for hybrid depth regularization |
US11699273B2 (en) | 2019-09-17 | 2023-07-11 | Intrinsic Innovation Llc | Systems and methods for surface modeling using polarization cues |
US11270110B2 (en) | 2019-09-17 | 2022-03-08 | Boston Polarimetrics, Inc. | Systems and methods for surface modeling using polarization cues |
US11525906B2 (en) | 2019-10-07 | 2022-12-13 | Intrinsic Innovation Llc | Systems and methods for augmentation of sensor systems and imaging systems with polarization |
US11842495B2 (en) | 2019-11-30 | 2023-12-12 | Intrinsic Innovation Llc | Systems and methods for transparent object segmentation using polarization cues |
US11302012B2 (en) | 2019-11-30 | 2022-04-12 | Boston Polarimetrics, Inc. | Systems and methods for transparent object segmentation using polarization cues |
CN113099146A (en) * | 2019-12-19 | 2021-07-09 | 华为技术有限公司 | Video generation method and device and related equipment |
US11580667B2 (en) | 2020-01-29 | 2023-02-14 | Intrinsic Innovation Llc | Systems and methods for characterizing object pose detection and measurement systems |
US11797863B2 (en) | 2020-01-30 | 2023-10-24 | Intrinsic Innovation Llc | Systems and methods for synthesizing data for training statistical models on different imaging modalities including polarized images |
US11290658B1 (en) | 2021-04-15 | 2022-03-29 | Boston Polarimetrics, Inc. | Systems and methods for camera exposure control |
US11683594B2 (en) | 2021-04-15 | 2023-06-20 | Intrinsic Innovation Llc | Systems and methods for camera exposure control |
US11954886B2 (en) | 2021-04-15 | 2024-04-09 | Intrinsic Innovation Llc | Systems and methods for six-degree of freedom pose estimation of deformable objects |
US11953700B2 (en) | 2021-05-27 | 2024-04-09 | Intrinsic Innovation Llc | Multi-aperture polarization optical systems using beam splitters |
US11689813B2 (en) | 2021-07-01 | 2023-06-27 | Intrinsic Innovation Llc | Systems and methods for high dynamic range imaging using crossed polarizers |
WO2023231138A1 (en) * | 2022-05-30 | 2023-12-07 | 元潼(北京)技术有限公司 | Multi-angle-of-view image super-resolution reconstruction method and apparatus based on meta-imaging |
Also Published As
Publication number | Publication date |
---|---|
CN1734500B (en) | 2010-04-14 |
WO2006012126A1 (en) | 2006-02-02 |
TWI298466B (en) | 2008-07-01 |
CN1734500A (en) | 2006-02-15 |
US20060104540A1 (en) | 2006-05-18 |
US7447382B2 (en) | 2008-11-04 |
TW200608310A (en) | 2006-03-01 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7447382B2 (en) | Computing a higher resolution image from multiple lower resolution images using model-based, robust Bayesian estimation | |
US7809155B2 (en) | Computing a higher resolution image from multiple lower resolution images using model-base, robust Bayesian estimation | |
US6285799B1 (en) | Apparatus and method for measuring a two-dimensional point spread function of a digital image acquisition system | |
US9692939B2 (en) | Device, system, and method of blind deblurring and blind super-resolution utilizing internal patch recurrence | |
Aly et al. | Image up-sampling using total-variation regularization with a new observation model | |
Lee et al. | Regularized adaptive high-resolution image reconstruction considering inaccurate subpixel registration | |
JP2003018398A (en) | Method for generating a super-resolution image from pixel image | |
JP2000188680A (en) | Method and system for producing high resolution image | |
JP2007072573A (en) | Image processor and image processing method | |
JP2008276393A (en) | Resolution enhancement device and method | |
JP2000244851A (en) | Picture processor and method and computer readable storage medium | |
JP2010187341A (en) | Image processing apparatus, imaging apparatus, image processing method, image processing program, and recording medium | |
JP5566199B2 (en) | Image processing apparatus, control method therefor, and program | |
JP4250237B2 (en) | Image processing apparatus, method, and computer-readable storage medium | |
JP4095204B2 (en) | Image processing apparatus, method, and computer-readable storage medium | |
US8571356B2 (en) | Image processing apparatus, image processing method, and image processing program | |
He et al. | Blind super-resolution image reconstruction using a maximum a posteriori estimation | |
Gevrekci et al. | POCS-based restoration of Bayer-sampled image sequences | |
Leitao et al. | Content-adaptive video up-scaling for high definition displays | |
Malczewski et al. | Super resolution for multimedia, image, and video processing applications | |
Messina et al. | Improving image resolution by adaptive back-projection correction techniques | |
Bätz et al. | Multi-image super-resolution for fisheye video sequences using subpixel motion estimation based on calibrated re-projection | |
KR100570630B1 (en) | Method for producing enhanced-resolution image by use of a plurality of low-resolution images | |
Wan et al. | Super-resolution image reconstruction | |
Salem et al. | Non-parametric super-resolution using a bi-sensor camera |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: MATSUSHITA ELECTRIC INDUSTRIAL CO., LTD., JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:DEGUCHI, FUTOSHI;YOSHINAGA, HIROSHI;TANAKA, MASAHIKO;AND OTHERS;REEL/FRAME:015558/0026 Effective date: 20040705 |
|
AS | Assignment |
Owner name: INTEL CORPORATION, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:NESTARES, OSCAR;HAUSSECKER, HORST W.;ETTINGER, SCOTT M.;REEL/FRAME:015817/0334;SIGNING DATES FROM 20040715 TO 20040720 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
FPAY | Fee payment |
Year of fee payment: 8 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 12TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1553); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 12 |
|
AS | Assignment |
Owner name: TAHOE RESEARCH, LTD., IRELAND Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:INTEL CORPORATION;REEL/FRAME:061175/0176 Effective date: 20220718 |