WO2014160142A1

WO2014160142A1 - Systems and methods for using alignment to increase sampling diversity of cameras in an array camera module

Info

Publication number: WO2014160142A1
Application number: PCT/US2014/025904
Authority: WO
Inventors: Jacques Duparre; Dan Lelescu
Original assignee: Pelican Imaging Corporation
Priority date: 2013-03-13
Filing date: 2014-03-13
Publication date: 2014-10-02

Abstract

Systems and methods in accordance with embodiments of the invention align lens stack arrays with arrays of focal planes to increase sampling diversity of the cameras in the resulting array camera modules. In one embodiment, a method for aligning a lens stack array with a sensor that includes a plurality of focal planes includes: initially aligning the lens stack array relative to the sensor; varying the spatial relationship between the lens stack array and the sensor so as to create at least two optical channels that each reflect a different central viewing and that can allow sub-pixel shifted images to be captured by their respective focal planes; selecting a spatial relationship between the lens stack array and the sensor that can provide a desired extent of variety of sub-pixel shifted images; and affixing the lens stack array and the sensor in the selected spatial relationship.

Description

Systems and Methods for Using Alignment to Increase Sampling Diversity of

Cameras in an Array Camera Module

FIELD OF THE INVENTION

[0001] The present invention generally relates to using alignment processes to increase sampling diversity of cameras in an array camera module.

BACKGROUND

[0002] In response to the constraints placed upon a traditional digital camera based upon the camera obscura, a new class of cameras that can be referred to as array cameras has been proposed. Array cameras are characterized in that they include an imager array, or sensor, that has multiple arrays of pixels, where each pixel array is intended to define a focal plane, and each focal plane has a separate lens stack. Typically, each focal plane includes a plurality of rows of pixels that also forms a plurality of columns of pixels, and each focal plane is contained within a region of the imager that does not contain pixels from another focal plane. An image is typically formed on each focal plane by its respective lens stack. In many instances, the array camera is constructed using an imager array that incorporates multiple focal planes and an optic array of lens stacks.

SUMMARY OF THE INVENTION

[0003] Systems and methods in accordance with embodiments of the invention align lens stack arrays with arrays of focal planes to increase sampling diversity of the cameras in the resulting array camera modules. In one embodiment, a method for actively aligning a lens stack array with a sensor that includes a plurality of focal planes, where each focal plane includes a plurality of rows of pixels that also form a plurality of columns of pixels and each focal plane is contained within a region of the imager array that does not contain pixels from another focal plane, includes: aligning the lens stack array relative to the sensor in an initial position, where the lens stack array includes a plurality of lens stacks and the plurality of lens stacks forms separate optical channels for each focal plane in the sensor; varying the spatial relationship between the lens stack array and the sensor so as to create at least two optical channels that each reflect a different central viewing direction relative to one-another and that can allow sub-pixel shifted images to be captured by their respective focal planes; capturing images of a known target using a plurality of active focal planes at different spatial relationships between the lens stack array and the sensor; evaluating to what extent the captured images reflect a variety of sub-pixel shifts for respective spatial relationships; selecting a spatial relationship between the lens stack array and the sensor that can provide a desired extent of variety of sub-pixel shifted images; and affixing the lens stack array and the sensor in the selected spatial relationship.

BRIEF DESCRIPTION OF THE DRAWINGS

[0004] FIG. 1 conceptually illustrates an array camera.

[0005] FIG. 2 illustrates an array camera module.

[0006] FIG. 3 illustrates an array camera module that employs a π filter.

[0007] FIG. 4 conceptually illustrates variations in focal length that can occur during the manufacture of an array camera module using a lens stack array and a sensor in accordance with embodiments of the invention.

[0008] FIG. 5 is a flowchart that illustrates a process for actively aligning a lens stack array and a sensor including an array of corresponding focal planes in accordance with an embodiment of the invention.

[0009] FIG. 6 illustrates rotating a lens stack array relative to a sensor in accordance with an embodiment of the invention

[0010] FIG. 7 illustrates tilting a sensor relative to a lens stack array in accordance with an embodiment of the invention

[0011] FIG. 8 schematically illustrates an initial configuration that may be used to actively align a lens stack array with a sensor in accordance with an embodiment of the invention.

[0012] FIG. 9 illustrates sweeping a lens stack array with respect to a sensor in accordance with an embodiment of the invention.

[0013] FIG. 10 illustrates a target that may be used during active alignment in accordance with many embodiments of the invention. [0014] FIG. 1 1 is a flowchart that illustrates an active alignment process that uses an iterative computation process to yield an array camera module that is capable of capturing and recording images that have sufficient on-axis and off-axis performance in accordance with an embodiment of the invention.

[0015] FIG. 12 illustrates a process for actively and passively aligning lens stack arrays and arrays of focal planes in accordance with an embodiment of the invention.

DETAILED DESCRIPTION

[0016] Turning now to the drawings, systems and methods for aligning lens stack arrays with arrays of focal planes on a monolithic sensor to increase the sampling diversity of the cameras in the resulting array camera module in accordance with embodiments of the invention are illustrated. Processes for constructing array camera modules using lens stack arrays are described in U.S. Patent Application Serial No. 12/935,504, entitled "Capturing and Processing of Images Using Monolithic Camera Array with Heterogeneous Imagers", Venkataraman et al. The disclosure of U.S. Patent Application Serial No. 12/935,504 is incorporated by reference herein in its entirety. The monolithic array camera modules illustrated in U.S. Patent Application Serial No. 12/935,504 can be constructed from an optic array of lens stacks, also termed a 'lens stack array', where each lens stack in the array defines an optical channel, and where the lens stack array is associated with a monolithic imager array, or 'sensor', including a plurality of focal planes corresponding to the optical channels in the lens stack array. Each focal plane can include a plurality of rows of pixels that also forms a plurality of columns of pixels, and each focal plane may be contained within a region of the imager array that does not contain pixels from another focal plane. An image may be formed on each focal plane by a respective lens stack. The combination of a lens stack array and a sensor can be understood to be an 'array camera module' and the combination of an individual lens stack and its corresponding focal plane within the sensor can be understood to be a 'camera.' Ideally, the lens stack array of an array camera is constructed so that each lens stack within it has the same focal length. However, the large number of tolerances involved in the manufacture of a lens stack array can result in the different lens stacks having varying focal lengths. The combination of all the manufacturing process variations typically results in a deviation of the actual ("first order") lens parameters - such as focal length - from the nominal prescription. As a result, each lens stack can have a different axial optimum image location. And consequently, since the sensor is monolithic, it typically cannot be placed a distance that corresponds with the focal length of each camera within an array camera module. There are a variety of processes in the manufacturing of conventional camera modules that can be utilized to align a lens stack array with a sensor to achieve acceptable imaging performance including active alignment processes and passive alignment processes.

[0017] In the context of the manufacture of camera systems, the term active alignment typically refers to a process for aligning an optical component or element (e.g. a lens stack array) with an image receiving component or element (e.g. comprising a monolithic sensor) to achieve a final desirable spatial arrangement by evaluating the efficacy of the image receiving component's ability to capture and record images as a function of the spatial relationship between the optical component and the image receiving component, and using this evaluation information to assist in the aligning process. Typically, this process is implemented by using the imaging system to capture and record image data (typically of a known target) in real time as the optical component is moving relative to the image receiving component. As the optical component is moved relative to the image receiving component, the spatial relationship between the two changes, and the characteristics of the recorded image data also change correspondingly. This recorded image data may then be used in aligning the optical component relative to the image receiving component in a desired manner. For example, active alignment can generally be used to determine a spatial relationship that results in a camera module that is capable of recording images that exceed a threshold image quality. Processes for actively aligning a lens stack array with an array of focal planes are described in U.S. Patent Application Serial No. 61/666,852, entitled "Systems and Methods for Manufacturing Camera Modules Using Active Alignment of Lens Stack Arrays and Sensors", Duparre et al. The disclosure of U.S. Patent Application Serial No. 61/666,852 is incorporated by reference herein in its entirety.

[0018] Ideally, when manufacturing camera modules in bulk, each camera module would be individually assembled using a rigorous assembly process, such as an active alignment process, to provide a quality configuration. However, performing such processes in bulk may be costly and time-consuming. An alternative to the use of an active alignment process to manufacture camera modules is the use of a passive alignment process. The term passive alignment typically refers to aligning an optical system with an imaging system to achieve a final desirable spatial arrangement using predetermined configuration parameters (e.g., the spacing between the lens stack array and the sensor is predetermined). In and of themselves, passive alignment processes can typically easily achieve a camera module that is in sufficient 'translational alignment,' i.e. the lens stacks of the lens stack array are lined up with their corresponding lenses (e.g. by using optical alignment marks also known as "fiducials" that are picked up by a vision system and used to guide the lateral alignment (artificially = in software) overlaying the alignment marks of the one component with those of the complementary one). Importantly, passive alignment processes may typically be implemented much more cost-effectively and much more rapidly than active alignment processes, since they typically do not involve iteratively, and laboriously, evaluating the efficacy of cameras during the alignment process, like active alignment processes do. However, precisely because passive alignment processes employ a less rigorous approach than active alignment processes, passive alignment processes may result in a less than optimal configuration because they cannot take into account material variations and/or tolerances. Processes for utilizing alignment information obtained during active alignment of one or more representative lens stack arrays and sensors to form array camera modules to manufacture array camera modules using passive alignment processes are disclosed in U.S. Patent Application Serial No. 61/772,443 entitled "Passive Alignment of Array Camera Modules Constructed from Lens Stack Arrays and Sensors Based Upon Alignment Information Obtained During Manufacture of Array Camera Modules Using an Active Alignment Process" to Duparre et al.. The disclosure of U.S. Patent Application Serial No. 61/772,443 is incorporated by reference herein in its entirety.

[0019] Processes for aligning lens stack arrays with sensors in accordance with many embodiments of the invention involve aligning the lens stack arrays with respect to sensors so as to enhance the resulting array camera module's ability to produce high-resolution images using super-resolution processes. Super-resolution refers to the process of synthesizing a plurality of low-resolution images of a particular scene - each image providing a sub-pixel shifted view of that scene (i.e. the object space sampled by the pixels is shifted relative to the other images captured by the array camera) - to derive a corresponding high-resolution image. Essentially, in a super-resolution process, sampling diversity between the low resolution images of a scene captured by an array camera module is utilized to synthesize one or more high resolution images of the scene. Thus, an array camera can capture and record a plurality of low-resolution images, and employ a super-resolution algorithm to generate a high-resolution image. Super-resolution processes that can be used to synthesize high resolution images from a plurality of low resolution images of a scene are described in U.S. Patent Application Serial No. 12/967,807 entitled "System and Methods for Synthesizing High Resolution Images Using Super-Resolution Processes" filed December 14, 2010, the disclosure of which is incorporated by reference herein in its entirety.

[0020] The extent to which super-resolution processing can be utilized to obtain an increase in resolution of an image synthesized from a plurality of low resolution images is dependent on the sampling diversity of the images. Importantly, the sampling diversity of the captured low resolution images is partly a function of the spatial relationship between the lens stack array and the sensor. Thus, many embodiments of the invention align the lens stack array with the array of focal planes to enhance the sampling diversity within the corresponding array camera module by discovering and implementing a spatial relationship between the lens stack array and the sensor that enables this result.

[0021] Array cameras and systems and methods for aligning lens stack arrays and sensors to increase sampling diversity between cameras in the resulting array camera modules in accordance with embodiments of the invention are discussed further below.

Array Camera Architectures

[0022] A variety of architectures can be utilized to construct an array camera using one or more array camera modules and a processor, including (but not limited to) the array camera architectures disclosed in U.S. Application Serial No. 12/935,504. A representative array camera architecture incorporating an array camera module and a processor is illustrated in FIG. 1 . The array camera 100 includes an array camera module 1 10, which is connected to an image processing pipeline module 120 and to a controller 130. In the illustrated embodiment, the image processing pipeline and the controller 130 are implemented using a processor. In various embodiments, the image processing pipeline module 120 is hardware, firmware, software, or a combination for processing the images received from the array camera module 1 10. The image processing pipeline module 120 is capable of processing multiple images captured by multiple focal planes in the camera module and can produce a synthesized higher resolution image. In a number of embodiments, the image processing pipeline module 120 provides the synthesized image data via an output 122.

[0023] In many embodiments, the controller 130 is hardware, software, firmware, or a combination thereof for controlling various operational parameters of the array camera module 1 10. The controller 130 receives inputs 132 from a user or other external components and sends operation signals to control the array camera module 1 10. The controller can also send information to the image processing pipeline module 120 to assist processing of the images captured by the focal planes in the array camera module 1 10.

[0024] Although a specific array camera architecture is illustrated in FIG. 1 , camera modules constructed using active alignment processes in accordance with embodiments of the invention can be utilized in any of a variety of array camera architectures. Camera modules that can be utilized in array cameras and processes for manufacturing camera modules utilizing active alignment processes in accordance with embodiments of the invention are discussed further below.

Array Camera Modules

[0025] An array camera module may be formed by aligning a lens stack array and an imager array in accordance with embodiments of the invention. Each lens stack in the lens stack array may define a separate optical channel. The lens stack array may be mounted to an imager array that includes a focal plane for each of the optical channels, where each focal plane includes an array of pixels or sensor elements configured to capture an image. When the lens stack array and the imager array are combined with sufficient precision, the array camera module can be utilized to capture image data from multiple images of a scene that can be read out to a processor for further processing, e.g., to synthesize a high resolution image using super-resolution processing.

[0026] An exploded view of an array camera module formed by combining a lens stack array with a monolithic sensor including an array of focal planes in accordance with an embodiment of the invention is illustrated in FIG. 2. The array camera module 200 includes a lens stack array 210 and a sensor 230 that includes an array of focal planes 240. The lens stack array 210 includes an array of lens stacks 220. Each lens stack creates an optical channel that resolves an image on the focal planes 240 on the sensor. Each of the lens stacks may be of a different type. For example, the optical channels may be used to capture images at different portions of the spectrum and the lens stack in each optical channel may be specifically optimized for the portion of the spectrum imaged by the focal plane associated with the optical channel. More specifically, an array camera module may be patterned with "π filter groups." The term π filter groups refers to a pattern of color filters applied to the lens stack array of a camera module and processes for patterning array cameras with π filter groups are described in U.S. Patent Application Serial No. 61/641 ,164, entitled "Camera Modules Patterned with π Filter Groups", Venkataraman et al. The disclosure of U.S. Patent Application Serial No. 61/641 ,164 is incorporated by reference herein in its entirety. FIG. 3 illustrates a single π filter group, wherein 5 lenses are configured to receive green light, 2 lenses are configured to receive red light, and 2 lenses are configured to receive blue light. The lens stacks may further have one or multiple separate optical elements axially arranged with respect to each other.

[0027] A lens stack array may employ wafer level optics (WLO) technology. WLO is a technology that encompasses a number of processes, including, for example, molding of lens arrays on glass wafers, stacking of those wafers (including wafers having lenses replicated on either side of the substrate) with appropriate spacers, followed by packaging of the optics directly with the imager into a monolithic integrated module.

[0028] The WLO procedure may involve, among other procedures, using a diamond- turned mold to create each plastic lens element on a glass substrate. More specifically, the process chain in WLO generally includes producing a diamond turned lens master (both on an individual and array level), then producing a negative mould for replication of that master (also called a stamp or tool), and then finally forming a polymer replica on a glass substrate, which has been structured with appropriate supporting optical elements, such as, for example, apertures (transparent openings in light blocking material layers), and filters.

[0029] Although the construction of lens stack arrays using specific WLO processes is discussed above, any of a variety of techniques can be used to construct lens stack arrays, for instance those involving precision glass molding, polymer injection molding or wafer level polymer monolithic lens processes. Issues related to variation in back focal length of the lens stacks within lens stack arrays are discussed below.

Back Focal Plane Alignment

[0030] An array camera module is typically intended to be constructed in such a way that each focal plane (i.e. an array of pixels configured to capture an image formed on the focal plane by a corresponding lens stack) is positioned at the focal distance of each lens stack that forms an optical channel. However, manufacturing variations can result in the lens stack in each optical channel varying from its prescription, and in many instances, these variations can result in each lens stack within a lens stack array having a different focal length. For example, parameters that may vary amongst individual lens stacks in a lens stack array because of manufacturing variations include, but are not limited to: the radius of curvature in individual lenses, the conic, higher order aspheric coefficient, refractive index, thickness of the base layer, and/or overall lens height. As one of ordinary skill in the art would appreciate, any number of lens prescriptions may be used to characterize the lens fabrication process, and the respective tolerances may involve departures from these prescriptions in any number of ways, each of which may impact the back focal length. Due to the monolithic nature of the sensor, the spatial relationship of the focal planes (with respect to the lens stacks) cannot be individually customized to accommodate this variability.

[0031] The variations in focal length that can occur in a conventional lens stack array are conceptually illustrated in FIG. 4. The array camera module 400 includes a lens stack array 402 in which lens stacks 404 focus light on the focal planes 406 of sensor 408. As is illustrated, variance between the actually fabricated lens stack and its original prescription can result in the lens stack having a focal length that varies slightly from its prescription and consequently an image distance that does not correspond with the distance between the lens stack array and the sensor. Accordingly, the images formed on the focal planes of the sensor can be out of focus. In addition, other manufacturing tolerances associated with the assembly of the array camera module including (but not limited to) variations in spacer thickness and alignment of the lens stack array relative to the sensor can impact all of the optical channels.

[0032] Therefore, as discussed in U.S. Patent Application Serial No. 61/666,852, active alignment processes may be incorporated in the manufacture of array camera modules to mitigate this effect. Moreover, in accordance with embodiments of the invention, active alignment processes can include aligning a lens stack array with a sensor to increase sampling diversity in the constituent cameras of the resulting array camera module. Aligning lens stack arrays with sensors to increase sampling diversity in these constituent cameras is discussed below.

Increasing the Sampling Diversity of Cameras in Array Camera Modules Using Active Alignment Processes

[0033] In many embodiments, lens stack arrays and sensors are actively aligned to increase sampling diversity in the constituent cameras of the resulting array camera modules. For example, in many embodiments, a lens stack array and sensor are aligned such that the resulting cameras reflect multiple fields of view, e.g. multiple viewing directions, that can capture images having sub-pixel offsets. In many instances, it is desirable that this field of view sampling be able to produce a variety of sub-pixel shifted images of a respective scene. The sub-pixel shifted images can thereby be harnessed to construct a super-resolved image of the respective scene. A greater variety in the captured sub-pixel shifted images can allow the super-resolution processes to be more effective. Moreover, it can be also be beneficial to align the lens stack array and the sensor so that local regions of the captured images also reflect sub- pixel shifts; in this way, each of the local regions can be properly super-resolved. Thus, in many embodiments, the alignment of the lens stack array and the sensor is controlled so as to control the extent of the variety of sub-pixel shifted images.

[0034] Active alignment processes are discussed in U.S. Patent Application Serial No. 61/666,852. A typical process that actively aligns a lens stack array with a sensor by generally aligning the two, varying their spatial relationship, evaluating the resulting configuration during the variation, and configuring the array camera module using the evaluation data in accordance with an embodiment of the invention is illustrated in FIG. 5. In particular, a lens stack array is generally aligned (510) with a corresponding sensor that has multiple focal planes; the spatial relationship of the lens stack array with respect to the sensor is varied (520); the quality of the captured image data is evaluated (530) at the varied spatial relationships; the array camera module is configured (540) using the information obtained during evaluation.

[0035] In accordance with embodiments of the invention, active alignment processes can be tailored to increase sampling diversity in the constituent cameras of the resulting array camera modules. In particular, in many embodiments, the spatial relationship between the lens stack array and sensor is varied (520) in ways that would expectedly increase the sampling diversity in the images captured and recorded by the plurality of cameras. For example, rotating a lens stack array relative to the sensor may be expected to increase the sampling diversity in the images captured by the constituent cameras in the resulting array camera module. The rotation of a lens stack array relative to a sensor in accordance with embodiments of the invention is illustrated in FIG. 6. In the illustrated embodiment, the spatial relationship between the lens stack array 610 and the sensor 620 is varied by rotating the lens stack array 610 relative to the sensor 620 - the center of rotation being one corner of the configuration. Consequently, each camera obtains a different central viewing direction, and the extent of this 'de-centration' is a function of the radial distance between a particular lens and the center of rotation. Thus, as each camera can capture an image with a different central viewing direction, the rotation can be expected to increase sampling diversity for purposes of super- resolution. Note that the lens stack array can be rotated to any extent, and the center of rotation can be any point, all in accordance with embodiments of the invention. [0036] A similar effect can be obtained by tilting the sensor relative to the lens stack array. FIG. 7 illustrates varying the spatial relationship between the lens stack array and the sensor by tilting the sensor relative to the lens stack array about a central axis of the sensor in accordance with embodiments. In the illustrated embodiment, the sensor 720 is tilted relative to the lens stack array 710. Sample ray 730 illustrates the effect that this has on the image captured and recorded by the sensor. Specifically, in the tilted alignment, a ray that strikes focal plane 722, does so at a point 750 that is toward one end of the focal plane 722. Conversely, if the sensor 730 was in the un- tilted arrangement (indicated by the dashed lines), the same ray would have hit focal plane 722 at a more central point 751 . Essentially, the captured image becomes offset, and the extent of the offset is proportional to the change in the distance between a focal plane and its corresponding lens stack. In this way, tilting augments the central viewing direction. And since tilting a lens stack array relative to its sensor necessarily results in a plurality of different distances between a focal plane and its respective lens stack, tilting will necessarily result in a corresponding plurality of central viewing directions. As before, the increase in central viewing directions can increase sampling diversity, and thereby enhance super-resolution processes. Note that the lens stack array can be tilted to any extent, and can be tilted about any axis, all in accordance with embodiments of the invention.

[0037] Referring back to FIG. 5, the configuration of the array camera module can be evaluated (530) at these varied spatial relationships. For example, the lens stack array can be rotated through various degrees of rotation relative to the sensor, and the ability of the configuration to super-resolve low resolution image data and thereby generate a high resolution image can be evaluated at the varied degrees of rotation. Alternatively, both prior to and subsequent to the rotation, images may be captured by the constituent cameras, and they may be contrasted with one another to assess to what extent they convey a sub-pixel offset, which would reflect an increased sampling diversity, and expectedly result in greater super-resolving properties. In several embodiments, the sampling diversity can be measured using a known target spaced a predetermined distance from the array camera module. Low resolution image data captured by some or all of the cameras in a color channel (typically green) can be fused onto a high resolution grid based upon geometric transformations that can be determined from the testing configuration. The extent of the sampling diversity can be determined based upon the presence of pixel stacks within the fused image. The larger the number of pixel stacks, the lower the sampling diversity. In several embodiments, sampling diversity is ascertained by looking at the numbers of pixel stacks and/or empty locations on the high resolution grid in 3 x 3 portions of the high resolution grid. In other embodiments, any of a variety of techniques can be utilized to determine sampling diversity based upon the characteristics of fused image data captured by the cameras in the array camera module in accordance with embodiments of the invention. As explained above, it can be expected that the rotation of the lens stack array relative to the sensor will enhance the super-resolving abilities of the configuration, because the rotation can increase the sampling diversity. The array camera module can be configured (540) accounting for this evaluation information.

[0038] Although alignment of lens stack arrays with sensors to increase sampling diversity by using alignments that modify the central viewing direction and/or chief ray angle of each resulting camera are discussed above in the context of active alignment processes, similar techniques can be utilized during passive alignment to vary the central viewing direction and/or chief ray angle of each camera in the resulting array camera modules. Additionally, while array camera modules are discussed, it is not meant to be suggested that the above-described processes are limited to the formation of monolithic array cameras. In many embodiments, lens stack arrays and sensors are aligned to form modules that can be used in the construction of non-monolithic array cameras. U.S. Pat. App. No. 14/188521 , entitled "Thin Form Factor Computational Array Cameras and Modular Array Cameras", to Venkataraman et al. discusses non- monolithic array camera modules, and is hereby incorporated by reference in its entirety. Additionally, U.S. Provisional Pat. App. No. 61/901 ,378, entitled "Non- Monolithic 3 x 3 Array Module with Discrete Sensors and Discrete Lenses" and U.S. Provisional Pat. App. No. 61/904,947, entitled "Array Camera Modules and Methods of Manufacturing Array Camera Modules Incorporating Independently Aligned Lens Stacks", to Mark Rodda et al. also discuss non-monolithic array camera modules. Thus, for instance, in many embodiments, during the sequential active alignment of the lenses on the respective sensors not only MTF and centering is used as an alignment criterion but also "sampling diversity" against the previously aligned channels. Active alignment processes, passive alignment processes and passive alignment processes that utilize alignment information obtained during the active alignment of a representative array camera module that can be utilized to manufacture array cameras including increased sampling diversity in accordance with embodiments of the invention are discussed in more detail below.

Active Alignment Processes

[0039] In many embodiments, processes for actively aligning a lens stack array with a sensor to construct an array camera module involve reading image data captured by multiple focal planes on the sensor as the lens stack array is moved relative to the sensor thereby acquiring image data at varied spatial relationships. The image data can be utilized to evaluate the resulting image quality at different spatial relationships between the sensor and the lens stack array and the spatial relationship that provides a predetermined threshold level of image quality can be utilized to construct the camera module. As referenced above, a process that actively aligns a lens stack array with a sensor by generally aligning the two, varying their spatial relationship, evaluating the resulting configuration during the variation, and configuring the array camera module using the evaluation data in accordance with an embodiment of the invention is illustrated in FIG. 5.

[0040] Referring back to FIG. 5, a lens stack array is generally aligned (510) with a corresponding sensor that has multiple focal planes. The combination is aligned so that each camera within the configuration is capable of capturing and recording images. The spatial relationship of the lens stack array with respect to the sensor is varied (520). In several embodiments, the variation is achieved by sweeping the lens stack array with respect to the sensor. Sweeping can be understood to mean moving one component (i.e. either the lens stack array or the sensor) in relation to the other over time. Sweeping may be in one degree of freedom or it can be across many degrees of freedom. As can readily be appreciated, the array nature of the camera module means that variations in the x, y, and z-directions, and tip/tilt and rotation of the lens stack array with respect to the sensor can all have significant impact on the imaged data captured by the focal planes on the sensor. Note that in many array cameras, focus and consequently sharpness of the cameras is primarily affected by the z-direction and the tip/tilt of the lens stack array with respect to the sensor, with the tip/tilt principally affecting the performance of the corner cameras. Conversely, in a conventional camera that comprises only a single lens stack, the image quality of the camera is primarily driven by the optical system's 'z-position' with respect to the sensor. In many embodiments, the path of the sweep is predetermined. Additionally, as already discussed above in the preceding section, the variation can also include rotating the lens stack relative to the sensor, so as to increase the sampling diversity of the cameras.

[0041] The quality of the captured image data is evaluated (530) at the varied spatial relationships. For example, in several embodiments of the invention, the configuration is intermittently evaluated during a sweep of the lens stack array with respect to the sensor. In many embodiments, the configuration is evaluated by evaluating multiple cameras' captured and recorded images of a known target at the varied spatial relationships. In several embodiments, only a subset of the configuration's cameras is used for evaluation purposes. An MTF score may be determined for each recorded image and used to evaluate a respective camera at a respective spatial orientation. The recorded images may also be evaluated at its different ROIs. For example, an MTF score may be assigned to each ROI within a recorded image.

[0042] Moreover, as discussed in the preceding section, evaluating the captured image data at the varied spatial relationships may involve super-resolving the images, and evaluating the resulting high-resolution image, or alternatively evaluating the sampling diversity within the low resolution images. As stated above, an array camera's ability to provide a high-resolution image using super-resolution processes is related to the sampling diversity of the captured images; and the sampling diversity can be related to the spatial relationship of the lens stack array relative to the sensor. Thus, the ability of the lens stack array and the sensor to provide a high-resolution image using super- resolution processes may be determined as a function of the spatial relationship between the lens stack array and the corresponding sensor. [0043] The array camera module is configured (540) using the information obtained during evaluation. In many embodiments, the configuration involves concluding a spatial relationship between the lens stack array and the sensor that results in the focal planes being able to capture and record images that exceed a threshold quality. In numerous embodiments, the configuration involves concluding a spatial relationship between the lens stack array and the sensor that results in the array camera module being able to provide a high-resolution image of a captured scene using super- resolution processes, with the high-resolution image exceeding a threshold quality. The configuration may also involve disabling cameras that do not surpass a threshold quality. Again, because array camera modules include a plurality of cameras, they can still function even when one or more of the cameras are disabled. The advantage of being able to disable a camera is that the average performance of the array including the camera may be much lower than the average performance of the remaining cameras when the disabled camera is excluded from consideration in determining the appropriate alignment of the lens stack array and sensor.

[0044] Although a process, and its variants, that actively aligns a lens stack array with a corresponding array of focal planes has been described, any of a number of different processes may be used to actively align a lens stack array with an array of focal planes in accordance with embodiments of the invention. An initial configuration for an active alignment process in accordance with embodiments of the invention is discussed below.

Initial Configuration for Aligning a Lens Stack Array with an Array of Focal Planes

[0045] Active alignment processes may begin from any number of initial configurations in accordance with embodiments of the invention. An initial configuration for an active alignment process where a device that is capable of orienting a lens stack array is connected to a lens stack array of a corresponding array camera module, a processor is connected to the corresponding sensor, and a target is positioned and illuminated so that the array camera module can capture and record it in accordance with an embodiment of the invention is illustrated in FIG. 8. The array camera module 810 includes a lens stack array 820 and a sensor 830 that has corresponding focal planes. The lens stack array and the sensor are generally aligned so that they are capable of capturing and recording images of the target 840. A device that is capable of spatially orienting the lens stack array 840 is connected to the lens stack array 820, and a processor 860 is connected to the sensor. Thus, the processor 860 is capable of capturing and recording images from the sensor 830, while the orientation of the lens stack array 820 is being varied, and the active alignment process can thereby be implemented. The combination of the device for spatially orienting the lens stack array 850 and the processor 860 can be understood to be an active alignment machine 870.

[0046] In some embodiments, the initial configuration involves generally aligning the lens stack array 820 and the sensor 830 so as to ensure that the lens stack array 820 and the sensor 830 are in sufficient translational and rotational alignment such that each lens stack is generally aligned with its corresponding focal plane. Translational motion here refers to motion of a system (i.e. the lens stack array 820 or the sensor 830) in a direction parallel to its respective surface. Rotation here refers to rotation of a system about the Z-axis (i.e. the axis defining the distance between the sensor and the lens stack array) relative to the other. General alignment may be achieved by, for example, monitoring a central feature on a test chart, and moving either the lens stack array or the sensor in translation (with respect to the other system) such that the central feature is centrally located within the central camera modules; this would indicate that the systems are in sufficient translational alignment. Either system may then be rotated with respect to the other so that the midpoints of each lens stack array and its corresponding focal plane define a line that runs generally parallel to the Z-axis. During this rotational adjustment, the systems may also be readjusted to preserve (or enhance) adequate translational alignment. In this way, each lens stack array may be generally aligned with its corresponding focal plane.

[0047] Although many embodiments of the invention employ the initial configuration illustrated in FIG. 8, many other embodiments employ other initial configurations appropriate to the requirements of specific applications. In accordance with embodiments of the invention, any initial configuration may be implemented that allows the spatial relationship between the lens stack array and the sensor to be varied, and further allows the corresponding array camera module to be evaluated, manipulated, and configured based on an evaluation of it. The varying of spatial relationships between the lens stack array and the sensor in accordance with embodiments of the invention is discussed below.

Varying the Spatial Relationship of the Lens Stack Array with Respect to the Sensor

[0048] The spatial relationship between a lens stack array and a corresponding sensor may be varied in any number of ways. For example, an active alignment process where a lens stack array is swept in a direction substantially normal to the sensor's planar surface in accordance with embodiments of the invention is illustrated in FIG. 9. An array camera module 900 includes a lens stack array 910 and a corresponding sensor 920 with an array of focal planes, and the active alignment process sweeps the lens stack array 910 in a predetermined direction 930 substantially normal to the sensor's surface (the z-direction). Note that sweeping the lens stack array in this fashion systematically varies the focus of each camera - typically cameras will be swept in focus and then out of focus. The array camera module may be evaluated on the varied spatial relationships along this sweep. Active alignment processes in accordance with embodiments of the invention can also include tipping, tilting, and/or rotating the lens stack array with respect to the sensor. In many embodiments, only the distance between the lens stack array and the sensor is varied in a sweep referred to as a "through focus sweep" and all relevant calculations to determine the optimum alignment, with respect to mitigating the variance in focal length (including centering as well as focus and tip/tilt), are made from images captured during the through focus sweep using the respective curve fittings and center of gravity calculations, respectively. As can be appreciated, a through focus sweep of a skewed lens stack array already provides information about the optimum tip/tilt of the lens stack array relative to the sensor by the appropriate plane fitting calculations of the peak focus positions or equalized MTF, respectively. These calculations are discussed further below.

[0049] Additionally, as already discussed above in the section above titled "Increasing the Sampling Diversity of Cameras in Array Camera Modules Using Active Alignment Processes," the spatial relationship between a lens stack array and a sensor may be varied by rotating the lens stack array with respect to the sensor. As also discussed above, this rotation may result in greater sampling diversity in the images captured by the cameras.

[0050] In numerous embodiments, the manner in which the spatial relationship varies is computationally determined. For example, the manner in which the spatial relationship varies may be determined computationally based upon an initial evaluation of the array camera module. Additionally, the manner in which the spatial relationship varies may change during an active alignment process. For instance, after the lens stack array has been swept in a direction substantially normal to the sensor's planar surface, a processor may compute a different sweeping path that may facilitate a better configuration of the array camera module.

[0051] Although several examples have been described related to how the spatial relationship between the lens stack array and the sensor may be varied, the spatial relationship may also be varied in any number of other ways in accordance with embodiments of the invention. The evaluation of the array camera module at the varied spatial relationships is discussed below.

Evaluating the Array Camera Module

[0052] In numerous embodiments, evaluating the array camera module during the active alignment process involves having multiple cameras capture and record images of a known target, and evaluating these images. The images may be evaluated by assessing their focus, for example. The assessment of the focus may be performed in any number of ways in accordance with embodiments of the invention. For example, in many embodiments, an MTF score may be determined for a given recorded image. Generally speaking, an MTF score is an advantageous metric insofar as MTF scores amongst different cameras can be directly compared with one another. In some embodiments, a recorded image may be given a 'focus score' which can similarly be used to evaluate the recorded image. For example, a focus score may be determined by convolving a kernel over contrasting features in an image, where the resulting value is related to the camera's ability to focus. Unlike the MTF score, a focus score may not necessarily be directly comparable to such scores from different cameras; instead a focus score may be more useful in evaluating a single camera. [0053] The selection of which scoring metric to use may be determined, in part, by the speed in which the scores can be calculated. For instance, if it takes longer to compute an MTF score than to compute a focus score, the focus score may be used in the evaluation. The selection of which scoring metric to use may also be determined, in part, by the accuracy and precision of the score. For instance, if the MTF score is a more precise means for evaluating image quality, then it may be used to evaluate the camera images. Moreover, the active alignment process may utilize several methods of evaluating a recorded image, and these methods may not necessarily be concurrent. For example, an evaluation based on focus scoring may be initially used, whereas an evaluation based on an MTF score may later be used. Additionally, the active alignment process may involve relating the different scoring metrics. For example, focus scoring may be used to evaluate the set of images recorded by an array camera, and MTF scoring may be used to evaluate a representative subset of those images. The MTF scores for the subset may then be normalized to the respective focus scores. And this determined relationship may be used to determine MTF scores for the remaining images.

[0054] Additionally, different regions of recorded images may be evaluated, thereby providing information on a camera's quality as to specific regions. For example, in certain embodiments, images are recorded of a known target that has multiple "Regions of Interest" (ROIs), and the cameras' recorded images of the known target are evaluated with respect to each region of interest. FIG. 10 illustrates a known target used in accordance with many embodiments of the invention. The known target 1000 includes a central feature 1010 that highlights a central ROI, also known as an "on-axis" ROI. The known target further includes features 1020 that highlight "off-axis" ROIs. The target in FIG. 10 is advantageous in so far as the edges of the features are oriented in such a way that the tangential and sagittal components of the MTF score, and thus also the astigmatism, can be directly derived and compared to prior lens test data. Thus, many embodiments utilize the known target illustrated in FIG. 10 by evaluating the quality of each camera with respect to each of the five ROIs.

[0055] The target illustrated in FIG. 10 may also be used in determining a focus score. Specifically, the determination of a focus score in conjunction with this target may involve convolving a kernel over areas of the image with contrasting features for each region of interest (e.g. the checkerboard patterns 1040 or the dark slanted square against the light background 1050), wherein the resulting value is proportional to the contrast between the features. For example, the following convolution kernel may be employed:

-1 , -1 , -1 , -1 , -1 I

-1 , -1 , 24, -1 , -1 I

-1 , -1 , -1 , -1 , -1 I

[0056] This convolution kernel will yield values that are proportional to a camera's ability to resolve contrast. Note that the value will either be positive or negative depending on whether the region being evaluated is transitioning from light to dark or dark to light. However, whether a region of interest is transitioning from light to dark or vice versa is irrelevant to a camera's ability to focus; therefore the absolute value of these values should be obtained. Then, a focus score for each ROI may be obtained by averaging these absolute values for each ROI.

[0057] Although FIG. 10 illustrates a particular known target that may be used in accordance with embodiments of the invention, many other embodiments utilize other known targets appropriate to the requirements of specific applications. For instance, the off-axis ROIs may be placed in the corners of the target - this allows the performance of the camera to be tested at larger field heights. In the illustrated embodiment, the ROIs have the advantage that the edges of the features are oriented in such a way that the tangential and sagittal components of the MTF and thus also the astigmatism can be directly derived and compared to prior lens test data. Moreover, although specific examples of how a focus score may be generated are provided, any of a variety of techniques can be used to generate a focus score.

[0058] Additionally, as already discussed above in the section titled "Increasing the Sampling Diversity of Cameras in Array Camera Modules Using Active Alignment Processes," the evaluation of the lens stack array and the sensor at the varied spatial relationships may entail super-resolving captured and recorded images, and evaluating those images. Alternatively, as discussed above, images captured and recorded by the constituent cameras may be contrasted with one another to determine the extent that they reflect sub-pixel offsets. Thus, the ability of the lens stack array and the sensor to provide a high-resolution image using super-resolution processes may be determined. As noted above, the result of the super-resolution process is related to the sampling diversity in the plurality of images, and the sampling diversity is related to the spatial arrangement of the lens stack array relative to the sensor. Thus, the array camera module's ability to produce a high-resolution image as a function of the spatial relationship of a lens stack array relative to the sensor may be determined.

[0059] More generally, the evaluation techniques herein described are merely illustrative. Any techniques and any criterion for evaluating the efficacy of an array camera module may be incorporated in accordance with embodiments of the invention. Using the evaluation data to configure the array camera module is discussed below.

Configuring the Array Camera Module

[0060] Evaluation data may be used to configure the array camera module in a number of respects. In many embodiments the array camera module is configured to minimize the detrimental impact caused by variance of focal length within a lens stack array. In a number of embodiments, the array camera module is configured to increase sampling diversity. As described above, variance within a lens stack array may be caused by manufacturing process variations including (but not limited to) those that affect the following parameters: the radius of curvature in individual lenses, the conic, higher order aspheric coefficient, refractive index, thickness of the base layer, and/or overall lens height. Additionally, as described above, the following manufacturing variations related to the fabrication of multiple lens stack arrays and camera modules may further exacerbate the variability in back focal lengths: the thickness of the lens substrates and spacers employed in the stack, especially those toward the sensor cover glass, the thickness of the sensor cover glass used, bond line thickness between the lens spacer and the sensor cover glass, and any air gap between the sensor and the sensor cover glass. Thus, many embodiments evaluate the quality of each camera as a function of its spatial relationship to the sensor; thereafter, the information is used to orient the lens stack array with respect to the sensor so that any deterioration in the quality of the array camera due to the variance in focal length within the lens stack array is lessened.

[0061] Several embodiments generate mathematical equations that approximately characterize data related to camera quality as a function of spatial relationship, and use the derived equations to compute a desired spatial relationship that lessens the detrimental impact of variance in focal length and/or increase sampling diversity. For example, some embodiments generate polynomial equations that approximately model the focal scoring data. Note that because of the nature of optics, each lens will typically have a peak focal value, and therefore polynomial equations are well suited to characterize the data. In many embodiments, the polynomial equations are generated by determining coefficients for predetermined generic polynomial equations (i.e. those with undetermined coefficients), such that the resulting equation approximately characterizes the data relating the camera quality to the spatial relationship. Many embodiments then use these derived equations to compute a best fit plane that characterizes a spatial relationship that reduces the detrimental impact of variance in focal length.

[0062] Notably, the best-fit planes may be computed in any number of ways. For instance, the best-fit plane may be computed to be a plane that includes an approximation of the peak values of the polynomial equations that characterize focal scoring data as a function of the spatial relationship. But, as described above, focal scoring data may not necessarily be directly comparable across different cameras. Therefore, best-fit planes may also be computed by generating equivalent MTF scores, and determining a plane that maximizes the mean MTF score while minimizing its variance. Specifically, the best-fit planes may be computed to determine a plane wherein the MTF scores amongst the different lens stacks are equalized within some specified tolerance. Moreover, any number of balancing algorithms may be employed to effectuate this computation as appropriate to the requirements of a specific application. The determination of these planes may then be used to facilitate the configuration of the array camera module.

[0063] In several embodiments, the configuration process involves orienting the lens stack array with respect to the sensor to form an array camera module that is capable of achieving pictures that have desired characteristics. In some embodiments, the lens stack array is oriented with respect to sensor so as to achieve an array camera module that is capable of recording images, wherein the quality of the on-axis aspects of the recorded image exceeds a specified threshold criterion. In several embodiments, the lens stack array is actively aligned with respect to the sensor to achieve an array camera module that is capable of recording images, wherein the quality of the off-axis aspects of the recorded image exceeds a specified threshold criterion. Note also that in various embodiments, the configuration process may involve disabling cameras that are above a certain threshold quality so as to avoid biasing the best fit plane determination. In numerous embodiments, the lens stack array is actively aligned with respect to the sensor to achieve an array camera module that is capable of recording images, wherein the quality of both on-axis and off-axis regions of interest exceed respective specified threshold qualities. In many embodiments, the lens stack array is aligned relative to the sensor so as to enhance the ability of the array camera module to provide a high quality image using super-resolution processes. As is evident from the above discussion, this can be done, for example, by selecting a spatial relationship that allows the cameras to capture and record a set of images with enhanced sampling diversity. In numerous embodiments many of the above-described concepts are applied in configuring the array camera module. For example, polynomial equations may be used in computing a spatial relationship that reduces the variance in back focal plane distance, the spatial relationship may be implemented, and then the lens stack array may be rotated relative to the sensor to increase sampling diversity.

[0064] In many embodiments, the configuration process involves disabling cameras that perform above or below a certain defined threshold quality. Again, because an array camera module has many cameras, it is possible for it to maintain functionality even when some of its cameras are non-functional. In several embodiments, cameras are disabled when their quality, as determined by their ability to focus sharply when in a given spatial orientation, is above or below a threshold value. For example, some embodiments determine whether a camera should be disabled by evaluating an MTF score of its respective recorded images. In many embodiments, if the number of disabled cameras exceeds a specified value, then the array camera module is designated unacceptable. In several embodiments, different threshold values can be specified for different types of cameras within the array camera module. For example, in a number of embodiments that employ π filter groups, different threshold values can be specified for the green cameras, the red cameras, and the blue cameras.

[0065] In various embodiments, information obtained during the evaluation aspect of the active alignment process is used to configure the functionality of the each camera. For example, if it is determined that a particular camera has a focal length that makes it better suited to record images of objects that are at a further distance, the array camera module can be configured to rely more heavily on that camera when synthesizing recorded images of objects at further distances.

[0066] The above descriptions regarding configuring an array camera module in accordance with embodiments of the invention is not meant to be exhaustive. Indeed, array camera modules can be configured in any number of ways based on evaluations of the configuration in accordance with embodiments of the invention. Active alignment processes that configure array camera modules so that they are capable of capturing and recording images that have desirable image properties are discussed below.

Active Alignment Processes that Yield Array Camera Modules Capable of Recording Images that have Desirable Characteristics

[0067] Active alignment processes in accordance with embodiments of the invention can use a variety of metrics to evaluate the image data that is captured during the active alignment process. In several embodiments, the active alignment process can optimize image quality in specific regions of the captured images, can optimize image quality in multiple regions of interest and/or can utilize a variety of metrics including (but not limited to) focus scoring and MTF scoring. An active alignment process that uses an iterative computation process to yield an array camera module that is capable of capturing and recording images that have sufficient on-axis and off-axis performance in accordance with an embodiment of the invention is illustrated in FIG. 1 1 .

[0068] The process is initially configured (1 102) so that a lens stack array and a corresponding sensor are mounted to an active alignment machine in a manner similar to that seen in FIG. 6, so that they are generally operable as an array camera. This may include generally aligning the lens stack array with its corresponding sensor, which itself may include verifying that the lens stack array and the sensor are in sufficient rotational alignment such that each lens stack is generally aligned with its corresponding focal plane, as described above. A known target with an on-axis ROI and off-axis ROIs (similar to that depicted in FIG. 10) is positioned and illuminated so that the array camera module may capture and record its image. The initial configuration may also include deactivating specific cameras in a predetermined fashion so that they do not record images during the alignment process.

[0069] The lens stack array is swept (1 104) in a direction normal to the sensor's planar surface, in a manner similar to that seen in FIG. 7, and may be swept for a predetermined distance. During the sweep, the active cameras intermittently capture and record (1 106) images of the known target. The processor evaluates (1 108) the recorded images and assigns a 'focus score' for each region of interest in each recorded image for each camera. Polynomial equations are derived (1 1 10) for each region of interest captured by each camera that best characterizes the focus score as a function of the camera's distance from the sensor. In some embodiments, the polynomial equations are derived by calculating coefficients for a given a predetermined generic polynomial equation (i.e. a polynomial equation with undetermined coefficients). The polynomial equations will typically have a peak value.

[0070] An "on-axis best fit plane" is derived (1 1 12) using the peak values of the polynomial equations. The on-axis best fit plane, is characterized in that it maximizes the peak values corresponding to the active cameras and/or minimizes the variance in the peak values.

[0071] The lens stack array is then aligned (1 1 14) with the computed best fit on-axis plane. Each active camera captures and records (1 1 16) an image of the known target. Each recorded image is then evaluated (1 1 18) by determining an MTF score for each ROI. Cameras that do not meet a threshold MTF score are disabled (1 120). For example, any cameras that do not have an MTF score within 20% of the median on-axis MTF score may be disabled, and subsequently excluded from further alignment position calculations. This threshold may of course be configurable. In other embodiments, other criteria are utilized to determine which cameras should be disabled. Moreover, if a specified number of cameras are disabled, the array camera is deemed unacceptable.

[0072] Assuming the camera is not deemed unacceptable, the previously acquired focus scoring data is scaled (1 122) using the peak focus score and MTF scores. For example, the MTF Score may be scaled in accordance with the following formula:

Scaled Focus Score_z = (Focus Score_z/Peak Focus Score) ^* MTF Score where the z subscript reflects the score at a particular z-position.

[0073] The focus scoring data (absolute values) are exposure/signal-level dependent. Thus different cameras (e.g. blue, green, red cameras) will have different absolute focus score peak values due to their different signal levels. However, MTF is a metric that is invariant to signal level. Thus, MTF enables the curves for focus score to be normalized such that the curve derived from focus score can also be used to compare each camera's peak performance and not only the position at which peak performance occurs. In other embodiments, any of a variety of metrics appropriate to a specific application can be utilized in determining camera peak performance.

[0074] As before, polynomial curves may then be derived (1 124) that characterize the scaled focus scores. Thus, each active camera will be characterized by polynomial equations that characterize the camera's ability to resolve each respective region of interest. Given these new polynomial equations, a best-fit on axis plane and a best-fit off axis plane are derived (1 126); in this instance, the best-fit planes are characterized in that they approximately maximize the mean MTF scores while minimizing their variance. A configurable number of planes that are evenly spaced between the two best-fit planes (on-axis and off-axis) are computed (1 128). Scaled focus scores for each camera at their respective corresponding positions along each of those planes are calculated (1 130). A best-fit plane determined (1 132) wherein any deviation toward the best-fit off axis plane causes a gain in the off-axis scaled focus score and a loss in the on-axis scaled score, wherein the ratio of the off-axis score gain to the on-axis score loss falls below a configurable threshold. The lens stack array is then re-aligned (1 134) with this computed plane.

[0075] The efficacy of the process is verified 1 136. This may be accomplished by, for example, having each active camera record an image of the known target, determining an MTF score for each ROI within that image, and ensuring that each MTF score surpasses some threshold calculation.

[0076] The processes described may be iterated (1 138) until a desired configuration is achieved. The lens stack array may be rotated (1 140) relative to the sensor to increase sampling diversity.

[0077] Although a particular process, and its variants, is discussed above, any number of processes may be used to achieve an array camera module that is capable of capturing and recording images that have adequate on-axis and off-axis performance in accordance with embodiments of the invention. Moreover, although the discussed process regards adequately balancing on-axis and off-axis performance of an array camera module, active alignment processes can be tailored to achieve any number of desirable picture characteristics in accordance with embodiments of the invention. Furthermore, active alignment processes, including any of the above-described active alignment processes, may be used to compute configuration parameters to be employed by passive alignment processes, and this concept is discussed below.

Incorporating Passive Alignment in the Bulk Manufacture of Array Camera Modules

[0078] Passive alignment processes in accordance with embodiments of the invention may be utilized in the bulk manufacture of array camera modules based on configuration parameters derived from alignment information obtained from one or more similar array camera modules manufactured using active alignment processes. Specifically, in numerous embodiments, alignment information is obtained by actively aligning one or more representative lens stack array(s) and sensor(s). The alignment information can then be used to derive configuration parameters that are utilized in the passive alignment of a plurality of lens stack arrays and sensors that are similar to the representative lens stack(s) and sensor(s). Lens stack arrays and sensors that are fornned on the same respective wafers, or alternatively are formed in the same positions on different spacers, may be sufficiently similar such that an alignment configuration determined during the active alignment of the representative lens stack array and the imager array may be similarly effective across the remaining lens stack arrays and sensors. Alternatively, if the representative lens stack array and sensor deviate from the remaining lens stack arrays and imager arrays in a known manner, the passive alignment parameters may be developed accordingly in view of the known extent of the deviation.

[0079] A process that aligns a plurality of lens stack arrays and sensors by actively aligning a representative lens stack array and a representative sensor, recording data characterizing the active alignment, computing configuration parameters for the passive alignment of the remaining lens stack arrays and sensors using the recorded data, and passively aligning the lens stack arrays and sensors using the computed parameters is illustrated in FIG. 12.

[0080] A representative lens stack array is actively aligned (1210) with a corresponding representative sensor that has multiple focal planes. Any active alignment process may be employed, including (but not limited to) any of the above- described active alignment processes. In many embodiments, the representative lens stack arrays and sensors are found to be sufficiently similar to their respective constituents, such that it can reasonably be expected that the final configuration achieved by actively aligning the representative lens stack array and the sensor may reasonably be expected to apply just as effectively to the remaining lens stack arrays and sensors. For example, lens stack arrays and sensors formed on the same respective wafers, may be sufficiently similar such that a configuration derived for one pair, can be expected to be similarly effective for the remaining pairs.

[0081] Data characterizing the active alignment is recorded (1220). For example, the final spatial arrangement of the lens stack array relative to the sensor, which (if any) cameras were deactivated, and the results of any through focus sweeps may be recorded. The overall performance of the actively aligned lens stack array and sensor may also be recorded, and may serve as a performance benchmark. The recording may be conducted in any suitable fashion. For example, the machine conducting the active alignment can record the data. Alternatively, any other such machine capable of recording the characterization data can do so.

[0082] Based on this recorded data, configuration parameters are computed (1230) for the passive alignment of the remaining lens stack arrays and sensors. In many embodiments, the passive alignment configuration parameters are computed so as to replicate the final configuration of the actively aligned representative lens stack array and sensor. The configuration parameters may include parameters relating to the spatial arrangement of the lens stack array relative to the sensor, including any rotation of the lens stack array relative to the sensor to increase sampling diversity, and may also include parameters related to the deactivation of cameras.

[0083] In a number of embodiments, the performance of the actively aligned camera is assessed in view of the through focus sweeps, which contain information regarding the best case performance of the cameras. This comparison may inform the extent of the impact that any adhesive curing processes used to affix the spatial relationship of lens stack arrays and sensors may have on the array camera's final imaging abilities. Accordingly, passive alignment configuration parameters may be developed to mitigate any anticipated adverse consequences of such adhesive procedures for affixing the final spatial relationship. For example, the configuration parameters may be developed so as to call for spacers of a greater or lesser thickness, based upon the impact of the adhesive curing process.

[0084] In various embodiments, the computation of the configuration parameters comprises using an optics measuring tool (such as a measuring tool manufactured by TriOptics Optical Test Instrument) to measure the (average) back focal length, or any other parameters, of the various lens stacks in the to-be-actively aligned lens stack array before it is aligned to the sensor, and relate these measurements to the spatial arrangement of the actively aligned lens stack array and sensor. In several embodiments, the measurements are performed during and/or after alignment. Then, with these relationships known, the optics measuring tool may be used to measure the back focal lengths, or any other measurable parameters, in the remaining lens stack arrays; other measurement devices may be applied to determine any mechanical variations in the packages of the sensor arrays that affect e.g. the focusing quality (cover glass thickness, air gap). A desired spatial arrangement for these remaining lens stack arrays and sensors can be computed using the known relationships, and configuration parameters for them can be computed accordingly. For example, in one instance, the distance between each lens stack and sensor within an actively aligned array camera module is measured and related to the respective lens stack's back focal length. A relationship may then be established between a given lens stack's back focal length and a preferable distance between the lens stack array and a sensor. Then, the back focal lengths of remaining lens stack arrays may be measured, and the lens stack array may be passively aligned in view of the relationship between a back focal length and a preferred distance. These embodiments are advantageous insofar as they do not necessarily rely on the similarity of the lens stack arrays in passively aligning the remaining lens stack arrays and sensors. Instead, any differences in back focal length distributions within the plurality of lens stack arrays are measured, and configuration parameters are developed accordingly to accommodate these differences. Additionally, in numerous embodiments, computing the passive alignment configuration parameters also accounts for through focus curves insofar as the through focus curves may be indicative of the sensitivity of a camera's performance to its spatial relationship. Hence, the through focus curves may be used to determine the tolerances within which the spacers and bond lines used in the passive alignment processes should be implemented. Moreover, in cases where the back focal lengths are measured and accounted for in computing the passive alignment configuration parameters, the data contained in the through focus curves may assist in computing the passive configuration parameters as they characterize sensitivity of the cameras to the spatial relationship of their respective lens stack array relative to their respective focal plane. Thus, the through focus curves may be indicative of the sensitivity of a camera's performance to its spatial relationship. Of course, the data contained in the through focus curves can be used in a variety of ways to facilitate the development of passive alignment configuration parameters in accordance with embodiments of the invention.

[0085] The configuration parameters may also specify how to achieve the desired spatial relationship. For example, the following thicknesses within a lens stack/sensor combination may be manipulated to achieve a desired spatial relationship between a lens stack and sensor; the spacer between the lens stack and the cover glass that shields the sensor, the cover glass thickness, the air gap between the cover glass and the sensor, and any spacer beads/adhesive bond lines used in affixing the configuration. The developed configuration parameters can specify the desired respective thicknesses of these parameters to achieve the desired spatial relationship. Note that, in many instances, an actively aligned lens stack array and sensor may use a small spacer to separate the lens stack from the sensor cover glass. However, when passively aligning lens stack, it may be desirable to have a relatively larger spacer separating the lens stack from the cover glass. Accordingly, the configuration parameters may be developed to achieve a desired spatial relationship while operating within the constraint of having the spacer thickness between the lens stack and the cover glass be constrained to within a particular range. Of course, this principal can be applied more generally in that a desired spatial relationship can be attained in any manner, e.g. manipulating any dimensions within a lens stack/sensor combination, in accordance with embodiments of the invention.

[0086] The thickness of the elements within the actively aligned lens stack array and sensor may be used in developing these configuration parameters. For example, a cross-section of the actively aligned lens stack array and sensor may be obtained, and the thickness of the relevant components can be measured. Alternatively, in the case where the thickness of the relevant components are all known, except that the adhesive bond line thickness is not known, this thickness may be obtained during the active alignment process by driving the lens stack against the sensor such that it mechanically contacts it, and subsequently repositioning the lens stack in its desired position; the distance between the point where the lens stack is contacting the sensor and its desired position can be approximated as the thickness of the adhesive bond gap. Passively aligned lens stack arrays and sensors may also be cross-sectioned so that the thickness of their elements can be measured. In particular, those passively aligned lens stack arrays and sensors that are exhibiting the best performance traits may be cross- sectioned, so that the thickness of their components may be determined. Accordingly, configuration parameters may be determined in view of these thickness measurements. [0087] With the configuration parameters developed, the remaining lens stack arrays and sensors may be passively aligned (1240) to form array camera modules. This passive alignment process may involve machining spacers and/or employing appropriately sized spacer beads in the adhesive to spatially orient the lens stack arrays and sensors in a desired manner, and may further involve deactivating specified cameras. For example, specific cameras may be deactivated if the final spatial arrangement is such that those cameras deviate in their performance abilities from remaining within the configuration. More generally, any of the above-described principles regarding enhancing the quality of the configuration (e.g. deactivating specific cameras, and augmenting the spatial relationship between the lens stack array and sensor) may be incorporated in passive alignment processes.

[0088] Of course, the passively aligned array camera modules may be assessed to determine to what extent their performance abilities are sufficient in view of the imaging abilities of the actively aligned array camera module and/or in view of the through focus curves obtained during the active alignment process. Additionally, as before, the performance of the passively aligned array camera modules may be evaluated in view of the through focus curves, and this comparison may again inform the extent of the impact on the spatial arrangement that the adhesive curing processes may cause. Accordingly, the configuration parameters may be augmented in view of this information.

[0089] For example, in many embodiments, the actively and passively aligned array camera modules are assessed relative to the through focus curves, and this assessment is used to augment further alignments. As is evident from the above discussion, the z-positioning of the lens stack array relative to the sensor is a key inquiry in the alignment process, and the through focus curves obtained during the active alignment process can reveal how camera performance varies as a function of relative distance between the lens stack array and the sensor. Hence, the through focus curves obtained during an active alignment can be used to predict the performance of the lens stack array and the sensor as actively aligned. When the representative lens stack array and sensor are actively aligned, the performance of the resulting array camera module can be compared against the predicted performance determined by evaluating the through focus curves, and any discrepancy between the predicted performance and the actual performance of the actively aligned lens stack array and sensor can be attributed to the impact that curing/adhesive processing (and/or any other finishing processing) had on the actively aligned lens stack array and sensor. Thus, for instance, MTF scores for the actively aligned camera may be obtained, and compared against the through focus curve, to infer the actual final z-positioning of the lens stack array to sensor. It can then reasonably be assumed to the extent that the actual final z- positioning of the lens stack array relative to the sensor deviated from the targeted z- positioning, this deviation is a function of the impact curing/adhesive processing, and/or any other finishing processing. Hence, the impact of these finishing processes becomes can be inferred, and further alignment processes can be augmented (e.g. adjusting spacer bead size) to compensate for this determined impact.

[0090] Similarly, passively aligned lens stack arrays and sensors may also be evaluated against the through focus curves obtained during the active alignment of the representative lens stack array and sensor to again gauge what the impact any finishing processes may have on the desired z-distance between the lens stack array and the sensor. Thereafter, further alignment processes can be also augmented in view of this determined impact. Of course, any measurement information obtained from optics measuring tools may also be used in augmenting the alignment procedures. Additionally, as mentioned before, the performance of passively aligned lens stack arrays may be assessed against through focus curves obtained from active alignment; in this way, the efficacy of the passive alignment processes can be gauged.

[0091] Although the present invention has been described in certain specific aspects, many additional modifications and variations would be apparent to those skilled in the art. It is therefore to be understood that the present invention may be practiced otherwise than specifically described. Thus, embodiments of the present invention should be considered in all respects as illustrative and not restrictive.

Claims

WHAT IS CLAIMED IS:

1 . A method for actively aligning a lens stack array with a sensor that includes a plurality of focal planes, where each focal plane comprises a plurality of rows of pixels that also form a plurality of columns of pixels and each focal plane is contained within a region of the imager array that does not contain pixels from another focal plane, the method comprising:

aligning the lens stack array relative to the sensor in an initial position, where the lens stack array comprises a plurality of lens stacks and the plurality of lens stacks forms separate optical channels for each focal plane in the sensor;

varying the spatial relationship between the lens stack array and the sensor so as to create at least two optical channels that each reflect a different central viewing direction relative to one-another and that can allow sub-pixel shifted images to be captured by their respective focal planes;

capturing images of a known target using a plurality of active focal planes at different spatial relationships between the lens stack array and the sensor;

evaluating to what extent the captured images reflect a variety of sub-pixel shifts for respective spatial relationships;

selecting a spatial relationship between the lens stack array and the sensor that can provide a desired extent of variety of sub-pixel shifted images; and

affixing the lens stack array and the sensor in the selected spatial relationship.