Array camera systems and methods

Abstract

An array camera system comprises plural imaging modules, each including a lens and an image sensor, fixedly joined together by a housing structure. The imaging modules include local memory for storing multiple frames of pixel imagery and processing circuitry with convolution capabilities that apply filtering functions to pixel data blocks, generating compressed filter result data alongside stored pixel values. The system employs imaging modules with different focal lengths, sensor dimensions, and filter arrays, with some modules oriented at non-uniformly spaced azimuthal and polar angles optimized for subject sampling resolution across varying scene distances. Integrated imaging modules feature permanently affixed lenses with voice coil actuators enabling rapid focus adjustment at frequencies exceeding 100 Hz for temporal focal scanning and depth capture. The system supports real-time video production with dual memory architecture storing both uncompressed pixel data for instant replay and compressed filter data for long-term storage, enabling virtual camera control and customized viewing experiences for stadium suites and remote viewers.

Description

ARRAY CAMERA SYSTEMS AND METHODSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This patent application claims priority to US Provisional Applications 63 / 740,985, filed December 31, 2024, 63 / 761,969 filed February 22, 2025, 63 / 878,352, filed September 9, 2025, 63 / 880,317, filed September 11, 2025, 63 / 917,971, filed November 14, 2025, and 63 / 920,260, filed November 18, 2025. This patent application also claims priority to US Application 19 / 234,706, filed June 11, 2025. This patent application expands on, and incorporates-by-reference, the work disclosed by applicant in the above priority applications and the following patent publications and applications: US12047692, WO2024147826, 19 / 013,418, filed January 8, 2025 (published as US20250234104), and 63 / 853,933, filed July 30, 2025.BACKGROUND

[0002] Array camera systems are known in the prior art and are detailed in documents cited herein.Exemplary forms of array camera systems are shown in Figs. 1-3.

[0003] Sports broadcasting and video production rely on multiple individual camera operators positioned at various locations around a venue, each operating gimballed, zoom-capable cameras and attempting to follow instructions relayed from remote producers about desired views to capture. This approach requires substantial labor costs, coordination overhead, and physical infrastructure for camera positioning. Moreover, conventional multi -camera installations provide limited flexibility for capturing simultaneous alternative views or retrospective views of events that were not prioritized during live broadcast.

[0004] Array camera systems employ imaging modules with uniform angular spacing in both azimuthal and polar dimensions. For example, modules within a row may be oriented at azimuths progressively incremented by a fixed angle such as -20°, -10°, 0°, 10°, and 20° relative to a reference direction, while different rows are splayed at uniformly-spaced polar angles such as 10°, 0°, and -10° relative to a reference plane. This uniform spacing approach fails to optimize coverage efficiency for scenes where subject distances vary significantly across the field of view. When imaging modules are positioned at varying distances from different parts of a scene, uniform angular spacing results in either over-sampling of near regions with excessive module count or under-sampling of distant regions with insufficient resolution. Conventional systems also typically employ imaging modules with identical or very similar optical characteristics, including common focal lengths, sensor dimensions, and spectral filtering properties. This uniformity limits the system's ability to efficiently allocate imaging resources across scene regions with different capture requirements, such as areas requiring high-speed capture of fast-moving subjects versus areas where lower frame rates suffice for contextual coverage.

[0005] Array camera systems capturing high-resolution imagery at high frame rates generate massive volumes of data. For example, a system with fifty imaging modules, each having a ten-megapixel sensor capturing frames at 200 frames per second, produces over one terabit of data every second. This data volume creates severe bottlenecks in storage, transmission, and processing. Existing approaches to managing this data involve trade-offs between compression and latency. Uncompressed storage enables instant replay without decompression delays but is impractical for long-term archival due to storage capacity requirements. Compressed storage reduces storage requirements but introduces latency in compression and decompression operations that can be problematic for instant replay applications requiring near-instantaneous access to recent video.

[0006] Furthermore, conventional broadcast systems provide limited flexibility for individual viewers to control their viewing experience. In stadium suites and other premium viewing environments, viewers typically receive the same broadcast feed selected by producers, without ability to independently select different views, zoom levels, or subjects of interest from the comprehensive scene coverage that could be available from a multi -camera installation.

[0007] There remains a need in the art for array camera systems that optimize imaging resource allocation through non-uniform angular spacing matched to scene geometry and subject distances, that efficiently manage the massive data volumes through intelligent storage architectures balancing instant replay requirements with long-term storage economy, and that enable flexible virtual camera control allowing multiple concurrent viewers to access personalized views from comprehensive fixed-camera scene capture.SUMMARY

[0008] According to an aspect of the invention, an array camera system comprises plural imaging modules, each including a lens and an image sensor, the plural imaging modules being fixedly joined together by a structure including a housing, at least two of the plural imaging modules being identical in factors including lens focal length, focus distance, color filter array, and image sensor dimensions, and at least two of the plural imaging modules being different in at least one of said factors, each of said plural imaging modules including a local memory operative to store pixel values for more than one frame of imagery captured by the image sensor, each of said plural imaging modules further including processing circuitry operative to process pixel values representing imagery captured by the image sensor, the processing circuitry including convolution circuitry that is operative to convolve sets of said pixel values with each of plural filter kernels, each of said filter kernels comprising an array of integer values, the processing circuitry outputting plural filter result data for storage in a memory, wherein imagery captured by each imaging module is characterized by both stored pixel values and stored plural filter result data. This configuration enables efficient data compression while maintaining access to both compressed and uncompressed imagery, supporting both instant replay capabilities and long-term storage economy in array camera applications.

[0009] According to an embodiment, the array camera system further includes a central processor that is coupled to provide access to both said stored pixel values and said stored plural filter result data. This centralized access enables coordinated processing and distribution of both compressed and uncompressed video data across multiple imaging modules.

[0010] According to an embodiment, the array camera system further includes a memory that stores information from the plural imaging modules, including a first memory part that stores pixel values from the plural imaging modules, and a second memory part that stores filter result data from the plural imaging modules. This dual memory architecture balances the competing needs of quickly- available replays and economy in storage costs by maintaining both lightweight uncompressed data for instant access and compressed data for long-term storage.

[0011] According to an embodiment, for at least one of the plural imaging modules, the convolution circuitry is operative to convolve a 3D set of pixel values, including pixel values from two or more different frames of imagery captured by the first imaging module image sensor, with a filter kernel comprising a 3D array of integer values. This temporal analysis capability enables compression of video data across both spatial and temporal dimensions, achieving higher compression ratios while preserving motion information critical for sports video applications.

[0012] According to an embodiment, the array camera system further includes an image synthesis module operative to recreate image data using the filter result values, and an error detection module operative to assess a difference between the recreated image data and counterpart pixel data. This feedback mechanism enables quality assessment of the compression-decompression process, ensuring that compressed data maintains sufficient fidelity for reconstruction of imagery when needed.

[0013] According to an embodiment, the array camera system further includes a feedback module operative to change one or more of said filter kernels in response to said assessed difference. This adaptive capability allows the system to optimize filter functions dynamically based on actual scene content, improving compression efficiency and reconstruction accuracy overtime.

[0014] According to an embodiment, the array camera system includes imaging modules A and B that are identical in factors of lens focal length, focus distance, color filter array, and image sensor dimensions, the convolution circuitry of imaging module A being operative to convolve imaging module A pixel values with a first filter kernel comprising a first array of integer values, and with a second filter kernel comprising a second, different, array of integer values, and the convolution circuitry of imaging module B being operative to convolve imaging module B pixel values with said first filter kernel and with said second filter kernel, wherein common filter kernels are applied to pixel values in both imaging modules A and B, the system further including first and second parallel memories, the first parallel memory storing filter result data corresponding to convolution of imaging module A pixel values with the first filter kernel, and storing filter result data corresponding to convolution of imaging module B pixel values with the first filter kernel, the second parallel memory storing filter result data corresponding to convolution of imaging module A pixel values with the second filter kernel, and storing filter result data corresponding to convolution of imaging module Bpixel values with the second filter kernel, wherein filter result data from both imaging modules A and B are stored in both the first and second parallel memories. This distributed memory architecture alleviates bottlenecks in storage and retrieval by enabling simultaneous access to data from multiple imaging modules across multiple independent memory units.

[0015] According to an embodiment, the array camera system includes at least three imaging modules A, B and C that are identical in factors of lens focal length, focus distance, color filter array, and image sensor dimensions, each of said at least three imaging modules A, B and C having a common modular form that includes a front surface parallel to the imaging module image sensor, coupled to at least first and second side structures, at least one of said side structures defining a coupling angle of less than 90 degrees relative to the front surface, wherein one of the side structures of imaging module A is coupled to a first of said side structures of imaging module B to thereby cause lens axes of imaging modules A and B to diverge by K degrees, and a second of said side structures of imaging module B is coupled to one of the side structures of imaging module C to thereby cause lens axes of imaging modules B and C to diverge by K degrees, said divergences of K degrees being established, at least in part, by said coupling angle of less than 90 degrees. This modular coupling approach enables precise angular positioning of imaging modules through mechanical design rather than individual adjustment, simplifying assembly while ensuring accurate angular spacing.

[0016] According to an aspect of the invention, an array camera system comprises N imaging modules where N is at least 4, fixedly joined together by a structure including a housing, each imaging module including a lens and an image sensor, each imaging module being oriented in an imaging direction characterized by an azimuthal angle cp relative to a reference direction, the N imaging modules being oriented at N different azimuthal angles that collectively form a progressively-ordered set S = {cpi, q>2, q>3, ... q>n} where cpi>c 2>c 3> ... >cpn, said azimuthal angles having differences therebetween that progressively change in magnitude, so that Ai>A2>...>An-i. This non-uniform angular spacing optimizes field of view coverage by matching angular separation between modules to their respective angular fields of view, which vary based on focal length and distance to imaged subjects, enabling comprehensive coverage of playing fields with consistent subject sampling resolution.

[0017] According to an aspect of the invention, a sports stadium is equipped with an array camera system that views a playing field, the array camera system comprising plural imaging modules, each including a lens and an image sensor, the plural imaging modules being fixedly joined together by a structure including a housing, the stadium including a computer system that stores imagery from the array camera system, the stadium further including first and second suites, each equipped with a respective viewing screen and a user interface coupled to the computer system, the user interface in the first suite enabling users to select first array camera system imagery for display on the viewing screen in the first suite, and the user interface in the second suite enabling users to select second array camera system imagery for display on the viewing screen in the second suite, the selected first and second imagery being different. This configuration enables personalized viewing experiences fordifferent stadium locations, allowing each suite to access customized views of the sporting event independently while drawing from the same comprehensive video capture system.

[0018] According to an embodiment, the user interface in the first suite enables users to select imagery depicting an area on the playing field, or a player on the playing field, for display on the screen in the first suite, to the exclusion of imagery depicting another area or player captured by the array camera system at a same time as said selected imagery depicting the selected area or player. This selective viewing capability allows viewers to focus on specific regions or players of interest while the array camera system simultaneously captures the entire field, providing individualized viewing experiences from comprehensive field coverage.

[0019] According to an embodiment, the array camera system comprises N imaging modules where N is at least 4, each imaging module being oriented in an imaging direction characterized by an azimuthal angle cp relative to a reference direction, the N imaging modules being oriented at N different azimuthal angles that collectively form a progressively-ordered set S = {cpi, q>2, q>3, ... q>n} where (pi>( 2>q>3> ... >cpn, said azimuthal angles having differences therebetween that progressively change in magnitude, so that Ai>A2>...>An-i. This non-uniform angular spacing ensures optimal coverage of the playing field visible from the stadium installation, matching module angular separation to varying subject distances and maintaining consistent image quality across the entire captured scene.

[0020] According to an embodiment, the array camera system includes imaging modules having azimuthal angles q>i>q>2>q>3>q>4>q>5, said azimuthal angles having differences therebetween characterized in that AI>A2>A3>A4. This specific five-module angular progression provides systematic coverage of extended playing field areas while maintaining the benefits of non-uniform spacing for consistent subject resolution.

[0021] According to an embodiment, the array camera system includes imaging modules having azimuthal angles q>i>q>2>q)3>q)4>q)5>q)6, said azimuthal angles having differences therebetween characterized in that AI>A2>A3>A4>A5. This six-module configuration extends the progressive angular spacing principle to provide even more comprehensive field coverage while maintaining optimal subject sampling resolution across all imaged regions.

[0022] According to an embodiment, the array camera system includes imaging modules having three, four, or more different focal lengths. This focal length diversity enables the system to optimize each module for its specific viewing distance and angular coverage requirements, balancing field of view breadth with subject resolution across the composite captured scene.

[0023] According to an embodiment, the array camera system includes imaging modules having three, four, or more different image sensor dimensions. This sensor size diversity provides additional flexibility in optimizing each module's capture characteristics for its specific role within the overall system architecture.

[0024] According to an embodiment, each imaging module is oriented in an imaging direction that is further characterized by a polar angle 0 relative to a reference plane, and all of said N modules are oriented in an imaging direction characterized by a common polar angle. This uniform polarorientation simplifies system design and calibration when vertical angular variation is not required for the application.

[0025] According to an embodiment, each imaging module is oriented in an imaging direction that is further characterized by a polar angle 0 relative to a reference plane, the array camera system including at least three imaging modules having respective polar angles 0i>02>03, said polar angles having differences therebetween where Aa>Ap. This non-uniform polar spacing optimizes vertical coverage of scenes such as elevated sports fields, matching vertical angular separation to varying subject distances and module focal lengths.

[0026] According to an aspect of the invention, an array camera system comprises plural imaging modules, each including a lens and an image sensor, the plural imaging modules being fixedly joined together by a structure including a housing, each of the imaging modules being oriented in an imaging direction characterized by an azimuthal angle cp relative to a reference direction, the plural imaging modules being oriented at F >3 different azimuthal angles that collectively form a progressively-ordered successive set of azimuthal angles SF = {cpi, q>2, ... c F} where (pi>q>2>... >q>F, wherein a first pair of the successive azimuthal angles in set SF differ by a first delta-azimuth value AA1, and a second pair of the successive azimuthal angles in set SF differ by a second delta-azimuth value AA2 greater than the first delta-azimuth value. This progressive variation in angular spacing enables efficient coverage of scenes where subject distances vary across the field of view, optimizing the trade-off between number of modules required and consistent subject sampling resolution.

[0027] According to an embodiment, the first and second delta-azimuth values have values where 5° < AA1 < 10° and 10° < AA2 < 25°. These specific angular ranges provide practical spacing values suitable for typical sports venue installations while maintaining the benefits of progressive angular variation.

[0028] According to an embodiment, a third pair of the successive azimuthal angles in set SF differ by a third delta-azimuth value AA3 greater than the second delta-azimuth value. This extension of the progressive spacing principle to additional modules enables coverage of larger fields or more complex venue geometries while maintaining optimal subject resolution characteristics.

[0029] According to an embodiment, the first, second and third delta-azimuth values have values where 5° < AA1 < 10°, 10° < AA2 < 14°, and 14° < AA3 < 25°. These refined angular ranges provide more specific guidance for four-module progressive spacing configurations in practical implementations.

[0030] According to an embodiment, a fourth pair of the successive azimuthal angles in set SF differ by a fourth delta-azimuth value AA4 greater than the third delta-azimuth value. This further extension accommodates five-module configurations with continued progressive angular spacing optimization.

[0031] According to an embodiment, the first, second, third and fourth delta-azimuth values have values where 5° < AA1 < 9°, 9° < AA2 < 12°, 12° < AA3 < 15°, and 15° < AA4 < 25°. These tightly specified angular ranges provide precise design parameters for five-module progressive spacing implementations, ensuring optimal field coverage with consistent subject resolution.

[0032] According to an embodiment, the imaging directions of the plural imaging modules are further characterized by G > 3 different polar angles that collectively form a progressively-ordered successive set of polar angles SG = {0i,02, ...0G} where 0i>02>... OG, wherein a first pair of the successive polar angles in set SG differ by a first delta-polar value API, and a second pair of the successive polar angles in set SG differ by a second delta-polar value AP2 greater than the first deltapolar value. This progressive polar spacing complements the azimuthal spacing optimization, enabling comprehensive three-dimensional coverage of complex scenes with varying vertical and horizontal subject distances.

[0033] According to an embodiment, the first and second delta-polar values have values where 0° < API < 23° and 23° < AP2 < 70°. These polar angle ranges accommodate typical elevation variations in sports venue installations while maintaining progressive spacing benefits.

[0034] According to an embodiment, a third pair of the successive polar angles in set SP differ by a third delta-polar value AP3 greater than the second delta-polar value. This extension enables four-level vertical coverage with continued progressive optimization of polar angular spacing.

[0035] According to an embodiment, the first, second and third delta-polar values have values where 0° < API < 23°, 23° < AP2 < 31°, and 31° < AP3 < 70°. These refined polar angle specifications provide practical design parameters for multi-level vertical coverage configurations.

[0036] According to an aspect of the invention, an array camera system comprises F>4 imaging modules, each including a lens and an image sensor, the F imaging modules being fixedly joined together by a structure including a housing, the housing positioning entrance apertures of the F imaging module lenses in a shared plane, each of the F imaging modules being oriented in an imaging direction characterized by an angle within said shared plane, wherein angles of said imaging modules are splayed non-uniformly within said shared plane. This non-uniform angular distribution enables optimized coverage of scenes where subject distances and required angular fields of view vary across the composite field of view, improving efficiency of module utilization.

[0037] According to an aspect of the invention, an array camera system for capturing image data depicting a scene includes at least first, second and third imaging modules, each having a lens with a focal length and being characterized by a respective resolution, frame-rate and bit-depth, the first and second imaging modules each having lens focal lengths greater than 40mm and being pointed in different viewing directions to capture first and second fields of view, the third imaging module having a lens focal length of less than 20 mm and having a field of view that spans said first and second fields of view, wherein said first and second fields of view are isolated, such that a line between their centers passes through a scene region that is imaged by the array camera system only at a resolution, frame rate and / or bit-depth that is lower than that of both the first and second imaging modules. This configuration enables efficient allocation of imaging resources by providing high- fidelity capture of isolated regions of particular interest while maintaining lower-fidelity contextual coverage of intervening areas, optimizing the balance between data volume and capture quality for applications such as baseball scouting.

[0038] According to an embodiment, said different viewing directions are characterized by azimuthal angles that differ by between 60 and 75 degrees. This angular separation is particularly suited for baseball diamond geometry, enabling simultaneous high-resolution capture of the pitcher's mound and batter's box from a single fixed camera position.

[0039] According to an embodiment, an analytics system comprises first and second array camera systems positioned at a baseball playing field, the first array camera system being positioned off the playing field along a third base line, and the second array camera being positioned off the playing field along a first base line. This dual-system configuration enables frontal imagery capture of both left-handed and right-handed pitchers and batters during their respective motions, providing comprehensive coverage for baseball analytics applications.

[0040] According to an aspect of the invention, a method comprises analyzing video imagery from each of plural imaging modules in an array camera system to detect imagery depicting one of the following events: a motion of a tennis serve, a motion of a baseball pitch, or a motion of a baseball swing, in response to detection of one of said events in video imagery from a first of said imaging modules, copying a first interval of image data that was previously written from said first imaging module into a circular buffer into long-term storage, and in response to said event detection in video imagery from the first imaging module, also copying a second interval of image data that was previously written from a second imaging module into a circular buffer into long-term storage. This event-triggered storage approach enables efficient capture of relevant high-speed sporting events while avoiding continuous storage of less relevant video data, optimizing storage resource utilization.

[0041] According to an embodiment, the method further includes in response to said event detection in video imagery from the first imaging module, writing a third interval of following video from the first imaging module into long-term storage, and in response to said event detection in video imagery from the first imaging module, also writing a fourth interval of following video from the second imaging module into long-term storage. This forward-looking storage ensures capture of the complete event sequence following the triggering motion, providing comprehensive video clips of sporting events.

[0042] According to an embodiment, the first, second and third intervals are less than five seconds, and the fourth interval is more than five seconds. This differential interval approach recognizes that high- fidelity modules capturing isolated regions require shorter clip durations while wider-angle contextual modules benefit from longer capture intervals to provide complete event context.

[0043] According to an aspect of the invention, a system for distributing video data to M>=1 videoreceiving clients, based on image data depicting a scene produced from N>=1 imagers in a camera system, said image data being stored in a memory, includes a first processor and a memory controller adapted to receive data indicating a client-requested view of a sub-part of the scene, generate corresponding floating point coordinates for each of plural sample points in a camera system frame of reference, identify from the floating point coordinates of each sample point one or more memory addresses, and to fetch image data from said memory addresses, and a second processor adapted to compile an output frame of imagery that depicts the client-requested sub-part of the scene from thefetched image data. This dual-processor architecture enables efficient virtual camera control by separating the floating-point coordinate generation from the integer-based interpolation operations, optimizing processing speed and power consumption while supporting multiple concurrent client video streams.

[0044] According to an embodiment, the received data indicates a client requested view of a sub-part of the scene that is captured by a first of said N imagers, said first imager being configured to output a frame of image data organized as rows and columns of pixels, wherein the received data indicates a center location of said client-requested view within said frame of image data, and the first processor is adapted to generate floating point coordinates identifying plural sample points corresponding to a line that passes through said center location within said frame of image data, wherein said line is tilted relative to said pixel rows of the first imager. This tilted sampling capability enables virtual rotation of output imagery to simulate camera pivoting, correcting for geometric distortions that arise when extracting off-axis views from fixed imager fields of view.

[0045] According to an embodiment, the floating point coordinates correspond to multiple lines through said frame of image data, each of said lines being tilted relative to said pixel rows of the first imager. This multi-line tilting enables complete frame generation with consistent geometric correction across all output rows, producing natural -appearing imagery from off-axis extractions.

[0046] According to an embodiment, M>=2, and the first processor is adapted to generate a first set of floating point coordinates for a first view requested by a first video-requesting client, and is adapted to generate a second set of floating point coordinates for a second, different, view requested by a second video-requesting client. This multi-client capability enables simultaneous service of multiple viewers with different viewing preferences, supporting diverse viewing experiences from a single camera system capture.

[0047] According to an embodiment, the received data indicates that the first client-requested view is at a first resolution and indicates that the second client-requested view is at a second, different, resolution. This resolution flexibility allows the system to optimize bandwidth and processing resources for each client's specific requirements and display capabilities.

[0048] According to an embodiment, the received data indicates that the first client-requested view is at a first frame rate and indicates that the second client-requested view is at a second, different, frame rate. This frame rate flexibility enables efficient resource allocation by providing higher frame rates only to clients requiring such temporal resolution while conserving bandwidth for other clients.

[0049] According to an embodiment, the second processor is adapted to interpolate plural image values fetched from the memory to produce a single output image value in said output frame of imagery. This interpolation capability enables generation of output pixels at arbitrary angular locations not corresponding to sensor pixel positions, providing smooth virtual camera control with sub-pixel positioning accuracy.

[0050] According to an embodiment, the memory controller is adapted to fetch P image values for use by the second processor to produce a first output image value in said output frame of imagery, and tofetch Q image values for use by the second processor to produce a second output image value in said output frame of imagery, where P > Q > 1. This variable kernel size approach optimizes processing efficiency by adapting interpolation complexity to the geometric relationship between output pixel locations and sensor pixel positions, reducing unnecessary computation.

[0051] According to an embodiment, P > Q > 2. This constraint ensures that interpolation operations utilize multiple sensor pixels even in simplified cases, maintaining output image quality while preserving processing economy benefits.

[0052] According to an embodiment, the first processor comprises a field programmable gate array (FPGA) and the second processor comprises a graphics processing unit (GPU). This hardware specialization optimizes system performance by assigning floating-point coordinate calculations to FPGA logic while leveraging GPU capabilities for repeated interpolation kernel operations.

[0053] According to an embodiment, the system includes N>=2 imagers, wherein the memory stores rows of image data from each of the imagers, and image data captured by pixels in a row of a first of the imagers projects to a first linear line of physical locations in x,y,z space, and image data captured by pixels in a row of a second of the imagers projects to a second linear line of physical locations in x,y,z space, where the first and second linear lines of locations in x, y, z space are non-parallel. This geometric relationship arises from different imager orientations and necessitates the tilted sampling approaches described above to produce geometrically consistent output imagery spanning multiple imager fields of view.

[0054] According to an embodiment, the received data indicating said client-requested view of the subpart of the scene indicates pan, tilt and zoom information for the requested view. This parameter set provides an intuitive and complete specification of virtual camera positioning, enabling producers and automated systems to precisely control output view characteristics.

[0055] According to an aspect of the invention, a method comprises using first and second spaced-apart imagers to sense light flashes emitted from light sources affixed to an ice skater skating on ice, including from skates worn by the ice skater, and determining correspondence between pixels in a first frame of imagery captured by the first imager and pixels in a second frame of imagery captured by the second imager, based on locations of the sensed light flashes in the first and second frames of imagery. This light-based correspondence technique enables precise calibration and registration between imagers by providing easily identifiable reference points in captured imagery.

[0056] According to an embodiment, said first and second imagers are mounted in a common housing.This integrated mounting ensures stable geometric relationships between imagers, improving reliability of the correspondence determination process.

[0057] According to an aspect of the invention, an array camera system includes first, second and third camera modules that respectively have first, second and third focal lengths and first, second and third fields of view, where the third focal length is longer than the second focal length and the second focal length is longer than the first focal length, and an area within the third field of view is also within the first field of view and within the second field of view. This nested field of view configurationprovides multiple resolution options for critical scene regions, enabling the system to serve clients requesting different zoom levels from the same scene area without requiring virtual zoom operations.

[0058] According to an embodiment, the third field of view is entirely encompassed within the second field of view. This complete nesting ensures that all high-resolution detail captured by the longest focal length module is also available at intermediate resolution, providing seamless zoom capability across the nested region.

[0059] According to an embodiment, the second field of view is not entirely encompassed within the first field of view. This partial overlap configuration extends intermediate-resolution coverage beyond the wide-angle contextual view, optimizing the balance between comprehensive coverage and targeted high-resolution capture.

[0060] According to an embodiment, at least one of said camera modules has a square field of view. This square configuration reduces lens distortions by limiting the angular extent of light entry into the lens, improving image quality particularly at field edges.

[0061] According to an embodiment, the array camera system includes at least two camera modules having the first focal length, at least two camera modules having the second focal length, and at least two camera modules having the third focal length. This replication of each focal length class provides redundancy and extended coverage, enabling comprehensive scene capture with multiple resolution tiers across larger fields of view.

[0062] According to an aspect of the invention, an array camera system includes at least two camera modules of a first type, at least two camera modules of a second type, and at least two camera modules of a third type, the first, second and third types of camera modules respectively having first, second and third focal lengths, where the third focal length is longer than the second focal length and the second focal length is longer than the first focal length, and at least one area within a field of view of a camera module of the third type is also within a field of view of a camera module of a first type and within a field of view of a camera module of the second type. This multi -type, multi -module configuration provides scalable coverage with multiple resolution tiers, supporting diverse viewing requirements across extended scene areas.

[0063] According to an embodiment, an area within a field of view of a camera module of the third type is within a field of view of no camera module of the second type. This gap in intermediate coverage demonstrates that complete nesting of all focal length tiers is not required, allowing flexible system designs that optimize module allocation based on specific scene characteristics and viewing requirements.

[0064] According to an aspect of the invention, an array camera system comprises plural imaging modules, each including a lens and an image sensor, the plural imaging modules being fixedly joined together by a structure including a housing, wherein the plural imaging modules include a first set of imaging modules having lenses with focal lengths greater than 30 mm, a second set of imaging modules having lenses with focal lengths less than 20 mm, and a third set of imaging modules having lenses with intermediate focal lengths between the first and second sets. This three-tier focal lengtharchitecture enables the array camera to capture both fine distant detail and wide-angle contextual coverage, supporting downstream digital zoom at arbitrary points throughout the depth of field.

[0065] According to an embodiment, the first set of imaging modules have focal lengths greater than 50 mm, and the second set of imaging modules have focal lengths less than 12 mm. These more extreme focal length ranges provide enhanced differentiation between telephoto detail capture and wide-angle contextual coverage capabilities.

[0066] According to an embodiment, the third set of imaging modules have focal lengths in a range selected from: 10-15 mm, 15-22 mm, 22-30 mm, 30-40 mm, or 40-55 mm. These intermediate focal length options provide flexibility in optimizing the middle tier of coverage for specific venue geometries and viewing requirements.

[0067] According to an embodiment, one implementation employs focal lengths of 60 mm for the first set, 24 mm for the third set, and 12 mm for the second set. This specific combination provides a practical balance of telephoto, intermediate, and wide-angle coverage suitable for typical sports venue applications.

[0068] According to an embodiment, another implementation employs focal lengths of 50 mm for the first set, 25 mm for the third set, and 6 mm for the second set. This alternative combination provides similar tiered coverage with different specific focal length values, demonstrating the flexibility of the three-tier approach.

[0069] According to an embodiment, the third set of imaging modules comprises imaging modules having two distinct intermediate focal lengths. This subdivision of the intermediate tier provides additional resolution gradation, enabling finer optimization of coverage across the mid-range distance zones.

[0070] According to an embodiment, one of the two distinct intermediate focal lengths is paired with a color sensor and the other is paired with a monochrome sensor. This sensor type pairing enables optimization of different intermediate modules for color fidelity versus light sensitivity and spatial resolution, providing complementary capabilities within the intermediate focal length tier.

[0071] According to an embodiment, the longer of the two intermediate focal lengths is paired with the color sensor. This pairing prioritizes color accuracy for more distant intermediate -range subjects while accepting lower light sensitivity at those distances.

[0072] According to an embodiment, the longer of the two intermediate focal lengths is paired with the monochrome sensor to maximize low-light sensitivity and spatial acuity. This alternative pairing prioritizes detail capture and low-light performance for more distant intermediate -range subjects, with color information inferred from neighboring color modules.

[0073] According to an embodiment, a 25 mm lens is paired with a color sensor and a 12 mm lens is paired with a monochrome sensor. This specific pairing provides a practical implementation of the dual intermediate focal length approach with complementary sensor type optimization.

[0074] According to an embodiment, monochrome sensors provide higher photon efficiency and superior detail rendition compared to color sensors, and color values for imagery captured bymonochrome sensors are inferred from contemporaneously captured color data from neighboring modules. This approach leverages the superior light-gathering and resolution capabilities of monochrome sensors while maintaining color imaging capability through inter-module color transfer.

[0075] According to an embodiment, the array camera system further comprises imaging modules operating at variable frame rates for motion-estimation or depth-fusion. This frame rate diversity enables optimization of temporal sampling for different scene regions and processing purposes, supporting advanced video analysis capabilities.

[0076] According to an embodiment, the array camera system further comprises imaging modules equipped with polarization filters for glare suppression or surface -normal estimation. This polarization capability extends the system's imaging modalities beyond spectral diversity, enabling enhanced scene analysis in challenging lighting conditions.

[0077] According to an embodiment, the array camera system further comprises multispectral or hyperspectral imaging modules for enriched colorimetry or material-reflectance discrimination. This spectral imaging capability provides enhanced color measurement and material identification beyond conventional RGB imaging, supporting advanced quality assurance and scene analysis applications.

[0078] According to an embodiment, the array camera system is configured to achieve full-field capture of sports venues with the ability to digitally zoom into any region of interest at any depth within a reconstructed view volume. This comprehensive capture and flexible extraction capability enables post-production control of viewing perspectives and zoom levels, supporting diverse broadcast and streaming requirements from a single fixed camera installation.

[0079] According to an aspect of the invention, an array camera system comprises plural imaging modules, each including a lens and an image sensor, the plural imaging modules being fixedly joined together by a structure including a housing, wherein the system includes first, second and third camera modules that respectively have first, second and third focal lengths and first, second and third fields of view, where the third focal length is longer than the second focal length and the second focal length is longer than the first focal length, at least two camera modules of each of the first, second and third types, wherein an area within a field of view of a camera module of the third type is also within a field of view of a camera module of the first type and within a field of view of a camera module of the second type, and wherein a region falls within the collective fields of view of camera modules of the third type, but is outside the collective fields of view of camera modules of the second type, creating a gap in coverage by the intermediate focal length modules. This configuration demonstrates that complete continuous coverage by all focal length tiers is not required, enabling efficient module allocation that concentrates intermediate-tier resources around specific scene regions while maintaining telephoto and wide-angle coverage of gap areas.

[0080] According to an aspect of the invention, an array camera system comprises plural imaging modules, each including a lens and an image sensor, the plural imaging modules being fixedly joined together by a structure including a housing, wherein at least one of the imaging modules comprises an integrated imaging module (IIM) that includes a static lens assembly and a lightweight movable lenselement positioned closest to the sensor, the movable lens element being actuated by a voice coil motor capable of rapid focus adjustment at frequencies of 100 Hz or greater, wherein the IIM enables fast, precise, and repeatable focal transitions across the depth of field by separating optical power between the heavier static lens assembly and the low-mass movable element, wherein an array of such IIMs performs temporal focal scanning, sweeping through focal planes of a view volume at high speed to capture depth-indexed image slices. This rapid focus control capability enables time- multiplexed depth capture, allowing the array camera to acquire three-dimensional scene information through sequential focal plane sampling without requiring multiple fixed-focus modules at different focus distances.

[0081] According to an embodiment, the array camera system includes a high-speed mechanism for time-multiplexed depth capture using integrated modules capable of dynamic focus control across successive exposures. This temporal multiplexing approach enables comprehensive depth sampling while minimizing the number of physical imaging modules required, optimizing system cost and complexity.

[0082] According to an embodiment, the voice coil motor enables the movable lens element to be repositioned at rates of several hundred hertz. This high repositioning rate allows multiple focal planes to be sampled within typical video frame intervals, enabling depth capture without sacrificing temporal resolution of the overall video stream.

[0083] According to an aspect of the invention, an array camera system comprises plural imaging modules, each including a lens and an image sensor, the plural imaging modules being fixedly joined together by a structure including a housing, wherein the system employs a heterogeneous combination of fixed-focal-length integrated imaging modules (IIMs) and variable-focal-length IIMs with high-speed focus control to optimize sampling of a 3D optical data cube. This hybrid architecture combines the simplicity and image quality of fixed-focus modules with the depthsampling capabilities of variable-focus modules, enabling efficient acquisition of both spatial and depth information throughout the view volume.

[0084] According to an embodiment, modules capable of rapid focal transitions are integrated alongside modules configured for fixed, high-quality imaging at specific depths or fields of view. This integration enables the system to allocate expensive variable -focus capability only where depth sampling is required while using simpler fixed-focus modules for regions where depth information is less critical.

[0085] According to an embodiment, the array camera system is configured to acquire depth-resolved image data throughout a view volume. This depth resolution capability supports advanced applications including three-dimensional scene reconstruction, depth-based video effects, and computational refocusing of captured imagery.

[0086] Further inventive features will become apparent with reference to the following detailed description and accompanying drawings.BRIEF DESCRIPTION OF DRAWINGS

[0087] Figure 1 shows a prior art array camera system with multiple imaging modules arranged in rows within an elongated housing.

[0088] Figure 2 shows a prior art array camera system with imaging modules arranged in a curved configuration.

[0089] Figure 3 shows prior art array camera configurations with different numbers of imaging modules and varying fields of view.

[0090] Figure 4 shows a block diagram of an array camera system including plural imaging modules connected to a central unit with processing and memory components.

[0091] Figure 5 shows optical elements of an illustrative lens system for an imaging module.

[0092] Figure 6 shows a prior art arrangement of imaging modules with uniform angular spacing in azimuthal direction.

[0093] Figure 7 shows a coordinate system defining azimuthal and polar angles for characterizing imaging module orientations.

[0094] Figure 8 shows a top-down view of a sports field indicating potential array camera system placement locations.

[0095] Figure 9 shows a top-down schematic view of an array camera system with eight imaging modules oriented at non-uniformly spaced azimuthal angles.

[0096] Figure 10 shows a top-down view of a sports field with an array camera system positioned at a comer, illustrating fields of view of imaging modules with progressively varying azimuthal spacing.

[0097] Figure 11 shows a top-down view of a sports field with an array camera system positioned at the center of a short edge, illustrating reversal in azimuthal angle progression.

[0098] Figure 12 shows a side view of an array camera system elevated on a pole with imaging modules oriented at different polar angles.

[0099] Figure 13 shows polar angle orientations of imaging modules in an array camera system with non-uniform polar spacing.

[0100] Figure 14 shows a front view of an array camera system with twenty-two imaging modules arranged in four rows at different polar angles.

[0101] Figure 15 shows an alternative spatial arrangement of imaging modules with the same angular orientations as those in Figure 9.

[0102] Figure 16 shows a schematic diagram of a distributed memory architecture for storing pixel data and compressed data from multiple imaging modules.

[0103] Figure 17 shows a composite view from an array camera system with a cross-hatched region indicating areas for high frame rate storage.

[0104] Figure 18 shows a mask defining regions of an image for differential frame rate storage based on content importance.

[0105] Figure 19 shows a baseball diamond layout with indicated array camera system placement positions for capturing pitcher and batter imagery.

[0106] Figure 20 shows a perspective view of an array camera system with multiple imaging modules including both narrow field and wide field modules.

[0107] Figure 21 shows a diagram illustrating effective focal length definition using subject distance and projected image size relationships.

[0108] Figure 22 shows a conceptual illustration of fields of view from seven imaging sensors in a camera head arranged in two rows.

[0109] Figure 23 shows memory allocation for pixel data from multiple imaging sensors in an aggregate scene memory.

[0110] Figure 24 shows composite imagery captured by a camera head positioned at a football stadium.

[0111] Figure 25 shows an enlarged view from an imaging sensor with a dotted rectangle indicating a producer-specified view for broadcast.

[0112] Figure 26 shows a block diagram of a dual-processor system for generating output video frames from aggregate scene memory.

[0113] Figure 27 shows sensor pixel locations with interpolation kernels of different shapes for generating output pixels at various positions.

[0114] Figure 28 shows a timing diagram illustrating frame rate conversion between sensor frame capture and output frame delivery.

[0115] Figure 29 shows a detailed system diagram for processing multiple client view requests from aggregate scene memory using cache memory.

[0116] Figure 30 shows a diagram of stair-stepped elongated horizontal pixel blocks forming ribbons for efficient memory block reading.

[0117] Figure 31A shows a client-requested view spanning imagery from two imaging sensors with different field orientations.

[0118] Figure 3 IB shows tilted output pixel rows relative to sensor pixel rows for generating output imagery from a left-oriented imager.

[0119] Figure 31C shows parallel output pixel rows relative to sensor pixel rows for generating output imagery from a center-oriented imager.

[0120] Figure 32 shows imagery from an imager with a boresight oriented above a playing field with action visible in the lower left quadrant.

[0121] Figure 33 shows a client-requested closeup view with output rows tilted relative to sensor rows to simulate camera pivoting.

[0122] Figure 34 shows a system diagram with pyramidal encoding module producing multiple chroma subsampling formats for storage in aggregate scene memory.

[0123] Figure 35 shows fields of view of eight imaging modules in a baseball-oriented camera head with three focal length classes.

[0124] Figure 36 shows fields of view of imaging modules in a basketball-oriented camera head with three focal length classes and a gap in intermediate coverage.

[0125] Figure 37 shows fields of view of intermediate focal length modules highlighting a central gap in collective coverage.

[0126] Figure 38 shows a region of overlap between fields of view of three module types forming a closed connected region surrounding a coverage gap.

[0127] Figure 39 shows an exploded perspective view of an integrated imaging module illustrating the assembly sequence and component relationships including static lens assembly, movable lens element, voice coil motor, VCM mount, and sensor board.

[0128] Figure 40 shows a system architecture diagram of a camera head with sensor interfaces, backplane, FPGA SOM, and connections to rendering and storage systems.

[0129] Figure 41 shows a system architecture diagram with four sensor interfaces connected to a backplane with FPGA SOM, DDR memory, and solid state drives.

[0130] Figure 42 shows a system architecture diagram with four sensor interfaces, compressors, backplane with FPGA SOM, DDR memory, H265 compressors, and solid state drives.DETAILED DESCRIPTION OF THE INVENTION

[0131] Fig. 4 shows one embodiment of an array camera system 10 incorporating aspects of the present technology. The illustrated system includes plural imaging modules 12 connected to a central unit 14. In typical use, the central unit provides data, e.g., over fiber optic cable 16, to a remote video production system 18. For a professional soccer or football game, the remote video production system may be in a trailer and linked to several such array camera systems positioned around the playing field or arena.

[0132] Each imaging module in the depicted array camera system 10 includes a lens system 20, an image sensor 22, and a module processor 24. The lens system 20 includes one or more glass or plastic optical elements — most having at least one curved (often aspherical) surface designed to focus light in a desired manner. The lens system may include further elements (flat or curved) that attenuate infrared light or define an aperture stop. We term the lens element closest to the image sensor the backmost element, and the lens element closest to the subject the frontmost element. Optical elements of an illustrative lens system are shown in Fig. 5, taken from patent 10,921,558. (In some embodiments, the distance between the backmost element and the image sensor is greater than 2, 5, or 10mm, but typically less than 30mm.)

[0133] The lens system 20 may include a variable focus mechanism. One such mechanism is a liquid lens arrangement employing an electric field to change shape and curvature of a liquid, altering the lens focus distance. Another mechanism is a motor drive that physically moves one or more lens elements to change focus distance. Such mechanisms can also change lens focal length, yielding telephoto control.

[0134] The lens system 20 further includes a barrel or other support member to which optical elements are affixed.

[0135] The image sensor 22 includes a semiconductor device defining an array of photo detectors and associated circuitry, such as charge storage and readout logic. The semiconductor device is commonly mounted to a packaging substrate, e.g., made of epoxy resin or ceramic, which connects electrical pads of the semiconductor to external circuitry. This packaging substrate may be mounted to a printed circuit board. An optical filter array (or color filter array, CFA), comprising a tiled pattern of optical filters having different transmission characteristics, is often positioned in the optical path before the photodetector array. The image sensor has a top surface nearest the lens, which may be a glass cover slip covering an optical input surface of the semiconductor device or the top surface of a filter array if present.

[0136] In one embodiment, the lens system 20 is secured directly to the image sensor 22, such as by glue. In such cases we term the imaging module an "integrated imaging module." More generally, the lens in an integrated imaging module omits a mount (e.g., a flange) that mechanically mates with a corresponding feature on the packaging substrate or circuit board. Instead, the lens is permanently affixed to one of the components of the image sensor, such as the top surface (e.g., a glass cover slip), the packaging substrate, or the circuit board.

[0137] The plural imaging modules 12 are fixedly joined together by a structure including an exterior housing. Suitable housings are known from the prior art, such as that shown in Fig. 1.

[0138] In one embodiment, at least two of the plural imaging modules are identical in factors including (a) lens focal length; (b) focus distance (for fixed-focus lenses); (c) optical filter array; and (d) image sensor dimensions. In this embodiment, at least two of the plural imaging modules also differ in at least one of these factors. As is familiar to artisans, lens focal length is a measure of how strongly a lens converges light. As used herein, the term specifically refers to what is sometimes termed “effective focal length,” as defined below. Focus distance refers to the distance between the imaging module and the subject plane of focus. This distance may be measured relative to the photodetector plane, the optical lens center, or other fixed reference point in the imaging module. Color filter array refers to the patterned layout of filters in a color filter overlaying the photodetector semiconductor device. The term “sensor dimensions” encompasses the height and width of the photodetector array, as well as its pixel dimensions.

[0139] In one embodiment, the module processor 24 comprises a data analysis stage 26, which examines each of plural sets of pixel data to determine strengths of different included pixel patterns. Each set of pixel data may be analyzed for dozens or hundreds (or more) of patterns. The analysis stage thus produces a set of dozens or hundreds of output values — each corresponding to a different pattern against which the pixel data was tested.

[0140] Such analysis stage can be implemented as a bank of filtering stages or, more commonly, as a neural stage that applies filtering functions by performing convolution operations between a set (e.g., block) of pixel data and different filter kernels. Dedicated convolution circuitry can be employed, or aprocessor can be configured to perform such operations. Each filter kernel commonly comprises an array of integer values. Each filtering operation yields one output datum indicating the degree to which the tested pattern is present in the input pixel data. The pixel data is divided into blocks for analysis, e.g., of size 8x8 pixels, 32x32 pixels, etc. The filter kernels are commonly each of the same size as the pixel blocks.

[0141] In some embodiments, part or all of the module processor is integrated on the same semiconductor substrate as the photodetector array. The module processor can also be implemented separately, such as by a microprocessor or GPU, which may be mounted on a circuit board shared with the image sensor.

[0142] The module processor provides the plural filter result data for storage in a memory, which may be in the imaging module, in the central unit, or in cloud storage. Regardless of where stored, the central unit has access to the stored filter result data from the subject imaging module and from each imaging module in the array camera system.

[0143] In one embodiment, each imaging module includes local cache memory 28 large enough to store more than a single frame of pixel imagery. In particular embodiments, this local cache memory may store two, three, five or ten frames of imagery, or up to five or ten seconds of video data. This cache memory can be implemented, in whole or part, on the semiconductor device that also includes the photodetector array, or not. This cache memory can form part of the module processor or be independent of it.

[0144] Central unit 14 includes a processor 30 and a memory system 32 and is connected to each imaging module by a data bus 34. This bus provides both pixel data from the local cache memory and filter result data from the analysis stage. A first part 32a of the memory system stores pixel values from the plural imaging modules; a second part 32b stores filter result data from the plural imaging modules. Access to both types of data adds to capabilities of the array camera system.

[0145] The first memory part may store pixel data captured by each imaging module within the preceding few (e.g., 1-10) minutes. This allows recent video depicting any part of the composite field of view to be replayed near-instantly (including at slow motion), without latency delays in compressing and decompressing data. This first memory part may take the form of a large circular buffer.

[0146] Long-term storage of such pixel data is usually impractical. Thus, after a minute or few in the first memory part 32a, such pixel data is overwritten with new pixel data. To retrieve older imagery, the system recalls compressed data stored in the second memory part 32b and decompresses it for use. Such operation does not require the near-instantaneous performance required for "instant replay" and can thus tolerate slower latency associated with compression and decompression.

[0147] Thus, by storing two types of video data — one via a lightweight process permitting both storage and client-delivery in milliseconds, and the other requiring more involved processing — the competing needs of quickly-available replays and economy in storage costs can be balanced.

[0148] In one embodiment, one or more imaging modules has a lens focal length exceeding 10 mm, and preferably greater than 20mm or 50mm (e.g., 100mm). By point of reference, mobile phones commonly employ lens focal lengths of 10mm or less.

[0149] It should be emphasized that focal length references herein are "effective" focal lengths and not "35mm equivalent" focal lengths. (Integrated imaging modules having effective focal lengths of 25- 100 mm correspond to 150-800 mm 35mm equivalent focal lengths.)

[0150] One or more lenses in the detailed array camera may have a large aperture (i.e., a low f-number faster than f / 2.8, such as f / 2), allowing more light to reach the sensor.

[0151] Some imaging modules within the array camera system may include conventional RGB Bayer color fdter arrays. Other imaging modules may include color fdter arrays detailed in commonly- owned PCT patent document WO2024147826. Still other imaging modules can employ arrayed fdters having different polarizations rather than spectral responses. The color fdter array tile size may be 2x2 pixels, 3x3 pixels or any other size, including non-square configurations. Some imaging modules can omit any filter array (yielding panchromatic imagery) or employ identical filtering at each pixel location.

[0152] Although not essential, some imaging modules apply corrections to captured imagery before providing image data to the analysis stage or providing pixel data to the central unit. Prior art white balance, gamma correction, and hot pixel correction can be used. It is preferable is to apply correction steps as detailed in co-pending provisional patent application 63 / 621,986, filed January 17, 2024 ("Shadow Chrome" processing), now pending as US Application 19 / 013,418, filed January 8, 2025 (published as US20250234104).

[0153] The integration of camera lenses with camera photosensors is a field advanced by the mobile phone industry for short focal length cameras (i.e., 10mm and less). In one prior art system, the lens elements are injection-molded plastic, and some include peripheral spacer rings establishing correct spacing to a next lens element. The elements are commonly of different diameters and packed into a barrel (also injection molded) shaped with stair-step features dimensioned to receive the differently- sized elements and further assure their correct spacing. The composite lens system thus-formed is then provided to a machine that performs active optical alignment while the image sensor captures pixel data from a laser pattern. The machine adjusts relative alignment between the lens and image sensor to achieve optimum results (e.g., highest MTF measurements), as indicated by data from the image sensor (i.e., a feedback loop). The machine then glues the barrel of the lens to the image sensor at the optimum alignment using a UV-curing adhesive. Such machines are available from ASM International NV, IsMedia Co. Ltd., and Pioneer FA Corp.

[0154] Applicant has found that use of integrated imaging modules yields superior results compared with non-integrated arrangements. Removable lenses cannot be aligned as precisely as a lens in an integrated module. An integrated module, in which the lens and image sensor are coupled together (preferably during manufacturing) and never separated, improves f-number by 5-10 times. Thismeans 5-1 Ox beter light sensitivity, lOO-lOOOx smaller volume per captured pixel, and 10-100x lower manufactured cost per pixel.

[0155] To illustrate, an integrated imaging module employing a 62mm / f2.8 lens is computed to achieve a spatial frequency resolution of 250 lines per mm, contrasted with 50 lines per mm for such a lens in a conventional "mount" arrangement. To achieve 250 line / mm performance with a conventional mounted camera lens requires resort to a lens of 300 mm focal length (e.g., the Sigma 300).

[0156] In one embodiment, some or all image sensors are Sony IMX 334 devices, with 2 micron pixels and a size of 2592x1944 (5 megapixels). Alternatively, or additionally, a sensor such as the OnSEMI AR1335 can be employed (1.1 micron pixels, with a size of 4208x3120).

[0157] As noted, lenses in one or more imaging modules can include a variable focus arrangement. Such lenses can employ voice coil actuators, such as are available from Commonlands (e.g., the CLA321- VCM) and cnAICO (e.g., the ACVCM181210). Alternatively, piezoelectric ("squiggle") actuators can be employed, such as are available from NewScale Technologies (e.g., the M3-F) and Piezolution (e.g., the PZM-M12-06-DB).

[0158] Some embodiments employ imaging modules arrayed and directed to collect imagery from a full 360 degrees around the array camera system. Such embodiments, as well as other arrangements detailed herein, find utility in ADAS (Advanced Driver Assistance Systems) applications used in a vehicle.

[0159] It will be understood that the data analysis stage 26 in each imaging module 12 operates as a form of data compressor — reducing the volume of pixel data captured by the imaging module to a smaller set of data. For example, if pixel data is divided into blocks of 8x8 pixels for analysis, and 24 filtering functions are applied (each filtering function corresponding to one particular patern of pixels, and producing a single output datum indicating the strength or degree to which such pixel patern is present in the analyzed imagery), the 64 input pixel values are reduced to 24 output filter values (i.e., a compression ratio of 2.67). If the pixel data is divided into blocks of 16x16 pixels and 48 filter functions are applied, the 256 input values are reduced to 48 output values (compression ratio of 5.3). If the pixel data is divided into blocks of 32x32 pixels and 100 filter functions are applied, the 1024 input values are reduced to 100 output values (compression ratio of 10.2). Such compression is critical given the volume of data captured in an array camera system and the botlenecks that arise in processing and transferring such data.

[0160] In one embodiment, at least one imaging module includes an analysis stage 26 that examines an input set of pixel data for 3D paterns, i.e., extending not just in 2D scene space but also across time. For example, a neural analysis stage can filter a 3D set of input pixel data with a 3D kernel (again comprising an array of integer values) defining a 3D filtering function. The 3D set of input pixel values includes pixel values from two or more different frames of imagery captured by the sensor from a sequence of image frames depicting a given scene. From this 3D input data cube, a single output datum is again produced for each filtering function.

[0161] Again, data compression results from such 3D filtering operation. For example, if the input pixel data cube comprises 16x16 pixel blocks from three consecutive image frames (taken from the same spatial frame location), and 100 different 3D filtering functions are applied, the 768 input pixel values are transformed to 100 filter output values (compression ratio of 7.7).

[0162] Still higher compression ratios can be achieved by appropriate selection of the size of input pixel data blocks (cubes) and the number of filtering functions.

[0163] The filter output data stored in the array camera system (or a mirrored counterpart stored in a remote data repository) can be used to generate synthesized imagery. That is, since the stored filter values indicate strengths of the multitude of different pixel patterns for which the captured data was analyzed, these component patterns can be combined — at their indicated strengths — to recreate an estimated counterpart of the original pixel data. (In analogous fashion, JPEG compression represents image components by strengths of their principal DCT components, and decompression involves combining patterns represented by these DCT components to recreate an estimated counterpart of the original image.)

[0164] An image synthesis module that operates in this manner to recreate image data using stored filter result values desirably has a graphical user interface enabling an operator (or a viewer) to denote which part of the field of view captured by the array camera system is to be synthesized. For example, the operator may use a stylus or other input arrangement to define a rectangle in the scene space from which imagery is to be recreated.

[0165] It will be recognized that the fidelity of re-created imagery depends on the richness of the filter data. If relatively few filter data are used to produce re-created imagery, the resulting imagery will be relatively less accurate. With progressively more filter data, progressively more accuracy in recreation is achieved.

[0166] The accuracy of recreated imagery depends not just on the number of filter data extracted from each set of input image data, but also on the filter patterns used. In one embodiment, a principal component analysis is performed on blocks of historical imagery earlier collected of a scene or venue. This analysis discerns a set of basis functions into which the historical image blocks can be decomposed. These basis functions are ranked by their frequency of occurrence (or strength) in the analyzed imagery. Then the top-ranked functions (e.g., the top-ranked 100 functions) are selected to serve as the filter functions. (The lower-ranked filter functions can be stored in a library for use as later-detailed.) Such compression techniques are further detailed in the Yan paper cited below (Compressive sampling for array cameras).

[0167] In some embodiments, different filter functions are used in processing different spatial regions of image data captured by an array camera system. Consider an array camera system that images a sports field from the sidelines, comprising five rows of imaging modules. Within each row, imaging modules are angularly-splayed to point in different directions. The bottom rows are oriented towards the playing field and most commonly capture imagery of green grass, white field markings, and players. Above that may be a row or two that typically capture imagery depicting the grandstands —1often populated with spectators. The top row may capture imagery including a margin of the sky above the grandstands.

[0168] The sets of filter functions selected for compression of imagery from these modules can be different. A first set selected for the bottom rows can be optimized for compression of imagery depicting green grass, white lines, and players. A second set, chosen for the intermediate rows, can be optimized for compression of imagery depicting the grandstands and their occupants. A third set, selected for the top row, can be similar to the second set but include filter functions optimized for compression of blue, grey, and night skies. The first set will include filters of high spatial frequencies sensitive to green imagery. The third set will include filters of low spatial frequencies sensitive to blue, i.e., filter functions that are suitable for compressing feature-less regions of blue sky.

[0169] During operation, an image synthesis module 36 can re-create synthesized imagery from the output filter functions. An error stage 38 can compare this re-created imagery with a copy of the originally-captured pixel imagery (e.g., from the first part 32a of the central unit memory) and output an error metric signal indicating the degree of difference. That is, the error metric indicates the accuracy of the compression-decompression process. This error metric can be computed on a perimaging module basis or across a group of several imaging modules — such as across plural imaging modules in a row. The error metric serves to control filter update logic 40 that alters filter functions used by one or more modules to more closely correspond to the captured imagery, thereby increasing compression / decompression accuracy.

[0170] One way to alter a set of filter functions is to randomly replace one (or more) filter function in a set used by an imaging module analysis stage 26 with a substitute function (or more) and repeat the error analysis. If the error metric drops, the substitute fimction(s) is kept; if not, the previous filter fimction(s) is returned, or a different substitution is tried. An imaging module that applies 100 different filter functions may have an associated filter function library 42 of 500 alternate filter functions from which substitute functions can be drawn. These alternate filter functions can comprise those functions discerned in the analysis of historical imagery that were not highly-enough ranked to be selected for inclusion in the original set of filter functions.

[0171] Consider a sports field that hosts football games, as well as both men’s and women’s soccer games. The spectators of the football game commonly wear red - the football team’s color. The spectators of the soccer games commonly wear blue for the men’s team, and yellow for the women’s team. The imaging modules that are oriented towards the grandstands may initially be configured to apply a set of filter functions optimized for compression of imagery depicting spectators wearing red. But the error metric may show accuracy of the compressed data is unsatisfactory (e.g., below a threshold value). In such cases, the system can substitute one or more different filter functions, e.g., filter functions optimized for imagery depicting spectators wearing blue. If no improvement in the error metric is found, further filter functions may be substituted, e.g., optimized for imagery depicting spectators wearing yellow.

[0172] The just-given example, in which the filters differ in color, is somewhat contrived. More typically, the substitute filter functions will differ from those they replace by additional or alternative features, such as by pixel patterns or - in the case of 3D patterns - by temporal features.

[0173] In another embodiment, the set of filter functions applied by an imaging module (or a subset thereof) is determined during camera operation by periodically performing a principal component analysis on recent frames of imagery to determine what set of filter functions best characterizes the scenery recently captured by the imaging module. A group of top-ranked filter functions identified in such analysis is added to the set of functions employed by the module, replacing a group of filter functions earlier employed.

[0174] In both just-described embodiments, the in-use filter functions to be replaced can be identified by examining the output values of all filters in-use over some interval of frames (e.g., the past 10 or 10,000 frames) to identify those that do not seem particularly suited to the captured imagery. For example, the average or aggregate of the output values from each filter function over the interval of frames can be computed. The filter functions with the lowest computed values are identified as candidates for replacement.

[0175] Data identifying the filter patterns presently in-use by each imaging module is stored in a memory accessible to systems that need access to such information to decompress (re-create) the compressed imagery. Additionally or alternatively, packets of the compressed data can include a data preamble including a series of bit flags identifying which filters, in a library of known filters, are present in use.

[0176] Array camera systems may be generally conceived as having a plurality of imaging modules arranged in rows of several modules each. In the prior art, the modules within each row are typically splayed uniformly in angle, oriented at azimuths progressively incremented by a fixed angle, e.g., - 20°, -10°, 0°, 10°, and 20°, relative to a reference direction (see Fig. 6). Similarly, the different rows of modules are splayed in uniformly-spaced polar (elevation) angles, e.g., 10°, 0°, and -10° relative to a reference plane. In accordance with certain embodiments of the present technology, different spacing arrangements are employed.

[0177] The imaging direction of each module can be characterized by two angles, cp and 0. The first specifies the azimuthal angle of the module's lens axis relative to a reference direction, and the second specifies its polar angle, typically relative to a reference plane. See Fig. 7. (The reference direction and plane may be referenced, e.g., to the array camera system housing or to the venue in which the array camera system is mounted.)

[0178] Within a row of imaging modules can be imaging modules oriented in N different azimuthal angles, which collectively form a progressively-ordered set S = {(pl, (p2, cp3, ... cpN}, where cpl>(p2>(p3>... >cpN. These azimuthal angles have differences therebetween of:Al= cpl -q>2;A2= q>2-q>3;AN-1= cpN-l-cpN.

[0179] In accordance with some embodiments, some or all these differences progressively change in magnitude, e.g., so that A1>A2>...>AN-1. In some embodiments, N is at least 5, and the array camera system includes imaging modules having azimuthal angles (pl>(p2>(p3>(p4>q>5. In such embodiments, these azimuthal angles have differences of:Al= cpl -q>2;A2= q>2-q>3;A3= cp3-(p4; andA4= cp4-(p5where A1>A2>A3>A4.

[0180] In other embodiments, N is at least 6, and the array camera system includes imaging modules having azimuthal angles (pl>(p2>(p3>(p4>(p5>(p6.These azimuthal angles have differences of:Al= cpl -q>2;A2= q>2-q>3;A3= (p3-(p4;A4= (p4-(p5; andA5= (p5-(p6where A1>A2>A3>A4>A5

[0181] Array camera systems, including those detailed above, may have all modules oriented at a common polar angle 0 relative to a reference plane. Alternatively, some of the modules may be oriented at different polar angles.

[0182] In some embodiments, an array camera system includes at least three imaging modules having respective polar angles 01>02>03, where the polar angles have differences of:AA= 01-02; andAB= 02-03and where AA>AB

[0183] The array camera systems detailed herein may include imaging modules having three, four, or more different focal lengths - sometimes within the same row, and sometimes with different focal lengths in different rows. Similarly, the imaging modules in these array camera systems may include modules having three, four, or more different image sensor dimensions - again sometimes within the same row, and sometimes with different dimensions in different rows.

[0184] The relevance of the different angles, focal lengths and sensor dimensions referenced above will become clear from the following discussion, e.g., of Figs. 8-13.

[0185] Fig. 8 is a top-down view of a sports field showing, by the small circles and star, different locations at which array camera systems may be positioned. Fig. 9 is a top-down schematic view of an array camera system suitable for placement at the field position denoted by the star in Fig. 8. This array camera system includes eight different imaging modules that are (a) fixedly joined together by a structure including a housing, and (b) oriented in eight different imaging directions. Each such direction is characterized by an azimuthal angle q> relative to a reference direction, which is shown bythe arrow at the upper right. (Commonly, the imaging modules - or more particularly their lens apertures - are spaced uniformly across the face of the camera system. A different spacing is illustrated here to emphasize the angles involved.)

[0186] A first imaging module is oriented at an azimuthal angle cp of 5.8°. A second imaging module is oriented at an azimuthal angle cp of 15°. A third is oriented at an azimuthal angle cp of 23°. A fourth is oriented at an angle of 31°.A fifth is oriented at an angle of 40.1°; a sixth at 51.8°, a seventh at 66.7°, and an eighth at 84.7°.As can be seen, these modules are oriented at F >3 different azimuthal angles that collectively form a progressively-ordered set of azimuthal angles SF = {(pl, cp2, ... cpF}, where cpl>(p2>.. ,>cpF. In such embodiment, one pair of the successive azimuthal angles in set SF differs by a first delta-azimuth value AA1, and a next pair of the successive azimuthal angles in set SF differs by a second delta-azimuth value AA2 that is greater than the first delta-azimuth value. In this particular example, the full set of successive azimuths yields the set of successive delta-azimuth values {9.2°, 8°, 8°, 9.1°, 11.7°, 14.9° and 18°}, as annotated in Fig. 9. As can be seen, if the azimuthal difference value 9.1° is taken as the first value, and the next value of 11.7° is taken as the second value, the second value is larger than the first. (Likewise with values 8° and 9.1°, 11.7° and 14.9°, and 14.9° and 18°.)

[0187] In this case, it is also true that there are successive delta-azimuth values in this set in which a first value is greater than (rather than less than) a next, second value (e.g., 9.2° > 8°).

[0188] Fig. 10 helps illustrate why this matters. This figure depicts atop-down view of a sports field. An array camera system placed at the lower right comer comprises eight imaging modules directed at different parts of the field. The field of view of each module, in a horizontal dimension, is shown by the solid line on the far side of the field from the camera and is identified by the circled module number. Each imaging module is configured, e.g., by selection of lens focal length and by sensor dimensions, to assure that a subject (e.g., a player) at the most-distant subject distance (e.g., at the far side of the field) is imaged with a subject sampling resolution of at least 1 image pixel per 2 mm of subject extent. If a player at the far edge of the field is 2m tall, this assures the player's height spans a pixel distance of at least 1000 pixels. (This degree of resolution enables zooming-in for close-up shots, e.g., for replays, without visible pixelation.) The fields of view of the imaging modules are tailored to overlap to provide coverage of all parts of the playing field.

[0189] Imaging module 3 views subjects from the greatest distance, i.e., from diagonally across the field.If the image sensor in this module is of pixel dimensions 4200 (w) x 2400 (h) (which may be regarded as a 10 megapixel sensor), and the lens focal length is selected to achieve a scene sampling density (subject resolution) of 1 pixel per 2 mm, then the horizontal span of this sensor serves to image a horizontal physical distance 2mm x 4200 = 8.4 meters on the far side of the field.

[0190] It will be understood that any subject within the field of view of this third imaging module that is closer to the sensor (i.e., within the cross-hatched region) will be sampled at a subject resolution greater than 1 pixel per 2 mm. (The depth of focus of each imaging module spans the full range of field distances imaged by the module.)

[0191] Others of the imaging modules view scenes from lesser distances. For example, the far side of the field viewed by module 7 is only about half so distant as the far side viewed by module 3. Thus, a subject of a given size (e.g., a 2m -tall player) at the far side of the field subtends about twice the angular distance when viewed by module 7 as compared to when viewed from module 3. This allows the focal length of imaging module 7 to be shortened, and / or the pixel dimensions of the sensor in module 7 to be reduced, while still sampling subjects at the far side of the field with a subject resolution of at least 1 pixel per 2 mm.

[0192] In the embodiment depicted in Fig. 10, the imaging modules all employ sensors of the same pixel dimensions (e.g., 4200 x 2400). But several modules employ lenses of different focal lengths. Lenses of smaller focal lengths capture scenes of larger angular breadth; they have larger angular fields of view. Thus, for example, module 7 has a larger angular field of view than module 3. Progressively, as the maximum scene distance diminishes (e.g., progressing from module 3 to 4, to 5, to 6, to 7, to 8), the angular field of view of each imaging module can increase. (This assumes a constant sensor size and a subject resolution of 1 pixel per 2 mm at the maximum scene distance.) It is for this reason that the azimuthal angles of the imaging modules are selected to differ by progressively changing amounts (e.g., AA7>AA6>AA5>AA4>AA3 in Fig. 10).

[0193] In an exemplary embodiment there is at least one successive pair of azimuth viewing angles, in the progressively-ordered set of F azimuthal viewing angles, for which their difference AAJ has a value of between 5 and 10 degrees, and there is at least one second successive pair of viewing angles, in the progressively-ordered set, for which their difference AAK has a value between 10 and 25 degrees. Some embodiments, in which F > 4, have at least one third successive pair of azimuthal angles for which their difference AAL > AAK. Some embodiments, in which F > 5, have at least one fourth successive pair of azimuthal angles for which their difference AAM > AAL.

[0194] In another exemplary embodiment, these difference angles are as follows:5° < AAJ < 10°10° < AAK < 14°; and in some embodiments in which F > 4:14° < AAL < 25°

[0195] In yet another exemplary embodiment, these difference angles are as follows:5° < AAJ < 9°9° < AAK < 12°; and, in some embodiments in which F > 4:12° < AAL < 15°; and, in some embodiments in which F > 5:15° < AAM < 25°

[0196] It will be noted that the progressive changing of azimuth angle differences need not continue from each module to the next (spatially-neighboring) module. In the Fig. 10 example, the difference angles progressively diminish only between modules 8 and 3. At module 2 the progression reverses, with the difference angles beginning to increase. This is due to the maximum scene distances no longer increasing but beginning to diminish (i.e., the maximum scene distance of module 2 is less than that of module 3, and the maximum scene distance of module 1 is less than that of module 2).

[0197] Fig. 11 shows a sports field viewed by an array camera positioned at the center of the short edge of the field. The reversal in azimuth differences occurs here too. That is:AA1>AA2>AA3>AA4>AA5 but AA6<AA7<AA8<AA9<AA 10.

[0198] To review, one embodiment of the technology is an array camera system comprising a housing secured to a support adjacent to a sports field, the system further including F>4 imaging modules, each including a lens and an image sensor, the F imaging modules being fixedly joined together by a structure including said housing, the housing positioning entrance apertures of the F imaging module lenses in a shared plane, each of the F imaging modules being oriented in an imaging direction characterized by an angle (e.g., an azimuthal angle) within the shared plane, such that the angles are splayed non-uniformly within the shared plane.

[0199] Just as imaging directions can be splayed non-uniformly within an azimuthal plane, they can also be splayed non-uniformly within a polar dimension. Figs. 12 and 13 illustrate.

[0200] Fig. 12 depicts an array camera system elevated on a pole at one end of a sports field. Four modules in the system (or four rows of modules) are oriented at different polar angles downward towards the field.

[0201] The different angles are identified in Fig. 13.A first module is oriented in a direction 64.4° below the horizon (indicated by the arrow in the upper right) and serves to image an area of the field near the array camera system. Second, third and fourth modules are respectively oriented at polar angles 43.7°, 30.5° and 22.6° below the horizontal and serve to image areas of the field progressively more- distant from the array camera system.

[0202] As can be seen, these modules are oriented at G >3 different polar angles that collectively form a progressively-ordered set of polar angles SG = {01, 02... 0G}, where 01>02... >0G.

[0203] As in the case illustrated in Fig. 10, the modules are configured to spatially sample scenes at a subject resolution of at least 1 pixel per 2 mm. For distant subjects, a lens of longer focal length is used, which has a commensurately smaller angular field of view. For nearer subjects, a lens of shorter focal length can be used, with a commensurately larger angular field of view. The progressive differences in polar angles of the imaging modules can vary in accordance with their respective angular fields of view. In the Fig. 13 case, that means AP3>AP2>AP1.

[0204] In an exemplary embodiment, there is at least one successive pair of polar angles, in the progressively-ordered set of polar angles, for which their difference APJ has a value of between 0 and 23 degrees relative to a reference plane, and there is at least one second successive pair of polar angles, in the progressively-ordered set, for which their difference APK has a value between 23 and 70 degrees. Some embodiments, in which G > 4, have at least one third successive pair of polar angles for which their difference APL > APK.

[0205] In another exemplary embodiment, these difference angles are as follows:0° < APJ < 23°23° < APK < 31°; and, in some embodiments in which G > 4:31° < APL < 70°

[0206] In some embodiments, the use of modules having lenses of shorter focal lengths (with their larger angular fields of view) means that fewer such modules can be used to capture imagery of a given physical expanse of scene. Fig. 14 shows an array camera system employing this principle.

[0207] The Fig. 14 camera system has 22 imaging modules arranged in four rows. Each row is characterized by a different polar angle. There are seven modules in the top row (oriented at a polar angle of 22.6°); six modules in the second row (oriented at a polar angle of 30.5°); five modules in the third row (oriented at a polar angle of 43.7°); and four modules in the bottom row (oriented at a polar angle of 64.4°). Since the modules oriented at the larger polar angles serve to image portions of the playing surface at closer distances to the camera system, they employ lenses of relatively short focal length. Such relatively wide-angle lenses have relatively larger angular fields of view, permitting fewer such modules to be employed to image a given scene.

[0208] Other array cameras have different numbers of modules with different row counts. A few examples include (A) 80 modules, arranged as rows of 20, 18, 16, 14 and 12; (B) 60 modules, arranged as rows of 30, 20 and 10; and (C) 9 modules arranged as rows of 4, 3 and 2.

[0209] It should be noted that the azimuthal angles and polar angles at which imaging modules are directed need not each progress in magnitude corresponding to spatial location within the camera system housing, e.g., from one module to the next in rows or columns. For example, while the eight azimuthal angles depicted for the array camera of Fig. 9 change monotonically, each to the next, this is not required. The modules of Fig. 9 could instead be re-arranged so they point in the directions shown in Fig. 15. The same directions are illustrated, just in a different positional or spatial order within the housing.

[0210] While this is a contrived example, the point remains: spatial ordering of the modules in accordance with their respective imaging directions is not required. In judging progressive differences in azimuth angles (or polar angles), we first order the angles numerically — smallest to largest or vice versa (if not already so-ordered) to yield an ordered progression of angles and then determine a corresponding progression of angle differences from this ordered sequence. This is the sense in which the term "progressively-ordered" is used herein, e.g., when referring to a set of angles.

[0211] It will be recognized that embodiments of the technology provide image data depicting large physical scenes from which selected excerpts can be rendered for display, broadcast, and / or replay. Replay can be particularly technically demanding, as slow-motion replay is commonly desired. To permit slow motion replays of high-speed events (such as of a baseball pitcher's motions or of a tennis ball striking a racket), frames must be captured at rates of 100, 200, or 300 frames per second or more.

[0212] The net result is a huge volume of data. If there are 50 imaging modules in an array camera, and each module has a 10 megapixel sensor, and frames are captured at 200 / second, then each sensor outputs 2 gigapixels of imagery per second. Across all 50 imaging modules, the single array camera system produces 100 gigapixels each second. Each pixel can comprise ten or twelve (or more) bits, leading to generation of more than a terabit of data every second.

[0213] Bottlenecks abound in such a system. Bottlenecks in storage and retrieval of the data can be alleviated, to some extent, by using multiple memories (e.g., RAM memories) that are independently operable to permit simultaneous access. Pixel data from a given image sensor module can be distributed for storage across several of the memories. And a given memory can store pixel data from several different imaging modules of the array camera system. Compressed data (e.g., fdtered values) derived from the pixel data can be stored similarly, with compressed data from a given image sensor module distributed for storage across several memories, and each memory storing compressed data from several different imaging modules.

[0214] One such arrangement is schematically shown in Fig. 16. In this arrangement, pixel data from each imaging module is distributed between six memories — three for pixel data and three for compressed data. Similarly, each memory stores data from three imaging modules. These parameters are illustrative only, and other values can be used. For example, pixel data from each module can be distributed between four memories, where each of these memories stores pixel data from two modules. Additional details on suitable memory architectures are provided in commonly-owned US patent 10,477,137.

[0215] In some embodiments, the torrent of data is further addressed by selective storage. For instance, the full set of pixel and / or compressed data generated from a frame of captured imagery may be stored only every Pth frame (e.g., where P is greater than or equal to 3, 5, 10 or 20). For intervening frames, pixel and / or compressed data is stored only for region-of-interest frame excerpts identified by detection of interframe motion. By this arrangement, movement of a sports ball, players, and referees is depicted with full image fidelity — storing associated pixel and / or compressed data, e.g., at 100 frames / second or more. But image data for static portions of the scene are stored only for every Pth frame (e.g., at or less than 60, 30 or 15 frames per second).

[0216] Known techniques from MPEG, H.264 and H.265 video compression can be employed to identify scene motion and to define regions of interest (e.g., rectangular image excerpts defined by comer pixel coordinates or object tracking).

[0217] Moreover, the field of view captured by an array camera system can include not just a sports field and associated activity but also grandstand seating and associated activity. It is not necessary to store image data at 200 frames per second that depicts spectators drinking beer. Accordingly, a spatial mask can be defined to identify pixel regions in the system field of view that merit (or that do not merit) high speed image capture. Other regions can be sampled and / or stored at lower rates. So even when motion is detected in the grandstands, full-fidelity image data is not stored, but rather an abbreviated set of data — such as data corresponding to every Pth frame.

[0218] This feature is conceptually shown in Figs. 17 and 18, in which the cross-hatched region denotes a mask indicating regions of the image data that should be stored with high frame rates (e.g., 100 frame per second or greater), as contrasted with the lower frame rate data stored elsewhere.

[0219] Another demanding aspect of many embodiments concerns exposure interval. As noted, many sports involve high speed motion. Examples include baseballs, tennis balls, and ice hockey pucks,each of which can travel at speeds approaching or exceeding 100 miles per hour. In order for imagery of such subjects to be depicted clearly, it is necessary to use exceedingly short exposure intervals, such as less than 250, 100 or 50 microseconds. For instance, to limit motion blur on a subject moving 100 mph to one pixel or less, it is necessary to use an exposure interval of 50 microseconds (assuming focus distances typically encountered in sporting venues, with a subject sampling resolution of 1 pixel per 2 mm).

[0220] In other contexts, strobe lighting might be used to allow such short exposure intervals. But constant strobe lighting is not practical in sporting venues; ambient lighting must be used.

[0221] To permit such short exposure intervals (e.g., 250 microseconds or less) with ambient lighting, it is beneficial to use image sensors employing one or more arrangements detailed in applicant's US patent 12,047,692. Such sensors excel in low light applications. In one such sensor, the color filter array employs one or more filters that does not substantially eliminate any portion of the visible light spectrum but instead passes at least 20% of incident light (and preferably passes 40%, 60% or 80% or more of incident light) at all visible wavelengths.

[0222] Another aid to use of short exposure intervals is to employ one or more image enhancement techniques detailed in applicant's pending patent application 63 / 621,986 (e.g., "ShadowChrome" processing), now pending as US Application 19 / 013,418, filed January 8, 2025. Such methods, for example, reduce the image noise that arises with short exposure intervals.

[0223] A principal application of the array camera technologies detailed herein is in capturing sports content for broadcast and streaming. Streaming includes not just packetized delivery of sporting events to remote viewers at home (on-demand or in real-time via the internet) but also to viewers within sports venues, such as in suites of a football stadium where a game is being played. Each suite can be equipped with a display screen and associated control apparatus that presents a user interface to enable viewers to select desired excerpts of the field or moving action on the field for viewing. That is, guests in one suite may select, for display, imagery depicting a first area or player(s) on the field (to the exclusion of other areas and / or other players simultaneously imaged by the array camera system), while guests in another suite may make different such selections. Eikewise for remote viewers — each may define their respective viewing experience.

[0224] As in the prior art, each of the array camera systems detailed herein can additionally include one or more wider-angle imaging modules (i.e., having lenses of shorter focal lengths), which captures a single field of view that spans the smaller fields of view of several other imaging modules to provide a larger spatial context from a single sensor's output.

[0225] While imaging modules in the array camera systems detailed above are commonly selected and arranged to capture high frame rate (e.g., 100 frame per second or more) imagery at 2mm-per-pixel resolution across overlapping fields of view to provide image data of substantially uniform quality (fidelity) across a large composite field of view, other embodiments are different. For example, different zones in the composite field of view can be captured with different degrees of image fidelity, such as frame rate, resolution, and / or bit-depth. In some embodiments, the imaging modulesmay be pointed in directions that, in places, do not capture overlapping (nor even adjoining) fields of view, leaving gaps in coverage.

[0226] The regions where image data of lower fidelity is captured (or no image data is captured) are chosen to be areas in which content is of lesser interest (or no content of interest is expected to occur). Costs and processing resources can thereby be conserved and reallocated to the regions of greatest interest.

[0227] One example is in scouting players for major league baseball teams. Baseball occurs at multiple scales — both spatial and temporal. On a fine scale, a pitcher takes about one second to throw the ball, and a batter's swing takes place in about 200 milliseconds. The pitch moves at about 90 miles per hour. If sampled at conventional frame rates of 60 frames per second, the ball advances more than two feet between frames, and the batter's swing takes place in the span of just a dozen frames.Scouting reports require more accuracy than this — preferably down to spatial and temporal scales on the order of millimeters and milliseconds.

[0228] To achieve such resolution of fast events, the frame rate must be increased to 200 or more (preferably 300 or 500 or more) frames per second. As before, resolution of at least 2mm per pixel is desired at captured scene, i.e., the pitcher's mound and batter's box.

[0229] But other areas of the baseball infield and outfield are of relatively less importance and can be captured in imagery with lower fidelity. A game may last three hours or more, so economy of data storage becomes a consideration.

[0230] In accordance with one embodiment, an array camera system simultaneously captures both the fine scales and large scales of a baseball game. To capture the fine scales, the array camera system includes at least first and second imaging modules. One of these modules is directed to the pitcher's mound. The other is directed to the batter's box. These modules are equipped with lenses of first focal lengths LI and L2, and imaging sensors SI and S2, chosen to capture imagery at 2mm-per-pixel resolution at a frame rate of 100 / second or above. The array camera system further includes a third imaging module equipped with a wider angle lens of focal length F3, where F3<F1 and F3<F2.

[0231] In a particular embodiment, the higher fidelity imaging modules can have focal lengths, Fl and F2, greater than 20mm (e.g., 21.8, 25 or 50mm), while the lower fidelity imaging module can have a focal length F3 of less than 20 mm (e.g., 5 or 12 mm). Both types of imaging modules desirably produce pixel frames of size 1920 x 1080 or larger.

[0232] The first and second imaging modules can be steerable, either manually (e.g., in a pivot mount) or electronically (e.g., steered by processor-controlled stepper motors) to point them to the pitcher's mound and batter's box (the high-resolution subjects). Alternatively, the array camera system can be built in contemplation of known placements on the baseball diamond (e.g., off the playing field, outside the first and third base lines, such as at the positions indicated by stars in Fig. 19, equidistant from both high-resolution subjects), in which case the two modules can be fixedly-mounted, pointed in azimuthal directions separated by between 60 and 75 degrees, and preferably separated by between 65-70 degrees.

[0233] Such array camera systems can be quickly set-up on tripods and positioned so that their two imaging modules are directed to the two high resolution subjects (as shown by the two solid-line arrows from the left-most star in Fig. 19).

[0234] The third imaging module of the array camera system can be directed in a third direction, different than the first two directions, chosen to provide a view of most or all of the ball field (e.g., as shown by the dashed-line arrow from the left-most star in Fig. 19), spanning the views of the first and second imaging modules. It can operate at a lower frame rate, e.g., below 100 or 60 frames per second.

[0235] The array camera system can additionally, or alternatively, be equipped with still other imaging modules for capturing high resolution imagery from other isolated regions (fields of view) around the ballfield, such as the bases, the shortstop position, and the batting circles. Second or further wide- angle imaging modules can also be provided, again with focal lengths shorter than LI and L2, to provide additional lower-fidelity contextual imagery.

[0236] It is desirable that a ball field be equipped with at least two such array camera systems, one on each side of the field, so that frontal imagery of both left- and right-handed pitchers and batters can be captured during their respective motions.

[0237] After the game, a tripod-based system can be quickly disassembled and packed for moving to another ballfield.

[0238] In some embodiments, the higher fidelity modules normally operate at a first frame rate, e.g., 60 or 100 frames per second, but can be switched to a burst mode, e.g., of 300 or 500 frames per second for a predetermined interval (e.g., 1 or 2 seconds). Depending on the sensor, burst mode can involve changing the capture resolution to accommodate the more demanding frame rate. Burst mode can be manually triggered (e.g., by a tablet user interface) or automatically triggered (e.g., by machine vision-based recognition of a pitcher's movement from an initial pitching stance). Such triggered burst-mode captures serve to limit data storage requirements. (As noted, a baseball game can last many hours, which makes storage of continuous 500 i s video onerous.)

[0239] In an alternative embodiment, the higher fidelity modules capture video continuously at full speed but write the data to a circular (ring) buffer that is overwritten in 5 or 10 seconds. Video event recognition serves to detect the start of the pitcher's pitch or the batter's swing. Such event recognition triggers transfer of the previous 2 seconds of video from the circular buffer into long-term storage and directs video captured for a first further interval, e.g., the next 3 seconds, into that long-term storage. Thereafter, the captured imagery is again directed to the circular buffer until another event recognition occurs and the cycle repeats. By this arrangement, the long-term storage collects 5 second clips of relevant pitcher / batter motions without intervening video data. (The long-term storage may be, e.g., a physical or solid state disk drive.)

[0240] A similar arrangement can be used with imagery captured by the lower fidelity (wider angle) imaging module, with detection of a pitcher / batter event triggering storage of recent video from a circular buffer into long-term storage, and imagery captured during a further second interval,typically longer than the first interval (e.g., more than the next five seconds, such as 10 or 20 seconds), also being written to long-term storage. Thereafter, the imagery from the wider angle imaging module resumes writing to the circular buffer. (All noted time intervals can be user- adjustable.)

[0241] Desirably, video from each imaging module is analyzed for a triggering event (e.g., the start of a pitcher's pitching motion). Upon detection of such an event, video in the circular buffers for all imaging modules in the system can be written to long-term memory, and all their captured video for a following interval can be written to the long-term memory.

[0242] In applications where two or more array camera systems are used, the same arrangement can be extended across all such systems. That is, when an event is detected in video from one imaging module in one system, it can trigger long-term storage of data not just from imaging modules in that system but also from imaging modules in the other system(s). Such event notification can be achieved by wireless or wired communication.

[0243] By arrangements such as the foregoing, all imaging modules can trigger at essentially the same instant, rather than having different modules triggering at different instants due to their different views of the pitcher / batter. Time-synchronized clips captured by each imaging module is thereby stored in long-term memory.

[0244] Machine vision-based event detection can be implemented with a neural network arrangement trained with a corpus of video depicting pitch- and swing-motions and further with a corpus of other video depicting other motions that are not pitch- and swing-motions, to enable the network to judge which motions should trigger capture of a video clip. Such event detection can be performed by each imaging module. In other embodiments a central unit, such as processor 30 in Fig. 4, can perform such detection on video data from all imaging modules.

[0245] One or more array camera systems of the described type, equipped with a wide field-of-view imaging module and two or more other imaging modules, where the latter modules capture narrower, isolated fields of view within the wide field of view, can be positioned other than as described above. For example, such a system can be placed behind home plate, with one imaging module pointed at the batter and another pointed at first base.

[0246] Of course, such array camera systems are useful for more than just baseball scouting. They can, for example, be used to capture imagery for other analytics or for broadcast. Similarly, such array camera systems are not limited to use just with baseball. Other applications also present isolated regions of particular interest within larger scenes of lesser interest. Tennis (with its two fixed service zones on opposite ends of the court) and hockey (with its two fixed goal nets) are examples. (Fields of view of first and second imaging modules in an array camera system are considered isolated if a line between their centers passes through a scene region that is imaged by the array camera system, if at all, at a resolution, frame rate and / or bit-depth that is lower than that of both the first and second imaging modules.)

[0247] In applications using more than one array camera system, the units can be wirelessly networked to a hub, which can enable synced timing and integration with other analytic accessories, e.g., radar.

[0248] Fig. 20 illustrates one implementation of the technology. The lens at the top left is part of the wider field (lower fidelity) imaging module, while the other three lenses form part of three narrower field (higher fidelity) imaging modules (e.g., steerable to point at the batting box, the pitcher's mound, and second base).

[0249] In some embodiments, a large (e.g., 20 megapixel) sensor is operated to capture a reduced-size (e.g., 2.5 megapixel) region of interest (ROI) in order to capture frames at the high rates desired. For example, the OnSemi AR2020 sensor is a 5120 x 3840 pixel sensor that captures frames at a nominal rate of 30 / second. However, by defining a smaller ROI (e.g., 1900 x 1200), it can be operated to capture hundreds of 10-bit image frames per second. (The ROI can be centered at coordinates determined to be a center of mass of inter-frame motion within the larger frame.)

[0250] It will be understood that the imaging modules in the just-described arrangements are preferably integrated imaging modules, as described earlier. It will further be understood that the modules can store both compressed and uncompressed versions of the imagery (also as described earlier), although this is not essential. Other aspects of the earlier-detailed embodiments are likewise applicable in the just-detailed embodiments, including use of technologies detailed in applicant's earlier patent filings to enhance image quality when capturing very short exposure frames (as occurs, e.g., at capture rates of hundreds of frames per second).

[0251] Among other features, the foregoing discussion detailed systems that enable sporting events to be captured for broadcast by a few fixed camera systems. With the resulting video data, producers outside the stadium (in the parking lot or across the continent) can specify and steer virtual cameras to produce desired views of the action on the field. Such arrangements avoid the need for individual camera operators positioned at gimballed, zoom-capable cameras around the field — each trying to follow instructions relayed from remote producers about views they should try to capture. Instead, embodiments of the present technology permit the producers to directly control the direction and zoom of camera views without working through the intermediary of individual camera operators (and without their associated costs).

[0252] The arrangement also provides new flexibility in post-production, as contrasted with realtime broadcast. That is, after the fact (e.g., a minute or a year later), a user can direct a virtual camera view to an on-field event that was not shown during the real-time broadcast, such as a later-alleged foul in a play or a view of an emotional player, and obtain video footage tailored by the user (e.g., in pan, tilt, zoom, and video format) to depict such event.

[0253] The following discussion elaborates associated hardware systems, e.g., continuing the discussion of Fig. 4, and particularly considering the processing of pixel data as stored in memory 32a by associated processor(s).

[0254] As noted, the pixel data in memory 32a is collected by a plurality of imaging sensors mounted in a common housing. This arrangement, with multiple sensors, is sometimes termed a "camera head."

[0255] Fig. 22 conceptually illustrates the views of seven imaging sensors A-G in a camera head. This head has two rows of imagers: four imagers on the top row and three on the bottom. All employ the same physical sensors (e.g., the OnSemi AR2020). However, the imagers in the bottom row have lenses of shorter focal length, giving them larger fields of view. (The imagers in the top row may all have focal lengths of, e.g., 50-70mm, while the imagers in the bottom row may all have focal lengths of, e.g., 40-55 mm.) The bullseye marker in the middle of Fig. 22 indicates an arbitrary but important center direction (bore- or chief-ray) of the camera head, being mid-way between the highest and lowest pixels in the composite data set (in pixel-space) and midway between the leftmost and rightmost pixels in the data set. This bore ray direction can serve as the origin of a coordinate system for the camera head, as the geometric center of its pixels' rays. Each pixel of the individual cameras in the head can be mapped to a direction within that coordinate system. (Each individual camera additionally has its own bore ray.)

[0256] Fig. 23 shows how each imaging sensor writes its pixel data to a corresponding range of memory 32a. In this example, each imaging sensor is allocated 40 megabytes of memory — enough to store 2-4 frames of pixel data.

[0257] In one application, the camera head is positioned in the grandstands of a football game at the 50 yard line, and the imaging sensors capture views like that shown in Fig. 24.

[0258] Fig. 25 is an enlargement from the middle imager in the bottom row of Fig. 24 and shows, by the dotted rectangle, a view specified by a producer to be sent to the video switcher for incorporation into a broadcast of the football game.

[0259] The producer can specify views in several different ways. One way is to define a rectangle by swiping a finger on a touch-sensitive screen (with a display akin to that shown in Fig. 24) to specify a rectangular area by two comer locations (shown by plus symbols in Fig 25). Another is to single-tap the screen, and the processor employs a default zoom value to define the size of the rectangle. (A default aspect ratio, such as 4:3 can be used, or several different formats of output can be produced with different aspect ratios. Likewise with frame rate; a default frame rate of 30 i s can be employed, and other or additional frame rates can also be produced.) The producer can double-tap to select another default zoom value (perhaps for a tighter close-up shot). The producer can also use two fingers to expand or contract a default zoom value, using gestures familiar with smartphones.

[0260] Still other navigation paradigms can be employed, e.g., drawing from those used by gamers in navigating 3D scenes. Such other techniques can make use of joysticks, trackballs, etc., to define a desired pan / tilt / zoom view of a scene.

[0261] The producer's instructions are translated into view coordinate parameters usable by the system processors, such as pan and tilt angles for the center point of the view and a zoom factor. The pan and tilt angles can be in an angular frame of reference like that shown in Fig. 7. The pan angle is the azimuthal angle. In an exemplary embodiment, a pan angle of zero corresponds to the direction faced by the front center of the camera head, i.e., as indicated by the bullseye in Figs. 22, 24 and 25.Positive pan angles can be to the right of the bullseye in the figures, while negative pan angles can beto the left. A tilt angle of zero can likewise correspond the vertical direction the fixed camera head is pointing, again as indicated by the bullseye in the noted figures. Positive tilt angles can be below the bullseye, and negative tilt angles can be above. Alternatively, a tilt angle of zero can be defined by the physical world, i.e., oriented to the horizon (and perpendicular to the force of gravity). In such arrangement, a tilt angle of ten degrees specifies a view oriented ten degrees below the physical horizon. Except as otherwise apparent, the location of a pixel — whether a pixel from an image sensor or a pixel computed for inclusion in a stream of output data — refers to its angular coordinates, such as pan and tilt.

[0262] After the producer has indicated a desired shot, e.g., in one of the manners described above, the processor can automatically move the specified view, frame-to-frame, to track action on the field, e.g., in accordance with known motion detection and prediction algorithms as used in MPEG and other video compression.

[0263] ("Producer" has historically denoted a person overseeing composition of a video production.Producers referenced in this disclosure can, indeed, be individuals. But the term should also be understood to encompass automated systems, e.g., a producer agent implemented with Al, that selects particular video shots for delivery as an output stream to a particular client. In some implementations, a dozen or more different clients are present, most — if not all — being served different content streams. Sometimes we speak of clients requesting an output stream without any express involvement of a producer.)

[0264] Fig. 26 shows aspects of a processing system like that shown in Fig. 4. The processing is divided into two phases, performed by first and second processors 30a and 30b.

[0265] The seven (N) imagers provide pixel data to the memory 32a, here called the aggregate scene memory since it contains all the pixels from all the imagers. The first processor 30a receives data from a producer indicating what virtual scene view is to be produced from the aggregate scene memory. (This data can be output by a user interface with which an individual producer interacts or can be output by a producer agent.) This received data may specify, e.g., the pan and tilt coordinates of the center of the view, together with zoom, aspect ratio, and frame rate of the desired video stream.

[0266] The first processor 30a determines which image data from the aggregate scene memory 32a is needed to fulfill the producer's request. In some embodiments, the first processor maps the pan / tilt / etc. view request into memory addresses where the needed image data is stored. In other embodiments, the first processor simply provides floating point coordinates of the output pixel locations, and a memory controller determines memory addresses where the needed image data is stored.

[0267] In a rare case, the needed image data is simply a cropped excerpt of the high-resolution image data provided from an imager. For example, the AR2020 sensor outputs frames of 5120 pixels wide (column count) and 3840 pixels high (row count). The producer may request a 1080p-sized frame, which is 1920 pixels wide by 1080 pixels high, at maximum imager resolution, within the field of view of a single imager. (The dotted view in Fig. 25 is illustrative of this situation.) In this case, thefirst processor takes the input pan / tilt coordinates and identifies the memory addresses that correspond to a rectangular block of image data centered at these angular viewing coordinates. It then instructs the memory to deliver this block of image data to the second processor 30b. The second processor simply outputs this image data to a subsequent stage (e.g., which may perform video encoding), since no further processing is required on the image data fetched from the memory.

[0268] More commonly, the needed image data is not simply a cropped excerpt of image data from an imager.

[0269] Each pixel in the camera head images a point in the scene characterized by an associated pan and tilt angle. (This association between each pixel and associated pan / tilt angles can be computed on the fly based on sensor parameters and camera geometry or may be computed in advance and stored in a data structure such as a table.) The pan / tilt / zoom view specified by the producer may be found to have comer point coordinates, in angular space, that don't match the angular coordinates of any sensor pixel. Instead, the system must interpolate between sensor pixel values stored in the memory. Likewise for other output pixels in the producer-requested view.

[0270] (Considering Figs. 22 and 24, it may be understood that, from a projective geometry viewpoint, the scene coordinates are actually defined in a spherical angular space relative to the bore ray direction (the bullseye target). Any given rectangle of sensor pixels will thus project out, in absolute space, to an area bounded by four curved edges. With rare exception, the same is generally true for any straight line of pixels on a sensor — they map to a curved line in the real -world space. This phenomenon can be taken into account in embodiments of the present technology, such as by mapping the curve to a sequence of line segments (e.g., pixel locations) of different tilt, as noted below.)

[0271] In an illustrative process, the first processor 30a of Fig. 26 operates on the input pan / tilt / zoom parameters to determine the location, in angular space, of the upper left pixel of the desired scene view. This is done with floating point arithmetic and yields real number angular coordinates having several decimals of precision. The processor then determines, e.g., using the zoom state, how far to the right and up or down the next requested output pixel in the top row is located in angular space, again by floating point coordinates. And successively likewise for all the other pixels. The desired scene view is thereby mapped to a sampling of pixel locations within the aggregate scene memory 32a. These desired pixel locations in angular space commonly do not match the pixel locations sampled by the sensors and stored in the aggregate scene memory, so interpolation is used to obtain the sampled image points from the stored data.

[0272] It is in this mapping of the desired view into the aggregate scene memory 32b that curvature can be taken into account — identifying piecewise linear segments of interpolated pixel locations in aggregate scene memory in which each pixel segment is charactered by fixed row- and columnoffsets between successive pixels, e.g., in a pixel row, thereby defining a pixel segment tilt. After the curvature error due to this linear approximation accumulates to a given degree (e.g., after a stretch of 32 or 64 pixels), these fixed offsets are adjusted to define a next segment of pixel locations having adifferent tilt. And so on. The increment of row and / or column offsets within the scene memory, from one pixel to the next, may thus be different at the left end of a pixel row in the desired view than at the right end.

[0273] Such operation will be further clarified by illustrative examples. Assume the pixel pitch for a given imager is constant in angular space for all pixels in the imager sensor. For example, if the imager has a horizontal field of view of 10 degrees and the imagery captured by the sensor is 5120 pixels in width, then the pan angle difference between adjacent sensor pixels is 10 / 5120, or 0.001953 degrees. Similarly for the tilt angle difference between adjacent pixels in a column. Accordingly, instead of determining — in angular space — how far to the right and up / down each successive pixel should be, the processor can determine (from the parameters specifying the producer's desired scene view) the location of all successive pixels in the frame in terms of offsets from the upper left comer in pixel space.

[0274] Imagine, by happenstance, that the upper left pixel location exactly corresponds, in angular space, to one of the pixels captured by an imager. Further assume that the producer requests a view that is at a zoom factor of one-half relative to the native resolution of the imaging sensor. In such case, the first processor 30a may instruct the memory to fetch the upper left comer pixel, and then ignore the next pixel to the right in the memory but fetch the pixel that is two to the right of the comer pixel (rightleft being the orientation of pixel rows in this example). And then fetch the pixel that is four to the right, and the six to the right, and so on across the memory. When the requested output row is filled (e.g., of 1920 pixels), the processor instructs the memory to ignore an intervening row of data in the memory but fetch a pixel that is two pixels down from the comer pixel. And then fetch the pixel that is two to the right, and then four to the right, and then six to the right, etc. By such arrangement, an output frame is composed from scene data sampled at a zoom factor of one-half the native sensor resolution.

[0275] Naturally, such happenstance occurrences — of the comer pixel's angular coordinates exactly corresponding to those of a sensor pixel, and the needed spacing between output pixels being an integer number of sensor pixels — are rare. More commonly, the angular coordinates of the desired output comer pixel do not match any pixel, and the needed pixel sampling interval is not an integer value.

[0276] In this case, interpolation is used. In particular, the upper left comer of the producer-specified output frame is determined in angular space, and a group of pixels whose associated angular coordinates are clustered around this comer pixel location are fetched from the aggregate scene memory. An interpolation kernel is then applied to values of these clustered pixels to determine a pixel value at the upper left comer of the specified output frame. Similarly for each other pixel of the output frame.

[0277] In the Fig. 26 embodiment, the first processor 30a determines the angular coordinates of the upper left pixel by floating point arithmetic. This first processor also determines the memory addresses of the cluster of pixels that most closely correspond to these determined angularcoordinates. Values of these clustered pixels are then fetched (by memory control circuitry 33) and provided to the second processor 30b. The second processor applies an interpolation kernel to values of these clustered pixels to determine the comer output pixel value. In one particular embodiment, this interpolation operation is performed using integer instructions rather than floating point.

[0278] From the producer-specified desired view parameters, the first processor 30a also calculates the offset between the comer pixel and the next pixel in the desired output row. As noted, this calculation can be performed in angular space or pixel space. The first processor may determine that the next pixel is at an offset of 7.17 pixels to the right of the comer pixel. Again, this processor determines the memory addresses of the cluster of pixels nearest this location, and these data are fetched from memory. The second processor 30b then applies an interpolation kernel to values of these clustered pixels to determine the next output pixel's value.

[0279] In the same fashion, the system determines values for each of the other pixels in the top output row — by identifying a group of pixels nearest the location of each desired output pixel and then interpolating their values to produce output data. After determining the, e.g., 1920 output pixel values in the first row of the output frame, the system calculates the position of the leftmost pixel in the second row of the output frame, typically as an offset from the upper left output pixel — either in angular space or pixel space. The system then continues as described above — using interpolation to compute output values for the leftmost and then for each successive pixel in the second row. And so on across and down through the output frame.

[0280] (While the foregoing process is described as proceeding in incremental fashion, starting by determining the value of the upper left comer output pixel and proceeding from that location, it will be understood that actual implementations can proceed differently. For example, the value of any pixel in the output frame can be determined first, and all other pixels can be located — and their values determined — based from that starting point. Moreover, while the process is described in a pixel-at-a- time manner, it will be understood that parallelism and vector operations can be employed to yield multiple output pixel values in a single instruction. One application of parallelism is to determine X- and Y-floating point pixel locations for all desired-view pixels in a given pixel segment, given the X- and Y -offsets that characterize that segment.)

[0281] Aspects of the foregoing are illustrated in Fig. 27. This figure shows an excerpt of the scene data, with each plus sign indicating a pixel location for a given sensor. The sloping lines with the periodic black dots show (by the dots) the locations of pixels in the producer-specified output frame relative to the sensor pixel locations. As can be seen, each successive output pixel is offset from the previous output pixel by a distance of roughly 7 pixels horizontally and one half pixel vertically. Again, processor 30a computes these offsets with high precision.

[0282] The bold square 271 in the upper left quadrant of Fig. 27 identifies 16 sensor pixels whose values are applied to an interpolation kernel (e.g., a bicubic interpolation kernel) to yield a value for the output pixel at the location of the black dot near the center of this square. (As is familiar, operation ofsuch an interpolation kernel involves multiplying each of the 16 sensor pixel values with a corresponding kernel coefficient and summing the resulting products.)

[0283] In the upper right quadrant is a rectangular bold box 272 identifying 12 sensor pixels whose values are interpolated to yield a value for the output pixel at the location of the black dot near the center of this rectangle.

[0284] In the lower left quadrant is another bold rectangle 273 identifying 12 sensor pixels whose values are interpolated to yield a value for the output pixel at the location of the associated central black dot.

[0285] In the lower right quadrant is a bold square 274 identifying a single sensor pixel whose value is used to determine a value for the output pixel at the location of the associated central black dot. (This can be regarded as a unity interpolation kernel — the output value is equal to the input value.)

[0286] Fig. 27 illustrates that different interpolation kernels may be used in different circumstances. In the upper left, the output pixel location is within a quarter pixel of the point where four pixels touch at a point. Since the output location is roughly equidistant from the centers of these four pixels, the values of all four pixels are used in interpolating the output pixel value. Additionally, values of each of the pixels that edge- or comer-adjoin one of these four pixels is also used in a bicubic kernel (i.e., 16 in all).

[0287] In the upper right, the output pixel location is less than three-quarters of a pixel distant from the centers of pixels to the right and left but is much further distant to centers of the nearest pixels above and below. Given these circumstances, a horizontal rectangular (non-square) kernel can be used, spanning four pixels in a central row region that includes the output pixel location, together with four pixels above and below.

[0288] Kernel 273, in the lower left, is used for similar reasons. But here, the output pixel location is within three-quarters of a pixel of pixel centers above and below. In this circumstance, a vertical rectangular (non-square kernel) can be used.

[0289] Kernel 274 spans a single pixel because the output pixel location is within a quarter pixel distance of the center of the pixel. In this circumstance, the value of the sensor pixel itself is a good approximation for the value of the output pixel.

[0290] It will be understood that the four kernel shapes in Fig. 27 are exemplary only. In other embodiments, kernels of the same or other shapes can be used in the same or other circumstances.

[0291] Desirably, the first processor 30a instructs the control circuitry 33 to fetch only the clusters of sensor pixel values required for the interpolations to be performed by the second processor, e.g., 16 pixel values to compute kernel 271.

[0292] The choice of kernel weights is left to the artisan. Commonly the weights are symmetrical around both diagonals, but this is not required. An illustrative set of kernel weights for kernel 271 is:

[0293] Imagers in the camera head may provide image frames at 60 or 100 frames per second. But the producer may request an output stream at a lower frame rate, e.g., 30 frames per second. In this case, the first processor may identify and fetch sensor pixel data from only certain of the frames, such as from every second frame. (Recall that memory 32a stores pixel data for more than a single frame of image data.)

[0294] Sometimes sensor pixel values from two (or more) frames are interpolated to yield a requested output stream frame rate. Consider the timing diagram of Fig. 28, showing the instants of sensor frame capture along an upper time line. The bottom time line shows a slower output frame rate requested for delivery to one of the clients.

[0295] In this case, output frame 1 A can be generated solely from the pixel data in sensor frame 1.Output frame 2A can be produced from a weighted average of spatially-corresponding pixel data in sensor frames 2 and 3 (e.g., using a one-third, two-thirds weighting). Output frame 3A can be produced from a weighted average of spatially-corresponding pixel data in sensor frames 3 and 4 (e.g., using a two-thirds, one-third weighting). Output frame 4A can be generated from the pixel data in sensor frame 5 alone. And so on.

[0296] Given the tremendous volume of pixel data being captured and processed by a multi-sensor camera head, applicant has found it advantageous to limit the amount of data processed and to process it in the simplest manner practical. These are termed "processing economies." The use of floating-point processing to determine the output pixel locations but use of integer processing to perform the kernel operations is one example of a processing economy. System operation is speeded and heat is reduced, since integer operations are faster and consume less energy than floating point operations.

[0297] Further processing economy is gained by use of interpolation kernels of different shapes, depending on the precise location of output pixels relative to the locations of sensor pixels (as discussed in connection with Fig. 27). Interpolation based on 12 pixels instead of 16 is a savings. So is occasional unity interpolation based on a single pixel. Likewise with frame rate conversion.Producing some output frames based on two sensor frames and others based just on one sensor frame is a further processing economy.

[0298] Fig. 29 shows a further level of detail of an illustrative system 291.

[0299] At the top, the system receives view-specifying data for multiple different video output streams for multiple different view clients. One specified video stream might track the quarterback of a football game and call for 1080p image data at 30 frames per second. A second might track the quarterback but call for 4K image data at 60 frames per second. A third may track the referee whooversees the game, at 1080p and 30 fps. A fourth may follow the head coach on the opposite sidelines, at 4k / 30 fps. Other streams may be specified by guests in premium suites of the stadium for presentation on displays in such suites. Still other streams may be specified by patrons at remote immersive viewing venues, such as COSM facilities. Etc., etc. Concurrency control techniques and other methods for avoiding memory collisions can be employed when large numbers of viewing clients are accessing the same data from memory at the same time.

[0300] The first processor 30a takes the pan / tilt / zoom parameters associated with each requested client stream and, with use of the zoom value, determines the location of the upper left pixel of the output frame in angular space and then the locations of all the other pixels in the output frame (e.g., as angular or pixel offsets from the initially-determined output pixel location). From these output pixel locations, the first processor determines which clusters of sensor pixel data stored in the aggregate scene memory will be necessary for interpolation and instructs the memory control circuitry to fetch those pixel values.

[0301] Additional processing economies can be realized by exploiting block-read capabilities of semiconductor memories. Instead of fetching individual sensor values needed for the interpolation operation from the aggregate scene memory, system operation is made faster and lower power by fetching a block of sensor values and copying them from the scene memory to a smaller cache memory 292.

[0302] Consider, again, Fig. 27. It will be seen that the sensor pixels needed for the interpolations are stored within swaths of memory locations oriented along the output rows, e.g., rows 275 and 276. These swaths are angled in Fig. 27, but such data can be captured within a stair-stepped pattern of elongated horizontal pixel blocks, sometimes termed ribbons.

[0303] Fig. 30 shows stairsteps of elongated horizontal pixel blocks 301 forming a ribbon. Each block may be, e.g., 32 pixels wide and 8 pixels high. Block read instructions enable such a block of data to be fetched from the aggregate scene memory nearly as economically as a single pixel value can be read from the memory. Yet each such block contains sensor pixel values for, e.g., one or two of the interpolation kernels shown in Fig. 27. Thus, instead of fetching 16 sensor pixel values for kernel 271 and 12 sensor pixel values for kernel 272 (i.e., 28 memory fetches), a single fetch is made from the aggregate scene memory to retrieve an 8 x 32 pixel block.

[0304] These blocks are written to the cache memory 292. (The system may include multiple cache memories, each serving one or more different output stream clients, but for simplicity's sake, only one cache memory 292 is shown.) The individual sensor pixel values needed for interpolation are then fetched from the cache memory and processed by the second processor 30b to yield pixel values for the output streams.

[0305] The 8 x 32 pixel block size is illustrative only, and different sizes can naturally be used in different implementations. In some embodiments, a block is only two pixels high and may correspond to an excerpt from a horizontal row of Bayer cells from a sensor. While in many embodiments, de- mosaicing is performed on Bayer cell data before the pixel data is written to the aggregate scenememory, in other embodiments the scene memory stores Bayer cell data prior to de-mosaicing. De- mosaicing can then be performed in a later stage of the system. (In like fashion, other corrections such as white balancing, color correction and gamma correction are commonly made before storage in the scene memory, but some embodiments may defer such adjustments to a later stage.) In still other embodiments, a block is only a single pixel high and comprises just an excerpt of a pixel row, e.g., of length 8, 16 or 32 pixels.

[0306] In Figs. 26 and 29, the first processor 30a may be a field programmable gate array (or ASIC) configured to calculate, e.g., the floating-point locations of requested output pixels in angular and / or pixel space, while the second processor 30b may be a graphics processing unit suited to repeated application of fixed interpolation kernels to image data e.g., by multiple integer multiply / add operations.

[0307] The kernel parameters applied in the interpolation operations are commonly fixed in advance (such as the 4 x 4 kernel matrix given above). In this case, rather than executing an interpolation operation for each output pixel, a look-up table can be used instead. That is, the interpolation operation is pre-computed for each possible combination of input data, and the results stored in a table in association with the input data. Interpolation then becomes a matter of performing a look-up in the stored table to determine the output value corresponding to a given set of input values. (If, as is usually the case, the interpolation kernel is symmetrical, then the input space of possible states is greatly reduced. For example, the pixel values at the four comers of the 4x4 pixel block are all commonly multiplied by the same parameter, which is often. With 8-bit imagery, there are 256 x 256 x 256 x 256 different combinations of the four comer pixel values. But since they are all weighted by the same value of 1, there are only 256+256+256+256 different states. Similarly for other pixels in the 4x4 input block that are all weighted by a common factor.)

[0308] In some embodiments, the second processor 30b is simplified by performing ID interpolations instead of 2D interpolations to produce output pixel values. This can comprise using the value of a first sensor pixel nearest the computed output pixel location together with one or more other sensor pixels on each side of the first pixel (i.e., in the same row). Again, a lookup table approach can be used. In still other embodiments, interpolation and table lookups can be avoided altogether by assigning to each output pixel the value of the sensor pixel nearest to it in angular location (i.e., a nearest-neighbor approximation).

[0309] It will be noted that the rows of output pixels 275 and 276 shown in Fig. 27 do not follow in the direction of rows of sensor pixels stored in the depicted excerpt of the scene memory. This situation sometimes arises in embodiments of the present technology and merits elaboration.

[0310] Consider, for example, the fields of view of the three lower imagers in Fig. 24. The field of view of the left imager 241 is tilted to the left. The field of view of the right imager 242 is tilted to the right. As a consequence, the sideline of the football field depicted towards the bottom of imagery captured by these imagers is not parallel to the bottom of their image frames. That is, the field sideline is not parallel to their sensor rows.

[0311] The tilting of the fields of view is due to the orientations of the three lower imagers. All have a positive tilt — both relative to the physical horizon and relative to the tip / tilt origin denoted by the bullseye. That is, they are tilted down towards the field. Moreover, each of the imagers is pointed in a different azimuthal (pan) direction. Given this geometry, the side edges of the frames are not parallel to each other, and neither are the bottom edges.

[0312] If a client requests a video stream that spans imagery from both the lower right imager 241 and the lower center imager 243, this discrepancy between relative orientation of the field sideline in the two imager fields of view must be addressed.

[0313] Fig. 31A shows such a case. A client has requested delivery of a content stream shown by the dotted rectangle. This virtual field of view incorporates imagery captured by imager 241 and imagery captured by imager 243. Yet features that are parallel in the physical world (e.g., the sideline depicted in both imagery) are not parallel in pixel space of the two imagers.

[0314] To address this discrepancy, output pixels generated for the left frame, 241, are inclined relative to pixel rows in that frame. Fig. 3 IB illustrates — by the dashed outline — the part of the stored imagery that is needed for the left part of the requested output stream. The orientation of the image frame captured by imager 241 is shown by the bold lines. The arrows show the direction of the pixel rows captured by imager 241. The lines within the dotted area show that the output pixels generated from this data are taken in a direction that is inclined relative to the sensor pixel rows. That is, the interpolation operations that generate output pixels in a given row of output imagery draw from a swath of pixels in the scene memory that is tilted relative to the rows of imager 241.

[0315] Fig. 31C show the imagery captured by imager 243. No tilting of the output pixel rows generated from this imagery, relative to the sensor pixel rows, is needed. Instead, the interpolation operations that generate output pixels in a given row draw from a swath of pixels in the scene memory that is parallel to the rows of imager 243.

[0316] The area of overlap between the imagery captured by imagers 241 and 243 can be handled in different ways. One way is to interpolate two output pixels for each pixel position in the overlap portion of the requested client stream — one using sensor data from imager 241 and one using sensor data from imager 243. A weighted average can then be computed from each pair of pixels, with the weights dependent on the output pixel's position within the overlap region. For example, if an output pixel is at a position 10% in from the left edge of the overlap region, the output pixel generated from imagery 241 data can be weighted 90% and the output pixel generated from imagery 243 can be weighted 10%. Other blending algorithms, e.g., as known from panorama stitching, can alternatively be employed, such as Laplacian pyramid blending.

[0317] Sometimes, output pixel rows that are not parallel to sensor rows may be requested for an output stream even if the client-requested imagery is sourced entirely from one sensor. Fig. 32 helps illustrate.

[0318] The bore axis of this imager (indicated by the bullseye) is oriented well above the field. In the lower left quadrant of the image there is action around the goal. If the camera was pivoted to the leftand downwards, this action could be repositioned to the center of the frame, near the bullseye. If the camera was physically pivoted left and down, the depiction of the goal in the sensor imagery would rotate a few degrees clockwise. (If doubtful, consider the wall with display advertising at the end of the field, behind the goal. In the Fig. 32 depiction, this wall extends up and to the right at an angle of about 15 degrees relative to the bottom edge of the frame. If, however, the camera is pivoted to the left, the wall would become progressively closer to parallel to the bottom edge of the frame. Its depiction would rotate clockwise with pivoting of the camera.)

[0319] When a horizontal surface (e.g., a sports field) is viewed obliquely from above by an imager, the imager's boresight view angle can be regarded as a boresight vector, defining one direction, or dimension, in a 3D reference frame. Where the boresight vector intersects the field can be regarded as the origin of this 3D reference frame. A second dimension of this 3D reference frame can be up, vertically, in the direction of the gravity vector. And the third dimension can be the direction perpendicular to the first two dimensions, i.e., in the plane of the field and perpendicular to the boresight vector. This third dimension can be termed a virtual horizon. Rows of imagery captured by the imager are parallel to this virtual horizon.

[0320] In most embodiments of the present technology, the camera head does not pivot; it is fixed. So the imager cannot be repositioned to bring the goal in Fig. 32 to the center of the frame. But a similar effect can be achieved by tilting the output image rows relative to the sensor rows.

[0321] This is shown in Fig. 33. A client has requested a closeup shot centered on the goal. In the Fig. 32 image, the goal is located several degrees (in angular space) to the left and down of the imager boresight. To produce an image akin what would be captured if the camera was pointed with its boresight toward the goal, a view as indicated by the dashed rectangle can be defined. The rows in this output image are tilted up, to the right, relative to the sensor rows. The optimum amount of tilt can be determined by the angular field of view of the imager and the position of the virtual boresight within this angular space. However, the human eye is not very discriminating of this effect, so accuracy is not critical. The output rows in the illustrated dashed rectangle view are tilted three degrees relative to the sensor rows. Such framing of the output stream effects a three degree virtual clockwise rotation of the goal relative to the image frame — simulating the effect of pivoting the camera to point towards the goal.

[0322] In the examples of Fig. 33 and 3 IB, output row pixels are interpolated from sensor samples taken around straight lines in the sensor pixel space, tilted relative to the sensor rows (e.g., lines 275 and 276 in Fig. 27).In other embodiments, the output rows can be interpolated from sensor samples taken along non-straight lines in the sensor pixel space, e.g., lines defined by polynomial equations. Such sampling is useful, e.g., to address lens distortions, which have curved aspects.

[0323] In dealing with multiple viewing clients that desire different qualities of output video, it can be helpful to process imagery from the different imagers into different pyramidal encodings and to store the different outputs in different scene memories for retrieval, as needed, by different clients.

[0324] Fig. 34 is an excerpt of such a system 341, taken from Fig. 29 but modified to enable use of pyramidal encoding. Pyramidal encoding exploits the fact that the human eye is more sensitive to brightness (luma) variations than color (chroma) variations, so color data (commonly Cb and Cr channels) can be stored at lower spatial resolutions. This is sometimes termed chroma subsampling.

[0325] In the illustrated arrangement, imagery from each sensor is applied to a pyramidal encoding module 343 that outputs video data in multiple encodings, which can be stored separately. One encoding can be 4:4:4. That is, the luma and chroma values of each imager-sensed pixel are maintained and stored in an area 342a of the aggregate scene memory devoted to 4:4:4 data (i.e., 24 stored values for a 4x2 pixel block).A second encoding can be 4:2:2.That is, the luma information is maintained but the chroma resolution is halved horizontally and maintained vertically (i.e., 8 luma, 4 Cb, and 4 Cr values per 4x2 pixel block), with the resulting data stored in a corresponding area 342b of the scene memory. A third encoding can be 4:2:0. That is, the luma information is maintained but the chroma resolution is halved horizontally and vertically (i.e., 8 luma, 2 Cb, and 2 Cr values per 4x2 pixel block), with the resulting data stored in a corresponding area 342c of the scene memory. The control module can then retrieve stored data, e.g., to interpolate values at floating-point pixel coordinates, from corresponding areas of the scene memory in accordance with the requirements of the requesting clients.

[0326] While not shown, a similar approach can be used to additionally, or alternatively, implement a multiscale pyramid. That is, the native imagery from each image sensor can be blurred and sampled, usually by a factor of 2 along both rows and columns, to generate counterpart images at different spatial resolutions. For example, a multiscale pyramid can include one level at the full sensor resolution, a second level at 1 / 4 the sensor resolution, and a third level at 1 / 16 the sensor resolution. Data for each representation can be stored in a corresponding area of the aggregate scene memory. The control module can then retrieve stored data from different such memory areas depending on the zoom requirements of the requesting clients, e.g., with wide angle view data retrieved from the 1 / 16 resolution data and closeup view data retrieved from the native resolution data.

[0327] Sample Embodiments

[0328] One sample embodiment 351 employs a camera head mounted behind home plate of a baseball diamond. The head includes eight imaging modules, identified in Fig. 35 by their respective fields of view. Two identical camera modules 352a, 352b have overlapping fields of view that collectively span the playing area and elements of the surrounding stadium. The lenses of these modules have relatively short focal lengths. A series of five other camera modules 354a-354e have overlapping fields of view that also collectively span the playing area but with relatively longer, intermediate, focal lengths to yield better definition of players on the field (e.g., at a subject sampling resolution of 1 pixel per 2 mm of height for a player at the batter box). A final camera module 356 is directed to the pitcher's mound with a still-longer focal length to yield still-better definition of the pitcher than is provided by camera 354c.

[0329] Note that the field of view of camera module 356 is totally encompassed within the field of view of another camera module, 354c of a shorter focal length (i.e., it is 100% overlapped). In other implementations, depending on placement of the camera head, the overlap between modules 356 and 354c may be less than 100% but above the 0.5-15% overlap by which fields of view 352a / 352b and 354a-354e overlap (e.g., more than 30%, 60%, or 80% overlap).

[0330] Relatedly, the fields of view of two camera modules, 354a and 354b, are totally encompassed within the field of view of another camera module, 352a of shorter focal length. And the camera module 354c field of view is partially (40%+) encompassed within the field of view of camera module 352a. Thus, part of the playing field (shown by crosshatch in Fig. 35) is depicted in imagery captured by three different camera modules, each employing a lens of a different focal length. So, too, the right side of field of view 356 (which is overlapped by camera modules 354c and 352b). Taken together, all of the scenery captured by camera module 356 with its long focal length lens is also captured by modules having intermediate and short focal length lenses. Three depictions of such region of pixels are thereby provided with different subject resolutions, e.g., of the pitcher.

[0331] Such an arrangement may be characterized as one including first, second and third camera modules of first, second and third types, respectively. These three types of modules respectively have first, second and third focal lengths and first, second and third fields of view. The third focal length is longer than the second focal length and the second focal length is longer than the first focal length. The modules are oriented so that an area within the third field of view is also within the first field of view and within the second field of view.

[0332] In the illustrated implementation of system 351, the field of view of the third camera module is entirely encompassed within the field of view of the second camera module, while the field of view of the second camera module is not entirely encompassed within the field of view of the first camera module.

[0333] Note, further, that each of the fields of view shown in Fig. 35 is square. This is found desirable because elongated, rectangular fields of view require entry of light into the camera module lens from angles that are relatively remote from the lens axis, increasing lens distortions. Either a square image sensor can be employed, or a square area of pixels in the center of a rectangular image sensor can be employed. (In other implementations, one-, two-, or more-camera modules in the camera head provide square fields of view but other modules — whether a majority or minority of all the camera modules in the head — can provide rectangular fields of view.)

[0334] As noted, the lens used in camera module 356 has a longer focal length than the lenses used in camera modules 354a-354e, which in turn have longer focal lengths than the lenses used in camera modules 352a / 352b. The image sensors employed in these three classes of camera modules can all be identical, or two or all three classes of modules can employ image sensors of different types.Additionally, or alternatively, the different classes of camera modules can capture image frames at the same rates, or different rates can be employed. For example, imagery of the pitcher from cameramodule 356 may be captured at 180 fps, versus 120 fps for imagery captured by modules 354a-354e, versus 30 fps for imagery captured by modules 352a / 352b.

[0335] In a variant embodiment there are one or more additional camera modules that each includes the longer focal length lens of module 356. One such additional module is oriented to capture imagery of first base. Other such modules may be oriented to capture imagery of second base and third base.

[0336] In one application, system 351 is used to capture data of a Little League baseball game. After conclusion of the game, the stored data is processed to extract player "reels." Each reel comprises video from a zoomed region of interest excerpted from the stored data that tracks a single player for all the time that he / she is on the playing field.

[0337] Another exemplary system 361 is shown in Fig. 36 and is arranged to capture imagery of a basketball game. Again, the system includes camera modules of three types, or classes, respectively having three different focal lengths.

[0338] The first type comprises a pair of camera modules 362a / 362b having relatively short focal length lenses. These camera modules capture fields of view that span substantially all of the playing area. In practice, these modules often serve to provide imagery of near-court action, so their focus planes are set accordingly (e.g., at a distance of 15 feet).

[0339] The second class of camera modules, 364a-364h, has relatively longer, intermediate focal length lenses. These are arrayed so their fields of view are clustered around, but not including, the far basketball backboard. Camera modules of this second class commonly have focus planes set to capture mid-court action, often at a distance two-or-more times that of modules in the first class, e.g., 40 feet.

[0340] The third class of camera modules, 366a-366c, has still longer focal length lenses. These are directed to the far end of the court and have focus planes set to capture far-court action, often at a distance two-or-more times that of modules in the second class, e.g., 90 feet.

[0341] Unlike just-discussed system 351, no cameramodule of the third class (i.e., with the longest lenses) has a field of view that is totally encompassed by a camera module having an intermediate length lens (i.e., a camera module of the second class). While the fields of view of third class modules 366a-366c are each partially-overlapped by one or more fields of view of second class modules (including majority-overlapped, in the case of 366b overlapped by 364c), none is totally overlapped.

[0342] Indeed, some parts of the collective field of view captured by the long lens modules (of the third type) are captured by none of the intermediate lens modules (of the second type). Fig. 37 shows the fields of view of the intermediate lens modules and highlights that there is a central gap 371 in their collective coverage. This area of the scene is imaged only by modules having the longest and shortest lenses.

[0343] Embodiment 361 is related to embodiment 351, e.g., by including modules of three types of different focal lengths, where an area within the field of view of a module of the third type overlaps (i.e., is within) the fields of view of modules of the first and second types. However, embodiment 361may further be characterized as including at least two modules of each of the first, second and third types.

[0344] The illustrated implementation may additionally be characterized as including a region (gap 371) that falls within the collective fields of view of camera modules of the third type but is outside the collective fields of view of camera modules of the second type.

[0345] In the depicted implementation, the areas of overlap between the collective fields of view of the third modules and the collective fields of view of the first and second modules form a closed, connected region 381 that surrounds and defines gap 371. This region of overlap between modules of the three types is shown in crosshatch in Fig. 38 (with the boundaries of the third fields of view bolded, to avoid confusion).

[0346] As with the implementation of embodiment 351, the camera modules used in the depicted version of embodiment 361 all have square fields of view, although this is not essential. Similarly, the camera modules of embodiment 361 can have different frame rates, with the long lens modules operating at frame rates higher than the intermediate lens modules and the short lens modules operating at frame rates lower than the intermediate lens modules (e.g., 180, 120 and 30 i s).

[0347] In particular implementations, the image sensors used in embodiments 351 and 361 may be Sony IMX536 devices, which are CMOS sensors capable of capturing up to 180 frames per second. The long lens camera modules of the third type can employ f2 lenses of 60mm focal length. In embodiment 361, this arrangement causes such an imaging module to produce a far court image depicting a 13 x 13 foot area, sampling a seven foot tall player across 1540 pixels (better than 1 pixel per 1.4mm of player height). As action moves towards the camera head, the players are depicted larger in the frame, yielding higher subject resolutions. At mid-court, the players are at a distance where camera modules of the second type, with intermediate-length lenses, yield subject resolution that is again better than 1 pixel per 1 ,4mm of player. At near court, camera modules of the first type provide such resolution.

[0348] In embodiments 351 and 361, lenses may be selected from multiple focal-length classes to provide complementary capture characteristics across the view volume. In representative configurations, "longer" focal -length lenses typically include focal lengths greater than approximately 30 mm, 40 mm, 50 mm, or 60 mm. "Shorter" focal -length lenses may include focal lengths below approximately 20 mm, 15 mm, 12 mm, or 8 mm. Between these ranges, intermediate focal lengths can be employed, such as lenses having focal lengths of about 10-15 mm, 15-22 mm, 22-30 mm, 30-40 mm, or 40-55 mm.

[0349] By way of example, one implementation employs three lens classes corresponding to focal lengths of 60 mm (long), 24 mm (intermediate), and 12 mm (short). Another uses focal lengths of 50 mm, 25 mm, and 6 mm, respectively. These combinations allow the array camerato capture both fine distant detail and wide-angle contextual coverage, enabling downstream digital zoom at arbitrary points throughout the depth of field.

[0350] In some variants, the intermediate-length optics themselves may comprise two distinct focal lengths. These two intermediate focal lengths may be paired with different sensor types to optimize overall system performance. For instance, one intermediate-length lens may be mounted to a color (RGB) sensor while another intermediate-length lens is mounted to a monochrome (panchromatic) sensor. In one variant, the longer of the two intermediate focal lengths is paired with the color sensor; in another, the longer focal length is paired with the monochrome sensor to maximize low-light sensitivity, spatial acuity, and Modulation Transfer Function (MTF).

[0351] A particular implementation uses a 25 mm lens with a color sensor and a 12 mm lens with a monochrome sensor. In another, the longer and shorter lenses may be 50 mm and 6 mm, respectively, with either assigned to a mono or color sensor depending on the desired balance of resolution, sensitivity, and color reconstruction requirements. Monochrome sensors may be preferred for certain modules because they provide higher photon efficiency, superior detail rendition, and higher MTF. Color values for imagery captured by such mono modules can be inferred or transferred from contemporaneously captured color data from neighboring modules.

[0352] In further embodiments, additional imaging modalities may be incorporated. These include modules operating at variable frame rates (e.g., high-speed capture for motion-estimation or depthfusion), modules equipped with polarization filters (e.g., for glare suppression or surface-normal estimation), and multispectral or hyperspectral modules (e.g., for enriched colorimetry or materialreflectance discrimination). Any such combination of focal lengths, sensor types, and imaging modalities may be used within the array camerato achieve full-field capture of sports venues with the ability to digitally zoom into any region of interest at any depth within the reconstructed view volume. When embodiments such as 351 and 361 provide image data from different types of camera modules (captured through lenses of different focal lengths) into an aggregate scene memory (e.g., as in Fig. 34), different of the stored data will have been captured at different physical subject scales. Metadata is stored with the image data to indicate which data was captured through which type of camera module. Control circuitry that governs retrieval of the data, e.g., for interpolation to yield pixels of a client-requested output stream, then adjusts the readout of data from the scene memory in accordance with this metadata. For instance, the control circuitry uses relatively larger X- and Y- offsets when selecting data captured with relatively longer focal length lenses. This assures that the subject scale represented in the streamed output data stays consistent, even if, for part of a pixel row, data captured by a long lens module (third module type) is used, while for another part of the same pixel row, data captured by a shorter lens module (e.g., first module type) is used.

[0353] Similarly, data written to the aggregate scene memory from different camera modules may have been captured at different frame rates. Again, stored metadata indicates the frame rate used during video capture, so the control circuitry can sample the stored data appropriately to generate the client- requested output stream. Thus, a row of pixels represented in an output stream may be based on some data collected at a high frame rate and other data collected at a low frame rate.

[0354] Applicant's patent application No. 19 / 234,706 describes array-camera architectures that perform temporal scanning or slicing of a view volume by sweeping the focal, spectral, and / or temporal sampling parameters of an array camera across multiple exposures to acquire slices of the multidimensional data cube representing the scene. As disclosed, a tunable fdter may be sequentially stepped to capture different spectral planes, and focal sweeping may be used to capture successive focus planes across the depth of field, thereby producing a time-multiplexed focal stack covering different z-planes of the view volume. These sequential measurements are then used in computational reconstruction of the 3D optical data cube, enabling depth-resolved imaging of complex scenes.

[0355] Applicant's related patent application No. 63 / 880,317, entitled, "Integrated Imaging Module with Separated Static and Movable Lens Elements for Enhanced Focus Control" discloses Integrated Imaging Modules (IIMs) designed for use in such array cameras. The IIM incorporates a lightweight movable lens element positioned closest to the sensor, actuated by a voice coil motor (VCM) capable of rapid focus adjustment at frequencies of 100 Hz or greater. By separating the optical power between a heavier static lens assembly and a low-mass movable element (e.g., 25-250 mg) and using high-speed VCM actuation, the IIM enables fast, precise, and repeatable focal transitions across the depth of field. This design allows an array of such IIMs to perform temporal focal scanning, sweeping through the focal planes of the view volume at high speed and thereby capturing depth-indexed image slices suitable for reconstruction of the 3D scene structure. The combined teachings of 19 / 234,706 and 63 / 880,317 therefore provide array cameras with a practical, high-speed mechanism for time- multiplexed depth capture using integrated modules capable of dynamic focus control across successive exposures.

[0356] Figure 39 shows an exploded perspective view of the integrated imaging module, illustrating the complete assembly sequence and component relationships. This view demonstrates how a static lens assembly 326a, 326b, and 326c, movable lens element 328, voice coil motor 330, VCM mount 340, and sensor board 332 are arranged and connected to form the complete integrated imaging module.

[0357] The exploded view reveals the hierarchical assembly structure, with the static lens assembly positioned at the top containing lens elements 326a, 326b, and 326c within lens barrel 334. Below the static lens assembly, the movable lens element 328 is shown in its operational position between the static assembly and the sensor 322.

[0358] The voice coil motor 330 surrounds the movable lens element 328 and provides the electromagnetic actuation necessary for focus control. The movable lens element 328 has a mass optimized for rapid voice-coil motor actuation, typically in the range of 25 to 250 milligrams, with a preferred range of 30 to 50 milligrams. This mass range is specifically matched to the capabilities of commercial off-the-shelf voice coil motors designed for mobile phone applications, enabling the system to leverage existing voice-coil motor technology while achieving focus adjustment frequencies of 100 Hz or greater. The movable lens element 328 may be constructed from glass or can alternatively be fabricated from plastic materials to further reduce mass and enable even faster focus response.

[0359] The VCM mount 340 provides structural support and precise positioning for the voice coil motor 330, ensuring proper alignment with the optical axis while facilitating the integration of the voice coil motor 330 with the sensor board 332. The VCM mount 340 serves as an interface component that mechanically couples the voice coil motor 330 to the overall assembly structure, providing stable mounting points that maintain the positioning relationships between the movable lens element 328 and the sensor 322. The mount 340 is designed to accommodate the electromagnetic operation of the voice coil motor 330 while providing the mechanical rigidity necessary to maintain optical alignment during focus operations. Additionally, the VCM mount 340 facilitates the electrical connections between the voice coil motor 330 and the focus control electronics on the sensor board 332, enabling the three-wire interface that provides power and control signals for the rapid focus adjustment capabilities of the system.

[0360] The sensor board 332 forms the foundation of the assembly, supporting both the sensor 322 and providing the electrical interface for the voice coil motor 330 through the integrated focus control electronics. The mounting structure 340 provides additional mechanical support and alignment features that enable the active alignment process during manufacturing. This exploded view illustrates how each component contributes to the overall system performance while maintaining the critical separation between the heavier static lens assembly 326a, 326b, and 326c and the lightweight movable lens element 328 that enables rapid focus adjustment capabilities exceeding 100 Hz.

[0361] Array-camera embodiments may employ fixed-focal-length IIMs, variable-focal-length IIMs with high-speed focus control, or heterogeneous combinations of both, to optimize sampling of the 3D optical data cube. By integrating modules capable of rapid focal transitions alongside modules configured for fixed, high-quality imaging at specific depths or fields of view, the array camera can efficiently acquire depth-resolved image data throughout the view volume. This architectural flexibility enables fast and efficient capture, storage, and retrieval of image information from arbitrary regions within the view volume, supporting dynamic imaging tasks, adaptive focal scanning, and computational reconstruction of scene geometry and appearance.

[0362] Array-camera embodiments may employ fixed-focal-length IIMs, variable -focal-length IIMs with high-speed focus control, or heterogeneous combinations of both, to optimize sampling of the 3D optical data cube. By integrating modules capable of rapid focal transitions alongside modules configured for fixed, high-quality imaging at specific depths or fields of view, the array camera can efficiently acquire depth-resolved image data throughout the view volume. This architectural flexibility enables fast and efficient capture, storage, and retrieval of image information from arbitrary regions within the view volume, supporting dynamic imaging tasks, adaptive focal scanning, and computational reconstruction of scene geometry and appearance.

[0363] The array camera system architecture comprises three principal components that work in concert to support client view requests within the view volume. The first component consists of a set of N imagers, each providing a distinct video stream under the principle of diversity. These streams vary in spatial region, focal length, frame rate, and sensor type (monochrome or color), collectively providingdiverse capture of the full view volume of a playing field. Each imager contributes unique characteristics to the overall system capability, with some modules optimized for wide-angle contextual coverage while others focus on high-resolution detail capture of specific regions.

[0364] The second component is an image stream fusion system that provides a geometrically defined space of the view volume. This fusion component encompasses the projections of the fields of view of each of the imagers into a unified geometrically defined coordinate system. The fusion system maintains spatial relationships between overlapping and adjacent fields of view, enabling seamless integration of imagery from multiple sources with different optical characteristics. This geometric framework serves as the foundation for virtual camera positioning and view synthesis within the captured volume.

[0365] The third component comprises a view request processing system for receiving requests from M view clients seeking specific views from within the geometrically defined space. Upon receiving a client request, this component fetches relevant source data from the input streams stored in buffers and synthesizes the requested view. The system addresses memory contention issues that arise when multiple view clients simultaneously request views of overlapping imager output from short-term memory. This is accomplished through intelligent buffer management, priority queuing, and efficient data sharing mechanisms that minimize redundant memory access while maintaining real-time performance for all connected clients.

[0366] Summarizing and conceptualizing these three components is helpful: Component 1 is the N- plural sourcing of motion-visual information of a scene where numerous imagers spatially overlap in their visual sampling of the scene, with each imager possessing a unique and well known geometric relationship with the scene and with all companion, cooperative imagers; Component 2 is the parallel ingestion and temporary storing of simultaneous N streams of visual data, keeping such data streams discrete, and processing these streams in an efficient manner for subsequent view-client data fetching; and Component 3, the setting up of M so-called ‘viewing agent processes’ or simply ‘viewing agents’ that represent the real-time viewing desires of a multitude of viewers and broadcast clients.Component 3 then directs very specific and geometrically guided memory fetching from the temporary storage locations within Component 2.

[0367] In some implementations, a viewing agent process further includes analysis logic configured to infer scene structure, object depth, or subject distance based on multi-view image data retrieved from the temporary storage. Such inference may be performed using geometric triangulation, disparity estimation, machine learning models, or combinations thereof. The viewing agent may generate distance or depth estimates associated with one or more subjects or regions of interest within the scene. In some implementations, the viewing agent communicates derived depth or distance information to a capture control subsystem of the array camera system. The capture control subsystem uses such information to adjust parameters of time-multiplexed depth capture, including the sequencing, timing, or waveform governing dynamic focus control of one or more integrated imaging modules. In this manner, viewer intent and scene-derived depth information influence thetemporal focal scanning behavior of the array camera system. For example, in a sporting event scenario, a viewing agent may identify one or more players, estimate player distance from the camera array over time, and provide such distance information to the capture control subsystem to bias or synchronize focus-cycling waveforms with anticipated player motion.

[0368] The integration of these three components enables the array camera system to serve multiple concurrent clients with customized views extracted from the comprehensive view volume capture. Each client can specify viewing parameters such as pan, tilt, zoom, frame resolution (e.g., SD, HD, 4K, etc.) and frame rate, while the system dynamically sources the optimal combination of imager data to fulfill each request. This architecture supports applications ranging from broadcast production to individual viewer experiences in stadium suites, where each location can access personalized views of the sporting event.

[0369] This three -component architecture is supported by the existing array camera infrastructure described throughout this specification. The diverse imager set leverages the non-uniformly spaced imaging modules detailed in Figs. 8-15, where modules with different focal lengths, sensor dimensions, and angular orientations collectively capture the full view volume. As described in connection with Fig. 10, imaging modules are configured with progressively different azimuthal angles and focal lengths optimized for varying scene distances, ensuring comprehensive spatial coverage with subject sampling resolution of at least 1 pixel per 2 mm across the playing field.

[0370] The image stream fusion component builds upon the geometrically defined coordinate system established by the camera head architecture shown in Figs. 22-25. The bullseye marker serves as the geometric center of the camera head's pixel rays, with each pixel mapped to a direction within the coordinate system. This unified framework enables the seamless integration of imagery from multiple sources with different optical characteristics, as detailed in the processing system of Figs. 26-29.

[0371] The view request processing subsystem utilizes the dual-processor architecture described in Fig.26, where the first processor 30a receives client view specifications and generates floating point coordinates for sample points, while the second processor 30b compiles output frames through interpolation of fetched image data. This approach enables on-the-fly fusion of images from various imagers based on pre-computed mappings, where each pixel in the camera head is mapped to a direction within the coordinate system either computed on the fly based on sensor parameters and camera geometry or pre-computed and stored in data structures such as tables. The system performs real-time geometric transformation by determining angular coordinates of desired output pixels in floating point arithmetic, then fetching clusters of sensor pixel values from the aggregate scene memory and applying interpolation kernels to synthesize the requested view from multiple source imagers with different optical characteristics. Memory contention management is addressed through the concurrency control techniques and cache memory systems detailed in Fig. 29, enabling multiple clients to access overlapping imager output from the aggregate scene memory 32a without performance degradation. The subsystem's ability to serve diverse client requests is further enhancedby the pyramidal encoding capabilities shown in Fig. 34, which provides multiple data representations optimized for different viewing requirements.

[0372] Each of these subsystems can be implemented across various hardware components, including memory units, field programmable gate arrays (FPGAs), processors within the camera head, or servers remote from the camera head, providing architectural flexibility for different deployment scenarios and enabling scalable performance optimization based on specific application requirements.

[0373] Depth of Field, as a generically defined property of real imagers and real lens systems, remains an important design consideration for array cameras in general. In short, there is an unavoidable (if using lens / CMOS sensor imagers) trade-off between F-stop settings which tend to promote higher light gathering sensitivity for an imaging unit - the lower the F-stop number the better- and the focus properties of, for example, athletes at various distances from an imaging unit - a player at 30 feet distance may be in focus while a player 20 feet further in distance is slightly out of focus. A partial remedy for this trade-off involves higher frame rate imagers whereby the cycling of the Z-axis focal plane can happen many times per second. Component 1 of the trio of components must perform this physical cycling of the VCM (or other focusing mechanism of an imager), generally as instructed by Component 2 which contains a form of realtime scene-dynamic geometric knowledge of the locations of player-athletes, where then finally Component 3 with its M viewing clients are all putting in their requests to have, for example, player A in focus or player B in focus, while other viewing clients may wish to have the background fans in focus. The important consideration for inclusion of this depth of field dynamism is that our three component architecture needs to explicitly include inter-component communication and instructions in order to facilitate this extended depth-of-field feature of an array camera.

[0374] Figure 40 shows an embodiment of an array camera backplane 500 with a high-speed optical link to a rendering and storage PC, illustrating a distributed processing architecture that separates image capture from computationally intensive processing tasks. This embodiment enables the camera head to remain compact and power-efficient while leveraging remote computing resources for advanced image processing and long-term data storage.

[0375] The camera head portion of the system includes four sensor interfaces 502a, 502b, 502c, and 502d that receive image data from the imaging modules described in previous figures. Each sensor interface connects to backplane 506, which serves as the central interconnection substrate for the camera head electronics. The backplane 506 houses an FPGA system-on-module (SOM) 508 that performs initial image processing, data formatting, data routing, and communication protocol management.

[0376] In one embodiment, the sensor interfaces 502a, 502b, 502c, and 502d comprise MIPI CSI-2 interfaces that provide standardized high-speed serial communication between the image sensors and the FPGA SOM 508. The MIPI CSI-2 protocol enables efficient transmission of raw image data with minimal latency, supporting the high frame rates and resolutions required for industrial inspection applications. In an alternative embodiment, at least one of the sensor interfaces comprises an SLVS-EC interface configured to receive image data from an IMX926 image sensor. The SLVS-EC interface provides enhanced bandwidth capabilities that enable transmission of high-resolution image data at elevated frame rates, particularly advantageous for imaging modules requiring maximum data throughput. The backplane 506 is configured to support both MIPI and SLVS-EC interface types, enabling flexible configuration of the imaging system based on the specific sensors and performance requirements of each imaging module.

[0377] In one embodiment, the FPGA SOM 508 comprises a Zynq UltraScale+ FPGA, such as the Trenz XZU90-19EG or similar alternatives. The Zynq architecture combines programmable logic fabric with integrated ARM processor cores, enabling both hardware-accelerated image processing and software-based control within a single module. The programmable logic implements high-throughput operations including image synchronization, demosaicing, color correction, lens distortion correction, and data packetization for transmission, while the ARM processor cores execute sensor control, manage communication interfaces, and coordinate system operation. In some embodiments, the FPGA SOM 508 sends preprocessed image streams directly to an NVIDIA Jetson GPU over MIPI or PCIe interfaces for advanced processing tasks. Alternative implementations may include other Zynq UltraScale+ variants or comparable FPGA platforms that provide similar combinations of programmable logic and embedded processing capabilities suitable for real-time multi-camera image processing applications.

[0378] Control tablet 510 comprises a personal computer or tablet computer executing control software that provides a user interface for system configuration, monitoring, and control. The control tablet 510 communicates wirelessly with the camera head via communication module 512, which may comprise a Wi-Fi interface, a cellular interface, or both. This wireless connection enables operators to remotely adjust capture parameters, initiate calibration procedures, monitor real-time system status, and access diagnostic information without requiring physical connection to the camera head. The control tablet 510 may also communicate with the rendering and storage PC 520 to access stored images, configure processing parameters, and retrieve quality assurance reports.

[0379] A synchronization module 526 provides timing signals to coordinate image capture on the backplane 506 with rendering and storage operations on the PC 520. The synchronization module 526 generates master clock signals and trigger pulses that are distributed to each sensor interface, maintaining precise temporal alignment between the imaging modules during capture. The synchronization signals are also transmitted to the rendering and storage PC 520 via the optical connector 524, enabling the PC to synchronize its processing pipeline with the incoming image data streams and ensuring that frames from all imaging modules are processed and stored in proper temporal sequence.

[0380] The processed image data is transmitted from the camera head to a rendering and storage PC 520 via optical connector 524, which works in tandem with NVMe storage modules 522 through 100Gb optical data links. In this embodiment, the optical connector 524 is a multi-fiber push on (MPO) connector implementing 100Gb Ethernet communication between the camera head and PC with QuadSmall Form factor Pluggable (QSFP28) optical modules. The optical link provides high-bandwidth, low -latency communication that enables real-time transmission of the multi-megapixel image streams from all imaging modules simultaneously to the NVMe storage modules 522. The use of optical fiber transmission eliminates electromagnetic interference concerns and enables flexible placement of the rendering and storage PC 520 at distances up to several hundred meters from the camera head.

[0381] The NVMe storage modules 522 provide high-speed solid-state storage for the captured image data received via the 100Gb optical data links. The NVMe modules 522 enable sustained write speeds sufficient to record the continuous multi -camera video streams without frame drops or buffer overflows. The storage capacity may be configured based on application requirements, with typical configurations providing several terabytes of storage for extended recording sessions.

[0382] Power module 516 receives electrical power from power input 514, which may be provided via a standard industrial power connection such as Power over Ethernet or a dedicated power supply. The power module 516 distributes regulated power to the sensor interfaces 502a, 502b, 502c, and 502d, the FPGA SOM 508, and other electronic components on the backplane 506. Power management circuitry within the power module 516 ensures stable operation across varying load conditions and protects the system from power supply transients.

[0383] This distributed architecture provides several advantages for industrial inspection applications.By offloading computationally intensive processing tasks to the rendering and storage PC 520, the camera head remains compact and generates minimal heat, enabling installation in space -constrained production environments. The high-bandwidth optical link ensures that the full resolution and frame rate capabilities of the imaging modules are preserved during transmission, while the remote processing resources enable sophisticated image analysis algorithms that would be impractical to implement in the camera head. The separation of capture and processing functions also facilitate system upgrades, as improvements to processing algorithms or storage capacity can be implemented by upgrading the rendering and storage PC 520 without requiring modifications to the camera head hardware.

[0384] Figure 41 shows an alternative embodiment of an array camera backplane 500 with local storage, illustrating a self-contained processing architecture that eliminates the need for a separate rendering and storage PC. This embodiment enables the camera head to operate as a standalone unit, performing all image capture, processing, and storage functions within a single integrated system suitable for applications where remote processing infrastructure is unavailable or undesirable.

[0385] The camera head portion of the system includes four sensor interfaces 502a, 502b, 502c, and 502d that receive image data from the imaging modules, connecting to backplane 506 which serves as the central interconnection substrate. The backplane 506 houses FPGA SOM 508 that performs image processing, data formatting, and storage management functions. In this embodiment, the FPGA SOM 508 comprises aZynq UltraScale+ FPGA such as the Trenz XZU90-19EG, providing both programmable logic for hardware -accelerated image processing and ARM processor cores for system control and storage management.

[0386] In contrast to the embodiment shown in Figure 40, which transmits processed image data to a remote rendering and storage PC via optical connector 524, the embodiment of Figure 41 stores the image data locally on solid-state drives 532 mounted directly on the backplane 506. The solid-state drives 532 comprise NVMe storage modules that provide high-speed data storage capabilities, enabling the system to record continuous multi-camera video streams without requiring external storage infrastructure. The elimination of the optical connector and remote PC reduces system complexity and enables deployment in environments where network connectivity or remote processing resources are limited.

[0387] DDR memory 530 provides high-bandwidth temporary storage for image data during processing operations. The DDR memory 530 serves as a buffer between the sensor interfaces 502a, 502b, 502c, and 502d and the solid-state drives 532, enabling the FPGA system-on-module 508 to perform image processing operations on captured frames before writing the processed data to persistent storage. The DDR memory 530 capacity is sized to accommodate multiple frames from all imaging modules simultaneously, preventing buffer overflows during peak data rates and ensuring smooth operation during continuous recording sessions.

[0388] Communication module 512 provides wireless connectivity to control tablet 510, enabling remote configuration and monitoring of the camera system without requiring physical connection. The communication module 512 may comprise a Wi-Fi interface, a cellular interface, or both, allowing operators to access system controls, adjust capture parameters, and retrieve stored images from the solid-state drives 532 via wireless communication. This wireless interface enables flexible system deployment and operation while maintaining the self-contained nature of the camera head.

[0389] Power module 516 receives electrical power from power input 514 and distributes regulated power to all electronic components on the backplane 506, including the sensor interfaces 502a, 502b, 502c, and 502d, the FPGA SOM 508, the DDR memory 530, and the solid-state drives 532. The power management circuitry within power module 516 ensures stable operation across varying load conditions and protects the system from power supply transients.

[0390] This self-contained architecture provides several advantages. The elimination of the optical link and remote PC reduces system cost and complexity while improving reliability by reducing the number of interconnected components. The local storage capability enables the system to operate independently of network infrastructure, making it suitable for deployment in remote locations or environments where network connectivity is unreliable. The wireless interface to control tablet 510 maintains operational flexibility while preserving the standalone nature of the camera head. The integrated processing and storage architecture also simplifies system installation and maintenance, as all critical components are housed within a single unit that can be deployed and serviced as a complete assembly.

[0391] Figure 42 shows an alternative embodiment of an array camera backplane 500 with image compression modules, illustrating a processing architecture optimized for bandwidth-constrained applications where real-time transmission of uncompressed multi-camera video streams would exceedavailable network capacity. This embodiment enables the camera head to compress image data prior to transmission, reducing bandwidth requirements while maintaining image quality suitable for quality assurance and inspection applications.

[0392] The camera head portion of the system includes four sensor interfaces 502a, 502b, 502c, and 502d that receive image data from the imaging modules, connecting to backplane 506 which serves as the central interconnection substrate. The backplane 506 houses FPGA SOM 508 that performs image processing, data routing, and compression coordination functions. In this embodiment, the FPGA SOM 508 comprises a Zynq UltraScale+ FPGA such as the Trenz XZU90-19EG, providing both programmable logic for hardware-accelerated image processing and ARM processor cores for system control and compression management.

[0393] In contrast to the embodiment shown in Figure 40, which transmits uncompressed image data to a remote rendering and storage PC via optical connector 524, and the embodiment shown in Figure 41, which stores uncompressed data locally on solid-state drives 532, the embodiment of Figure 42 incorporates H.265 compression modules 534 that compress the image data prior to storage or transmission. The H.265 compression modules 534 implement the High Efficiency Video Coding standard, achieving compression ratios of 10: 1 to 50: 1 while maintaining image quality sufficient for industrial inspection applications. The compressed video streams require significantly less bandwidth for transmission and less storage capacity for archival, enabling deployment in environments where network infrastructure or storage resources are limited.

[0394] DDR memory 530 provides high-bandwidth temporary storage for image data during compression operations. The DDR memory 530 serves as a buffer between the sensor interfaces 502a, 502b, 502c, and 502d and the H.265 compression modules 534, enabling the FPGA SOM 508 to perform preprocessing operations on captured frames before the frames are compressed. The DDR memory 530 capacity is sized to accommodate multiple uncompressed frames from all imaging modules simultaneously, preventing buffer underflows during compression operations and ensuring smooth operation during continuous recording sessions.

[0395] The H.265 compression modules 534 receive preprocessed image data from the DDR memory 530 and apply video compression algorithms to reduce the data rate while preserving image quality. Each compression module 534 may be dedicated to processing the video stream from one or more imaging modules, enabling parallel compression of multiple video channels. The compression parameters may be configured based on application requirements, with adjustable quality settings that balance compression ratio against image fidelity. For quality assurance applications requiring lossless or near-lossless compression, the compression modules 534 may be configured to operate at high quality settings that preserve fine details and color accuracy, while applications tolerating greater compression may use lower quality settings to achieve higher compression ratios and reduced bandwidth requirements.

[0396] The compressed video streams are stored on solid-state drives 532 mounted directly on the backplane 506. The solid-state drives 532 comprise NVMe storage modules that provide high-speeddata storage capabilities, with the reduced data rate of the compressed video enabling longer recording sessions within a given storage capacity compared to uncompressed storage. The compression enables the system to store hours or days of continuous multi-camera video within practical storage capacities, facilitating long-term quality monitoring and retrospective analysis of production processes.

[0397] Communication module 512 provides network connectivity to control tablet 510 and may also provide connectivity to remote systems for accessing stored compressed video. The communication module 512 may comprise an Ethernet interface, a Wi-Fi interface, or both, allowing operators to access system controls, retrieve compressed video files from the solid-state drives 532, and monitor system status via network communication. The reduced bandwidth requirements of the compressed video streams enable transmission over standard network infrastructure without requiring the highspeed optical links used in the embodiment of Figure 40.

[0398] Power module 516 receives electrical power from power input 514 and distributes regulated power to all electronic components on the backplane 506, including the sensor interfaces 502a, 502b, 502c, and 502d, the FPGA SOM 508, the DDR memory 530, the H.265 compression modules 534, and the solid-state drives 532. The power management circuitry within power module 516 ensures stable operation across varying load conditions and protects the system from power supply transients.

[0399] This compression-enabled architecture provides several advantages for industrial inspection applications. The reduced bandwidth requirements enable deployment in environments where highspeed network infrastructure is unavailable or cost-prohibitive, while the compressed storage extends the practical recording duration within given storage capacities. The local compression and storage capability enables the system to operate independently of remote processing infrastructure, making it suitable for deployment in distributed manufacturing facilities or remote inspection stations. The compressed video format also facilitates efficient transmission of inspection data to central quality control systems for aggregation and analysis across multiple production lines or facilities. The integration of compression, processing, and storage within a single camera head unit simplifies system installation and maintenance while providing the flexibility to operate in standalone mode or as part of a networked inspection system.

[0400] Concluding Remarks

[0401] Having described and illustrated principles of the present technology with reference to various examples, it should be apparent that applicant's inventive work is not so limited.

[0402] For example, while reference was repeatedly made to imaging at a subject sampling resolution of 1 pixel per 2mm of subject extent, it will be understood that this parameter is application-dependent. In other implementations, the resolution may be, e.g., 1 pixel per 1, 3, 5 or 10mm of subject extent.

[0403] Similarly, while certain detailed embodiments achieve compression of imagery by applying the image data to different filtering functions, it will be understood that other compression techniques can be used. These include lossless compression (e.g., HuffYUV and lossless modes of H.264, H.265 and AVI compression) and lossy compression (e.g., conventional H.264 and MPEG compression).

[0404] Although the detailed embodiments often employ plural imaging modules that differ in focal length but share the same sensor size, it will be recognized that the same considerations can instead lead to use of imaging modules that differ in sensor size but share the same focal length. And some embodiments include imaging modules of both different focal lengths and sensor sizes.

[0405] Still further, while reference has been made to streaming selected content to stadium suites and to viewers' homes, it can likewise be streamed to public viewing venues, such as concert halls, auditoriums, theatres, and the Las Vegas Sphere.

[0406] Moreover, systems of the described sort have applications other than in sports, e.g., in surveillance monitoring of plural exterior doors to a facility, while also capturing a wide-angle view. In many such other applications, lower frame rates can naturally be used (in some instances employing the same frame rate for both the longer- and shorter-focal length imaging modules).

[0407] Effective focal length of a lens system is defined with reference to the example of Fig. 21 (not to scale). A subject (depicted by the stickman) is remote from the lens at a distance D shown as 30m. (The lens system is regarded as negligibly-thin as compared to distance D.) The physical size of the subject S is, e.g., 2m.

[0408] The lens projects a corresponding, inverted image onto the plane of the image sensor (the focal plane). This projected size of the image I is, e.g., 0.85mm.

[0409] The ratio of the subject's physical size S to its distance from the lens D is the same as the ratio of the projected image size I on the sensor to the effective focal length EFL. That is:S / D = I / EFL, soEFL = D*I / S.

[0410] In the example just-given, with the units in meters, EFL = 30*.00085 / 2, or ,01275m, or 12.75mm.

[0411] (Effective focal distance may be better determined with reference to image rays from an infinite distance. However, infinity does not work well in ratios, and a distance of 30m or 100 ft is closeenough.)

[0412] Some implementations of the present technology employ "integrated" imaging modules (IIMs), in which the lens and sensor are permanently attached to each other. A preferred integrated imaging module is variable-focus, as described in co-pending application 63 / 880,317. The described arrangement achieves variable focus by using a movable element that is lightweight enough (e.g., under one gram) that it can be re-positioned by a voice coil actuator, and the module re-focused, at rates of several hundred hertz. In some embodiments, the lens actuator comprises a voice coil motor (VCM) coupled to a movable lens element having a moving mass of less than 1 gram. The VCM and suspension define a closed-loop positioning system configured for high-speed focus adjustments. As used herein, ‘repositioned at rates of several hundred hertz’ refers to a commanded position-update frequency and / or closed-loop tracking bandwidth (e.g., -3 dB) of at least 200 Hz, such as 200-500 Hz, for small-signal focus corrections.

[0413] It will be understood that reference to imaging modules in a "row" commonly refers to modules positioned so that their lens apertures are in a common horizontal plane, even if the face of the array camera system is curved.

[0414] Reference was made to video in different formats. Formats are typically distinguished by one or more of: frame dimensions (in pixels), aspect ratio, frame rate, type of chroma subsampling, compression type, and fde type.

[0415] Technology that tracks a person, playing equipment (e.g., ball) and / or activity within video data is known to artisans, so is only briefly referenced herein. Exemplary technologies are detailed in US patent publications 20080192116, 20100026809, 20150268822, 20160101358 and 20210260442.

[0416] The following publications form part of this disclosure and detail particular implementations of systems (e.g., image sensors) used in certain embodiments of the present technology.

[0417] Pang, W. and Brady, D.J., 2019. Distributed Focus and Digital Zoom. arXiv preprint arXiv: 1909.06451;

[0418] Brady, D.J., Pang, W., Li, H., Ma, Z., Tao, Y. and Cao, X., 2018. Parallel cameras. Optica, 5(2), pp.127-137.

[0419] Additional details concerning lens design and sensor construction that can be employed in embodiments of the present technology are found in Blahnik et al, Smartphone imaging technology and its applications, Advanced Optical Technologies, 2021 Jun 1; 10(3): 145-232.

[0420] Additional details concerning array camera technologies that can be employed in embodiments of the present technology, including particular arrangements for compressing imagery, synthesizing imagery, and rendering imagery, are found in Yan et al, Compressive sampling for array cameras, SIAM Journal on Imaging Sciences. 2021 ; 14( 1): 156-77; Wang et al, Integrated photonic encoder for low power and high-speed image processing. Nature Communications. 2024 May 27; 15 ( 1 ) : 4510; Huang et al, Array camera image fusion using physics-aware transformers. arXiv preprint arXiv:2207.02250. 2022 Jul 5; Yuan et al, A modular hierarchical array camera. Light: Science & Applications. 2021 Feb 18; 10(l):37; and applicant's patent documents US10462343, US10944923, and US20200059606.

[0421] Deep learning technologies can be employed in embodiments of the present technology. Suitable deep learning technologies are detailed in Brady et al, Smart cameras. arXiv preprint arXiv:2002.04705. 2020 Feb 11.

[0422] Applicant hereby expressly teaches that the imaging modules and array camera systems detailed above can be implemented to further include the sensors, color fdter arrays, lenses and focus arrangements detailed in the above-cited and the incorporated-by-reference documents. Likewise, the methods and systems detailed above can be implemented to further include the digital zoom, filtering, and other image signal processing methods detailed in the above-cited and the incorporated-by- reference documents.

[0423] Each of the non-patent documents cited herein is attached as an appendix to application 63 / 740,985 and forms part of the present disclosure. The patent documents cited herein are expressly incorporated by reference, as if set forth fully herein.

Claims

Claims1. An array camera system comprising plural imaging modules, each including a lens and an image sensor, the plural imaging modules being fixedly joined together by a structure including a housing, wherein the system employs a heterogeneous combination of:fixed-focal-length integrated imaging modules (IIMs); andvariable-focal-length IIMs with high-speed focus control to optimize sampling of a 3D optical data cube.

2. The array camera system of claim 1 wherein modules capable of rapid focal transitions are integrated alongside modules configured for fixed, high-quality imaging at specific depths or fields of view.

3. The array camera system of claim 1 configured to acquire depth-resolved image data throughout a view volume.

4. An array camera system of any of the foregoing claims comprising plural imaging modules, each including a lens and an image sensor, the plural imaging modules being fixedly joined together by a structure including a housing, wherein at least one of the imaging modules comprises an integrated imaging module (IIM) that includes:a static lens assembly having a first mass; anda lightweight movable lens element positioned closest to the sensor, the movable lens element having a second mass, lower than the first mass, and being actuated by a voice coil motor capable of rapid focus adjustment at frequencies of 100 Hz or greater;wherein the IIM enables fast, precise, and repeatable focal transitions across the depth of field by separating optical power between the static lens assembly and the movable element;wherein an array of such IIMs performs temporal focal scanning, sweeping through focal planes of a view volume at high speed to capture depth-indexed image slices.

5. The array camera system of claim 4 including a high-speed mechanism for time-multiplexed depth capture using integrated modules capable of dynamic focus control across successive exposures.

6. The array camera system of claim 5 including an Al viewing agent watching a sporting event being captured by the array camera system and contributing player distance information to the time- multiplexed depth capture cycling waveform.

7. The array camera system of claim 4 wherein the voice coil motor and movable lens element define a closed-loop positioning system configured to reposition the movable lens element with a commanded position-update frequency and / or closed-loop tracking bandwidth of at least 200 Hz.

8. The array camera system of claim 7 wherein the closed-loop tracking bandwidth is at least 300 Hz.

9. The array camera system of claim 7 wherein the movable lens element has a mass between 25 mg and 250 mg.

10. An array camera system for a stadium that views a playing a field, the array camera system comprising plural imaging modules, each including a lens and an image sensor, the plural imaging modules being fixedly joined together by a structure including a housing, the stadium including acomputer system that stores imagery from the array camera system, the stadium further including first and second suites, each equipped with a respective viewing screen and a user interface coupled to the computer system, the user interface in the first suite enabling users to select first array camera system imagery for display on the viewing screen in the first suite, and the user interface in the second suite the enabling users to select second array camera system imagery for display on the viewing screen in the second suite, the selected first and second imagery being different.

11. The array camera system of claim 10 in which the user interface in the first suite enables users to select imagery depicting an area on the playing field, or a player on the playing field, for display on the screen in the first suite, to the exclusion of imagery depicting another area or player captured by the array camera system at a same time as said selected imagery depicting the selected area or player.

12. An array camera system comprising N imaging modules where N is at least 4, fixedly joined together by a structure including a housing, each imaging module including a lens and an image sensor, each imaging module being oriented in an imaging direction characterized by an azimuthal angle cp relative to a reference direction, the N imaging modules being oriented at N different azimuthal angles that collectively form a progressively-ordered set S = {(pl, (p2, q>3, ... cpN} where (pl>(p2>q>3>... >cpN, said azimuthal angles having differences therebetween of:Al= cp 1 -cp2A2= q>2-q>3AN-1= cpN-l-cpN, characterized in that said differences progressively change in magnitude, so that A1>A2>...>AN-1.

13. The array camera system of claim 12 where N is at least 5, the array camera system including imaging modules having azimuthal angles ( i>( 2>( 3>( 4>( 5, said azimuthal angles having differences therebetween of:Al= (pl-(p2; A2= q>2-q>3; A3= cp3-(p4; and A4= cp4-(p5; characterized in that A1>A2>A3>A4.

14. The array camera system of claim 13 where N is at least 6, the array camera system including imaging modules having azimuthal angles c I>(p2>(p3>(p4>(p5>cp6, said azimuthal angles having differences therebetween of:Al= q> l-(p2; A2= q>2-q>3; A3= cp3-(p4; A4= cp4-(p5; and A5= cp5-(p6;characterized in that A1>A2>A3>A4>A5.

15. The array camera system of any of claims 12-14 including imaging modules having three, four, or more different focal lengths.

16. The array camera system of any of claims 12-15 including imaging modules having three, four, or more different image sensor dimensions.

17. The array camera system of any of claims 12-16 in which each imaging module is oriented in an imaging direction that is further characterized by a polar angle 0 relative to a reference plane, and all of said N modules are oriented in an imaging direction characterized by a common polar angle.

18. The array camera system of any of any of claims 12-17 in which each imaging module is oriented in an imaging direction that is further characterized by a polar angle 0 relative to a reference plane, the array camera system including at least three imaging modules having respective polar angles 01>02>03, said polar angles having differences therebetween of:AA= 0I-02 and AB= 02-03, where AA>AB.

19. An array camera system comprising plural imaging modules, each including a lens and an image sensor, the plural imaging modules being fixedly joined together by a structure including a housing, each of the imaging modules being oriented in an imaging direction characterized by an azimuthal angle cp relative to a reference direction, the plural imaging modules being oriented at F >3 different azimuthal angles that collectively form a progressively-ordered successive set of azimuthal angles SF = {cpl, q>2, ... (pF } where <p2> ... >q>F, wherein a first pair of the successive azimuthal angles in set SF differ by a first delta-azimuth value AA1, and a second pair of the successive azimuthal angles in set SF differ by a second delta-azimuth value AA2 greater than the first delta-azimuth value.

20. The array camera system of claim 19 in which the first and second delta-azimuth values have values as follows:5° < AA1 < 10°; and 10° < AA2 < 25°.

21. The array camera system of claims 19 or 20 in which F > 4, wherein a third pair of the successive azimuthal angles in set SF differ by a third delta-azimuth value AA3 greater than the second deltaazimuth value.

22. The array camera system of claim 21 in which the first, second and third delta-azimuth values have values as follows:5° < AA1 < 10°; 10° < AA2 < 14°; and 14° < AA3 < 25°.

23. The array camera system of claims 21 or 22 in which F > 5, wherein a fourth pair of the successive azimuthal angles in set SF differ by a fourth delta-azimuth value AA4 greater than the third deltaazimuth value.

24. The array camera system of claim 23 in which the first, second, third and fourth delta-azimuth values have values as follows:5° < AA1 < 9°; 9° < AA2 < 12°; 12° < AA3 < 15°; and 15° < AA4 < 25°.

25. The array camera system of any of claims 19-24 in which the imaging directions of the plural imaging modules are further characterized by G > 3 different polar angles that collectively form a progressively-ordered successive set of polar angles SG = {01,02,...0G} where 01>02>...0G, wherein a first pair of the successive polar angles in set SG differ by a first delta-polar value API, and a second pair of the successive polar angles in set SG differ by a second delta-polar value AP2 greater than the first delta-azimuth value.

26. The array camera system of claim 25 in which the first and second delta-polar values have values as follows:0° < API < 23° and 23° < AP2 < 70°.

27. The array camera system of claims 25 or 26 in which G >4, wherein a third pair of the successive polar angles in set SP differ by a third delta-polar value AP3 greater than the second delta-polar value.

28. The array camera system of claim 27 in which the first, second and third delta-polar values have values as follows:0° < API < 23°; 23° < AP2 < 31°; and 31° < AP3 < 70°.

29. An array camera system comprising F>4 imaging modules, each including a lens and an image sensor, the F imaging modules being fixedly joined together by a structure including a housing, the housing positioning entrance apertures of the F imaging module lenses in a shared plane, each of the F imaging modules being oriented in an imaging direction characterized by an angle within said shared plane, wherein angles of said imaging modules are splayed non-uniformly within said shared plane.

30. An array camera system for capturing image data depicting a scene, the system including at least first, second and third imaging modules, each having a lens with a focal length and being characterized by a respective resolution, frame-rate and bit-depth, the first and second imaging modules each having lens focal lengths greater than 40mm and being pointed in different viewing directions to capture first and second fields of view, the third imaging module having a lens focal length of less than 20 mm and a having field of view than spans said first and second fields of view, wherein said first and second fields of view are isolated, such that a line between their centers passes through a scene region that is imaged by the array camera system only at a resolution, frame rate and / or bit-depth that is lower than that of both the first and second imaging modules.

31. The array camera system of claim 30 in which said different viewing directions are characterized by azimuthal angles that differ between 60 and 75 degrees.

32. An analytics system comprising first and second array camera systems according to claim 31 positioned at a baseball playing field, the first array camera system being positioned off the playing field along a third base line, and the second array camera being positioned off the playing field along a first base line.

33. A method comprising the acts:analyzing video imagery from each of plural imaging modules in an array camera system to detect imagery depicting one of the following events: a motion of a tennis serve, a motion of a baseball pitch, or a motion of a baseball swing;in response to detection of one of said events in video imagery from a first of said imaging modules, copying a first interval of image data that was previously written from said first imaging module into a circular buffer, into long-term storage; andin response to said event detection in video imagery from the first imaging module, also copying a second interval of image data that was previously written from said second imaging module into a circular buffer, into long-term storage.

34. The method of claim 33 that further includes:in response to said event detection in video imagery from the first imaging module, writing a third interval of following video from the first imaging module into long-term storage; andin response to said event detection in video imagery from the first imaging module, also writing a fourth interval of following video from the second imaging module into long-term storage.

35. The method of claim 34 in which the first, second and third intervals are less than five seconds, and the fourth interval is more than five seconds.

36. A system for distributing video data to M>=1 video-receiving clients, based on image data depicting a scene produced from N>=1 imagers in a camera system, said image data being stored in a memory, the system including:a first processor and a memory controller adapted to receive data indicating a client-requested view of a sub-part of the scene, generate corresponding floating point coordinates for each of plural sample points in a camera system frame of reference, identify from the floating point coordinates of each sample point one or more memory addresses, and to fetch image data from said memory addresses; anda second processor adapted to compile an output frame of imagery that depicts the client-requested sub-part of the scene from the fetched image data.

37. The system of claim 36 in which the received data indicates a client requested view of a sub-part of the scene that is captured by a first of said N imagers, said first imager being configured to output a frame of image data organized as rows and columns of pixels, wherein the received data indicates a center location of said client-requested view within said frame of image data, and the first processor is adapted to generate floating point coordinates identifying plural sample points corresponding to a line that passes through said center location within said frame of image data, wherein said line is tilted relative to said pixel rows of the first imager.

38. The system of claim 37 in which the floating point coordinates correspond to multiple lines through said frame of image data, each of said lines being tilted relative to said pixel rows of the first imager.

39. The system of claim 36 in which M>=2, and the first processor is adapted to generate a first set of floating point coordinates for a first view requested by a first video-requesting client, and is adapted to generate a second set of floating point coordinates for a second, different, view requested by a second video-requesting client.

40. The system of claim 39 in which the received data indicates that the first client-requested view is at a first resolution, and that indicates the second client-requested view is at a second, different, resolution.

41. The system of claim 39 in which the received data indicates that the first client-requested view is at a first frame rate, and indicates that the second client-requested view is at a second, different, frame rate.

42. The system of claim 36 in which the second processor is adapted to interpolate plural image values fetched from the memory to produce a single output image value in said output frame of imagery.

43. The system of claim 42 in which the memory controller is adapted to fetch P image values for use by the second processor to produce a first output image value in said output frame of imagery, and to fetch Q image values for use by the second processor to produce a second output image value in said output frame of imagery, where P > Q > 1.

44. The system of claim 43 in which P > Q > 2.

45. The system of claim 42 in which the first processor comprises a field programmable gate array (FPGA) and the second processor comprises a graphics processing unit (GPU).

46. The system of claim 36 including N>=2 imagers, wherein the memory stores rows of image data from each of the imagers, and image data captured by pixels in a row of a first of the imagers projects to a first linear line of physical locations in x,y,z space, and image data captured by pixels in a row of a second of the imagers projects to a second linear line of physical locations in x,y,z space, where the first and second linear lines of locations in x, y, z space are non-parallel.

47. The system of claim 36 in which the received data indicating said client-requested view of the subpart of the scene indicates pan, tilt and zoom information for the requested view.

48. A method comprising the acts:using first and second spaced-apart imagers to sense light flashes emitted from light sources affixed to an ice skater skating on ice, including from skates worn by the ice skater; anddetermining correspondence between pixels in a first frame of imagery captured by the first imager and pixels in a second frame of imagery captured by the second imager, based on locations of the sensed light flashes in the first and second frames of imagery.

49. The method of claim 48 wherein said first and second imagers are mounted in a common housing.

50. An array camera system including first, second and third camera modules that respectively have first, second and third focal lengths and first, second and third fields of view, where the third focal length is longer than the second focal length and the second focal length is longer than the first focal length, and an area within the third field of view is also within the first field of view and within the second field of view.

51. The array camera system of claim 50 in which the third field of view is entirely encompassed within the second field of view.

52. The array camera system of claim 50 in which the second field of view is not entirely encompassed within the first field of view.

53. The array camera system of claim 50 in which at least one of said camera modules has a square field of view.

54. The array camera system of claim 50 including at least two camera modules having the first focal length, at least two camera modules having the second focal length, and at least two camera modules having the third focal length.

55. An array camera system including at least two camera modules of a first type, at least two camera modules of a second type, and at least two camera modules of a third type, the first, second and third types of camera modules respectively having first, second and third focal lengths, where the thirdfocal length is longer than the second focal length and the second focal length is longer than the first focal length, and at least one area within a field of view of a camera module of the third type is also within a field of view of a camera module of a first type and within a field of view of a camera module of the second type.

56. The array camera system of claim 55 in which an area within a field of view of a camera module of the third type is within a field of view of no camera module of the second type.

57. An array camera system comprising plural imaging modules, each including a lens and an image sensor, the plural imaging modules being fixedly joined together by a structure including a housing, wherein the plural imaging modules include:a first set of imaging modules having lenses with focal lengths greater than 30 mm;a second set of imaging modules having lenses with focal lengths less than 20 mm; anda third set of imaging modules having lenses with intermediate focal lengths between the first and second sets.

58. The array camera system of claim 57 wherein the first set of imaging modules have focal lengths greater than 50 mm, and the second set of imaging modules have focal lengths less than 12 mm.

59. The array camera system of claim 57 wherein the third set of imaging modules have focal lengths in a range selected from: 10-15 mm, 15-22 mm, 22-30 mm, 30-40 mm, or 40-55 mm.

60. The array camera system of claim 57 wherein one implementation employs focal lengths of 60 mm for the first set, 24 mm for the third set, and 12 mm for the second set.

61. The array camera system of claim 57 wherein another implementation employs focal lengths of 50 mm for the first set, 25 mm for the third set, and 6 mm for the second set.

62. The array camera system of claim 57 wherein the third set of imaging modules comprises imaging modules having two distinct intermediate focal lengths.

63. The array camera system of claim 62 wherein one of the two distinct intermediate focal lengths is paired with a color sensor and the other is paired with a monochrome sensor.

64. The array camera system of claim 63 wherein the longer of the two intermediate focal lengths is paired with the color sensor.

65. The array camera system of claim 63 wherein the longer of the two intermediate focal lengths is paired with the monochrome sensor to maximize low-light sensitivity and spatial acuity.

66. The array camera system of claim 63 wherein a 25 mm lens is paired with a color sensor and a 12 mm lens is paired with a monochrome sensor.

67. The array camera system of claim 55 wherein monochrome sensors provide higher photon efficiency and superior detail rendition compared to color sensors, and color values for imagery captured by monochrome sensors are inferred from contemporaneously captured color data from neighboring modules.

68. The array camera system of claim 57 further comprising imaging modules operating at variable frame rates for motion-estimation or depth-fusion.

69. The array camera system of claim 57 further comprising imaging modules equipped with polarization filters for glare suppression or surface-normal estimation.

70. The array camera system of claim 57 further comprising multispectral or hyperspectral imaging modules for enriched colorimetry or material-reflectance discrimination.

71. The array camera system of claim 57 configured to achieve full-field capture of sports venues with the ability to digitally zoom into any region of interest at any depth within a reconstructed view volume.

72. An array camera system comprising plural imaging modules, each including a lens and an image sensor, the plural imaging modules being fixedly joined together by a structure including a housing, wherein the system includes:first, second and third camera modules that respectively have first, second and third focal lengths and first, second and third fields of view, where the third focal length is longer than the second focal length and the second focal length is longer than the first focal length;at least two camera modules of each of the first, second and third types;wherein an area within a field of view of a camera module of the third type is also within a field of view of a camera module of the first type and within a field of view of a camera module of the second type; andwherein a region falls within the collective fields of view of camera modules of the third type but is outside the collective fields of view of camera modules of the second type, creating a gap in coverage by the intermediate focal length modules.

73. An array camera system comprising:plural imaging modules, each including a lens and an image sensor, the plural imaging modules being fixedly joined together by a structure including a housing, at least two of the plural imaging modules being identical in factors including: (a) lens focal length; (b) focal length; (c) color filter array; and (d) image sensor dimensions, and at least two of the plural imaging modules being different in at least one of said factors (a), (b), (c) or (d);each of said plural imaging modules including a local memory operative to store pixel values for more than one frame of imagery captured by the image sensor;each of said plural imaging modules further including processing circuitry operative to process pixel values representing imagery captured by the image sensor, the processing circuitry including convolution circuitry that is operative to convolve sets of said pixel values with each of plural filter kernels, each of said filter kernels comprising an array of integer values, the processing circuitry outputting plural filter result data for storage in a memory; andwherein imagery captured by each imaging module is characterized by both (i) stored pixel values, and (ii) stored plural filter result data.

74. The array camera system of claim 73 further including a central processor that is coupled to provide access to both said stored pixel values and said stored plural filter result data.

75. The array camera system of any of claims 73-74 that further includes a memory that stores information from the plural imaging modules, including a first memory part that stores pixel values from the plural imaging modules, and a second memory part that stores filter result data from the plural imaging modules.

76. The array camera system of any claims 73-74 in which, for at least one of the plural imaging modules, the convolution circuitry is operative to convolve a 3D set of pixel values, including pixel values from two or more different frames of imagery captured by the first imaging module image sensor, with a filter kernel comprising a 3D array of integer values.

77. The array camera system of any of claims 73-76 that further includes an image synthesis module operative to recreate image data using the filter result values, and an error detection module operative to assess a difference between the recreated image data and counterpart pixel data.

78. The array camera of claim 77 that further includes a feedback module operative to change one or more of said filter kernels in response to said assessed difference.

79. The array camera system of any claims 73-78 claims including imaging modules A and B that are identical in factors (a), (b), (c) and (d), the convolution circuitry of imaging module A being operative to convolve imaging module A pixel values with a first filter kernel comprising a first array of integer values, and with a second filter kernel comprising a second, different, array of integer values, and the convolution circuitry of imaging module B being operative to convolve imaging module B pixel values with said first filter kernel and with said second filter kernel, wherein common filter kernels are applied to pixel values in both imaging modules A and B;the system further including first and second parallel memories, the first parallel memory storing filter result data corresponding to convolution of imaging module A pixel values with the first filter kernel, and storing filter result data corresponding to convolution of imaging module B pixel values with the first filter kernel; the second parallel memory storing filter result data corresponding to convolution of imaging module A pixel values with the second filter kernel, and storing filter result data corresponding to convolution of imaging module B pixel values with the second filter kernel; wherein filter result data from both imaging modules A and B are stored in both the first and second parallel memories.

80. The array camera system of any claims 73-79 includes at least three imaging modules A, B and C that are identical in factors (a), (b), (c) and (d), each of said at least three imaging modules A, B and C having a common modular form that includes a front surface parallel to the imaging module image sensor, coupled to at least first and second side structures, at least one of said side structures defining a coupling angle of less than 90 degrees relative to the front surface, wherein one of the side structures of imaging module A is coupled to a first of said side structures of imaging module B to thereby cause lens axes of imaging modules A and B to diverge by K degrees, and a second of said side structures of imaging module B is coupled to one of the side structures of imaging module C to thereby cause lens axes of imaging modules B and C to diverge by K degrees, said divergences of K degrees being established, at least in part, by said coupling angle of less than 90 degrees.

Images