• 专利附录
  • 专利说明
  • 权利要求
  • 类似技术
  • 同族专利
  • 引用文献
  • 法律状态
  • 优先权
    • 专利摘要:
    The method for determining a depth map (100) of a scene (10) comprises developing a distance map (400) of the scene (10) obtained by flight time measurements, an acquisition of two scene images (201, 202) from two different viewpoints, and stereoscopic processing (300) between said two images (201, 202) taking into account said distance map (400). The development of said distance map (400) comprises elaboration of distance histograms (410) by acquisition zones (420) of the scene, and said stereoscopic processing (300) comprises, for each region (120) of the depth map (100) corresponding to an acquisition zone (420), an elementary processing (310) taking into account the corresponding histogram (410).
    公开号:FR3063374A1
    申请号:FR1751539
    申请日:2017-02-27
    公开日:2018-08-31
    发明作者:Manu ALIBAY；Olivier Pothier；Victor MACELA；Alain BELLON；Arnaud Bourge
    申请人:STMicroelectronics SA；
    IPC主号:
    专利说明:

    © Publication no .: 3,063,374 (to be used only for reproduction orders)
    ©) National registration number: 17 51539 ® FRENCH REPUBLIC
    NATIONAL INSTITUTE OF INDUSTRIAL PROPERTY
    COURBEVOIE
    ©) Int Cl 8 : G 06 T 7/50 (2017.01), G 06 T 7/593, H 04 N 13/00
    A1 PATENT APPLICATION
    ©) Date of filing: 27.02.17. © Applicant (s): STMICROELECTRONICS SA Company ©) Priority: anonymous— FR. (72) Inventor (s): ALIBAY MANU, POTHIER OLIVIER, MACELA VICTOR, BELLON ALAIN and BOURGE (43) Date of public availability of the ARNAUD. request: 31.08.18 Bulletin 18/35. ©) List of documents cited in the report preliminary research: Refer to end of present booklet @) References to other national documents ©) Holder (s): STMICROELECTRONICS SA Company related: anonymous. ©) Extension request (s): ©) Agent (s): CASALONGA.
    Pty METHOD AND DEVICE FOR DETERMINING A DEPTH MAP OF A SCENE.
    FR 3 063 374 - A1
    The method for determining a depth map (100) of a scene (10) comprises drawing up a distance map (400) of the scene (10) obtained by time of flight measurements, an acquisition of two images of the scene (201, 202) from two different points of view, and stereoscopic processing (300) between said two images (201, 202) taking into account said distance map (400). The elaboration of said distance map (400) comprises an elaboration of histograms of distances (410) by acquisition zones (420) of the scene, and said stereoscopic processing (300) comprises, for each region (120) of the depth map (100) corresponding to an acquisition area (420), an elementary processing (310) taking into account the corresponding histogram (410).

    i
    Method and device for determining a depth map of a scene
    Embodiments and embodiments relate to the determination of a depth map by stereoscopy and by measurement of combined flight times.
    A depth map is a digital representation of the distances between the positions of different objects in a scene and a receiver, the rendering of which is comparable to a photograph that supports depth information, not brightness.
    There are different techniques for acquiring or determining a depth map, such as stereoscopy or time of flight measurement.
    Time-of-flight measurement involves emitting an identifiable electromagnetic wave signal on a stage, typically pulsed laser illumination, and detecting signals reflected by objects in the scene. The time difference between the instant of emission of a signal and the instant of reception of this signal reflected by an object of the scene makes it possible to calculate the distance separating the transceiver from the object.
    Stereoscopy is a so-called passive measurement and makes it possible to determine a depth map of a scene from two images of the scene, of photographic type, taken from different points of view.
    Figure 1 illustrates the principle of depth determination by stereoscopy. Two images 201, 202 of a same scene 10 comprising an object 11 are obtained from different and known points of view, for example by means of image sensors fitted with optics. Usually the two points of view have a parallel optical axis and are aligned horizontally to correspond to a left image 201 and a right image 202, spaced apart by a so-called fundamental difference.
    The parallax resulting from the fundamental deviation means that the object 11 is projected onto the images 201, 202 at respective positions 111, 112 different. The projections 111, 112 are located on the same epipolar line 12, which is typically horizontal for points of view located in the same plane, with parallel optical axes and receiving surfaces aligned vertically.
    Thus, the determination of the distance between the receiver and the object 11, namely the depth of the object 11, comprises for each point of the image 201 a calculation of similarity with the points of the image 202 located on the same epipolar line 12. The distance dd separating the positions of the similar points of the two images 201, 202 is called disparity. The value of the disparity ôd between the two projections 111, 112 makes it possible to extrapolate the depth of the corresponding object 11, taking into account in particular the fundamental difference, and the optical and technological characteristics of the acquisition means.
    Techniques for identifying similarities between projections and extrapolating depths from disparities generally include a sub-sampled initialization calculation, a less sampled calculation using initialization and refinement processing, in order to obtain a correspondence by a so-called decreasing granularity method. These techniques are more fully detailed in particular in the scientific publication: “HIRSCHMULLER, Heiko. Stereo processing by semi global matching and mutual information. IEEE Transactions on pattern analysis and machine intelligence, 2008, vol. 30, no 2, p. 328-341. "
    The calculations implemented in the usual stereoscopic treatments therefore require a large amount of computing resource introducing long execution times and sometimes leading to errors.
    A fusion of time-of-flight and stereoscopy technologies can limit and refine conventional stereoscopic processing.
    For example, it is possible to perform an a posteriori estimate of the depth map using both the time of flight measurement and the stereoscopic image, or to initiate the stereoscopic processing with the depth information obtained by time measurement. of flight using one or more time of flight sensors (ToF: "time of flight" in English), for example in place of sub-sampled processing. In addition, it is also possible to calculate each disparity value taking into account the variations in depths measured by flight time.
    This type of combination of technologies is more fully detailed, notably in the scientific publication: “Vineet Gandhi, Jan Cech, Radu Horaud. High-Resolution Depth Maps Based on TOFStereo Fusion. ICRA 2012 - IEEE International Conference on Robotics and Automation, May 2012, Saint-Paul, Minnesota, United States. IEEE, pp.4742-4749, 2012 ”
    It is recalled here that a time-of-flight sensor makes it possible to measure the distance between an object and the sensor by measuring the time difference between the emission of a signal and the reception of this signal after being reflected by the object.
    Current combinations of stereoscopic determination and time-of-flight measurement techniques require extremely good mutual calibration and pixel-to-pixel correspondences between stereoscopic images and time-of-flight measurement. In other words, a high-resolution time-of-flight sensor, for example at least 25 kilopixels, is required to implement these depth map determination improvements.
    However, high-resolution time-of-flight sensors are bulky, consuming energy and computing resources, which is generally undesirable. They are also expensive. Finally, these inconveniences are incompatible with in particular autonomous and on-board, compact and battery-powered technologies, which are becoming more and more common.
    However, there are lower resolution time-of-flight sensors compatible with the requirements of autonomous and on-board technologies. In addition, they are less expensive than their high resolution competitors.
    This type of time of flight sensor is very compact, autonomous and economical in energy (notably consuming less than 20pW in standby, and less than 35mW in operation). This type of sensor measures a distance map, in which the depths are obtained by acquisition zones, for example according to a matrix of 5 * 3 zones or 15 * 9 zones of an equivalent image of a scene. For each acquisition area, a distribution of the depths of the objects present in the corresponding part of the scene is developed in the form of a histogram.
    Thus, compared to high-resolution time-of-flight sensor technologies, the information obtained by this type of sensor is less precise but can be integrated into a processing chain of an autonomous or embedded system.
    There is a need to improve stereoscopic treatments with the help of such time-of-flight measurements, obtained with a sensor compatible with an autonomous or on-board technology, in a simple and effective manner.
    It is thus proposed according to modes of implementation and embodiment of using such a distance map comprising histograms of distances by acquisition zones to optimize a determination of a depth map obtained by stereoscopic acquisition, in a system. autonomous or on-board.
    According to one aspect, there is thus proposed a method for determining a depth map of a scene, comprising an elaboration of a map of distances of the scene obtained by time of flight measurements, an acquisition of two images of the scene from two different points of view, and stereoscopic processing between said two images taking into account said distance map. According to a general characteristic of this aspect, the preparation of said distance map comprises the preparation of distance histograms by areas of acquisition of the scene, and said stereoscopic processing comprises, for each region of the depth map corresponding to a acquisition zone, elementary processing taking into account the corresponding histogram.
    The term of correspondence between a region of the depth map and an acquisition zone means for example that said regions and zones includes the distance information of a same part of the scene. Indeed, it is possible to establish geometric relationships, depending on the conditions of the different acquisitions and measurements, such as the positions of the viewpoints and the optical conditions.
    The method according to this aspect thus makes it possible to acquire an improved spatial understanding of a scene, in a manner compatible with autonomous or embedded technologies.
    This spatial understanding can, for example, be exploited in the context of adaptive management of a photographic flash and / or very effective autofocus, while optimizing the determination of the depth map of a scene.
    The stereoscopic processing can comprise an extrapolation of the distances of the scene from disparities between said two images.
    For example, the elementary processing limits the calculations of said extrapolation (that is to say for example reduces the quantity and the time of calculations of said extrapolation) and / or improves the reliability of the result of said extrapolation (that is to say i.e. avoid errors) and / or add additional depth information to the extrapolation of scene distances in the depth map.
    In particular in order to be implemented in autonomous or on-board technologies, the resolution of the depth map can for example be at least a thousand times greater than the resolution of the distance map measured by flight time, the resolution of the map of distances measured by flight time being equal to the number of acquisition zones.
    Also, for example, the scene distance map obtained by time-of-flight measurements can include ten to a thousand acquisition zones.
    Advantageously, the respectively maximum and / or minimum distances measurable by flight time are respectively greater and / or less than respectively a ceiling value of stereoscopic range and / or a floor value of stereoscopic range.
    The stereoscopic range ceiling value corresponds to the maximum extrapolable distance in stereoscopic processing, while the stereoscopic range ceiling value corresponds to the minimum extrapolable distance in stereoscopic processing.
    These ceiling and floor values of stereoscopic range depend in particular on the fundamental difference between the positions of the points of view of said two images of the scene.
    According to one mode of implementation, the elementary processing includes, if necessary, an identification of at least one region of the depth map, known as the out-of-range region, the corresponding histogram of which does not include a respectively shorter distance and / or greater than respectively said ceiling value of stereoscopic range and / or said floor value of stereoscopic range, the extrapolation then not being carried out in this at least one out of range region.
    This implementation mode makes it possible to limit the calculations of said extrapolation by avoiding the implementation of unnecessary calculations in areas previously detected out of range (which would not have obtained a result by stereoscopic extrapolation).
    Advantageously, the elementary processing includes an allocation of a default depth to said at least one out-of-range region, the default depth being taken from the distances of the corresponding histogram.
    For example, the default distance can be equal to or greater than the longest distance from the corresponding histogram, or even the mean or median of the distances from the corresponding histogram, or a value chosen according to other criteria from among said distances.
    This makes it possible to add to the depth map additional depth information to that obtained by stereoscopic extrapolation.
    The elementary processing can also include an allocation of a constant depth to respectively at least one region of the depth map, known as the plane region, the corresponding histogram of which comprises a single group of distances whose width at half-height is less than a threshold width, the constant depth being equal to a distance from the single group and the extrapolation then not being carried out in this at least one plane region.
    This also makes it possible to avoid the implementation of extrapolation calculations in a region comprising a substantially flat surface and substantially perpendicular from the point of view of image acquisitions and time of flight measurement, and possibly comprising no texture (none detail) which allows extrapolating a disparity, or comprising repetitive patterns, sources of error in calculating disparities.
    In addition, this also makes it possible to add additional depth information to the depth map to that obtained by stereoscopic extrapolation.
    According to one mode of implementation, the elementary processing comprises an elaboration of a range of values of disparities by region from the extreme values of the distances of the corresponding histogram, and the extrapolation of the distances from the scene is made by region from disparity values included in the corresponding range.
    This implementation mode makes it possible to limit the calculations implemented by the extrapolation of the stereoscopic processing, by limiting the identification of the disparities in an interval obtained beforehand by the histogram of the flight time measurements.
    In addition, said disparity values included in the corresponding range can be taken at a step corresponding to the accuracy of said time of flight measurement, further limiting the amount of calculation of the distance extrapolation of the stereoscopic processing.
    In addition, this mode of implementation makes it possible to improve the reliability of the calculations of said extrapolation, avoiding common errors in the case of uniform or patterned textures repeated in the scene.
    According to one embodiment, the elementary processing comprises a determination of the regional distributions of the distances obtained by said extrapolation and a comparison of the outlines of the histograms of distances by acquisition areas with the outlines of the distributions of distances by respective regions, said comparison producing a rate of agreement.
    In other words, this mode of implementation comprises a comparison of the histograms of the depths obtained by stereoscopy and obtained by measurement of time of flight, and makes it possible to provide, by means of the agreement rate, information on the quality and the reliability of the depth map.
    According to another aspect, a device for determining a depth map of a scene is proposed, comprising a time-of-flight sensor configured to draw up a distance map of the scene, a means of stereoscopic image acquisition. configured to acquire two images of the scene from two different points of view, and a stereoscopic processing means configured to carry out a stereoscopic processing between said two images taking into account said distance map. According to a general characteristic of this aspect, the time-of-flight sensor is configured to develop histograms of distances by scene acquisition zones, and said stereoscopic processing means comprises an elementary processing means configured, for each region of the depth map corresponding to an acquisition area, to perform an elementary processing taking into account the corresponding histogram.
    The stereoscopic processing means can be configured to carry out a stereoscopic processing comprising an extrapolation of the distances of the scene from disparities between said two images.
    For example, the elementary processing means can be configured to limit the calculations of said extrapolation, and / or improve the reliability of the result of said extrapolation and / or add additional depth information to the extrapolation of the distances from the scene in the depth map.
    The resolution of the stereoscopic image acquisition means can be at least a thousand times greater than the resolution of the time-of-flight sensor, the resolution of the time-of-flight sensor being equal to the number of acquisition zones.
    For example, the time-of-flight sensor has ten to a thousand acquisition zones.
    Advantageously, the respectively maximum and / or minimum distances measurable by the time of flight sensor are respectively greater than and / or less than respectively a stereoscopic range ceiling value and / or a stereoscopic range floor value extrapolable by the stereoscopic processing means.
    According to one embodiment, the elementary processing means is configured to, if necessary, identify at least one region of the depth map, known as the out-of-range region, the corresponding histogram of which does not have a distance less than a threshold. of stereoscopic range, the stereoscopic processing means being also configured not to carry out the extrapolation in this at least one region out of range.
    Advantageously, the elementary processing means is configured to assign a default depth to said at least one out-of-range region, the default depth being taken from the distances in the corresponding histogram.
    ίο
    The elementary processing means can also be configured to assign a constant depth to respectively at least one region of the depth map, called the plane region, the corresponding histogram of which comprises a single group of distances whose width at half-height is less. at a threshold width, the constant depth being equal to a distance from the single group, the stereoscopic processing means being also configured not to carry out the extrapolation in this at least one plane region.
    According to one embodiment, the elementary processing means is configured to develop a range of disparity values by region from the extreme values of the distances of the corresponding histogram, and the stereoscopic processing means is configured to carry out the extrapolation of the scene distances by region from disparity values included in the corresponding range.
    Advantageously, the stereoscopic processing means is configured to carry out the extrapolation of the distances from the scene by region from values of disparities taken at a step corresponding to the precision of said measurement of time of flight.
    According to one embodiment, the elementary processing means is configured to determine distributions by regions of the distances obtained by said extrapolation and to compare contours of histograms of distances by acquisition zones with the contours of distributions of distances by respective regions, said comparison producing a rate of agreement.
    An electronic device, such as a camera, a mobile phone or a touch pad, is also proposed, comprising a device for determining a depth map as defined above.
    Other advantages and characteristics of the invention will appear on examining the detailed description of embodiments, in no way limiting, and the appended drawings in which:
    - Figure 1, previously described, illustrates the principle of determining depths by stereoscopy;
    - Figures 2 to 6 show examples of embodiments and embodiments of the invention.
    FIG. 2 illustrates an exemplary implementation of a method for determining a depth map of a scene 10 involving stereoscopic and time of flight acquisitions.
    The stereoscopic acquisition includes an acquisition of two images 201, 202 of scene 10 from two different viewpoints.
    The respective projections 111 and 112 of the same object-point of scene 10 consequently have a parallax, that is to say a shift between their relative positions in the images 201, 202, otherwise called disparity 211.
    On the other hand, the method comprises an elaboration of a distance map 400 obtained by measuring time of flight.
    The development of the distance map 400 of the scene 10 furthermore includes a measurement of time of flight, that is to say a measurement of the time elapsed between the emission of a light signal on the scene and the reception of this signal reflected. This time measurement is proportional to the distance between a transceiver 40 and the various objects in scene 10, a distance otherwise called the depths of the various objects in the scene.
    This measurement is carried out by acquisition zones 420, which may be 10 to 500 according to a matrix distribution in the field of vision of the transceiver 40.
    Geometric relations between the positions of the points of view of two images 201, 202 and the point of view of the measurement of time of flight on the one hand, and the optical characteristics such as the field of vision and the deformations of the different acquisitions of on the other hand, make it possible to establish precise correspondences of the different acquisition zones 420 with regions 120 of a depth map 100. In other words, to each acquisition zone 420 of the time of flight measurement corresponds to a region 120 of the depth map.
    In each acquisition zone 420, on certain time of flight measuring devices, or “time of flight sensor”, the distribution of the distances is communicated in the form of a histogram 410, in addition to an overall measurement of the distance on said area.
    Thus the development of the distance map 400 includes an elaboration of histograms of distances 410 by acquisition areas 420 of the scene.
    Furthermore, a stereoscopic processing 300, comprising an extrapolation 210 of the distances from the scene from the disparities 211 between said two images 201, 202, is implemented.
    The stereoscopic processing 300 comprises an elementary processing 310, which can be implemented before, during and / or after the implementation of the distance extrapolation 210.
    For each region 120 of the depth map 100 corresponding to an acquisition area 420, the elementary processing 310 takes into account the histogram 410 of the acquisition area 420 and makes it possible in particular to limit and / or improve the reliability of the result of said extrapolation 210, and / or of adding additional depth information to that obtained by said extrapolation 210.
    According to a first example, the elementary processing 310 advantageously takes advantage on the one hand of the fact that the maximum measurable distance per flight time is generally greater than a ceiling value 312 of stereoscopic range corresponding to the maximum distance identifiable in said extrapolation 210 of the distances from the scene, and secondly from the fact that the minimum measurable distance by time of flight is generally less than a floor value 313 of stereoscopic range corresponding to the minimum distance identifiable in said extrapolation 210 of the distances from the scene.
    Thus, as shown in FIG. 3, the elementary processing 310 includes an identification of the acquisition zones for which a corresponding histogram 412 does not have a distance less than a stereoscopic range threshold 312, the acquisition zones then being said zones out of range 122.
    Indeed, the disparity in stereoscopy is proportional to the inverse of the depth of the corresponding object. Consequently, at a sufficiently large distance, a disparity will no longer be detectable, an optical object located at infinity not introducing parallax.
    On the other hand, at sufficiently short distances, an object can introduce an undetectable disparity value because it is greater than a maximum disparity, or due to optical deformations due to the angle of view.
    It is understood, for this example as for the following, that an expression of the type “the histogram does not have a distance less than a ceiling value” must be understood as “the histogram does not have a distance less than a value sufficient ceiling to be relevant ”. In fact, in the time of flight sensors, a uniform noise can exist at all distances, and parasitic occlusions can introduce erroneous measurements, in small quantity in the histograms.
    For example, for a fundamental difference of 12mm between the two respective viewpoints of images 201 and 202, with a resolution of 12 megapixels, the stereoscopic range ceiling value 312 can be approximately 1.5 meters to approximately 2.5 meters, while the maximum range of the measurement by time of flight can be greater than 4 meters. The minimum range of measurement by time of flight can be equal to or very close to 0 meters.
    The elementary processing 310 also implements the identification of the corresponding regions, called out of range regions 122.
    Thereafter, the extrapolation 210 is then not carried out in the region (s) out of range 122 identified, if necessary (indeed it is possible that there is no region out of range in scene 10).
    In addition, the elementary processing 310 advantageously comprises an allocation of a default depth 314, originating from the distances of the corresponding histogram 412, to said at least one out of range region 122.
    For example, the default depth can be equal to or greater than the greatest distance of the corresponding histogram 412, or equal to the average of the distances of the corresponding histogram 412, to the median of the distances of the corresponding histogram 412, or still at the distance from the corresponding histogram 412 having the largest population.
    Thus, in this first example, the elementary processing 310 is implemented before the implementation of the extrapolation of distances 210, and allows on the one hand to save resources and computation time, and on the other hand adding depth information to regions 122 of the depth map which would not have it without the time of flight measurement.
    The elementary processing 310 can also include an allocation of a constant depth 316 to a region called the flat region 126. The constant depth 316 comes from a histogram 416 representative of a measurement of a surface that is substantially planar and substantially perpendicular to the optical axis of the measurement.
    A histogram 416 representative of such a situation comprises a single distance measurement, or substantially a single distance measurement, that is to say a histogram comprising a single group of distances whose width at half height 417 is less than a threshold width. The threshold width can for example be less than five distance intervals, or five bars, from the histogram.
    The constant depth 316 can be chosen as the distance of greatest measured population or the average or the median of the histogram 416.
    Due to the reliability of such a measurement as to the planar nature of the part of the scene corresponding to the corresponding acquisition area 426, extrapolation 210 is not necessary. However, if the distance group 416 representative of a flat surface is not the only measurement present in the histogram (in the sense of measurement obtained in sufficient quantity to be relevant), then it will be necessary to carry out the extrapolation 210 in the corresponding region.
    FIG. 4 illustrates a second example of implementation of an elementary processing 310 making it possible to improve the stereoscopic processing 300.
    In this second example, the elementary processing 310 isolates the extreme values 324 of the distances present in the histograms 410 of the different acquisition zones 420.
    These extreme distance values 324 are translated into extreme disparity values, which form a range 322 of possible disparity values for any point in the corresponding region 120.
    Indeed, as a disparity value can be extrapolated to a depth, a depth measurement makes it possible to calculate the equivalent disparity.
    Thus, this range of values of possible disparities 322 makes it possible to implement the extrapolation 210 of the distances from the scene on values of disparities limited to this range 322 of values of possible disparities.
    In other words, the elementary processing 310 comprises in this example the development of a range of disparity values 322 by region from the extreme values 324 of the distances from the corresponding histogram 410. The extrapolation 210 of the distances from the scene is then made, for each region, from values of disparities 211 included in the corresponding range 322.
    This implementation is advantageous on the one hand in terms of the amount of calculation used for each extrapolation 210 of the distances from scene 10 by region 420.
    On the other hand, this implementation makes it possible to reinforce the reliability of the determination of the depth map, in particular in surface conditions without texture, or in surface with repeated patterns, for which the identification of projections 111, 112 of the same object 11 of the scene is delicate and often leads to errors.
    In addition, the resolution (in the sense of the quantification step) of a time-of-flight measurement is constant with respect to the distance from the measured object.
    However, the disparity values are proportional to the inverse of the corresponding distance. Thus an elementary variation of disparity will induce a resolution (in the sense not of quantification) of the estimated distance less and less precise with the increase in the distance of the measured object.
    Consequently, by virtue of the range 322 of value of possible disparities, it is possible to evaluate a step of disparity corresponding to the precision of the measurement of time of flight, for example 2 cm, for the interval of measured distance 324. The correspondence between the two projections 111, 112 of the same object 11 in the two images 201, 202 will then be carried out according to this step of disparity, making it possible to optimize the extrapolation calculations by region 120 for a given order of magnitude of distance .
    FIG. 5 illustrates a third example of implementation of an elementary processing 310 making it possible to improve the stereoscopic processing 300.
    In this example, the depth information obtained by stereoscopy and by time of flight measurement are compared in order to generate a criterion for evaluating the measurement, in this case a rate of agreement between two contours of histograms.
    Thus the elementary processing 310, here implemented after the stereoscopic extrapolation of depths 210, reconstructs by region 120 a histogram 330 of the extrapolated distances.
    For each acquisition area 420 and corresponding region 120, the contour 334 of the histogram 330 of the extrapolated distances is compared to the contour 414 of the histogram 410 of the distances developed by measurement of flight time. By contour is meant here an envelope of the histogram, which can be obtained by interpolation of the values. This eliminates differences in resolution between the histograms 330 and 410.
    This comparison can for example be implemented by a conventional comparison method, such as the least squares method.
    The result of the comparison gives a concordance rate which makes it possible to assess the similarity between the two types of depth measurement for the same scene.
    A match rate close to 1 indicates successful stereoscopic processing, while a match rate below a match threshold indicates a discrepancy between the two measurement methods, and therefore an unreliable depth map .
    FIG. 6 represents an electronic device APP, here a mobile phone or a touch pad, comprising a DIS device for determining a depth map of a scene. The APP electronic device is advantageously a portable device operating on battery, and the DIS device is for example included in an on-board system for acquisition and stereoscopic processing.
    The device DIS includes a stereoscopic image acquisition means 20 configured to implement a stereoscopic acquisition as previously detailed, a time-of-flight sensor 40 configured to produce a map of the distances 400 of the scene in such a way as previously detailed, a stereoscopic processing means 30 configured to implement the stereoscopic processing 300 in a manner as previously detailed and an elementary processing means 31 configured to implement the elementary processing 310 in a manner as previously detailed.
    For example, the stereoscopic acquisition means 20 comprises two objectives with a focal length between 20mm and 35mm, a field of vision between 60 ° and 80 °, and whose optical axes are parallel, as well as two image sensors of 12 megapixels, forming two horizontally aligned viewpoints spaced 12mm apart.
    For example, the time-of-flight sensor 40 is placed between the two viewpoints and is of the compact all-in-one sensor type. The time-of-flight sensor 40 can operate in the infrared spectrum at 940nm, have a field of vision compatible with that of the stereoscopic acquisition means 40, a range of 3.5m, an accuracy of
    2cm, low energy consumption (20pW at rest and 35mW in operation), a matrix of 5 * 3 acquisition zones or 15 * 9 acquisition zones, and a 32-bit autonomous calculation unit.
    The stereoscopic processing means 30 and elementary processing means 31 may or may not be integrated into the same integrated circuit of a microcontroller type calculation unit.
    Furthermore, the invention is not limited to these embodiments and implementation but embraces all the variants, for example the different examples described above can be adapted to the constraints of a particular stereoscopic treatment and are moreover cumulable. between them. In addition, the various quantitative information given, such as the performance of the various equipment, has been presented by way of example in the context of a definition of a technological context.
    权利要求:
    Claims (25)
    [1" id="c-fr-0001]
    1. Method for determining a depth map (100) of a scene (10), comprising:
    - drawing up a distance map (400) of the scene (10) obtained by time of flight measurements,
    - an acquisition of two images of the scene (201, 202) from two different points of view, and
    - a stereoscopic processing (300) between said two images (201, 202) taking into account said distance map (400), the preparation of said distance map (400) comprising the preparation of distance histograms (410) by acquisition areas (420) of the scene, and said stereoscopic processing (300) comprising, for each region (120) of the depth map (100) corresponding to an acquisition area (420), an elementary processing (310 ) taking into account the corresponding histogram (410).
    [2" id="c-fr-0002]
    2. Method according to claim 1, in which the stereoscopic processing (300) comprises an extrapolation (210) of the distances of the scene from disparities (211) between said two images (201, 202).
    [3" id="c-fr-0003]
    3. Method according to claim 2, in which said elementary processing (310) limits the calculations of said extrapolation (210), and / or improves the reliability of the result of said extrapolation (210) and / or adds additional depth information to extrapolation (210) of the distances from the scene (10) in the depth map (100).
    [4" id="c-fr-0004]
    4. Method according to any one of the preceding claims, in which the resolution (208) of the depth map (100) is at least a thousand times greater than the resolution (408) of the distance map measured by time of flight ( 400), the resolution of the map of distances measured by time of flight (400) being equal to the number of acquisition zones (420).
    [5" id="c-fr-0005]
    5. Method according to any one of the preceding claims, in which the distance map (400) of the scene (10) obtained by time of flight measurements comprises from ten to a thousand acquisition zones (420).
    [6" id="c-fr-0006]
    6. Method according to any one of the preceding claims taken in combination with claim 2, in which the respectively maximum (409) and / or minimum distances measurable by flight time are respectively greater and / or less than a ceiling value respectively ( 312) of stereoscopic range and / or a floor value (313) of stereoscopic range.
    [7" id="c-fr-0007]
    7. The method as claimed in claim 6, in which the elementary processing (310) comprises, where appropriate, an identification of at least one region of the depth map (100), called the outport region (122), including the histogram. corresponding (412) does not have a distance respectively less than said ceiling value (312) of stereoscopic range (312) and / or greater than said floor value (313) of stereoscopic range, the extrapolation (210) then not being produced in this at least one out-of-range region (122).
    [8" id="c-fr-0008]
    8. The method of claim 7, wherein said elementary processing (310) comprises assigning a default depth (314) to said at least one out of range region (122), the default depth (314) being derived distances from the corresponding histogram (412).
    [9" id="c-fr-0009]
    9. Method according to any one of the preceding claims taken in combination with claim 2, wherein said elementary processing (310) comprises an allocation of a constant depth (316) to at least one region of the depth map respectively ( 100), known as the flat region (126), the corresponding histogram (416) of which comprises a single group of distances whose half-height width (417) is less than a threshold width (418), the constant depth (316) being equal to a distance from the single group and the extrapolation (210) then not being carried out in this at least one planar region (126).
    [10" id="c-fr-0010]
    10. Method according to any one of the preceding claims taken in combination with claim 2, in which said elementary processing (310) comprises an elaboration of a range of disparity values (322) by region from the extreme values (324 ) of the distances of the corresponding histogram (410), and the extrapolation (210) of the distances of the scene is made by region from values of disparities (211) included in the corresponding range (322).
    [11" id="c-fr-0011]
    11. The method of claim 10, wherein said disparity values (211) included in said corresponding range (322) are taken in a step (326) corresponding to the accuracy of said time of flight measurement.
    [12" id="c-fr-0012]
    12. Method according to any one of the preceding claims taken in combination with claim 2, wherein said elementary processing (310) comprises a determination of the distributions (330) by regions (120) of the distances obtained by said extrapolation (210) and a comparison of the contours (414) of the histograms of distances (410) by acquisition zones (420) with the contours (334) of the distributions (330) of the distances by respective regions (120), said comparison producing a rate of agreement ( 336).
    [13" id="c-fr-0013]
    13. Device for determining a depth map (100) of a scene (10), comprising:
    a time-of-flight sensor (40) configured to produce a distance map (400) of the scene (10), a stereoscopic image acquisition means (20) configured to acquire two images of the scene (201, 202) from two different points of view, and
    - a stereoscopic processing means (30) configured to carry out a stereoscopic processing (300) between said two images (201, 202) taking into account said distance map (400), the time of flight sensor (40) being configured to developing histograms of distances (410) by acquisition zones (420) of the scene, and said stereoscopic processing means (30) comprising an elementary processing means (31) configured, for each region (120) of the map depths (100) corresponding to an acquisition area (420), to perform an elementary processing (310) taking into account the corresponding histogram (410).
    [14" id="c-fr-0014]
    14. Device according to claim 13, in which the stereoscopic processing means (30) is configured to carry out a stereoscopic processing (300) comprising an extrapolation (210) of the distances of the scene from disparities (211) between said two images (201, 202).
    [15" id="c-fr-0015]
    15. Device according to claim 14, in which the elementary processing means (31) is configured to limit the calculations of said extrapolation (210), and / or improve the reliability of the result of said extrapolation (210) and / or add a additional depth information to the extrapolation (210) of the distances from the scene (10) in the depth map (100).
    [16" id="c-fr-0016]
    16. Device according to any one of claims 13 to
    15, in which the resolution (208) of the stereoscopic image acquisition means (20) is at least a thousand times greater than the resolution (408) of the time-of-flight sensor (40), the resolution (408) of the sensor flight time being equal to the number of acquisition zones (420).
    [17" id="c-fr-0017]
    17. Device according to any one of claims 13 to
    16, in which the time-of-flight sensor (40) has ten to one thousand acquisition zones (420).
    [18" id="c-fr-0018]
    18. Device according to any one of claims 13 to 17 taken in combination with claim 14, in which the respectively maximum (409) and / or minimum distances measurable by the time-of-flight sensor are respectively greater than and / or less than respectively a ceiling value (312) of stereoscopic range and / or a floor value (313) of stereoscopic range which can be extrapolated by the stereoscopic processing means (30).
    [19" id="c-fr-0019]
    19. Device according to claim 18, in which the elementary processing means (31) is configured to, if necessary, identify at least one region of the depth map (100), called the out-of-range region (122), of which the corresponding histogram (412) does not have a distance respectively less than said ceiling value of stereoscopic range (312) and / or greater than said floor value (313) of stereoscopic range, the stereoscopic processing means (30) being moreover configured not to carry out the extrapolation (210) in this at least one out of range region (122).
    [20" id="c-fr-0020]
    20. Device according to claim 19, wherein said elementary processing means (31) is configured to assign a default depth (314) to said at least one out-of-range region (222), the default depth (314) being from distances in the corresponding histogram (412).
    [21" id="c-fr-0021]
    21. Device according to any one of claims 13 to
    20 taken in combination with claim 14, wherein said elementary processing means (31) is configured to assign a constant depth (316) to respectively at least one region of the depth map (100), said plane region (126) , of which the corresponding histogram (416) comprises a single group of distances whose half-height width (417) is less than a threshold width (418), the constant depth (316) being equal to a distance from the single group, the stereoscopic processing means (30) being also configured not to carry out the extrapolation (210) in this at least one planar region (126).
    [22" id="c-fr-0022]
    22. Device according to any one of claims 13 to
    21 taken in combination with claim 14, wherein said elementary processing means (31) is configured to develop a range of disparity values (322) by region from the extreme values (324) of the histogram distances (410 ) corresponding, and the stereoscopic processing means (30) is configured to extrapolate (210) the distances of the scene by region from disparity values (211) included in the corresponding range (322).
    [23" id="c-fr-0023]
    23. Device according to claim 22, in which the stereoscopic processing means (30) is configured to carry out the extrapolation (210) of the distances of the scene by region from values of disparities (211) taken in a step (326 ) corresponding to the accuracy of said time of flight measurement.
    [24" id="c-fr-0024]
    24. Device according to any one of claims 13 to
    5 23 taken in combination with claim 14, wherein said elementary processing means (30) is configured to determine distributions (330) by regions (120) of the distances obtained by said extrapolation (210) and to compare contours (414 ) histograms of distances (410) by acquisition zones (420) with the
    10 contours (334) of the distributions (330) of the distances by respective regions (120), said comparison producing a rate of concordances (336).
    [25" id="c-fr-0025]
    25. Electronic device (APP), such as a camera, mobile phone or touch pad, with a device
    15 (DIS) for determining a depth map according to any one of claims 13 to 24.
    3 ° 6 ^;
    2/4
    类似技术:
    公开号 | 公开日 | 专利标题
    FR3063374B1|2019-06-07|METHOD AND DEVICE FOR DETERMINING A DEPTH MAP OF A SCENE
    Shin et al.2016|Photon-efficient imaging with a single-photon camera
    BE1022486B1|2016-05-03|A TOF TUNING APPARATUS SYSTEM AND A METHOD FOR MEASURING DISTANCE WITH THE SYSTEM
    EP2549434B1|2018-10-17|Method of modelling buildings from a georeferenced image
    EP3513552B1|2021-08-25|Systems and methods for improved depth sensing
    TW201044091A|2010-12-16|Apparatus and method for measuring depth and method for computing image defocus and blur status
    EP3073441B1|2019-07-10|Method for correcting an image of at least one object presented remotely to an imager and illuminated by an illumination system and camera system for carrying out said method
    US20200320725A1|2020-10-08|Light projection systems
    EP2943935B1|2017-03-08|Estimation of the movement of an image
    EP2825995B1|2017-03-15|System for identification of a digital camera starting from a picture and method implemented with said system
    EP3555848A1|2019-10-23|Method for implementing a cd-sem characterisation technique
    WO2017093057A1|2017-06-08|Method for characterising a scene by calculating the 3d orientation
    US20200182971A1|2020-06-11|Time of Flight Sensor Module, Method, Apparatus and Computer Program for Determining Distance Information based on Time of Flight Sensor Data
    FR3103301A1|2021-05-21|Method for detecting specularly reflected light beam intensity peaks
    Ma et al.2018|An inverse problem approach to computational active depth from defocus
    WO2015071426A1|2015-05-21|System and method for characterising objects of interest in a scene
    FR3079041A1|2019-09-20|METHOD FOR DETERMINING DISTANCE SEPARATING AN OBJECT AND AN OPTICAL DETECTION SYSTEM, AND CORRESPONDING SYSTEM
    EP3069299B1|2021-12-29|Photography system and method including a system for lighting the scene to be shot
    Wang et al.2022|A systematic non-uniformity correction method for correlation-based ToF imaging
    WO2021213936A1|2021-10-28|Method for georeferencing of optical images
    WO2018161270A1|2018-09-13|Speckle three-dimensional imaging method and device based on spatio-temporal combination
    WO2020136180A1|2020-07-02|Method for segmenting an image
    WO2019077010A1|2019-04-25|Data processing method and associated onboard system
    FR3109628A1|2021-10-29|Digital elevation model geometric calibration process
    FR3097975A1|2021-01-01|PASSIVE TELEMETRY METHOD AND DEVICE BY IMAGE PROCESSING
    同族专利:
    公开号 | 公开日
    FR3063374B1|2019-06-07|
    US20180247422A1|2018-08-30|
    US10445892B2|2019-10-15|
    CN207835674U|2018-09-07|
    US11138749B2|2021-10-05|
    US20190385320A1|2019-12-19|
    CN108513121A|2018-09-07|
    US20220005214A1|2022-01-06|
    CN108513121B|2021-02-05|
    引用文献:
    公开号 | 申请日 | 公开日 | 申请人 | 专利标题
    US20150287192A1|2012-12-20|2015-10-08|Olympus Corporation|Image processing device, electronic device, endoscope apparatus, information storage device, and image processing method|
    US20150041625A1|2013-08-06|2015-02-12|Stmicroelectronics Limited|Time to digital converter and applications thereof|
    US7852461B2|2007-11-15|2010-12-14|Microsoft International Holdings B.V.|Dual mode depth imaging|
    WO2011005316A1|2009-07-07|2011-01-13|Thomson Licensing|Method and system for brightness correction for three-dimensional projection|
    US9297889B2|2012-08-14|2016-03-29|Microsoft Technology Licensing, Llc|Illumination light projection for a depth camera|
    US20140363097A1|2013-06-06|2014-12-11|Etron Technology, Inc.|Image capture system and operation method thereof|
    US20160360185A1|2015-06-03|2016-12-08|Empire Technology Development Llc|Three-dimensional imaging sensor calibration|
    US9927253B2|2016-05-11|2018-03-27|GE Lighting Solutions, LLC|System and stereoscopic range determination method for a roadway lighting system|
    FR3063374B1|2017-02-27|2019-06-07|Stmicroelectronics Sa|METHOD AND DEVICE FOR DETERMINING A DEPTH MAP OF A SCENE|FR3063374B1|2017-02-27|2019-06-07|Stmicroelectronics Sa|METHOD AND DEVICE FOR DETERMINING A DEPTH MAP OF A SCENE|
    CA3066502A1|2017-06-21|2018-12-27|Vancouver Computer Vision Ltd.|Determining positions and orientations of objects|
    WO2020185446A1|2019-03-08|2020-09-17|Array Photonics, Inc.|Electronic devices having displays with infrared components behind the displays|
    CN110097592B|2019-04-07|2020-12-15|杭州晶一智能科技有限公司|Semantic description method of ground information|
    JP2021165692A|2020-04-08|2021-10-14|ソニーセミコンダクタソリューションズ株式会社|Ranging device and ranging method|
    法律状态:
    2018-01-22| PLFP| Fee payment|Year of fee payment: 2 |
    2018-08-31| PLSC| Publication of the preliminary search report|Effective date: 20180831 |
    2020-01-22| PLFP| Fee payment|Year of fee payment: 4 |
    2021-11-12| ST| Notification of lapse|Effective date: 20211005 |
    优先权:
    申请号 | 申请日 | 专利标题
    FR1751539|2017-02-27|
    FR1751539A|FR3063374B1|2017-02-27|2017-02-27|METHOD AND DEVICE FOR DETERMINING A DEPTH MAP OF A SCENE|FR1751539A| FR3063374B1|2017-02-27|2017-02-27|METHOD AND DEVICE FOR DETERMINING A DEPTH MAP OF A SCENE|
    CN201710763275.4A| CN108513121B|2017-02-27|2017-08-30|Method and apparatus for depth map evaluation of a scene|
    CN201721097448.5U| CN207835674U|2017-02-27|2017-08-30|Electronic device and electronic equipment|
    US15/692,794| US10445892B2|2017-02-27|2017-08-31|Method and apparatus for depth-map estimation of a scene|
    US16/548,138| US11138749B2|2017-02-27|2019-08-22|Method and apparatus for depth-map estimation of a scene|
    US17/478,643| US20220005214A1|2017-02-27|2021-09-17|Method and apparatus for depth-map estimation of a scene|
    [返回顶部]