• 专利附录
  • 专利说明
  • 权利要求
  • 类似技术
  • 同族专利
  • 引用文献
  • 法律状态
  • 优先权
    • 专利摘要:
    The present invention relates to a method of characterizing a sample, using a set of spectral images of the sample to be characterized previously acquired, in particular by infrared thermography or spectral imaging, and at least one neural network, the method comprising the steps comprising: - generating at least one volume of values of an observed parameter from said spectral images, for a plurality of pixel coordinates of the images and a plurality of acquisitions, - extracting at least one input data set from from said data volume, these input data corresponding to the values of the observed parameter, for a pixel of the same coordinates according to different acquisitions, values to which at least one transformation function has been applied, - to drive said at least one neural network into using the input data to extract at least one characteristic from the sample terize.
    公开号:FR3074596A1
    申请号:FR1761522
    申请日:2017-12-01
    公开日:2019-06-07
    发明作者:Valeriu Vrabie；Eric Perrin；Sihem Mezghani
    申请人:Universite de Reims Champagne Ardenne URCA；
    IPC主号:
    专利说明:

    NEURONES NETWORKS
    The present invention relates to methods and devices for characterizing samples from spectral images, in particular acquired by infrared thermography, and using deep neural networks.
    Most known surveillance systems, in particular for accident prevention, traffic routing, or decision, for example for the non-destructive detection and / or control of components and / or various phenomena, rely on the use of numerous sensors and on the use of known detection techniques. Infrared, near-infrared (NIR) and ultraviolet (UV) radiation can be used. In particular, far infrared electromagnetic radiation, also called radiative heat, permanently emitted by any body having a temperature above absolute zero (-273.15 ° C) is used. Specific detectors make it possible to capture this radiation in certain wavelengths and to transcribe it into luminance values linked to the surface temperature of the object, creating thermal images.
    The miniaturization of infrared thermographic cameras, the drop in their acquisition cost and the development of computer computing capacities have encouraged the use of such cameras as a non-destructive alternative technique in several applications, such as industrial inspection, evaluation damage, fatigue of materials, or estimation of coating thickness. The possibility of penetrating the coating layer without having any influence on the pigments justifies the use of infrared techniques for the inspection of the thicknesses of coatings such as paint.
    This technique has certain limitations, such as a high sensitivity to external reflection, variations in emissivity, and the use of a heat source as an excitation source which cannot be considered uniform from an energy point of view, for example because of the use of one or more high power flashes. This inhomogeneity will directly influence the thermal signature of the coating of the target observed both during the heating and cooling periods. In order to solve this problem, the temperature distribution during a thermographic inspection was studied and measures aimed at reducing the effects of the non-uniform temperature distribution were suggested, such as the use of a reconstruction algorithm. images based on a Fourier transformation to inhibit the effect of non-uniform heating. Other methods are used to improve thermal contrast and overcome these external artifacts, including the use of thermal contrast, absolute thermal contrast, or modified absolute differential contrast. It is also known to use an algorithm based on partial least squares regression to automatically improve the visibility of defects in samples by partially eliminating background noise. Other methods based on the use of higher order statistics or the decomposition into singular values have been developed. However, these denoising results are not yet optimal.
    A perceptron multilayer neural network was used to detect and characterize defects using pulsed infrared thermography. The results show that phase images are less sensitive to noise, but an increase in sampling frequency is highly recommended for this study. An illustration of such results is found, for example, in the article Defect detection in pulsed thermography: a comparison of Kohonen and Perceptron neural networks, by Steve Vallerand et al. Proc. SPIE 3700, Thermosense XXI, March 1999.
    It has recently been demonstrated that it is possible to use deep learning algorithms, known as “deep learning” in English, to classify data, for example images, sounds, or texts, by extracting characteristics to represent the data at different levels of abstraction. The use of deep learning makes it possible to obtain robust results and to envisage various applications. These algorithms implement models made up of several supervised or unsupervised layers, where the non-linear stages of information processing are by nature hierarchical. Convolutional neural networks (CNNs) are, in known manner composed of several layers, each layer acting as a filter and causing a reduction in dimensionality of the data which is then transmitted to the next layer. The layers are composed of neurons, themselves composed of an activation function, a weight and a bias on each of its inputs.
    As shown in FIG. F illustrating F state of the art, in such structures, input data DE are received at the level of sensors SI, S2, S3, each sensor being able to collect your signals representative of a characteristic F ( or "feature" in English) of the sample to be analyzed. These signals are transmitted to a neural network M made up of layered muitipies. This model M is trained with F using one or more iots of training data DApp, in particular made up of images, at the level of which characteristics F have been annotated and which are subjected to a learning algorithm AApp allowing the learning to recognize these characteristics. A classification C is obtained at output, according to a set of classes C1, C2, C3, of the data supplied as input DE.
    These algorithms (and in particular convolutional neural networks) are based on the learning of many data models, they are therefore very greedy in computation and require a large database for their learning. The implementation of these algorithms on embedded systems is limited by the computing capacity and by the resources, notably memory, available. Recently developed systems, using for example a Raspberry Pi-type nanocomputer, are currently able to analyze between 2 and 4 frames of image per second from a video stream. It’s not yet fast enough to handle video streams satisfactorily.
    CN 10 6022365 discloses a method for detecting defects on the surface of a material, using a network of neurons RBF ("radial basis function" in English) and infrared thermography images to create a classifier.
    Application CN 10 5760883 describes, in the field of monitoring the operating condition of mining equipment, a method for automatically identifying the key components of a belt conveyor, using infrared thermography and a so-called neural network BP (“back-propagation” in English) to extract the characteristics of the components.
    Application CN 10 2621150 relates to a method for identifying damage to an aircraft coating, using a Large Margin Separator (SVM) algorithm based on a gray level co-occurrence matrix and a signal characterizing different types of damage to establish a classifier. Detection of damage to the aircraft skin and the damage itself can be classified and identified for further maintenance processing.
    Other approaches for performing learning tasks based in particular on specific software and DSP systems (“digital signal processor” in English), or else using FPGAs (“Field Programmable Gâte Array” in English) as presented by example in the article “Optimizing Convolutional Neural Network on DSP” by S. Jagannathan et al., IEEE International Conference on Consumer Electronics 2016, allow to reach interesting performances, approximately three times faster than systems based on graphics processors GPU ("graphie processor unit" in English).
    It is known to use pre-trained CNN networks by modifying only their final layers, which makes it possible to offer hybrid networks made up of a pre-trained CNN network, in particular on a machine with high computing power, and d '' a supervised classifier to train, for example an SVM algorithm. This makes it possible to reduce the necessary learning base and to envisage its implementation on an embedded system at low cost, for example on a nanocomputer of the Raspberry Pi type. The article by Huang FJ et al, “Large-scale learning with SVM and convolutional nets for generic object categorization ”, IEEE CVPR 2006, vol 2, pp 4, describes a hybrid network made up of a convolutional network and an SVM, dedicated to the classification of objects.
    International application WO 99/05487 describes the use of a fiber optic probe and a hybrid neural network to increase the accuracy of the analysis of tissue lesions.
    There therefore remains a need to further improve the reliable estimation of certain characteristics of samples from spectral images obtained using a device whose excitation conditions are variable and not uniform, in particular via a characterization device by infrared thermography.
    The invention responds to the need mentioned above thanks, according to one of its aspects, to a method for characterizing a sample, using a set of spectral images of the sample to be characterized previously acquired, in particular by infrared thermography or spectral imaging, and at least one neural network, the method comprising the steps consisting in:
    generate at least one volume of values D (Nx, Ny, Ne) of a parameter observed from said spectral images, for a plurality of coordinates (x, y) of the pixels N of the images and a plurality of acquisitions Ne,
    extracting at least one set of input data D'x, y (Ne) from said volume of data D (Nx, Ny, Ne), these input data corresponding to the values of the observed parameter, for a pixel of same coordinates according to different Ne acquisitions, values to which at least one transformation function has been applied,
    - training said at least one neural network, in particular at least one layer of the network, using the input data to extract at least one characteristic of the sample to be characterized, and
    - use said at least one characteristic extracted by the neural network to perform a classification of the input data according to a plurality of classes, each class being representative of at least one characteristic of the sample to be characterized.
    Preferably, said at least one neural network, in particular at least one layer of the network, has been trained beforehand on images other than spectral images, called “natural images”, in particular natural images of animals, objects , plants, people. Only the final layer of the network needs to be trained with the input data.
    Preferably, the classification is carried out by a classifier independent of the neural network. The classifier can be of the wide margin separator (SVM) type. In a variant, the classifier is of the Softamx or RBF type with a Gaussian nucleus. The classification can be done at the level of at least one layer of a perceptron. In a variant, the classification is carried out by the neural network used for the extraction of the characteristics, preferably by the final layer of this network.
    Thanks to the use of a hybrid structure comprising a pre-trained network and a classifier, the invention allows rapid implementation of the system, the database and the learning time being reduced. The implementation of the method according to the invention can thus be done on an on-board system at low cost, for example a nanocomputer, also called nano-PC, or a dedicated card, allowing the operation of resource-intensive applications and / or requiring real-time.
    The use of an SVM classifier makes it possible to work with large data, which makes it possible to process a large number of data of different types, to improve recognition performance significantly, and to considerably reduce computation times .
    The robustness of the hybrid architecture according to the invention makes it possible to have a post-processing technique of infrared thermographic data weakly sensitive to the non-uniformity of the energy deposit generated by the excitation system and to the measurement conditions, for example, different placements of the spectral image acquisition camera in terms of distances or angles of the lens relative to the sample, or lighting conditions for different acquisitions, depending on the time, the temperature, or the season of the year.
    One-dimensional signals
    The different acquisitions can correspond to different acquisition times over a predefined acquisition period, in particular in the case of infrared thermography.
    In a variant, in particular in the case of spectral imaging, the different acquisitions correspond to acquisitions according to different wavelengths, carried out at the same time.
    Spectral imagery groups multispectral or hyper spectral imagery. Multispectral imaging consists of acquiring a small and limited number of discrete bands, and does not require the use of a spectrometer to analyze the data. Hyperspectral imaging allows the acquisition of a large number of narrow spectral bands through the use of a spectral band separation system such as a spectrometer.
    The data volume D advantageously contains P pixels, for each pixel N in the x plane, there correspond the coordinates (Nx, Ny, Ne), where Ne is the coordinate of the acquisition. This is the same pixel P from one plane to another, recorded at different times or for different wavelengths.
    The spectral evolution of a pixel, for example the temporal evolution of the temperature, is thus considered as a one-dimensional signal, forming a n-tuple of values, and used directly for classification.
    The input data are advantageously transmitted to the neural network in the form of images representing curves corresponding to the values D’x, y (Ne) of the input data set as a function of the acquisition Ne. This allows the transposition to one-dimensional signals of the principles of deep learning type networks for the study of natural images, in particular convolution and reduction of dimensionality for the extraction of the characteristics. The fact that the neural network is pre-trained on natural images reduces the amount of training data required.
    For learning, the images can be resized according to the standard dimensions imposed by the neural network used.
    The transformation function applied to the values Dx, y (Ne) of the observed parameter can be the identity function, the values remaining unchanged and being used as such by the neural network.
    In a variant, said at least one transformation function applied to the values of the observed parameter Dx, y (Ne) is a centering, normalization and / or smoothing function. The spectral responses Dx, y (Ne) can thus each be centered, for example with respect to an average value calculated over all of the images used for learning said network, or with respect to the first image resulting from the first acquisition, and / or normalized with respect to their maximum, or with respect to a reference value, corresponding in particular to a predefined wavelength.
    The smoothing method used can be the so-called “Savitzky and Golay” (SG) method, consisting in approximating on a sliding window of size m the spectral response segment using a polynomial of degree n, with m included. between 10 and 20 points and n between 1 and 6, for example m = 15 points and n = 4. Smoothing makes it possible to reduce the irregularities and singularities of the responses. Using smooth spectral responses to calculate derivatives helps avoid the appearance of artifacts or amplification of noise due to the derivative in the resulting signals.
    A function for calculating the first derivative can be applied to the values D (Nx, Ny, Ne) of the observed parameter to obtain the input data (D’x, y (Ne)). In the case where the observed parameter is the temperature of the sample, this calculation makes it possible to take into account the rate of cooling of the sample.
    A function for calculating the second derivative can be applied to the values D (Nx, Ny, Ne) of the observed parameter to obtain the input data (D’x, y (Ne)). In the case where the observed parameter is the temperature of the sample, this calculation makes it possible to take into account the acceleration of the cooling of the sample.
    Infrared thermography
    Preferably, the spectral images used are thermal images acquired by infrared thermography, the parameter observed being the temperature of the sample.
    The principle of infrared thermography is based on the measurement of the energy emitted by the surface of a body in a given interval of wavelengths, corresponding to the temporal acquisition of thermal radiation in the infrared bands of the electromagnetic spectrum. This energy is transmitted through appropriate optics to a detector. These so-called radiometric systems allow non-contact measurement of surface temperature fields at rates reaching several hundred Hertz for images whose average size is around 80,000 pixels. The measurement process usually only causes a slight increase in temperature, which is unlikely to interfere with data acquisition. In known manner, the detectable temperature variations range from a few tenths to a few tens of degrees Celsius / Kelvin. A thermal excitation of the specimen can be carried out. The digital analysis of the thermal data is then carried out.
    The surface of the sample to be characterized can be thermally excited prior to the acquisition of thermal images.
    Neural network and feature extraction
    Said at least one neural network can be a convolutional neural network.
    The neural network can include one or more convolutional layers and / or one or more fully connected layers.
    As is known, each convolution layer produces an activation of an input image, the first layers extracting basic characteristics, such as the outline, and the upper layers extracting higher level characteristics, such as texture information. .
    The characteristics extracted from the input data can be the thickness or the range of thickness of the sample or of certain parts of the sample, an amount representative of a property of the sample, for example a thickness layer of paint, a thickness of an intermediate layer, for example of the Sol Gel type originating from solution-gelling processes, the level of water stress of a plant, the variation in pigmentation of plants, for example leaves, flowers or fruits of plants.
    Characterization device
    Another subject of the invention, according to another of its aspects, is a device for characterizing a sample, in particular by infrared thermography or spectral imaging, comprising:
    - a means of acquiring a set of spectral images of the sample to be characterized, in particular a thermal camera,
    - a data processing module capable of:
    • generate at least one volume of values (D (Nx, Ny, Ne)) of a parameter observed from said spectral images, for a plurality of coordinates (Nx, Ny) of the pixels of the images and a plurality of acquisitions ( Ne), • extract at least one set of input data (D'x, y (Ne)) from said data volume (D (Nx, Ny, Ne)), these input data corresponding to the values of the observed parameter, for a pixel with the same coordinates (x, y) according to different acquisitions (Ne), to which at least one transformation function has been applied, and
    an analysis module, comprising at least one neural network, capable of at least training said at least one neural network by using the input data to extract at least one characteristic of the sample to be characterized, and to using said characteristic extracted by the neural network to classify the input data according to a plurality of classes, each class being representative of at least one characteristic of the sample to be characterized.
    Said at least one neural network has preferably been trained beforehand on images other than spectral images, in particular natural images of animals, objects, plants, people.
    The neural network can include one or more convolutional layers and / or one or more fully connected layers.
    The device according to the invention may include a classifier independent of the neural network to perform the classification.
    The device may include a means of thermal excitation of the sample to be characterized, in particular a means of surface excitation by illumination, preferably a means of surface excitation by pulsed illumination such as a flash lamp.
    The device can further comprise a decision-making module communicating with the analysis module, and an action means able to act on the sample, said decision-making module being able to slave said action means retroactively. according to the classification results obtained from said analysis module and to trigger an appropriate action towards the sample. This allows for sample control and reliable monitoring, for example in non-destructive testing applications.
    The means of action may be a nozzle for spraying a plant protection product on crops, capable of spraying a quantity of product adapted to the classification results. The characteristics to be extracted can be the leaf mass, the type of vegetation, or even the type of disease of the plant examined.
    The characteristics set out above for the process apply to the device and vice versa.
    The characterization device according to the invention can be easily installed on an on-board system either at low cost, for example a nano-PC for example of the Raspberry PI type, or specific, for example a dedicated card of the TX1 type of Nvidia®, according to the intended application. Most of the processing can take place in the intelligent embedded system, for example a nano-PC that can be equipped with a high-definition camera.
    Control process
    Another subject of the invention, according to another of its aspects, is a method for controlling a sample, comprising the step of generating with the device for characterizing a sample as defined above, as a function of the results of classification, information relating to the sample with a view to making a decision consisting in deciding on an action to be taken towards the sample to be characterized, and in particular in transmitting an action instruction to a means of action capable of implement it.
    Computer program product
    Another subject of the invention, according to another of its aspects, is a computer program product for implementing the method for characterizing a sample as defined above, using a set of spectral images of the sample. to characterize previously acquired, in particular by infrared thermography or spectral imaging, and at least one neural network, the computer program product comprising a support and recorded on this support instructions readable by a processor for when executed:
    - at least one volume of values (D (Nx, Ny, Ne)) of a parameter observed from said spectral images is generated, for a plurality of coordinates (Nx, Ny) of the pixels of the images and a plurality of acquisitions (Born),
    - at least one set of input data (D'x, y (Ne)) is extracted from said volume of data (D (Nx, Ny, Ne)), these input data corresponding to the values of the observed parameter , for a pixel with the same coordinates (x, y) according to different acquisitions (Ne), to which at least one transformation function has been applied,
    - said at least one neural network is trained using the input data to extract at least one characteristic of the sample to be characterized, and
    - Said at least one characteristic extracted by the neural network is used to classify the input data according to a plurality of classes, each class being representative of at least one characteristic of the sample to be characterized.
    detailed description
    The invention can be better understood on reading the description which follows, of non-limiting examples of implementation of the invention, and on examining the appended drawing, in which:
    FIG. 1, previously described, illustrates a classification method using a deep learning algorithm according to the prior art,
    FIG. 2 represents an example of steps for preparing the data in the method according to the invention, with a view to providing them to at least one neural network,
    FIG. 3 illustrates steps for classifying data by a neural network in the method according to the invention,
    FIG. 4 represents an example of a device for characterizing a sample according to the invention, and
    - Figures 5 to 10 are tables showing the classification performance of the process according to the invention.
    FIG. 2 shows an example of steps for preparing data in the method according to the invention, with a view to providing them to at least one neural network. In this example, spectral images are acquired by a thermal camera, thus forming so-called thermal images. The parameter observed from these images is the temperature of the surface of sample E.
    From a sample E to be characterized, during a step A, a set of thermal images 2 is acquired over a predefined acquisition period T, for example using an infrared thermal camera.
    During a step B, a volume of data D (Nx, Ny, Ne) corresponding to the instantaneous temperature values is generated from the thermal images 2, in a frame where Nx and Ny correspond to the coordinates of the pixels N of the images 2 in the directions (x, y) and Ne corresponds to the acquisition, expressed either in number of the image or in time, or in wavelength in the case of multispectral or hyperspectral imagery.
    A one-dimensional input data set D’x, y (Ne) is extracted from the data volume D (Nx, Ny, Ne), during a step EP. In the example considered, these input data D'x, y (Ne) correspond to the instantaneous temperature values, for a pixel with the same coordinates (x, y) according to different recording times Ne, at which at least one function transformation is applied, detailed below.
    During a step I, images Ux representing curves corresponding to the values D'x, y of the input data set as a function of the recording time Ne are generated for each data set, and are transmitted to a network of neurons RN (cnn), in convolution in the example considered. The neural network used in the method according to the invention can nevertheless be of any type.
    Advantageously, the method according to the invention further comprises a step PT for preprocessing the data following the extraction step EP. Preferably, this step PT of data preparation consists in applying at least one transformation function to the values Dx, y (Ne), in particular a centering, normalization and / or smoothing function. During a step NL, different functions can be applied to the input data set, for example the identity function forming the OJ set corresponding to the original data set, pre-processed or not, and / or a function for calculating the first derivative forming the set J1, and / or a function for calculating the second derivative forming the set J2.
    In the example considered, in an application for evaluating the heterogeneity of a layer of paint, the sample E to be inspected is a metallic steel plate of dimension 370 × 500 mm with a layer of paint coating deposited in 4 strips. the thicknesses of which vary from 59 to 95 μm, as can be seen in FIG. 2. This sample was placed horizontally against an insulating support in order to avoid any phenomenon of conduction between the sample and the ground. A global surface thermal excitation of the sample can be carried out in order to have a high speed of heating of the entire surface. Preferably and as in the example illustrated, the pulsed infrared thermography technique is used: a thermal wave is sent to the sample surface whose excitation profile is as close as possible to a Dirac pulse. Halogen lamps can be used, but the temperature rise caused by long-term lighting can damage the surface of the sample. Alternatively and preferably, several flash lamps generating a large amount of energy in a very short time, positioned at different angles, are used.
    In the example considered, the thermal camera records every 5 milliseconds a thermal image, or thermogram, of the front face of the surface of the sample. Following the acquisition of these thermal images, a volume of data is generated as described above. The acquisition period is between ... 0.5 seconds and 2 seconds, being for example equal to 1 seconds.
    As described above and as visible in FIG. 3, each pixel of the data volume recorded by the camera is a one-dimensional signal which is represented by an image Ux used as input of a neural network RNcnn. This network of neurons RNcnn has preferably been trained beforehand on images other than spectral images, said to be natural, originating for example from the database “www.image.net” containing more than 1000 classes of images and more a million images. This neural network can belong to an MP analysis model further comprising a P perceptron, comprising for example an input layer, one or more hidden layers and an output layer. This MP analysis model is initially able to classify images by the neuron network RN and then the perceptron P according to classes K (Kl, K2 ...).
    As described above, the neural network is trained using the input data Ux to extract FE characteristics, corresponding to different thicknesses of paint in the example considered. In the case where the neural network belongs to an MP analysis model comprising a P perceptron, the extraction of the characteristics can be carried out by at least one of the layers of the perceptron.
    The extracted characteristics are used to classify the thermal responses according to a plurality of classes C1, C2, each class being representative of a characteristic of the sample to be characterized, here different thicknesses. The classification is, in this example and preferably, carried out by a classifier independent of the neural network, of the wide margin separator (SVM) type.
    In the example under consideration, each of the paint coating thicknesses visible in FIG. 2 is associated with a class C1, C2, C3, C4, of different spatial dimensions. Acquisitions on this coating were made at two different times Tl and T2, creating two volumes of data, the first of dimensions 110x611x400, and the second of dimensions 103x631x400. Reconstituting identical measurement conditions between each measurement being difficult, that is to say reproducing the same positioning of the samples, the lamps, and the camera, the data volumes do not have the same dimensions, except for the temporal dimension since the acquisition period is always the same. As previously described, one-dimensional datasets J0, J1 and J2 were generated from these volumes.
    Figure 5 shows the classification performance for the J0 dataset from the second volume, including a set of 8000 signals which were randomly selected (2000 for each class). 70% of the data was randomly selected for learning, corresponding to the "training data set", and the remaining 30% for the classification test, corresponding to the "test data set". We observe on the diagonal that 97% of the measurements of class 1, 95% of those of class 2, 92% of those of class 3 and 91% of those of class 4 are well classified and that the average precision is 93.5%.
    Similarly, Figure 6 shows the classification performance for the data set Jl. We see by looking at the diagonal that 96% of the measurements of class 1, 91% of those of class 2, 91% of those of class 3 and 87% of those of class 4 are well classified and the average precision is 91.25%.
    Figure 7 shows the classification performance for the J2 dataset. We see by looking at the diagonal that 94% of the measurements of class 1, 86% of those of class 2, 85% of those of class 3 and 89% of those of class 4 are well classified and that the average precision is therefore 88.1%. The comparison of the classification results on the 3 games JO, Jl and J2 shows that the best results are those obtained by taking the standardized and smoothed JO input data.
    The method according to the invention makes it possible to recognize the thermal response of each pixel and to associate it with the correct class, in order to find in a reliable manner, in the example described, the thickness of each strip of coating of the sample.
    The table in FIG. 8 shows the independence of the classification results with respect to the energy deposition, that is to say the verification of the fact that a variation in the homogeneity of the energy deposition on the sample has a very small impact on the classification results of each of the pixels in the data volume. This allows experimental measurements to be carried out in which the uniformity parameter of the surface energy deposition is not very troublesome.
    These classification performances were established for an OJ data set containing 16,000 signals, ie 4,000 signals per class, each class comprising 1,000 signals acquired with four flash lamps, 1,000 signals acquired with three flash lamps, 1,000 signals acquired with two flash lamps, and 1000 signals acquired with a single flash lamp. Each of the classes thus comprises pixels lit by a different number of “flash” lamps. This type of data selection significantly increases the different types of thermal responses depending on the lighting and therefore allows the RNcnn network to extract the most significant characteristics possible, increasing the performance of the algorithm and therefore the reliability of classification results.
    We see by looking at the diagonal that 97% of measures in class 1, 92% of measures in class 2, 86% of measures in class 3 and 91% of measures in class 4 are well classified. The average precision is 91.5%.
    The table in FIG. 9 was produced by applying the classifier to 4000 signals different from those used for learning and extracting the characteristics by the neural network RNcnn, but acquired at the same time period T2, for which each class is constituted in the same way as during learning, i.e. 1000 signals with 4 flashes, 1000 signals with 3 flashes, 1000 signals with 2 flashes, and 1000 signals with 1 flash. As shown in Figure 9, 93.075% of the measures in class 1, 97.65% of the measures in class 2, 97.8% of the measures in class 3, and 94.8% of the measures in class 4 are correctly classified.
    The table in Figure 10 shows the independence of the classification results from the measurement conditions. The achievement of two strictly identical measurements spaced over time on the same sample is indeed almost impossible to obtain. Lamp positions, distance from the camera to the sample, tilt angles, and rotation of the sample relative to the camera are variable parameters that can affect classification results.
    In this example, the training data set, acquired at a time period T2, is identical to that used to produce the table in FIG. 9, the classification performance being evaluated by applying the algorithm to 4000 signals different from the signals used for learning, and acquired at a different time period Tl, and for which each class is constituted in the same way as during learning, that is to say 1000 signals with 4 flashes, 1000 signals with 3 flashes, 1000 signals with 2 flashes, and 1000 signals with 1 flash.
    As shown in Figure 10, 98.6% of the measures in class 1, 91.65% of the measures in class 2, 91.975% of the measures in class 3, and 95.65% of the measures in class 4 are well classified. The classification results of the different pixels belonging to the 4 different classes is very weakly dependent on the variations in measurement conditions.
    We will now describe, with reference to FIG. 4, an example of a device 20 for characterizing a sample E according to the invention. This device 20 is, in the example considered, a non-destructive control drone by spectral imaging.
    The characterization device 20 comprises a means AQ for acquiring a set of spectral images of the sample E, in particular a thermal or spectral camera, scanning all the spectral ranges, for example infrared, visible, near infrared, the medium infrared, the far infrared corresponding to thermal or thermographic infrared, and any combination of these ranges, in particular visible and near infrared.
    The characterization device 20 also comprises a data processing module MAM capable of generating at least one volume of instantaneous values of data D (Nx, Ny, Ne), and of extracting at least one set of input data D'x , y (Ne) from said data volume D (Nx, Ny, Ne), as described above.
    In the example considered, the characterization device 20 further comprises an MP analysis module, comprising at least one neural network RNcnn making it possible to extract characteristics from the input data, the MP analysis module being able to using the extracted characteristics to classify the input data according to a plurality of classes.
    The characterization device 20 may include a means of thermal excitation EX of the sample E to be characterized.
    The characterization device 20 can further comprise a decision-making module MD communicating with the analysis module MP and an action means ACT capable of acting on the sample E. The decision-making module MP is advantageously capable to enslave said action means ACT retroactively as a function of the classification results obtained and to trigger an appropriate action towards the sample E.
    The invention is not limited to the examples which have just been described.
    One possible application of the invention is the design of an intelligent spraying device 20, within the framework of intelligent agriculture or smart-agriculture. In such an intelligent sprayer, each spray nozzle, corresponding to the ACT action means, can be associated with an on-board hybrid network according to the invention, for example, on a nano-PC using a low-cost camera, for example in the field visible and / or near infrared which makes it possible to detect the leaf mass, the type of vegetation, even the type of disease of the plant examined, so as to spray the right phytosanitary product with the quantity just necessary. Multispectral or hyperspectral imagery can then be used to acquire the images. The plants can be corn, wheat, or vine. Plant growth can also be monitored by the invention and the possible occurrence of a disease can be predicted.
    The invention is not limited to a convolutional neural network. “Deep Neural Network” (DNN) or “Deep Belief Network” (DBN) networks can be considered.
    It is possible to process several groups of data in parallel by parallel networks.
    The invention can be implemented on any type of hardware, for example a personal computer, a smart phone, a nanocomputer, or a dedicated card.
    The invention is not limited to applications for characterizing coatings by infrared thermography. Radiation in the visible, near-infrared, mid-infrared, far infrared, Terahertz or ultraviolet domains could be used. The invention is particularly suitable for non-destructive testing applications, in order to preserve the quality of the samples tested.
    The invention can be used in various applications, for example in low-cost intelligent on-board sensors, or in decentralized fog computing infrastructures, in which the objective is to gain in efficiency and reduce the amount of data transferred.
    The invention can be used in many other fields, such as the military, electrical monitoring, geology, or even biology or bioinformatics, in particular for monitoring manufacturing processes and the quality of materials.
    权利要求:
    Claims (22)
    [1" id="c-fr-0001]
    1. Method for characterizing a sample (E), using a set of spectral images of the sample to be characterized previously acquired, in particular by infrared thermography or spectral imaging, and at least one neural network (RNcnn), the method comprising the steps of:
    generating at least one volume of values (D (Nx, Ny, Ne)) of a parameter observed from said spectral images, for a plurality of coordinates (x, y) of the pixels (N) of the images and a plurality of 'acquisitions (Ne),
    - extract at least one set (Jx) of input data (D'x, y (Ne)) from said data volume (D (Nx, Ny, Ne)), these input data corresponding to the values of observed parameter, for a pixel with the same coordinates according to different acquisitions (Ne), values to which at least one transformation function has been applied,
    - train said at least one neural network (RNcnn) using the input data (Jx) to extract at least one characteristic of the sample (E) to be characterized, and
    - using said at least one characteristic extracted by the neural network (RNcnn) to classify the input data according to a plurality of classes, each class being representative of at least one characteristic of the sample (E) to be characterized .
    [2" id="c-fr-0002]
    2. Method according to claim 1, in which said at least one neural network (RNcnn) has been trained beforehand on images other than spectral images, in particular images of animals, objects, plants, or people.
    [3" id="c-fr-0003]
    3. Method according to claim 1 or 2, wherein said at least one neural network (RNcnn) is a convolutional neural network (CNN).
    [4" id="c-fr-0004]
    4. Method according to any one of claims 1 to 3, in which the classification is carried out by a classifier independent of the neural network.
    [5" id="c-fr-0005]
    5. Method according to the preceding claim, wherein the classifier is of the wide margin separator type.
    [6" id="c-fr-0006]
    6. Method according to any one of the preceding claims, in which the input data (Jx) are transmitted to the neural network (RNcnn) in the form of images (Ux) representing curves corresponding to the values (D'x , y) of the input data set as a function of the acquisition (Ne).
    [7" id="c-fr-0007]
    7. Method according to any one of the preceding claims, in which said at least one transformation function applied to the values of the observed parameter (D (Nx, Ny, Ne)) is a centering, normalization and / or smoothing function. .
    [8" id="c-fr-0008]
    8. Method according to any one of claims 1 to 6, in which the transformation function applied to the values of the observed parameter (D (Nx, Ny, Ne)) is the identity function.
    [9" id="c-fr-0009]
    9. Method according to any one of claims 1 to 7, in which a function for calculating the first derivative is applied to the values of the observed parameter (D (Nx, Ny, Ne)) to obtain the input data (D 'x, y (Ne)).
    [10" id="c-fr-0010]
    10. Method according to any one of the preceding claims, in which a function for calculating the second derivative is applied to the values of the observed parameter (D (Nx, Ny, Ne)) to obtain the input data (D'x y (Ne)).
    [11" id="c-fr-0011]
    11. Method according to any one of the preceding claims, in which the characteristics extracted from the input data are the thickness or the range of thickness of the sample (E) or of certain parts of the sample, a quantity representative of a property of the sample, in particular a thickness of paint layer, a thickness of an intermediate layer, the level of water stress of a plant, the variation in pigmentation of plants.
    [12" id="c-fr-0012]
    12. Method according to any one of the preceding claims, in which the spectral images used are thermal images acquired by infrared thermography, the parameter observed being the temperature of the sample (E).
    [13" id="c-fr-0013]
    13. Method according to the preceding claim, wherein the surface of the sample (E) to be characterized is thermally excited prior to the acquisition of thermal images.
    [14" id="c-fr-0014]
    14. Device (20) for characterizing a sample (E), in particular by infrared thermography or spectral imaging, comprising:
    a means of acquisition (AQ) of a set of spectral images of the sample (E) to be characterized, in particular a thermal camera,
    - a data processing module (MAM) capable of:
    • generate at least one volume of values (D (Nx, Ny, Ne)) of a parameter observed from said spectral images, for a plurality of coordinates (x, y) of the pixels (N) of the images and a plurality of 'acquisitions (Ne), • extract at least one set of input data (D'x, y (Ne)) from said data volume (D (Nx, Ny, Ne)), these data d' input corresponding to the values of the observed parameter, for a pixel with the same coordinates according to different acquisitions (Ne), values to which at least one transformation function has been applied, and
    - an analysis module (MP), comprising at least one neural network (RNcnn), capable of at least training said at least one neural network by using the input data to extract at least one characteristic of it sample (E) to be characterized, and to use said characteristic extracted by the neural network to perform a classification of the input data according to a plurality of classes, each class being representative of a characteristic of the sample (E) to be characterized .
    [15" id="c-fr-0015]
    15. Device according to the preceding claim, wherein said at least one neural network (RNcnn) has been trained beforehand on images other than spectral images, in particular images of animals, objects, plants, or people.
    [16" id="c-fr-0016]
    16. Device according to claim 14 or 15, the neural network (RNcnn) comprising one or more convolutional layers and / or one or more fully connected layers.
    [17" id="c-fr-0017]
    17. Device according to any one of claims 14 to 16, comprising a classifier independent of the neural network (RNcnn) to perform the classification.
    [18" id="c-fr-0018]
    18. Device according to any one of claims 14 to 17, comprising a means of thermal excitation (EX) of the sample (E) to be characterized, in particular a surface excitation means by illumination, preferably a means of surface excitation by pulsed lighting such as a flash lamp.
    [19" id="c-fr-0019]
    19. Device according to any one of claims 14 to 18, further comprising a decision-making module (MD) communicating with the analysis module (MP) and an action means (ACT) capable of acting on the sample (E), said decision-making module being capable of slaving said means of action retroactively as a function of the classification results obtained from said analysis module and of triggering an appropriate action towards the sample.
    [20" id="c-fr-0020]
    20. Device according to the preceding claim, in which the action means (ACT) is a nozzle for spraying a phytosanitary product on crops, capable of spraying an amount of product adapted to the classification results.
    [21" id="c-fr-0021]
    21. A method of checking a sample (E), comprising the step of generating with the device (20) for characterizing a sample according to any one of claims 14 to 19, as a function of the classification results, information relating to the sample with a view to making a decision consisting in deciding on an action to be taken towards the sample to be characterized, and in particular in transmitting an action instruction to a means of action capable of putting it implemented.
    [22" id="c-fr-0022]
    22. Computer program product for implementing the method for characterizing a sample (E) according to any one of claims 1 to 13, using a set of spectral images of the sample to be characterized previously acquired, in particular by infrared thermography or spectral imaging, and at least one neural network (RNcnn), the computer program product comprising a support and recorded on this support instructions readable by a processor for when executed:
    - at least one volume of values (D (Nx, Ny, Ne)) of a parameter observed from said spectral images is generated, for a plurality of coordinates (x, y) of the pixels (N) of the images and a plurality acquisitions (Ne),
    - at least one set of input data (D'x, y (Ne)) is extracted from said volume of data (D (Nx, Ny, Ne)), these input data corresponding to the values of the observed parameter , for a pixel with the same coordinates according to different acquisitions (Ne), values to which at least one transformation function has been applied,
    - said at least one neural network (RNcnn) is trained using the input data to extract at least one characteristic of the sample (E) to be characterized, and said at least one characteristic extracted by the neural network ( RNcnn) is used to classify the input data according to a plurality of classes, each class being representative of a characteristic of the sample (E) to be characterized.
    类似技术:
    公开号 | 公开日 | 专利标题
    FR3074596A1|2019-06-07|SAMPLE CHARACTERIZATION METHOD USING NEURON NETWORKS
    EP2427752B1|2013-04-17|Method for identifying a scene from multiple wavelength polarised images
    CA2778682A1|2011-05-05|Method and device for analysing hyper-spectral images
    Yao et al.2013|Hyperspectral image classification and development of fluorescence index for single corn kernels infected with Aspergillus flavus
    Barman et al.2020|Smartphone image based digital chlorophyll meter to estimate the value of citrus leaves chlorophyll using Linear Regression, LMBP-ANN and SCGBP-ANN
    Huang et al.2020|Phenotypic techniques and applications in fruit trees: a review
    Hongyu et al.2015|Rapid diagnosis of tomato NPK nutrition level based on hyperspectral technology.
    Maier et al.2019|Estimating chlorophyll a concentrations of several inland waters with hyperspectral data and machine learning models
    FR3069940B1|2019-09-06|METHOD AND SYSTEM FOR CARTOGRAPHY OF THE HEALTH CONDITION OF CULTURES
    WO2019053364A1|2019-03-21|Device for capturing a hyperspectral image
    Ahmad et al.2018|Monitoring insect pest infestation via different spectroscopic techniques
    Zhang et al.2020|Classification method of CO2 hyperspectral remote sensing data based on neural network
    Okamoto et al.2008|Unified hyperspectral imaging methodology for agricultural sensing using software framework
    Barrows et al.2017|Development of a low-cost multispectral camera for aerial crop monitoring
    Zhao et al.2016|Application of hyperspectral imaging in measurement real-time of seeds
    Pedraza et al.2019|Sweet citrus fruit detection in thermal images using fuzzy image processing
    Li et al.2019|Rapid Detection and Visualization of Mechanical Bruises on “Nanfeng” Mandarin Using the Hyperspectral Imaging Combined with ICA_LSQ Method
    Men et al.2021|A residual network with attention module for hyperspectral information of recognition to trace the origin of rice
    Singh et al.2010|Near-infrared hyperspectral imaging for quality analysis of agricultural and food products
    FR3081590A1|2019-11-29|METHOD FOR INCREASING THE SPATIAL RESOLUTION OF A MULTI-SPECTRAL IMAGE FROM A PANCHROMATIC IMAGE
    Lang et al.2021|Classification of materials using a pulsed time-of-flight camera
    Seo et al.2021|Soil Moisture Retrieval Model Design with Multispectral and Infrared Images from Unmanned Aerial Vehicles Using Convolutional Neural Network. Agronomy 2021, 11, 398
    FR2733596A1|1996-10-31|Method for blood sample analysis using a neural network.
    EP3937628A1|2022-01-19|Agricultural treatment control device
    Puumalainen2019|Computational spectral imaging and band-pass filters for apple bruising degree assessment
    同族专利:
    公开号 | 公开日
    EP3717877A2|2020-10-07|
    WO2019106179A3|2019-07-25|
    CA3083079A1|2019-06-06|
    JP2021504864A|2021-02-15|
    FR3074596B1|2019-12-06|
    WO2019106179A2|2019-06-06|
    CN111630356A|2020-09-04|
    US20200333185A1|2020-10-22|
    引用文献:
    公开号 | 申请日 | 公开日 | 申请人 | 专利标题
    WO1998043042A1|1997-03-25|1998-10-01|Applied Spectral Imaging Ltd.|Spectral bio-imaging methods for cell classification|
    WO1999005487A1|1997-07-25|1999-02-04|Physical Optics Corporation|Accurate tissue injury assessment using hybrid neural network analysis|
    EP2696191A1|2011-04-06|2014-02-12|Universitat Autònoma De Barcelona|Method for the characterisation and classification of kidney stones|
    CN102621150B|2012-03-23|2014-02-05|南京航空航天大学|Airplane skin damage identification method based on gray level co-occurrence matrix and support vector machine|
    CN105760883A|2016-02-15|2016-07-13|西安科技大学|Belt conveyer key component automatic identification method based on infrared thermography|
    CN106022365B|2016-05-16|2019-04-02|电子科技大学|Surface defect depth estimation method based on data fusion and RBF neural|EP3608701A1|2018-08-09|2020-02-12|Olympus Soft Imaging Solutions GmbH|Method for providing at least one evaluation method for samples|
    CN110333260A|2019-07-19|2019-10-15|北京锐百凌科技有限公司|Explosion-proof EO-1 hyperion far infrared gas cloud imaging control transmission and information processing, orientation delivery system|
    DE102020205456A1|2020-04-29|2021-11-04|Volkswagen Aktiengesellschaft|Method, device and computer program for generating quality information about a coating profile, method, device and computer program for generating a database, monitoring device|
    CN113063751A|2021-03-25|2021-07-02|司法鉴定科学研究院|Forensic medicine pulmonary fat embolism analysis method based on infrared spectrum imaging technology|
    法律状态:
    2018-12-31| PLFP| Fee payment|Year of fee payment: 2 |
    2019-06-07| PLSC| Publication of the preliminary search report|Effective date: 20190607 |
    2019-12-30| PLFP| Fee payment|Year of fee payment: 3 |
    2020-12-28| PLFP| Fee payment|Year of fee payment: 4 |
    2021-12-30| PLFP| Fee payment|Year of fee payment: 5 |
    优先权:
    申请号 | 申请日 | 专利标题
    FR1761522|2017-12-01|
    FR1761522A|FR3074596B1|2017-12-01|2017-12-01|METHOD FOR CHARACTERIZING SAMPLES USING NEURON NETWORKS|FR1761522A| FR3074596B1|2017-12-01|2017-12-01|METHOD FOR CHARACTERIZING SAMPLES USING NEURON NETWORKS|
    US16/768,573| US20200333185A1|2017-12-01|2018-11-30|Method for characterising samples using neural networks|
    CA3083079A| CA3083079A1|2017-12-01|2018-11-30|Method for characterizing samples using neural networks|
    CN201880077916.3A| CN111630356A|2017-12-01|2018-11-30|Method for characterizing a sample using a neural network|
    PCT/EP2018/083210| WO2019106179A2|2017-12-01|2018-11-30|Method for characterizing samples using neural networks|
    JP2020547304A| JP2021504864A|2017-12-01|2018-11-30|How to characterize a sample using a neural network|
    EP18814839.9A| EP3717877A2|2017-12-01|2018-11-30|Method for characterizing samples using neural networks|
    [返回顶部]