Image processing

Neural networks are especially suited for image processing tasks, whether it is for feature extraction, multi-scale signal analysis or pattern recognition.

Using Saimple for an image processing use-case

Neural networks are a powerful tool that can exceed other methods' performances. But they might be a challenge to train and to validate.

All along your production cycle Saimple can help your teams by speeding-up development and improving the quality of the AI.

Early on during the specification phase it helps:

Through the design phase by:

Finally during the integration phase by:

For example let us consider the following workflow that could be implemented for one of your AI projects. The project aims at integrating an automated threat recognition around a secure building. For example for every moving object it has to classify its nature (drone, car, person, animal, etc). We assume that a preliminary on-the-shelf architecture has been selected for the task, and a first dataset is available.

1- Assemble your training set, train on it, see what the network is learning

Starting from an on-the-shelf and a raw training set is every AI engineer's first step. Before customization the engineer needs to select a promising architecture and see how it performs on the dataset. However training sets can be incomplete and biased which can alter the performance assessment done. Usually the engineer only sees the performance score of the network on the dataset, for example using a confusion matrix (see Fig. 1). At first an engineer is blind to any performance cause, as the neural network does not explain much what is impacting its performance. Each guess the engineer is doing needs to be tested and the network trained again, this iterative process can take a very long time in some cases.

(Fig. 1 - A confusion matrix, which is good to measure performance but not very helpfull to interpret the results)
(Fig. 1 - A confusion matrix, which is good to measure performance but not very helpfull to interpret the results)

Here is how Saimple can help your engineer adapt the training dataset.

Using its unique explainability process for neural network Saimple can extract for every tested image why it is correctly or wrongly classified. For example in Fig. 2 the engineer knows that this image is wrongly classified as Truck despite being annotated as Street. With Saimple he can check and see that some features of Truck images are recognized on this image. Knowing that he checks correctly classified images of Trucks (Fig. 3) to discover that the main feature that is recognized of Trucks seems to be the brown boxes that are loaded in, which is not what he intended. Therefore he updates the database by forcing the boxes to appear less on the images of Trucks. After retraining, the image is correctly classified and the features motivating this classification are the correct ones.

By doing so your engineer discovers which class is more likely to bring confusion in the classification process. He also learns why some classes are stepping onto others. He can therefore take appropriate action to correct the dataset to quickly improve the performance of your network.

 

 

(Fig. 2 - A street image wrongly classified as


(Fig. 2 - A street image wrongly classified as "Truck")

 

(Fig. 3 - Features of Truck mostly recognized by the AI system on the original image)
(Fig. 3 - Features of Truck mostly recognized by the AI system on the original image)

 

(Fig. 4 - Example of image of Truck in training set with the features learned by the AI system.)
(Fig. 4 - Example of image of Truck in training set with the features learned by the AI system.)

 

2- Improve the architecture to ensure better performance

Once the training set is stable enough you can now try to optimize the neural network architecture to improve the overall performance over the test dataset. Normally the optimization is done manually using intuition and experience. Each optimization is a trial and error approach and much time can be invested without any certainty of any significant improvement.

Here is how Saimple can help your engineer find a more performant architecture.

After training the neural network it appears that the performances are not always good on certain images of the class Helicopter. The engineers suspect that in the internal process of the network some layers fail to intercept the feature of the landing gears. Usually it has to change iteratively the layer, modifying their size, their number until finding an adequate solution. Some use automated processes to find this solution, but each time the training and the evaluation are necessary, which can still be long and tedious. With Saimple your engineer can visualize at each layer how the process is handling the input information (Fig. 1). By doing this the engineer can discover that the features of the landing gears he is interested in are lost at the second hidden layer. Therefore any attempt to preserve these features should be done either on this layer or at least near this one.


 

By doing this the engineer can discover that the features of the landing gears he is interested in are lost at the second hidden layer. Therefore any attempt to preserve these features should be done either on this layer or at least near this one.

Using this process your engineer discovers quickly where to change its network to improve its performance.

 

(Fig. 1 - A view of what is discovered at each layer by the neural network)
(Fig. 1 - A view of what is discovered at each layer by the neural network)

3- Perform an optimized data augmentation

In order to improve a dataset engineers can use data augmentation techniques. While being well understood they are sometimes used despite having the need. They can largely improve the robustness of a network however they will increase rapidly the cost of training.

 

Applying whenever it is necessary and only to the extent needed are essential to avoid costly delay. Saimple can help control the need for data augmentation techniques. For further information please refer to the page on Data Augmentation.

4- Test your system against the future conditions of use

The intended surveillance system is designed to operate outdoors in any condition of brightness and weather. However, testing a dataset is usually not enough to validate the performance on every condition and even less to validate it on every combination of conditions. The real life scenario might not be covered by the testing dataset. As a result the performance of the system may vary greatly with no explanation once in production.

To avoid such situations any engineer should increase its database coverage, either manually or through data augmentation [lien vers page Data Augmentation]. But this is very time consuming. With Saimple it is possible to test the robustness of your neural network against a variety of perturbation automatically. In this example let’s assume that the camera can suffer from blurry inputs (Fig 1.) or the presence droplets of water on the protective glass (Fig 2.). Normally an engineer would have to perturb all images in the dataset using multiple settings of the noise (different settings of the blurring kernel, or different numbers of droplets). This would require an exponential number of tests, but it won’t be able to prove whether or not the system is performant beyond this test. For that you engineer need Saimple to cover all the space around the images he has with all the possible noise he can apply (Fig 3.).

Let us assume that the camera use can add up to a certain maximum intensity of blurring noise. With Saimple, the engineer can model this blurring noise using standard libraries. Then he uses Saimple to check the robustness of its networks on each image he has. Saimple will automatically prove whether or not the classifier will be correct for any noise up to a certain maximum value of intensity. He has no need to build an exponential testing process for each image, he just has to prove once the robustness of each image.

(Fig. 1 - CCTV image with gaussian blur)


 

(Fig. 2 - CCTV image with droplet mask)

(Fig. 2 - CCTV image with droplet mask)

 

(Fig.3 - Multiple noises in on one image contained in our Abstract Domain)

(Fig.3 - Multiple noises in on one image contained in our Abstract Domain)

But not all noises are mathematical, for example droplets on images can be challenging to model. For those Saimple allow the use of masks to model specific noises applied onto an image, for example droplets (Fig 4). The tool will help prove that the classifier can handle the application of a mask. After setting up a mask the engineer can see how much the classifier resists the perturbation model by the mask (Fig 5).

 

(Fig. 4 - Droplet mask)

(Fig. 4 - Droplet mask)

 

Knowing when the network classification can fail gives the engineer the knowledge how much noise can the system handle (Fig 5). It is critical to know as soon as possible if a system will resist to avoid costly deployment phases that could fail.

Using Saimple the engineer was able to check during the design phase if its system will handle the volume of noise the camera can add. He was also able to test before putting in production the limit of its system to environmental perturbations.

5- Check the impact of the hardware you plan your integration on

Changing hardware between the training phase and the production phase can imply severe cutbacks to the performance of neural networks. As Neural networks adapt their training to the precision they have, modifying it is bound to have an impact on their performance. Modern frameworks for training neural networks do not consider this impact since for them production and design environnement are on one and the same.

To better manage your operational risks, Saimple helps your engineer measure the impact of the hardware on performance. Specifically it can check the robustness even when changing the underlying numerical precision. It allows engineers to anticipate and adapt both training and architecture to better match the hardware specifications down the road. Thus avoid costly and time-consuming retrofit actions which are needed otherwise.

Assuming that the classification system was trained on 64-bits architecture it takes a lot of memory space. To allow integration into the camera the system has to use only 16-bits operations. Changing just before going into production the precision causes a drop in the performance of the classifier of 3.9% (Fig. 1). Using Saimple earlier on this situation would have been avoided since the robustness of this neural network was not guaranteed upon a change that goes as down as to 16-bits operations. However using Saimple the engineer can find how mixed precision can be used to have both a reduction of the size of the neural network and the conservation of performance (Fig. 2).

Evolution of the classification performance vs the precision used and memory usage
(Fig. 1 - Evolution of the classification performance vs the precision used and memory usage)

 

 


Using Saimple your engineer can test ahead the integration in your system and avoid a retrofit. When the hardware constraint degrades too much the performance, Saimple helps by automatically finding a tradeoff between performance and memory size of the neural network.

Assuming that the classification system was trained on 64-bits architecture it takes a lot a memory space. To allow integration into the camera the system have to use only 16-bits operations. Changing just before going into production the precision causes a drop in the performance of the classifier of 3.9% (Fig. 1). By using Saimple earlier on this situation would have been avoided since the robustness of this neural network was not guarantee upon a change that goes as down as to 16-bits operations. However but using Saimple the engineer can found how mixed precision can be used to have both a reduction of the size of the neural network and the conservation of performance (Fig. 2).

Using Saimple your engineer can test ahead the integration in your system and avoid a retrofit. When the hardware constraint degrade to much the performance Saimple helps by finding automatically a tradeoff between performance and memory size of the neural network.

 

(Fig. 2 - Evolution of the classification performance vs. also with a mixed precision)(Fig. 2 - Evolution of the classification performance vs. also with a mixed precision)

6- Document your neural network validation

Once your product is moving into production the performance and reliability of your AI have to be documented. Using only classical performance scores might not be enough to inspire trust in your product or discuss with regulatory or certification bodies.

Numalis is at the forefront of the standardization of AI. Working within the ISO group on AI to define what the robustness of neural networks is, see more about our work here. (in process)

Saimple allows your engineers to build quickly documentation and profit reproducible benchmarks of robustness and explainability. With the built-in features to export any result of Saimple you can assemble quickly any chart or graphics you need for your commercial or regulatory documentation.

7- Validate continuously

Continuous integration is an important practice to ensure the quality of your future product all along its development cycle. Testing early on and at every step will help you avoid any regression in your training and integration. The ability to construct tests that are both relevant and sufficient to ensure the quality of your product is not easy. Each time you use Saimple to test something you can document and archive this test. From there you can build your own robustness and explainability test bench using Saimple that you can launch on any test server. Using a simple batch run methodology you can launch at any time every test you have and look for regression automatically. Batch of tests made using Saimple can be integrated into your continuous integration framework in order to add the robustness and explainability functional tests on top of others.

Testing every aspect of AI is crucial, with Saimple you can automate your testing of the robustness and the explainability within your current continuous integration framework.

Numalis

We are an innovative software editor company providing tools and services to make your neural networks reliable and explainable

Contact us

Follow us