Machine learning and embedded systems. Demonstration of possibilities

Machine learning and embedded systems. Demonstration of possibilities

One of the advantages of machine learning technologies is their ability to learn and adapt based on new data, which allows solving the tasks of pattern recognition, natural language processing, and finding patterns and anomalies. Traditionally, collected data is processed on servers, but for some tasks it is more appropriate (and sometimes necessary) to use another approach, called edge machine learning.

In this article, we will talk about the practical application of our machine learning framework for embedded systems, using the example of a demonstration stand that we showed at the Army-2023 forum.

One of the advantages of machine learning technologies is their ability to learn and adapt based on new data, which allows solving the tasks of pattern recognition, natural language processing, and finding patterns and anomalies. This set of advantages has made it possible to use neural networks together with smart devices. Traditionally, collected data is processed by machine learning algorithms on servers. With this approach, you have to put up with the waiting time for data transfer and processing. The explosive growth of the number of devices leads to an increase in the amount of generated data, and, as a result, the load on the server infrastructure.

However, for a number of tasks, it is more appropriate to use another approach, which is called edge machine learningwhich combines such concepts as:

  • edge computing — the paradigm of distributed computing, carried out by the devices themselves, or their networks;
  • machine learning – A class of adaptive algorithms using learning methods based on the received data.

Thus, the advantages of this approach give us:

  • security — because there is no need to transfer data outside the device or its network;
  • autonomy – The absence of a delay in sending data to the server reduces the execution time of the logical conclusion, reduces bandwidth requirements, and also allows for prompt decision-making with or without human intervention;
  • possibility of adjustment behavior of the neural network during operation.

The capabilities of built-in devices are constantly growing, which allows you to use them to solve new tasks. In our company, there was a need to create tools for extreme machine learning.

To solve these problems, we have developed the intelligent platform “Synaptika” for the Neutrino real-time operating system. It is a framework for building data collection, processing and analysis systems using machine learning technologies. One of the examples of the practical application of “Synaptics” is the program complex for monitoring abnormal processes (PC MAP) (more detailed information can be obtained from another article of ours).

This year, for the “Army-2023” exhibition, we prepared a special stand with the “Synaptika” platform.

In order to demonstrate the capabilities of the “Synaptika” platform, we developed a hardware and software complex for recognizing persons in order to control access to objects of critical information infrastructure (CII). The following components were used in the demonstration stand: an IR camera, a computing module with the “Neutrino” ZOSRV, a router.

Photo of the stand

The camera receives the stream from the camera in the IR spectrum and transmits it according to the protocol RTSP (real time streaming protocol). The developed application receives a stream, preprocesses the received frame and, if there is a person in the frame, analyzes it. The analysis of the image received from the IR camera allows you to extract additional details from the shades of the two-dimensional gray image. After identifying a person, information is displayed on whether or not he has access.

Schematically, the work algorithm is shown in the image:

An example of identification of the person of one of our company’s employees (Colored areas are the extracted features for the recognized person)

The application consists of the following structural components:

  • Control module — performs interaction between lower-level modules;
  • Image capture module – Reads the rtsp stream and receives a frame at the request of the control module, controls the conveyor;
  • Data supply module – Continuous flow of data received from the camera;
  • Preprocessing module – Performs image preprocessing using OpenCV;
  • Analysis module – Processes the frame preprocessed by the previous module;
  • Graphical interface — program interface, displays images from the camera and information about the recognition process and results;
  • Database – A repository of biometric data about persons who have access to a critical infrastructure object.

Structural components of the program

In general, the recognition of a person’s identity is:

  • processing and analysis of the image received from the camera,
  • comparison with the existing database of biometric data.

The first stage of processing the received image includes searching for and highlighting a person’s face on it. For this, the method is used detectMultiScale() from the OpenCV library. It allows you to search for the location of a person’s face and eyes in the frame. If a person is found, the resulting area containing the face image is highlighted from the entire frame. Further transformations are carried out only with this area.

By analyzing the gray gradient, you can get additional information about the three-dimensional contour of the face, since the brightness of a particular pixel correlates with the distance from the camera to the point — the smaller the distance, the darker the pixel. Based on the data on the brightness of pixels, special areas of the face are “highlighted” (pixels that fall into certain brightness ranges) that most characterize its contour are highlighted.

Thus, the ratio of the sides of the selected area of ​​the frame containing the image of the person, information about the “illuminated” areas of the person, information about the location of the eyes – all these are excellent unique features that allow identifying the identity of a specific person.

“Synaptika” does not limit us in choosing a machine learning framework – you can choose TensorFlow Lite, ONNX Runtime, mlpack and other frameworks. However, for the image analysis in the demonstration project, we chose a non-standard neural network model — an interference neural network. One of its features is the use of a mathematical model of a biological neuron. The use of the neural network interference model (and the corresponding Interference library) in our project provides the following advantages:

  • the possibility of retraining and changing the architecture of the neural network “on the fly”, which will allow updating data on employee access if necessary;
  • low resource intensity of computational learning and recognition processes;
  • the possibility of accelerating calculations by transferring calculations to GPUs or neuroaccelerators.

You can read more about the application of an interference neural network for dynamic real-time data analysis at the link.

Based on the results of the interference model, a network architecture (in essence a classifier) ​​was developed directly for this task. The architecture assumes that before analysis, the image is split into four equal parts (research has shown that this is the optimal number), as shown in the figure.

Division of the image into four parts: forehead, bridge of the nose, nose and mouth, chin.

Schematically, the architecture of the network is presented in the figure. Four neurons (N1-N4) constitute one ensemble. One ensemble is responsible for recognizing the identity of one person, assigning his biometric data to a certain class. Each class can store a set of multiple images for a single person.

Architecture of one ensemble in the network

During the learning process of each person, a new ensemble of neurons is created. Then, in order to remember his biometric data, each of the listed parts of the image is fed two pixels (R, G, B channels) to the input of the corresponding neuron (depending on the area of ​​the face). As already mentioned above, each person has four such neurons. After that, the state of the neurons is preserved.

In the process of recognition, the algorithm for submitting the image to the network input is similar. The state of the neurons in the ensemble is calculated for the recognized image and then compared to all the stored ensembles in the network. According to the level of correspondence, it is possible to come to a disappointing conclusion as to how much the recognized data matches the stored data.

We are constantly developing the architecture of the intelligent platform “Synaptika”. It is planned to improve the API of the framework to make it more convenient for developers, to add more machine learning frameworks, to expand the list of supported hardware accelerators for neural networks, which will allow us and our customers to solve a wider range of tasks. We are also continuing the development of our own synaptic-based products, planning to improve the facial recognition booth for further demonstrations.

In the near future, a cycle of articles on the interference model of neural networks, its capabilities, architecture, mathematical basis and practical application is planned, and for now you can familiarize yourself with the scientific work “Parametric synthesis of the interference model of a neural network”.

Subscribe to our channel to know the latest news

Related posts