HARDWARE AND SOFTWARE VIDEO PROCESSING COMPLEX

Abstract

The article is devoted to solving the current scientific and applied problem of creating a hardware-software complex for processing video streams in autonomous observation systems, focused on real-time operation in conditions of limited computing and energy resources. It is shown that traditional approaches to transmitting a "raw" video stream to control stations do not provide the necessary delay and resistance to radio-electronic interference. The feasibility of transferring calculations directly to the autonomous observation system using the Edge AI concept is substantiated. An analysis of modern hardware architectures for building machine vision systems is carried out, in particular microcontrollers, FPGAs and heterogeneous systems on a crystal with AI accelerators. A device heterogeneous architecture based on the NVIDIA Jetson Orin NX module and a video sensor with a MIPI CSI 2 interface is proposed, which provides minimal image capture delay. The algorithmic support for object detection based on the YOLOv8 neural network and methods for its optimization using the TensorRT library with INT8 quantization are described. Using the YOLOv8 model ensured high detection accuracy. The integration of optimization algorithms allowed us to significantly improve system performance compared to standard use of PyTorch. The results of experimental studies are presented, confirming the achievement of performance of more than 50 frames per second with a total processing delay (latency) of less than 40 ms and power consumption of up to 12 W. Low latency is a critical factor for using the complex in the control loop of high-speed observation systems. The results obtained indicate the possibility of practical application of the hardware and software complex as part of modern autonomous observation systems.