FPGA simulated MIPI camera, successfully connectedNVIDIAAGX Developer

1. preface

Visual cameras are key to machine intelligence/automation, and NVIDIA Jetson supports cameras from multiple partners, including cameras with USB, Ethernet, MIPI, and other interfaces. Jetson provides a complete Jetpack SDK for these cameras, enabling users to quickly connect to these cameras and implement corresponding designs.

The list of cameras supported by Jetson can be found on the official website as follows:

https://developer.nvidia.com/embedded/jetson-partner-supported-cameras?t -1 supported jet son products=Origin&t1 supported jet son products=Xavier

Select the CSI/MIPI interface and Orin suite, and the webpage will display a list of Jetson Partner supported cameras, mainly Sony's IMX290 IMX334、IMX179、IMX335、IMX415 ， And Aptina's AR0234, AR0144, AR0822, etc. Therefore, for example, if users use NVIDIA's Jetson AGX Orin developer suite, the SDK of the suite natively supports CSI access to the sensors listed above, allowing users to quickly develop their own solutions.

If you choose the camera from the above list and use it with the accompanying CSI adapter board, as the Camera Partner has already pre docked with various models of NVIDIA Jetson, the driver in the original factory image is already supported by default, and can be plug and play.

howeverIf we choose a sensor model outside the list, which is not a Camera Partner product, we cannot directly use it,For example, if I want to use Sony's IMX472, IMX252, Sitewe's SC2210, SC235HGS, etc., thenIt is necessary to upgrade the supporting hardware and driver software.For this, Aoweisi provides you with a complete solution, which can be equipped with any camera or choose FPGA integrated ISP.

2. FPGA analog MIPI camera accessJetsonprogramme

As shown in the following figure: The scheme adopts Yilingsi Titanium 16nm FPGA Ti60F225, collects image sensor data through MIPI CSI RX, undergoes a series of complex ISP operations inside the FPGA, and then sends the image data to the backend through MIPI CSI TX.

Among them, VC-MIPI-AGX is the MIPI access sub card of the NVIDIA Jetson AGX Orin developer kit developed by Aoweisi. As shown in the figure below, it is the MIPI card slot of AGX and the access sub card we designed. We use a high-speed and stable TypeC 3.0 data cable to transmit MIPI signals. The current solution can support 2 MIPI CSI channels, with a maximum bandwidth of 6Gbps per channel. At present, the solution has been successfully integrated into AGX, stable and reliable, and the baking machine has passed the test.

2.1Introduction to Yilingsi MIPI

The low power consumption, high bandwidth, and small size characteristics of the Yilingsi 16nm titanium series FPGA fully empower terminal products to make them innovative and competitive. The titanium series Ti60 can support multi camera input, multi type screen driver, high-performance RISC-V soft core and other functions with MIPI/LVDS interfaces, suitable for various camera and sensor systems and other scene applications.

Therefore, we chose Yilingsi's FPGA as the main controller, fully utilizing its high-speed and low-power characteristics to complete the collection of MIPI CSI, ISP image processing, and transmission of MIPI CSI. The MIPI PHY of Yilingsi Ti60 FPGA is 1.5Gbps single channel, and 4 lanes support 6Gbps. At 85% effective bandwidth, the effective transmission resolution/frame rate is evaluated as follows:

Adopting RGB888 or Bayer 8-bit transmission has its advantages and disadvantages:

1) RGB888 8-bit transmission, AGX can directly obtain image data, with the advantage of no additional source data processing overhead; The downside is that due to Bayer's 3x bandwidth, the current FPGA scheme can only support up to 1080P60 scheme.

2) Bayer 8-bit transmission, with a bandwidth of 1/3 of RGB888, can support higher resolutions (4K60), and at the same resolution, can support longer transmission distances than RGB888; The shortcoming is that AGX's CPU (Neon) is needed to process the Bayer2RGB algorithm. Currently, testing 1080P120 consumes 43% of the resources of one CPU (a total of 12 CPUs, with single core resource consumption as shown in the following figure).

Based on comprehensive evaluation, due to the current use of only one AGX CPU and the user's algorithm mainly running CUDA on the GPU, the advantage of using Bayer transmission is more obvious.

2.2Introduction to Aoweisi ISP

NVIDIA can also perform complete ISP image processing, and even the ISP function and performance implemented by ASIC are more powerful than FPGA. However, in this solution, we have to use FPGA for complete ISP processing, mainly for the following reasons:

1) NVIDIA still requires a lot of tuning work for the selected sensors;

2) NVIDIA's ISP does not support 8-bit RAW input mode;

3) Some application engineers are not familiar with the ISP field;

4) Some ISP algorithms are not included in NVIDIA;

5) The demand for low latency requires a pipeline to complete ISP.

In view of this, we have completed a complete ISP processing process using FPGA. MIPI CSI directly outputs the data after FPGA image processing. After NVIDIA AGX receives the RGB data, users can directly carry out subsequent application algorithms. The ISP image processing pipeline based on Yilingsi FPGA is as follows:

The self-developed assembly line ISP by Aoweisi and its related features are shown in the table below:

2.3AGX Orin Access Driver

In terms of hardware, the VC-MIPI-AGX sub card designed by Ovis is used to achieve user MIPICSI to AGX Samtec input. In terms of software, it is necessary to develop specific Sensor driver adaptations based on hardware board and module design. If there are special customized control requirements, such as ISP parameter control, it is necessary to modify the Camera framework for deep customization development.

For the protection of system security, Jetson's kernel compilation was turned on when it left the factoryDriver Signature OptionsIf the newly developed driver signature certificate does not match, the driver ko file cannot be loaded. However, most users are unable to obtain the signature certificate from NVIDIA when compiling factory images, which cannot solve the ko signature problem. If you need to use a new sensor module, you will need toRecompile the kernel, generate a full system image, and burnsystemThe workload is quite enormous.

Aoweisi solves the above-mentioned software and hardware problems for customers and creatively uses typeC 3.0 cables as data transmission media, with reliable connections and stable communication. Based on the NVIDIA Jetson AGX Orin developer platform, the self-developed FPGA MIPI ISP camera connection environment is connected as follows:

3. Effect display