Deploy Special NVIDIA Jetson Model

Last Updated：2021-05-18

This article introduces how to obtain the adaptive end model file for the NVIDIA Jetson nano device and deploy the model to BIE by combining the NVIDIA L4T Base image to complete the AI image recognition.

Setp1 Get SDK for Jetson nano environment using EasyEdge

Upload original model

After the model is submitted, it generates the end model on my model interface. Select the corresponding chip and operating system, and click "Release" to get the SDK file.

Setp2 SDK compilation and verification

Compile and authenticate the SDK file, and generate the model that can run on the Jetson device.

Note: To use the Jetson series SDK of EasyDL, you need to install the specified version of JetPack and related components. At present, the supported JetPack versions include JetPack4.4 and JetPack4.2.2.

The model resource file has been packaged in the downloaded SDK package by default. You can compile and run the Demo project directly.

cd demo
mkdir build && cd build
cmake .. && make
sudo ./easyedge_demo {Model RES folder}  {Test image path} {Serial number}

Compilation process reference:

`nano@jetson-nano:~/Downloads/easydl-sdk/cpp/baidu_easyedge_linux_cpp_aarch64_JetPack4.4_gcc7.4_v0.5.5_20200811/demo/build$ ls
CMakeCache.txt  CMakeFiles  cmake_install.cmake  easyedge_batch_inference  easyedge.log  easyedge_multi_thread  easyedge_serving  install_manifest.txt  Makefile
nano@jetson-nano:~/Downloads/easydl-sdk/cpp/baidu_easyedge_linux_cpp_aarch64_JetPack4.4_gcc7.4_v0.5.5_20200811/demo/build$ ./easyedge_batch_inference /home/nano/Downloads/easydl-sdk/RES /home/nano/Downloads/easydl-sdk/imgs/mix008.jpeg
2020-08-24 03:51:09,915 WARNING [EasyEdge] 548102246416 Only compile_level 2 is supported for this model on JetPack 4.4 DP version. Please read documentation for the details
2020-08-24 03:51:09,915 INFO [EasyEdge] 548102246416 Compiling model for fast inference, this may take a while (Acceleration)
2020-08-24 03:53:08,488 INFO [EasyEdge] 548102246416 Optimized model saved to: /home/nano/.baidu/easyedge/jetson/mcache/26119049355/m_cache, Don't remove it
Results of image /home/nano/Downloads/easydl-sdk/imgs/mix008.jpeg:
1, tomato, p:0.999717 loc: 0.672033, 0.405379, 0.80056, 0.569352
2, kiwi, p:0.999273 loc: 0.154121, 0.0717932, 0.393312, 0.399547
2, kiwi, p:0.999206 loc: 0.45734, 0.0788124, 0.735513, 0.390724
1, tomato, p:0.998933 loc: 0.385497, 0.0432633, 0.51477, 0.22167
1, tomato, p:0.998886 loc: 0.520549, 0.384235, 0.654002, 0.552556
1, tomato, p:0.998507 loc: 0.295452, 0.452287, 0.422471, 0.588263
1, tomato, p:0.998191 loc: 0.181686, 0.568888, 0.295577, 0.712147
1, tomato, p:0.996489 loc: 0.386017, 0.250961, 0.504955, 0.408554
2, kiwi, p:0.991238 loc: 0.350772, 0.568207, 0.61708, 0.87927
Done`

Authentication process reference:

`nano@jetson-nano:~/Downloads/easydl-sdk/cpp/baidu_easyedge_linux_cpp_aarch64_JetPack4.4_gcc7.4_v0.5.5_20200811/demo/build$ ls
CMakeCache.txt  CMakeFiles  cmake_install.cmake  easyedge_batch_inference  easyedge.log  easyedge_multi_thread  easyedge_serving  install_manifest.txt  Makefile
nano@jetson-nano:~/Downloads/easydl-sdk/cpp/baidu_easyedge_linux_cpp_aarch64_JetPack4.4_gcc7.4_v0.5.5_20200811/demo/build$ ./easyedge_serving /home/nano/Downloads/easydl-sdk/RES "57F7-0892-3057-6D58" 0.0.0.0 24401
2020-08-24 03:56:17,802 WARNING [EasyEdge] 548017537040 Only compile_level 2 is supported for this model on JetPack 4.4 DP version. Please read documentation for the details
2020-08-24 03:56:17,802 INFO [EasyEdge] 548017537040 Compiling model for fast inference, this may take a while (Acceleration)
2020-08-24 03:57:34,965 INFO [EasyEdge] 548017537040 Optimized model saved to: /home/nano/.baidu/easyedge/jetson/mcache/26119049355/m_cache, Don't remove it
2020-08-24 03:57:34,968 INFO [EasyEdge] 548017537040 HTTP is now serving at 0.0.0.0:24401, holding 1 instances
2020-08-24 03:57:55,502 INFO [EasyEdge] 548000241392 [access_log] "GET /" 200 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
2020-08-24 03:57:55,582 INFO [EasyEdge] 548000241392 [access_log] "GET /api/info" 200 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"
2020-08-24 03:58:31,103 INFO [EasyEdge] 546816874224 [access_log] "POST /" 200 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36"

Compilation reference: https: //ai.baidu.com/ai-doc/EASYDL/Dka3ky0an
Model SDK download address: links: https: //pan.baidu.com/s/1k7IOxsBAyIZXXPXpmle91Q Password: suy6 `

Step 3 Generate model package and upload it to object storage

1. Create run.sh. This file is the startup file of the container.

#Model serial number, need to apply online, here is the test serial number
license_key=E60A-5124-5ACD-3C9B
#curr_dir/sdk，Later used in the BIE inference service configuration, as the working directory in the container
curr_dir=/sdk
demo_dir=${curr_dir}/cpp/sdk/demo/build
lib_dir=${curr_dir}/cpp/sdk/lib
res_dir=${curr_dir}/RES
export LD_LIBRARY_PATH=${lib_dir}:${LD_LIBRARY_PATH}
#run
${demo_dir}/easyedge_serving ${res_dir} ${license_key}

2. Pack the CPP directory, RES directory, and run.sh file as an easyedge-fruit-model.zip file, and upload it to BOS.

Note: Here, multiple directories and files are compressed into a zip file, but the outermost directory is not compressed directly. If the compression level is wrong, it will cause that the decompressed directory does not match after the model is downloaded to the edge device.
Compiled model package reference: links: https: //pan.baidu.com/s/1IDpHehBzUC58-xv2N4KByQ Extraction code: 26uf, file name: fruit-ai-model.zip (you need to change the absolute path $ (pwd) pointed by curr_dir in run.sh to the specific container directory/SDK.)

Step4 Distribute model to the edge

1. Create edge node

Create the edge node jetson-nano as shown in the figure below.

2. Create AI model configuration item

Create the configuration item fruit-demo as shown in the figure below. Import the model file to upload to BOS.

3. Create AI inference service

Create the application fruit-detection as shown in the figure below.

Add a container service in the application as shown in the figure below:

Image address: nvidia provides a container image NVIDIA L4T Base for NVIDIA Jetson series. nvidia official website: https: //ngc.nvidia.com/catalog/containers/nvidia: l4t-base This container image provides the running environment of nvidia jetson, and can run the AI model service that supports the jetson device in the container. After the containerization is supported, we can manage the edge node device and improve the efficiency of model service deployment through BIE.

Official website mirroring address:nvcr.io/nvidia/l4t-base:r32.4.3
If the download speed in China is very slow, you can download the mirror image from Baidu cce:hub.baidubce.com/nvidia/l4t-base:r32.4.3-arm64

Volume configuration: The type is a configuration item. The parameter selects the file configuration item created previously. Container directory: /sdk (correspond to the run.sh file). Bind the AI inference service with the model file through the volume configuration. When the inference service is distributed to the edge node, download the AI model file automatically.
Port mapping of container service: port 24401, protocol TCP
Startup parameter of container service: bash, /sdk/run.sh

4. Deploy an application to the node

Add the target node tag in the edge application.

5. Install node on jetson nano device

5.1 Execute the docker info to view the edge node container environment and ensure the Default Runtime: nvidia

nano@jetson-nano:~/Downloads/fruit-ai-model$ docker info
Client:
 Debug Mode: false
 
Server:
 Containers: 38
  Running: 21
  Paused: 0
  Stopped: 17
 Images: 12
 Server Version: 19.03.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: nvidia runc
** Default Runtime: nvidia**
 Init Binary: docker-init
 containerd version:
 runc version:
 init version:
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.9.140-tegra
 Operating System: Ubuntu 18.04.5 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 3.871GiB
 Name: jetson-nano
 ID: O7GP:DDD5:5CIR:LEWJ:2BQ3:4WIW:VA4H:JDCP:5VGL:L2K3:PLZ7:KBHO
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

If the Default Runtime is nvidia, change the /etc/docker/daemon.json file, and add "default-runtime": "nvidia", the /etc/docker/daemon.json file after change.

nano@jetson-nano:~$ cat /etc/docker/daemon.json
{
    "default-runtime": "nvidia",
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

After the change, restart the docker.

sudo systemctl restart docker

5.2 Run installation command

Step 5 Verify AI inference service of an edge node

Open the online inference service through browser: http: //「ip」 : 24401/, and upload the test image. The inference result is as follows, which demonstrates that the AI service is started normally.

Historical Versions

EasyEdge Overview

百度智能云

Intelligent Edge