This chapter uses the open-source ssd_mobilenet_v1_coco_2017_11_17 model to demonstrate the demo of edge video AI fully.

I. Prerequisites

• There is an edge node device. This document use Chenyao POC-351VTC to install ubuntu-18.04-desktop-amd64.
• There is an Intel Movidius NCS 2.
• There is a camera, which may be the USB Camera or Webcam. The latter is used in this experiment. The webcam is powered via the POE port of the edge node.
• There is an object recognition model ssd_mobilenet_v1_coco_2017_11_17 is used in this experiment, which supports to detect 90 objects. For the detailed list, referr to rmscoco_label_map.
• The edge node has connected to the Baidu AI Cloud per the [[Getting](BIE/Getting Started/Getting Started Guide.md) Started](BIE/Getting Started/Getting Started Guide.md) tutorial.

Build the edge hardware environment shown in the figure below:

The steps are as follows:

1. Supply the power to the edge node hardware.
2. Connect the webcam to the POE port.
3. Insert the wireless NIC and Movidius into any USB port.

The overall environmental information is shown as follows:

1. Edge node

   • Wireless NIC IP: 172.30.196.93
   • IP connected to webcam: 192.168.100.14

2. Webcam IP

   • Webcam IP: 192.168.100.10
   • RTSP stream address:rtsp://b:a1234567@192.168.100.10:554/Streaming/channels/1/, which will be used later.

   The general format of RTSP protocol address is rtsp://<username>:<password>@<ip>:<port>/<Streaming/channels/stream_number> whose parameters are explained as follows:

   • <username> : Login user name of camera, which can be usually found in the base of camera.
   • <password> : Login password of camera, which can be usually found in the base of camera.
   • <ip> : IP address assigned to camera by router/switch.
   • <port> : Port number of RTSP protocol, generally defaults to 554,
   • <Streaming/channels/stream_number> : Camera channel

III. Simulation Scenario

1. The camera is connected to the edge box to detect objects in the field of vision (FOV) in real-time.
2. When it detects the target object, it will save the posterize time image and send a message to the edge broker service synchronously. If it does not detect the target object, it will discard the posterize time image.
3. It supports the detection of multi-target objects, and the objects detected in this scenario experiment include scissors, laptop, Book, keyboard, and person.

In addition that two system applications baetyl-core and baetyl-function are automatically deployed when the edge node is connected to the cloud, it is necessary to deploy the following four applications on the edge node.

Finally, there will be 6 edge applications on the edge node as shown in the figure below:

V. Edge Application Relationship

The figure below shows the call relationship among modules:

The whole video inference process is as follows:

1. Video posterize time
2. Load the model to perform the AI inference.
3. Call the function to handle the inference results.
4. Save the image that reaches the specified threshold based on the function return results.
5. Send the inference results.
6. Subscribe to the local broker message, and upload the inference images that meet the conditions to the BOS minio.

vi-broker configuration

1. Create the application: vi-broker, add the container service:baetyl-broker, and configure the application as shown in the figure below:

• Service name:baetyl-broker
• Image: hub.baidubce.com/baetyl/broker: v2.0.0

• Port Mapping

  • Host port: 1883
  • Container port: 1883
  • Protocol: TCP

vi-function configuration

1. Create the function configuration item: vi-function-code as shown in the figure below:

• Tag: baetyl-function: python3-opencv
• Configure the data variable name: analyse.py. This variable nameanalysewill be used as the function entry.
• Configure the data variable value as the python code as follows:

#!/usr/bin/env python
# -*- coding:utf-8 -*-
"""
function to analyse video infer result in python
"""

import time
import numpy as np

location = "var/lib/baetyl/image/{}.jpg"
classes = {
        1: 'person',73: 'laptop',76: 'keyboard',77: 'cell phone',84: 'book',87: 'scissors'
}

def handler(event, context):
    """
    function handler
    """

    data = np.fromstring(event, np.float32)
    mat = np.reshape(data, (-1, 7))
    objects = []
    scores = {}
    for obj in mat:
        # print("obj:", obj)
        clazz = int(obj[1])
        if clazz in classes:
            score = float(obj[2])
            if classes[clazz] not in scores or scores[classes[clazz]] < score:
                scores[classes[clazz]] = score
            if score < 0.6:
                continue
            objects.append({
                'class': classes[clazz],
                'score': score,
                'left': float(obj[3]),
                'top': float(obj[4]),
                'right': float(obj[5]),
                'bottom': float(obj[6])
            })

    res = {}
    res["imageDiscard"] = len(objects) == 0
    res["imageObjects"] = objects
    res["imageScores"] = scores
    path = location.format(time.time())
    if len(objects) != 0:
        res["imageLocation"] = path
        res["publishTopic"] = "video/infer/result"
        res["type"] = "UPLOAD"
        content = {}
        content["localPath"] = path
        content["remotePath"] = path
        res["content"] = content

    return res

2. Create the function application: vi-function, and add the function service: vi-function-service as shown in the figure below:

• Service name: vi-fucntion-service
• Service type: Function service, other than container service
• Function configuration item: Create the configuration item: vi-function-conf before use.
• Runtime: python3-opencv
• Function entry: alyse.handler.

The full path of function is [Service Name/Function Entry], and the full call path to analyze this python function above isvi-fucntion-service/analyse.

video-infer configuration

1. Create the configuration item: video-infer-conf as shown in the figure below:

• create the configuration data with the variable name: conf.yml. The variable is shown as follows:

video:
  uri: 'rtsp://b:a1234567@192.168.100.10:554/Streaming/channels/1/'
  limit:
    fps: 1
infer:
  model: var/lib/baetyl/model/frozen_inference_graph.pb
  config: var/lib/baetyl/model/ssd_mobilenet_v1_coco_2017_11_17.pbtxt
  backend: openvino
  device: vpu
process:
  before:
    swaprb: true
    width: 300
    hight: 300
  after:
    function:
      name: vi-fucntion-service/analyse # Corresponds to the vi-function function created earlier.
logger:
  path: var/lib/baetyl/app-log/video-infer.log
  level: debug
broker:
  address: 'tcp://baetyl-broker:1883' # The vi-broker application created earlier corresponds to.
  clientid: video-infer
function:
  address: 'http://baetyl-function.baetyl-edge-system'

• The application video-infer uses the VPU computing power of Movidius NCS to perform the edge inference computing through the configuration items of backend: openvino and device: vpu . If there are no these two configuration items, the CPU inference is used by default.

• After it performs the AI inference, video-infer will call the analysis function vi-fucntion-service/analyse.
• The message returned by the analysis function is published to the local MQTT Broker tcp://baetyl-broker:1883.

2. Create the configuration item: video-infer-model as shown in the figure below:

• endpoint: Object storage access address

• bucket name: Bucket name created in the object storage, which is model-upload here.
• File name: Name of model file in object storage. If it is in the directory, you need to fill in the directory name + file name. It is ssd_mobilenet_v1_coco_2017_11_17.zip here.
• AK/SK: Object storage access credential
• MD5: MD5 value of model file ssd_mobilenet_v1_coco_2017_11_17.zip:acc050a4e8fcea32edcb30ab510e63b7. After it downloads the model file ssd_mobilenet_v1_coco_2017_11_17.zip successfully, the edge node will perform the MD5 verification on the model file to verify the integrity of downloaded file.
• Decompress: Select Yes, and the decompression format is ZIP. After it downloads the model file ssd_mobilenet_v1_coco_2017_11_17.zip, the edge node will be responsible for decompression. In this case, you will get the model files frozen_inference_graph.pb and ssd_mobilenet_v1_coco_2017_11_17.pbtxt after the model file ssd_mobilenet ssd_mobilenet_v1_coco_2017_11_17.zipp is decompressed.

3. Create the application: video-infer, and add the container service: video-infer-openvino as shown in the figure below:

• service name: video-infer-openvino

• Image: hub.baidubce.com/baetyl/video-infer: v2.0.0-openvino-amd64

• Volume configuration:

  • conf: Add the video-infer-conf configuration item as the module configuration.
  • model: Add the video-infer-model configuration item. The edge node will automatically download the model file ssd_mobilenet_v1_coco_2017_11_17 and decompress it to the/var/lib/baetyl/modeldirectory, and then the video-infer application will load the model files frozen_inference_graph.pb and ssd_mobilenet_v1_coco_2017_11_17.pbtxt from here.
  • image: Map the image saved by the AI inference to the host directory from the directory/var/lib/baetyl/imagein the container. The video-infer-imageparameter is filled in here, and the corresponding host directory is/var/lib/baetyl/app-data/video-infer-image.
  • dev: After the Movidius USB is inserted into the edge box, there will be one more device in the host/dev directory, and the directory/devof host will be mapped to the directory/devin the container.

• Privilege Option:

  • Privilege mode enabling: Yes

remote-object configuration

1. Create the configuration item: remote-mino-conf as shown in the figure below:

Add the configuration data variable name: service.yml. The corresponding variable values are shown as follows:

clients:
  - name: minio
    kind: S3
    endpoint: 'http://ip:port' # Replace with your own object storage address
    ak: AKIAIOSFODNN7EXAMPLE
    sk: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    timeout: 10m
    pool:
      worker: 100
      idletime: 30s
    bucket: image-upload
rules:
  - name: remote-minio-1
    source:
      topic: video/infer/result
      qos: 1
    target:
      client: minio
logger:
  path: var/lib/baetyl/app-log/minio.log
  level: debug

2. Create the application: remote-object, and then add the container service: remote-object as shown in the figure below:

• Service name: remote-object
• Image address: hub.baidubce.com/baetyl-sandbox/remote-object: git-d37fded

• Volume configuration

  • minio-conf: Mount the configuration item of remote-object service.
  • image: Map the inference image in the host to the directory in the container. In this way, the remote-object service can get the image and upload it to the object storage minio.
  • log: If you need to map the log file of modules to the host, you need to configure this item.

VII. Verify Edge AI Detection Results

step1: Deploy the application to the edge node.

1. Deploy all of the above applications to the edge node as shown in the figure below:

2. Check the deployment of applications on the edge node as shown in the figure below:

step2: Subscribe to the local broker service of the edge node with the MQTT Box.

It mentions that it will save the posterize time image and send a message to the edge broker service synchronously when it detects the target object in the Simulation Scenario. In order to monitor the messages sent to the broker service, we subscribe the message of the video/infer/result topic in advance through the MQTT Box as shown in the figure below:

step3: Detect the object with the camera.

1. Hold the camera and rotate it for one circle, so that the camera can scan the Scissors, laptop, book, keyboard, and Peopleon the desk and the on the station.
2. View the message interface of MQTT Box subscribing to the hub module in real-time. When it detects a target object every time, the MQTT Box can subscribe to a message.

3. Carry out the Json formatting of messages subscribed with the MQTT Box to get the results as follows:

{
    "content":{
        "localPath":"var/lib/baetyl/image/1602639487.0222735.jpg",
        "remotePath":"var/lib/baetyl/image/1602639487.0222735.jpg"
    },
    "imageCaptureTime":"2020-10-14T01:38:06.971394704Z",
    "imageDiscard":false,
    "imageHight":720,
    "imageInferenceTime":0.046391136,
    "imageLocation":"var/lib/baetyl/image/1602639487.0222735.jpg",
    "imageObjects":[
        {
            "bottom":0.9267578125,
            "class":"scissors",
            "left":0.166259765625,
            "right":0.505859375,
            "score":0.97412109375,
            "top":0.233154296875
        }
    ],
    "imageProcessTime":0.082386501,
    "imageScores":{
        "book":0.09637451171875,
        "cell phone":0.0443115234375,
        "person":0.0282135009765625,
        "scissors":0.97412109375
    },
    "imageWidth":1280,
    "publishTopic":"video/infer/result",
    "type":"UPLOAD"
}

We can draw the following conclusions from the above information:

• It detects that the object is scissor:"class": "scissors"
• The score that AI infers it as scissor is 0.974:"scissors": 0.974121093751
• Image saved"imageLocation": "var/lib/baetyl/image/1602639487.0222735.jpg": corresponding host directory:/var/lib/baetyl/app-data/video-infer-image/1602639487.0222735.jpg

step4: Verify the objects in the saved image.

1. SSH logs in to the edge node to view that multiple posterize time images have been saved as shown in the figure below:

2. Download 1602639487.0222735.jpg to the local computer, and confirm that the object in the image is a scissor, which is consistent with the message received by the MQTT Box as shown in the figure below:

step5: Verify that the inference result images have been uploaded to BOS.

Open the Minio Browser, select the bucket image-upload, and then progress to the var/lib/baetyl/image directory. We can see the image uploaded from the edge node as shown in the figure below: