Through IntelliEdge - BIE - Desensitize and then Store Data in the Cloud - TSDB

Last Updated：2021-10-28

Preface

The device system used for test in this article is Darwin.
The client that simulates the MQTT client subscribing to messages to Baidu AI Cloud IoTHub is MQTT.fx.
The client that simulates the MQTT client sending messages to the local Hub service is MQTTBOX.
It is mandatory to use the TLS/SSL security certificate for communication authentication between the services in this article and the local Hub service.

Preparation Before Test

In actual application scenarios, data generated by the device have to be desensitized locally before displayed in the cloud.

This article takes the temperature sensor in an actual production workshop as an example. The Baidu OpenEdge framework and related products and services of Baidu AI Cloud SkyEngineering are applied to desensitize data collected by the temperature sensor (such as removing workshop number, device model and device ID) and then upload the desensitized data to the remote cloud for visual display.

The path/service through which the data flow is:

MQTT.fx -> Local Hub -> Function SQL Runtime -> Local Hub -> MQTT Remote Module -> Baidu IoT Hub -> Baidu IoT Rule Engine -> Baidu IoT TSDB -> Baidu IoT Visualization

Therefore, before officially starting the test, you have to perfect related configurations in the cloud such as IoT Hub, Rule Engine, TSDB and Visualization.

Create an IoT Hub Endpoint

For the process to create it, please see Quickly Create an IoT Hub Endpoint (including creating users, identities, policies and topic authorities). A diagram of the results after it is created is given here.

As above, an IoT Hub project named baetyl_demo has been created. The user name is guqgsr9/test, the identity information is principal, the authentication method is certificate authentication, the policy is policy, and it has privilege to release and subscribe messages to the topic data/filter. (See the module configuration for the MQTT Remote service during test in the following content.)

Create a Rule Engine Rule

For the process to create it, please see Quickly Create a Rule Engine Rule (including compilation of conversion SQL statements, setting of constraints, designation of data destination). 2 rules have to be created here. One is for filtering the original data generated by the local device; the other is for extracting the data messages received from the IoT Hub predetermined topic in real time, converting them into data content that can be received by TSDB and then transmitting them to TSDB. The diagram of the results after the rules are created is as follows:

Rule for filtering device production data:

As above, a rule named filter has been created, and it is used to filter and desensitize the original data actually generated by the local device. The figure shows the process to filter the temperature, ts, unit and other fields in the actual production data which satisfy temperature> 50 and then upload them to the Hub module of the remote cloud.

Note: You do not have to set data destination for the rule. It is only used for cloud SQL statement debugging.

Rule for receiving messages from the IoT Hub predetermined topic:

As above, a rule named baetyl-demo has been created. This rule extracts messages from the data/filter topic of the IoT Hub Endpoint by default, convert them into data that conform to TSDB Specification through Convert SQL Statements, and store them in the TSDB database named baetyl.

Create a TSDB Database

For the process to create it, please see Quickly Create a TSDB (including query category, time period and time series measurement). A diagram of the results after it is created is given here.

As above, a TSDB time series database named baetyl has been created. The database queries by default the time series data information that satisfies the time series measurement identifier in the selected time period, and the first 1000 values that meet the above conditions are displayed by default.

Create an IoT Visualization Display Board

For the process to create it, please see Quickly Create an IoT Visualization (including setting source data table and time series measurement information). A diagram of the results after it is created is given here.

As above, an IoT Visualization display board named baetyl_demo has been created. The displayed data come from the time series database baetyl, the time series measurement information is device_temperature, the X axis is the timestamp, and the Y axis is the device temperature value.

With this, related services in the cloud before the official test are created and set.

Note: The regions of the IoT Hub Endpoint, rule engine Rule, TSDB database and IoT Visualization display board created above should be the same. For example, all the regions are Beijing or Guangzhou.

Basic Steps

The steps to upload the device production data to the cloud after desensitization, write them into TSDB and display them in the IoT Visualization mainly include:

Step 1: Create a core and download the configuration (including the master program). Select a region (Beijing or Guangzhou) on the BIE cloud management suite page to create a core, complete the configuration information required for creating the core, click “Download a configuration”, and then select to include the master program. For details, please see BIE Getting Started.
Step 2: Start baetyl locally. After the master program (including configuration) is decompressed locally, start baetyl, and then click the core connection status button. If the baetyl program starts normally, you can see that the core connection status has changed to connected.
- Reference command to start BAETYL:
  - mkdir baetyl-demo
  - cd baetyl-demo && unzip -d . baetyl-xxx.zip
  - sudo chmod +x bin/baetyl
  - sudo bin/baetyl start
Step 3: Establish a service configuration. Enter the created core, and then start to create the service configuration information (Hub service configuration, Function Manager service configuration, Function Filter service configuration and Remote service configuration) required for the test one by one. For details, please see BIE Operation Practices.
- Hub service configuration: 4 storage volumes - conf, data, cert and log have to be mounted to store configurations of the Hub service, persistent data, ssl certification resources and logs respectively. For details about using and mounting storage volumes, please see How to import a storage volume correctly.
- Function Manager service configuration: 3 storage volumes - conf, cert and log have to be mounted to store configurations, client certificates connected to the Hub service and log data respectively.
- Function Filter service configuration: The conf storage volume has to be mounted to store configuration information.
- Remote service configuration: 4 storage volumes - conf, localhub-cert, iothub-cert and log have to be mounted to store configurations, SSL certificates of the localhub service, SSL certificates of the cloud iothub service and logs.
Step 4: Release and distribute a service configuration. After each service required for the core is configured, click “Generate a configuration” button to generate a configuration in the current version, and then click “Distribute a configuration” button to distribute the generated version configuration to the local device. The distributed configuration is automatically switched and loaded in the agent service. For details, please see BIE Operation Practices.
- In this process, the edge core is required to keep connected. If the edge core is disconnected before the configuration is distributed, the baetyl master program has to be restarted, and the new configuration can be distributed after the connection is restored to connected(recommended); or click Download a configuration button to download the new configuration to the local device, replace it locally, and then start the baetyl master program.
Step 5: Configure MQTTBOX connection information. Start MQTTBOX and configure the information required to establish a connection with the local Hub module
Step 6: Send test data. After the connection between MQTTBOX and the local Hub module is established, send test data to the topic data, open the TSDB panel to check whether data are successfully written, and open the IoT Visualization display Board to observe the status of data writing.
Step 7: Verify the result If the above process goes well, you can see that the test just sent has been successfully written into TSDB and displayed in the IoT Visualization.

Test and Verification

In this section, combined with the Baidu IntelliEdge (BIE) Cloud Management Suite, all the configuration information required to execute the local operating package of Baidu IntelliEdge is created from the cloud and then distributed for local deployment through the cloud management suite. Lastly, the local operating package of Baidu IntelliEdge is started locally to complete the test and verification of the whole test case.

Master Program Configuration

version: V2 
services: 
  - name: agent 
    image: 'hub.baidubce.com/baetyl/baetyl-agent:latest' 
    replica: 1 
    mounts: 
      - name: agent-conf-wgon28r2x-V1 
        path: etc/baetyl 
        readonly: true 
      - name: agent-cert-wgon28r2x-V1 
        path: var/db/baetyl/cert 
        readonly: true 
      - name: agent-volumes-wgon28r2x-V1 
        path: var/db/baetyl/volumes 
      - name: agent-log-wgon28r2x-V1 
        path: var/log/baetyl 
  - name: localhub 
    image: 'hub.baidubce.com/baetyl/baetyl-hub:latest' 
    replica: 1 
    ports: 
      - '1883:1883' 
      - '8883:8883' 
    mounts: 
      - name: baetyl-localhub-conf-V1 
        path: etc/baetyl 
        readonly: true 
      - name: baetyl-log-V1 
        path: var/log/baetyl 
      - name: baetyl-localhub-cert-V1 
        path: var/db/baetyl/cert 
        readonly: true 
      - name: baetyl-persist-data-V1 
        path: var/db/baetyl/data 
  - name: function-manager 
    image: >- 
      hub.baidubce.com/baetyl/baetyl-function-manager:latest 
    replica: 1 
    mounts: 
      - name: baetyl-func-manager-conf-V1 
        path: etc/baetyl 
        readonly: true 
      - name: baetyl-mqtt-client-cert-V1 
        path: var/db/baetyl/cert 
        readonly: true 
      - name: baetyl-log-V1 
        path: var/log/baetyl 
  - name: function-filter 
    image: 'hub.baidubce.com/baetyl/baetyl-function-sql:latest' 
    replica: 0 
    mounts: 
      - name: baetyl-function-filter-conf-V1 
        path: etc/baetyl 
        readonly: true 
  - name: remote-mqtt 
    image: 'hub.baidubce.com/baetyl/baetyl-remote-mqtt:latest' 
    replica: 1 
    mounts: 
      - name: baetyl-remote-mqtt-conf-V1 
        path: etc/baetyl 
        readonly: true 
      - name: baetyl-remote-iothub-cert-V1 
        path: var/db/baetyl/iothub/cert 
        readonly: true 
      - name: baetyl-mqtt-client-cert-V1 
        path: var/db/baetyl/localhub/cert 
        readonly: true 
      - name: baetyl-log-V1 
        path: var/log/baetyl 
volumes: 
  - name: agent-conf-wgon28r2x-V1 
    path: var/db/baetyl/agent-conf-wgon28r2x/V1 
    meta: 
      url: https://edge.bos.gz.xxxxxxx 
      md5: 6wKhAQhA/H1nymnO6G7u3g==
      version: V1 
  - name: agent-cert-wgon28r2x-V1 
    path: var/db/baetyl/agent-cert-wgon28r2x/V1 
    meta: 
      url: https://edge.bos.gz.xxxxxxx 
      md5: ZzM0vIavNPP66En8Cog3Wg== 
      version: V1 
  - name: agent-volumes-wgon28r2x-V1 
    path: var/db/baetyl 
  - name: agent-log-wgon28r2x-V1 
    path: var/db/baetyl/agent-log 
  - name: baetyl-log-V1 
    path: var/db/baetyl/baetyl-log 
  - name: baetyl-localhub-cert-V1 
    path: var/db/baetyl/baetyl-localhub-cert/V1 
    meta: 
      url: https://edge.bos.gz.xxxxxxx 
      md5: El/cnLg2nGC1/UfTZl6NLw==
      version: V1 
  - name: baetyl-persist-data-V1 
    path: var/db/baetyl/baetyl-persist-data 
  - name: baetyl-mqtt-client-cert-V1 
    path: var/db/baetyl/baetyl-mqtt-client-cert/V1 
    meta: 
      url: https://edge.bos.gz.xxxxxxx 
      md5: 8/McnX+CC3dQoyYCtN+99Q== 
      version: V1 
  - name: baetyl-function-filter-conf-V1 
    path: var/db/baetyl/baetyl-function-filter-conf/V1 
    meta: 
      url: https://edge.bos.gz.xxxxxxx 
      md5: TshyXdCfRZqAnS57MCzIRw== 
      version: V1 
  - name: baetyl-localhub-conf-V1 
    path: var/db/baetyl/baetyl-localhub-conf/V1 
    meta: 
      url: https://edge.bos.gz.xxxxxxx 
      md5: 4SCOtrGHd8fAnx7nkJ5pGw== 
      version: V1 
  - name: baetyl-remote-iothub-cert-V1 
    path: var/db/baetyl/baetyl-remote-iothub-cert/V1 
    meta: 
      url: https://edge.bos.gz.xxxxxxx 
      md5: XzfchDcE3xUcWa/PwqLGCw==
      version: V1 
  - name: baetyl-remote-mqtt-conf-V1 
    path: var/db/baetyl/baetyl-remote-mqtt-conf/V1 
    meta: 
      url: https://edge.bos.gz.xxxxxxx 
      md5: ncrHak7Le3nodgVoIrEmMQ== 
      version: V1 
  - name: baetyl-func-manager-conf-V1 
    path: var/db/baetyl/baetyl-func-manager-conf/V1 
    meta: 
      url: https://edge.bos.gz.xxxxxxx 
      md5: 65ksedn9d6jUcLEtHJl/0Q==
      version: V1

HubService Configuration

listen: 
  - tcp://0.0.0.0:1883
  - ssl://0.0.0.0:8883
certificate: 
  ca: var/db/baetyl/cert/ca.pem 
  cert: var/db/baetyl/cert/server.pem 
  key: var/db/baetyl/cert/server.key 
principals: 
  - username: two-way-tls 
    permissions: 
      - action: 'pub' 
        permit: ['#'] 
      - action: 'sub' 
        permit: ['#'] 
  - username: test 
    password: hahaha 
    permissions: 
      - action: 'pub' 
        permit: ['#'] 
      - action: 'sub' 
        permit: ['#'] 
logger: 
  path: var/log/baetyl/localhub-service.log 
  level: "debug"

Function ManagerService Configuration

hub: 
  address: ssl://localhub:8883 
  username: two-way-tls 
  ca: var/db/baetyl/cert/ca.pem 
  cert: var/db/baetyl/cert/client.pem 
  key: var/db/baetyl/cert/client.key 
  insecure: true 
rules: 
  - clientid: localfunc-1 
    subscribe: 
      topic: data 
      qos: 1 
    function: 
      name: filter 
    publish: 
      topic: data/filter 
      qos: 1 
functions: 
  - name: filter 
    service: function-filter 
    instance: 
      max: 10 
logger: 
  path: var/log/baetyl/func-service.log 
  level: "debug"

Function FilterService Configuration

functions: 
  - name: filter 
    handler: 'SELECT temperature, ts, unit WHERE temperature > 50'

As configured above, messages sent to the topic data are processed (desensitized and filtered) by the SQL runtime, and the processing results are fed back to the topic data/filter.

RemoteService Configuration

hub: 
  address: ssl://localhub:8883 
  username: two-way-tls 
  ca: var/db/baetyl/localhub/cert/ca.pem 
  cert: var/db/baetyl/localhub/cert/client.pem 
  key: var/db/baetyl/localhub/cert/client.key 
  insecure: true 
remotes: 
  - name: iothub 
    address: 'ssl://guqgsr9.mqtt.iot.gz.baidubce.com:1884' 
    clientid: remote-iothub-1 
    username: guqgsr9/test 
    ca: var/db/baetyl/iothub/cert/ca.pem 
    cert: var/db/baetyl/iothub/cert/client.pem 
    key: var/db/baetyl/iothub/cert/client.key 
rules: 
  - hub: 
      subscriptions: 
        - topic: data/filter 
          qos: 0 
    remote: 
      name: iothub 
      subscriptions: 
logger: 
  path: var/log/baetyl/remote-service.log 
  level: "debug"

As configured above, the local Hub service sends messages of the topic data/filter to the Remote service (subscription privilege for the topic data/filter has been granted when the IoT Hub Endpoint is created above), the remote Hub service (referring to the Baidu AI Cloud IoTHub platform) receives the messages of the topic data/filter and triggers the rule baetyl-demo (created above), the rule engine encapsulates the messages (to meet the TSDB specification) and transmits them to TSDB, and finally the messages are displayed in the IoT Visualization.

Testing

Set the connection of MQTTBOX according to the Hub service configuration, as shown in the figure below.

Similarly, set the connection of MQTT.fx according to the cloud IoT Hub configuration, as shown in the figure below.

Send messages to the topic data through MQTTBOX. Please see the following for the message format:

{ 
    "temperature": 67.5, //Temperature 
    "ts": 1568817178, // UnixTime stamp 
    "deviceid": "dev001", //Device ID 
    "factory": "shanghai", //Production address 
    "type": "temperature tensor", //Device type 
    "unit": "℃" // Unit 
}

According to the above message processing logic, this message is filtered out and sent back to the local Hub service, the local Hub service sends data to the Remote service, and then it is uploaded to the cloud IoT Hub, encapsulated through the rule baetyl -demo, transmitted to TSDB and finally displayed in the IoT Visualization. The diagram is shown below.

MQTTBOX receiving the processed message indicates that the message has been processed by the Function Filter service, and the result has been returned to the Hub service.

MQTT.fx receiving the cloud IoT Hub message indicates that the message has been sent to the cloud IoT Hub through the Remote service.

If the following message is sent to the topic data through MQTTBOX:

{ 
    "temperature": 35.6, //Temperature 
    "ts": 1568817182, // UnixTime stamp 
    "deviceid": "dev001", //Device ID 
    "factory": "shanghai", //Production address 
    "type": "temperature tensor", //Device type 
    "unit": "℃" // Unit 
}

Then neither MQTTBOX nor MQTT.fx receives the processed message ( temperature <50 is filtered out). Similarly, the rule engine baetyl-demo, TSDB and IoT Visualization do not receive the processed message.

To display the processed results in the cloud more clearly, we wrote multiple pieces of data that meet the requirements to get the corresponding TSDB and the IoT Visualization display effect, as shown below.

TSDB receives multiple processed results

IoT Visualization receives multiple processed results

With this, data have been written into Baidu AI Cloud TSDB through Baetyl and visually displayed through IoT Visualization.

Configuration File Description

FAQs

百度智能云

Intelligent Edge