百度智能云

All Product Document

          Intelligent Edge

          baetyl-remote-object Module

          Introduction

          The baetyl-remote-object module is used to upload the end side file to the cloud storage (BOS and Awss3). Awss3 refers to the storage compatible with the s3 interface, and it can support the standard awss3, ceph, and minio. baetyl-remote-object subscribes the message from the broker on the end side, parses the message content, and uploads the specified file in the message.

          Configuration

          The full configuration files of baetyl-remote-object are shown as follows, and the configuration field is explained correspondingly:

          clients: # Client configuration
            - name: # Client name, [must] specify, the client of the target configured in the rules must be selected from the list of clients configured here
              kind: # Support upload object storage service type, currently supports bos and awss3. The enumeration values are BOS, S3
              endpoint: # ervice address
              region: # Only valid for awss3 type, which means region and whose default value is  us-east-1. The priority of endpoint is higher than a region
              ak: # ak
              sk: # sk
              bucket: # bos、awss3 的 bucket
              timeout: # Default value: 30s, the timeout period for the client to connect to the bos and awss3 services
              backoff: # Upload strategy, effective for bos
                max: # Maximum number of retries, whose default value is 0
                delay: # The maximum upload retry time, whose default value is 20 s
                base: # Upload retry base time, whose default value is 0.3s. Retry will be done in an exponential growth mode of retry base multiplied by 2 until the maximum number of retries or the longest retry time is reached
              multipart: # The large file block configuration
                partsize: # The block size, whose default value is 10m, which is 10MB, which is divided into blocks over 10MB
                concurrency: # The number of concurrent upload blocks, whose default value is 10
              pool: # Coroutine pool configuration
                worker: # The maximum number of coroutines in the coroutine pool, whose default value is 1000
                idletime: # Idle cleaning time, whose default value is 30 s
              temppath: # Package type temporary storage address
              limit: # Monthly current limit strategy
                enable: # Whether to open the current limit, whose default value is false
                data: # Maximum upload flow/month, [must] be greater than 0, the unit supports k,m,g,t,p
                path: # Persistent statistics file path
          
          rules:
            - name: rule1 # Rule name, must be unique
              source: # News source
                topic: broker/topic1 # Message subject
                qos: 1 # Message quality
              target: # Message destination
                client: baidubos # object Target node
          
          logger: # Log configuration
            path: # Log path
            level: # Log level

          The following provides a configuration demo of BOS:

          clients:
            - name: baidubos
              kind: BOS
              endpoint: bj.bcebos.com
              ak: XXXXXXXXXXXXXXXXXXXXXXXXXX
              sk: XXXXXXXXXXXXXXXXXXXXXXXXXX
              multipart:
                partsize: 10m
                concurrency: 10
              pool:
                worker: 1000
                idletime: 30s
              bucket: bos-remote-demo
              temppath: var/lib/baetyl/tmp
              limit:
                enable: true
                data: 9g
                path: var/lib/baetyl/data/stats.yml
          
          rules:
            - name: rule1
              source:
                topic: broker/topic1
                qos: 1
              target:
                client: baidubos
          
          logger:
            path: var/log/baetyl/service.log
            level: "debug"

          Configuration demo of minio:

          clients:
            - name: minio
              kind: S3
              endpoint: http://127.0.0.1:8080
              ak: XXXXXXXXXXXXXXXXXXXXXXXX
              sk: XXXXXXXXXXXXXXXXXXXXXXXX
              multipart:
                partsize: 10m
                concurrency: 10
              pool:
                worker: 1000
                idletime: 30s
              bucket: bos-remote-demo
              temppath: var/lib/baetyl/tmp
              # max 1000g
              limit:
                enable: true
                data: 9g
                path: var/lib/baetyl/data/stats.yml
          
          rules:
            - name: rule3
              source:
                topic: broker/topic3
                qos: 1
              target:
                client: minio
          
          logger:
            path: var/log/baetyl/service.log
            level: "debug"

          Note: baetyl-remote-object will be connected to the broker on the end side by default. Refer to broker for relevant information.

          remote-object message template for reference:

          {
              "type":"UPLOAD",
              "content":{
                  "remotePath":"dir/abc.zip", // Path in bos bucket
                  "zip":true, // Pack zip true to pack as zip, under false, no changes will be made to the file, and tar for the folder
                  "localPath":"var/pic/2019-01-01.png", // Local directory, but file/folder, the folder can be packed in zip/tar
                  "meta": { // k-v map[string]string
                      "key1":"value1",
                      "key2":"value2"
                  }
              }
          }
          Previous
          Download Area
          Next
          baetyl-function Module