baetyl-remote-object Module
Introduction
The baetyl-remote-object module is used to upload the end side file to the cloud storage (BOS and Awss3). Awss3 refers to the storage compatible with the s3 interface, and it can support the standard awss3, ceph, and minio. baetyl-remote-object subscribes the message from the broker on the end side, parses the message content, and uploads the specified file in the message.
Configuration
The full configuration files of baetyl-remote-object are shown as follows, and the configuration field is explained correspondingly:
clients: # Client configuration
- name: # Client name, [must] specify, the client of the target configured in the rules must be selected from the list of clients configured here
kind: # Support upload object storage service type, currently supports bos and awss3. The enumeration values are BOS, S3
endpoint: # ervice address
region: # Only valid for awss3 type, which means region and whose default value is us-east-1. The priority of endpoint is higher than a region
ak: # ak
sk: # sk
bucket: # bos、awss3 的 bucket
timeout: # Default value: 30s, the timeout period for the client to connect to the bos and awss3 services
backoff: # Upload strategy, effective for bos
max: # Maximum number of retries, whose default value is 0
delay: # The maximum upload retry time, whose default value is 20 s
base: # Upload retry base time, whose default value is 0.3s. Retry will be done in an exponential growth mode of retry base multiplied by 2 until the maximum number of retries or the longest retry time is reached
multipart: # The large file block configuration
partsize: # The block size, whose default value is 10m, which is 10MB, which is divided into blocks over 10MB
concurrency: # The number of concurrent upload blocks, whose default value is 10
pool: # Coroutine pool configuration
worker: # The maximum number of coroutines in the coroutine pool, whose default value is 1000
idletime: # Idle cleaning time, whose default value is 30 s
temppath: # Package type temporary storage address
limit: # Monthly current limit strategy
enable: # Whether to open the current limit, whose default value is false
data: # Maximum upload flow/month, [must] be greater than 0, the unit supports k,m,g,t,p
path: # Persistent statistics file path
rules:
- name: rule1 # Rule name, must be unique
source: # News source
topic: broker/topic1 # Message subject
qos: 1 # Message quality
target: # Message destination
client: baidubos # object Target node
logger: # Log configuration
path: # Log path
level: # Log level
The following provides a configuration demo of BOS:
clients:
- name: baidubos
kind: BOS
endpoint: bj.bcebos.com
ak: XXXXXXXXXXXXXXXXXXXXXXXXXX
sk: XXXXXXXXXXXXXXXXXXXXXXXXXX
multipart:
partsize: 10m
concurrency: 10
pool:
worker: 1000
idletime: 30s
bucket: bos-remote-demo
temppath: var/lib/baetyl/tmp
limit:
enable: true
data: 9g
path: var/lib/baetyl/data/stats.yml
rules:
- name: rule1
source:
topic: broker/topic1
qos: 1
target:
client: baidubos
logger:
path: var/log/baetyl/service.log
level: "debug"
Configuration demo of minio:
clients:
- name: minio
kind: S3
endpoint: http://127.0.0.1:8080
ak: XXXXXXXXXXXXXXXXXXXXXXXX
sk: XXXXXXXXXXXXXXXXXXXXXXXX
multipart:
partsize: 10m
concurrency: 10
pool:
worker: 1000
idletime: 30s
bucket: bos-remote-demo
temppath: var/lib/baetyl/tmp
# max 1000g
limit:
enable: true
data: 9g
path: var/lib/baetyl/data/stats.yml
rules:
- name: rule3
source:
topic: broker/topic3
qos: 1
target:
client: minio
logger:
path: var/log/baetyl/service.log
level: "debug"
Note: baetyl-remote-object will be connected to the broker on the end side by default. Refer to broker for relevant information.
remote-object message template for reference:
{
"type":"UPLOAD",
"content":{
"remotePath":"dir/abc.zip", // Path in bos bucket
"zip":true, // Pack zip true to pack as zip, under false, no changes will be made to the file, and tar for the folder
"localPath":"var/pic/2019-01-01.png", // Local directory, but file/folder, the folder can be packed in zip/tar
"meta": { // k-v map[string]string
"key1":"value1",
"key2":"value2"
}
}
}