Logstash Data Storage to BOS

Updated at：2025-11-03

Tool overview

Logstash is an open-source data collection, processing, and analysis tool that captures events from various sources and streams them to the desired repository. This document describes how Logstash can store data in the BOS bucket.

Configuration guide

Download and install the Logstash installation package from the official website, select the latest version for your OS. After download, unzip the package and create configuration files.
Shell
```
1wget https://artifacts.elastic.co/downloads/logstash/logstash-8.15.0-linux-x86_64.tar.gz
2tar zxvf logstash-oss-8.15.0-darwin-aarch64.tar.gz
3cd logstash-8.15.0
4touch file2bos.conf
```

Modify the configuration file file2bos.conf, use local files as input source and BOS as output source. For detailed configuration, refer to Plugin

                Shell
                
            

                input {
   file {
     path => "/tmp/access.log"
     start_position => "beginning"
   }
}
output {
   s3 {
 endpoint => "bos endpoint"     # e.g.https://s3.bj.bcebos.com
     access_key_id => "bos ak"
     secret_access_key => "bos sk"
 bucket => "bos-bkt"        # Required field
     size_file => 1024
     time_file => 1
     prefix => "logstash/"
   }
}
            

Parameter description

Parameter name	Parameter meaning
path	Enter the absolute path of the source file
start_position	Determine the position where the reading data can be started from the file; beginning means reading from the beginning of the file
endpoint	BOS Endpoint，Reference document
access_key_id	For BOS Access Key ID, Refer to How to Obtain AKSK
secret_access_key	BOS Secret Access Key
bucket	BOS bucket name
size_file	Set the file size; when the file exceeds the specified size_file value in bytes, a new file is generated.
time_file	Set the file rolling period in minutes. When the data writing duration surpasses the rolling period, a new file is generated.
prefix	Specify the cloud file system directory, e.g., “logstash/”; files will be written to the bucket’s logstash/ directory

Run the program: bin/logstash -f file2bos.conf
Result verification: Newly written objects can be checked in the specified directory in the BOS bucket, with the name format: ls.s3.ffe6de57-539a-4953-adb1-2807d604d1ec.2023-08-23T14.19.part0.txt

Fluentd Collects Data Storage to BOS

Elasticsearch Snapshot Data Storage to BOS