BOS-FTP Tool

Introduction

BOSFTP is a Python-based FTP server tool for BOS that translates FTP protocol operations into actions on objects and directories stored in BOS. It supports file uploads, downloads, deletions, and folder creation.

Key functions

Key functions supported by BOSFTP include:

• File streaming upload, large files are uploaded in parallel by multiparts
• File streaming download
• Support most FTP commands to meet FTP requirements
• Service configuration to meet performance requirements

System environment

• OS: LINUX
• Python version: Python 2
• Dependency: pyftpdlib>=1.5.0 bce-python-sdk>=0.8.23

Install

1. Install pyftpdlib using pip

   Shell Copy

   1pip install pyftpdlib

2. Install the Python SDK along with its dependencies.

   Shell Copy

   1pip install pycryptodome
   2pip install future
   3pip install futures
   4pip install bce-python-sdk

3. Download BOSFTP and unzip.

4. Verify whether dependencies are properly installed

   Shell Copy

   1cd ..
   2python setup.py install

Start FTP server

Run the command to initiate an FTP server, with the listener port set to 2121 by default.

1bash start.sh
2> running ftp server at port 2121...

Disable

• Execute the ctrl + c command to disable the FTP Server service. -Run sh start.sh& at backend to enable the service; execute sh stop.sh to disable it.

Configuration file

1[BOS_ACCOUNT]
2# Set ak, sk, bucket
3access_key_id = <your access_key_id>
4secret_access_key = <your secret_access_key>
5bucket = <BucketName>
6 Working directory of # FTP Server.
7bos_user_home_dir = /home/users/xxx
8
9[FTP_ACCOUNT]
10# FTP account configuration. Configuration format: <username:password:read-write permission>; multiple accounts are separated by semicolons.
11login_users = usr1:psw1:RW;usr2:psw2:R
12
13[NETWORK]
14# Set the endpoint, defaulting to http://bj.bcebos.com
15#endpoint = http://xx.bcebos.com
16# FTP listener port number, defaulting to 2121
17listen_port = 8081
18# HTTP connection timeout duration, defaulting to 3600
19connection_timeout_in_mills = 10000
20# Maximum parallel upload threads (number of threads of the thread pool), defaulting to cpu_count * 4
21parallel_upload_num = 9
22# FTP server maximum connections, defaulting to 1024
23max_connection_num = 1000
24# Three-step upload splits the file into several multi_part_size for parallel uploading, defaulting to 10485760 (10M)
25multi_part_size = 10485760
26masquerade_address = 127.0.0.1
27# passive_port can specify the port selection range in passive mode, defaulting to the (60000, 65535) range
28passive_port = 60000,65535
29
30[LOG]
31# Set the log level
32log_level = INFO
33# Set the log directory with the log name bosftp.log
34log_dir = ./log/

Configuration file description

• bos_user_home_dir: Local folder path: All uploads and downloads will be carried out under this directory.
• login_users: User account information: When enabling the service, it will be used to verify whether the username and password exist in the “login_users1”. For example: When open 127.0.0.1 2121 connects to the server, the system will prompt users to enter the username and password, which will only pass the permission authorization if it exists in the login_users.
• parallel_upload_num: Large file will be split into multiparts (defaulting to 10 M) for concurrent loading. This parameter is used to control the concurrency level
• masquerade_address: Use the masquerade_address option to uniquely bind an IP for reply in passive mode
• log_dir: The directory where BOSFTP operation log files are stored.

Testing

1sh start.sh&” enables the backend service
2cd test
3sh run_test.sh
4sh stop.sh

Note: When configuring AK/SK, the service will only be activated on the backend if the bucket actually exists.

FAQs

• What should the user do if deleting a created directory results in the parent directory being deleted?

For example, perform the following operations:

1mkdir dir1/dir2
2cd dir1
3rm dir2
4cd ..

Since creating a directory involves generating an empty object named dir1/dir2, this action will also result in the deletion of dir1. Recommended method for directory creation:

1mkdir dir1
2cd dir1
3mkdir dir2

• When executing the size file_name, it displays '550 SIZE not allowed in ASCII mode'

Execute the type binary command to modify the transmission method.

• When the connection_timeout_in_mills configuration parameter is set too small, rename process of a large file object may fail due to timeout

It is recommended to set the connection_timeout_in_mills configuration parameter value to 10000, i.e., 10 s.

• A zero-sized object uploaded successfully may not appear during list operations.

The pyftpdlib library does not process zero-sized files by default, making it impossible to upload such files.

• After enabling the FTP service, other machines show Connection Refused when trying to connect. What should the user do?

Display as follows

1227 Entering passive mode (127,0,0,1,97,183).
2ftp: connect: Connection refused

Execute the passive command.

Version records

• BOSFTP toolkit [2024-01-24] v1.3.1

  • Compatible with pyftpdlib v1.5.9+ version

• BOSFTP toolkit [3/8/2018] v1.2.0

  • Fix Python SDK version issues during installation and optimize log directory configuration

• BOSFTP toolkit [3/8/2018] v1.0.1

  • Support the APPE API for resumable upload function

• BOSFTP toolkit [11/9/2018] v1.0.0

  • Support file upload, download, deletion, and folder creation
  • Support put, get, delete, mkdir, ls, cd, quite, size, rename commands
  • Custom FTP configuration
  • Upload large file in parts