Introduction

BOSFTP is a Python-based FTP server tools of BOS that converts the operations of FTP protocol into operations to the objects and directories stored in BOS. Support operations such as upload, download, deletion of file and creation of folders

Main Functions

The main functions supported by BOSFTP include the followings:

• Streaming upload of files, multi-part and parallel upload of large files
• Streaming download of files
• Support most of the FTP commands, and meet the demands of FTP
• Configurable service that can meet the performance requirements

System Environment

• Operating system: LINUX
• Python version: Python 2.7
• Dependency: pyftpdlib>=1.5.0 bce-python-sdk>=0.8.23

Installation

1.Use pip to install pyftpdlib

pip install pyftpdlib 

2.Install python sdk

cd bce-python-sdk-0.8.23 
python setup.py install 

3.Verify if the dependent package is well installed

cd ..
python setup.py install

4.Download BOSFTP, and extract it.

Enable FTP Server

Execute the command to enable an FTP server, and the default listening port is 2121.

bash start.sh 
> running ftp server at port 2121... 

Disable

• Execute the command ctrl + c to close FTP Server service.
• Run sh start.sh& to enable the service at the designated background, and run sh stop.sh to disable the service

Configuration Files

[BOS_ACCOUNT] 
# Configure ak, sk, bucket 
access_key_id =<your access_key_id> 
secret_access_key =<your secret_access_key> 
bucket =<BucketName> 
# Working directory of FTP Server 
bos_user_home_dir = /home/users/xxx

[FTP_ACCOUNT] 
# Configuration of FTP account The configuration format is shown as<User name: Password: read-write access permission>, and multiple accounts are separated by semicolons. 
login_users = usr1:psw1:RW;usr2:psw2:R 

[NETWORK] 
# Configure endpoint, the default is http://bj.bcebos.com 
#endpoint = http://xx.bcebos.com 
# The default FTP listening port number is 2121.
listen_port = 8081
# The default HTTP connection time-out is 3600. 
connection_timeout_in_mills = 10000
# The default maximum number of parallel upload, namely the thread count of thread pool is cpu_count * 4 
parallel_upload_num = 9
# The default maximum connections of FTP server are 1024.
max_connection_num = 1000
# The three-step upload splits the file into each multi_part_size, and then uploads in parallel, with the default size of 10485760 (10M) 
multi_part_size = 10485760
masquerade_address = 127.0.0.1

[LOG] 
# Set log level. 
log_level = INFO 
# Configure the log directory, with the log name of bosftp.log 
log_dir = /home/users/lihangyu01/baidu/bos/bosftp/log

Configuration Files Description

• bos_user_home_dir: indicating the local folder path, either upload or download is made under such directory
• login_users: indicating the information on user account, it is to verify if the username and password are stored in login_users1 when the service is enabled, for example: open 127.0.0.1 2121 to connect to server, which prompts to enter username and password, and the information entered here must be in login_users to be verified
• parallel_upload_num: In case of uploading a large file, the file will be split into multiparts (default to be 10M), and then be uploaded in parallel, and such parameter is used to control the concurrence.
• masquerade_address: Use the option of masquerade_address to bind a unique IP for reply under the passive mode

Testing

sh start.sh& " to enable the background service 
cd test 
sh run_test.sh 
sh stop.sh 

Note: When AK/SK is configured, the bucket must exist actually to enable the service in the background

FAQs

• What if the parent directory is deleted after the created directory is deleted?

For example, execute the following operation:

mkdir dir1/dir2 
cd dir1 
rm dir2 
cd .. 

Since the way to create a directory is to create an empty object named dir1/dir2, the above-mentioned operation will cause dir1 to be deleted as well. So, it is recommended to use the following way when creating a directory:

mkdir dir1 
cd dir1 
mkdir dir2 

• Execute size file_name, display '550 SIZE not allowed in ASCII mode'

Execute type binary command to modify the transmission mode

• When the configuration parameter connection_timeout_in_mills is relatively small, the rename of a large file object fails.

It is recommended that the configuration parameter connection_timeout_in_mills is 10000, namely 10s.

• Put an object with the size of 0 shows success, but list does not.

The pyftpdlib library itself does not read a file with the size of 0, so it can upload a file of such size (0).

• After the FTP service is enable, what about Connection Refused as shown on other machine?

Show as follows

227 Entering passive mode (127,0,0,1,97,183). 
ftp: connect: Connection refused 

Just execute the command passive

Version History

• BOSFTP kit [2018-03-08]Version No. 1.0.1

  • Support APPE interface to realize breakpoint resume function
• BOSFTP kit [2018-11-09]Version No. 1.0. 0
• Support file upload, download, deletion, and support for folder creation
• Support the command like put, get, delete, mkdir, ls, cd, quite, size and rename
• Customize ftp configurations
• Multi-part uploading of large files