Application Scenarios

Hybrid cloud, which integrates public cloud and private cloud, is the main mode and development direction of cloud computing in recent years. For the sake of security, enterprises prefer storing data in private cloud, but at the same time, they want to get computing resources of public cloud. In the Baidu AI Cloud hybrid cloud solution, the customer IDC and Baidu AI Cloud VPC are connected through express tunnels/VPN to realize close connection between the cloud and local business. They can not only take advantage of the on-demand and easy expansion features of the cloud resources, but also utilize the local IDC to meet the compliance requirements.

Express tunnel/VPN can provide high-speed and secure transmission channel for IDC to access BOS. However, in the hybrid cloud scheme, when the customer IDC accesses BOS (for example, backing up the local data to the cloud), as the customer IDC resolves the BOS domain to get the public network IP address of BOS, the data flow is still transmitted through the public network, and the express tunnel/VPN cannot be used. This tutorial provides a solution for using express tunnel/VPN to access BOS in hybrid cloud.

Solution

Overview

The core of using express tunnel/VPN to access BOS in hybrid cloud is to use BCC as access agent:

1.First apply for a BCC and install Nginx as the reverse agent to BOS;

2.Modify the local DNS record of the customer IDC machine, and point the request for accessing BOS to the Nginx agent.

Note: This scheme can only guarantee access to BOS in the same area is realized by express tunnel/VPN and Intranet. For example, if the customer IDC is connected with Baidu AI Cloud's Beijing VPC, this scheme can use express tunnel/VPN to access the BOS bucket located in Beijing; however, if the customer visits the BOS bucket located in Guangzhou, the data will still be transmitted through the public network.

Configure Nginx Reverse Agent

1.Apply for a BCC first. Recommended configuration: 2-core CPU, 4G memory, CentOS 7.1, public IP not needed.

2.Install Nginx. As we need additional stream module to provide TCP protocol proxy, we shall also install stream module.

a) yum install nginx nginx-mod-stream

b) Configure Nginx:

cd /nginx/
vim nginx.conf

Change nignx.conf to the following configuration. Note that the following configuration is applicable to Beijing region. If you want to access BOS in Guangzhou region, replace bj in the configuration file with gz:

#user  work;
worker_processes  auto;

events {
    worker_connections  1024;
}

stream {
    error_log logs/access.log info;

    # The backend address of BOS in Beijing
    upstream httpbosbj {
        server bj.bcebos.com:80;
    }

    upstream httpsbosbj {
        server bj.bcebos.com:443;
    }

    server {
        listen       80;

        proxy_pass httpbosbj;
        # Proxy settings
        proxy_connect_timeout   5s;
        proxy_timeout           90s;
        proxy_buffer_size       4k;
    }

    server {
        listen       443;

        proxy_pass httpsbosbj;
        # Proxy settings
        proxy_connect_timeout   5s;
        proxy_timeout           90s;
        proxy_buffer_size       4k;
    }
}

c) Verify the configuration file and start Nginx (root needed):

 nginx -t 
 mkdir -Vp /usr/share/nginx/logs 

Confirm that Nginx started successfully:

Configure Customer IDC Nodes

Configure the local DNS record to let the customer IDC machine point the request of accessing BOS to the Nginx for transfer. For example, in CentOS environment, add records in/etc/hosts:

nginxIp bj.bcebos.com

Here NginxIp is the intranet IP address of the Nginx reverse agent configured above, eg. 192.168.1.5.The same is true for modifying local DNS records in Windows and other environments.

Testing

After the configuration is completed, you can use express tunnel/VPN to access BOS from IDC machine. You can use BOS SDK, CLI, BOS desktop and other tools to upload and manage files.

In addition, you can log in to the Nginx agent node to check the log to confirm whether the Nginx reverse agent works normally:

less /usr/local/nginx/logs/access.log

Note

The solution provided in this tutorial can meet the needs of accessing BOS in most cases. In the 2-core 4G configuration, the proxy node can support 1000 concurrent connections (this is because BOS limits the number of concurrent access connections of a single IP for anti DDoS reasons) and 1Gbps access speed.

However, the scheme has the following limitations:

• Nginx agent is a potential single-point failure. You need to maintain this BCC instance to ensure its normal operation;
• For higher upload speed and more concurrent connections, you can further improve the configuration of Nginx nodes or add additional Nginx nodes.

In addition, VPC Endpoint is about to go online. VPC Endpoint is a highly available and scalable VPC access BOS agent. Please look forward to it!