Compatible Tool
Compatibility Description
Access addresses can be set on most tools developed based on AWS S3.Set the access addresses of these tools as AWS S3, the service domain name of BOS, to access BOS through these tools. Take some commonly used SDK and tools as examples to illustrate how to interface BOS.
Note:
Word string Meaning $ACCESS_KEY Access key of Baidu AI Cloud account $SECRET_KEY Secret key of Baidu AI Cloud account
AWS SDK for Python
1.Install Boto library:
pip install boto3 2.Access Bos through AWS SDK for python.
import boto3
from botocore.client import Config
s3 = boto3.client(
's3',
aws_access_key_id=$ACCESS_KEY,
aws_secret_access_key=$SECRET_KEY,
endpoint_url='http://s3.bj.bcebos.com',
region_name='bj',
config = Config(
signature_version='s3v4',
)
)
# Use S3 client
s3.create_bucket(...) AWS SDK for Java
1.Add dependency packages to pom.xml.
//Add the following AWS Java SDK dependency package to pom.xml.
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk</artifactId>
<version>1.11.82</version>
</dependency> 2.Access Bos through AWS SDK for java.
import java.io.IOException;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.*;
import com.amazonaws.services.s3.S3ClientOptions;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.SDKGlobalConfiguration;
public class S3Sample {
public static void main(String[] args) throws IOException {
System.setProperty(SDKGlobalConfiguration.ENABLE_S3_SIGV4_SYSTEM_PROPERTY, "true");
AmazonS3 s3 = new AmazonS3Client(new BasicAWSCredentials($ACCESS_KEY,$SECRET_KEY));
s3.setEndpoint("s3.bj.bcebos.com");
S3ClientOptions options = new S3ClientOptions();
options.withChunkedEncodingDisabled(true); // Must to have
s3.setS3ClientOptions(options);
// Use S3 Client
s3.createBucket(...);
}
}3.Compiled code
mvn package AWS PHP SDK
1.Installation: Download aws.phar. See AWS PHP SDK Installation Method for more installation methods. 2.Access Bos through AWS SDK for PHP.
<?php
require 'aws.phar';
use Aws\S3\S3Client;
use Aws\Exception\AwsException;
$s3Client = new S3Client([
'version' => 'latest',
'region' => 'bj',
'credentials' => [
'key' => $ACCESS_KEY,
'secret' => $SECRET_KEY,
],
'endpoint' => 's3.bj.bcebos.com',
'signature_version' => 'v4',
]);
$buckets = $s3Client->listBuckets();
foreach ($buckets['Buckets'] as $bucket){
echo $bucket['Name']."\n";
}AWS Golang SDK
1.Installation
go get -u github.com/aws/aws-sdk-go 2.Access Bos through AWS SDK for Golang.
import (
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
"github.com/aws/aws-sdk-go/aws/credentials"
)
conf := &aws.Config{
Region: aws.String("bj"),
Endpoint: aws.String("s3.bj.bcebos.com"),
Credentials: credentials.NewStaticCredentials($ACCESS_KEY, $SECRET_KEY,, ""),
}
sess := session.Must(session.NewSessionWithOptions(session.Options{Config:*conf}))
svc := s3.New(sess)
getObjectParams := &s3.GetObjectInput{
Bucket: aws.String("my-bucket"),
Key: aws.String("my-object"),
}
getObjectResp, err := svc.GetObject(getObjectParams)
if err != nil {
fmt.Println(err.Error())
return
}AWS CLI Tools
1.Install AWS CLI tools.
pip install awscli 2.Access the BOS using the AWS CLI
-
Edit Profile
$ cat ~/.aws/config [default] s3 = signature_version = s3v4 region = bj $ cat ~/.aws/credentials [default] aws_access_key_id = $ACCESS_KEY aws_secret_access_key = $SECRET_KEY -
Execute command
aws --endpoint-url http://s3.bj.bcebos.com s3 ls
Hadoop S3A Tools
S3A is hadoop's official toolkit for using S3 in hadoop system. Through S3A, you can operate S3 storage like operating hdfs. Currently BOS has already supported most commonly used S3A functions. You can find a more detailed introduction to S3A in: S3 Support in Apache Hadoop and Hadoop-AWS module: Integration with Amazon Web Services.
1.Download dependency packages
-
Ensure that the following dependency packages exist in the hadoop system:
- aws-java-sdk-1.11.82.jar - aws-java-sdk-core-1.11.82.jar
- aws-java-sdk-kms-1.11.82.jar - aws-java-sdk-s3-1.11.82.jar
- Use Jar Package which is applicable to BOS: As BOS is not compatible with some functions of S3 at present, for better product experience, please be sure to use this customized toolkit.
| Jar Package which is applicable to BOS | MD5 code |
|---|---|
| hadoop-aws-2.8.0.jar | 6ffbdc9352b9399e169005aeeb09ee98 |
2.Modify the relevant configuration of S3A
<property>
<name>fs.s3a.endpoint</name>
<value>http://s3.bj.bcebos.com</value>
</property>
<property>
<name>fs.s3a.signing-algorithm</name>
<value>AWSS3V4SignerType</value>
</property>
<!--Enable bos backend interface-->
<property>
<name>fs.s3a.bos.compat.access</name>
<value>true</value>
</property>
<property>
<name>fs.s3a.access.key</name>
<value>$ACCESS_KEY</value>
</property>
<property>
<name>fs.s3a.secret.key</name>
<value>$SECRET_KEY</value>
</property> 3.Use S3A to experience BOS service and execute command in hadoop environment
hadoop fs -ls s3a://$YOUR_BUCKET_NAME 4.How to modify Hadoop-aws
(1) Download hadoop source code (2) Modify hadoop-2.8.0-src/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ClientFactory.java file. Since BOS has not yet supported Chunked Upload, which is written by default in Java sdk, therefore, this operation of sdk shall be turned off in the code.
// Find the aws client used by s3a, for example:
// Under the createAmazonS3Client function
// AmazonS3 s3 = new AmazonS3Client(credentials, awsConf);
// Add a ban on chunkedencoding for aws client
s3.setS3ClientOptions(new S3ClientOptions().withChunkedEncodingDisabled(true)); (3) Compile to get a jar package
cd hadoop-2.8.0-src/hadoop-tools/hadoop-aws
mvn package CloudBerry Explorer for Amazon S3
CloudBerry Explorer for Amazon S3 is S3 graphical data management software provided by CloudBerry Lab. It owns powerful graphical interface and supports management operations such as uploading, downloading, synchronizing and deleting data from local to cloud.
Now Baidu AI Cloud also supports managing BOS resources through CloudBerry Explorer.
1.Select the software version that matches your operating system, download and install the software.
Note: The software supports both free and paid versions. You can use the free version directly.
2.Configure cloud storage after installation and select "S3 Compatible":
3.Add a S3 Compatible account, in which the display name can be filled in BOS, and the service point, i.e. the service domain name of BOS, access key and password Key shall be filled in AK/SK of Baidu AI Cloud. AWS S3 compatible service domain names do not support HTTPS for the time being. Please remove the "Use SSL" check and select 4 for Signature version.
4.Click "Test Connection" to test connectivity. The display of "Connection success" indicates successful connectivity.
5.CloudBerry Explorer can be used for various data management below.