Target scenario

AWS Lambda is a computing service that can run code without pre-configuring or managing the server. You can configure AWS Lambda's triggers to execute your uploaded function code. Therefore, we can use AWS Lambda to achieve real-time synchronization of the Objects uploaded by the user to the S3 Bucket to the BOS Bucket.

Main steps

1 Logging in the AWS console, and entering the Lambda Service Console to click “Create Function”.

2 Creating a function from scratch. In this example, selecting Go 1.x as the common language runtime (the subsequent example lambda program is also a Go program, and the user can choose the language he is good at to implement the lambda function). The character select is to create a new character from the template.

3 Configuring Lambda function Desiger, function code, concurrency, memory allocation, timeout, environment variables, etc.. Configuring as needed.

• Basic configuration: Configuring as needed, such as the memory size, timeouts, and environment variables required for your code to execute.

Note:

1. It is recommended to fill in the timeout time of more than 1 minute. The Lambda function downloads the file from S3 and uploads it to BOS. The time taken is related to the size of the Object, and sometimes it would be long, so setting the time according to actual situation.
2. Lambda function saves Object to disk, only the / tmp / directory can be selected, other directories have no write permission;
3. The memory configuration is related to the memory size required when the function runs, and it can be set according to actual needs.

• Uploading lambda code: The uploaded code must be in zip format, and the handler must fill in the file name of the Go file.

• Configuring trigger: We choose the S3 trigger, and further select the S3 bucket that needs to be monitored. The scope of the monitoring event can also be narrowed by using the object prefix and suffix. After adding, you will see the trigger content is: ObjectCreated

4 Saving the configuration. Each time the user uploads the object to the S3 bucket, the uploaded Go lambda code will be executed to synchronize the object to the BOS; entering CloudWatch console, you can check the log of program executed.

Lambda function code

• Library environment preparation: Referring to AWS S3 Golang SDK and BOS Golang SDK tutorial

  go get -u github.com/aws/aws-sdk-go/...
  go get github.com/aws/aws-lambda-go/lambda
  go get github.com/baidubce/bce-sdk-go

• The code is saved in the s32bos.go file, after it was compiled into a binary file, compressing it into a zip file and upload it, the command is as follows

   GOOS=linux GOARCH=amd64 go build -o s32bos s32bos.go
   zip s32bos.zip s32bos

• s32bos.go code:

   package main
  
   import (
   	"context"
   	"fmt"
   	"os"
   )
  
   import (
   	"github.com/aws/aws-lambda-go/events"
   	"github.com/aws/aws-lambda-go/lambda"
   	"github.com/aws/aws-sdk-go/aws"
   	"github.com/aws/aws-sdk-go/aws/credentials"
   	"github.com/aws/aws-sdk-go/aws/session"
   	"github.com/aws/aws-sdk-go/service/s3"
   	"github.com/aws/aws-sdk-go/service/s3/s3manager"
   	"github.com/baidubce/bce-sdk-go/services/bos"
   )
  
   var (
   	S3_REGION = "ap-northeast-1"
   	S3_AK     = "AKIAJ*****LWGRYAULYTA"
   	S3_SK     = "ABZrI****RqhrE8VzvuESvQMWXcsOPiMJ7"
  
   	BOS_AK       = "388e4e0b*****cfff3cbecb5"
   	BOS_SK       = "2db53e648c53456e***5b7d378"
   	BOS_ENDPOINT = "http://bj.bcebos.com"
   	BOS_BUCKET   = "bucket-name"
   )
  
   func downloadFromS3(bucket string, object string) {
   	file, err := os.Create("/tmp/" + object)
   	if err != nil {
   		fmt.Printf("Unable to open file, err %v", err)
   	}
   	defer file.Close()
   	// init s3 downloader
   	sess, err := session.NewSession(&aws.Config{
   		Region:      aws.String(S3_REGION),
   		Credentials: credentials.NewStaticCredentials(S3_AK, S3_SK, ""),
   	})
   	downloader := s3manager.NewDownloader(sess)
  
   	numBytes, err := downloader.Download(file,
   		&s3.GetObjectInput{
   			Bucket: aws.String(bucket),
   			Key:    aws.String(object),
   		})
   	if err != nil {
   		fmt.Printf("Unable to download object %s, size %d, err %v", object, numBytes, err)
   	}
   	fmt.Println("download from s3 success")
   }
  
   func upload2Bos(bucket string, object string) {
   	bosClient, err := bos.NewClient(BOS_AK, BOS_SK, BOS_ENDPOINT)
   	if err != nil {
   		fmt.Printf("create bos client err %v", err)
   		return
   	}
  
   	etag, err := bosClient.PutObjectFromFile(bucket, object, "/tmp/"+object, nil)
   	if err != nil {
   		fmt.Printf("put object to bos err %v", err)
   		return
   	}
   	fmt.Printf("upload2Bos success etag: %s", etag)
   }
  
   func HandleLambdaEvent(ctx context.Context, s3Event events.S3Event) {
   	fmt.Println("start to handle lambda event")
   	for _, record := range s3Event.Records {
   		s3 := record.S3
   		fmt.Printf("[%s - %s] Bucket = %s, Object = %s \n", record.EventSource, record.EventTime, s3.Bucket.Name, s3.Object.Key)
  
   		downloadFromS3(s3.Bucket.Name, s3.Object.Key)
   		upload2Bos(BOS_BUCKET, s3.Object.Key)
  
   	}
  
   	fmt.Println("handle lambda event success")
   }
  
   func main() {
   	lambda.Start(HandleLambdaEvent)
   }

Reference file

1. Lambda Function Handler (Go)
2. AWS SDK for Go Developer Guide
3. BOS Golang SDK
4. aws-lambda-go