Overview

Files in BOS often undergo lifecycle-related operations, such as archiving, sinking, and deletion. Typically, files are frequently accessed shortly after creation. Over time, their access frequency decreases, eventually designating them as "cold files" that are infrequently accessed and ultimately deleted. Manually maintaining the data lifecycle is time-consuming and labor-intensive, whereas neglecting it results in permanent storage in standard storage, leading to high costs. To address this, BOS offers a lifecycle management feature that automates data lifecycle processes, from creation to archiving to deletion, saving users both time and storage expenses.

File lifecycle management method

• To make data management more user-friendly and intelligent, in addition to the Basic Lifecycle Management feature, BOS now supports Intelligent Tiering. This feature not only sinks files to appropriate tiers based on access frequency but also intelligently transitions files between "hot" and "cold" states according to their access patterns. It can automatically upgrade files to Standard Storage or Infrequent Access Storage, providing an optimal balance between storage costs and performance.
• In Basic Lifecycle Management, beyond the ability to customize rules using prefixes, it now supports features such as setting object tags (Tag), directory exclusion (Not), file size filtering (Filesize), and lifecycle management for object versions, allowing for more refined control over the lifecycle process.

Differences between intelligent tiering and basic lifecycle management

Function description

A bucket can only be configured for either Basic Lifecycle Management or Intelligent Tiering at a time; the two cannot be enabled simultaneously.

Intelligent tiering

• When Intelligent Tiering is enabled for a bucket, the system periodically monitors data access. If there is no access for an extended time, the data is moved to a lower-cost storage tier. If the data is accessed later, it will be returned to a high-frequency access tier to ensure optimal read performance.
• It supports files to be transferred from standard storage - infrequent access storage - cold storage or cold storage - infrequent access storage - standard storage;
• It can configure regular deletion of files and fragments (unfinished file fragments in multipart upload tasks), and supports deleting them after a specified number of fixed days. The starting time in the rule is calculated based on the creation time of the object. Deletion has higher priority than conversion, and deletion will be executed when it expires;
• After configuring the rules, you can monitor the volume of sinking and warming data for a single bucket via the bucket data analysis feature on the console.

Notes on intelligent tiering

• Do not support conversion to archive storage, standard storage - multi-AZ, infrequent access storage - multi-AZ;
• A bucket can only have one rule configured at a time.

Basic lifecycle

Basic Lifecycle Management supports the following functions

• Custom time to change storage classes, from standard storage to infrequent access storage, cold storage, archive storage; or from standard storage - multi-AZ to infrequent access storage - multi-AZ;
• Automatically delete data that is no longer needed at a scheduled time;
• Clear expired data from the three-step upload process.

Divided by scenarios, Basic Lifecycle Management supports two modes

• Automatic archiving of data after reaching a certain age: For example, define that all data created more than 30 days ago is automatically converted to infrequent access storage with lower storage costs;
• Automatically delete data when it reaches a certain age. For example, define that all data created more than 30 days ago is automatically deleted.

Divided by rule configuration, Basic Lifecycle Management supports simple and complex modes

• Simple mode: The defined rules only include the setting of the scope of effective files, which only configures the prefix or all files in the bucket (excluding multi-version objects);
• Complex mode: The rules include defining the effective scope of files, such as including files with specific object tags, excluding certain files, specifying file sizes, and managing the lifecycle of historical object versions.

Functions supported by complex lifecycle rule management

• Supports setting object tags and allows specifying tags to filter and delete objects.
• Supports excluding specific directories, filtering objects outside the directory, and specifying object tags.
• Supports defining file size thresholds and filtering objects for deletion within the specified size range.
• Set multi-version lifecycle rules, which support the deletion of historical versions.

Special reminder

The differences in execution logic between complex and simple lifecycle management modes for buckets are reflected in how files are converted and deleted.

• Simple mode: For conflicting simple rules, the “refined execution” logic is applied. For example: Rule 1 - Files in the a/b/ directory will be deleted 180 days after creation; Rule 2 - Files in the a/b/c/ directory will be deleted 365 days after creation. In this case, the file a/b/c/1.txt will be deleted 365 days after creation, while a/b/2.txt will be deleted 180 days after creation.
• Complex mode: For conflicting complex rules, the “global optimal execution” logic is applied, giving deletion higher priority. For example: Rule 1 - Files in the a/b/ directory (excluding historical versions) will be deleted 180 days after creation; Rule 2 - Files in the a/b/ directory will be deleted 365 days after creation but excludes the a/b/test/ directory. In this scenario, the file a/b/c/1.txt will be deleted 180 days after creation, and the file a/b/test/2.txt will also be deleted 180 days after creation.
• Special reminder: If a bucket has a complex rule created, the bucket lifecycle management will overall apply the complex rule execution

Notes on basic lifecycle

General notes

• Each bucket can have up to 1,000 rules;
• BOS lifecycle rules will take effect within one day after being set;
• After the rules take effect, BOS will process eligible objects accordingly, but the processing requires some time, so the results may not be immediately visible. Under normal circumstances, the time for sinking or deletion is a few hours, but if the amount of sinking data is large, it may take several days or even longer to complete the sinking or deletion;
• The time calculated in the rules (i.e., the "age" of an object) is based on the object's creation time, not the creation/modification time of the lifecycle rule;
• BOS only stores the last modified time of files, i.e., the last-modified time. If Meta is not updated or files are not overwritten, the last-modified time will be the creation time. Therefore, the "creation time" in the lifecycle is actually the last-modified time.
• Policies based on file access time are currently supported only in the Beijing and Suzhou regions.
• The minimum storage period for infrequent access, cold storage, and archive storage is 30, 60, and 180 days respectively. The lifecycle sinking/deletion rules you configure must adhere to these minimum storage durations. If your configuration is below the minimum duration, the console will prompt you with an error. Please adjust accordingly as per the prompt.
• Files of single AZ type can only be sunk to single AZ type files, and cannot be sunk to multi-AZ type files; Standard storage - multi-AZ type files can only be sunk to infrequent access storage - multi-AZ type files.
• When lifecycle sinking occurs, restoring the original storage type will incur restoration fees. Additionally, after sinking, the original file type will be deleted, which will result in a request fee. For instance, if a file from the infrequent access storage class is moved to cold storage, BOS will first retrieve the file from infrequent access storage, incurring a restoration fee. Once successfully moved to cold storage, the original file will be deleted, incurring a delete request fee that is part of the write request fees.

Notes on complex lifecycle rules

Notes on complex lifecycle rules

• In the multi-version lifecycle, the effective date for historical versions is based on when they became historical, not the modification time of their metadata.
• Multi-version lifecycle does not support archive types.
• Tag rule settings: Each rule can define up to 10 object tags, with each key being ≤ 128 characters and each value being ≤ 256 characters long.
• Each rule has only one excluded directory. When the Not option is enabled, at least one of the prefix and tag must exist, that is, set both prefix and tag, or set only prefix or tag. There is only one prefix or tag in Not; the excluded prefix must be under the set prefix and have the same structure as the prefix. For example, if the prefix is bucket/prefix, the not prefix can be bucket/prefix1
• Specify the minimum file size for the lifecycle rule to apply. The minimum file size must be ≥ 0 bytes and < 5 TB.
• Specify the maximum file size for the lifecycle rule to apply. The maximum file size must be ≥ 0 bytes and < 5 TB.
• Basic lifecycle semantics and complex lifecycle semantics differ in certain ways. Basic lifecycle semantics follow the "longest prefix" principle—if a longest prefix match is found, no further rules are checked. Complex lifecycle semantics, however, aim for the "global optimal" rule, applying the earliest executable rule based on the combination of prefix, exclusions, file size, and tags.

Configuration method

• To manage the lifecycle through the console, please refer to Manage Lifecycle.

• Managing lifecycle through API:

  • PutBucketlifecycle
  • GetBucketLifeCycle
  • DeleteBucketLifeCycle

The following section focuses on the syntax of rule files for managing lifecycles through the API.

Rule definition

The lifecycle management service offered by BOS is executed by establishing rules and assigning actions to those rules.

Note:

• BOS basic lifecycle is executed on a daily basis;
• BOS's basic lifecycle rules are carried out within one day of being set. However, if conditions for execution are not met immediately, it could take up to two days in the worst-case scenario.
• Once the rules become effective, BOS processes the eligible objects as specified. However, this processing takes some time (typically several hours), so the results may not be immediately visible after the rules are configured.
• The time used in the rules (such as the "age" of an object) is based on the object's creation time, not the time when the lifecycle rule was created or modified.
• BOS only records the last modified time of files, which doubles as the creation time if metadata is not updated or files are not overwritten. Hence, the "creation time" referred to in the lifecycle actually represents the last-modified time.

Rule definition follows the following principles:

• Rules are defined based on buckets;
• Basic lifecycle management can have up to 1,000 rules per bucket, and intelligent tiering only supports one configuration, and they cannot be configured at the same time;

The following is an example of a basic lifecycle rule, which is attached to a bucket named samplebucket and contains 3 rule items:

• Delete all files under prefix1 after September 7, 2016;
• Automatically transfer files under prefix2 that have been created for more than 30 days to infrequent access storage;

• Automatically delete incomplete three-step uploads under the prefix3 directory that have remained unfinished for more than 5 days.

  Plain Text Copy

  1 {
  2"rule": [
  3    {
  4        "id": "sample-rule-delete-prefix",
  5        "status": "enabled",
  6        "resource": [
  7            "samplebucket/prefix1/*"
  8        ],
  9        "condition": {
  10            "time": {
  11                "dateGreaterThan": "2016-09-07T00:00:00Z"
  12            }
  13        },
  14        "action": {
  15            "name": "DeleteObject"
  16        }
  17    },
  18    {
  19        "id": "sample-rule-transition-prefix",
  20        "status": "enabled",
  21        "resource": [
  22            "samplebucket/prefix2/*"
  23        ],
  24        "condition": {
  25            "time": {
  26                "dateGreaterThan": "$(lastModified)+P30D"
  27            }
  28        },
  29        "action": {
  30            "name": "Transition",
  31            "storageClass": "STANDARD_IA"
  32        }
  33    },
  34    {
  35        "id": "sample-rule-abort-multiupload-prefix",
  36        "status": "enabled",
  37        "resource": [
  38            "samplebucket/prefix3/*"
  39        ],
  40        "condition": {
  41            "time": {
  42                "dateGreaterThan": "$(lastModified)+P5D"
  43            }
  44        },
  45        "action": {
  46            "name": "AbortMultipartUpload"
  47        }
  48    }
  49]
  50}

The following is a configuration rule for complex lifecycle, including two configurations

• The latest version includes objects within the prefix/ directory but not within the prefix/prefix1/ directory, that have tags 'key1':'value1' or 'key2':'value2', do not have the tag 'key3':'value3', are sized between 1,024 bytes and 2,048 bytes, and have not been modified for over 3 days. These objects will be deleted. Additionally, if the latest version of objects in this directory only has a deletion marker, they will be cleaned in the next lifecycle round.
• Historical versions, which are objects across the entire directory with tags 'key1':'value1' or 'key2':'value2' and have existed as historical versions for more than 5 days, will be deleted.

1{
2  "rule": [{
3  "id": "rule-id",
4  "status": "enabled",
5  "ExpiredObjectDeleteMarker" :"true",
6  "resource": [
7      "bucket/prefix/*"
8  ],
9  "condition": {
10      "time": {
11          "dateGreaterThan": "$(lastModified)+P3D"
12      },
13      "objectSize":{
14          "minSize":1024,
15          "maxSize":2048
16      },
17      "tag": {
18          "key1":"value1",
19          "key2":"value2"
20      }
21  },
22  "action": {
23      "name": "DeleteObject"
24  },
25 "not":{ // Each rule contains only one not prefix+ not tag, where the prefix and tag must be under the same item
26      "resource": "bucket/prefix/prefix1*",
27      "tag": {"key3":"value3"}
28  }
29 },{
30  "id": "rule-id2",
31  "status": "enabled",
32  "resource": [
33      "bucket/*"
34  ],
35  "condition": {
36      "time": {
37          "dateGreaterThan": "$(lastModified)+P5D"
38      },
39      "tag": {
40          "key1":"value1",
41          "key2":"value2"
42      }
43  },
44  "action": {
45      "name": "NonCurrentVersionDeleteObject"
46  }
47  
48 }]
49}

Each rule includes an ID, status, resource, condition, and action. For detailed explanations, please refer to the table below.

action

BOS supports five operations: Transition, DeleteObject, AbortMultipartUpload, NonCurrentVersionDeleteObject, and NonCurrentVersionTransition. The detailed explanations of the five actions are as follows:

dateGreaterThan

BOS supports two forms of dateGreaterThan settings: absolute time date and relative time days. The notes are as follows:

Description:

• Each rule is limited to a single action.
• Each resource may have only one rule per action. For instance, bucket/prefix1/* can include a single transition rule and a single expiration rule, but not two transition rules.
• If the prefixes defined in the rules overlap, the action aligned with the smallest scope of the included prefix will take precedence. For example, Rule 1 specifies that objects with prefix p1/p1/ will be deleted after 10 days, while Rule 2 specifies that objects with prefix p1/p1/p1/ will be deleted after 15 days. Since Rule 1's prefix encompasses Rule 2's prefix, objects under p1/p1/p1/ will be deleted after 15 days, and objects only under p1/p1/ will be deleted after 10 days.
• When a date is used as the Condition, such as converting files in “folder/” to infrequent access after 30 days and deleting them on August 17, 2016, the system will compute the timing for each operation based on the rules and compare them. If a file's transition and expiration fall on the same day, it will be directly deleted; similarly, if the deletion date occurs prior to the transition date, it will be directly deleted as well.
• It’s important to distinguish complex lifecycle rules from basic ones. For overlapping prefixes, the minimum time principle applies. For example, Rule 1 deletes objects with prefix p1/p1/ after 10 days, and Rule 2 deletes objects with prefix p1/p1/p1/ after 13 days. Rule 1 is executed first. To ensure Rule 2 deletes after 13 days, Rule 1 should exclude p1/p1/p1/ by adding it as an exception.