Data management includes data import, export, and data cleaning.

Data Import

Import Demonstration Data

The demonstration data helps users experience various functions of the time series data. The demonstration data contains meteorological simulation data (non-real data) such as temperature, humidity, wind force, PM2.5, and ultraviolet index collected at 8 a.m. each day on air monitoring stations in Beijing, Shanghai and Guangzhou in 2015, the names and zip codes of each city to which each data point belongs, the monitoring station's geographical location (latitude and longitude) and other tags, to help you quickly experience and understand various functions of query, aggregation and chart display of the TSDB.

1. Select Product Service > Time Series Database (TSDB) > Database Name, and enter the database details page.

2. Click Import > Import Demonstration Data > Import Data and click OK in the pop-up dialog box to complete the operation of importing demonstration data. You can view the corresponding import record below after importing.

   

For an operation example of the demonstration database, see Generate Demonstration Database Chart.

Import Data

Users can re-import data from the previous Export to BOS to TSDB, or import local data. If you import owned data from a local path, you need to upload the data file to BOS in the file format .CSV.

Note:

Different fields of the same metric are not required to be completely imported in a same import task, but can be imported multiple times. As long as the imported metric, timestamp and tag value are the same, you can simultaneously query multiple fields through multiple imports in one query.

The user's own data must contain "metric," "timestamp," at least one "field" and at least one "tag". Wherein:

• The first column header must be "metric", representing the metric of the data point.
• The second column header must be "timestamp", representing the timestamp of the data point, with units of a total of milliseconds from Jan 1, 1970 00:00:00 GMT (Jan 1, 1970, 08:00:00 Beijing time) to the present.

• The third column and the columns afterwards:

  If the header contains a colon":", it means that this column is field of a data point, the colon is preceded by the field name, and the colon is followed by the field type (supporting Number, String and Bytes, where the values in a column of the Bytes type need to be encoded by base64). For example, "value:Number" in the following image.

  If the expression does not contain a colon, it indicates that this column is a tag of the data point, with the header as the tag name and the contents of the column as the tag value. The red box below contains tag names.

1. Select Product Service > Time Series Database (TSDB) > Database Name, and enter the database details page.
2. Click Import > Import Backup Data > Import Data, select the specified Bucket name and file in the pop-up dialog box and set an encoding format, and click OK to complete the operation of importing demonstration data.

Data Export

TSDB provides two methods as manual and automatic data export.

Manual Data Export

The export function allows you to export current data to BOS Bucket. BOS (Baidu Object Storage) is Baidu object storage that provides users with stable, secure, efficient, and highly scalable storage services. Bucket is a namespace in BOS.

Before exporting data, users should Enable Object Storage Service and Create Bucket.

The specific operation steps are as follows:

1. Select Product Service > Time Series Database TSDB, and enter the Database List page.
2. Click database name, and enter the database details page; click Import/Export > Export > Manual Data Export tab, and enter the export operating page.

3. Click Create Export and configure the following information in the pop-up window, including:

   • Export address: BOS Bucket address where exported files are stored. If a BOS Bucket has been created, you can select the specified Bucket name in the drop-down list of Export Address, or if a BOS Bucket has not been created, you can quickly create a Bucket by clicking Create Bucket.
   • Time range: Export all data or only export data within a specified time range.
   • Metric: Export data for all metrics or only data for specified metrics.
   • Export file format: Only CSV format is currently supported.
   • Number of export files: Support exporting all data into one file (because of the bos file size limit, a large amount of data is split into multiple files) or exporting the data to multiple files in parallel (suitable for big data export).

   Click OK when the configuration is done, to complete the operation of data export.

4. Once the export operation is complete, the user can go back to the export operating page to view the export results. Click View Details to get the export details and the corresponding BOS Bucket address.

Automatic Data Export

When auto-export function is enabled, the data within a time range from the previous day to 00:00 of the present day is automatically exported in the early morning of each day, and the specific operations are as follows:

1. Select Product Service > Time Series Database TSDB, and enter the Database List page.
2. Click database name, and enter the database details page; click Import/Export > Export > Automatic Data Export tab, and enter the export operating page.

3. Enable the Auto-Timed Export switch and set Export Address and Export File Number.

   • Export address: BOS Bucket address where exported files are stored. If a BOS Bucket has been created, you can select the specified Bucket name in the drop-down list of Export Address, or if a BOS Bucket has not been created, you can quickly create a Bucket by clicking Create Bucket.
   • Number of export files: Support exporting all data into one file (because of the bos file size limit, a large amount of data is split into multiple files) or exporting the data to multiple files in parallel (suitable for big data export).
4. Click Save for the configuration to take effect.

Data Cleaning

Note: Data cleaning is mainly used for removing debug test data. Do it on formal data with caution.

Select Product Service > Time Series Database TSDB, and enter the Database List page.

Click database name, and enter the database details page; click Data Cleaning tab, and enter the data cleaning page.

The system provides data cleaning information of the last 1 hour by default, and if you need to clean more information, click the drop-down menu on the right to switch the time range.

Data Time Limit

Select Product Service > Time Series Database TSDB, and enter the Database List page.

Click database name, and enter the database details page; click Data Time Limit tab, and enter the data time limit page.

Set the data time limit to ON, set the time limit range with units as days, and after saving the data within the time limit is stored by the system.

Note:

Enable the data time limit function to automatically delete data outside the time limit range by the system.