Create Topic

1. Register and log in to the Baidu AI Cloud platform. For more information on how to operate, please see [Registration](UserGuide/Account Password/Register Account.md# Register Baidu Account) and Login.
2. After your login is successful, select “Product Service -> Baidu Messaging System” and then click the “Topic” to enter the page of the topic list.

3. Click "Create a Topic" and then set up the topic name and the number of partitions:

   • If the entered topic name is "demo", the system is going to create a topic of <accountID>__demo.
   • The number of partitions determines the message throughput that can be processed by this topic. Please select an appropriate number.

   

4. Click "OK" to create a Baidu Messaging System (BMS) topic, which can be used directly.

   

5. Click "View a Certificate" to view/edit the certificate privileges for this topic.

   

6. The “Privilege certificate”, “General certificate” and “Other user’s certificate” can be shown in the certificate privileges for this topic.

   

Privilege certificate: Its privileges are applicable to the self-owned certificate for all topics.

General certificate: Its privileges are applicable to the self-owned certificate for all topics.

Other user's certificate: The user can authorize its topic-related read/write privileges to other user through adding other user’s certificate. After authorization, the authorized user is entitled to read/write the topics of the authorizing user.

Note: The current data storage time of the BMS is 24 hours by default. After the time has expired, the data involved is deleted automatically. If you have any special requirements, please contact us with your ticket.

Create Consumer Group

The consumer group is a scalable and fault-tolerant consumer system provided by Kafka. The consumer group created via the console has a complete security protection system. Users are strongly recommended to use the consumer group feature.

1. Register and log in to the Baidu AI Cloud platform. For more information on how to operate, please see Registration and Login.
2. After your login is successful, select “Product Service -> Baidu Messaging System” and then click the “Consumer Group Management” to enter the list page.
3. Select a region (optional), and then switch to the one as required.

4. Click "Create a Consumer Group", and then enter the consumer group name.

   Note: The consumer group created via the console is automatically prefixed with the Account ID before the name. When using it, you need to enter the complete name of the consumer group.

   

5. The usage privilege of the consumer group needs to be bound to the certificate. Click "View a Certificate" to view/edit the certificate privileges of this consumer group.

   

6. The “Privilege certificate”, “General certificate” and “Other user’s certificate” can be shown in the certificate privileges for this consumer group. When this consumer group is used, the user is going to be verified whether the certificate has the privilege to use this consumer group.

   Note: For the consumer group created via non-console, its privileges do not need to be controlled, but data security problems may be available. Therefore, users are strongly recommended to create a consumer group via the console and use it.

   

Download Certificate

Before connecting to the BMS, the Kafka client needs to provide a certificate for verification and authorization. Please go to "Products Service > Baidu Messaging System - Certificate” from the Baidu AI Cloud platform to download the certificate kafka-key.zip, which is used to verify the identity of the customer and its cloud service.

Notes:

• Please keep your certificate in a safe place. This certificate is used to build the SSL connection and verify the client’s AccountId. Once it is lost or disclosed to others, please go to the Baidu AI Cloud platform to regenerate another one.
• If you use Hong Kong service, please apply to enable Hong Kong service with your ticket first, and then go to the Hong Kong region to download the corresponding certificate.
• The certificates in different regions are independent of each other and cannot be used across regions.
• The certificates in Guangzhou and Beijing can be used across regions due to historical reasons.

Certificate Management

You can view the certificate-related information in the certificate list, such as the name, serial number, type, key and creation time. Additionally, you can select "Regenerate" to update a certificate, select "Delete a Certificate" to delete a certificate; select "View a Topic" to view/edit the certificate privileges for a topic, and select "View a Consumer Group" to view/edit the certificate privileges for a consumer group.

Connection Service

You can connect to the BMS with two methods. The first one is to use the code samples provided by the Baidu AI Cloud. The second one is to download the Kafka binary package from the Kafka official website and then gradually configure the relevant information.

Connect to the Service with the Code Samples

1. Download the code samples. Please log in to the console, open the "Product Service > Baidu Messaging System - Certificate", click the "Code Samples" in the upper right corner of the page, and then download it to the local drive. After that, decompress it.
2. The code samples provided by the BMS are a Maven project package.
3. Run the code samples.cd Go to the code directory of "sample-with-kafka-key" with the command cd and execute the command run.bat <accountID>_demo. After that, the connection to the BMS can be built.

In addition, the code samples in Python and Golang languages we provide for Kafka service have been uploaded to Python Sample and Golang Sample in GitHub. You can clone these code samples to your local drive, and then design your own programs with these samples.

Connect to the Service Through Step-By-Step Configuration

1. Download the Kafka binary package from the Kafka official website at http://kafka.apache.org/downloads.html. Currently, Version 0.10 is supported.
2. Because the users who access the Kafka service through Java need to use pom of the Java client kafka-client/0.10, we recommend that for the users who use JDK 7 or JDK 8, their minimum configuration must be Java 1.7.0-b147.

3. Configure client.properties. Please go to "Product Service > Baidu Messaging System - Certificate” on the Baidu AI Cloud to download the certificate kafka-key.zip, which is used to verify the identity of the customer and its cloud service. Afterwards, please decompress the zip package under the root directory of the kafka binary package, during which client.properties, client.keystore.jks and client.truststore.jks need to be placed in the same directory. After that, open UTF8-encoded client.properties, which is shown as follows:

    security.protocol=SSL
    ssl.truststore.location=client.truststore.jks
    ssl.truststore.password=<your certificate password>
    ssl.keystore.location=client.keystore.jks
    ssl.keystore.password=<your certificate password>

4. The scripts in the client code and the BMS are used for communication, and connection is encrypted with SSL to ensure that data transmission cannot be monitored and tampered with. Take the BMS in Beijing as an example. The detailed operations are as follows (before any operations are made, make sure that client.properties, client.keystore.jks and client.truststore.jks are in the root directory of the Kafka binary package):

   1. Enable a consumer. Open a command line tool and enter the following command:

      sh bin/kafka-console-consumer.sh --topic <accountID>__demo --bootstrap-server kafka.bj.baidubce.com:9091 --consumer.config client.properties --from-beginning --new-consumer

      • For the Windows operation system, the startup command is as follows:

        bin\windows\kafka-console-consumer.bat --topic <accountID>__demo --bootstrap-server kafka.bj.baidubce.com:9091 --consumer.config client.properties --from-beginning --new-consumer

   2. Release a message. Open another command line tool and enter the following command:

      sh bin/kafka-console-producer.sh --producer.config client.properties --topic <accountID>__demo --sync --broker-list kafka.bj.baidubce.com:9091

   • For the Windows operation system, release a common as follows:

     bin\windows\kafka-console-producer.bat --producer.config client.properties --topic <accountID>__demo --broker-list kafka.bj.baidubce.com:9091

   3. After the message is sent, the returned message in the consumer’s command line indicates that the message has been well received.

      Notes:

      • The domain name and port number corresponding to the Beijing region is kafka.bj.baidubce.com:9091.
      • The domain name and port number corresponding to the Guangzhou region is kafka.gz.baidubce.com:9092.
      • The domain name and port number corresponding to the Suzhou region is kafka.su.baidubce.com:9091.
      • The domain name and port number corresponding to the Shanghai region is kafka.fsh.baidubce.com:9091.
      • The domain name and port number corresponding to the Hong Kong region is kafka.hk02.hk.baidubce.com:9092.
      • The domain name and port number corresponding to the Baoding region is kafka.bd.baidubce.com:9071.