Purpose

• Diagnose the failed step, find the failure cause in the log, and find the accurate location of the error in your program.
• Tune the successful steps, and assess the step configuration and the parameter rationality based on experience to give you tuning suggestions.

Applicable Scope

Diagnosis or tuning of Hadoop MR steps, Hadoop Streaming steps, and Spark steps. Diagnosis or tuning of Hive, Pig, and HBase at a later time.

View Diagnosis Information on Failed steps

1. In the cluster step page, click the name of the failed step to view the details of that step.

2. In the step details page, click "Diagnosis" to enter the diagnosis page.

   As the above figure shows, the diagnosis includes the information on user configuration or program error and gives suggestions. If the error is in your program, the diagnosis function can scan the exception stack or error stack in the user’s codes, delete the error information in the framework which does not help you to find the problem, and directly find the incorrect code in your program. View the error file information in bos if you want to know the error further.

View Tuning Information on Successful steps

1. In the cluster step page, click the name of the completed step to view the details of that step.
2. In the step details page, click "Tuning" to enter the tuning page.

The parameters preset in the cluster are specific to the tuning balance for most steps. The optimal parameters vary with the steps. BMR tuning reasonably adjusts the step parameters according to the step information. BMR has more than ten built-in tuning rules, including the balance of Map/Reduce tasks, memory overflow check, and Reduce bucketing uniformity. BMR collects the cluster environment data and step running information when the step is running, and then figures out the score of the tuning rules based on the information collected when the step ends. If the score is higher than the threshold, BMR gives a warning and provides the failure causes and tuning directions. The failure causes generally include the unreasonable setup of step parameters and the unreasonable selection of packages. BMR’s tuning automatically collects the configuration parameters and system parameters set by you, compares the influences of different parameters on step running, and gives the suggestions. For example, during the balance check of Map/Reduce tasks, BMR weighs the volume and time of data processing in every Map/Reduce task, and then suggests the number of Map/Reduce tasks according to Baidu’s experience over the years.

3. View the counter information. You can view different counter information after the step ends. Such information gives you a better understanding of step running conditions and helps you improve the programs.