Each line in the plot represents the performance profile of a single submission, summarizing its performance across all tested workloads and comparing it to the best submission for each workload. The final benchmark score is the integrated performance profile, i.e. the area under the curve, with higher benchmark scores being better. A step in the performance profile occurs at 𝜏 if, for one workload, the submission achieves the target performance within 𝜏 times the fastest submission’s time. For example, a step at 𝜏=2.0 indicates that, for one workload, this submission takes exactly twice as long to reach the target performance as the workload’s fastest submission.

Fractions of the submission times with respect to the workload budgets. Submissions that did not reach the target on a workload have a fraction of ‘inf’ in the table. Submissions for which the submission code raised an error before any evaluations are marked with ‘*’.

Benchmark Workloads

To test that submitted training algorithms can train a variety of deep learning models, the benchmark contains eight fixed workloads. Submissions are scored based on their time-to-results across all eight workloads. Each workload consists of a dataset, model, loss function, and performance target. Submissions cannot modify any part of the workloads and must train all workloads without any additional workload-specific settings (see the Ruleset section). The following table summarizes the workloads in this iteration of the benchmark.

Summary of the fixed base workloads used in the AlgoPerf benchmark. The possible losses are the cross-entropy loss (CE), the mean absolute error (L1), and the Connectionist Temporal Classification loss (CTC). The evaluation metrics additionally include the structural similarity index measure (SSIM), the error rate (ER), the word error rate (WER), the mean average precision (mAP), and the bilingual evaluation understudy score (BLEU).

Ruleset

To ensure generally useful training algorithms, submissions must automate and account for any required workload-specific tuning. Submissions to the benchmark compete under two separate tuning rulesets:

1. The external tuning ruleset simulates hyperparameter tuning with a fixed amount of parallel resources. Multiple hyperparameter settings can be tried in parallel, and training stops once one such setting reaches the desired target performance.
2. In contrast, the self-tuning ruleset simulates tuning on a single machine, where all workload-specific tuning must be performed within the training time of a single run.

Both rulesets have individual leaderboards and winners.

Standardized Benchmarking System

The AlgoPerf: Training Algorithms benchmark uses fixed hardware to ensure that submitters innovate on the training algorithm part of the ML pipeline. For this iteration of the benchmark, all results are gathered on a standardized benchmarking hardware system with 8x V100 GPUs (16GB of VRAM each), 240GB in RAM, and 2TB in storage. Standardizing the hardware and software stack allows fair comparison of wall-clock training times across all submissions. All results were computed by the competition organizers.