Skip navigation

BenchCouncil: International Open Benchmark Council

 

BenchCouncil’s View On Benchmarking AI and Other Emerging Workloads (Technical Report, Slides presented by Prof. Jianfeng Zhan at BenchCouncil SC BoF). This paper outlines BenchCounci’s view on the challenges, rules, and vision of benchmarking modern workloads.

BenchCouncil Top Level Projects

(1) BenchCouncil BigDataBench

Homepage: http://www.benchcouncil.org/BigDataBench/

.

(2) BenchCouncil AIBench

Homepage: http://www.benchcouncil.org/aibench/.

  • AIBench Scenario - proxies to industry-scale real-world applications scenarios.
    • Edge AIBench - an instance of the scenario benchmark suites, modeling end-to-end performance across IoT, edge, and Datacenter.
  • AIBench Training - nineteen representative AI tasks with state-of-the-art models to guarantee diversity and representativeness.
    • HPC AI500 - large-scale HPC AI system benchmarking.
  • AIBench Inference - nineteen representative AI tasks with state-of-the-art models to guarantee diversity and representativeness.
    • AIoTBench - an open-source benchmark suite for Mobile and Embedded device Intelligence.
  • AIBench Micro - intensively-used hotspot functions for simulation-based architecture researches.
  • AIBench Synthetic - complementary to real-world benchmarks, with scalable problem sizes to model learning dynamics.


BenchCouncil Incubator Projects

(1) BenchCouncil BENCHCPU

Homepage: http://www.benchcouncil.org/benchcpu/

.

(2) BenchCouncil WPC

Homepage: http://www.benchcouncil.org/WPC/

.

(3) BenchCouncil FLBench

Homepage: http://www.benchcouncil.org/flbench.html/

.

(4) BenchCouncil EleGeneration

Homepage: http://www.benchcouncil.org/electricity.html

.