Skip navigation

BenchCouncil: International Open Benchmark Council

 

BenchCouncil's Mission

International Open Benchmark Council (BenchCouncil) is a non-profit research institute which aims to promote the standardization, benchmarking, evaluation, incubation, and promotion of open-source chip, AI, and Big Data techniques.


The current BenchCouncil activities include:

  • Establish and maintain a repository of benchmark specifications for quantitative system and algorithm evaluation and analysis.

  • Review, shepherd, and release open-source benchmark implementations.

  • Set up testbeds for innovative open-source chip, AI, and Big Data techniques.

  • Organize conferences, workshops, and teleconferences fostering the transfer of knowledge between industry and academia in the areas of benchmarking.

  • Organize challenges using released benchmarks.

BenchCouncil organizes BenchCouncil main conference (Bench), BenchCouncil International Symposium on Intelligent Computers, and BenchCouncil International Symposium on Chips (Chips 2019).


BenchCouncil's Benchmarking Procedure

Each benchmarking pipeline consists of the following six steps: benchmark proposal, formation of working group, publish benchmark specifications, open-source implementations of benchmark specifications, challenges, and archive performance numbers.

  • Benchmark proposal. BenchCouncil encourages any company, institute, organization or individual to submit a benchmark proposal. Once a proposal is accepted, the authors will be invited to give a talk at BenchCouncil main conferences. On the same theme, different groups are encouraged to submit their proposals.

  • Formation of working groups. For each benchmark, a working group will be formed. Two leaders will be designated, one from industry and another from academia. The working group has the responsibilities to seek funding to support the research and development of benchmarks. Generally, a series of discussions will be held at BenchCouncil main conferences, follow-up workshops and teleconference.

  • Publish benchmark specifications. BenchCouncil encourages publishing benchmark specification.

  • Open-source implementations of benchmark specifications. BenchCouncil encourages anyone to contribute to the open-source implementations of benchmarks specifications.

  • Challenges. BenchCouncil will organize challenges using the released benchmarks.

  • Archive performance numbers. The performance numbers will be archived on the BenchCouncil web site.