The Second BenchCouncil International Symposium, held in November, 2019, in Denver, CO, USA. To advance the state of the art of AI and other benchmarking, the International Open Benchmark Council (BenchCouncil) has three fundamental responsibilities. First, it releases influential benchmarks and index to prevent from chaotic competitions within AI, computer, finance, education, medical, and other technological industries. Second, it encourages data-driven, intelligence-inspired, and benchmark-based quantitative approaches to tackling multi-disciplinary challenges. Finally but not least, BenchCouncil incubates benchmark projects and hosts the BenchCouncil benchmark projects, and further encourages reliable and reproducible research using the BenchCouncil benchmark projects or incubator benchmark projects.
This year, BenchCouncil released four top level AI benchmarking projects, including AIBench – a scenario-distilling benchmarking methodology and an AI benchmark suite, HPC AI500 – a benchmark suite for HPC AI systems, Edge AIBench – an end-to-end edge AI benchmark Suite, and AIoTBench – an AI benchmark suite for benchmarking mobile and embedded device intelligence. Using AIBench as baseline, BenchCouncil hosted the 2019 AI System and Algorithm Challenge (http://www.benchcouncil.org/competitions.html) and organized the 2019 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench 2019) (http://www.benchcouncil.org/bench19/index.html ). The 2019 AI Challenge consists of four challenge tracks: International AI System Challenge based on RISC-V, International AI System Challenge based on Cambricon Chip, International AI System Challenge based on X86 Platform, and International 3D Face Recognition Algorithm Challenge. The Bench 2019 symposium solicits papers that address hot topic issues in benchmarking, measuring, and optimizing systems. This book includes 31 regular papers from the Bench 2019 conference, which were selected from 79 submissions, yielding an acceptance rate of 39%. The tutorials about AI benchmarks for datacenter (AIBench), edge (Edge AIBench), and HPC (HPC AI500) were presented at the conference, but are not included in this book.
The call for papers for Bench 2019 attracted a number of high-quality submissions. During a rigorous review process, in which each paper was reviewed by at least three experts. In addition, we invited five keynote speakers, including Dr. Dan Stanzione from The University of Texas at Austin; Prof. Dhabaleswar K. (DK) Panda from The Ohio State University; Mr. Gilad Shainer from Mellanox; Prof. Geoffrey Fox from Indiana University; and Prof. Felix Wolf from TU Darmstadt in Germany. Bench 2019 also hosted five invited talks, including Dr. Zheng Cao from Alibaba; Dr. Dong Li from ICT, Chinese Academy of Sciences, and Seaway Technology Co., LTD.; Prof. Bo Wu from Colorado School of Mines; Dr. Weijia Xu from The University of Texas at Austin; and Dr. Gabriel Antoniu from Inria.
Bench 2019 conference invites manuscripts describing original work in the area of benchmarking, evaluation methods and tools in Big Data, Artifical Intelligence, High-Performance Computing and Computing Architectures (Call for Papers ) . All accepted papers will be presented at the Bench 2019 conference and will be published in a special issue of the BenchCouncil Transactions on Benchmarks, Standards and Evaluation (TBench) .
Sponsored by BenchCouncil, Bench 2019 conference will present numerousawards, including the BenchCouncil Achievement Award ($3000), the BenchCouncil Rising Star Award ($1000), the BenchCouncil Distinguished Doctoral Dissertation Award ($1000)，and the BenchCouncil Best Paper Award ($1000). To encourage reliable and reproducible research using the benchmarks from all organizations, the Bench conference presents the BenchCouncil Award for Excellence for Reproducible Research to the papers using publicly available benchmarks. Each article receives a $100 prize, for up to 12 articles.