Tutorial: BenchCouncil AIBench
---An Agile Domain-specific Benchmarking Methodology and
an Al Benchmark Suite
News: All the AlBench slide presentations and hands-on tutorial videos are publicly available from Tutorial_Link. The separate link for each talk is also provided in Schedule part, Additionally, the videos are provided to show howto use our benchmarks on a publicly available Testbed.
Open Benchmark Council (BenchCouncil) is a non-profit international benchmark organization, whichaims to promote the standardization, benchmarking, evaluation, incubation, and promotion of open-source chipAl, and Big Data techniques. This tutorial is aimed at presenting AlBench. We are glad to introduce the following interesting topics:
(1) The challenges and motivation for characterizing AI workloads.
(2) Benchmarking methodology, models, and metrics.
(3) AIBench scenario benchmarks
(4) Edge AIBench: towards Comprehensive End-to-end Edge Computing Benchmarking
(5) AIBench training and its rankings
(6) HPC AI500: A benchmark suite and HPC AI ranking for HPC AI systems
(7) AIBench Inference and its rankings
(8) AIoTBench for benchmarking mobile and embedded device Intelligence and its rankings
(9) An open testbed for AI in HPC, Datacenter, IoT, and Edge.
(10) Hands-on demos on how to use AIBench on BenchCouncil testbed.
Location and Date
We will give a tutorial on
AIBench at ASPLOS 2020 in San Diego, CA, USA.
March 17th, 2020 (Saturday),09:00 -
12:00 (Half Day)
ROOM:TBD
Organizers and Presenters
| Organizer: |
Jianfeng Zhan |
ICT, CAS & BenchCouncil (zhanjianfeng@ict.ac.cn) |
| Organizer: |
Wanling Gao |
ICT, CAS & BenchCouncil (gaowanling@ict.ac.cn) |
| Presenter: |
Jianfeng Zhan |
ICT, CAS & BenchCouncil (zhanjianfeng@ict.ac.cn) |
| Presenter: |
Lei Wang |
ICT, CAS & BenchCouncil (wanglei_2011@ict.ac.cn) |
| Presenter: |
Wanling Gao |
ICT, CAS & BenchCouncil (gaowanling@ict.ac.cn) |
| Presenter: |
Chunjie Luo |
ICT, Chinese Academy of Sciences and University of Chinese Academy of Sciences (zhengchen@ict.ac.cn) |
| Presenter: |
Tianshu Hao |
ICT, Chinese Academy of Sciences and University of Chinese Academy of Sciences (haotianshu@ict.ac.cn) |
| Presenter: |
Zihan Jiang |
ICT, Chinese Academy of Sciences and University of Chinese Academy of Sciences (jiangzihan@ict.ac.cn) |
| Presenter: |
Fei Tang |
ICT, Chinese Academy of Sciences and University of Chinese Academy of Sciences (tangfei@ict.ac.cn) |
Abstract
Modern real-world application scenarios like Internet services not only consist of diversity of AI and non-AI modules with very long and complex execution paths, but also have huge code size, which raises serious benchmarking or evaluating challenges. Using AI components or micro benchmarks alone can lead to error-prone conclusions. Together with seventeen industry partners, we extract nine typical application scenarios, and identify the primary components. As the proxy to real-world applications, the AIBench scenario benchmarks let the software and hardware designers obtain the overall system performance and find out the key components within the critical path. Following the same methodology, we propose Edge AIBench for benchmarking end-to-end performance across IoT, edge and Datacenter.
Earlier-stage evaluations of a new AI architecture/system need affordable AI training benchmarks, while using a few AI component benchmarks alone in the other stages may lead to misleading conclusions. We present a balanced AI benchmarking methodology for meeting the conflicting requirements of different stages. We identify and implement seventeen representative AI tasks with the state-of-the-art models to guarantee the diversity and representativeness of the benchmarks. Meanwhile, we keep a benchmark subset to a minimum for affordability. Furthermore, on the basis of the AIBench training subset, we present the HPC AI500 benchmarks for evaluating HPC AI systems for both affordability and representativeness. For AI Inference, as its cost is trivial, we provide comprehensive AI inference benchmarks. Meanwhile, we propose AIoTBench for considering diverse light-weight AI frameworks and models.
Biographies
Jianfeng Zhan
Prof. Jianfeng Zhan is a Full Professor at Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS), and University of Chinese Academy of Sciences (UCAS), and director of the Software Systems Labs, ICT, CAS. He received his B.E. in Civil Engineering and MSc in Solid Mechanics from Southwest Jiaotong University in 1996, and 1999, and his Ph.D. in Computer Science from Institute of Software, CAS and UCAS in 2002. He has supervised over 90 graduate students, post-docs, and engineers in the past two decades. His research areas span from Chips, Systems, to Benchmarks. A common thread is benchmarking, designing, and implementing, and optimizing parallel and distributing systems. He has made strong and effective efforts to transfer his academic research into advanced technology to impact general-purpose production systems. Several technical innovations and research results, including 36 patents, from his team have been widely adopted in benchmarks, operating systems and cluster and cloud system software with direct contributions to the advancement of the parallel and distributed systems in China or even in the world. Prof. Jianfeng Zhan founds and chairs BenchCouncil. He served as IEEE TPDS Associate Editor since 2018. He received the second-class Chinese National Technology Promotion Prize in 2006, the Distinguished Achievement Award of the Chinese Academy of Sciences in 2005, and IISWC Best paper award in 2013, respectively.
Lei Wang
Dr. Lei Wang received the Ph. D degree in computer engineering from University of Chinese Academy of Sciences, Beijing, China, in 2016. He is currently a senior engineer with the Institute of Computing Technology, Chinese Academy of Sciences. His current research interests include datacenter software systems, workload characterization and benchmarking. He was a recipient of the Distinguished Achievement Award of the Chinese Academy of Sciences in 2005.
Wanling Gao
Dr. Wanling Gao is an Assistant Professor in computer science at the Institute of Computing Technology, Chinese Academy of Sciences and University of Chinese Academy of Sciences. Her research interests focus on big data and AI benchmarking, computer architecture, and workload characterization. She received her B.S. degree in 2012 from Huazhong University of Science and Technology and her PhD degree in 2019 from Institute of Computing Technology, Chinese Academy of Sciences and University of Chinese Academy of Sciences in China.
Chunjie Luo
Chunjie Luo is an Engineer and PHD candidate of the Institute of Computing Technology, Chinese Academy of Sciences and University of Chinese Academy of Sciences. His research interests focus on machine learning and benchmarking. He received his B.S. degree in 2009 from Huazhong University of Science and Technology and his master degree in 2012 from Institute of Computing Technology, Chinese Academy of Sciences and University of Chinese Academy of Sciences in China.
Tianshu Hao
Tianshu Hao received the B.S. degree from Nankai University, Tianjin, China, in 2015. She is currently pursuing Ph. D. degree in ICT, CAS. Her research interests focus on big data, edge computing, IoT and AI benchmarking.
Zihan Jiang
Zihan Jiang is a doctoral student in computer architecture at Institute of Computing Technology, Chinese Academy of Sciences and University of Chinese Academy of Sciences. His research interests include AI benchmark and distributed deep learning. Currently, he works on HPC AI500 project.
Fei Tang
Fei Tang received the B.S. degree from Zhengzhou University, Zhengzhou, China, in 2016. He is currently pursuing Ph. D. degree in ICT, CAS. His research interests focus on big data, benchmarking and search engine.