Component benchmarks
Table 1: The Component Benchmarks Specifications.
| Problem Domain | Models | Dataset | Target Quality | AI Framework | Comm Lib | AI Acc Lib | Epochs |
|---|---|---|---|---|---|---|---|
| Image Classification | ResNet-50 | ImageNet | Top1 Accuracy:0.763 | TensorFlow | Horovod | CUDA, CUDNN, NCCL | 90 |
| Extreme Weather Analytics | Faster-RCNN | The Extreme Weather Dataset | mAP@[IoU=0.5]=0.35 | TensorFlow | Horovod | CUDA, CUDNN, NCCL | 50 |
Problem Domain
Image Classification
Image Classificationis widely used in many applications ofcommercial fields, which is a funda-mental task in AI research. With the developing of large-scale deep earning, Image Classification hasbecome a well-known showcase optimizing HPC AI system.
Extreme Weather Analysis poses a great challenge to human society. It brings severe damage to people health and economy every single year. For instance, the heatwaves in 2018 caused over 1600 deaths according to the UN report. And the landfall of hurricane Florence and Michael caused about 40 billion dollars worth of damage to US economy. In this context, understanding extreme weather life cycle and even predicting its future trend become a significant scientific goal. Achieving this goal always requires accurately identifying the extreme weather patterns to acquire the insight of climate change based on massive climate data analysis. Now, the reference implementation is available, which is the benchmark for BenchCouncil 2019 System award.
Reference Models
ResNet-50
ResNet is a milestone in Image Recognition, marking the ability of AI to identify images beyond humans. It solves the degradation problem, which means in the very deep neural network the gradient will gradually disappear in the process of propagation, leading to poor performance. Due to the idea of ResNet, researchers successfully build a 152-layer deep CNN. This ultra deep model won all the awards in ILSVRC'15.
Faster-RCNN
Faster-RCNN targets real-time object detection. Unlike the previous object detection model (RCNN,fastRCNN), it replaces the selective search by a region proposal network that achieves nearly cost-free region proposals. Further more, Faster-RCNN combines the advanced CNN model as their base network for extracting features and is the foundation of the 1st-place winning entries in ILSVRC'15 (ImageNet Large Scale Visual Recognition Competition)
The Selected Dataset
The Extreme Weather Dataset
The ExtremeWeather Dataset is made up of 26-year of climate data. The data of every year is available as one HDF5 file. Each HDF5 file contains two data sets: images and boxes. Images data set has 1460 example dense images (4 per day, 365 days per year) with 16 channels. Each channel is 768 * 1152 corresponding to one measurement per 25 square km on earth. Boxes dataset records the coordinates of the four extreme weather events in the corresponding images: tropical depression, tropical cyclone, extratropical cyclone and the atmospheric river. This dataset is now avaliable, see The Extreme Weather Dataset.
ImageNet
ImageNet is large visual database designed for use in visual object recognition research.More than 14 million images have been hand-annotated according to the WordNet hierarchy. Both theoriginal images and bounding boxes are provided. The data size is more than 100 GB.Metrics
The main metric of HPC AI500 is VFLOPS, which is defined as the follow:
VFLOPS = FLOPS * (achieved_quality/target_quality)^n
achieved_quality represents the actual model quality achieved in the evaluation; target_quality is the state-of-the-art model quality that has been predefined in HPC AI500 benchmark. The value of n is apositive integer, which is used to define the sensitivity to the model quality.