The rapid growth of the number of the client side devices brings great challenges to computing power, data storage, and network, considering the variety and quantity of these devices. The traditional way to solve these problems is the cloud computing framework. And with the development of 5G network technology, the Internet of Things(IoT) has been another solution. Moreover, edge computing shows a trend of rapid development these years, which combines cloud computing and IoT framework. In the edge computing scenarios, the distribution of data and collaboration of workloads on different layers are serious concerns for performance, security, and privacy issues. So for edge computing benchmarking, we must take an end-to-end view, considering all three layers: client-side devices, edge computing layer, and cloud servers.

Edge AIBench is a benchmark suite for end-to-end edge computing including four typical application scenarios: ICU Patient Monitor, Surveillance Camera, Smart Home, and Autonomous Vehicle, which consider the complexity of all edge computing AI scenarios. In addition, Edge AIBench provides an end-to-end application benchmarking framework, including train, validate and inference stages. Table 1 shows the component benchmarks of Edge AIBench. Edge AIBench provides an end-to-end application benchmarking, consisting of train, inference, data collection and other parts using a general three-layer edge computing framework.

ICU Patient Monitor. ICU is the treatment place for critical patients. Therefore immediacy is significant for ICU patient monitor scenario to notify doctors of the patients’ status as soon as possible. The dataset we use is MIMIC-III. MIMIC-III provides many kinds of patients data such as vital signs, fluid balance and so on. Moreover, we choose heart failure prediction and endpoint prediction as the AI benchmarks.
Surveillance Camera. There are many surveillance cameras all over the world nowadays, and these cameras will produce a large quantity of video data at all times. If we transmit all of the data to cloud servers, the network transmission bandwidth will be very high. Therefore, this scenario focus on edge data preprocesses and data compression.
Smart Home. Smart home includes a lot of smart home devices such as automatic controller, alarm system, audio equipment and so on. Thus, the uniqueness of the smart home includes different kinds of edge devices and heterogeneous data. We will choose two AI applications as the component benchmarks: speech recognition and face recognition. These two components have heterogeneous data and different collecting devices. These two component benchmarks both collect data on the client side devices (e.g., camera and smartphone), infer on the edge computing layer and train on the cloud server.
Autonomous Vehicle. The uniqueness of the autonomous vehicle scenario is that the high demand for validity. That is to say, it takes absolute correct action even without human intervention. This feature represents the demand of some edge computing AI scenarios. The automatic control system will analyze the current road conditions and make a corresponding reaction at once. We choose the road sign recognition as the component benchmark.

Table 1. The summary of Edge AIBench

AI Application Benchmarks	End-to-end Application Scenarios	Cloud Server	Edge Computing Layer	Client Side Device
Heart Failure Prediction	ICU Patient Monitor	Train	Infer Send Alarm	Generate Data
Endpoint Prediction	ICU Patient Monitor	Train	Infer	Generate Data
Unexpected Respiratory Decompensation Prediction	ICU Patient Monitor	Train	Infer Send Alarm	Generate Data
Person Re-Identification	Surveillance Camera	Decompress Data Train	Compress Data Infer	Generate Data
Lane Keeping	Autonomous Vehicle	Train	Infer	Generate Data
Road Sign Recognition	Autonomous Vehicle	Train	Infer	Generate Data
Speech Recognition	Smart Home	Train	Infer	Generate Data
Face Recognition	Smart Home	Train	Infer	Generate Data

A Federated Learning Framework Testbed. We have developed an edge computing AI testbed to provide support for researchers and common users, which is publicly available from http://www.benchcouncil.org/testbed.html. Security and privacy issues become significant focuses in the age of big data, as well as edge computing. Federated learning is a distributed collaborative machine learning technology whose main target is to preserve the privacy. Our testbed system will combine the federated learning framework.