论文标题
在异质基础架构上服务和优化机器学习工作流程
Serving and Optimizing Machine Learning Workflows on Heterogeneous Infrastructures
论文作者
论文摘要
随着智能设备和物联网无处不在的部署的出现，机器学习推断的数据源已越来越多地转移到网络的边缘。现有的机器学习推理平台通常假设一个均匀的基础架构，并且不考虑包括边缘设备，本地集线器，边缘数据中心和云数据中心的更复杂和分层的计算基础架构。另一方面，最近的自动努力为异质环境提供了可行的解决方案，用于模型压缩，修剪和量化。对于机器学习模型，现在我们很容易找到甚至生成一系列在准确性和效率之间进行不同权衡的模型。我们设计和实施Jellybean，这是一种用于在异构基础架构上服务和优化机器学习推理工作流程的系统。给定的服务级目标（例如，吞吐量，准确性），杰利比（Jellybean）选择了满足准确性目标的最具成本效益的模型，并决定如何在基础架构的不同层次上部署它们。评估表明，与最先进的模型选择和工人分配解决方案相比，Jellybean的视觉问题回答总成本最高可达58％，而NVIDIA AI City Challenge的车辆跟踪最多可达36％。 Jellybean在服务成本上也胜过先前的ML服务系统（例如在云上的火花）。
With the advent of ubiquitous deployment of smart devices and the Internet of Things, data sources for machine learning inference have increasingly moved to the edge of the network. Existing machine learning inference platforms typically assume a homogeneous infrastructure and do not take into account the more complex and tiered computing infrastructure that includes edge devices, local hubs, edge datacenters, and cloud datacenters. On the other hand, recent AutoML efforts have provided viable solutions for model compression, pruning and quantization for heterogeneous environments; for a machine learning model, now we may easily find or even generate a series of models with different tradeoffs between accuracy and efficiency. We design and implement JellyBean, a system for serving and optimizing machine learning inference workflows on heterogeneous infrastructures. Given service-level objectives (e.g., throughput, accuracy), JellyBean picks the most cost-efficient models that meet the accuracy target and decides how to deploy them across different tiers of infrastructures. Evaluations show that JellyBean reduces the total serving cost of visual question answering by up to 58%, and vehicle tracking from the NVIDIA AI City Challenge by up to 36% compared with state-of-the-art model selection and worker assignment solutions. JellyBean also outperforms prior ML serving systems (e.g., Spark on the cloud) up to 5x in serving costs.
