Researchers from Intel have introduced a method to enable distributed inference of large language models (LLMs) on fleets of Intel AI PCs. The approach, outlined in a paper titled "Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PC Fleets" on arXiv, addresses the memory limitations of individual AI PCs by distributing a single LLM across several devices. This allows a collection of AI PCs to collectively run models, such as 70-billion-parameter LLMs, that would otherwise be too large for any one machine's unified memory.
Modern Intel AI PCs feature integrated GPUs and Neural Processing Units (NPUs) with at least 16 GB of unified memory. While capable, this memory is insufficient for directly hosting very large LLMs. The paper proposes using pipeline parallelism, a technique that splits a model by its layers into distinct stages. Each stage is then pre-compiled into an OpenVINO graph, an optimized intermediate representation for inference on Intel hardware. Each participating AI PC runs one of these pre-compiled shards and passes activation data to the next machine in the pipeline.
The researchers highlight three key techniques that make this distributed inference method efficient. First, they focused on recovering the inference speed of an unsplit model. A naive per-stage export of the model would typically perform below monolithic inference due to a missed OpenVINO GPU optimization. By injecting a `beam_idx Gather` operation into each shard, the researchers were able to trigger this optimization, restoring performance.
OpenVINO is Intel's toolkit for optimizing and deploying deep learning models, designed to enhance inference performance across various Intel hardware, including CPUs, GPUs, and NPUs. It supports different parallelization strategies like model parallel, data parallel, pipeline parallel, and tensor parallel, which distribute parts of a large model across multiple computing nodes to accelerate training and inference. Intel has also developed OpenVINO GenAI, a library that extends the OpenVINO runtime for more efficient use with generative AI models.
The concept of distributing LLM inference across multiple devices is a subject of ongoing research. Other work, such as "PipeMax: Enhancing Offline LLM Inference on Commodity GPU Servers," explores pipeline parallelism combined with offloading inactive KV cache to expand effective GPU memory and improve throughput on GPU servers. Intel has also been exploring "On-Device-First Hybrid LLM Inference on AI PCs," which examines how client devices can contribute to scalable enterprise generative AI by handling smaller language models locally and offloading complex tasks to the cloud.
The paper also builds on previous work in optimizing LLM inference on Intel hardware, including efforts to push the envelope of LLM inference on AI-PCs and Intel GPUs through ultra-low-bit quantization and specialized microkernels. These advancements aim to reduce memory requirements and accelerate inference, making LLMs more accessible on resource-constrained environments.
The authors of the arXiv paper include Kunda Xu, Xiuchuan Zhai, Xiaoxia Sun, Tingqian Li, and Wanglei Shen. Their work suggests a practical pathway for enhancing the capabilities of individual AI PCs by enabling them to collaborate on computationally intensive LLM tasks, potentially expanding the reach of advanced AI applications to a broader range of client devices.