The Hugging Face Transformers library is widely used by data scientists globally for exploring state-of-the-art NLP models and developing new NLP features. With over 5,000 pre-trained and fine-tuned models available across more than 250 languages, it offers a versatile environment accessible regardless of the framework in use.
While experimenting with models in 🤗 Transformers is straightforward, deploying these large models into production with optimal performance and managing them within a scalable architecture presents a significant engineering challenge for any Machine Learning Engineer.
The substantial performance gains and built-in scalability offered by the hosted Accelerated Inference API attract users seeking to build NLP features. Achieving the final 10x performance boost requires low-level optimizations tailored to specific models and hardware.
This article explores some of the methods used to maximize computational efficiency for users.
Getting to the first 10x speedup
The initial phase of optimization is the most straightforward, focusing on utilizing the most effective techniques available within the Hugging Face libraries, irrespective of the specific hardware.
Efficient methods integrated into Hugging Face model pipelines are utilized to minimize computation during each forward pass. These techniques are tailored to the model’s architecture and task. For example, in text generation with a GPT architecture, attention matrix computation is optimized by focusing on the last token’s new attention in each pass:
Naive version
Optimized version
Tokenization frequently acts as a performance bottleneck during inference. The 🤗 Tokenizers library provides highly efficient methods, utilizing a Rust implementation combined with intelligent caching, which can lead to up to a 10x speedup in overall latency.
By leveraging the latest features of Hugging Face libraries, a consistent 10x speedup can be achieved compared to a standard, unoptimized deployment for a given model and hardware configuration. With monthly updates to Transformers and Tokenizers, users of the API benefit from continuous performance improvements without needing to adapt their deployments.
Compilation FTW: the hard to get 10x
Achieving peak performance becomes more complex, requiring model modification and compilation specifically for the target inference hardware. Hardware selection depends on the model’s memory footprint and the demand profile, including request batching. Different API users might find greater benefits from accelerated CPU or GPU inference, each demanding distinct optimization techniques and libraries.
After selecting the appropriate compute platform for a specific use case, several CPU-specific techniques can be applied with a static graph:
- Optimizing the graph (Removing unused flow)
- Fusing layers (with specific CPU instructions)
- Quantizing the operations
Standard functions from open-source libraries, such as 🤗 Transformers with ONNX Runtime, may not yield optimal results or could lead to substantial accuracy loss, especially during quantization. There is no universal solution, and the ideal approach varies for each model architecture. However, thorough investigation into the Transformers codebase and ONNX Runtime documentation can facilitate achieving an additional 10x speedup.
Unfair advantage
The Transformer architecture marked a pivotal moment for Machine Learning performance, initially in NLP, leading to a rapid and accelerating rate of improvement in Natural Language Understanding and Generation over the past three years. Concurrently, the average model size has grown significantly, from BERT’s 110 million parameters to GPT-3’s 175 billion.
This trend presents considerable challenges for Machine Learning Engineers deploying the latest models into production. A 100x speedup, while ambitious, is often necessary to deliver predictions with acceptable latency in real-time consumer applications.
Achieving such performance is aided by close collaboration with the maintainers of the Hugging Face Transformers and Tokenizers libraries. Additionally, strong partnerships forged through open-source collaborations with hardware and cloud vendors like Intel, NVIDIA, Qualcomm, Amazon, and Microsoft allow for fine-tuning models and infrastructure with the most recent hardware optimization techniques.
