- Zoomer is a comprehensive, automated debugging and optimization platform for AI.
- The platform operates across various training and inference workloads, offering deep performance insights that enable energy savings, workflow acceleration, and efficiency gains in AI infrastructure.
- Zoomer has achieved reductions in training time and significant QPS improvements, establishing itself as a key tool for AI performance optimization across the entire AI infrastructure.
Operating AI infrastructure at a large scale means that inefficient performance debugging can lead to substantial energy waste, increased operational expenses, and underutilized hardware, particularly across numerous GPUs. The core challenge lies in maximizing computational efficiency while minimizing waste. Even a small percentage point improvement in utilization translates into significant capacity gains that can be redirected towards innovation and growth.
Zoomer functions as an automated, all-in-one platform for profiling, debugging, analyzing, and optimizing AI training and inference workloads. Since its creation, Zoomer has become a primary tool for GPU workload optimization, generating thousands of profiling reports daily for various teams.
Why Debugging Performance Matters
AI infrastructure supports large-scale and advanced workloads across a global fleet of GPU clusters, continuously evolving to meet the growing scale and complexity of generative AI.
At the training level, it supports a diverse range of workloads, including models for ads ranking, content recommendations, and GenAI features.
At the inference level, hundreds of trillions of AI model executions are served daily.
Operating at this scale necessitates prioritizing the elimination of GPU underutilization. Training inefficiencies can delay model iterations and product launches, while inference bottlenecks can limit the ability to serve user requests at scale. Removing resource waste and accelerating workflows helps in training larger models more efficiently, serving more users, and reducing environmental impact.
AI Performance Optimization Using Zoomer
Zoomer is an automated debugging and optimization platform that functions across all AI model types (such as ads recommendations, GenAI, computer vision) and both training and inference paradigms. It provides deep performance insights that enable energy savings, workflow acceleration, and efficiency gains.
Zoomer’s architecture comprises three essential layers that collaborate to deliver comprehensive AI performance insights:
Infrastructure and Platform Layer
This foundation provides the enterprise-grade scalability and reliability required to profile workloads across massive infrastructure. It includes distributed storage systems utilizing
(a blob storage platform) for trace data, fault-tolerant processing pipelines capable of handling huge trace files, and low-latency data collection with automatic profiling triggers across thousands of hosts simultaneously. The platform maintains high availability and scale through redundant processing workers and can manage numerous profiling requests during peak usage periods.
Analytics and Insights Engine
This core intelligence layer offers deep analytical capabilities through multiple specialized analyzers. These include: GPU trace analysis via Kineto integration and NVIDIA DCGM, CPU profiling through StrobeLight integration, host-level metrics analysis via dyno telemetry, communication pattern analysis for distributed training, straggler detection across distributed ranks, memory allocation profiling (including GPU memory snooping), request/response profiling for inference workloads, and more. The engine automatically detects performance anti-patterns and provides actionable recommendations.
Visualization and User Interface Layer
This presentation layer transforms complex performance data into intuitive, actionable insights. It includes interactive timeline visualizations showing GPU activity across thousands of ranks, multi-iteration analysis for long-running training workloads, drill-down dashboards with percentile analysis across devices, trace data visualization integrated with Perfetto for kernel-level inspection, heat map visualizations for identifying outliers across GPU deployments, and automated insight summaries that highlight critical bottlenecks and optimization opportunities.
The three essential layers of Zoomer’s architecture.
How Zoomer Profiling Works: From Trigger to Insights
Understanding how Zoomer conducts a complete performance analysis offers insight into its sophisticated approach to AI workload optimization.
Profiling Trigger Mechanisms
Zoomer operates using both automatic and on-demand profiling strategies, tailored to different workload types. For training workloads, which involve multiple iterations and can run for days or weeks, Zoomer automatically triggers profiling around iteration 550-555 to capture stable-state performance while avoiding startup noise. For inference workloads, profiling can be triggered on-demand for immediate debugging or through integration with automated load testing and benchmarking systems for continuous monitoring.
Comprehensive Data Capture
During each profiling session, Zoomer simultaneously collects multiple data streams to construct a holistic performance picture:
- GPU Performance Metrics: SM utilization, GPU memory utilization, GPU busy time, memory bandwidth, Tensor Core utilization, power consumption, clock frequencies, and power consumption data via DCGM integration.
- Detailed Execution Traces: Kernel-level GPU operations, memory transfers, CUDA API calls, and communication collectives via PyTorch Profiler and Kineto.
- Host-Level Performance Data: CPU utilization, memory usage, network I/O, storage access patterns, and system-level bottlenecks via dyno telemetry.
- Application-Level Annotations: Training iterations, forward/backward passes, optimizer steps, data loading phases, and custom user annotations.
- Inference-Specific Data: Rate of inference requests, server latency, active requests, GPU memory allocation patterns, request latency breakdowns via Strobelight’s Crochet profiler, serving parameter analysis, and thrift request-level profiling.
- Communication Analysis: NCCL collective operations, inter-node communication patterns, and network utilization for distributed workloads.
Distributed Analysis Pipeline
Raw profiling data flows through sophisticated processing systems that deliver multiple types of automated analysis, including:
- Straggler Detection: Identifies slow ranks in distributed training through comparative analysis of execution timelines and communication patterns.
- Bottleneck Analysis: Automatically detects CPU-bound, GPU-bound, memory-bound, or communication-bound performance issues.
- Critical Path Analysis: Systematically identifies the longest execution paths to focus optimization efforts on highest-impact opportunities.
- Anti-Pattern Detection: Rule-based systems that identify common efficiency issues and generate specific recommendations.
- Parallelism Analysis: Deep understanding of tensor, pipeline, data, and expert parallelism interactions for large-scale distributed training.
- Memory Analysis: Comprehensive analysis of GPU memory usage patterns, allocation tracking, and leak detection.
- Load Imbalance Analysis: Detects workload distribution issues across distributed ranks and provides recommendations for optimization.
Multi-Format Output Generation
Results are presented through multiple interfaces tailored to different user needs: interactive timeline visualizations showing activity across all ranks and hosts, comprehensive metrics dashboards with drill-down capabilities and percentile analysis, trace viewers integrated with Perfetto for detailed kernel inspection, automated insights summaries highlighting key bottlenecks and recommendations, and actionable notebooks that users can clone to rerun jobs with suggested optimizations.
Specialized Workload Support
For massive distributed training for specialized workloads, such as GenAI, Zoomer includes a purpose-built platform supporting LLM workloads. This platform offers specialized capabilities including GPU efficiency heat maps and N-dimensional parallelism visualization. For inference, specialized analysis covers everything from single GPU models, with plans to expand to massive distributed inference across thousands of servers.
A Glimpse Into Advanced Zoomer Capabilities
Zoomer offers an extensive suite of advanced capabilities designed for various AI workload types and scales. While a comprehensive overview of all features would require multiple blog posts, here is a glimpse at some of the most compelling capabilities that demonstrate Zoomer’s depth:
Training Powerhouse Features:
- Straggler Analysis: Helps identify ranks in distributed training jobs that are significantly slower than others, causing overall job delays due to synchronization bottlenecks. Zoomer provides information that helps diagnose root causes like sharding imbalance or hardware issues.
- Critical Path Analysis: Identification of the longest execution paths in PyTorch applications, enabling accurate performance improvement projections.
- Advanced Trace Manipulation: Sophisticated tools for compression, filtering, combination, and segmentation of massive trace files (2GB+ per rank), enabling analysis of previously impossible-to-process large-scale training jobs.
Inference Excellence Features:
- Single-Click QPS Optimization: A workflow that identifies bottlenecks and triggers automated load tests with one click, reducing optimization time while delivering QPS improvements of +2% to +50% across different models, depending on model characteristics.
- Request-Level Deep Dive: Integration with Crochet profiler provides Thrift request-level analysis, enabling identification of queue time bottlenecks and serving inefficiencies that traditional metrics miss.
- Realtime Memory Profiling: GPU memory allocation tracking, providing live insights into memory leaks, allocation patterns, and optimization opportunities.
GenAI Specialized Features:
- LLM Zoomer for Scale: A purpose-built platform supporting 100k+ GPU workloads with N-dimensional parallelism visualization, GPU efficiency heat maps across thousands of devices, and specialized analysis for tensor, pipeline, data, and expert parallelism interactions.
- Post-Training Workflow Support: Enhanced capabilities for GenAI post-training tasks including SFT, DPO, and ARPG workflows with generator and trainer profiling separation.
Universal Intelligence Features:
- Holistic Trace Analysis (HTA): An advanced framework for diagnosing distributed training bottlenecks across communication overhead, workload imbalance, and kernel inefficiencies, with automatic load balancing recommendations.
- Zoomer Actionable Recommendations Engine (Zoomer AR): Automated detection of efficiency anti-patterns with machine learning-driven recommendation systems that generate auto-fix diffs, optimization notebooks, and one-click job re-launches with suggested improvements.
- Multi-Hardware Profiling: Native support across NVIDIA GPUs, AMD MI300X, MTIA, and CPU-only workloads with consistent analysis and optimization recommendations regardless of hardware platform.
Zoomer’s Optimization Impact: From Debugging to Energy Efficiency
Performance debugging with Zoomer creates a cascading effect that transforms low-level optimizations into massive efficiency gains.
The optimization pathway flows from: identifying bottlenecks → improving key metrics → accelerating workflows → reducing resource consumption → saving energy and costs.
Zoomer’s Training Optimization Pipeline
Zoomer’s training analysis identifies bottlenecks in GPU utilization, memory bandwidth, and communication patterns.
Example of Training Efficiency Wins:
- Algorithmic Optimizations: Power savings were delivered through systematic efficiency improvements across the training fleet, by resolving reliability issues for low-efficiency jobs.
- Training Time Reduction Success: In 2024, a 75% training time reduction was observed for Ads relevance models, leading to a 78% reduction in power consumption.
- Memory Optimizations: One-line code changes for performance issues due to inefficient memory copy, identified by Zoomer, delivered 20% QPS improvements with minimal engineering effort.
Inference Optimization Pipeline:
Inference debugging focuses on latency reduction, throughput optimization, and serving efficiency. Zoomer identifies opportunities in kernel execution, memory access patterns, and serving parameter tuning to maximize requests per GPU.
Inference Efficiency Wins:
- GPU and CPU Serving Parameters Improvements: Automated GPU and CPU bottleneck identification and parameter tuning led to a 10% to 45% reduction in power consumption.
- QPS Optimization: GPU trace analysis was used to boost serving QPS and optimize serving capacity.
Zoomer’s GenAI and Large-Scale Impact
For massive distributed workloads, even small optimizations compound dramatically. 32k GPU benchmark optimizations achieved 30% speedups through broadcast issue resolution, while 64k GPU configurations delivered 25% speedups in just one day of optimization.
The Future of AI Performance Debugging
As AI workloads expand in size and complexity, Zoomer is advancing to meet new challenges focused on several innovation fronts: broadening unified performance insights across heterogeneous hardware (including MTIA and next-gen accelerators), building advanced analyzers for proactive optimization, enabling inference performance tuning through serving parameter optimization, and democratizing optimization with automated, intuitive tools for all engineers. As AI infrastructure continues its rapid growth, Zoomer plays an important role in innovating efficiently and sustainably.
