DrP: Meta’s Root Cause Analysis Platform at Scale

Investigating incidents can be a challenging endeavor in today’s digital environment, particularly within large-scale systems composed of numerous interconnected components and dependencies.

DrP, a root cause analysis (RCA) platform developed by Meta, automates the investigation process. This automation significantly decreases the mean time to resolve (MTTR) incidents and reduces the workload for on-call personnel.

Currently, over 300 teams at Meta leverage DrP, conducting 50,000 analyses daily. The platform has proven effective in cutting MTTR by 20-80%. Understanding DrP’s capabilities can open new avenues for efficient incident resolution and enhanced system reliability.

What It Is

DrP functions as an end-to-end platform that automates the investigation process for extensive systems. It addresses the inefficiencies inherent in manual investigations, which often depend on outdated playbooks and ad-hoc scripts. These traditional approaches can lead to extended downtimes and increased on-call effort as engineers spend considerable time triaging and debugging incidents.

DrP provides a comprehensive solution through an expressive and adaptable SDK, which allows for the creation of investigation playbooks known as analyzers. These analyzers are executed by a scalable backend system that integrates smoothly with standard workflows, including alerts and incident management tools. Furthermore, DrP incorporates a post-processing system to automate actions based on investigation outcomes, such as mitigation steps.

DrP’s primary components include:

1. Expressive SDK: The DrP SDK enables engineers to codify investigation workflows into analyzers. It offers a rich collection of helper libraries and machine learning (ML) algorithms for data access and problem isolation analysis, covering areas like anomaly detection, event isolation, time series correlation, and dimension analysis.
2. Scalable Backend: This system executes the analyzers, providing both multi-tenant and isolated execution environments. It ensures that analyzers can operate at scale, managing thousands of automated analyses daily.
3. Integration with Workflows: DrP integrates with alerting and incident management tools, allowing analyzers to be automatically triggered during incidents. This integration ensures that investigation results are immediately available to on-call engineers.
4. Post-processing System: Following an investigation, this system can initiate automated actions based on the analysis results. For instance, it can generate tasks or pull requests to address issues identified during the investigation.

How It Works

Authoring Workflow

The creation of automated playbooks, or analyzers, begins with the DrP SDK. Engineers outline the investigation steps, detailing inputs and potential paths to pinpoint problem areas. The SDK offers APIs and libraries to codify these workflows, enabling engineers to capture all necessary input parameters and context in a type-safe manner.

1. Enumerate Investigation Steps: Engineers start by listing the steps required for incident investigation, including inputs and potential routes to isolate the problem.
2. Bootstrap Code: The DrP SDK provides bootstrap code to generate a template analyzer with pre-filled boilerplate code. Engineers then extend this code to capture all essential input parameters and context.
3. Data Access and Analysis: The SDK includes libraries for data access and analysis, such as dimension analysis and time series correlation. Engineers use these libraries to code the main investigation decision tree into the analyzer.
4. Analyzer Chaining: For dependent service analysis, the SDK’s APIs facilitate seamless chaining of analyzers, allowing for context transfer and output retrieval.
5. Output and Post-processing: The output method captures findings from the analysis, utilizing specialized data structures for both text and machine-readable formats. Post-processing methods automate actions based on analyzer findings.

Once developed, analyzers undergo testing and code review. DrP incorporates automated backtesting into code review tools, ensuring high-quality analyzers before deployment.

Consumption Workflow

In a production environment, analyzers integrate with tools such as UI, CLI, alerts, and incident management systems. Analyzers can automatically activate upon alert triggers, delivering immediate results to on-call engineers and improving response times. The DrP backend manages a queue for requests and a worker pool for secure execution, with results returned asynchronously.

1. Integration with Alerts: DrP is integrated with alerting systems, allowing analyzers to automatically trigger when an alert is activated. This provides immediate analysis results to on-call engineers.
2. Execution and Monitoring: The backend system manages a queue for analyzer requests and a worker pool for execution. It monitors execution, ensuring that analyzers run securely and efficiently.
3. Post-processing and Insights: A separate post-processing system handles analysis results, annotating alerts with findings. The DrP Insights system regularly analyzes outputs to identify and rank the primary causes of alerts, assisting teams in prioritizing reliability improvements.

Why It Matters

Reducing MTTR

DrP has demonstrated significant improvements in reducing MTTR across various teams and use cases. By automating manual investigations, DrP enables quicker triage and mitigation of incidents, leading to faster system recovery and enhanced availability.

1. Efficiency: Automated investigations reduce the time engineers spend on manual triage, allowing them to concentrate on more complex tasks. This efficiency translates to faster incident resolution and reduced downtime.
2. Consistency: By codifying investigation workflows into analyzers, DrP ensures consistent and repeatable investigations. This consistency lowers the likelihood of errors and improves the reliability of incident resolution.
3. Scalability: DrP can manage thousands of automated analyses daily, making it suitable for large-scale systems with intricate dependencies. Its scalability ensures it can support the needs of expanding organizations.

Enhancing On-Call Productivity

The automation provided by DrP reduces the effort required from on-call personnel during investigations, saving engineering hours and mitigating on-call fatigue. By automating repetitive and time-consuming steps, DrP allows engineers to focus on more complex tasks, thereby improving overall productivity.

Scalability and Adoption

DrP has been successfully deployed at scale within Meta, serving over 300 teams and utilizing 2000 analyzers to execute 50,000 automated analyses daily. Its integration into mainstream workflows, such as alerting systems, has facilitated widespread adoption and demonstrated its value in real-world scenarios.

1. Widespread Adoption: DrP has been adopted by hundreds of teams across various domains, showcasing its versatility and effectiveness in addressing diverse investigation requirements.
2. Proven Impact: DrP has been in production for over five years, with demonstrated results in reducing MTTR and enhancing on-call productivity. Its impact is evident in positive user feedback and significant improvements in incident resolution times.
3. Continuous Improvement: DrP is constantly evolving, with ongoing enhancements to its ML algorithms, SDK, backend system, and integrations. This commitment ensures DrP remains a cutting-edge solution for incident investigations, while its growing adoption across teams allows existing workflows and analyzers to be reused, compounding the shared knowledge base and increasing its value throughout the organization.

What’s Next

Looking forward, DrP aims to evolve into an AI-native platform, playing a central role in advancing Meta’s broader AI4Ops vision. This transformation will enable more powerful and automated investigations, enhancing analysis by delivering more accurate and insightful results. It will also simplify the user experience through streamlined ML algorithms, SDKs, UI, and integrations, facilitating effortless authoring and execution of analyzers.