Uber built an automated system that correlates caller and downstream failures using RPC middleware and production metrics. The tool classifies dependencies as fail-close, fail-open, or unknown to overcome the limitations of sampled tracing for rare incidents.
- Correlates caller and downstream failures via RPC middleware and production metrics
- Classifies dependencies as fail-close, fail-open, or unknown for better handling
- Addresses gaps in sampled distributed tracing for rare failure scenarios
- Automates dependency mapping across Uber’s large-scale service mesh