Case study
Logging, MDC, Trace ID, ALB Correlation ID and Log Aggregator
Architect Observability Distributed Tracing
Trace ID Propagation Across Services
flowchart LR
A("Load Balancer\nALB / NLB") -->|"Injects X-Trace-Id header"| B("Reverse Proxy\nnginx / Apache")
B -->|"Forwards Trace ID"| C("Application 1\nTomcat / SLF4J + MDC")
C -->|"Propagates Trace ID"| D("Application 2\nTomcat / SLF4J + MDC")
classDef accent fill:#0047FF,stroke:#0036C2,color:#FFFFFF
class A accent
Log Aggregation and Analysis
flowchart LR
subgraph Sources["Log Sources"]
A("Application Logs")
B("System Logs")
C("Reverse Proxy Logs")
D("ALB Access Logs")
end
E("Log Aggregation\nSolution")
subgraph Outputs["Monitoring and Observability"]
F("Dashboard\nLog Visualization")
G("Alerting")
G --> H("Paging\nPagerDuty / OpsGenie")
G --> I("Chat\nSlack / Teams")
G --> J("Email\nNotifications")
end
Sources -->|Log Data| E
E --> Outputs
classDef accent fill:#0047FF,stroke:#0036C2,color:#FFFFFF
class E accent
Background
A set of Java applications were running on Tomcat servers, each using a different logging framework with no standardized format. There was no way to correlate logs across applications — no trace IDs, no correlation IDs, and no structured log output. This made it extremely difficult to trace a request end-to-end or search and analyze logs effectively.
Problem
- No standard logging framework or configuration across the applications.
- No trace IDs or correlation IDs to correlate logs across services.
- No end-to-end log tracing from the load balancer through reverse proxies to each application.
- No log aggregation solution to collect and centralize logs from different sources.
- No log monitoring or alerting to detect and respond to issues in a timely manner.
Solution
- Standardize on a common logging framework and configuration across all applications.
- Implement MDC (Mapped Diagnostic Context) to attach trace IDs at the request level and propagate them across service boundaries.
- Select and deploy a log aggregation solution.
- Configure ALB, reverse proxies, and applications to inject and forward trace IDs consistently.
- Set up log monitoring and alerting using the aggregation solution to detect issues in real time.
- Enable log analysis and visualization to gain insights into application performance and user behavior.
- Define log retention policies and log rotation to manage storage and ensure availability for analysis.
- Standardize logging metadata, categories, and date formats across all services for consistency.
- Train and educate teams on the new logging standards, trace propagation, aggregation, monitoring, and retention practices.
Outcome
- Standardized on SLF4J with Logback as the logging framework across all applications.
- Implemented MDC to attach and propagate trace IDs at the request level across service calls.
- Selected and deployed a centralized log aggregation solution.
- Achieved end-to-end log tracing from the load balancer and reverse proxies through all applications using trace IDs.
- Implemented log monitoring and alerting to detect and respond to incidents in a timely manner.
- Delivered log analysis dashboards to surface insights into application performance and user behavior.
- Established log retention policies and rotation schedules to manage storage efficiently.