mujahidk

Case study

Logging, MDC, Trace ID, ALB Correlation ID and Log Aggregator

Architect Observability Distributed Tracing

Trace ID Propagation Across Services

flowchart LR
    A("Load Balancer\nALB / NLB") -->|"Injects X-Trace-Id header"| B("Reverse Proxy\nnginx / Apache")
    B -->|"Forwards Trace ID"| C("Application 1\nTomcat / SLF4J + MDC")
    C -->|"Propagates Trace ID"| D("Application 2\nTomcat / SLF4J + MDC")

  classDef accent fill:#0047FF,stroke:#0036C2,color:#FFFFFF
    class A accent

Log Aggregation and Analysis

flowchart LR
    subgraph Sources["Log Sources"]
        A("Application Logs")
        B("System Logs")
        C("Reverse Proxy Logs")
        D("ALB Access Logs")
    end

    E("Log Aggregation\nSolution")

    subgraph Outputs["Monitoring and Observability"]
        F("Dashboard\nLog Visualization")
        G("Alerting")
        G --> H("Paging\nPagerDuty / OpsGenie")
        G --> I("Chat\nSlack / Teams")
        G --> J("Email\nNotifications")
    end

    Sources -->|Log Data| E
    E --> Outputs

  classDef accent fill:#0047FF,stroke:#0036C2,color:#FFFFFF
    class E accent

Background

A set of Java applications were running on Tomcat servers, each using a different logging framework with no standardized format. There was no way to correlate logs across applications — no trace IDs, no correlation IDs, and no structured log output. This made it extremely difficult to trace a request end-to-end or search and analyze logs effectively.

Problem

  1. No standard logging framework or configuration across the applications.
  2. No trace IDs or correlation IDs to correlate logs across services.
  3. No end-to-end log tracing from the load balancer through reverse proxies to each application.
  4. No log aggregation solution to collect and centralize logs from different sources.
  5. No log monitoring or alerting to detect and respond to issues in a timely manner.

Solution

  1. Standardize on a common logging framework and configuration across all applications.
  2. Implement MDC (Mapped Diagnostic Context) to attach trace IDs at the request level and propagate them across service boundaries.
  3. Select and deploy a log aggregation solution.
  4. Configure ALB, reverse proxies, and applications to inject and forward trace IDs consistently.
  5. Set up log monitoring and alerting using the aggregation solution to detect issues in real time.
  6. Enable log analysis and visualization to gain insights into application performance and user behavior.
  7. Define log retention policies and log rotation to manage storage and ensure availability for analysis.
  8. Standardize logging metadata, categories, and date formats across all services for consistency.
  9. Train and educate teams on the new logging standards, trace propagation, aggregation, monitoring, and retention practices.

Outcome

  1. Standardized on SLF4J with Logback as the logging framework across all applications.
  2. Implemented MDC to attach and propagate trace IDs at the request level across service calls.
  3. Selected and deployed a centralized log aggregation solution.
  4. Achieved end-to-end log tracing from the load balancer and reverse proxies through all applications using trace IDs.
  5. Implemented log monitoring and alerting to detect and respond to incidents in a timely manner.
  6. Delivered log analysis dashboards to surface insights into application performance and user behavior.
  7. Established log retention policies and rotation schedules to manage storage efficiently.
 

Get in touch.

Contact