The Three Pillars of Observability

Understand logs, metrics, and traces — and how test engineers use them to debug production failures.

The Three Pillars

Logs

Logs are timestamped, structured records of discrete events.

// Structured log (JSON)
{
  "timestamp": "2024-01-15T14:32:01.234Z",
  "level": "ERROR",
  "service": "checkout-api",
  "traceId": "abc-123",
  "message": "Payment processing failed",
  "userId": "user-456",
  "orderId": "order-789",
  "error": "Card declined: insufficient funds",
  "duration_ms": 1203
}

Log levels:

• DEBUG — verbose, development only
• INFO — normal operations
• WARN — degraded but operational
• ERROR — failed operation, investigation needed
• FATAL — service cannot continue

Metrics

Metrics are numeric measurements aggregated over time.

The RED method for services:

• Rate — requests per second
• Errors — error rate
• Duration — latency distribution

Traces

A trace represents a complete request across multiple services:

Trace: user-checkout-flow (traceId: abc-123)
├── [0ms]    POST /api/checkout        (12ms total)
│   ├── [1ms]  DB: SELECT cart          (3ms)
│   ├── [4ms]  API: validate inventory  (5ms — external)
│   └── [9ms]  DB: INSERT order         (2ms)
└── [12ms]  → Response: 201 Created

Each step is a span with start time, duration, and metadata.

Using Observability in Test Investigation

test('payment failure produces structured log with correct fields', async ({ request }) => {
  // Trigger a payment failure
  const res = await request.post('/api/checkout', {
    data: { cardNumber: '4000000000000002', amount: 100 },  // test card that declines
  });
  expect(res.status()).toBe(402);

  // Assert the log was written (via log API or test helper)
  const logs = await getRecentLogs({ service: 'checkout-api', level: 'ERROR' });
  const paymentLog = logs.find(l => l.message.includes('Payment processing failed'));
  expect(paymentLog).toBeDefined();
  expect(paymentLog.userId).toBeDefined();     // PII context logged
  expect(paymentLog.traceId).toBeDefined();    // correlatable with traces
  expect(paymentLog.duration_ms).toBeDefined(); // latency captured
});

Correlation: Logs + Traces

test('failed request trace correlates with error log', async ({ request }) => {
  // Force an error
  const res = await request.post('/api/orders/invalid-id/process');
  expect(res.status()).toBe(400);

  // Both log and trace should share the same traceId
  const traceId = res.headers()['x-trace-id'];
  expect(traceId).toBeDefined();

  const log = await getLogByTraceId(traceId);
  expect(log.level).toBe('ERROR');
  expect(log.traceId).toBe(traceId);
});

SLI, SLO, SLA — Revisited with Observability

SLI: p95 latency measured by metrics system = 342 ms
SLO: p95 must be < 500 ms (internal target)
SLA: p99 < 1 s (contractual — violation triggers penalty)

Observability pipeline:
Metrics → Grafana dashboard → Alert → PagerDuty → On-call engineer