Architecture & Scalability

Answer senior-level architecture questions about scaling test suites, parallelism, sharding, reporting, and multi-team test ownership.

The Scalability Interview

Senior and lead roles get questions about what happens when a test suite grows from 50 to 5,000 tests. They want to know:

• How do you keep CI feedback loops short?
• How do you manage test ownership across teams?
• How do you handle shared infrastructure (DBs, auth, environments)?
• How do you measure and improve test reliability?

Execution Scalability

Parallelism vs Sharding

// Parallelism: multiple workers on ONE machine
// playwright.config.ts
export default defineConfig({
  workers: process.env.CI ? 4 : 2,  // 4 CPUs on CI agent
});

// Sharding: split suite across MULTIPLE machines
// Machine 1: npx playwright test --shard=1/4
// Machine 2: npx playwright test --shard=2/4
// Machine 3: npx playwright test --shard=3/4
// Machine 4: npx playwright test --shard=4/4

When to shard:

• Suite takes >15 minutes on a single machine
• CI costs allow multiple parallel agents
• Tests are independent (no shared mutation)

Blob Reporter + Merge

# GitHub Actions matrix sharding
jobs:
  test:
    strategy:
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - run: npx playwright test --shard=${{ matrix.shard }}/4 --reporter=blob
      - uses: actions/upload-artifact@v4
        with:
          name: blob-report-${{ matrix.shard }}
          path: blob-report/

  merge-reports:
    needs: test
    steps:
      - uses: actions/download-artifact@v4
        with: { path: all-blob-reports/, pattern: blob-report-* }
      - run: npx playwright merge-reports --reporter=html ./all-blob-reports

Test Suite Architecture at Scale

Layered Test Strategy

Suite layers (each has different run frequency):

Layer 1: Smoke (10-20 tests)      → every commit, <5 min
Layer 2: Regression (200-500)     → every PR, <20 min (sharded)
Layer 3: Full suite (1000+)       → nightly, <60 min (sharded)
Layer 4: Performance/Security     → weekly or release

// Tag-based filtering
test('critical path checkout @smoke @regression', async ({ page }) => { ... });
test('edge case empty cart @regression', async ({ page }) => { ... });

// playwright.config.ts projects
projects: [
  { name: 'smoke', grep: /@smoke/ },
  { name: 'regression', grep: /@regression/ },
  { name: 'full', grepInvert: /@skip/ },
]

Multi-Team Ownership

Mono-repo with team namespacing:
e2e/
├── core/        → platform team owns
├── payments/    → payments team owns  
├── catalog/     → catalog team owns
└── shared/      → shared fixtures, utilities (governance required)

playwright.config.ts: each team has a project config
payments team PR: only runs e2e/payments/ tests in PR, full suite nightly

Reliability at Scale

Flakiness Tracking

// Custom reporter: track flaky tests
class FlakinesReporter implements Reporter {
  onTestEnd(test: TestCase, result: TestResult) {
    if (result.status === 'flaky') {
      // POST to metrics service: test title, file, duration, retry count
      reportFlakiness({
        title: test.title,
        file: test.location.file,
        retries: result.retry,
        duration: result.duration,
      });
    }
  }
}

Flakiness SLO: Teams own their test reliability. A test that flakes >5% of runs in 7 days is auto-filed as a P2 bug and quarantined (tagged @flaky, excluded from smoke).

Quarantine Pattern

// quarantine.ts — shared list
export const QUARANTINED = [
  'checkout > payment > should handle timeout',
  'catalog > search > should filter by price',
];

// playwright.config.ts
grepInvert: new RegExp(QUARANTINED.map(t => t.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')).join('|'))

Environment Architecture

Environments:
- local: developer machine, real services via docker-compose
- preview: per-PR ephemeral environment (Vercel/Railway preview deploy)
- staging: shared, seeded weekly, integration test target
- production: smoke only (never mutate, never create test data)

// playwright.config.ts — environment-aware baseURL
const ENV = process.env.TEST_ENV ?? 'staging';
const BASE_URLS = {
  local: 'http://localhost:3000',
  preview: process.env.PREVIEW_URL!,
  staging: 'https://staging.example.com',
  production: 'https://example.com',
};

export default defineConfig({
  use: { baseURL: BASE_URLS[ENV] },
});