Observability and speeding up slow pipelines

Find where the time goes, cache what is safe to cache, build only what changed, use remote execution, track DORA measures, and alert on the failures that matter.

Find where the time actually goes

# GitHub Actions: per-step timing from the API
gh run view 123456 --json jobs \
  --jq '.jobs[] | .name as $j | .steps[] | "\\($j)\\t\\(.name)\\t\\(.conclusion)\\t\\(.startedAt)\\t\\(.completedAt)"'

# GitLab: job durations across the last 20 pipelines
glab api "projects/:id/pipelines?per_page=20" \
  | jq -r '.[].id' \
  | xargs -I{} glab api "projects/:id/pipelines/{}/jobs" \
  | jq -r '.[] | "\(.name)\t\(.duration)"' \
  | sort -k2 -nr | head -20

# locally: Gradle, Jest and pytest all report slow suites
gradle build --profile          # build/reports/profile
npx jest --verbose --silent=false

Queue wait is invisible in the workflow file and often the largest term. Measure wall-clock time from commit to first green result, not the sum of step durations.

Caching and affected-only builds

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }        # needed to diff against the base

      - name: Detect changed packages
        id: changed
        run: |
          if [ "${{ github.event_name }}" = "pull_request" ]; then
            BASE="origin/${{ github.base_ref }}"
            git fetch origin "${{ github.base_ref }}" --depth=1
            PACKAGES=$(git diff --name-only "$BASE"...HEAD -- 'packages/*' | cut -d/ -f2 | sort -u | paste -sd, -)
          else
            PACKAGES=all
          fi
          echo "packages=${PACKAGES:-none}" >> "$GITHUB_OUTPUT"

      - uses: actions/setup-node@v4
        with: { node-version: 22, cache: npm }

      - name: Restore build cache
        uses: actions/cache@v4
        with:
          path: |
            .turbo
            node_modules/.cache
          key: build-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}-${{ github.sha }}
          restore-keys: build-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}-

      - run: npx turbo run build test --filter="[${{ steps.changed.outputs.packages }}]"

• A cache key must include the lockfile hash, or a dependency change is served a stale node_modules.
• Use restore-keys for a partial match so a new commit still warms from the previous cache.
• Never cache anything that must be reproducible byte for byte, such as a release artefact — build it fresh and upload it as an artefact instead.
• Cache only across trusted branches. A pull request that can write a cache another pull request restores is a delivery path for malicious code.

Metrics and alerting

# notify when the default branch has been red for more than 30 minutes
- name: Alert on long red
  if: failure() && github.ref == 'refs/heads/main'
  run: |
    ./notify.sh "Pipeline broken on main since ${{ github.run_started_at }}: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"

# and a weekly report of what is slow, so it gets fixed rather than endured
on:
  schedule:
    - cron: '0 9 * * 1'

FAQ

Related

Testing strategy inside the pipeline CI/CD concepts and choosing a platform

Last refreshed 2026-09-18.