Chris Wiegman

Battling Link Rot on a Hugo Site

| 5 min read
Battling Link Rot on a Hugo Site

One of the hardest parts of keeping this site running as it ages has been staying on top of link rot. Having 18 years of links adds up, especially when you’re talking about tech that counts in days what the rest of the world counts in years.

For a number of years I used Dr. Link Check to give me a report of broken links, which I’d then fix by hand. It’s been effective but this site has grown to a point where I need an ever-more-expensive plan to keep up with it all. That’s OK, but not ideal.

Since moving back to Hugo, whose links are much easier to parse programmatically, I thought there had to be a better way.

There is, and it’s called Lychee.

Lychee is a Rust CLI utility that checks links programmatically and generates a report I can use to fix them. With Dr. Link Check I ran link checks once a year when I paid for a month of the service. With Lychee I run link checks automatically once a month or on demand, either as a GitHub Action or on my laptop. This means I can stay ahead of the inevitable link rot over time and make sure this site continues to function as I intended.

The Lychee Configuration

Regardless of where I run Lychee, locally or in the GitHub Action, it uses the same configuration file:

# Shared config for `make check-links` and .github/workflows/link-check.yml.
# lychee loads this automatically from the working directory.

cache = true
max_cache_age = "25d"
no_progress = false

# Accept 403s: many sites block bots outright, which isn't a real broken link.
accept = ["200..=299", "403"]

exclude = [
    '^(?:/|#|\.)',
    '^file:',
    '^mailto:',
    'https?://(www\.)?chriswiegman\.com',
]
exclude_path = ["public/library/"]

# Identify ourselves so hosts don't silently throttle/block an anonymous bot UA.
user_agent = "Mozilla/5.0 (compatible; chriswiegman.com link checker; +https://chriswiegman.com)"

# Give slower sites more time before giving up, and retry transient/rate-limit
# failures with a real gap instead of hammering them again immediately.
timeout = 30
max_retries = 4
retry_wait_time = 3

# Keep overall load low enough that we don't trip rate limits or saturate
# our own connection pool (defaults are 128 total / 10 per host / 50ms gap).
max_concurrency = 32
host_concurrency = 4
host_request_interval = "500ms"

Running Lychee Locally

To run Lychee locally I use a Make target. This generates a fresh report each time I run it:

.PHONY: check-links
check-links:
	rm -f lychee-report.md
	@command -v lychee >/dev/null 2>&1 || { echo "lychee not found. Install it with: brew install lychee"; exit 1; }
	hugo --gc --minify
	@lychee -vv --root-dir '$(CURDIR)/public' --format markdown --output lychee-report.md './public/**/*.html';

With this I only run make check-links from my terminal and it generates a Markdown report I can use to clean up any bad links.

Running as a GitHub Action

I also use the following GitHub Action to check the links on my site monthly and create an issue with any bad links found:

name: Check for Broken Links

on:
  schedule:
    # Run monthly at 7 AM UTC on the 1st of every month
    - cron: '0 7 1 * *'

permissions:
  contents: read
  issues: write

concurrency:
  group: ${{ github.workflow }}
  cancel-in-progress: true

jobs:
  check-links:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v7
        with:
          submodules: true
          fetch-depth: 2

      - name: Setup Hugo
        uses: peaceiris/actions-hugo@v3
        with:
          hugo-version: "latest"
          extended: true

      # Hugo's resources/ (processed images, bundles) and its module/build
      # cache (.hugo_cache) are cheap to share across runs but expensive to
      # rebuild from scratch, so both are restored from one cache entry
      # keyed on everything that can change the build output.
      - name: Cache Hugo build artifacts
        uses: actions/cache@v6
        with:
          path: |
            resources
            .hugo_cache
          key: ${{ runner.os }}-hugo-${{ hashFiles('**/go.mod', '**/go.sum', '**/package-lock.json', '**/pnpm-lock.yaml', '**/yarn.lock', 'hugo.toml', 'hugo.yaml', 'hugo.json', 'config/**', 'assets/**') }}
          restore-keys: |
            ${{ runner.os }}-hugo-

      - name: Build site
        shell: bash
        run: hugo --gc --minify --cacheDir "${{ github.workspace }}/.hugo_cache"

      - name: Cache Lychee output
        uses: actions/cache@v6
        with:
          path: .lycheecache
          key: cache-lychee-${{ hashFiles('.lycheecache') }}
          restore-keys: cache-lychee-

      - name: Check for broken links
        id: lychee
        uses: lycheeverse/lychee-action@v2
        with:
          args: --root-dir '${{ github.workspace }}/public' './public/**/*.html'
          fail: false
          format: markdown
          jobSummary: true
        env:
          GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}

      - name: Upload lychee report
        if: always()
        uses: actions/upload-artifact@v7
        with:
          name: lychee-report
          path: ./lychee/out.md
          if-no-files-found: warn

      - name: Create or update issue with broken links report
        if: steps.lychee.outputs.exit_code == '1'
        uses: actions/github-script@v9
        with:
          script: |
            const fs = require('fs');
            const maxBodyBytes = 60000;
            const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;

            let report = '';
            try {
              const lycheeOutput = fs.readFileSync('./lychee/out.md', 'utf8');
              const outputBytes = Buffer.byteLength(lycheeOutput, 'utf8');
              if (outputBytes > maxBodyBytes) {
                const trimmed = lycheeOutput.slice(0, Math.floor(maxBodyBytes * 0.9));
                report = `${trimmed}\n\n...output truncated. [See full report in workflow logs](${runUrl}).`;
              } else {
                report = lycheeOutput;
              }
            } catch (error) {
              report = `Could not read lychee output. [See workflow logs](${runUrl}) for details.`;
            }

            const body = `## Broken Links Report\n\n_Generated by weekly link check — [workflow run](${runUrl})_\n\n${report}`;

            const issues = await github.rest.issues.listForRepo({
              owner: context.repo.owner,
              repo: context.repo.repo,
              state: 'open',
              labels: ['broken-links']
            });

            if (issues.data.length === 0) {
              await github.rest.issues.create({
                owner: context.repo.owner,
                repo: context.repo.repo,
                title: 'Broken Links Detected',
                body: body,
                labels: ['broken-links', 'bug']
              });
            } else {
              await github.rest.issues.createComment({
                owner: context.repo.owner,
                repo: context.repo.repo,
                issue_number: issues.data[0].number,
                body: body
              });
            }

Upload this to your .github/workflows folder in your repo and it will handle everything except actually fixing the links.

Taken together, the config and these run methods are helping me clean up this site and keep it running well going forward. If you run a Hugo site of your own, both the config and the workflow above should drop in with only a little modification.