Saving the Debian package cache on CircleCI

Many CircleCI builds go through a stage where they need to install build dependencies into a docker container. Paid users of CircleCI are able to snapshot Docker layers, to avoid repeating this work on every build. But for the rest of us, those packages get downloaded for every build.

CircleCI don’t provide Debian repos, so this means hitting the Debian servers each time. This can be slow, but seems like is must also waste a lot of bandwidth. Seems like the right thing to do here is to download the packages once and use the CircleCI caching facilities to keep a copy for future builds.

This should be easy, except that the Debian docker image has caching turned off in multiple places. I’m summarising them here so that people don’t neeed to spend ages searching like I did.

version: 2
jobs:
  build:
    docker:
      - image: debian:stretch
    steps:
      - checkout
      - run:
          name: Update packages
          command: apt update

So far, so standard. First thing, we need to tell apt to keep the packages it downloads, by adding a couple of lines to the configuration file.

      - run:
          name: Configure apt cache
          command: |
              echo 'APT::Keep-Downloaded-Packages "true";' >/etc/apt/apt.conf.d/01keep-debs
              echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' >>/etc/apt/apt.conf.d/01keep-debs

Docker also adds some config which deletes the cache afterwards, even if it is created. So we also need to remove that.

      - run:
          name: Remove apt cache purge configuration
          command: rm /etc/apt/apt.conf.d/docker-clean

Stretch doesn’t have the right certificates to talk to the URL inside CircleCI that stores the cache! We need to get these installed before we restore the cache, otherwise it will fail.

      - run:
          name: Install certificates
          command: apt install -y ca-certificates

At last, we can restore the cache, if there is one.

      - restore_cache:
          keys:
              - debian-package-cache-{{ checksum ".circleci/config.yml" }}
              - debian-package-cache

Now install your dependencies. This will either reuse the files downloaded previously, or populate them otherwise.

      - run:
          name: Install build dependencies
          command: apt install -y texlive-latex-base texlive-latex-recommended texlive-latex-extra biber

Over time, we might get an accumulation of out of date dependencies, causing the cache to grow. Not a huge problem, but for neatness, this line removes them.

      - run:
          name: Clean unneeded packages from cache
          command: apt autoclean

Now we have the right stuff in the cache, save it.

      - save_cache.
          key: debian-package-cache-{{ checksum ".circleci/config.yml" }}
          paths:
              - "/var/cache/apt/archives"

Finally, you can do your build.

      - run:
          name: Do your build
          command: ./build.sh

That’s it! Happy building.