Home Lab Journal Part 2 - Destruction and Regeneration

My CI/CD pipeline for this blog was starting to have issues last year. I built this blog with a static site generator called Hakyll, which is built using Haskell and Stack. Building the static site generator during blog publication is pretty expensive since all of the Haskell packages are statically linked to the binary. For a while, I was able to cache the binary and only have to do a full rebuild of the generator once per month.

Unfortunately, the caching method that I was using stopped working in GitHub Actions. Wanting more control, and seeing this as good opportunity to set up a service in my home lab, I set out to configure my own CI/CD pipeline on one of my lab machines.

I spent some time researching different options for this task and ended up picking Forgejo. I know that Forgejo is more than just CI/CD, but I thought it would be a good idea to use something that is backwards compatible with with GitHub Actions so I don’t have to spend too much time up front converting my pipeline specification to the new pipeline.

One thing I’m glad for now is that I set about setting up an Ansible playbook that would make sure that a Docker role was applied to the system, and then configure the Forgejo image, a Docker in Docker sidecar, and the Forgejo runner were set up. That’s because I did something stupid. I know this is a bad idea, but since my daily driver is Arch based, I thought it would be interesting to set up my 2 Lenovo Tinies with Arch Linux installs. I wanted to see how long they would last before a package update would kill them.

I went on vacation. When I came home I ran some system updates, and Arch corrupted many of it’s /etc files. The corruption was so bad, I couldn’t get a configured network interface despite going through the advice on the Arch forum for this particular problem. With my Ansible playbook in hand, I decided to just swap out pacman for apt in the yaml. Then I installed Ubuntu Server on my home lab machines and rebuilt them with Ansible.

I had my CI/CD pipeline project back to where it used to be within the afternoon. Then I added some Terraform for some AWS roles and got the pipeline working like it used to. I still have the caching problem, but I now have the ability to publish my blog again.

What’s next for the Home Lab?

During this process, I identified some more improvements to make. The Docker image that I’m using for the base container within my runner didn’t have some specific tools that I needed. I ended up resorting to building and deploying the image to the server as part of the playbook. I think one of my next steps will be to set up a quick Docker registry in my home lab so that I can pull custom images using Ansible and other tools.

Next up, I have some issues with secrets management. Right now I have things working between KeePass and Ansible Vault. This means that I have encrypted secrets committed into Git, which I don’t necessarily like. I would prefer to have a secrets manager deployed that way if I ever get to the point where I have a CI/CD pipeline for services in my home lab, then I have a way of securely distributing secrets.

I also need to settle on a reverse proxy. I have already dabbled a little bit at setting up a split horizon domain in my home lab. If I can get that working, and have a good reverse proxy set up then I can get friendlier URLs and I can have an easier time managing certs for HTTPS.