Home Lab Journal Part 1 - Getting Started Again

I have a relatively simple home lab. It’s mainly just a NAS, media server running PLEX, and a remote VPS that runs a service that I want to keep up more reliably. I’ve been watching more Home Lab videos on YouTube lately, and decided to start sprucing up my home lab so that I do some more tinkering at home. I wanted to get to the point where I could run a Kubernetes cluster at home. But first I had few issues to solve.

Power Distribution

At some point shortly after moving house, my the batteries died in the UPSes for my set up. Feeling overwhelmed with other things at the time, I just pulled out the UPSes from my set up and had everything plugged into surge protectors. I didn’t quite have the bandwidth for figuring out what kind of batteries that I would need for each UPS system, so I just put it off.

Unfortunately, that left the available power sockets in my home office completely saturated. I needed to solve my battery problem to make sure that I had enough power to accommodate the three new machines that I was about to set up.

Now that I had a clearer goal and some more mental bandwidth, I set about to figure out what batteries that I needed and find instructions from the UPS manufacturer on how to replace the batteries. It turns out that sourcing the batteries was easier than I had previously thought. They’re just off the shelf sealed lead acid batteries.

I found a decent deal on Amazon for two batteries with free same day delivery. I figured cool I have two UPSes, so I just need two new batteries. I could get them delivered that Friday, and install the batteries on Saturday to get both units back up and running. It wasn’t until I opened the first unit that I realized that each UPS was going to take 2 lead acid batteries. It’s a good thing that I ordered two!

I got the one UPS back online and started rewiring what I could of the power distribution. Thankfully, I was doing this in stages, so with one UPS, had enough power for the first piece of new hard war that I needed to set up.

Network Port Saturation

Much like with my power distribution problem, I had an issue with the availability of network ports. Every single ethernet port was occupied. Thankfully, it was simple to address by getting a new 8 port unmanaged switch. I ordered it at the same time as the batteries, and then re-wired my network while everything was powered down.

Updates

I also noticed while I was re-configuring things that my servers and my laptop were all able to update to the next LTS of Ubuntu. I decided to try and upgrade 1 machine at a time, and things went a bit sideways.

The cleanest upgrade was my home server that I use to run PLEX. It upgraded pretty much without issue. With a successful upgrade under my belt, I thought cool it will be easy to just get this done. I’m going to upgrade my cloud server and my laptop at the same time.

The laptop kept failing it’s initial checks. I kept having to reconfigure my system to try and prepare it for the upgrade. It took multiple iterations of upgrading packages and removing others until finally the LTS update script would let me proceed. After it finished (my laptop happened to be the longest run because of all of the packages that I have installed), it rebooted and immediately seized up. I was locked out enough that I couldn’t kill X and try to get into a TTY session. Thankfully, after a reboot, I was able to get in and use the system normally.

My cloud server while being the fastest system to upgrade, took the longest to get it’s services back up and running. I have a couple of python services that I’m running on this VPS. None of them were properly isolated by their developer’s in virtual environments. Ubuntu has disallowed the global installation of pip packages due to supply chain attack issues (can’t say that I blame them). That left me with services that were down and would not come back up.

I tried to force them into isolation using venv and some other configuration tricks. Unfortunately, while I was able to get some of these services up to the point of having their admin panels available in the browser, I was not able to get appropriate communication between services.

I eventually tore the whole mess down, and wrote a docker compose file to set up the services on my server. After a little bit of tinkering, the services were accessible and talking to each other.

Conclusion

Now that I’ve got all of that sorted out my next steps will be:

• Set up a jump box.
• Get my hands on some Lenovo Tiny PCs to use as nodes in my Kubernettes cluster.
• Set up Kubernetes using Kubernetes The Hard Way.