Craig Weber

This week I’ve been looking at overlayroot as a potential solution to reduce the effort to make changes to the nodes in my Raspberry Pi cluster. In this post I want to brain-dump about the problem I’m hoping it solves and the problems I’m running into with respect to implementing overlayroot as well as potential solutions that I’m exploring.

We recently moved from Chicago to Des Moines, and we’re staying in an AirBnB for a couple months while we look for the right house to buy. In the meanwhile, most of our stuff (including critical components of my homelab) are in storage, which means my blog wasn’t running. In this transient period, I figured I would try to run my blog in the cloud, and while there are easier and even cheaper options, I decided to try out running it on EC2 in order to learn a bit more about traditional Linux system administration. This post will document the approach I arrived at.

I’m starting a new series where I briefly discuss what I’ve been working on, what I’ve read, and what I’d like to explore. I’m just calling it “Update” because I know I can’t commit to any particular schedule. To find out what I’ve been thinking about recently, read on.

Disclaimer: This is not a production-grade solution

At home I have a Raspberry Pi Kubernetes cluster running, among other things, this blog (or at least at the time of this writing). One of my goals for this cluster is to keep my cloud/SaaS/etc costs down below $5/month. Another goal is to avoid poking holes in my home router’s firewall.

One of the maddening things about MacOS is that it lacks ext4 file system support. This is a bummer because I use ext4 on many of my external hard drives as well as boot volumes for various headless Linux machines. Once in a while, these boot drives become corrupted (due to user error) and I find myself wanting to mount the file system on another system to repair the error. Ideally I can just pop it in a SATA<->USB adapter and mount it on my Mac, but alas…

I worked on this blog for several hours this weekend, releasing a new post, implementing a couple new features and fixing some bugs. Here are the highlights:

• Published the first entry in my Homelab series
• Fixed a bug on iPad that was causing a ~300% zoom
• Implemented syndication (atom feed)
• Fixed broken relative links in post snippets
• Reduced coupling between markdown and site configuration

For hardware, I settled on Raspberry Pi 4Bs. They support up to 8GB of RAM (enough power to run the k3s master nodes) and USB 3.0 for fast external SSD I/O. The only downside of the 4Bs is that they require more power than the 3Bs, and the same multiport USB power supplies that could support a 3B cluster couldn’t support a 4B cluster. To solve for this, I decided on PoE hats and a PoE switch. This halves the number of cables that need to be run to each Pi, which makes the Pi cluster that much more enjoyable and easy on the eyes.

For whatever reason, macOS Catalina and Big Sur were both tinting my external monitor pink. Some research indicated that it had to do with the color mode, notably that I needed to force RGB. MacOS’s UI doesn’t give the user the ability to change the color mode directly, so you have to hack around the display profile files directly.

This post and its comments from 2013 seem to be the authoritative guide on forcing RGB mode; however, these steps (and the variations found in the comments) make you do a lot of things, including disabling the System Integrity Protection (basically the stuff that prevents even the super user from changing certain files and directories), booting into recovery mode, changing boot files (which can put your system into a boot loop, as I discovered the hard way), and a number of other dangerous, arcane things.

Fortunately, I found a sequence that is much safer and easier (tested on both Big Sur and Catalina on two distinct MacBook Pros):

As I alluded to in my last post, I’ve finally decided to pull the trigger and build my own homelab: a personal computing environment for playing around with new tools and approaches for developing or operating software, including software that is personally useful.

For my homelab, I want to build a Raspberry Pi Kubernetes cluster for hosting my own applications and experimenting with different tools and approaches for operating software. However, bare metal (i.e., not running on a public cloud provider, such as AWS) is a second-class citizen for Kubernetes, so one must bring their own providers for storage, networking, load balancing, ingress (roughly “HTTP/layer-7 routing”), and much more.

One day, I have no doubt that there will be Kubernetes distributions targeting bare metal which are mature, robust, and open source; in the meanwhile, this series will document my efforts to work around those limitations so that others can build their own personal cloud platform more easily (or at least know what they’re considering getting into!).

Next time, I’ll delve into the hardware I’m using for my cluster.

• Part I: Hardware

I’ve recently been working on my Raspberry Pi Kubernetes cluster. I also use Tailscale for my home VPN (because it’s performant and absurdly easy to setup and configure). I wanted to run Kubernetes services on my VPN using private DNS names (e.g., foo.local) and addresses from the Tailscale address space (e.g., 100.*) as opposed to the host network address space (e.g., 192.168.*).