Basically my entire Homelab is build upon HashiCorp’s products. On August 10th, HashiCorp announced that they would switch all of their products to the BSL, the Business Source License, where they had been licensed under the Mozilla Public License before.

From my (rather rudimentary!) understanding, the license basically says that all “non-production” use is perfectly fine. I’m pretty confident that that covers all of my own personal usage. But as it was pointed out to me today, that formulation also creates a lot of uncertainty for commercial entities of all kinds.

For some products, it looks pretty clear on first sight. When you offer hosted Nomad, perhaps offering Vault for secrets and Consul for service discovery and mesh networking as well, you can’t use the products anymore, period.

If, on the other hand, you are using a Nomad cluster to run your SaaS offering, it seems that you are fine (as long as the software you are aSS’ing isn’t another HashiCorp product. 😉).

But there are other use cases for other products that look to me to be far less clear cut, for example what about offering Vagrant based VMs to your customers for training on your own software?

Even more problematic seems to be the “production use allowed as long as the product doesn’t compete with a HashiCorp offering” clause. As far as I know, HashiCorp does not currently offer Nomad or Consul hosted services. But they do offer Terraform Cloud.

Terraform seems to be the biggest worry at the moment, at least from the posts I saw. There is already an OpenTF effort ongoing. It is first an appeal to HashiCorp to return Terraform to an Open Source license. But it also threatens to fork Terraform from the last MPL release if the license is not changed back:

If HashiCorp is unwilling to switch Terraform back to an open source license, we propose to fork the legacy MPL-licensed Terraform and maintain the fork in the foundation. This is similar to how Linux and Kubernetes are managed by foundations (the Linux Foundation and the Cloud Native Computing Foundation, respectively), which are run by multiple companies, ensuring the tool stays truly open source and neutral, and not at the whim of any one company.

What grates the most on me right now is how they did it from one day to the other, with pretty much zero warning and no grace period at all. Granted, they said they will backport fixes under the MPL to older releases for a while, but they could have at least gone for a more user-friendly approach with a proper grace period.

It’s going to be interesting to see how this sudden course change works out for them. It looks like they got hit as hard as everybody else by the end of the zero interest rate era and the subsequent drying up of capital.

Or it might have something to do with them going public sometime last year. They also had some lay-offs recently.

Might be that their business model simply doesn’t work out and they are now hoping for a cash infusion.

So what about the HashiCorp Homelab?

I’m honestly still undecided how to continue with my Homelab. From my understanding, the license change doesn’t actually impact me at all. And the one licensing aspect important to me is still there: Being able to look at the source code, which helped me a lot when investigating some weird behaviors over the years.

Of course, with community contributions very likely sharply declining, I’ve got no idea how the tools I’m currently using will develop.

But right now, I can only say this: At least for my Homelab use cases, the HashiCorp tools are absolutely excellent. Both their usability and their functionality are excellent, and there are a lot of far larger projects which would be served well by docs half as good as HashiCorp’s.

Their tools have been absolutely rock solid for me, and offered all of the features I could have dreamed up.

So my biggest problem is not that I expect anything to stop working, or that I don’t like their tools. Rather, it leaves a bad taste in my mouth using them now.

Alas…

So what to do now? My Orchestrator is HashiCorp Nomad. My service discovery and mesh networking is provided by HashiCorp Consul. My secrets for my Ansible playbooks and Nomad jobs are stored in HashiCorp Vault. The disk images for all of the hosts in my Homelab are created with HashiCorp Packer. I use HashiCorp Terraform to configure all of the other HashiCorp tools.

A part of me has been thinking about taking the lab into a really weird direction, namely deploying RedHat OpenShift. That might be a fun project. But tying my lab to yet another Corporation is probably not the smartest of ideas. 😉

So while I don’t want to tear everything down right now (I only finished the current iteration of the Homelab last year!), this is still a welcome trigger to dive into the deep end of Kubernetes.

Back when I first decided to expand my Homelab stack with an Orchestrator, I looked at Kubernetes first - and then bounced from the sheer complexity of the cluster setup. Specifically, I got to the “Choose a CNI plugin” section of the docs. And after reading all the glossy pages of the different offerings, I did not only not know which one to choose - I didn’t even know how to make that choice. But I want to give it at least a try now.

I’m not yet entirely decided on which Kubernetes Distribution to choose, but I’m leaning towards just bog-standard k8s. I don’t know why, but I’ve always leaned towards “give me the vanilla experience”, even when it’s more painful than the alternatives.

I would end up running the cluster on the same HW I’m using now. With eight Raspberry Pi CM4 making up the core of the workers, three Raspberry Pi 4 4GB as control nodes and three x86 machines which currently run my Ceph cluster.

That’s one thing I’m genuinely interested in: Rook Ceph, which is a Ceph cluster run inside a Kubernetes cluster. This would make the rather more beefy x86 machines available for other workloads as well, in what’s called a “hyperconverged” infra, if I’m not mistaken. One of my biggest worries about this part of the plan is whether I can somehow transform my current Ceph cluster into a Ceph Rook cluster running on Kubernetes.

But until then, there’s still a lot of research I have to do. The goal would be to end up with a cluster that does pretty much the same as my current one.

In no particular order, here is what I will have to figure out:

  • Service discovery, preferably via DNS and usable outside the cluster as well
  • Cluster networking with per-service permissions, to replace Consul Connect
  • Secrets management that’s not just usable inside the cluster, but also for things like my Ansible playbooks
  • Config file handling. I don’t really like the “Dump everything into env vars” approach for cloud native apps, and make heavy use of Nomad’s consul-template integration, also to e.g. insert secrets into config files
  • Ingress, which I would like to keep using Traefik for. Here it looks like that should be possible, as e.g. k3s already ships with Traefik for ingress by default
  • Job definitions in one place/file instead of the half a dozen Yaml files it looks like Kubernetes needs? Perhaps my very own Helm repo is the solution here?
  • Does k8s actually run properly on Raspberry Pis?
  • Cluster shutdowns and reboots, without having to stop every single job by hand
  • How to set up the networking? From what I understand, Kubernetes uses a separate cluster network

What I don’t think I will do is going the GitOps route via e.g. ArgoCD. To setup something like that, I would most likely need to run the necessary tools, including Gitea and all of its dependencies, outside the cluster. A thought I’ve never liked, because it means having to maintain two completely separate environments.

The Plan for now

To begin with, I will dig up my homeserver from the days when my Homelab only consisted of a single machine. That has an Intel i7 10700 8c/16t CPU. And if my inventory is correct, I should have 64 GB of unused RAM somewhere, as well as about 1TB worth of SATA SSDs.

I will most likely just put Ubuntu on it and run LXD for the VMs, just because I’m familiar with LXD VMs from my previous Homelab iteration. Then I will set up 3 4GB VMs for the controller nodes, because that’s how much RAM my three controller Raspberry Pis have. The rest will then go to the worker VMs.

The idea is to then experiment with the Kubernetes cluster until I have all moving parts I want in place, preferably including an example Ceph Rook cluster. That will give me enough time and space to spend some weekends experimenting with a Kubernetes Homelab.

Also, for the first time in a very long time, at least part of my Homelab will be for genuine experimentation, instead of being purely a HomeProd setup. :slight_smile:

Once all of that is done, the future direction of HashiCorp’s tools will likely have become a lot clearer, and I can then - after not even two years - decide the fate of the HashiCorp Homelab. Regardless of the outcome of my Kubernetes experiment, I had a lot of fun with the HashiCorp tools.