I’ve just had a major success: My docker-compose like Nomad script can now use the nomad binary with the job run -output command to transform a HCL file into JSON for use in the Nomad API. Before, my tool was using the Nomad API’s /v1/jobs/parse endpoint.

This meant that I was not able to make use of any of the HCL2 functions recently introduced. I’m mostly interested in using the file and fileset functions, and I want to tell you why.

Handling service config files in Nomad

What I want to talk about are not Nomad’s own config files or job specs. Instead, what I’m going to talk about are the config files for the services I’m running on Nomad. For example Gitea’s app.ini or Fluentd’s config files.

Somehow, you need to make sure that those are available to the service you’re running when it starts.

There are several obvious solutions to it. I could put those files onto the service’s CSI volume, for example. But an approach like this is sub-optimal, because I would also like to have my config files in a Git repo. And I would like to have them in the same Git repo as my Nomad job specs. And I would like to not put the entire repo into every job’s CSI volume. And what about jobs which otherwise don’t need any volumes? Just create one for that one config file?

Doesn’t sound right, does it?

Of course, Nomad has a solution to that. Several in fact. Let’s start with the simplest one: You can just put your config files into the job spec. The template stanza has the data option, for example. But that also seems wrong to me, because it would mix concerns in a single file: Configuring the service, and telling the orchestrator how to run that service.

So the next approach is the artifact stanza. It looks something like this:

job "docs" {
  group "example" {
    task "server" {
      artifact {
        source      = "https://example.com/file.tar.gz"
        destination = "local/some-directory"
        options {
          checksum = "md5:df6a4178aec9fbdc1d6d7e3634d1bc33"
        }
      }
    }
  }
}

It makes use of the go-getter lib, also by HashiCorp. It has a lot of different places which can be put into its source option, including HTTP, Git and S3. What it cannot do, when used in Nomad, is access random files on the Nomad host. This is obviously for security reasons, but it is also very restricting.

So my only way to get config files into the service’s containers was to provide them through the artifact stanza.

Now, if I were a GitOps kind of guy, this would be simple. I could use the artifact stanza to check out a Git repository with the service configs and do the following when working on a service config change:

Commit the potential change into the repo locally
Push the commit to my Gitea instance
Update the Nomad job on my machine with nomad job run
See that the job fails due to an error in the config file
Make another change. Either create a new commit, probably called “Attempt 1”, or do a force push
Launch the job again
See it fail again
Make another change. Either create a new commit, probably called “Attempt 2”, or do a force push
…

I hope you can see the problem already. If not, allow me to compliment your patience.

But that’s a relatively minor problem. The major problem with this approach: Neither my Gitea instance nor its dependencies, Postgres and Redis, can now run on the cluster. Because for Nomad to get Gitea’s config file, it needs to access the Gitea instance. Which is a problem.

So to spell out my goal: I wanted to be able to have a single Git repo somewhere, with both the service configs and the Nomad job specs. And I wanted to be able to make a change on disk and then, without committing or pushing or anything else, I wanted to be able to run nomad job run and have the local state of files be reflected in the newly started job.

I also wanted some way to not have to specify every single service config file in the Nomad job spec. I wanted to be able to just say: Here is an entire directory of configs, make them all available to the job.

So what I ended up doing was to acquire a goat. And an ancient dagger from a very old friend. And then I went onto a misty hilltop and summoned the following eldritch horror forth from the Warp.

How to use S3 as a filesystem

You have been warned.

From the Nomad side, I ended up using the artifact stanza with S3. I’ve already got S3 available via my Ceph cluster, and that Ceph cluster is considered to be one layer below my Nomad cluster in the stack - meaning the Ceph cluster works and bootstraps without needing anything from the Nomad cluster or any services running on it.

So in my jobs, handling the service configs looked something like this:

    task "foo" {
      driver = "docker"

      config {
        image = "foo:1.0"

        mount {
          type = "bind"
          source = "local/artifact_dl/config"
          target = "/etc/foo"
        }
      }

      artifact {
        source = "s3::https://example.com:4711/configs/foo/"
        mode = "dir"
        destination = "./local/artifact_dl/"
        options {
          region = "homenet"
        }
      }
      template {
        source = "./local/artifact_dl/templates/foobar.conf.templ"
        destination = "./local/artifact_dl/config/foobar.conf"
        change_mode = "restart"
        perms = "644"
      }
    }

This means that Nomad will download the foo/ directory in the configs/ bucket and put its contents into local/artifacts_dl/. That directory would look like this in my config git repository:

config
|- foo
|  |- config/
|  |  |- bar.conf
|  |  |- baz.conf
|  |- templates
|  |  |- foobar.conf.template
|  |- nomad
|  |  |- foo.hcl

So I would have all my files for a particular job in a separate directory. And only that subdirectory would be made available to the job during startup.

Now the last problem: How to get the automatic sync between my local working copy and the S3 bucket? The answer is s3fs. A FUSE tool to mount S3 as a POSIX(ish) filesystem. So on my desktop, I would run the following command:

s3fs -o passwd_file=/file/with/pw -o url=https://example.com:4711 -o use_path_request_style -o use_cache=/tmp/s3fs -o enable_noobj_cache -o multireq_max=500 -o parallel_count=50 configs /local/mount/path

And then I’ve got the S3 bucket’s content available locally. And you can even work with Git on this S3 backed filesystem.

A short aside for other Ceph users. Ceph is able to provide an NFS filesystem backed by an S3 bucket. This sounds like a way better solution, but it sadly does not work. This S3 backed NFS provider does not support all FS operations, it seems. Running git status on a Git repo on such a mounted NFS returns “not implemented” errors.

When making local changes, I can then just run nomad job run and get the newly started job to use the locally changed files - because they were synced back to the S3 bucket.

There are a number of problems with this approach. First, performance. Normal FS operations are okayish, but Git does a lot of FS operations. Even just a git status takes a couple of seconds to complete, and the repo isn’t that big. Even opening files has a perceivable delay. Then, note that I wrote above how I would get the changed config files in a newly started job. Nomad only checks whether the job spec has changed when determining whether it needs to update the job itself when running nomad job run.

You can work around that with the nomad alloc stop command. Interestingly, that command doesn’t really stop anything - it’s more of a stop and then start again operation. And that operation also downloads the artifacts again.

But hopefully, you can see that this is a clutch.

Doing it better

To get rid of this entire setup, I changed my tool to make use of the nomad binary to convert HCL job files to JSON files for use by the Nomad API.

This allows me to change the above Nomad file example into something like this:

    task "foo" {
      driver = "docker"

      config {
        image = "foo:1.0"

        mount {
          type = "bind"
          source = "local/artifact_dl/config"
          target = "/etc/foo"
        }
      }
      dynamic "template" {
        for_each = fileset(".", "foo/config/*")
        content {
          data = file(template.value)
          destination = "local/config/${basename(template.value)}"
          change_mode = "restart"
        }
      }
      template {
        data = file("foo/templates/foobar.conf.templ")
        destination = "./local/config/foobar.conf"
        change_mode = "restart"
        perms = "644"
      }
    }

And there we are. The dynamic keyword is part of the HCL2 expansion of HashiCorp’s HCL language. It dynamically creates stanzas. In this case, the above dynamic stanza is equivalent to:

template {
  data = file("foo/config/bar.conf")
  destination = "local/config/${basename(/foo/config/bar.conf)}"
  change_mode = "restart"
}
template {
  data = file("foo/config/baz.conf")
  destination = "local/config/${basename(/foo/config/baz.conf)}"
  change_mode = "restart"
}

The fileset just expands to a list of all the files according to the glob in its argument, in this case to all files in ./foo/config.

The file function just loads a file’s content. In this case, that content is loaded into the data parameter of template stanzas, which means inline template definitions - just loaded from a file on the local disk, on the operator machine, instead of the Nomad client filesystem during job startup.

Which is exactly what I needed. As I can now send the job spec and all service config files off to Nomad for execution as one package. And because the config files are now part of the job spec, from Nomad’s PoV, the job is automatically restarted whenever I update a config file.

There is one potential catch with this: I have read somewhere, a long while ago, that there is a size limit to how large a transmitted job spec can be. I can’t find the bug report/documentation right now, but I seem to remember that it was done to prevent Denial of Service attacks by sending overly large job specs. My largest config currently is my Fluentd config, spread over a number of files. It didn’t result in any problem, so for now I seem to be safe.

My next steps:

Migrating all of my jobs to use the new approach of using file for config file management
Merging the services repository into my main homelab repo

And the final step: Taking aforementioned ancient dagger and sending this wandering S3 horror back whence it came.

Handling service config files in Nomad#

How to use S3 as a filesystem#

Doing it better#

Handling service config files in Nomad

How to use S3 as a filesystem

Doing it better