This is part 17 of my k8s migration series.

I’ve been using Gitea as my Git forge for a while now. What’s now the Gitea instance started life is a Gogs instance in 2016, when I had to downsize my Homelab to a Raspberry Pi 3B that couldn’t handle all the things I wanted to run on it. I decided to get rid of my Gitlab instance and exchange it for Gogs. The switch back to Gitea then happened because Gitlab started eating 12% of my new home server’s CPU even when idle.

This is what the front page looks like when logged in:

I’m quite liking it and, in contrast to Gitlab, I never had any problems with it. It’s pretty snappy (again, especially in contrast to Gitlab) and relatively light on resources. Most of the time I can’t even tell whether it got assigned one of my beefier x86 nodes or a Raspberry Pi.

I’ve got 82 repositories stored in it, from relatively small dead projects which never got much farther than a README to extremely large repos containing 3D models and such for a Sins of a Solar Empire mod I was once involved in. Most repos don’t see a lot of activity and I’m the only user at the moment. The instance is not publicly accessible, but I might change that when the ForgeFed project matures.

My way of working depends on the repository. For my Homelab, this blog and my Homelab docs for example I’m just pushing to the master branch. (Which again reminds me to finally get around to the main branch migration.) In my development projects though I’m mostly working with Pull Requests. I find Gitea’s interface pretty convenient, and like seeing all the information and CI runs for a specific feature in one place.

So I’m not using too many actual features of Gitea, it’s mainly a convenient UI for my Git repos. But I must admit: I’m rather fond of that activity heat map. The only Gitlab feature I’m genuinely missing are the repository stats. If any of you know a good web app, either dynamic or statically generated, that can show stats on a Git repo, I’d be very interested.

Database setup and migration

I promise, this is the last time one of my migration articles will have a long-winded section on databases. 😉 But in this case, it’s warranted, because this is the first time I’m actually migrating a database, instead of setting up a new one.

I’m using CloudNativePG to manage the Postgres databases in my k8s cluster. More details can be found here. CNPG has a number of methods for seeding a new DB cluster with data. Generally, those approaches are split into two. The first way involves another online cluster and full replication or restoration of a backup. The second method is more suited to what I needed, namely a one-time import from another cluster using initdb to bootstrap the CNPG cluster. This method uses pg_dump/pg_restore from another running cluster. This method suited me somewhat well, because my Nomad Postgres setup is still up and running. The docs for this method can be found here.

There was just one problem: In Nomad, I’m using Consul Connect Service Mesh to connect services and only allow access between specific services instead of having open ports everywhere. This has been working pretty nicely in the past several years. Remember, I’m switching away from HashiCorp’s stuff not because their software is bad, but rather for ideological reasons.

But in this instance, I was stumped. For using pg_dump, CNPG needs access to the other cluster. But of course no k8s service is currently inside the Consul Mesh, so there’s no way to access the Postgres DB. I thought: Well, I can just open up a node port temporarily. And I failed. As in: I spend an entire evening trying to figure this out and had to give up. For reference, the network config for my Postgres Nomad job looks like this:

  group "postgres" {

    network {
      mode = "bridge"
    }

    service {
      name = "postgres"
      port = 5432

      connect {
        sidecar_service {}
      }

      check {
        type     = "script"
        command  = "/usr/bin/pg_isready"
        args     = ["-U", "postgres"]
        interval = "30s"
        timeout  = "2s"
        task     = "postgres"
      }
    }
[...]

With that config, Consul launches an Envoy container next to the Postgres container in the network namespace, by default on a random port. Inside the network namespace, Postgres’ 5432 port is connected to Envoy. Envoy then listens on a public port, but only lets through connections with the right mTLS cert. Other services can then be allowed to access Postgres via their own Envoy proxy. As best as I’ve been able to figure out, there’s no way for a service outside the mesh to get through the Envoy proxy to the Postgres port.

But opening another port also did not work. I’m reasonably sure that’s because trying to connect Postgres’ socket to two other sockets (the temporary public one, and Envoy’s) is just not something that can ever work. I still tried though. Pretty hard even.

But in the end I threw up my hands and had to admit that I was trying something that’s simply not possible. I could either have that port accessible on the node, or via the Consul Mesh, but not both.

I also couldn’t just temporarily switch off the Consul Mesh for Postgres, because that would have impacted other workloads on my Nomad cluster. Took me quite a while to come up with the solution: I remembered that, during my initial migration to Nomad from my Docker Compose setup, I had set up an Ingress Gateway to provide access to the already migrated services from the apps still running in Docker Compose. That Ingress Gateway does pretty much what it says on the tin: It allows services from outside the mesh access to services inside the mesh. It was of course not as fine-grained as the service mesh itself. If a service could reach the gateway, it could access all services inside the mesh that the gateway was allowed to access.

Luckily, by the time I originally set up the Ingress Gateway, I had already started to put my Homelab under version control, and I was still able to find the old Ingress Gateway definition. I pared it down to only Postgres, and the Nomad job ended up looking like this:

job "ingress-gateways" {
  datacenters = ["homenet"]

  group "internal" {
    network {
      mode = "bridge"

      port "postgres" {
        static = 5577
        to = 5577
      }
    }

    service {
      name = "ingress-internal"
      port = "8080"

      connect {
        gateway {
          ingress {
            listener {
              port = 5577
              protocol = "tcp"
              service {
                name = "postgres"
              }
            }
          }
        }
      }
    }
  }
}

This definition starts by setting up a bridged network namespace, meaning no outside access by default. Then it creates a listener for Postgres. With that, the Envoy proxy of the service would create a socket at port 5577 in the namespace, connected to the Postgres service’s Envoy proxy. The gateway would also open a static port on 5577 on the node it is running on, which would be connected to port 5577 inside the network namespace. And with that, any service connecting to port 5577 on the host running the Ingress Gateway would be connected to the Postgres database. Pretty neat and simple setup, but took me a while to remember.

I ran test connections with this command to confirm that I finally had external access to the cluster:

psql -U gogs -h ingress-internal.service.consul -p 5577 -d gitea

With that, I finally had access to the Postgres cluster from inside my k8s cluster.

The CNPG Cluster manifest then looks like this:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: gitea-pg-cluster
  labels:
    homelab/part-of: gitea
spec:
  instances: 2
  imageName: "ghcr.io/cloudnative-pg/postgresql:17.2"
  bootstrap:
    initdb:
      database: gitea
      owner: gitea
      import:
        type: microservice
        databases:
          - gitea
        source:
          externalCluster: nomad-pg
  resources:
    requests:
      memory: 200M
      cpu: 150m
  postgresql:
    parameters:
      max_connections: "200"
      shared_buffers: "50MB"
      effective_cache_size: "150MB"
      maintenance_work_mem: "12800kB"
      checkpoint_completion_target: "0.9"
      wal_buffers: "1536kB"
      default_statistics_target: "100"
      random_page_cost: "1.1"
      effective_io_concurrency: "300"
      work_mem: "128kB"
      huge_pages: "off"
      max_wal_size: "128MB"
      wal_keep_size: "512MB"
  storage:
    size: 1.5G
    storageClass: rbd-fast
  backup:
    barmanObjectStore:
      endpointURL: http://rook-ceph-rgw-rgw-bulk.rook-cluster.svc:80
      destinationPath: "s3://backup-cnpg/"
      s3Credentials:
        accessKeyId:
          name: rook-ceph-object-user-rgw-bulk-cnpg-backup-gitea
          key: AccessKey
        secretAccessKey:
          name: rook-ceph-object-user-rgw-bulk-cnpg-backup-gitea
          key: SecretKey
    retentionPolicy: "30d"
  externalClusters:
    - name: nomad-pg
      connectionParameters:
        host: ingress-internal.service.consul
        port: "5577"
        user: gogs
        dbname: gitea
      password:
        name: olddb-secret
        key: pw

I’ve omitted some standard things like the backup bucket setup here. The important parts for the migration are the spec.bootstrap.initdb.import and spec.externalClusters keys.

Let’s start with the externalClusters definition. It’s documented here and describes the connection to another cluster. This doesn’t need to be a CNPG cluster. One problem was that seemingly, no documentation exists of the externalClusters.connectionParameters.port option. I spend quite a while trying to figure out whether the port was supposed to go on the end of the host parameter, or whether it was a separate key. I was finally saved by the fact that CNPG is open source, and so I could look at the code - specifically a Yaml file from their test setup here. The password for the connection was coming from a Secret with the old database credentials. As you can see in the user parameter, the Gitea database was originally created during the Gogs phase of my Git hosting. 😁

The second part of the config is in spec.bootstrap.initdb.import, which tells CNPG what it should import from the external cluster. The first choice to make here is the type of the import. This describes the destination cluster, meaning the new CNPG cluster. The choices are microservice, meaning that the cluster serves only one app with one user, or monolith, meaning a cluster hosting the databases of multiple services. Besides that, I just needed to provide the name of the database in the source cluster and the name of said cluster in the externalClusters list.

This import, as configured above, worked immediately. I was very positively surprised. All data was imported properly, and CNPG automatically created the customary Secret with the connection details and credentials for accessing the cluster. After the initial import, I was able to remove the spec.bootstrap.initdb.import and externalClusters keys completely from the manifest without any error.

Helm Chart

For the Gitea deployment itself, I made use of the official Helm chart. It is one of the better ones I’ve encountered since starting the migration, providing the ability to set config options in the values.yaml instead of having to maintain a separate app.init file. What I value extremely highly is that they provide the ability to add environment variables via env.ValueFrom, so I can directly use the automatically created Secrets from CNPG for the DB and Rook Ceph for the S3 bucket. This safes me the roundabout setup I had to do for other charts, where I had to use external-secrets to template the auto Secrets into new Secrets with a different format to conform to the Chart’s expectations.

One downside at the time of writing is that the chart is not on the newest Gitea version 1.23.1. But I just changed the image tag and chart version 10.6.0 worked without issue. Going by this GitHub issue the delay is simply due to some internal refactoring of the chart they want to finish before the next release.

I will split the exploration of my values.yaml file into two parts, one with the Gitea config under the gitea key and one for everything else.

Everything besides the Gitea config

Let’s start with the “everything else” part:

replicaCount: 1
image:
  rootsless: true
  tag: "1.23.1"

strategy:
  type: Recreate

containerSecurityContext:
  capabilities:
    add:
      - SYS_CHROOT

service:
  ssh:
    type: LoadBalancer
    port: 2222
    externalTrafficPolicy: Local
    annotations:
      external-dns.alpha.kubernetes.io/hostname: git.example.com
    labels:
      homelab/public-service: "true"

ingress:
  enabled: true
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: very-safe-entrypoint
  hosts:
    - host: gitea.example.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - hosts:
      - gitea.example.com

resources:
  requests:
    cpu: 1
  limits:
    memory: 1500Mi

persistence:
  enabled: true
  create: false
  mount: true
  claimName: gitea-data-volume

signing:
  enabled: false

actions:
  enabled: false

redis-cluster:
  enabled: false

postgresql-ha:
  enabled: false

postgresql:
  enabled: false

As I noted above, I had to hardcode the Gitea tag for now, because the newest chart version is still on 1.22.3. I also opted for using the rootless image, which I did not use in the Nomad job. It just is a little bit nicer than having a root-capable image, although Gitea automatically drops root privileges at startup and even with the root image, Gitea doesn’t actually run as root.

I also had to hardcode the update strategy to Recreate. I’m not sure why it’s set to rolling updates by default. This doesn’t really work, because Gitea is launched in a Deployment and has a PVC mounted, so the newly started instance won’t actually be able to start because the RWO volume can’t be mounted in two Pods at the same time.

Then comes an important one, the SYS_CHROOT capability. This is documented as required when using cri-o as the container runtime in the Helm chart’s extensive README.md.

External access is split between two different subdomains, one for Gitea’s web frontend going through my Traefik ingress and one for git access with SSH. I like setting my LoadBalancer services, provided by Cilium, to the Local externalTrafficPolicy. This ensures that the client IP those services see are the actual client IPs, and not the IPs of the machine which received the request and forwarded it to the service it was intended for. The homelab/public-service label is simply a sign for Cilium that it should handle the service.

The ingress config has one important point, the tls key. I initially did not set that key, because I’ve never set it before - my Traefik automatically uses HTTPS and I’m using a wildcard cert. But Gitea needs to generate publicly addressable URLs for some things, e.g. when providing clone URLs for HTTPS or providing callback URLs in webhooks, e.g. for Woodpecker. The domain itself is fine, but the chart determines the protocol like this:

{{- define "gitea.public_protocol" -}}
{{- if and .Values.ingress.enabled (gt (len .Values.ingress.tls) 0) -}}
https
{{- else -}}
{{ .Values.gitea.config.server.PROTOCOL }}
{{- end -}}
{{- end -}}

In there, https is only set as the protocol if the ingress.tls list has at least one entry. And setting the server.PROTOCOL config comes with it’s own problems, so I decided to just add the tls.hosts setting, even if it means that I have to repeat my Gitea domain a number of times in the config.yaml.

For the persistence setting I had to go with a pre-created volume, because I had to make the migrated Gitea data available. One thing to note here is that you should delete the app.ini file your previous setup might have left on the disk. The init container which puts together the app.ini from the values and env variables and so on doesn’t handle it well when there’s an existing app.ini it didn’t create itself.

I also disabled a number of features I didn’t need, like signing or actions or the Redis and Postgres instances the Helm chart can deploy, because I’ve already got my own deployments.

The Gitea config

Here’s the full gitea: section of the config.yaml, just for reference. I will post the relevant subsections as I go over them:

gitea:
  admin:
    existingSecret: null
    username: null
    password: null
  metrics:
    enabled: false
  oauth:
    - name: "Keycloak"
      provider: "openidConnect"
      existingSecret: oidc-credentials
      autoDiscoverUrl: "https://key.example.com/realms/homelab/.well-known/openid-configuration"
  config:
    APP_NAME: "My Gitea"
    RUN_MODE: "prod"
    server:
      SSH_SERVER_HOST_KEYS: "ssh/gitea.ed25519"
      APP_DATA_PATH: "/data/gitea_data"
      SSH_DOMAIN: "git.example.com"
      SSH_PORT: 2222
    database:
      DB_TYPE: "postgres"
      LOG_SQL: false
    oauth2:
      ENABLED: true
    service:
      DISABLE_REGISTRATION: true
      REQUIRE_SIGNIN_VIEW: true
      DEFAULT_KEEP_EMAIL_PRIVATE: true
      DEFAULT_ALLOW_CREATE_ORGANIZATION: true
      DEFAULT_ORG_VISIBILITY: true
      DEFAULT_ORG_MEMBER_VISIBLE: false
      DEFAULT_ENABLE_TIMETRACKING: true
      SHOW_REGISTRATION_BUTTON: false
    repository:
      ROOT: "/data/git-repos"
      SCRIPT_TYPE: bash
      DEFAULT_PRIVATE: private
      DEFAULT_BRANCH: main
    ui:
      DEFAULT_THEME: gitea-auto
    queue:
      TYPE: redis
      CONN_STR: "addr=redis.redis.svc.cluster.local:6379"
      WORKERS: 1
      BOOST_WORKERS: 5
    admin:
      DEFAULT_EMAIL_NOTIFICATIONS: disabled
    openid:
      ENABLE_OPENID_SIGNIN: false
    webhook:
      ALLOWED_HOST_LIST: private
    mailer:
      ENABLED: true
      SUBJECT_PREFIX: "[Gitea]"
      SMTP_ADDR: mail.example.com
      SMTP_PORT: "465"
      FROM: "gitea@example.com"
      USER: "gitea@example.com"
    cache:
      ADAPTER: "redis"
      INTERVAL: 60
      HOST: "network=tcp,addr=redis.redis.svc.cluster.local:6379,db=0,pool_size=100,idle_timeout=180"
      ITEM_TTL: 7d
    session:
      PROVIDER: redis
      PROVIDER_CONFIG: network=tcp,addr=redis.redis.svc.cluster.local:6379,db=0,pool_size=100,idle_timeout=180
    time:
      DEFAULT_UI_LOCATION: "Europe/Berlin"
    cron:
      ENABLED: true
      RUN_AT_START: false
    cron.archive_cleanup:
      ENABLED: true
      RUN_AT_START: false
      SCHEDULE: "@every 24h"
    cron.update_mirrors:
      ENABLED: false
      RUN_AT_START: false
    cron.repo_health_check:
      ENABLED: true
      RUN_AT_START: false
      SCHEDULE: "0 30 5 * * *"
      TIMEOUT: "5m"
    cron.check_repo_stats:
      ENABLED: true
      RUN_AT_START: true
      SCHEDULE: "0 0 5 * * *"
    cron.update_migration_poster_id:
      ENABLED: true
      RUN_AT_START: true
      SCHEDULE: "@every 24h"
    cron.sync_external_users:
      ENABLED: true
      RUN_AT_START: false
      SCHEDULE: "@every 24h"
      UPDATE_EXISTING: true
    cron.deleted_branches_cleanup:
      ENABLED: true
      RUN_AT_START: true
      SCHEDULE: "@every 24h"
    migrations:
      ALLOW_LOCALNETWORKS: true
    packages:
      ENABLED: false
    storage:
      STORAGE_TYPE: minio
      MINIO_ENDPOINT: rook-ceph-rgw-rgw-bulk.rook-cluster.svc:80
      MINIO_LOCATION: ""
      MINIO_USE_SSL: false

  additionalConfigFromEnvs:
    - name: GITEA__DATABASE__HOST
      valueFrom:
        secretKeyRef:
          name: gitea-pg-cluster-app
          key: host
    - name: GITEA__DATABASE__NAME
      valueFrom:
        secretKeyRef:
          name: gitea-pg-cluster-app
          key: dbname
    - name: GITEA__DATABASE__USER
      valueFrom:
        secretKeyRef:
          name: gitea-pg-cluster-app
          key: user
    - name: GITEA__DATABASE__PASSWD
      valueFrom:
        secretKeyRef:
          name: gitea-pg-cluster-app
          key: password
    - name: GITEA__SECURITY__SECRET_KEY
      valueFrom:
        secretKeyRef:
          name: secret-key
          key: key
    - name: GITEA__OAUTH2__JWT_SECRET
      valueFrom:
        secretKeyRef:
          name: jwt-secret
          key: jwt
    - name: GITEA__MAILER__PASSWD
      valueFrom:
        secretKeyRef:
          name: mail-pw
          key: pw
    - name: GITEA__STORAGE__MINIO_BUCKET
      valueFrom:
        configMapKeyRef:
          name: gitea-bucket
          key: BUCKET_NAME
    - name: GITEA__STORAGE__MINIO_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: gitea-bucket
          key: AWS_ACCESS_KEY_ID
    - name: GITEA__STORAGE__MINIO_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: gitea-bucket
          key: AWS_SECRET_ACCESS_KEY

Let’s start with the gitea.admin config:

gitea:
  admin:
    existingSecret: null
    username: null
    password: null

I’ve already got an admin account, so I didn’t want the Helm chart to create a new one. I thought I could do that by just setting admin: {}, but that of course doesn’t work. So the Helm chart created an admin user with the chart’s default gitea.admin.password. But I then figured out that setting all values to null does work. It’s important to note that Gitea doesn’t then remove the newly created admin user again. It needs to be deleted manually via the UI.

The gitea.oauth config is also worth a paragraph. First, it’s important to note that this is the config for Gitea as an Oauth2 client. The config for Gitea as an identity provider has to be done in another place. I’m using Keycloak as my identity provider in the Homelab. For more details, see this post. The issue is that Gitea’s OAuth2 client config can only be done in the UI or via the CLI, not via the config file. And I had already taken my Nomad instance down at this point. I could get the client ID and secret from Keycloak, but not the name under which it was saved in the database for example. It was also pretty unclear what options should be set under the gitea.oauth key. I finally ended up looking into the init container script, which is a bash script using the Gitea CLI to create the OAuth2 entry:

    function configure_oauth() {
      {{- if .Values.gitea.oauth }}
      {{- range $idx, $value := .Values.gitea.oauth }}
      local OAUTH_NAME={{ (printf "%s" $value.name) | squote }}
      local full_auth_list=$(gitea admin auth list --vertical-bars)
      local actual_auth_table=''

      # We might have distorted output due to warning logs, so we have to detect the actual user table by its headline and trim output above that line
      local regex="(.*)(ID\s+\|Name\s+\|Type\s+\|Enabled.*)"
      if [[ "${full_auth_list}" =~ $regex ]]; then
        actual_auth_table=$(echo "${BASH_REMATCH[2]}" | tail -n+2) # tail'ing to drop the table headline
      else
      [...]
      fi

      local AUTH_ID=$(echo "${actual_auth_table}" | grep -E "\|${OAUTH_NAME}\s+\|" | grep -iE '\|OAuth2\s+\|' | awk -F " "  "{print \$1}")

      if [[ -z "${AUTH_ID}" ]]; then
        echo "No oauth configuration found with name '${OAUTH_NAME}'. Installing it now..."
        gitea admin auth add-oauth {{- include "gitea.oauth_settings" (list $idx $value) | indent 1 }}
        echo '...installed.'
      else
        echo "Existing oauth configuration with name '${OAUTH_NAME}': '${AUTH_ID}'. Running update to sync settings..."
        gitea admin auth update-oauth --id "${AUTH_ID}" {{- include "gitea.oauth_settings" (list $idx $value) | indent 1 }}
        echo '...sync settings done.'
      fi
      {{- end }}
      {{- else }}
        echo 'no oauth configuration... skipping.'
      {{- end }}
    }

    configure_oauth

I’ve removed some unimportant bits for the sake of brevity (heh, brevity 😂). What we can see here is that the entries in the gitea.oauth section are converted into CLI flags and their parameters 1:1. For finding the right options I used for my Keycloak setup, I ended up looking into the database:

\c gitea
SELECT * FROM login_source;

id | type |   name   | is_sync_enabled |                                                                                                                                                                                               cfg                                                                                                                                                                                               | created_unix | updated_unix | is_active
----+------+----------+-----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+--------------+-----------
  1 |    6 | Keycloak | f               | {"Provider":"openidConnect","ClientID":"bar","ClientSecret":"foo","OpenIDConnectAutoDiscoveryURL":"https://key.example.com/realms/homelab/.well-known/openid-configuration","CustomURLMapping":null,"IconURL":"","Scopes":null,"RequiredClaimName":"","RequiredClaimValue":"","GroupClaimName":"","AdminGroup":"","RestrictedGroup":"","SkipLocalTwoFA":true} |   1678573526 |   1678573526 | t
(1 row)

But this still left the question of how the gitea.oauth.existingSecret should be formatted. Which keys was the chart expecting the Secret to have? I wasn’t able to find any info, so I ended up looking first for the place where the gitea.oauth_settings from the init script above was defined, which lead me to the chart’s helpers again:

{{- define "gitea.oauth_settings" -}}
{{- $idx := index . 0 }}
{{- $values := index . 1 }}

{{- if not (hasKey $values "key") -}}
{{- $_ := set $values "key" (printf "${GITEA_OAUTH_KEY_%d}" $idx) -}}
{{- end -}}

{{- if not (hasKey $values "secret") -}}
{{- $_ := set $values "secret" (printf "${GITEA_OAUTH_SECRET_%d}" $idx) -}}
{{- end -}}

{{- range $key, $val := $values -}}
{{- if ne $key "existingSecret" -}}
{{- printf "--%s %s " ($key | kebabcase) ($val | quote) -}}
{{- end -}}
{{- end -}}
{{- end -}}

Here, the key and secret values, if not defined in the chart, are set to the GITEA_OAUH_KEY_$ID and GITEA_OUATH_SECRET_$ID env variables. Looking for those variables then lead me to the Deployment template:

- name: GITEA_OAUTH_KEY_{{ $idx }}
  valueFrom:
    secretKeyRef:
      key:  key
      name: {{ $value.existingSecret }}
- name: GITEA_OAUTH_SECRET_{{ $idx }}
  valueFrom:
    secretKeyRef:
      key:  secret
      name: {{ $value.existingSecret }}

And here I finally had my answer: The Secret should have a key key and a key secret for the two values. Armed with that info I could finally define the OAuth2 options:

gitea:
  oauth:
    - name: "Keycloak"
      provider: "openidConnect"
      existingSecret: oidc-credentials
      autoDiscoverUrl: "https://key.example.com/realms/homelab/.well-known/openid-configuration"

One thing which annoyed me is in Gitea’s S3 config:

gitea:
  config:
    storage:
      STORAGE_TYPE: minio
      MINIO_ENDPOINT: rook-ceph-rgw-rgw-bulk.rook-cluster.svc:80
      MINIO_LOCATION: ""
      MINIO_USE_SSL: false

The MINIO_ENDPOINT needs to have the host and port in one value. But the ConfigMap created by Rook for a new bucket contains them only in separate keys, meaning I had to hardcode the value in the values.yaml instead of taking it from the ConfigMap. But at least I could still use the Secret Rook creates to get the S3 credentials:

gitea:
  additionalConfigFromEnvs:
    - name: GITEA__STORAGE__MINIO_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: gitea-bucket
          key: AWS_ACCESS_KEY_ID
    - name: GITEA__STORAGE__MINIO_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: gitea-bucket
          key: AWS_SECRET_ACCESS_KEY

An option like this is what more Helm charts should have: The ability to use the valueFrom form of defining env variables. With this, I can easily use autogenerated Secrets and ConfigMaps without having to jump through hoops.

Next stumbling block was the redis config:

gitea:
  config:
    cache:
      ADAPTER: "redis"
      INTERVAL: 60
      HOST: "network=tcp,addr=redis.redis.svc.cluster.local:6379,db=0,pool_size=100,idle_timeout=180"
      ITEM_TTL: 7d
    session:
      PROVIDER: redis
      PROVIDER_CONFIG: network=tcp,addr=redis.redis.svc.cluster.local:6379,db=0,pool_size=100,idle_timeout=180
    queue:
      TYPE: redis
      CONN_STR: "addr=redis.redis.svc.cluster.local:6379"
      WORKERS: 1
      BOOST_WORKERS: 5

Here I wasn’t aware that the connection string has to have a certain format and isn’t just host:port. Took me a while to figure out why I wasn’t able to make a connection with Redis.

And finally, another word on YAML: Check your indentation! 😅 I had to make sure that the entire network could reach the SSH service so I could actually use it for git operations. So I added fromEntities:\n - world to the network policy:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "gitea-access"
spec:
  endpointSelector:
    matchExpressions:
      - key: "app.kubernetes.io/name"
        operator: In
        values:
          - "gitea"
  ingress:
    - fromEndpoints:
      - matchLabels:
          homelab/ingress: "true"
          io.kubernetes.pod.namespace: traefik-ingress
      - fromEntities:
          - world

And when I still could not connect, I checked with Cilium’s monitoring:

kubectl -n kube-system exec -ti cilium-vh5jj -- cilium monitor --type drop
xx drop (Policy denied) flow 0x0 to endpoint 896, ifindex 5, file bpf_lxc.c:2067, , identity world->63410: 300.300.300.1:59774 -> 10.8.5.79:2222 tcp SYN
xx drop (Policy denied) flow 0x0 to endpoint 896, ifindex 5, file bpf_lxc.c:2067, , identity world->63410: 300.300.300.1:59774 -> 10.8.5.79:2222 tcp SYN
xx drop (Policy denied) flow 0x0 to endpoint 896, ifindex 5, file bpf_lxc.c:2067, , identity world->63410: 300.300.300.1:59774 -> 10.8.5.79:2222 tcp SYN
xx drop (Policy denied) flow 0x0 to endpoint 896, ifindex 5, file bpf_lxc.c:2067, , identity world->63410: 300.300.300.1:59774 -> 10.8.5.79:2222 tcp SYN
xx drop (Policy denied) flow 0x0 to endpoint 896, ifindex 5, file bpf_lxc.c:2067, , identity world->63410: 300.300.300.1:59774 -> 10.8.5.79:2222 tcp SYN
xx drop (Policy denied) flow 0x0 to endpoint 896, ifindex 5, file bpf_lxc.c:2067, , identity world->63410: 300.300.300.1:59774 -> 10.8.5.79:2222 tcp SYN
xx drop (Policy denied) flow 0x0 to endpoint 896, ifindex 5, file bpf_lxc.c:2067, , identity world->63410: 300.300.300.1:59774 -> 10.8.5.79:2222 tcp SYN
xx drop (Policy denied) flow 0x0 to endpoint 896, ifindex 5, file bpf_lxc.c:2067, , identity world->63410: 300.300.300.1:59774 -> 10.8.5.79:2222 tcp SYN

Fast forward through an hour of reading through Cilium’s network policy docs, and I took another look at the policy - and realized that I had screwed up the indentation. 🤦 It should of course look like this:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "gitea-access"
spec:
  endpointSelector:
    matchExpressions:
      - key: "app.kubernetes.io/name"
        operator: In
        values:
          - "gitea"
  ingress:
    - fromEndpoints:
      - matchLabels:
          homelab/ingress: "true"
          io.kubernetes.pod.namespace: traefik-ingress
    - fromEntities:
        - world

With the fromEntities entry in the ingress: list, not the fromEndpoints: list. And after that it was all up and running. Woodpecker, my CI, did not need any additional config to access Gitea, it worked out of the box. Likely because it uses HTTPS for Git access and goes through the standard Gitea URL. And I don’t think I can change that to have it use the internal service instead of going through the ingress. That’s because it also uses Gitea for auth, and I don’t think it will handle having two different URLs to access Gitea very well. But that still ended up on the rickety pile of Homelab tasks to look at at some point.

Overall, it was a good migration and allowed me to figure out my DB migration strategy with a service which I could do without for a couple of days. I also have to congratulate the Gitea community on their work on the Helm chart. It was definitely one of the better ones I’ve used.

And that’s it for today. I can’t say what’s going to be next on the migration list, as I haven’t decided yet. I first thought to migrate my IoT services, Mosquitto, zigbee2mqtt and friends, but I’d also like to tackle some of the bigger items, like Nextcloud. On the other hand, I’m really not looking forward to touching my Nextcloud deployment. It has been working so nicely.