This is the second post in my series on the YaCy distributed, self-hosted search engine. The main topic this time is getting pages into the search index via crawling. In contrast to search engines like Google, Bing or Kagi, the content of the search index in YaCy is driven by its users. YaCy has an integrated web crawler to crawl pages and add them to the search index. It can be invoked in one of three ways: ...
I’m going to vent a bit in this one, mostly about myself. Just a few minutes ago, the following post scrolled past me in my Fedi timeline: Anybody self host their blog on their sbc or pc at home? Is there a good tutorial I can follow? I’m especially concerned about opening up ports or exposing my home ip to randos. I read that post and immediately thought: Running my blog on an SBC at home? I’m doing that! Concern about opening ports and “exposing” the home IP to the wider internet? I’ve certainly got opinions on that! ...
Welcome to the newest rabbit hole I’ve found myself in. This post starts a new series where I’m taking a look at the YaCy self-hosted, distributed peer to peer search engine. And probably web crawling and search ranking algorithms. In this post, I will concentrate on how I deployed YaCy into my Kubernetes cluster, and a few pieces about my first steps with it. You won’t find answers to questions like “how good is it as a Google replacement?” in this post. There’s a lot more work ahead for me to actually make that judgment. ...
After implementing a lot of helpers and basic infrastructure for my Smokeweb app (see here if you’re interested in what the project is about), I finally arrived at creating the database connection. In this post, I will give an overview of the Golang test helpers I implemented for tests which need a database, show you how I implemented some scripting for a Postgres database using Podman and then I will tell you how I implemented database tests in WoodpeckerCI. ...
In my $dayjob, I’m a build engineer in the CI team of a large company. So I’m reasonably confident that this is going to be only the first post in a long series on the CI setup for my Smoking project. I like CIs and the automated testing they come with. I think it was one of the better ideas the tech industry has come up with. I’m seeing its benefit every day at work. So I also have CIs for most of my private projects. ...