This is the second post in my series on the YaCy distributed, self-hosted search engine. The main topic this time is getting pages into the search index via crawling. In contrast to search engines like Google, Bing or Kagi, the content of the search index in YaCy is driven by its users. YaCy has an integrated web crawler to crawl pages and add them to the search index. It can be invoked in one of three ways: ...
Welcome to the newest rabbit hole I’ve found myself in. This post starts a new series where I’m taking a look at the YaCy self-hosted, distributed peer to peer search engine. And probably web crawling and search ranking algorithms. In this post, I will concentrate on how I deployed YaCy into my Kubernetes cluster, and a few pieces about my first steps with it. You won’t find answers to questions like “how good is it as a Google replacement?” in this post. There’s a lot more work ahead for me to actually make that judgment. ...