In contrast to search engines like Google, Bing or Kagi, the content of the search index in YaCy is driven by its users. YaCy has an integrated web crawler to crawl pages and add them to the search index. It can be invoked in one of three ways:

1. Manually via a web interface in YaCy
2. Automatically via scheduled (re-) crawls
3. Automatically by using YaCy as a local web proxy

I will concentrate on the first way in this post. The second way, with scheduled repeats of a previous manual crawl will be described later, when I get into properly setting up and using YaCy longer term.

The third way is somewhat interesting. It avoids having to manually crawl interesting pages by configuring YaCy as a local web proxy and using it for your normal internet browsing. YaCy will then automatically launch crawls for any pages you visit. I find that a little bit too Big Brother, even though it’s all running in my own Homelab and won’t use this approach.

The manual crawl start page looks like this:

That’s certainly a lot of options. Let’s go through the most important ones. The first setting to configure is the starting point. This is a list of URLs, which can also contain paths to restrict the crawl to sub-paths. Once a URL is provided, YaCy will reach out to the page and see whether it contains a Sitemap. If so, it will allow using said Sitemap to start the crawl. Finally, the crawl can also be started from a file containing a list of URLs.

The index attributes are not too interesting, I just always used the default index.

The crawl filter section is where it really gets interesting. The configuration of the crawl depth is quite important, as it sets how deep the crawl will actually go. I will talk more about it later. I never used the page count configs, but I guess it’s a tool to get out of some sort of endless loop? The constraints will be their own topic later on. The load filter for URLs is a lot more interesting. It can either restrict to the initial domains or even sub-paths of the initial list of URLs. Instead, any regex can also be used to manually restrict the URLs. More on this also later on.

I never touched the ‘Document Filter’ at all, never saw a reason to. Same for the content filter.

The ‘Clean-Up before Crawl Start’ is more interesting. It takes the crawl start URLs and can delete all pages for those domains or sub-paths from the index. This also makes sure that they’re all re-crawled. Instead of deleting the content outright, the ‘Double-Check Rules’ section can be used to re-crawl only pages of a certain age.

I never did anything with the ‘Document Cache’ up to now. And the Robot Behaviour will be discussed in the next section.

So let’s dive a bit deeper into what happens when we click the ‘Start New Crawl Job’ button.

Crawling Basics

To start with: The crawler implementation of YaCy has a default delay between crawling the same domain of 250 ms, meaning it will produce at most 4 req/s and 240 req/min. This value can be increased by crawling multiple sites in parallel, because the restriction is only per domain, not for all crawling.

Another thing I find a little bit sad is that by default, YaCy uses random browser user agents. But there is an option to change that in the “Robot Behaviour” section. It does respect the robots.txt, including the Crawl-delay directive.

As the existence of the “Double-Check Rules” section indicates, YaCy does not normally visit pages which it already has in its index again. This can be worked around in two ways. The first is to delete all pages for the domain or path given in the crawl URL list. The second way is via re-crawling only pages which have last been crawled a certain time ago.

During my own crawls up to this point, I always decided to only crawl within a certain domain or even just a certain subpath. I don’t see too much usefulness in doing wide Internet crawls. For the few times I tried that, e.g. for this blog, there were a lot of random pages added to the index pretty fast. To make a per-domain crawl, I would advise not to use Restrict to start domains option directly. That’s because that option will also exclude subdomains, which I at least always want to include in the crawl. For example, when crawling ceph.io I would definitely also want to crawl docs.ceph.io. Instead, I use the Use filter option, with a filter of ^(.*\.)?ceph\.io.*$. You might be wondering about the (.*\.)? at the beginning. This is to make sure that only subdomains are caught, not variations on the domain. I came up with this when I wanted to crawl the homepage of my old University town, paderborn.de. I first launched it with a filter of .*paderborn.de. But this also caught stuff like company-paderborn.de, which I didn’t want.

When it comes to the crawling depth, I always use .* in the “Unlimited crawl depth for URLs matching with” field. Through the use of the filter, I already ensured that I will only get results from the one domain I want to crawl, so just using .* here is fine because I really want to crawl the entire page. One important thing regarding the crawl depth I still have to look up in the code is whether YaCy somehow stores the depth for pages which have already been crawled. As I’ve noted above, YaCy does normally not re-crawl pages it has already seen. But this would of course be unfortunate if the same page is later visited with a greater crawl depth, but then gets ignored because it was already seen.

Another point I learned pretty quickly: I have to keep an eye on the crawl and the URLs it visits. One example is GitHub, where my initial crawls happily crawled the /commits, /blobs and /tree endpoints, resulting in a very large amount of pages in the index which I really wasn’t interested in. Another example is faz.net, a German newspaper. Their page has the subpath /kaufkompass. It contains product pages for hundreds of thousands of products, and is again not something I’m even remotely interested in crawling. Yet another example was the homepage of the city I’m living it. It has an event calendar going back over a decade. Also lots and lots of pages to crawl. So it’s always worth it to click around a bit on any given page before launching the crawl, to see whether there’s something to exclude.

Let’s next take a deeper look at two additional examples.

Crawling a GitHub project

One of the places my Google searches lead me quite often are the issues and pull request pages of GitHub, especially when searching for error messages. So I wanted to crawl the GitHub pages of all of the projects I’m using, for example everything under https://github.com/yacy/yacy_search_server. As I’ve noted above, this needs some exclusions to avoid crawling the Git repository. In my case, what worked quite nicely was adding .*/(tree|commits|blob|changes|actions|commit)/.* as a must-not-match URL filter. One issue I came across while trying to crawl GitHub was that the pagination system has the pages as query parameters. This is a problem because I normally disable URLs with query parameters, as they generally represent dynamic content. But in the case of pagination for e.g. issues or pull requests, I do of course want the crawler to follow them. So I enabled the Accept URLs with query-part checkbox. This worked as expected, but now I had a new problem: The crawler started fetching all possible filter permutations for the issue and pull request filters, for example https://github.com/yacy/yacy_search_server/issues?q=is%3Aissue%20state%3Aopen%20author%3Apdstefan%20label%3Aindex.

These end up in the crawl queue because the links to the different filters are delivered by GitHub when opening the issues or PR pages. I will want to work around this, but haven’t had the time to come up with a proper regex yet. Another thing I need to check is whether those links might have the nofollow tag, which I’ve currently got the crawler ignoring.

Then there’s the question whether doing crawling via the GitHub API might not be “nicer” to GitHub than fetching the actual pages and listing all the time. Of course, I really couldn’t care less how much my crawling costs the LLM pushers from Microsoft GitHub. But then again, your ethics are determined by how you treat your enemies, not by how you treat your friends. This leads directly to the next question: How to handle this? I’m assuming that GitHub might not be the only page which might be easier to crawl via its API. But I also don’t think it’s really worth it to implement all sorts of special cases into YaCy’s crawler directly. This is something which obviously calls for some sort of plugin model. So I will have to figure out how to feed already crawled pages into YaCy’s index via an API. Or perhaps to introduce a YaCy API for crawling plugins.

Respecting HTTP code 429

Another target I tried to crawl was pkg.go.dev, because I’m currently working on a Go webapp and I find myself searching for Go packages and functions quite often. During this crawl, I was surprisingly getting back HTTP code 429, too many requests. Of course as I mentioned above, I want to be a well-behaved crawler. So I first verified again whether the YaCy crawler was properly crawling at, at most, a rate of 4 req/s. And it was. It just seems that pkg.go.dev has quite a low ceiling for engaging its rate limiters.

To my surprise, instead of simply slowing down, YaCy just retried the error’ing pages a few more times and then just gave up on them, meaning I was missing some pages and getting others.

This lead me to spelunking through the YaCy code again, and finding that it just didn’t have any special 429 return code handling. At all. Here is what I believe to be the relevant code, see the code on GitHub:

if (statusCode > 299 && statusCode < 310) {
// ...
} else if (statusCode == HttpStatus.SC_OK || statusCode == HttpStatus.SC_NON_AUTHORITATIVE_INFORMATION) {
// ...
} else {
    // client.close(); // explicit close caused: warning: [try] explicit call to close() on an auto-closeable resource
    // if the response has not the right response type then reject file
    this.sb.crawlQueues.errorURL.push(request.url(), request.depth(), profile,
            FailCategory.TEMPORARY_NETWORK_FAILURE, "wrong http status code", statusCode);
    throw new IOException("REJECTED WRONG STATUS TYPE '" + statusline
            + "' for URL '" + requestURLString + "'$");
}

So this behavior is imminently fixable by implementing exponential back-off once 429 return codes start showing up.

Special thanks also to Remy on mastodon.social, who pointed out the existence of the Retry-After header. Sadly, pkg.go.dev does not send along that particular header with the 429 responses.

Thoughts on launching crawls

As I’ve noted in the introduction, there are a few ways to launch a crawl. The main problem I see is that they’re all not too great for daily, long-term use.

Sure, I can launch crawls for all of the pages I regularly visit and fill the index up front. But what about re-crawls, e.g. for new versions of documentations, or catching the content of new issues in GitHub projects? How to make sure that I don’t have to somehow remember all the configurations for all of the pages?

YaCy does have a page for repeating previous crawls, and it supposedly also can create and edit crawling profiles, but I at least haven’t been able to actually make that work. Just for the reason that I haven’t found the button for actually creating a profile yet. 🤷

Beyond that, what I would really like is a way to trigger crawl with a specific profile for the page I’ve currently open in a browser. Because having to stop reading a page and switching to YaCy to trigger a crawl would probably get old pretty fast. So this is another thing I’d like to look into: First, making sure I can properly create and manage crawling profiles, and then perhaps introducing a browser extension from which to trigger a crawl for the current page, allowing me to chose the profile.

Another interesting feature to look into regarding automating (re-)crawls would be YaCy’s RSS import feature. It can seemingly ingest an RSS feed and then regularly fetch it and crawl all new entries. This could be very nice to not have to regularly re-crawl entire blogs, and would also be pretty nice for news sites. And it might even partially work for GitHub issues and PRs, as I think there are RSS feeds available for those as well. The main problem here would be that one would want to re-crawl issues from time to time to catch newer comments.

Some general thoughts on crawling

It’s not the early 2010s anymore. The Internet has become a lot more hostile to anything self-hosted since the advent of LLMs. Personal sites and open source projects alike are getting buried under the indiscriminate incompetence of the LLM bros with more money than brains, sending their scrapers to download the exact same site 1000x times in a row, or traversing the entirety of a Git repository via HTTP - multiple times per minute. As a consequence, many people and projects have started erecting barriers, reaching from CloudFlare to projects like Iocaine or Anubis to block crawlers. But of course the same blockers also work perfectly fine against my crawling.

So even the general viability of any search engine which doesn’t have the heft of Google or Microsoft behind it seems pretty questionable now.

Besides that, I’m also wondering about how YaCy handles the distributed nature of the crawling. Let’s take codeberg.org as an example. Considering the type of people who would be interested in a distributed search engine, I’d judge the probability that multiple of us will try to crawl codeberg reasonably high. But this seems patently unnecessary, as we would just add the same pages twice to the global index. While at the same time adding undue load on the infrastructure of a non-profit doing good things in providing an alternative to GitHub. Probably not an issue with the currently very low number of running YaCy instances. But still something to think about if it ever gets more popular.

Backburner

As you can see from all of the ideas for improvements from above, there’s a lot to potentially do to make YaCy more useful. But especially the receding willingness to allow crawling by random crawlers has given me pause. Plus, I would have to invest a lot more time and make more contributions to YaCy’s code. Which is fine, but introducing YaCy to my stack was only supposed to be a short break from my Smokeweb project.

So for now, I will put the project on the backburner, and return to it once I’ve finished the majority of my work on Smokeweb.

In this post, I will concentrate on how I deployed YaCy into my Kubernetes cluster, and a few pieces about my first steps with it. You won’t find answers to questions like “how good is it as a Google replacement?” in this post. There’s a lot more work ahead for me to actually make that judgment.

You can find this post and any future ones in the series under the YaCy tag.

A fair warning before I continue: The project accepts slopcoded contributions.

It currently doesn’t look like there’s a large team behind it, but there’s a community forum with some activity, although new signups are currently broken due to an issue with the mail server. The last release was in March, and PRs are regularly getting reviewed and merged.

What is YaCy?

While I’ve said this post is mostly about deployment, it’s probably a good idea to tell you all a bit about what YaCy is, so you know whether you actually want to read on.

YaCy is a self-hosted, peer to peer search engine. It has entirely its own index and does not rely on the likes of Bing or Google. On its main page, it presents a simple search mask:

When searching, the results page should also be pretty familiar:

This result was a bit unexpected right now. I hadn’t actually crawled my own blog yet, and it still found my posts. Looks like somebody has been pointing a YaCy crawl at it at some point. So this is actually what I would call a good-ish search result. It mostly found a series of blog posts about exactly what I was interested in - migrating from Nomad to Kubernetes, plus a few other results also related to migrating from something to Kubernetes.

The way YaCy works is that there is an index held locally in an embedded Apache Solr instance, which is also used for searching. This search index is filled by the instance’s own crawling of websites. I will go into more detail on crawling in a future post. By the way, if you’ve got any good blog posts or articles which explain how web crawling works these days, what to look out for and how to behave properly, I’d be very happy to hear about them, for example via the Fediverse.

The P2P aspect of YaCy is used in two different ways. The first one is during searching. In the screenshot above, on the left side, you can choose between ‘Peer-to-Peer’ or ‘Privacy’ mode. Privacy mode here means to only search the local instance’s index. The Peer-to-Peer mode searches the local index and goes out to other instances to do a remote search. The second way is via a constant gossip protocol which exchanges pieces of the local index with other instances, both sending and receiving. This is always ongoing in the background, without user intervention. This way, you will end up with a lot more entries in your local index than just what you yourself crawled, and the remote search adds to that on top.

I’m also of a mind to look into this a bit more deeply, because the official docs and what exactly is exchanged is not too detailed, and I want to look at the code a bit more.

Let’s end this section with a bit of a general vibe: The project does work. I do get search results from pages I’ve crawled myself, and I’m also getting results for pages which I definitely have not crawled myself. The network is active, showing about 600 peers seen over the past week, and I’m getting quite a few remote searches in. I’ve also had some success with a few of my searches. The example above was a pleasant surprise, getting served my own blog for a relevant search query. But there have also been other queries which were not too useful. I’ve for example just done a quick search for CloudNativePG. This did show CNPG’s GitHub page, but the home page was not in the index at all.

There are in the main two areas I will want to research more deeply. One being crawling. Most important to me is to make sure that YaCy’s crawler really respects all the rules around web crawling, like respecting robots.txt and keeping the per-site request rates low. I think it already does that, but I will need some testing. Then there’s the question of what to crawl? How deep to crawl? What’s the right way to get a breadth-first crawl going, instead of just indexing pages I already know? But without filling the index with too much garbage?

Then there’s search ranking. It doesn’t come out in the example search from above, but the ranking is really not great sometimes. But it’s also highly configurable. And there is an extended ranking called CitationRank, similar to Google’s PageRank. I really want to dig into that and how it’s implemented.

One nice thing to note: YaCy implements the necessary APIs to be used as a search provider in Firefox.

Deploying YaCy

YaCy is a Java application and comes with multiple ways of deploying it, both with and without Docker. It has a few warts, though.

Before I get to my Kubernetes deployment, a quick note: You can also run it locally on your own desktop machine. It works perfectly nice there, even without an externally open port. You won’t be fully participating in the P2P network, but you will be able to do remote searches. And when you’re triggering crawls, your resulting index will even be shared with other peers. But your instance won’t be serving other peer’s remote searches, and you won’t be able to receive index updates from other peers via the background gossip protocol.

Let’s start with the Docker images. At the moment, the newest versioned releases for the Docker image on Dockerhub are from 12 months ago, even though there was a YaCy release in March. The only current images are in the latest tag, which I don’t really like. So my first step was building the YaCy image myself. Here is the Containerfile:

## builder image
ARG alpine_ver
ARG jdk_ver
ARG wkhtmltopdf_ver
FROM eclipse-temurin:${jdk_ver}-jdk-alpine-${alpine_ver} AS builder

# Install needed packages not in base image
RUN apk add --no-cache curl git apache-ant

# set current working dir & copy sources
WORKDIR /opt
COPY . /opt/yacy_search_server/
RUN ant compile -f /opt/yacy_search_server/build.xml \
    && rm -fr /opt/yacy_search_server/.git

# Set initial admin password: "yacy" (encoded with custom yacy md5 function net.yacy.cora.order.Digest.encodeMD5Hex())
RUN sed -i "/adminAccountBase64MD5=/c\adminAccountBase64MD5=MD5:8cffbc0d66567a0987a4aba1ec46d63c" /opt/yacy_search_server/defaults/yacy.init && \
	sed -i "/adminAccountForLocalhost=/c\adminAccountForLocalhost=false" /opt/yacy_search_server/defaults/yacy.init && \
	sed -i "/server.https=false/c\server.https=true" /opt/yacy_search_server/defaults/yacy.init

## build final image
FROM surnet/alpine-wkhtmltopdf:${wkhtmltopdf_ver} AS wkhtmltopdf
FROM eclipse-temurin:${jdk_ver}-jre-alpine-${alpine_ver} AS app

RUN apk add --no-cache \
	imagemagick \
	xvfb \
	ghostscript \
	# Install dependencies for wkhtmltopdf
	libstdc++ \
	libx11 \
	libxrender \
	libxext \
	libssl3 \
	ca-certificates \
	fontconfig \
	freetype \
	ttf-dejavu \
	ttf-droid \
	ttf-freefont \
	ttf-liberation \
	# more fonts
	&& apk add --no-cache --virtual .build-deps \
	msttcorefonts-installer \
	# Install microsoft fonts
	&& update-ms-fonts \
	&& fc-cache -f \
	# Clean up when done
	&& rm -rf /tmp/* \
	&& apk del .build-deps

# Copy wkhtmltopdf files from docker-wkhtmltopdf image
COPY --from=wkhtmltopdf /bin/wkhtmltopdf /bin/wkhtmltopdf

# copy YaCy to app image
RUN addgroup yacy && adduser -S -G yacy -H -D yacy
WORKDIR /opt
COPY --chown=yacy:yacy --from=builder /opt/yacy_search_server /opt/yacy_search_server

# Expose HTTP and HTTPS default ports
EXPOSE 8090 8443

# Set data volume: yacy data and configuration will persist even after container stop or destruction
VOLUME ["/opt/yacy_search_server/DATA"]

# Next commands run as yacy as non-root user for improved security
USER yacy

# Start yacy as a foreground process (-f) to display console logs and to wait for yacy process
CMD ["/bin/sh","/opt/yacy_search_server/startYACY.sh","-f"]

It’s a light adaption of the official Alpine Dockerfile, with the only change being that I introduced configurable versions for the JDK, Alpine and other tooling. I build this via my internal pipeline. If you’re interested, have a look at this post.

With that done, I could create the Kubernetes deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: yacy
spec:
  replicas: 1
  selector:
    matchLabels:
      homelab/app: yacy
  strategy:
    type: "Recreate"
  template:
    metadata:
      labels:
        homelab/app: yacy
    spec:
      automountServiceAccountToken: false
      securityContext:
        fsGroup: 1000
        runAsNonRoot: true
        runAsUser: 100
        runAsGroup: 1000
      containers:
        - name: yacy
          securityContext:
            allowPrivilegeEscalation: false
            # Can't be done because htroot/ is written to and is outside DATA/ dir
            # At the same time, this dir contains files already, so can't just be remapped
            #readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL
          image: containers.homelab.example/homelab/yacy:{{ .Values.appVersion }}
          volumeMounts:
            - name: data
              mountPath: {{ .Values.mountDir }}
          resources:
            limits:
              cpu: 2000m
              memory: 3200M
            requests:
              cpu: 2000m
              memory: 3200M
          env:
            - name: YACY_PORT
              value: "{{ .Values.ports.bind }}"
            - name: YACY_PORT_PUBLIC
              value: "{{ .Values.ports.public }}"
            - name: YACY_STATICIP
              value: "{{ .Values.domain }}"
            - name: YACY_JAVASTART_XMX
              value: "Xmx3000m"
            - name: YACY_UPNP_ENABLED
              value: "false"
            - name: YACY_SERVER_HTTPS
              value: "false"
            - name: YACY_NETWORK_UNIT_PROTOCOL_HTTPS_PREFERRED
              value: "true"
            - name: YACY_UPDATE_PROCESS
              value: "manual"
            - name: YACY_PROMOTESEARCHPAGEGREETING
              value: "Meier's Search"
            - name: YACY_PROXYCLIENT
              value: ""
            - name: YACY_ADMINACCOUNTFORLOCALHOST
              value: "false"
            - name: YACY_SCAN_ENABLED
              value: "false"
            - name: YACY_BROWSERPOPUPTRIGGER
              value: "false"
            - name: YACY_TRAY_ICON_ENABLED
              value: "false"
            - name: YACY_NETWORK_UNIT_AGENT
              value: "mei-home-search"
          livenessProbe:
            httpGet:
              port: {{ .Values.ports.bind }}
              path: "/"
            initialDelaySeconds: 30
            periodSeconds: 30
          startupProbe:
            httpGet:
              port: {{ .Values.ports.bind }}
              path: "/"
            periodSeconds: 10
            failureThreshold: 24
            initialDelaySeconds: 60
          ports:
            - name: yacy-http
              containerPort: {{ .Values.ports.bind }}
              protocol: TCP
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: yacy-volume

So there’s a few things to say about the config. First, note that the Container cannot be configured with readOnlyRootFilesystem: false. This is because files are written to directories which already contain other files as part of the image, so it can’t be re-mounted.

Another thing worth mentioning is the rather long startup probe duration. This is due to the fact that sometimes, YaCy does some cleanup/compression during startup. Especially after a crash instead of a proper shutdown. This can take quite a while, especially when you’re working with a Pi 4 class CPU.

Then there’s a general problem with the setup of the environment variables. The issue is that the env variables correspond to settings in the YaCy config file. To set those via env variables, you have to prepend YACY_ to the front of the config key’s name and also upper case it and replace . with _. This works for values like some.setting.name, but fails for settings like someSettingName. That’s due to how a check to see whether a config key corresponding to the env variable name exists works, lower casing the whole env variable name and then searching for it in the configs. See this issue and my accompanying fix here.

Also note that the YACY_PORT_PUBLIC setting is not currently supported upstream, it’s a fix for an issue I’ve discovered earlier. I will go into more detail in the next section.

The YACY_PORT and YACY_PORT_PUBLIC settings set the port YaCy binds to and the port it communicates to other peers for connections, respectively. I’m disabling UPnP as well as HTTPS, as I don’t have UPnP enabled in my firewall and YaCy is fronted by a reverse proxy terminating TLS. The YACY_STATICIP setting defines the address reported to other peers trying to contact this one. Despite its name, it also happily takes a domain, not just an IP.

The YACY_PROMOTESEARCHPAGEGREETING setting configures the subheading shown under the YaCy logo on the search page.

YACY_ADMINACCOUNTFORLOCALHOST is an important setting. By default, YaCy launches with a configuration which allows all connections coming from local to do anything they want without any further authentication. Not a good default in my view, but likely intended to make it easier to handle when running an instance locally.

YACY_SCAN_ENABLED is disabled here, as that setting would scan the local network for other YaCy instances. Not useful, as this is the only instance I’m running. Well, right now at least.

Both YACY_BROWSERPOPUPTRIGGER and YACY_TRAY_ICON_ENABLED are irrelevant for deployments on Kubernetes, as they enable features only useful for local desktop deployments.

And finally, YACY_NETWORK_UNIT_AGENT defines the name of the peer in the YaCy P2P network. If not given, a name will be generated randomly.

One note on the volume I’m attaching here: This is a Ceph RBD from an SSD pool. I figured that Solr would likely not be happy with a network attached HDD volume. I haven’t had any performance or stability issues with this.

It’s also worth noting that YaCy can be configured via a config file as well, but it’s not really cloud native. There is the yacy.init file to begin with. During first startup, that’s copied to get the initial config file. It should work perfectly well to override this with e.g. a ConfigMap to change some settings. Any changes after that initial setup are more complicated without using env variables though. That’s because the real config file under DATA/SETTINGS/yacy.conf is also written to by YaCy when changes are made via the web UI. Those would be lost upon restart with a ConfigMap.

External accessibility and peering

Before getting into the details, it’s worth noting that YaCy has different peer levels an instance can have, ranging from “Virgin” (yes, I know 😔), where it hasn’t had contact to any outside instance, to “Principal”. Virgin instances are for example instances which are really only for local use, e.g. providing search for an intranet and its sites only. In this mode, the instance doesn’t connect to any other peers and doesn’t participate in the global search index.

The next level is “Junior”. Here, the instance can connect to external peers at least outgoing. This allows usage of P2P search and outgoing transfers of pieces of the local index, but not receipt of index pieces from other peers.

Next comes the “Senior” mode. This is what my instance is currently running in. It means full participation in the P2P index, being able to be contacted by external peers. If you’re curious, my instance is mei-home-search.

The final level, “Principal”, is a Senior instance which also provides an initial seed list. Some of those are hardcoded into the YaCy binary as a starting point for new instances. This is only required during initial setup. Afterwards, each instance keeps its own seed list and uses that after restarts.

To take part in the peer to peer aspect of YaCy, external peers need to have access to my instance. I was a bit apprehensive about just hanging the entire thing out in public. But I found that just making the /yacy/ path available seems to be enough to make peering work.

So the next thing to look at is the address and port YaCy hands to other peers for the P2P connection. Here, the YaCy docs in the yacy.init file are a bit confusing and don’t really work, at least for me. They document three different ports:

# port number where the server should bind to
port = 8090
[...]
#sometimes you may want yacy to bind to another port, than the one reachable from outside.
#then set bindPort to the port yacy should bind on, and port to the port, visible from outside
#to run yacy on port 8090, reachable from port 80, set bindPort=8090, port=80 and use
#iptables -t nat -A PREROUTING -p tcp -s 192.168.24.0/16 --dport 80 -j DNAT --to 192.168.24.1:8090
#(of course you need to customize the ips)
bindPort =
[...]
#publicPort if you use a different port to access YaCy than the one it listens on, you can use this setting
publicPort=

The port config is the expected configuration for the port YaCy actually binds to. Reading the bindPort config, you might expect that you could set the bindPort instead, and then YaCy would bind to that and only set the port as the port communicated to external peers for connections. I tried it with a setting like this:

port = 443
bindPort = 8090

This lead to errors during startup, because now YaCy was trying to bind to 443, which failed because it’s not running as root. After some searching, I found that bindPort doesn’t show up anywhere in the code. It seems to simply be unused. I’ve created a PR to remove it.

Then there’s the publicPort setting. I couldn’t use it via env variables due to the aforementioned issues with camelCase settings. But I tried setting it through the UI as well as manually editing the config file. Neither worked. External requests still shattered on my firewall, trying to access port 8090, or any other port I set in the port setting. But what I wanted here was a way to set the listening port of YaCy itself separate from the port that YaCy tells other peers to connect to. I also didn’t want to set the port setting to 443, because that would have meant extended permissions for the YaCy container.

I could have opened port 8090, but I also didn’t want to do that. I already have ports 80 and 443 open and wanted to use them. So I looked into the code instead. See this issue and the accompanying pull request. With that (as of yet unmerged) PR, there is now a new port.public setting, which only configures which port is send to other peers for external connections.

With that set to 80, I was hoping everything to work now. But other peers were still unable to reach mine. This time though, the issue was entirely of my own making. In my Bastion Traefik, I had two open ports, one NAT’ed to my external port 80 and one to external 443. But to again keep permissions for that Traefik instance restricted, those ports on the bastion host were not 80 and 443, but higher ports. But I used Traefik’s entrypoint redirection to point the HTTP entrypoint to the HTTPS entrypoint. This, of course, did never actually work. As this setting would reply to any request to the HTTP port with a permanent redirect. But not to port 443, but to the port where the internal HTTPS endpoint was listening. Which isn’t accessible publicly.

That took me quite a while to figure out.

But once I had finally configured that correctly, my peer started peering properly.

There’s still something hinky though. YaCy regularly reports the peering status in the logs, here’s an example from my peer:

PeerPing: I am accessible for 31 peer(s), not accessible for 24 peer(s).

So peering definitely works for some peers. I know that because I’m receiving remote search queries with no issue. But some other peers still cannot connect to me. And I just can’t figure out why not. Something to look into at a later date.

Resource consumption

Before I end this post, a short look at the resource needs is in order. My usage has been rather restricted up to now, but I have done at least a few crawls already, of a few random pages. I’ve for example crawled kubernetes.io, the German newspaper faz.net and the official pages of a few cities I’ve lived in the past, just to get a feeling. In total, that lead to a disk usage of about 13 GB. The top was at 16 GB, but I’m not sure why it’s suddenly so much reduced.

When it comes to networking, the need is not too much. The few crawls I’ve done up to now haven’t even saturated my “my country is a bit shit at the internet” 250 MBit/s connection.

Then there’s CPU usage. Here is the usage of the YaCy Pod since its launch:

Note that that this is the Kubernetes way of measuring CPU, meaning a utilization of 1.0 means one core fully used. Also, this is running on Raspberry Pi CM4. The two phases with higher utilization than 0.2 were while I was running crawls. Everything else is normal use, and the smaller/shorter spikes are restarts. So CPU power seems to be mostly used during crawling and ingestion of the results into the local index.

Here is the memory usage:

As is typical for a garbage collected language like Java, the memory consumption fluctuates a lot. Once I had gathered a bit of an index through my first crawls, the average consumption rose quite a bit. I only reached stability without OOM after increasing the limit all the way up to 3 GB and setting the JVM’s Xmx option to 3000 MB. I expect to have to increase the volume once the size of the local index increases when I crawl more sites.

After I’ve used YaCy properly for a while, I will likely write up another post on its scaling behavior, because I’m rather curious about that. But for now it seems to run quite happily in that 3 GB limit.

What’s next

The next part will be a deep dive into crawling. I’ve already found that just choosing a starting point and just doing a depth 3 crawl isn’t really that great. I’ve also found that you pretty much need to first explore the page you’re crawling a bit. Two examples: The faz.net page has a gigantic /kaufkompass/ category full of products which is really not worth crawling. And when crawling a GitHub project, you will likely want to exclude /tree, /commits and /blobs.

There also seem to be features for importing e.g. ZIM files or RSS feeds.

And there’s the question of how to get better search result ranking.

And finally, there’s quite some interesting metrics available, like crawled pages, number of peers, number of remote searches and so on. I’m feeling very tempted to either implement Prometheus metrics directly in YaCy, or writing an external exporter which scrapes YaCy’s existing APIs.