Repository navigation
[RFC] Support for horizontal scaling Sourcebot #439
Description
This discussion tracks adding support for horizontal scaling to Sourcebot.
TL;DR
- I think the existing single, self-contained docker container is great since it's super easy to use and is radically simple.
- But it's probably not practical in larger deployments, especially ones with thousands of repositories.
- We've seen instances where when a machine is under heavy indexing load, the webapp can become unresponsive or even OOM errors can occur.
- I think the first logical step here is to distribute our services as separate images (i.e.,
ghcr.io/sourcebot-dev/(webapp|worker|zoekt)) - Upstream dependencies like Postgres, and Redis would be separate.
- More images = more complexity, so I think the best mechanism to manage this complexity is moving towards declarative configs. We could do the following:
- I think it will be important to have sensible defaults (s.t., it's very easy to deploy), while having deep configuration options that are well documented (s.t., it fits the user's deployment story).
Open questions / investigations
- What will we need to do to make each one of our services (webapp, worker, zoekt) to support multiple replicas at the same time?
Is this Overkill?
I'm trying not to go too overboard with the design here. DX should not be sacrificed with this change: ideally it should improve. Sourcebot should be just as easy to deploy as a vertically scaling service, both locally and in production. Deploying it as a horizontally scaling service in production should also be easy, while including configuration depth to support a wide range of deployment scenarios.
Locally, using Docker Compose will streamline deployment since a) it's a single command, docker compose up, and b) it's declarative, so all of the options (w/ sensible defaults, or commented out) can be included inside the docker-compose.yml.
In production, using Helm, Terraform, NixOS, etc. will help get people setup quickly. Again, sensible defaults here are key.
We also should continue supporting the self-contained image since to avoid introducing a breaking change here. We can evaluate if it's still worth shipping whenever we do a v5.
Activity
abraverm commentedon Aug 7, 2025
The more deployment types you would support the more issues would be reported and divarage your attention from the core. The nature with Open Source project is that there would be many deployment types and I believe it is better for the project to be the gathering place for them. That said I think a better startucture would be having a seperate repository for each deployment type, but centralized integration test suite. The project could commit to two type of deployments, lets say Docker (single container) and Kubernetes (Helm), and to the test suite repository. Maturity and stability of other deployment repositories would depend on which version of Sourcebot and test suite it passed. That said, I believe the critical point for horizonal scaling of Sourcebot is the integration test suite, there needs to be an easy path for one to report and test issues with any type of deployment.
kquinsland01 commentedon Sep 4, 2025
My thoughts, more or less exactly.
I can say precisely zero nice or polite things about
helmand many, many, many many things about all ofhelm's many faults and poor design decisions that have inspired countless chart maintainers to come up with bespoke/hacky workarounds.For essentially that reason, i'd suggest not bothering and letting $otherPeople maintain the seemingly unbounded number of ways to deploy $something into k8s.
Publish well tested docker containers and a robust docker-compose file to show how each container interacts with the other(s) and comment it with good rules of thumb for scaling. With a high-quality lowest-common-denominator, people are then free to make their own kustomize stack(s) or helm chart(s) or even just run that docker compose file through tools like
komposeas per their individual needs and constraints.komapa commentedon Dec 12, 2025
As long as the important data can be externalized like the Postgres database, you can simply document that a super simple VerticalPodAutoscaling config with inplace resizing can more or less remove any issues around the single Pod nature of Sourcebot:
What I would like to see to be honest is zero downtime upgrades which should also handle rolling of the single container when we need to update base images or the nodes under the pod. VPA can handle the scaling but we still need a way for Sourcebot to not have downtime during normal maintenance.