Git at any scale

Cask is a Git host. It speaks ordinary Git over HTTPS, it runs as a single container behind a Cloudflare Worker, and it keeps the authoritative copy of every object and every ref in an append-only log in object storage. The bare repositories on its disk are caches. If the disk vanishes — and on this platform it does, routinely, without warning — nothing is lost, because the disk was never where the repository lived.

This is a write-up of how it works and why it is shaped this way. It is not a product pitch. Most of the interesting decisions come from taking disposable compute and object storage seriously rather than pretending Cask is a smaller version of something enormous.

Disposable compute, durable Git

The usual advice for self-hosting Git is to put a bare repository on a server and back it up. That advice is fine right up until you try it on modern serverless infrastructure, where the machine running your code is deliberately disposable. Containers get replaced during a deploy. They get replaced when the platform reschedules them. They get replaced when they have been idle and something needs the capacity. A bare repository on that container's filesystem is not a repository; it is a cache with an unusually convincing directory listing.

The other standard answer is a managed forge. That works, and for most people it is the right answer. It was not the answer here: a forge is a lot of product to accept when what you need is a remote you can push to and a way to read the code in a browser.

So the question is narrow: what is the smallest honest design for a Git host where the compute is disposable and the storage is an object store?

"Honest" is carrying weight in that sentence. It is easy to build something that appears to work: point Git at a directory, serve the smart HTTP protocol, return success. The failure mode is silent and delayed. You push on Tuesday, the container is replaced on Thursday, and the commit you pushed is gone with no error anywhere in the system, because at no point did anything claim responsibility for making that push durable. A host that loses a push it acknowledged is worse than no host at all, because it consumed the trust that would have made you check.

Truth lives in a log

The design decision everything else follows from is that the durable state of a repository is an ordered log of ref updates, plus the packs of Git objects those updates need, stored in an object store. Cask calls it the write-ahead log because that is what it is: nothing is acknowledged to a client until it is in the log.

A log entry is small and boring on purpose. It names the repository, the kind of mutation it represents — a push, a server-side ref update, a merge — a list of ref transitions with their old and new object IDs, and optionally the key of a pack object holding the new Git objects. There is one more object per repository, an index, which is the ordered list of committed entry IDs.

The index is the commit point, and this is the part worth being precise about. A .wal entry object sitting in the bucket is not a commit. A pack sitting next to it is not a commit. An entry is committed exactly when the index names it. That distinction sounds pedantic until you consider what happens when a process dies between uploading an entry and updating the index: on restart, an orphaned entry is simply an object nobody references, and the correct action is to ignore it. If instead the system inferred commitment from "an entry object exists", a half-finished write would resurrect itself as history.

Ordering the log this way also makes the recovery story trivial to state. To reconstruct a repository, create an empty bare repo, read the index, and replay every entry in order: unpack each pack into the object store, then apply that entry's ref transitions as a single git update-ref --stdin transaction. Nested branch names, tags, and deletions all work, because they are just ref names and object IDs and Git already knows how to apply them atomically.

An earlier iteration of this design stored a single ref name per entry and replayed everything under refs/heads/. It worked for the demo and was wrong in three separate ways: a push that updated two branches lost one of them, a tag came back as a branch, and a deletion could not be represented at all. The current entry carries a list of updates, treats an all-zero new object ID as a deletion, and preserves the full ref name. Old single-ref entries still replay, because the reader translates them into a one-element list. That backwards compatibility is not politeness; it is the difference between a schema change and a data migration.

Repositories on disk are projections

If the log is truth, then the bare repository under /repos is a projection of it — a materialized view that exists because Git's own tooling is very good and I would rather use it than reimplement it.

Every entry point into the system begins the same way: make the projection match the log, then do the work. There are only two interesting cases. If there is no local repository, build one in a staging directory, replay the whole index into it, and rename it into place when it is complete. The rename matters: a crash halfway through a replay leaves a half-built directory that a later request must not mistake for a warm cache, and an atomic rename means the canonical path either has a complete projection or has nothing.

If there is a local repository, it carries a small marker file recording how many entries it has applied and the ID of the last one. Catching up means checking that the marker is a prefix of the committed index and replaying only the suffix. If the marker is missing, unparseable, or names an entry the index does not have in that position, the projection is discarded and rebuilt from scratch. There is no attempt to reconcile a divergent cache, because a cache that disagrees with the log has no information worth preserving.

Replay is idempotent by construction. For each ref transition, the projection is inspected: if the ref is already at the new object ID, the update has been applied and is skipped; if it is at the old object ID, the update applies; if it is at neither, the projection has diverged and gets discarded. That last branch is the one that keeps a subtle bug from becoming a data-loss bug, because it refuses to force a ref into a state the log did not describe.

An earlier version of the cache had a check that amounted to "if a HEAD file exists, the cache is fine". A warm container therefore never noticed writes it had not made itself. That is fine on a single writer that never restarts, which is to say it is fine until the first time it matters.

The push path, and the ordering that makes it work

Push is where a Git host earns its keep, and it is the part of Cask I have rewritten the most.

The externally visible contract is simple: when git push prints ok, the objects and refs are in the log. Getting there requires the request to be handled in a specific order, and two of the steps exist because of things that went wrong.

First, the entire request body is spooled to a temporary file before a single response header is committed. This is not an optimization; it is the fix for a class of hang that shows up when a container sits behind a proxy. Once the handler starts writing a response, the remainder of the request body may never arrive, and both sides sit waiting for each other. Reading the body to completion first makes the rest of the handler straightforward, and spooling to disk rather than memory keeps a large push from being a memory event. It also means an oversized body can be answered with a real 413, because the headers have not been sent yet.

Second, the ref commands are parsed from that spooled file, not guessed from a suffix of the bytes. The pkt-line framing at the front of a receive-pack request tells you exactly which refs the client wants to change and from what. Those commands are the basis for everything that follows, so they are read from a file that is known to be complete.

There is one wrinkle here that cost me a real bug. When a push is larger than Git's http.postBuffer — a megabyte by default — the client cannot rewind the body to retry after an authentication challenge. So it first sends a probe: the same endpoint, with a body consisting of nothing but a flush packet, purely to learn whether the request will be accepted. Cask parsed that probe, found zero ref commands, concluded the request was malformed, and returned a 500. Every push over a megabyte failed, and the smaller pushes I had been testing with sailed through. The fix is four lines — a body that parses to no commands and is at most one flush packet long gets the empty response a stock Git server returns — but the lesson is the durable part: the protocol has behaviours that only appear at certain sizes, and "it works on my repository" is not coverage.

Third, under the repository's exclusive lease, the projection catches up to the log and every commanded ref is snapshotted. Then git receive-pack runs against that projection with the spool as its input, and its response is captured in a bounded buffer rather than streamed straight to the client. Holding the response is what makes the rest of the sequence possible: nothing has been promised yet.

Fourth, acceptance is classified by comparing each commanded ref's object ID before and after Git ran. This is deliberately not done by parsing Git's human-facing status text. The ref store is the authority on what changed, and reading it directly handles mixed results — one branch accepted, another rejected by a non-fast-forward — without depending on message formats. If a ref ends up somewhere neither the client asked for nor the snapshot recorded, the projection is invalidated and the request fails, because the system has lost track of what happened.

Fifth, a canonical pack is generated for the accepted new objects, excluding every ref tip that is already durable. Excluding the durable tips rather than just the commanded old IDs is what makes creating a branch at an existing commit carry no object payload at all. A delete-only push produces no pack.

Sixth, the entry and its pack are written, and the index is updated to name the entry. Only after the index write returns does the captured Git response go back to the client. If the log write fails, the accepted refs are rolled back to the snapshot, and the client gets a 500 — never Git's captured ok. The client's belief about durability and the log's contents cannot disagree, because the belief is created after the log write, not before.

Reading is a budget problem

The read side of a Git host is mostly an exercise in refusing to do unbounded work. A browser request that walks an arbitrarily large tree, renders an arbitrarily large blob, or diffs an arbitrarily large change is a request that can occupy the single container indefinitely, and the same container is serving clones and pushes.

So every reader in Cask is bounded and says so when it truncates. A directory listing caps its entries. A blob preview caps its bytes and reports that the preview is partial while raw download still returns exact bytes. A diff caps both changed files and aggregate patch bytes, and the UI distinguishes "no changes" from "binary" from "truncated" from "the branch is gone", because those are four different situations and collapsing them into an empty panel is how you end up debugging a non-problem.

Two of the newer readers show the pattern more clearly than the rest.

Blame is paginated by line range. A naïve implementation attributes the whole file and then renders it, which means the cost of the page is set by the largest file anyone ever commits. Cask attributes a window — five hundred lines by default — and offers previous and next links. Parsing Git's porcelain blame output has a small subtlety worth mentioning: the full commit header appears only the first time a commit is seen, and later lines from the same commit carry an abbreviated header. So commit metadata is cached by object ID while parsing, and runs of consecutive lines from one commit are grouped into a single header in the gutter, which is also how it reads best.

Search is the other one, and it is where I had to actually measure rather than guess. File path search is easy to bound: a recursive listing of one commit's paths, cached by commit object ID. Keying the cache by commit ID makes each entry immutable — a push produces a new commit and therefore a new entry, so there is no invalidation logic to get wrong — and the cache is capped by both entry count and total paths held so a few large repositories cannot quietly become the container's memory ceiling.

Content search is harder, because git grep has no persistent index and re-reads the searched content every time. Rather than declaring it too expensive or shipping it unconditionally, I measured it on this codebase and on a synthetic corpus, wrote the numbers down, and used them to pick a ceiling: a revision whose content exceeds the configured size is not scanned, and the page says so with the measured reason instead of timing out. The measurement is recorded in the repository along with its methodology, including the detail that repeated filler compresses so well it makes grep look far cheaper than it is on real source. I am not going to quote a throughput figure here as though it were a benchmark; it was one machine, warm cache, and the useful output was a budget, not a headline.

A control plane beside the log

Git's log is authoritative for objects and refs, and it should not be asked to carry anything else. But a usable host needs state that is not a Git object: the list of repositories, the default branch, merge settings, pull requests and their state, activity, comments.

Cask keeps those as small JSON documents in a reserved prefix in the same bucket. The repository document holds identity, settings, the default branch and a pull-request counter. Pull requests are one document each, numbered, under their repository. Activity events are immutable and keyed by the pull request revision they belong to. Comments are their own objects keyed by creation time, which means listing a prefix returns a conversation in order without maintaining a counter, and — more importantly — a comment never contends with the optimistic revision a merge depends on.

There are two rules that make this coherent. The first is that the catalog gates everything: a URL alone never creates a repository, and every Git and browser route loads the catalog record before touching a projection. The second is that metadata documents carry a monotonically increasing revision, and an edit submits the revision it was based on. A stale edit is refused rather than silently winning.

The default branch is a good example of why this state has to be durable rather than derived. Git's HEAD in a bare repository decides what a fresh clone checks out. If the default branch only exists as whatever the runtime happened to initialize, then a rebuilt projection can advertise a different branch than the one it advertised yesterday, and a clone quietly checks out the wrong thing. Cask stores the default branch in the repository document and reasserts it as the symbolic HEAD every time a projection is created or caught up.

Merging without a transaction

The one place where two systems must agree is merging a pull request, because a merge writes to the Git log and to the control plane, and object storage offers no transaction spanning both.

The way out is a journal. Before anything is written, a merge operation document records what is about to happen: the pull request, the method, the expected base and head object IDs, the resulting commit, and whether the head branch should be deleted. The merge commit is created with merge-tree and commit-tree — no worktree, no index — and its objects go into the log with the operation's ID attached to the entry. Only then is the pull request marked merged and the activity event appended, and only then is the operation completed.

Recovery reads the journal. "Committed to the log" has a precise definition: the index names the operation's entry, and that entry carries the matching operation ID and the expected ref update. An operation without that evidence is aborted, and the pull request stays open — which is the safe direction, because the refs were never changed. An operation with that evidence is finished: catch the projection up, mark the pull request merged, append the event, complete the journal. Every step is idempotent, so running recovery twice is uneventful, and recovery runs before any other mutation on that repository and before rendering a pull request, so an interrupted merge cannot sit there looking open forever.

Destroying data on purpose

Deletion deserves the same care as writing, and it is easy to get wrong in the opposite direction: a synchronous handler that deletes what it can and returns success leaves orphaned packs nobody can reach and a name that looks free.

Cask treats deletion as a resumable operation. It requires the full owner/repository typed in, writes a tombstone outside every prefix it is about to delete, marks the repository as deleting, and returns immediately with an operation ID. From that moment the repository disappears from listings and every route — browser, API and Git — answers 410 Gone rather than 404, because "this is being destroyed" and "this never existed" are different facts and a client should not conclude the name is free.

The work then proceeds in bounded, idempotent pages: evict the projection, page through the log prefix, page through the metadata prefix, and only once both are provably empty remove the catalog document. Progress is recorded in the tombstone, so a container replaced mid-deletion resumes at the step its predecessor reached. The tombstone is retained after completion, which both distinguishes a finished deletion from an interrupted one and holds the name for a while so an accidental recreate cannot silently take over a name whose data was just destroyed.

The ordering is the point. The catalog document goes last because it is the only thing that can reach the rest; deleting it first would strand every pack in the bucket with nothing pointing at it.

What single-user buys, and what it costs

Nearly everything above is simpler than its multi-tenant equivalent, and the simplification is intentional.

There is one container, and it is the only writer. Mutations for a repository are serialized by a keyed lock held across the entire read-validate-write-apply sequence, and different repositories do not block each other. That is a real consistency boundary, and it is honest as long as there is exactly one writer. The object store's check-then-write is not a compare-and-swap, and I have deliberately not exposed anything that pretends otherwise: before Cask could be sharded across containers, it would need genuine conditional writes or a durable coordinator. Writing that limitation down is more useful than a lock that only looks safe.

Capacity is bounded and stated rather than assumed. The log index is a flat list and the catalog is a flat prefix, so there are declared ceilings on repositories per host, pull requests per repository, log entries and refs per repository, and ref updates per push. Crossing one returns a clear capacity error. A segmented index would raise them; until that exists, an explicit error beats silently truncated data.

And Private on a repository badge is product copy, not access control. There is one account. There is no permission model to enforce, so there is no permission model claimed. Repository content is never served to an unauthenticated request, the browser session is an HttpOnly cookie, pages are rendered server-side so that untrusted repository content stays inside template escaping, and that is the whole security story. Anything more would be a story about software I have not written.

What I did not build

No object distribution layer. Clone performance is a function of one bucket and one container, and adding a distribution layer would add failure modes to solve a problem that is not the one this host has.

No standby copies, no custom transport, no gossip protocol. Cask speaks the Git protocol every client already speaks, over HTTPS, and the interesting engineering is in what happens after the bytes arrive.

No CI, no issues, no wiki, no packages. The pull request Checks tab is an empty state that links to nothing, because there is nothing to link to and a tab that implies a CI system you do not have is a lie with a nice icon.

No SSH. One clone URL, copied from one button.

What I would tell someone building the same thing

Three things, in order of how much they cost me.

Decide what "durable" means before you write the handler, and make the code say it out loud. The rule that a client never sees ok before the log write is one sentence, and it dictates the order of eight steps in the push path. Without the sentence, those steps drift into whatever order made the last bug go away.

Make the disposable thing obviously disposable. Naming the local repositories projections, giving them an applied marker, and rebuilding them from scratch whenever the marker is not a clean prefix removed a whole category of "how did it get into this state" questions. There is no state to reason about, because the cache is not allowed to have opinions.

Measure the thing you are unsure about, and write down the number and the method. The content search ceiling is the clearest example: the honest answer to "is this fast enough?" is not yes or no, it is "here is the cost, here is the budget it has to fit in, and here is what happens when it does not fit". That turned an argument into a configuration value, and a configuration value that fails a test when the assumption behind it stops holding.

Cask is a small system, and most of its interesting behaviour is about being careful at the boundaries where it could quietly lie to me. That seems like the right thing to spend effort on for something whose entire job is to still have my commits next week.