How we built Alaan's Brain, and how it lied to us

Broken retrieval does not look broken; it looks like an answer.

An Alaan’s product manager asked Synapse, our internal agent, where a behaviour was implemented. Synapse searched, read what came back, and said the code did not exist. It was wrong. The code was in the index, and the question had named the identifier directly.

Nothing in the system reported a problem. Synapse received a full result set at maximum confidence, did not find the answer in it, and said so honestly. The retrieval was broken, and from the outside it looked like knowledge.

Two names here. Brain is the store underneath: everything the company knows, ingested and indexed in one place. Synapse is the agent people talk to, and most of its job is working out what to ask the Brain. A synapse is how a signal gets across, so the division is a route rather than a stack: the Brain holds what we know, and Synapse is how anyone reaches it.

That division is why the bug took weeks to find. The failure was in the Brain and the only place it showed up was in Synapse, where it did not look like a failure at all. This is how the Brain works, and what the bug changed about the rest of it.

Why we built it

We know a lot as a company and almost none of it sits in one place: a Notion page someone wrote nine months ago, a Drive folder of call notes, the Slack thread where a decision actually got made, and the product code, which is usually the only version of the truth that has not gone stale.

We build a finance platform for finance teams, so internal questions tend to be exact rather than approximate. What counts as a valid receipt under a specific VAT rule. Which of two similar looking numbers is the one the customer sees on their statement. An answer that is roughly right is worse than no answer, because somebody will forward it to a customer and then we get to have a much longer conversation.

The real answer usually lives in the code. A policy doc describes what we intended; the code describes what happens to a transaction at two in the morning when a merchant sends a partial reversal. The two drift apart quietly, and the person who knows which one is currently winning is in a meeting. So the knowledge base had to treat code with the same weight as a written document, or it would describe intended behaviour confidently and know nothing about the real one.

One store, every source

Everything else rests on one decision: every source produces the same kind of row. A Notion page, a scanned PDF, a Slack attachment and a file from a product repo all become chunks in one store, under one vector index and one text index. No separate index per source. No separate retrieval path per source.

It is obvious written down and not at all obvious while building, because every source arrives in its own shape and a pipeline per source always looks like less work on the day.

The payoff arrived with code search. A repo is a collection, its files are documents, and the chunk to embed to retrieve path we already had turned out to be exactly what "how does this work" questions need. We added no new infrastructure; the only code specific parts are working out which files to read and diffing them so unchanged files are not embedded twice.

Connectors do almost nothing

A connector answers two questions: what files exist, and which of them changed. It does not parse, chunk, embed or index anything. All of that happens in one ingest function, which is also where the file type allowlist lives, where the per user quota is consumed, and where the audit row is written.

The single funnel has paid for itself repeatedly. Zip upload, so you can drop a whole folder in and have it indexed, inherited quotas, auditing and type checks for free, because the archive expander just calls the same function once per file. It also gave us one place for the unglamorous work an archive demands. Size caps are enforced while the bytes are still arriving, because the sizes declared in an archive header are written by whoever made the archive and can happily lie. Symlinks and nested archives are refused, because a symlink named notes.md pointing at /etc/passwd would walk host files into the knowledge base. Invisible characters are stripped from filenames, because a right to left override mid name makes the file list render something quite different from the file itself.

Quotas exist because bulk upload turned cost into a safety question. One large archive is hundreds of documents and thousands of embedding calls, so before quotas one enthusiastic person could get through the day's budget before lunch.

The expensive step worth paying for

Before a chunk is embedded, a small model writes a sentence or two placing it inside its parent document, and that sentence is prepended to the text we embed and index. It is the most expensive thing in the pipeline and the part I would argue hardest for.

A raw chunk from the middle of a document has lost its heading, its preconditions and its caveats, so it reads like a confident general statement when it was actually a footnote about one specific case. The extra sentence puts that context back, and it helps keyword search too, because the chunk now contains the words someone would actually use to ask about it.

Scanned files go to a vision model instead of a text extract, because a PDF with no extractable text used to sail through ingest and land as an empty document that answered nothing and reported nothing wrong.

Which model does which work

The context step makes a model call for every chunk we ingest, millions of calls, and the work is genuinely dull: read this paragraph, say where it sits. Paying frontier prices for that would be strange, so it runs on an open weight model we deploy inside our own cloud account.

The lower bill is the least interesting part. The document text never leaves our own tenancy, which is far easier to explain to a customer than contractual language about a third party. And every adapter sits behind the same interface, so moving a step to a newer model is a configuration change rather than a project. Open weight models closed the gap on narrow mechanical work a while ago, so we pick up each improvement for the cost of a deploy, and no single vendor decides what our most repeated step costs next quarter.

The conversation itself still runs on a frontier model, since choosing the right tool and writing the answer is the part that genuinely needs the intelligence. Mechanical volume goes to open models we host; hard reasoning stays with the best model we can get.

Two retrieval modes, and one bad number

If a collection is small enough we skip retrieval entirely and hand over all of it in reading order. There is nothing worth ranking when the model can read the whole thing, and it is the cheapest good answer available.

Anything bigger goes through hybrid retrieval. Vector search and text search run independently, get fused by rank position, then go to a reranker that scores them against the question actually asked. Anything below a relevance floor gets dropped rather than padded out to make the result count look respectable.

The two modes produce different kinds of number. A reranked score runs from roughly 0 to 1 and means relevance; the small collection path does no ranking at all and stamps a flat placeholder instead. Harmless alone, and it stopped being harmless the moment one query had to search several collections at once, which is the normal case for code search. We searched each collection separately and sorted the union on raw score, so the placeholder outranked every genuine relevance score, every single time.

In production that meant an unscoped code search returned the first few chunks of the smallest indexed repo, in file order, for every query, repeating the same chunks each time. Across ten queries in one conversation, 20 of the 22 excerpts came back from that one small repo, while the two multi million token repos that actually held the answer were never reached at all.

That is where the PM got told the code did not exist.

The fix, and the smaller fix that mattered more

The fix was to produce one ranked set instead of several. Candidates get gathered per collection but concurrently, the whole union gets reranked once, and if the reranker is unavailable we fuse by rank position, because a rank means the same thing everywhere and is comparable by construction.

The change alongside it mattered more. Every excerpt now carries a note about how it was scored:

{ "repo": "core", "filepath": "models/transaction.js",
  "score": 0.71, "scoreKind": "rerank", "text": "…" }

A scoreKind of inline means no ranking happened at all, and the model is told that directly, so an unranked result can no longer pass itself off as a confident hit. That is the lesson we kept applying afterwards: a system that cannot say how much it knows will fill the gap with confidence, so it has to be told what it has.

Telling the model what it has

Search used to be a blind guess. The model had a tool description and no idea whether anything relevant existed, so it both missed obvious documents and went hunting for things we had never indexed.

The prompt now carries a short manifest of what this particular person is allowed to search, built with the same scoping rules the tool itself enforces, so it can never advertise something the tool would then refuse to look at. It gives true counts rather than a truncated list pretending to be the whole thing. It flags documents that are still processing, so the model does not confidently deny something halfway through ingest. And it says outright that a filename is not its contents, because otherwise the model will cheerfully answer from a title.

Two different permission questions

Access control here is really two questions, and running them together is how internal tools end up leaking.

The first is who may search a collection. That is an ACL on the collection itself, resolved server side from the session and never taken from anything the client sends.

The second is where an answer built from those documents may be said out loud. A private document searched in a DM is fine; the same answer pasted into a shared channel has been disclosed to everyone in it. So each knowledge base carries its own allowlist of where it may speak, denies by default, inherits nothing from anywhere else, and only its owner can change it.

One more rule came out of an incident: deletion has to cascade to the chunks first. Chunks are what search actually reads, so removing the document row alone leaves a deleted document still answering questions and still citing a filename that has disappeared from the UI. The other order means the worst case is leftover metadata nobody can read.

Did it work

The clearest signal came from upstream of the knowledge base. Before code search existed, PM conversations spent most of their effort rediscovering the same schema over and over: across 43 conversations, 167 schema probes and 346 exploratory queries, with some conversations guessing at table and column names 15 to 30 times before answering anything at all. One semantic search over the models now returns the fact directly.

That guessing was never free. It came out of the same day as the customer call someone was preparing for or the spec someone was trying to finish. New joiners get the worst of it, because you can work here for a month before you are confident about what a settlement actually settles.

The hard questions are still hard. What has gone away is a whole category of expensive guessing, which is the ordinary tax of a company knowing more than any one person in it can hold, and it gets steeper as we hire. The Brain does not remove that tax so much as make it cheap to pay, and for a company whose product is other people's money, being able to point at the document or the line of code an answer came from is worth more than the speed.

What is next

Chunking is still generic. We split code on blank lines, which works well enough without being good; splitting on function or class boundaries would make the excerpts far more coherent.

Sync still runs on a timer, so code search can be up to a day out of date. A webhook on merge would fix that.

The bigger one is automated evaluation. The bug above survived for weeks because nothing was watching retrieval quality on a schedule. We are building a self healing harness for exactly that, and I will write about it separately.