← all writeups

Jun 2026isolationsecurityarchitecture

Start with what must never happen

How VolleyPacket keeps one customer's data away from another's, and why every rule is written as a thing that must never be true instead of a thing the system should do.

Picture a self storage place. One building, one roof, hundreds of customers. Everybody’s boxes are in the same warehouse, a few feet apart. The only thing keeping your things yours is that the building knows which unit belongs to who.

Software like mine is that building. VolleyPacket takes a spreadsheet of names and turns it into a document for every person on it, then emails and texts those documents out. One run of it went to over 17,000 government job candidates. Different customers upload different lists. Their data sits in the same database, on the same servers, a few rows apart.

So the question I care about is not “is it separated.” Everybody says their thing is separated. The question is: the day it stops being separated, does anything notice?

A quick word about the word “tenant”

Tenant is a landlord word. Somebody who rents a unit in a building. Software borrowed it and kept the picture: one building, lots of renters, everybody’s things locked in their own unit.

So technically, “multi tenant” just means one running app serving lots of separate customers with a wall between them. By that reading VolleyPacket qualifies fine. Every account is its own unit.

But words drift, and this one drifted. When engineers say multi tenant out loud they nearly always mean the renter is a company. A whole team shares one unit. There are people inside it holding different keys — an admin who can do more, a new hire who can do less, settings everyone in the unit inherits. Picture a company’s Slack, one workspace with forty people in it.

VolleyPacket isn’t that. One account is one person. Nobody shares a unit, because there is nothing above the account to share — no teams, no organizations, no “invite your colleague.” So technically it’s a tenant, socially it’s a multi user app, and the plainest word is just account. That’s the one I’ll use.

Worth saying, because the drifted meaning drags a whole family of problems along with it — two people at the same company who should see different things, admins with more power than members, permissions handed down from a group. All real, all hard, none of them in here. This is the smaller job: keeping one account’s things away from another account’s things. Still surprisingly easy to get wrong.

The trick that made this easier

Most people design software by writing down what it should do. Upload a file. Send an email. Show a progress bar.

I found it easier and safer to write down the opposite: the things that must never happen. Two customers must never see each other’s rows. A password must never appear in a log. That is a much shorter list, and a much more useful one.

Here is why the flip helps so much. “It should send the email” is a promise about the future. To know if it is true you have to wait, watch, and hope. “Two customers must never see each other’s rows” is a statement about right now. You can go and check. Today. With a command.

(People who study this properly call the second kind a safety property — something bad never happens — and the first kind a liveness property — something good eventually happens. I did not know the words when I started. I just noticed one kind was easy to check and the other one wasn’t.)

Everything below is one of those rules. Each one is a thing that must never be true, and each one has a way to find out if it just became true.

Never fetch a thing without saying who is asking

The rule: every job, every template, every piece of data is fetched for a specific logged in person. No shortcuts. No admin back door that skips the check.

That sounds obvious written down. It is not obvious in a codebase, because of how this particular mistake fails.

Say you ask a shop for your parcel, and instead of your parcel they wheel out the entire back room. Did anything go wrong, as far as anyone can tell? You asked for your parcel. You got your parcel. It was just in there with everyone else’s. Nobody’s alarm went off, because getting more than you asked for is not an error anywhere in the system.

That is exactly what a forgotten filter does. The page still loads. The tests still pass. Nothing turns red. You find out when a customer emails you about a name they should never have seen.

So I stopped relying on remembering. Fetching a job goes through one function that takes two things at once: which job, and which person is asking. There is no version of it that takes only the job. To write the unsafe one you would have to sit down and build it on purpose. The safe way is the lazy way, and the dangerous way is extra work — which is the only arrangement that survives a tired Friday.

Blank must never mean allowed

The worst ownership bugs are not the missing checks. Missing checks get caught. The worst ones are checks that run, pass, and were meaningless.

Imagine a bouncer with a guest list. His rule is: your name tag has to match a name on the list. Now hand him a blank name tag, and hand him a list with a blank line on it. Blank matches blank. The rule ran. The rule passed. He lets you in. Nobody did anything wrong and the door is open.

Empty values do this constantly. An unset owner and an unknown visitor can quietly agree with each other if all you do is compare the two.

In VolleyPacket the fix has two halves. For most things there is simply no such thing as a blank owner — the database itself refuses to store a job without one, so the bad case cannot exist to be compared. Making a state impossible beats checking for it every time.

But templates are more interesting, because there a blank owner is real and it means something: a template with no owner is one of the built in ones, meant for everybody. So the same blank value has to mean two opposite things depending on what you are doing. If you are reading, no owner means “this is a shared one, help yourself.” If you are changing or deleting, no owner means “this is not yours, hands off.”

Same empty box, opposite answers. The code has to come out and say which one it means. That is one line of difference and it is the entire meaning of the check.

One place a thing can come from

VolleyPacket runs several workers, on machines whose local disk gets wiped every time I deploy. Early on that forced a decision: the database is the truth about what a job is doing. Not a worker’s memory. Every request goes and asks again.

I made that call for boring reliability reasons. It turned out to be a separation decision too.

A cache is a sticky note. It is fast precisely because you do not have to go and ask again — and that is the whole problem. A sticky note does not remember who it was written for. It outlives the conversation that produced it. Every cache key is one more place to forget whose row this was, and it keeps being wrong long after the moment that made it.

Taking the cache out did not fix a bug. It deleted a category of question. There is one place a row can come from, and that place always knows who owns it.

Reliability and separation wanted the same design. When two different worries point at the same answer, the answer is usually right.

Files leak differently than rows

Rows are easy. A row can have a column that says who it belongs to.

Files do not have that. Files have paths, and a path is just a piece of text, and text is where this quietly goes wrong.

Think of a coat check. You hand over your coat, you get a ticket, the ticket has a number. The number is the only thing connecting you to your coat. Now issue two people the same number. No rule was broken. No one lied. Somebody is still going home in a stranger’s coat.

Two customers starting a run in the same second is not a freak event, it is a Tuesday afternoon. If the name of a stored file is built out of anything less than something guaranteed unique to that one run, the two collide — and one customer’s finished documents land exactly where the other customer’s app will go looking. No query was wrong. No permission was skipped. The filename did it.

So every file goes through one storage layer, and nothing else is allowed to reach the actual file store directly. Which gives me my favourite rule in the whole system.

That is what I mean by making failure visible. A rule you can only keep by remembering it is not a rule, it is a hope.

Secrets must never be written down

Customers plug in their own email and text message accounts, which means my system is holding other people’s login details. Those are scrambled before they are stored, and only unscrambled for the second they are actually used.

The storing part is the part everyone does. The part that actually bites is the writing down.

Most leaked passwords were never stolen. They were printed. Somebody adds a line while chasing a bug at 2am to see what the system is sending. An error handler helpfully dumps the whole request so there is something to look at later. Now the secret is sitting in a log file, in plain text, forever, and the log file gets copied to places nobody is thinking about.

So the rule is not “scramble the column.” That is the easy half. The rule is that the readable version must never exist anywhere it can be captured — and logs are the place people forget is a place.

The house counts, not the guest

Every plan has limits on it. Those limits live in exactly one table in the code, and they are enforced on the server, on every request.

Never in the browser. The browser is a convenience. It is the guest, and if you let the guest count how many people they brought, the guest will count generously.

The deeper point is not really about billing. Any rule that lives in two places will eventually give two different answers, and the wrong one will be the one that runs.

If you are starting this yourself

Write the rules down. Not the architecture diagram, the rules — the things that must never be true. Each one earns its place by having nearly broken something.

Then ask the harder question about each one: how would I know if this were violated right now?

If the answer is “I would spot it in review,” it is decoration. If the answer is a command you can run, a rule the database refuses to break, or a shape that makes the wrong version fail to build, then it is real.

Separation you can check beats separation you are hoping for. That is the whole thing.