The Single Context Problem
The Single Context Problem
All money in an account can only be described by the account itself. Individual pieces of money inside the account are indistinguishable from one another.
This is caused by what should be a good thing: having authoritative balances for an account. If the money in your account boils down to a single number, then it's just that: a number. It can't have properties of its own, or be described in parts. Instead we describe the number by adding properties to the account which describe the money it holds.
Let's illustrate this by creating an account to keep track of apples, and making some deposits.
| Line# | Credits | Debits | Balance |
|---|---|---|---|
| 1 | 0 | 0 | 0 |
| 2 | +3 | 0 | 3 |
| 3 | +6 | 0 | 9 |
We start with a 0 apples apples on line 1. Then on line 2 we add 3 green apples and on line 3 we add 6 red apples, for a total of 9 apples. Next, we'll add a 4th line and take some apples away
| Line# | Credits | Debits | Balance |
|---|---|---|---|
| 1 | 0 | 0 | 0 |
| 2 | +3 | 0 | 3 |
| 3 | +6 | 0 | 9 |
| 4 | 0 | -4 | 5 |
Line 4 removes 4 apples, and we have a balance of 5. Pretty easy to follow. What kinds of apples do we have in the balance? All that we can say deterministically is that we have 5 apples. Since the balance is just a number (5) it can only be described by its account, which just says "Apples". If we want finer grained information we'd have to break this into two accounts, one for green apples and another for red apples.
Can't we use the transactions?
A common first approach when encountering this problem is to try to use the transactions to break down the existing balance. The short answer is that this doesn't work. In order to make it actually function you need some sort of data structure that breaks down the balance into transactions, and keeps track of which credits are matched to which debits. This becomes incredibly complex as the account has more volume, but also means that the balance itself is no longer authoritative. Your new data structure is the actual source of truth for the account.
The specifics of this problem and some attempted solutions are covered in depth in The Attribution Problem
Why is it a problem?
It's not... at first. The insidious part of this problem is that as the systems mature it will cause a huge amount of work for you. Let's illustrate the problem with a simple account design and talk about the problems that arise as it gets more complex.
In this example we'll use a simple reseller store. You can imagine this being Amazon, or someone's Etsy shop. We need to take in money, and then pay the out the sellers for what we owe them. Let's start with a really simple breakdown: Payments (incoming money), Escrow (held money), and Seller (outgoing money to the sellers).
These are the only 3 accounts we have for the entire system. Our goal is to just understand, at a high level, the context of the money in our system.
This works well for a while. It's super simple, very intuitive, and is enough for us to create a balance sheet for the company (money out - money in = balance). It answers three questions:
- How much money do my buyers owe me total?
- How much money does the system hold?
- How much money do I owe my sellers total?
If you are a pre-seed startup this might be all you need to get off the ground, but likely won't hold up past that. Eventually your investors, and your CEO will want to know more about the money inside your system. Let's add a logical progression and try to answer the question "How much do I owe each seller?" instead of just total outgoing balance.
How do we evolve?
The single context problem says that we can't break down a number inside an account. So if we want to answer this question we only have two options:
- Create smaller accounts
- Solve the problem at product level instead (above the financial system)
I won't address the second option much, since it'll be covered extensively in The Attribution Problem, except to say that if you go this route you are signing yourself up for complicated reconciliation between the two systems.
Instead we'll choose the only pure account solution, and break up our larger seller account into smaller ones each for an individual seller.
A Revisionist History
Alright, so we've been running our system for a while with these three accounts and now we want to split out the individual seller accounts. We come up with a plan to have a user attribute on our account model and we'll change our business logic to move money from escrow to the seller specific accounts.
//Simple account model with user ownership
model Account {
name: string;
user: "System" | string;
balance: number;
}
This is where we'll hit some major snags with the single context problem. By changing our business logic we can route new money to the individual accounts, but what about the money currently in our global seller account? The global seller account doesn't have the individual information (single context) so how do we break down and get the money into the new accounts?
You can't. The authoritative balance is a sum. The cumulative property of addition makes it impossible to deterministically split a sum into its individual parts. The proof for this is covered in detail in The Attribution Problem, but just trust me if you don't feel like digging into the details.
The math is provable, but what we're missing is that there is information other than just the balance. We've got transactions, and probably some product data about what we sold, and how much the seller should have gotten. That should be enough, right?
The transactions on their own aren't enough to be able to deconstruct the account into its pieces, even if they have good detailed data already attached to them (also covered in The Attribution Problem). The product data (billing information, inventory, etc) likely knows what the pieces should have been, but not what should still be in the system.
In order to deconstruct the balance as deterministically as possible you'll need replay transactions joined to product data, keeping track of the individual parts until you get back to the current state, and then look at the pieces. Unfortunately, you'll have to replay all transactions either since the beginning of time, or since there was an exact 0 balance in order to get a close approximation. Even then it will depend on the quality of both your transaction and product data.
Even under perfect circumstances you are looking at a lot of work, which will only increase as your system grows to more accounts and more transactions over time.
The worst part? You'll end up doing this often. Not every new question will require breaking into smaller accounts, but most will. Here are some examples of questions you might want to ask the system which would require breaking down these accounts even further:
- "Did we pay a seller too much for a product?"
- "How much of the incoming is still a receivable, versus how much we've actually been paid?"
- "How much does a customer still owe us?"
- "Do we have any payouts that haven't fully cleared yet?"
- "Does this seller owe us any money?"
These are just some relatively simple questions to start with, but it's a never ending set of changes. Companies and systems are always looking for other dimensions to understand their data and most will require more detailed, more specific accounts. If you choose this route, but ready to have an operational team dedicated to refactoring and rerouting your account structures or telling your product/data organizations that you can't produce more detailed data.
The Workflow Problem
The endless procession of breaking down accounts may be annoying, but its not impossible. Especially if you only route new money to new accounts and never try to break out the money in the old accounts (which is the most common way I've seen it handled).
Remember our big account that was simply "How much money do we owe sellers?". That number wasn't useless. We came up with that account because its valuable information, but what do we do now that the account doesn't exist anymore?
The most straight forward approach is to sum up all the seller accounts and we get the overall balance for the system. Hurray, we've got the best of both worlds!
Or... do we?
Now we've added a couple of problems to the system. We have the concept of an overall seller balance, but its not authoritative anymore. Instead, its computed from many smaller balances. It depends on point-in-time information from those smaller balances, and there might be a huge number of them if your business has a lot of sellers. How do we keep it in sync? How do we know the difference between authoritative and non-authoritative balances in our system? How do we know which accounts should be summed to find this balance?
Compounding matters further, we now might need to change the underlying accounts to accommodate questions you want to ask of the higher level balance. As an example, let's say for legal reasons we want to know how much we owe just the sellers who are based in Europe, or will be paid in Euros. The seller accounts don't have the information, so thanks to The Single Context problem we'll need to split out smaller accounts again, so that we can roll up only the correct accounts into the new aggregate regional accounts.
After that change, now our individual seller balances themselves need to be aggregations of the region specific seller accounts. Additionally, all the systems that were moving money from escrow to seller accounts now need to be able to determine which regional seller accounts the money should be moved into. This ties your business logic to your accounts structure in way that makes changing either a very slow and risky endeavor. I call this The Workflow Problem.
Now we have to manage aggregate accounts, some of which will be critical to the workflows of our systems. We also need to implement some kind of discovery mechanism both for aggregate accounts to define their parts, and for workflows to discover the correct account to move money into.
Symptoms
Wondering how if your current system is suffering from issues rooted in The Single Context Problem? Here are some of the symptoms you might notice:
- You are constantly breaking accounts into smaller, more specific ones
- You need to create new, smaller accounts even when the information already exists in the system
- When you change your accounts you have to change upstream and downstream systems
- Upstream and downstream systems have logic in them to choose the right sub-account to move money from/to
- Aggregate accounts are often poor quality balances because the underlying accounts change, but the aggregate doesn't get updated
- Every few years you do a big overhaul from "user focused" accounts to "company focused" accounts or visa versa
- Changing your financial systems is very slow and complex, since small account breakdowns require rewiring other systems
Mitigation
I have an overall suggestion for dealing with most account issues holistically which I'll discuss in the Non-Fungible Money post but here are some strategies which will help you reduce the impact of The Single Context Problem specifically.
Discoverability
Accept that eventually your accounts will be quite small and fragmented. Focus on building a robust discovery system for your accounts. Your aggregate accounts should be able to declare their dependencies and not have to constantly update as the root accounts become smaller. Similarly, the systems moving money into/out of accounts should be able to rely on abstractions and have the money be routed behind the scenes to more specific accounts.
In order to help with this you'll want a robust data model for accounts themselves which give you lots of dimensions to query on.
Start Tiny
If you have the luxury of starting from scratch: define what you believe is the smallest possible account for your system, and build everything up from that. This will help you have a clearer definition of which accounts have authoritative balances, and what is computed.
Unfortunately, it's pretty easy to get this wrong. If you are newer to financial systems, however specific you think a "tiny" account needs to be, its probably even smaller. Also, the complexity of these dimensions tends to develop over time, so predicting exactly how it will grow is difficult. If you misjudge the right granularity of account you'll end up breaking down accounts and rewiring them as described earlier in the article.
Tiny accounts can be an effective strategy for solving several of the account problems, but require very specific constraints to address the different issues. Crafting the perfect tiny account constraints can be hard and is covered more in the Non-Fungible Money post at the end of the series.
Embrace Asynchronous Aggregation
It's impossible to synchronously update every aggregate account any time that any of the transitive dependency balances are updated. With only a couple of tiers of accounts your highest level balances are likely to depend on hundreds or thousands of small account balances. Doing all that updating transactionally will slow your systems to a crawl, if it functions at all.
Instead, focus on using strategies to address the right level of account and need for consistency.
- Build near real time pipelines for aggregate accounts that need to be mostly up-to-date (like seller balance)
- Build daily pipelines for higher level aggregates like company balance sheets, or executive metrics.
- Assume that aggregate accounts will always have at least slightly stale balance data when making balance decisions or displaying them.
- Figure out how you will identify aggregate accounts with overlapping balances (the contain balances from the same small account). This issue will come up very often when generating reports or metrics and is devious to try to unravel without knowledge of what's been counted.
These strategies will help you have performant and functioning aggregate accounts, but means you'll have to deal with the balances being stale (out of sync). Make sure your business logic can't make decision based on these stale balances or you're likely to end up with overcharge or over pay scenarios.
Up next
Still interested in account systems? Well, (un?)fortunately the problems don't stop here. Next we'll dig into why it's impossible to decompose balances deterministically and how that will affect your system in The Attribution Problem.