The Cost of Convenience
Part I: The Problem
1. The Bill Coming Due
Somewhere in your company, right now, someone is making a decision on data that is wrong.
Not maliciously wrong. Not catastrophically wrong. Wrong in the quiet, compounding way that enterprise data goes wrong: a customer record pointing to an address from three moves ago, an inventory count reflecting yesterday’s 3 AM batch instead of this morning’s receipts, a BOM listing a part number the engineering team retired six months ago but nobody removed from the manufacturing side. The decision will be made, the transaction will post, and nothing will visibly break. The consequences will show up two quarters later as a reconciliation variance, a customer complaint, a shipment that didn’t arrive, an audit finding that takes three people a month to close.
The standing cost of running a modern enterprise on dirty data is enormous. Gartner has estimated the annual cost of poor data quality at $12.9 million per organization. Experian has put the share of company data that is inaccurate at roughly 30%. Harvard Business Review has reported that 47% of newly created data records contain at least one critical error. These numbers are not news to anyone who has worked inside a mid-sized or larger company. They are the weather.
What is less widely acknowledged is that this weather is not natural. It was manufactured, deliberately, one convenient shortcut at a time, over the course of about thirty years. And the bill is coming due now, not because something new broke, but because the accumulated debt of three decades of “just add a field” and “we’ll write a quick integration” and “the users can customize it themselves” has finally grown large enough that it cannot be refinanced.
This essay is about how that debt was accumulated, who benefited from accumulating it, and what it would take to stop.
2. Three Symptoms of One Disease
The dysfunction shows up in three places, and most people treat them as separate problems. They are not.
Dirty data is the most visible symptom. It is the symptom the CFO can see when the close takes ten days instead of five. It is the symptom the VP of Sales can see when the CRM shows three different “lifetime values” for the same customer depending on which dashboard is open. It is the symptom the plant manager can see when the inventory system says there are 400 units on the shelf and the shelf has 237.
Spaghetti integrations are the second symptom. Ask any IT leader how many systems their company runs and the answer will be somewhere between five and twenty. Ask how those systems stay in sync and the honest answer will involve some combination of a nightly batch job that runs at 3 AM, a tangle of point-to-point APIs built at different times by different people now at different companies, and a few critical spreadsheets emailed around on a schedule nobody formally owns. The integration layer is load-bearing, undocumented, and maintained by attrition.
Decision latency is the third, and it is the one that hurts most but shows up least on any dashboard. It is the gap between the moment a fact becomes true in the physical world (a part arrives at receiving, a machine goes down, a customer places an order) and the moment that fact is trustworthy enough in the data to act on. In a well-run company that gap is minutes. In most companies it is hours to days. In bad cases it is permanent: the fact never becomes trustworthy, because by the time the batch has run and the reconciliation has completed and the exception has been cleared, the physical world has moved on and the data is describing a state that no longer exists.
These three symptoms look like separate problems (a data quality problem, an integration problem, and an analytics problem) and they are treated by separate vendors selling separate products. They are not separate. They are the same disease showing up in three different tissues. The disease is that somewhere, years ago, a decision was made to optimize for the convenience of the person entering the data or writing the integration or shipping the feature, at the expense of everyone downstream of them. And that decision was made not once but continuously, across every layer of the enterprise stack, for thirty years.
3. Quantifying the Invisible
The reason the debt has been allowed to accumulate is that nobody sends an invoice for it. The cost of a dirty data record does not arrive as a line item. It arrives as friction: slightly slower close cycles, slightly worse forecasts, slightly more time spent in reconciliation meetings, slightly lower trust in the number on the dashboard. Friction does not look like a cost until you measure it.
Consider a single custom field.
An operations director in 2019 asks for a field to track “shipping preference” on the customer record. It takes the admin twenty minutes to add. The field is free, or so it seems. But over its lifetime, that field accrues cost in at least seven places:
Maintenance. Every schema change in that table now has to consider the field. Every upgrade. Every migration script. Every test suite.
Documentation. Somebody has to explain to every new user, integrator, and BI analyst what the field means, what values are valid, and when it applies. Most of the time, nobody does, and the knowledge dies with the person who asked for the field.
Integration. Every system that reads or writes the customer record now has to decide whether to handle the field. Some will; most will quietly ignore it; the ones that handle it will do so inconsistently.
Data quality. The field is not in the required-fields list, is not validated, and is populated by a subset of users who remember it exists. Its values drift. Free-text variants proliferate. “FedEx,” “Fedex,” “FEDEX,” “FDX,” and “Fed Ex Ground” all appear in production within eighteen months.
Reporting. Someone builds a report that groups by shipping preference. The report is wrong, because the field is wrong, but nobody knows the report is wrong because nobody has audited the field.
Migration. Five years later, the company moves to a new ERP. The migration team finds the field, has no idea what it was for, cannot reach the person who requested it (they left in 2021), and has to make a judgment call: map it, drop it, or carry it forward as a free-text note. The judgment call will be wrong at least some of the time.
Trust decay. Every user who notices the field is unreliable trusts the customer record slightly less overall. The cost of that distrust is not measurable per-field, but it is the mechanism by which “the system” stops being treated as authoritative and the spreadsheets start.
Rough estimates on the lifetime cost of a single custom field in an enterprise system run between $1,500 and $2,500 depending on how it’s counted. Most organizations have somewhere between 50 and 500 custom fields across their primary systems of record. The math is not subtle. A midsize manufacturer with 200 accumulated custom fields is carrying between $300K and $500K of standing data-model debt, most of which will only materialize as a cost at migration time, when it will be ten times more expensive to resolve than it would have been to prevent.
That’s one shortcut, repeated a few hundred times, and it is the smallest of the convenience decisions worth examining. The others are larger.
Part II: The History
4. Convenience as a Product Feature
The accumulation of data debt did not happen by accident, and it did not happen because the people involved were careless. It happened because, beginning sometime in the early 1990s, convenience stopped being a side effect of enterprise software and started being a product feature, something customers asked for by name and vendors competed on directly.
The pivot is visible in the marketing material of the era. The first-generation ERP systems of the 1980s (the mainframe MRP-II suites, the early SAP R/2 installations) were sold on the promise of enforcing process discipline. The pitch was that the software knew what a manufacturer’s data should look like, and the company’s job was to learn to operate within that model. The implementation was painful precisely because the software was rigid. Rigidity was the point.
By the mid-1990s the pitch had inverted. SAP R/3 introduced configuration tables of a scope and depth that no prior system had contemplated. Oracle Applications offered a “flexfields” mechanism that let customers add structured custom fields to nearly any object. JD Edwards built an entire competitive position around being “configurable without programming.” Siebel, PeopleSoft, and later Salesforce extended the pattern into CRM and HR. The message to the buyer had changed from “learn the system” to “the system learns you.”
This was a real improvement in some respects. A rigid system that cannot accommodate genuine business variation will be worked around by its users or rejected by the market. But the pendulum swung past “enough flexibility to be adopted” and kept going. By the late 1990s the enterprise software industry had internalized an assumption that was never seriously examined: that more customization was always better, that every customer’s requirement was legitimately unique, and that any constraint imposed by the vendor’s data model was a limitation to be engineered around rather than a discipline to be honored.
Two commercial incentives drove the shift, and both are still operating.
The first was the implementation economy. By 1998, the services revenue attached to a large SAP or Oracle implementation exceeded the license revenue by a multiple of three to five. The systems integrators (Accenture, Deloitte, IBM Global Services, and their competitors) were not paid to install stock software. They were paid to configure, customize, extend, and integrate. Every custom field, every bespoke workflow, every parallel table was billable work. A rigid product would have collapsed the services market around it. A flexible product grew that market indefinitely.
The second was the RFP dynamic. Enterprise buyers in the 1990s and 2000s wrote requirements documents that listed hundreds or thousands of specific features. The procurement process rewarded the vendor whose sales engineer could say “yes” to the most line items, regardless of whether the feature was core to the product or a customization that would cost the buyer ten times its face value over the system’s life. Vendors who answered “no, you don’t need that, here’s how we handle it” lost the deal to vendors who answered “yes, we can add that as a custom field.” The buyer’s procurement team celebrated the win. The buyer’s data team inherited the debt.
Low-code and no-code platforms, which arrived in force during the 2010s, are the most recent expression of the same pattern. The pitch (“anyone can build it”) is a direct descendant of the 1990s flexfield pitch, and it fails in the same way. Anyone can build it means nobody owns it. Every citizen-developed application is a custom field with a user interface attached, and the cost of that application over its lifetime follows the same curve: cheap to create, expensive to maintain, impossible to migrate, and quietly corrosive to the data it touches.
The pattern across thirty years is consistent. The enterprise software industry sold convenience to the people who bought the software, and extracted the cost from the people who used it, maintained it, and eventually had to replace it. The two groups are not always the same group, and when they are not, the transaction is stable. When they are, as in small and mid-sized manufacturers where the owner is also the operator, the pain is felt directly and the purchase is usually regretted. The regret arrives too late to influence the market, because by the time it arrives the company is already locked in.
5. What Works: Systems That Chose Rigor
To understand what the enterprise data world failed to do, it is useful to look at domains that succeeded.
Four examples are worth examining in detail, because each one faced exactly the kind of diversity pressure that enterprise software has used to justify its customization culture, and each one resolved it by choosing the opposite strategy: impose a constrained, universal standard, accept the short-term pain of conformance, and let the network effects of shared vocabulary do the rest.
TCP/IP. In the 1970s, when networked computing was being built, the industry was fragmented across a dozen incompatible network protocols: IBM’s SNA, DEC’s DECnet, Xerox’s XNS, Novell’s IPX, Apple’s AppleTalk. Each vendor had strong commercial reasons to preserve its own protocol. Each protocol’s customers had strong operational reasons to keep theirs (training, tooling, existing deployments). The rational outcome, under vendor incentives alone, would have been permanent balkanization. TCP/IP succeeded anyway. Not because it was technically superior (in many ways it was inferior to the proprietary alternatives at first) but because it was radically simple, provider-neutral, and free to implement. The constraint was severe: everyone uses the same addressing, the same packet format, the same handshake semantics. The diversity that resulted (the entire internet, every protocol layered on top of it, every device that now connects) is a direct consequence of that constraint. If TCP/IP had allowed vendors to add custom header fields to the base protocol, there would be no internet. There would be a Balkanized collection of vendor-specific networks with translation gateways, which is exactly the situation the enterprise data world is in today.
The Dewey Decimal System and ISBN. Before Dewey, libraries organized their collections by local convention: by acquisition date, by accession number, by the librarian’s preference, by donor. A book cataloged in Cleveland was not findable in Boston without a Cleveland-trained librarian. Dewey imposed a universal taxonomy: every book in the world can be classified under one of ten top-level divisions, subdivided recursively. ISBN, added later, gave every published book a globally unique identifier. Together they turned “find this book” from an interaction with a local custodian into a machine-tractable lookup against a universal registry. The constraint was severe. Publishers had to adopt ISBN; libraries had to accept Dewey’s categorical choices (some of which are idiosyncratic or outdated). The payoff was interlibrary loan, global catalogs, the entire field of bibliometrics, and eventually the architecture of web search, which inherits Dewey’s premise that meaning lives in a shared namespace, not a private one.
The ISO shipping container. Before 1956, cargo was moved in the form the shipper chose: barrels, sacks, crates of irregular dimension, loose bulk. Every transfer between ship, rail, and truck required hand-labor to unpack and repack, and the transfer dominated the cost of shipping. Malcolm McLean’s insight was that if every cargo piece was packaged in a box of the same dimensions, every port and every truck and every train could be designed around that box, and the cost of transfer would collapse. The constraint, a few standard container sizes, rigorously enforced, was adopted over furious industry resistance from longshoremen, shippers, and port operators whose existing businesses depended on the inefficiency. Within twenty years, global shipping costs had fallen by more than 90%, and the volume of world trade had multiplied accordingly. Containerization is the single largest logistics improvement in human history, and it happened because one dimension of the problem, the shape of the box, was constrained absolutely.
SWIFT and the international banking network. Cross-border payments in the 1960s were a bespoke process of telexed instructions between correspondent banks, each using its own message conventions. SWIFT, launched in 1973, imposed a rigorously standardized message format (the MT series, later ISO 20022) that every participating bank had to implement. The standard was not optional, and it was not extensible by individual banks. The payoff was that a payment instruction composed in Tokyo was machine-readable in Zurich without translation. The global financial system runs on this constraint. Every attempt to layer “flexibility” onto SWIFT (proprietary message extensions, local dialects, vendor-specific add-ons) has been resisted by the governance body precisely because the value of the network is the uniformity of the protocol.
These four examples have three properties in common, and the presence of all three is what distinguishes a standard that succeeds from one that does not:
The vocabulary is fixed. Not mostly fixed. Not extensible by well-meaning participants. Fixed. A TCP/IP packet header is the shape it is. An ISBN is thirteen digits. A shipping container is forty feet by eight by eight (or one of two other sanctioned dimensions). A SWIFT MT103 message has the fields it has and no others. The participants in each system can do anything they want with what is inside the standard: the payload of the packet, the contents of the book, the goods in the container, the parties to the payment. They cannot modify the envelope.
The governance body does not benefit commercially from the standard’s adoption beyond its existence. The IETF does not sell TCP/IP implementations. The Library of Congress does not sell Dewey numbers. ISO does not profit from container shipments. SWIFT is a cooperative owned by its member banks. This neutrality is not incidental. It is the precondition that makes universal adoption rational for the participants. If the standard’s custodian were also the standard’s largest commercial beneficiary, every competitor would have a reason to propose an alternative.
The constraint is enforced at the infrastructure layer, not the application layer. A non-conforming packet is not delivered. A non-ISBN book is not findable in the catalog. A non-standard container does not fit the crane. A malformed SWIFT message is rejected by the network. The enforcement is automatic and happens before the participant’s intent is consulted. This is the feature that distinguishes an actual standard from a suggestion. Every successful standard in this list has a moment where a deviation is rejected by a mechanism the participant cannot override.
Hold these three properties in mind. They will reappear in Part III as the exact properties that the enterprise data standards of the last thirty years have tried, and failed, to achieve.
6. What Almost Worked: Canonical Data Models in Enterprise Software
The enterprise software world has not been unaware of the value of shared standards. Several serious attempts have been made, over more than thirty years, to produce a canonical data model for business operations, something that would do for the language of commerce what TCP/IP did for the language of networks. Each attempt has been technically credible, organizationally substantial, and commercially supported. Each has failed, and the specific manner of failure is instructive.
OAGIS (Open Applications Group Integration Specification) was founded in 1995 by a consortium of ERP vendors and large enterprise customers with the explicit goal of standardizing business object definitions (purchase orders, invoices, shipments, work orders) across vendor boundaries. OAGIS is still active today. It is used in production at thousands of companies. It would be wrong to call it a failure in absolute terms. But it did not do what its founders intended, and it did not become the enterprise data equivalent of TCP/IP. The reason is that OAGIS is fundamentally extensible. The standard provides a canonical envelope for a business object, and then permits arbitrary extension through a mechanism called user-area elements. Every OAGIS adopter has added its own extensions. The result is that an OAGIS message between two systems still usually requires a mapping translation, because the canonical fields are a minority of the message and the extensions are where the real business content lives. OAGIS succeeded at the envelope and failed at the vocabulary.
ISA-95 is the IEC/ISO standard for manufacturing operations management, first published in 2000. It defines a model for how enterprise systems (ERP) should exchange data with manufacturing execution systems (MES). It is rigorous, well-specified, and widely cited. It is also scoped narrowly enough that it addresses only one interface in one domain. A manufacturer that adopts ISA-95 has solved the ERP-to-MES problem and still faces the CRM problem, the supplier-portal problem, the quality-system problem, and the warehouse problem, each of which has its own standards-attempt with its own scope and its own extension mechanism, and none of which agree with each other on what a “customer” or a “part” is.
UN/CEFACT, ebXML, and RosettaNet were three overlapping initiatives in the late 1990s and early 2000s to define a universal XML-based vocabulary for B2B commerce. All three produced large, careful, academically impressive specifications. None of them achieved broad adoption. The failure mode in each case was a combination of governance drift (the specifications grew faster than the implementations) and the absence of an infrastructure layer that could enforce conformance. A company that produced a non-conforming ebXML message was not punished by the network, because there was no network. Each trading partner implemented its own interpretation, and “conformance” was whatever the two parties negotiated in a point-to-point integration.
The Microsoft–SAP–Adobe Open Data Initiative, announced in 2018 and quietly retired around 2023, was the most recent and in some ways the most ambitious attempt. Three of the largest enterprise software vendors in the world jointly announced a shared customer data model that would allow data to flow across their respective platforms without translation. The marketing was polished. The technical work was real. The outcome was that five years later, a customer record created in Dynamics, SAP, and Adobe Experience Cloud still requires bespoke mapping to move between the three, and the initiative has been superseded by each vendor’s own proprietary data platform strategy. The failure mode is worth naming clearly: the standard’s custodians were also its largest commercial beneficiaries, and each had a stronger incentive to differentiate its implementation than to honor the shared model. When differentiation pays more than cooperation, cooperation loses. This is a classical Nash-equilibrium outcome, and it is the outcome that any vendor-led standard will produce, every time, without exception.
The Kimball Group’s critique of industry-standard data models, published in 2010 and pointed enough to still be quoted, captures the pattern from the practitioner’s side. The critique was not that canonical models are bad in principle. It was that the canonical models actually being sold (by consultants, by vendors, by industry bodies) were not the time-savers they were marketed as. They had been designed in committees, carried the compromises of every constituency, and required so much local adaptation to be usable that the team would have been faster starting from a clean sheet. Kimball’s point was not that standardization is wrong. It was that the specific canonical models on the market had been so diluted by the incentive structure of their creation that they had lost the property of constraint that would have made them valuable.
Across all of these attempts, the failure mode is consistent and diagnostic. Every one of them failed the three tests that the successful standards of the previous section passed: the vocabulary was not fixed (it was extensible), the governance body benefited commercially from adoption (or its members did), and the constraint was not enforced at an infrastructure layer (conformance was honor-system). Any one of these three failures is fatal. All three together guarantee that the standard will exist only as a marketing artifact, not as operational reality.
7. Why the Pattern Holds
The lesson across thirty years of enterprise standards work is not that standards are hard (they are, but other domains manage them) or that vendors are malicious (they aren’t; they are rational actors responding to the incentives they face). The lesson is that an enterprise data standard is governed by a specific incentive topology, and that topology has to be structured deliberately, or the standard will degrade along a predictable trajectory.
The topology has three actors: the vendor of a system of record, the buyer of that system, and the consultant who implements it. Each has a local interest that, when pursued, produces a globally worse outcome.
The vendor’s local interest is to differentiate its product, which means providing extension mechanisms (custom fields, custom objects, custom workflows) that customers can use to express their uniqueness, and that, once used, make the customer unable to leave without a migration project. The extension mechanisms are the vendor’s retention strategy. A vendor that removes them is a vendor that shortens its contract renewals.
The buyer’s local interest is to satisfy the specific requirements of its operations team, its finance team, its sales team, and its executive sponsor, each of whom has real needs and little incentive to compromise for the sake of a data model they do not see. The buyer’s procurement process rewards vendors who say “yes” to every requirement. The buyer’s implementation team builds the custom fields because they were asked to. The buyer’s future migration team inherits the cost, but that team does not exist yet, and so its vote is not counted.
The consultant’s local interest is the continuation of the engagement. A consultant who delivers a stock implementation finishes in six months and is not re-engaged. A consultant who builds custom extensions and bespoke integrations is engaged for years, across multiple phases, with an annuity of maintenance work. Neither the consultant nor the consulting firm has an incentive to advocate for structural simplicity.
In this topology, the absence of a constrained, provider-neutral, infrastructure-enforced standard is not an oversight. It is the equilibrium. Every participant has rational reasons to prefer the status quo, and no single participant has the leverage to change it. This is why every attempt to produce a canonical model from within the industry has failed: the industry’s incentive structure is specifically aligned against the properties that would make the model work.
The domains that succeeded (TCP/IP, Dewey, containers, SWIFT) did not succeed because their participants were more virtuous. They succeeded because the standard’s custodian sat outside the commercial interest of any single participant, because the standard was structurally incapable of being extended by individual adopters, and because the consequence of non-conformance was imposed by the infrastructure rather than negotiated between parties. Those three conditions are necessary. They are also sufficient. And they are exactly the conditions that the enterprise software industry has, for thirty years, been structurally unable to produce from within itself.
So much for the history. Part III turns to what it has cost, and what it continues to cost, that these conditions have not been met.
Part III: The Connections
8. Dirty Data Is Downstream of Convenient Data Entry
The custom-field example in Part I was a single shortcut. The argument of this section is that every category of dirty data in the modern enterprise can be traced to a similar shortcut, a point at which somebody chose the convenient option for the person in front of them, and the cost was absorbed by every downstream system, process, and decision.
Consider the ways a customer record becomes wrong.
An account is created by a sales development representative at 4:47 PM on a Thursday because the lead has just asked to see a demo. The SDR types the company name as they heard it over the phone. The company’s legal name is “Acme Industries, Inc.” The SDR types “Acme Inc.” The CRM does not check. A second SDR, two weeks later, creates “Acme Industries” for the same company because nothing surfaced the duplicate. A third SDR, three months later, creates “ACME” because the search was case-sensitive and didn’t find the other two. The company is now three records, and nothing in the system considers this a problem until a CFO runs a lifetime-value query and gets a number that is a third of what it should be.
The root cause is not that the SDRs were careless. It is that the data entry form was designed to be fast, not correct. Checking for duplicates requires a search. Validating a legal name requires a lookup against a registry. Enforcing a canonical form requires constraints. Each of these checks has a cost in user friction: the form becomes slower, the user sometimes has to do extra work, the autocomplete sometimes surfaces results the user has to think about. The cost is real, and it is paid by the person entering the data. The benefit, a customer record that can be trusted, is paid to everyone else, and later, and indirectly. The incentives point the wrong way. A designer optimizing for the measurable thing (form completion rate, time-to-first-record, abandonment) will remove the checks. A designer optimizing for an unmeasurable thing, the integrity of the downstream record, will keep them, and will lose the argument to anyone who has numbers.
The pattern is general. Dirty data is almost never the result of users being sloppy; it is the result of systems having been designed, deliberately, to accept sloppy input in exchange for faster adoption. The sloppiness is priced in at the point of sale, and it shows up as a cost at the point of use, five years later, in a room where the people who sold the system are not present.
The pattern repeats in every layer of the stack. Part numbers get entered without validation because “operations needs to get the work order in.” Inventory adjustments get posted without reason codes because “the shift supervisor doesn’t have time to pick from a dropdown.” GL accounts get mapped to expense categories by AP clerks who have never been trained on the chart of accounts because “we need the bill paid.” In each case, a structural decision was made somewhere upstream (by a software vendor, a system implementer, or a business analyst) to make the input side easy at the cost of the output side. The downstream consumers of that data are not represented in the room where the decision is made, because they are often downstream in time as well as in process. Future employees, future auditors, future migration teams, future AI systems trying to make sense of the history: none of them have a voice in the design review.
Dirty data is what the discipline that was skipped looks like from the other side.
9. Spaghetti Systems Are Downstream of Convenient Integration
The integration layer of a modern enterprise deserves its own accounting, because it is the place where convenience debt compounds most aggressively.
Every integration in an enterprise system exists because two systems that should be talking to each other cannot do so natively. The question of why they cannot has a long answer and a short one. The long answer involves schema mismatches, API generations, authentication protocols, data volume constraints, and the historical accident of which system was implemented first. The short answer is that the two systems were not designed to share a vocabulary, and so every conversation between them is a translation.
Translations have a cost structure that does not scale gracefully. A point-to-point integration between two systems is a fixed construction cost plus an ongoing maintenance burden proportional to how often either system changes. That is manageable for two systems. For five systems, there are potentially ten integrations. For ten systems, there are forty-five. For twenty systems, not an unusual number in a company of any size, there are one hundred and ninety. Nobody builds all of them. What gets built is a subset prioritized by urgency, funded by whichever business unit needed the connection most, owned by whoever was the lead engineer at the time, and documented, if at all, in a Confluence page that has not been edited since 2021.
The result is a topology that resembles a spaghetti bowl not metaphorically but literally. The CRM talks to the ERP, but only the sales module, and only for accounts created after 2019. The ERP talks to the warehouse system, but the translation loses SKU variant information because the two systems model variants differently. The warehouse system talks to the shipping carrier’s API, but only for domestic shipments; international ones go through a separate process that involves a CSV export, a manual email, and a third-party broker. The finance system reads from all of the above via a nightly batch that runs at 3 AM and fails about once a month in ways that take a day to diagnose because the error handling was written by somebody who no longer works there.
This is not a pathology. This is the ordinary state of an enterprise that has been operating for more than five years. The integrations exist because each one was, individually, the fastest way to solve the problem in front of whoever was solving it. None of them were built with a shared vocabulary, because there was no shared vocabulary to build against. Each one encoded, in its translation logic, that engineer’s understanding of what the two systems meant by “customer” or “order” or “shipment” on the day the integration was written. Those understandings have drifted since. The translations have not been updated.
The cost shows up in two places. The first is the integration itself, which requires constant maintenance and a standing headcount to own. The second, larger, cost is the latency introduced by every hop. A fact in the physical world (a part was received, a machine went down, a customer cancelled) has to propagate through a chain of systems, each of which introduces a delay, a transformation, and an opportunity to fail silently. By the time the fact is visible in the dashboard the executive is looking at, it is hours or days old and has been reshaped enough times that the original signal is noisy.
The operational consequence of this latency is that the organization cannot act on its own data. It acts on approximations: spreadsheets assembled from partial exports, tribal knowledge held by the people closest to the systems, intuitions formed by managers who have learned, over years, not to trust the official numbers. The official system becomes a ceremonial object. The real work happens in the margins.
This is what a spaghetti integration topology costs. It does not appear as a line item. It appears as the reason the company cannot close its books in five days, cannot promise a ship date with confidence, cannot see its inventory in real time, cannot compete with a better-coordinated competitor on speed. The cost of the integrations was paid up front, in cash, to the consultants who built them. The cost of the topology they compose is paid continuously, forever, by the business that runs inside it.
10. Bad Decisions Are Downstream of Both
Dirty data and spaghetti integrations are not the disease. They are the mechanism by which the disease is transmitted to the place where it does the most damage: the decision layer.
Every non-trivial decision in a modern company is made, at least in part, on data. Pricing decisions are made against historical margin analysis. Hiring decisions are made against headcount plans built from operational projections. Capital decisions are made against payback calculations that rest on utilization data. Strategic decisions are made against market-sizing exercises that depend on customer segmentation. In each case, the decision is only as good as the data it is based on, and the data, in most companies, is what the dirty-data-and-spaghetti-integration pipeline has produced.
The most generous estimate is that the data is roughly right. The average executive, making a decision from a dashboard, is operating on numbers that are approximately correct for most segments, slightly wrong for some, badly wrong for a few, and silently broken in ways that will surface only when a particular edge case is queried. The decision will be made anyway, because decisions cannot wait for data to be clean. Over time, the executive learns which numbers to trust and which to mentally discount. This learning is itself an expensive thing. It takes years, it is not transferable to a successor, and it encodes the dysfunction into the organizational tacit knowledge in a way that makes the dysfunction harder to fix.
This was the state of affairs before large-language-model AI became a standard enterprise tool. The arrival of AI has not improved the situation; it has amplified it.
An LLM operating over clean, structured, semantically consistent data is a powerful tool. An LLM operating over the typical enterprise data estate is a confident hallucination machine. The model does not know that “Acme Inc,” “Acme Industries,” and “ACME” are the same customer. It does not know that the custom field added in 2019 was abandoned in 2021. It does not know that the warehouse system’s “on hand” number reflects yesterday’s batch and not this morning’s receipts. It will answer questions fluently anyway, because that is what it is built to do, and its answers will look authoritative. They will be based on the same dirty data the human analysts have learned to distrust, but the LLM has not learned to distrust it, and the user of the LLM is unlikely to know what to discount.
The enterprise AI problem, in most cases, is not an AI problem. It is a data problem that has been made visible by putting a fluent generator on top of it. The generator did not create the dirty data. It just made the dirty data consequential in a new way, by converting it efficiently into confident recommendations that non-expert users are not equipped to second-guess.
Every enterprise AI program that begins with the question “what can we do with AI?” and proceeds to build a prototype against the existing data estate is going to discover this in order: first the prototype works surprisingly well on the demo data; then it fails surprisingly badly on the production data; then the team realizes that the production data is not usable; then the team proposes a data cleansing project; then the data cleansing project runs into the same obstacles that have prevented every prior data cleansing project from succeeding, because the dirty data is a symptom of a system design, not a transient condition.
AI is the bill collector on thirty years of convenience. It is not the one being charged (the organization is) but it is the one that will not politely look away.
11. The Rational Dysfunction
It is tempting, at this point in the argument, to assign blame. The villain could be the software vendor who sold the customizable product, or the consultant who built the custom fields, or the executive who mandated the implementation without understanding its consequences, or the user who entered the dirty data. All of these actors have contributed, and all of them could, in principle, have acted differently.
None of them acted irrationally.
The software vendor built a customizable product because customizable products close deals. The vendor that refused to allow custom fields lost to the vendor that allowed them. Over thirty years, this selection pressure removed the vendors who tried to sell constrained products from the market. The ones that remained are the ones whose products are customizable, because the alternative was not remaining.
The consultant built custom fields because the client asked for them and was willing to pay. A consultant who refused on grounds of long-term data hygiene would have lost the engagement to a competitor who was happy to say yes. The consulting firms that survived are the ones that said yes, billed for it, and moved on to the next client before the consequences matured.
The executive mandated the implementation because the board asked for the system by name, the budget was approved, and the executive’s tenure is measured in years while the consequences of the implementation are measured in decades. An executive who pushed back on the mandate in order to preserve long-term data hygiene would have been perceived as an obstructionist, and would have been replaced by someone who was willing to execute the mandate.
The user entered dirty data because the system was designed to accept dirty data and the user had a queue to clear. A user who insisted on entering clean data would have missed their quota, or their shift deadline, or their approval window. The users who remained are the ones who entered the data as fast as the system would accept it.
Each of these actors, in each of these moments, made the rational local choice. The dysfunction is not a failure of any individual to behave correctly. It is a failure of the system to reward the correct behavior. In the language of game theory, this is a coordination failure, a situation in which every participant would be better off if everyone cooperated, but no individual participant has an incentive to cooperate unilaterally. Coordination failures do not resolve on their own. They resolve either through the imposition of an external constraint that changes the payoffs, or they do not resolve.
For thirty years, the enterprise data world has not resolved. The constraint that would change the payoffs, a provider-neutral, infrastructure-enforced, non-extensible data standard, has not been imposed, because the actors who would have to impose it are the same actors whose local interest is aligned against it. And so the dysfunction has continued, and the debt has compounded, and the bill has grown.
That is the position the industry is in now, and the remainder of this essay considers what it would take to change it.
Part IV: The General Solution
12. Two Kinds of Signaling
Biology solves the coordination problem in two distinct ways, and the difference between them is the same difference that separates a well-run enterprise from a badly run one.
The endocrine system is a broadcast network. A gland releases a hormone into the bloodstream, the hormone reaches every cell in the body eventually, and the cells that are equipped to respond to it do so on a timescale of minutes to hours. The signaling is diffuse, redundant, and tolerant of noise. It works because the message is slow, the targets are many, and approximate delivery is acceptable. It is also the mechanism by which the body regulates long-horizon states: growth, metabolism, stress response, reproduction. Endocrine signaling is powerful for what it does. It is not what you want running your reflexes.
The nervous system is a point-to-point network. A neuron fires, the signal travels along a specific axon to a specific target, the target responds within milliseconds, and a feedback loop from the result informs the next signal. The signaling is precise, addressed, and intolerant of translation error. It works because the sender and receiver share exactly the same molecular vocabulary: the same neurotransmitters, the same receptor shapes, the same ion-channel mechanics. Nervous-system signaling is what makes purposeful action possible. A hand reaches toward a cup because the brain can send a signal to the specific muscle fibers that move specific fingers, and the proprioceptors can report back on the position in real time. Neither the command nor the feedback would work if the signal had to be broadcast through the bloodstream and interpreted by receptors that might or might not be tuned to recognize it.
Both systems are essential. A body without an endocrine system cannot maintain homeostasis. A body without a nervous system cannot act on intent. A body with only the endocrine system, no nervous system at all, is approximately the operational state of most modern enterprises.
The symptoms described in Part I are endocrine symptoms. The nightly batch job is a hormonal release: the signal goes out at 3 AM and reaches its targets sometime before lunch. The reconciliation meeting is a feedback loop with a twenty-four-hour latency. The spreadsheet emailed around on a schedule is a chemical signal distributed through a shared medium. Dashboards that show yesterday’s numbers are endocrine dashboards. The company can survive on this kind of signaling for the same reason an organism can survive on endocrine alone: for slow, stable processes, it works well enough. What the company cannot do, running on endocrine signaling, is react. It cannot close a work order the moment the last operation completes. It cannot reserve inventory the moment a customer places an order. It cannot detect a quality excursion the moment the measurement goes out of spec. It is structurally incapable of nervous-system behavior, because the infrastructure to support nervous-system behavior does not exist.
Building that infrastructure is what Part IV is about.
13. Why DNA Makes the Nervous System Possible
The nervous system is a specific biological achievement, and it is worth asking why it was possible at all.
A neurotransmitter (acetylcholine, dopamine, glutamate) is a molecule of a precise shape. A receptor on the post-synaptic neuron is a protein with a binding site that recognizes exactly that shape. When the neurotransmitter binds, a specific conformational change in the receptor triggers a specific downstream event. The signal is carried by the fit between molecule and receptor, not by the quantity of the molecule or the vigor of its release. This is why a vanishingly small amount of a neurotransmitter can produce a deterministic response: the coding is in the shape, and the shape is unambiguous.
The reason acetylcholine in one organism binds an acetylcholine receptor in another organism, the reason neural signaling is consistent across every animal that has it, is that the shape of acetylcholine is determined by the same 20 amino acids, assembled by the same ribosomes, reading the same 64 codons, from the same 4 nucleotide bases. The receptor is determined by the same machinery. The constraint of the genetic code is the upstream condition that makes the precision of the downstream signaling possible. If every organism used a different amino acid alphabet, the receptors would not be portable, the neurotransmitters would not be standardized, and precise point-to-point signaling would not exist. The body would have only the endocrine equivalent (rough, slow, broadcast) because there would be no way to build a molecular key that fit a specific molecular lock in a reliable way.
It is worth pausing on what biology does with this constraint, because the numbers are not intuitive and the argument does not land without them. Every living organism on Earth, every bacterium, every plant, every fungus, every animal, runs on the same 4 nucleotide bases, the same 64 codons, and the same 20 amino acids. That is the entire alphabet. Four letters in the base code, twenty building blocks in the protein alphabet, universal across 3.5 billion years of evolution, unchanged since before oxygen was common in the atmosphere.
From that alphabet, biology has produced the most diverse, most specialized, most extensively customized system that the human mind can reliably hold as real. Bioluminescent fish in the Mariana Trench at pressures that would collapse a submarine. Thermophilic archaea thriving in volcanic vents at 250°F. Tardigrades that survive radiation doses a thousand times lethal to humans, vacuum, and desiccation for decades. Mycorrhizal networks that shuttle nutrients across acres of forest floor. A peregrine falcon in a 240-mile-per-hour dive. An octopus rewriting its own RNA in real time to tune its neurons to the ambient water temperature. A blue whale with a four-hundred-pound heart. There is no human technology, no market, no civilization that comes close to the range of specialization biology has produced. Life is the ceiling of what a human mind can credibly conceive of as a specialized thing, and we have no referent for anything beyond it.
All of it runs on four bases and twenty amino acids.
This has a direct consequence for anyone claiming a constrained data model cannot accommodate their business. The claim is that their operation is too unique for a fixed vocabulary. The honest form of that claim is that their operation is more specialized than the blue whale, the tardigrade, the octopus, and the archaeon combined. It is not. Nothing a human enterprise does approaches the specialization pressure that biology navigates routinely, and biology navigates it with a vocabulary smaller than most ERP systems’ custom-field lists.
The reason the constraint produces the diversity, rather than limiting it, is that diversity is a property of configuration, not of structure. A human genome differs from a fruit fly genome not because humans have a 21st amino acid but because the same 20 amino acids are arranged in different sequences, in different quantities, under different regulatory control. The vocabulary is identical; the literature is infinite. The combinatorial space of twenty-letter sequences is large enough to encode every organism that has ever lived and every organism that ever will, with room to spare. There was never a need for a 21st letter. The need was always for better arrangements of the existing twenty, and the fixed encoding is what made those arrangements searchable, heritable, and improvable over evolutionary time.
A constraint severe enough to produce the blue whale and the tardigrade is severe enough to produce a manufacturing company in Harrisburg and a food processor in Dallas. The difference between the two businesses is configuration, not vocabulary. It has to be, because the alternative is to insist that the variation between two manufacturers exceeds the variation between two phyla, and that is not a serious position.
The biological lesson for enterprise data follows from there, and it is decisive.
Precision signaling requires a shared vocabulary. A shared vocabulary requires a constrained encoding. The constraint is not the price of precision; the constraint is the mechanism of precision. A signaling system that permits each participant to extend the vocabulary produces, at best, an endocrine system: a broadcast medium in which approximate messages reach approximate targets and the receivers have to interpret what was meant. A signaling system whose encoding is fixed and universal produces a nervous system: a targeted medium in which a specific message reaches a specific receiver and the response is determined by the binding fit, not by interpretation.
An enterprise that wants to operate like a nervous system (fast, precise, feedback-coupled) needs the same property in its data that biology has in its molecules. The vocabulary has to be constrained, the encoding has to be universal, and the receptors (the systems and processes that consume the data) have to be tuned to a shape they can recognize without ambiguity. An enterprise that permits every system to extend the vocabulary has chosen, whether it realized it or not, to remain endocrine. No amount of investment in faster pipes, better dashboards, or smarter AI will convert an endocrine enterprise into a nervous-system enterprise, because the limit is not the bandwidth of the signaling. The limit is the absence of a molecular alphabet that sender and receiver agree on.
This reframes every principle that would otherwise sound like an abstract design preference.
Immutability of meaning is the condition that every receptor in the organization is tuned to the same shape. If the meaning of “customer” drifts between systems (one system treats it as a billing entity, another as a shipping entity, a third as a support entity) then the message “this customer has an open issue” does not produce a deterministic response, because there is no single receptor for the signal. The shape is ambiguous, and the response is therefore also ambiguous. Immutability is not rigidity. It is the precondition for precision.
Provider neutrality is the condition that the vocabulary is not owned by any participant in the signaling. Biology’s genetic code works because no organism owns it. If acetylcholine were proprietary, licensed by one species and available to others only under revocable terms, the nervous systems of every other species would be at the mercy of the licensor, and the evolutionary equilibrium would push every species toward developing its own incompatible neurotransmitter rather than depending on one controlled by a potential rival. This is exactly what has happened in enterprise software. Every vendor has developed its own proprietary vocabulary, because depending on a rival’s vocabulary is commercially irrational. The result is that no enterprise can signal across vendor boundaries without translation, and translation is the endocrine fallback that every cross-vendor process actually runs on today.
Operational grounding is the condition that the vocabulary describes something real. A receptor that is tuned to a shape that does not exist in nature is not a receptor; it is a fiction. A canonical data model that describes a committee’s abstraction of what an enterprise ought to be, rather than the concrete structure of what enterprises actually are, produces receptors that no real process binds to. This is why most academic data models have failed, not because they were logically wrong, but because the shapes they defined were not the shapes that the operational reality presents.
Deterministic classification is the condition that a given molecule is recognized as a given molecule, every time, by every receptor tuned to it. Biology accomplishes this through chemistry. The binding is not a judgment call; it is a physical fit. Enterprise data has to accomplish the same thing through rules, deterministic classification procedures that assign a given piece of data to a given canonical entity the same way, every time, against the same input. Machine-learning classification, for all its power, does not meet this bar: it produces answers that vary with training data, drift with time, and cannot be audited to the satisfaction of a regulator or a migration team. The receptors of an enterprise nervous system have to be built from rules, not probabilities.
These four conditions are not a design preference. They are the biological preconditions for any system that aspires to operate by nervous-system signaling rather than endocrine broadcast. An enterprise that meets them can act on its own data in real time. An enterprise that does not meet them can broadcast, and hope, and reconcile after the fact. The difference between the two modes is the difference between a reflex and a rumor.
14. How to Buy Software Differently
The principles above apply to the supply side. The demand side, the buyers, can accelerate or retard the arrival of a viable canonical model by changing the way they buy.
The first change is to ask a different first question. The question most enterprise software buyers ask is “what system should we buy?” The correct first question, in 2026 and for the foreseeable future, is “can we classify the data we already have?” A company that cannot describe its existing data in terms of a shared vocabulary is not ready to buy a new system. It is ready to do the classification work that has to happen before any system change can be non-destructive. Every ERP migration is a data migration; every data migration that begins without a classification of the source is a migration that will lose information. The buyer who skips this step will pay for the skip at cutover, which is the worst possible moment to discover the cost.
The second change is to treat extensibility as a liability, not an asset. The RFP practice of listing hundreds of functional requirements and awarding the deal to the vendor who says “yes” to the most of them is the mechanism by which custom fields enter the enterprise. The buyer who rewards that mechanism, by selecting the vendor who promises to accommodate every request, is buying the debt that will mature ten years later as a migration crisis. The buyer who asks, instead, “which of our requirements can you meet without extending the data model?” is buying a different kind of system, one that has been forced to justify its defaults and whose defaults are therefore worth more.
The third change is to demand reversibility. A system that cannot be exited (because the data cannot be extracted in a form a successor system can load, because the customizations are undocumented, because the vendor will not support export) is not a system the buyer owns. It is a system that owns the buyer. The cost of reversibility at the point of purchase is usually negotiable. The cost of its absence at the point of exit is unbounded. Reversibility should be a hard requirement, on the same tier as security and compliance, and it should be tested before the contract is signed, not with a marketing promise but with an actual export, loaded into an actual successor, and verified for fidelity.
The fourth change is to account for the total cost of every custom field. The convention of treating customization as free labor (it is just a configuration, after all) understates the cost by orders of magnitude. The buyer who asks, for every proposed customization, “what does this cost over ten years?” will accept fewer customizations. The buyer who does not ask will accept them all, and will pay for them all, in the order in which they mature.
15. How to Build Software Differently
The supply side has its own corresponding changes to make, which some vendors are beginning to make and most are not.
The first is to treat the data model as the product. A system of record whose distinguishing value is its data model (rigorously defined, operationally grounded, deliberately constrained) is a different commercial proposition than one whose distinguishing value is the breadth of its feature list. The data model is harder to sell, because it is invisible in a demo. It is also harder to copy, because it encodes years of decisions that cannot be reproduced by implementing the same features on a different foundation. The vendors who are going to matter in the next decade of enterprise software are the ones who recognize that the data model is the durable asset and behave accordingly.
The second is to refuse customization that affects structure. A vendor can permit arbitrary configuration (workflows, statuses, business rules, validation logic, reporting views) without permitting extension of the data model. The distinction is not subtle, and it is the distinction that separates the systems that can be upgraded from the systems that cannot. Customers will ask for structural customization. The vendor’s job is to understand the request well enough to express it as configuration instead, and to decline the request where configuration cannot express it. This is a harder conversation than “yes, we can add a field.” It is also the conversation that produces systems that survive their tenth year.
The third is to build integration against shared vocabulary rather than shared schemas. Two systems that have agreed on what a “customer” means can exchange customer records without either system needing to know the other’s internal representation. This is how SWIFT works, how TCP/IP works, how every successful integration standard works. It is not how most enterprise integration works today, and the reason it is not is that there has been no shared vocabulary to build against. The arrival of a viable canonical model would change this, not by forcing vendors to abandon their schemas but by giving them a neutral surface against which to interoperate. Vendors who begin designing toward that surface before it exists will be positioned to benefit from it when it does.
The fourth is to treat audit as a first-class concern. Every data modification, every state transition, every configuration change, every integration event should be recorded in a form that can be reconstructed, inspected, and explained. This is sometimes framed as a compliance requirement, and it is, but its more fundamental purpose is epistemic. An organization that cannot reconstruct how a piece of data came to have its current value cannot reason about whether that value is trustworthy. An audit trail is not a regulatory burden. It is the mechanism by which the organization retains the ability to know what its own systems are telling it.
16. The Uncomfortable Conclusion
The thesis of this essay has been that the current state of enterprise data is the accumulated consequence of thirty years of rational local decisions that produced a globally dysfunctional outcome, and that the outcome can only be changed by imposing constraints that the participants, acting individually, will not impose on themselves.
This is not a hopeful thesis. It does not suggest that the problem is close to solving itself. It does not identify a hero vendor or a breakthrough technology that will cut through the accumulated debt. It suggests, instead, that the situation will continue to deteriorate until the cost of continuing exceeds the cost of changing, and that the cost of changing, now, is very high, and the cost of continuing, now, is still tolerable for most organizations. The equilibrium is stable because the pain has not yet exceeded the threshold at which coordinated action becomes rational.
Two forces are operating, slowly, to shift the equilibrium.
The first is that the installed base of legacy customizations is aging into un-maintainability. The engineers who built the custom fields, the custom integrations, the custom reports, are retiring. Their institutional knowledge is not being replaced. The systems that depend on that knowledge are becoming, year over year, harder to change. At some point, the cost of continuing to operate a given system exceeds the cost of replacing it, and the replacement decision arrives whether the organization is ready or not. The 60-70% failure rate of ERP migrations, well documented across decades of Gartner and Standish research, is the shape of these forced decisions playing out against the debt accumulated by the previous generation of convenience choices.
The second is that the arrival of AI has raised the value of clean data from a nice-to-have to a competitive necessity. An organization whose data is clean enough to be reasoned over by a model gains a compounding advantage over an organization whose data is not. The organizations that figure this out, and a meaningful number of them are figuring it out, will begin to reward vendors, consultants, and internal teams who prioritize data discipline over feature velocity. This is a slow change in the selection pressure, but it is real, and it is the first time in thirty years that the commercial incentive has begun to favor the discipline side of the trade.
Neither of these forces is going to resolve the situation on its own, and neither is going to resolve it quickly. The bill for thirty years of convenience is very large, and it is not going to be retired in a single cycle. The question for the organizations that are still accumulating the debt is not whether they will eventually pay it. They will. The question is whether they continue to borrow, at compounding rates, against a balance they are already struggling to service, or whether they begin, now, to pay it down.
Paying it down looks unglamorous. It looks like classifying data before migrating it. It looks like saying no to custom fields. It looks like replacing point-to-point integrations with a shared vocabulary. It looks like choosing vendors on the strength of their data model rather than the breadth of their feature list. It looks like hiring data-systems leaders and giving them structural authority over executive decisions that would otherwise compound the debt further. None of these moves produce a quarterly win. All of them produce, over a decade, a company that can act on its own data.
The companies that do this will outcompete the companies that do not. This is not a prediction. It is what has already happened in every prior instance where a domain resolved a coordination failure by adopting a constrained, provider-neutral, infrastructure-enforced standard. Containerized shipping outcompeted break-bulk shipping. Networked computing outcompeted proprietary networks. Standardized finance outcompeted correspondent-banking translation layers. In each case, the transition was painful, the incumbents resisted, the equilibrium held longer than anyone expected, and then it broke, decisively, in a period shorter than anyone expected.
The transition in enterprise data has not yet broken. It is going to. The organizations that position themselves now, by adopting the principles above, by buying and building differently, by treating data discipline as a strategic priority rather than an operational afterthought, will be the ones that are standing when it does.
The bill is coming. The only question is who pays it deliberately, and who pays it reactively.