Do You Trust Your Data? 5 Steps to Data Quality

When I walk clients through our FASTIR framework, my favorite question is a simple one: “Do you trust your data?”

The answers run across the spectrum, but they are rarely a clean yes or no. What’s typical is a patchwork, where some areas are rock solid and others are not.

The cost of that patchwork is easy to miss. One client was wasting time across the entire organization because people didn’t trust the data, so they were double and triple checking it before using it. Nobody budgets for that, yet everybody pays for it. And beyond the operational drag, don’t underestimate the reputational risk of bad data reaching a client or a regulator.

So what makes some areas trustworthy and others not? It is rarely luck, and it is rarely just the tooling. We developed Olmstead’s Data Quality Framework to answer that question. Success ultimately takes a combination of People, Process, Technology, and Data.

The 5 Steps of Our Data Quality Framework

Our Data Quality Framework outlines five steps for achieving data quality.

1. Assign Accountability. Most frequently, when data in an area isn’t trusted, the root cause is an accountability gap. There is an old saying: “When no one person is accountable, then no one is accountable.” Typically, the Data Owner assigned to a Data Domain is accountable for its quality. As a general principle, the business owns the data (not IT), and we strive to place ownership in the hands of Subject Matter Experts.
2. Define Quality. What does quality actually mean? It depends on the Data Domain, which is why Subject Matter Experts need to define it. They articulate the business requirements and validation rules. Think of these as the specifications that the data must meet.
3. Measure Quality. Test every rule. For each rule, does it Pass or Fail? Knowing that a rule passes is valuable in its own right: it proves the rule was tested, and that evidence is what builds confidence in the data. Knowing which rules fail gives Data Stewards something actionable to investigate.
4. Enforce Quality. Failures are Exceptions (aka Data Quality Issues). Route them to the appropriate Data Stewards for timely resolution. Defining, automating, and monitoring these workflows is key. An exception that lands in nobody’s queue is an exception that never gets fixed.
5. Track Quality Metrics. Compute both current and historical Data Quality Metrics. You want to know how things look right now, and you want to know the trend: are things getting better or worse?

Notice that the first two steps have nothing to do with technology. This is why firms that buy a data quality tool without establishing ownership and definitions tend to be disappointed. The tool can only test the rules that someone is accountable for writing.

The most common way these programs stall is by trying to write rules for every field on day one. Instead, begin with your Critical Data Elements, the data that drives NAV, client reporting, regulatory filings, and investment decisions. Prove the five steps work there, then widen the circle.

What Does “High Quality” Even Mean?

Some people think that data is either good or bad. In reality, it is far more nuanced, which is why we like to introduce the concept of Quality Dimensions. Data can be poor quality for a number of different reasons:

• Accuracy: The degree to which data correctly represents real-life entities. Does the maturity date on the bond match the prospectus?
• Completeness: Whether all required data is present. Does every security have an asset class and a country of risk?
• Consistency: Whether data values are consistently represented within a data set and between data sets. Is a client classified the same way in the CRM as in the accounting system?
• Integrity: Internal consistency within a data set, such that there are no holes or missing parts. Does every position reference a portfolio and a security that actually exist?
• Reasonableness: Whether a data pattern meets expectations. Did a price move 40% overnight? Did the custodian file arrive with half the usual number of records?
• Timeliness: The degree to which data represents reality from the required point in time. Are yesterday’s closing prices loaded before the morning reports run?
• Uniqueness: No entity exists more than once within the data set. Is the same client in the CRM three times under slightly different names?
• Validity: Whether data values conform to a defined domain of values, inclusive of data type, format, and precision. Is that currency code a real ISO code?

This is not an exhaustive list of the ways data can be wrong. It helps you articulate what your instincts are already telling you about the data. It also makes Step 2 much easier: a Subject Matter Expert who walks through each dimension for their domain will come out the other side with a solid first draft of their quality rules.

How to Enforce Data Quality

There is rarely one thing that enforces data quality. It is usually a layered approach.

1. Data Modeling & Constraints. How you model your data matters, because a well-designed model makes certain errors impossible. Relational data models, for example, use tables, columns, data types, keys, and relationships to describe what valid data looks like. Storing a date in a true date field means nobody can enter a value that isn’t a date. Marking a field as required means nobody can leave it blank. Quality that is built into the structure is the cheapest kind, because nobody has to monitor it.
2. Data Validation at Capture or Ingestion. Validate data at the point where it enters your environment. When data is captured from users, that means validating each entry in the application before it is saved. When data is ingested from third parties, such as custodian, administrator, or vendor files, that means enforcing critical quality checks before the data is loaded, to prevent pollution and downstream operational risk.
3. Business Rule Enforcement. Firms end up loading data from different systems into something like a Data Warehouse, and inevitably need to reconcile across them. For example, do we have positions today for all active portfolios? These business rules need to be declared and tested, and there are now Data Quality platforms you can buy to solve this problem. Where possible, you want to empower Data Stewards to manage the rules themselves, without requiring IT intervention.

The technical team tends to enforce Layers 1 and 2 proactively. The business teams usually enforce Layer 3 reactively.

Making the Layers Work Together

Prevention is cheaper than detection, so the guiding principle is to push enforcement down to the lowest layer that can handle it. A bad value that is rejected by a constraint or stopped at ingestion never pollutes the Enterprise Data Warehouse, and nobody downstream has to wonder whether they can trust it. Some rules, like cross-system reconciliation, can only be tested once the data comes together, which is exactly why the third layer exists. But it should be the safety net, not the only line of defense.

The layers should also learn from one another, and this is where continuous improvement comes in. Every data issue is a chance to ask two questions: could a rule have caught this, and could a lower layer have prevented it? An issue that slipped through becomes a new or updated Data Quality Rule, so that the same problem is detected next time. An exception that keeps resurfacing at Layer 3 becomes a candidate to move down to Layer 1 or 2, so that it is prevented rather than repeatedly detected. A mature program gets stronger with every issue it resolves.

The same thinking applies to resolution: fix data at the source. A Data Steward who patches a value in the warehouse has solved the problem for one day and one consumer. The bad value returns with the next load, and every other system fed by that source still has it.

Which Metrics Are Worth Tracking?

You cannot manage what you do not measure, and a Data Quality program is no exception. A handful of metrics, each answering a plain question, goes a long way:

• Count of Data Quality Rules: What is our quality posture? A high number of passing rules indicates high quality, and a domain with very few rules is a domain nobody is really checking.
• Count of New Exceptions: Where are the hot spots?
• Age of Exceptions: What is new, and what is lingering?
• Mean Time to Resolution: Are issues resolved in a timely manner?
• Ratio of Exceptions to Rules: Are there operational opportunities, such as resourcing or automation?

Two things make these metrics far more useful. The first is slicing them by dimensions such as Data Domain and Data Owner, so that you can see which domains and which teams are working well and which are not. The second is history. Today’s numbers, combined with data lineage, tell you the blast radius of a current problem: which reports, downstream systems, and clients it touches. The trend tells you whether a domain is getting healthier or quietly deteriorating.

An Enterprise Data Governance Committee would typically monitor these metrics and direct resources toward the areas that need help. That closes the loop back to Step 1, because the metrics are how accountability becomes visible.

Trust, but Verify

Here is what this looks like in practice. One firm had outsourced its middle office and assumed that appropriate checks and balances were in place. Once we rolled out our Data Quality Framework and wrote 300 rules, we discovered that the service provider was falling down. More importantly, the client now knew immediately where and when.

The lesson extends well beyond that one engagement. You can outsource a function, but you cannot outsource accountability for the data. Custodians, administrators, and data vendors all feed your decisions, and measuring the quality of what they deliver turns provider oversight from an assumption into evidence.

AI Raises the Stakes, and Lowers the Effort

AI changes the data quality conversation in both directions.

On one hand, it raises the stakes. An experienced analyst will often notice when a number looks wrong. An AI agent may not. It will take the value it is given and build confidently on top of it, at a speed and scale no team of analysts can match. “Garbage in, garbage out” has never applied faster.

On the other hand, AI can take much of the labor out of the five steps. It can profile a data set and propose candidate rules for Subject Matter Experts to review. It can spot anomalies that no one thought to write a rule for, which is the Reasonableness dimension at work. It can triage exceptions, suggest likely root causes, and help Data Stewards resolve issues sooner. What it cannot do is be accountable. AI can propose a rule, but a Data Owner still has to stand behind it.

Conclusion

Trusted data is not an accident. The areas of your firm where data is trusted are almost certainly the areas where someone is accountable, quality is defined, rules are measured, exceptions are resolved, and somebody is watching the trend. The areas where data isn’t trusted are missing one or more of those steps.

Olmstead has deep expertise helping investment managers establish Data Governance and put a Data Quality Framework into practice, so that their data can be trusted. Do you trust your data? If the answer is “it depends,” contact us and let’s talk about closing the gaps.