We’re living in the golden age of data. But even at high volumes, many companies struggle with being data rich and insights poor. One approach to augmenting enterprise data is to buy it — a.k.a. data acquisition.
Data acquisition involves the set of activities that are required to qualify and obtain external data — and also data that may be available elsewhere in an organization — and then to arrange for it to be brought into or accessed by the company. This strategy is on the rise as organizations leverage this data to get access to prospects, learn information about customers they already work with, be more competitive, develop new products and more.
Data Acquisition’s Inherent Problems
Standardizing data acquisition can be an afterthought at the tail-end of a costly data journey — if it’s considered at all. Now we see companies starting to pay more attention to the finer, critical points of data acquisition as the need for more data grows. And this is a good thing because ignoring acquisition best practices and proper oversight will lead to a whole host of problems that can outweigh the benefits of bringing in new data.
The costly, problematic issues we see organizations grapple with include:
- Data purchases that, once brought into the organization, don’t meet the intended business needs.
- Data consumers don’t trust the acquired dataset.
- Acquired data duplicates what the company already owns (i.e., data that’s “hidden” in another line of business).
- No way to capture and manage the associated rich metadata, such as having an enterprise data catalog in place.
- Ineffective or nonexistent governance practices that aren’t set up to manage the data acquisition process.
Despite data acquisition’s inherent problems, when the process is thoughtfully developed and properly managed, it can facilitate a data purchase in a matter of weeks — not months, as it often takes some companies — that bypasses problems and delivers outcomes that meet or exceed business expectations.
Companies are starting to pay more attention to the finer, critical points of data acquisition as the need for more data grows.
Data Acquisition Best Practices
First San Francisco Partners’ client engagements that focus on data acquisition standardization and governance are sharply on the rise. We believe certain foundational aspects should be in place to maximize value, optimize cost, reduce risk, ensure business needs are met and allow the purchase to be more broadly used by the organization (when appropriate).
These foundational aspects require the organization to:
- Standardize and centralize the data request funnel.
- Partner more effectively with vendor management.
- Set clear expectations with suppliers.
- Translate data needs into formal business cases.
- Leverage data governance and a data catalog to manage the process and capture metadata of all kinds.
With these beliefs in mind, here are a few best practices for improving the data acquisition process at your organization.
Establish a Request Process
Your data consumers will benefit greatly from a standardized method for submitting data requests. For example, use a data acquisition inbox to direct the requests to a data owner or delegate steward who knows the data subject area or domain the best. This individual will help determine if the request has enough information to move forward — and, if so, confirms the data isn’t available in the organization before the data acquisition process kicks off.
Also, an established data acquisition process guides people through a series of steps that helps them develop a strong and compelling business case that supports a legitimate purchase.
Data consumers benefit greatly from a standardized method for submitting data requests.
Leverage Vendor Management and Build Supplier Relationships
In many organizations, the vendor management area is often understaffed, underfunded, underutilized or ill-equipped to effectively answer certain questions, such as:
- What relationships does the company already have with data suppliers?
- Have those relationships served a purpose or failed overtime?
- Is there an opportunity to improve the partnership with a supplier?
- Across the organization, what datasets are already owned?
- What are the data agreements and associated rules?
Vendor managers should understand the data acquisition request and advocate for the business. They should share the goal of building a strong business case for the purchase before going into negotiations and contracting, which helps to minimize risk and the cost.
It’s not only important to know what data supplier relationships your company already has, but it’s also critical to understand their full capabilities — how they share information, their quality standards, is the data mastered correctly, and the privacy guardrails in place (e.g., anonymized and de-identified data).
A standardized data acquisition approach can surface questions, an understanding of whether more data is needed or if there are quality concerns to take back to the supplier. This back-and-forth relationship shows the supplier you know your requirements and opens up the opportunity to test your relationship with them, including their responsiveness, consultative skills, breadth of offering and overall fit for your needs.
Formalize the Business Case
Translating business requirements into data requirements is your first best step toward building a strong business case and kicking off successful data acquisition. A business case typically includes request details, options, cost-benefit analysis, personnel needs, timeline and risks. The process also includes understanding the architectural and technical implications of acquiring or accessing the data — which can create unforeseen costs.
Validating and vetting a data sample set against your requirements is another critical acquisition task — and one that should be completed before the purchase. This includes data analysis and profiling steps that ensure you understand the data you’re buying.
Translating business requirements into data requirements is your first best step toward building a strong business case and kicking off successful data acquisition.
Govern the Acquisition Process and Kick-Start Your Catalog
(That is if you don’t already have a catalog.)
If your data governance area is new or not well-established, data acquisition is an excellent use case to formalize it. As I mentioned, the business need to acquire additional data will surely increase. As the volume increases (and the associated risks and opportunities increase, too), governance becomes an integral part of your data acquisition best practices, as well as being able to advocate for a data governance tool or catalog.
Before submitting a data acquisition request, a business user will typically try to identify if the data is already available. In large or particularly siloed organizations without a data catalog, it can be a challenge to know what data the company already has access to.
When a catalog is configured and managed correctly, data users can perform a simple search of the catalog to discover what’s available, how the data is defined, who knows the most about it, who is using it in the company and much more. The data catalog is the central repository that all data efforts stem from. It holds the inventory of data assets and is, essentially, a shopping tool people can use before they look outside the company for new data (sometimes called a “data marketplace”).
Once it’s determined additional data is needed, the data governance organization can ensure the acquisition framework is followed — from inquiry to ingestion and to help to minimize risk, increase fulfillment efficiencies, and ensure the data is in the best form and ecosystem to land and be used in an organization.
If your data governance area is new or not well-established, data acquisition is an excellent use case to formalize it.
A governed process means information about the purchased dataset (metadata) is gathered — and once the purchase is approved and data is integrated, the metadata is recorded in the catalog.
This typically includes key details from the data requirements, vendor agreement rules/controls, data characteristics, meaning, quality or profiling results, privacy, compliance and usage rules, aspects of the data model and source-to-target mapping.
Companies who don’t implement an enterprise data catalog attempt to store this information in assorted documentation libraries or in the brain of “that guy who knows so much about data.” This is a missed opportunity that will impact future data acquisition requests and negotiations.
In addition, many companies are starting to capture a vendor’s data catalog upfront and recording detailed findings as part of the data acquisition process, even if they decide not to go ahead with the purchase. There is a considerable amount of information which is valuable to long-term maintenance and understanding of the acquired data. This activity also helps inform what vendor data is available, what was already purchased, and where one may want to extend a dataset.
Rewards of a Data Acquisition Framework
Data acquisition is a multi-step process, and I’ve just touched on a few areas here. The number of activities can vary, depending on your organization and the data itself. And I haven’t even talked about all the acquisition model options, such as ingestion into a data lake, or access to the supplier’s analytical environment (i.e., data as a service), or how the acquisition process can apply to internal data as well.
Today, data acquisition is recognized as a significant front end of the data supply chain. Buying data is often a large financial commitment, and related problems — such as insufficient due diligence or acquiring data you already own — can be costly. Even acquiring “free” data can have a cost to ingest it and require oversight in order to adhere to usage requirements. And data acquisition can’t be successful without data governance monitoring the process. But when you rely on a standardized approach for data acquisition, this dependency can minimize financial loss and power operational efficiency.
The same benefits apply to an enterprise data catalog. This central repository helps you inventory your data (what you create and what you buy), understand the business need for data, be more informed about how best to use the data — and who to talk to with questions or concerns.
When data acquisition activities are relevant and right-sized for your business, there’s a measurable impact on the bottom line: Decreased costs. Reduced inefficiencies. Enhanced business agility. Digital transformation. External data that fits a business need.
In the future, we’ll share more on this topic and some real-world results we’ve seen when leveraging Collibra’s data catalog to solve data acquisition challenges.
If any aspect of what I shared resonates with you, let me ask: What’s holding you back from improving your data acquisition practices?
See my related article, A Framework for Data Acquisition with Collibra, for more on this topic.
