I recently shared my thoughts on data acquisition best practices, the value of an enterprise data catalog and data governance’s connection to the acquisition process. Building off of that foundation, I’d like to focus now on some specific recommendations for developing a standardized acquisition framework and the role Collibra can play in end-to-end data acquisition process management.
When First San Francisco Partners (FSFP) clients focus on building out data capabilities such as analytical insights, governance, privacy and quality, they often neglect the importance of data acquisition. This is when we challenge them to step back and evaluate the source of their data, whether it is purchased (fee-based) or just downloaded (free).
The majority of companies rely on external data to enhance day-to-day business operations, such as adding demographic data to the marketing database to improve segmentation. For these organizations, implementing a robust data acquisition process sets the foundation to govern their data beyond ingestion. In many cases, our clients also focus on tackling master data management or advanced analytics, while struggling to bring order and meaning to the data within.
With our recommended data acquisition framework, we’re able to shine the light on what’s really going on with our customer’s data and where they can quickly pivot to leverage the data acquisition process to increase data understanding, as well as better govern the data to reduce duplicates and ease integration. All this leads to improved productivity and reduced costs.
Critical Role of a Data Catalog
One of those quick pivots is capturing all sorts of valuable metadata upfront when assessing a new vendor or data source, including:
- Who is the vendor and have we already worked with them?
- What contractual or compliance agreements are associated with the data?
- What are the quality measures?
- What does the data mean and what is the format?
- What business area could it help? Can the enterprise access it or does it need restrictions?
- Do we already have similar data?
- What data domain does it belong to, and who can I go to with questions?
- Who are the people accountable in making acquisition decisions?
- What specific projects or strategic efforts does the data support?
- <Fill in the blank.> I’ll leave this bullet up to you to come up with at least one more. Think of your past struggles trying to find data to meet your needs. What would you want to know upfront?
And here is a trick we started working on with recent clients who already have Collibra Data Catalog: Capture the information identified above in the catalog, regardless of whether you decide to move forward with the data purchase (or download). This informs the business case for active and future purchase decisions and prevents another part of the organization duplicating the effort. Also, focusing on metadata naturally helps prepare for ingestion when moving forward with acquiring the data.
MDM’s Role in the Acquisition Process
A not-so-obvious data capability which also plays a role in data acquisition is master data management (MDM). When an organization acquires data, it may share a customer master data set with the data vendor. If they have a master data process, it is easier to match and append the additional purchased data to the existing customer record to meet an organization’s research, analytical or marketing needs. This enables you to truly integrate the data and get more value from that acquired data.
Part of building a trustworthy relationship with a data vendor is understanding its match rules and “matchability” ratings. When this part of data acquisition isn’t governed effectively, both initially as well as ongoing, an organization’s decision to purchase the data may introduce significant quality issues and risk.
Once receiving that appended information, the rules for viewing and storage must be addressed because it may be different than your existing data. Consider if it can be associated with an operational MDM system or de-identified and only accessible by certain roles (e.g., analyst or data scientist) or in certain organizations (e.g. finance or human resources).
In FSFP’s data acquisition framework, MDM-focused activity prompts the development of strong data-sharing and usage agreements and KPIs, such as a 95% match-accuracy rate, which can be written into contracts and governed within Collibra, further setting expectations the vendor relationship is a two-way street — and further ensuring you are getting value from your data acquisition.
Best Practices, Revisited
In a previous article, The Ins and Outs of Data Acquisition: Beliefs and Best Practices, I covered best practices that address some of the inherent problems of acquisition.
My recommendations included:
- Establish a request process
- Leverage vendor management and build supplier relationships
- Formalize the business case
- Govern the acquisition process and kick-start your catalog if you don’t have one already
Another best practice is to make sure the correct people are identified and trained to support the data acquisition process and that your data governance area also oversees the process.
Data acquisition is cross-functional and should involve many people when done well, including:
- Business stakeholders to identify a requirement for new data
- Data owners or stewards to determine if data already exists within the company and whether the new source is fit for purpose
- Vendor management to facilitate the assessment of vendor/data, as well as the acquisition process
- Privacy/legal/compliance/security/IT to chime in on their comfort with moving forward based on internal and external standards and regulations
As you likely already know, the importance of having enterprise and data architecture in the initial discussions helps to inform the best outcome.
Collibra Data Intelligence Solution for Data Acquisition
When our clients share their experience with the complexities of data acquisition, they do so mostly in an abstract manner. Once we demonstrate how Collibra can be used to knit all the pieces together, there’s often an “aha moment.” This is when clients see the data catalog in a new light —through the lens of data acquisition — and more importantly, they start to realize the metadata captured upfront jumpstarts other use cases within the organization (e.g. MDM, quality, analytical warehouses).
FSFP has used Collibra to make the case for why a robust data acquisition framework paves the way for data intelligence for years to come. Its tool functionality also allows us to deliver on FSFP’s mission statement: Information is your business. Making it actionable is ours.
Acquisition Framework’s Additional Benefits
Risk and privacy is another focus area for most of our clients. It is rare for a client’s privacy officer to skip out on the opportunity to weigh in on the decision to purchase (or download) and ingest external data — that is, if they know about it.
One of the wins we see with clients is that our data acquisition framework brings forth key questions and considerations, which allows for critical discussions and transparency early on in the process.
We’ve always recommended clients standardize their data acquisition processes, but now there’s a renewed urgency in classifying data correctly, tracking lineage and meeting privacy regulations. Establishing a data acquisition framework addresses all three.
Technologies such as Collibra enhance and support data-related activities and privacy, but we know the people and process side of acquiring data is far from going away. Just imagine getting data classification, data lineage and data privacy all working together, and you have a dependable data catalog. Your organization is trained up on data acquisition and governance, so what used to be a confusing process is now documented and supported consistently. When searching in Collibra, but coming up short, instead of being frustrated, you have exactly what you need to reach out to someone with a question or submit a new acquisition request.
Acquisition’s Path Forward
A repeatable, transparent, collaborative data acquisition process can reduce the time to onboard and ingest a new data set from 12 months to less than two, and ensure you are spending your money wisely. When a data acquisition framework is coupled with a data intelligence tool, such as Collibra’s, it’s a powerful combination. The pair brings order to an organization’s often chaotic data environment.
If the idea of a data acquisition framework has appeal to you, let us know. We can talk through your acquisition challenges and share success stories from clients who renewed their focus on data acquisition, including those new to Collibra or those who already use it.
