Working with clients to optimize their metadata management tool, which is often Collibra, I think I’ve seen it all — the good, the bad, the just OK. There are a few errors and pitfalls I see occurring time and time again, and I’d like to share some tips on how to avoid them.
Vision of a Metadata Management Tool
A metadata management tool has many different capabilities. You can document anything from a business term to a report to a database to physical data dictionaries. And there’s a lot you can capture within each of those individual components.
It’s easy to get caught up in the different metadata management tool capabilities and forget the overall goal or vision for why you’re using a metadata tool in the first place. It’s also easy to forget the tool is meant for many people to use in the organization, not just one team or department.
Typically, the vision of a metadata management tool is to make it organization-wide. While the tool might work great for one area, not catering to the masses means the initiative could fail.
Here is an example from an engagement I recently worked on: When we started the engagement, we learned one particular team was comprised of heavy users of the tool and they customized the metadata. It made sense to their team, but anyone else looking at the tool would find it hard to locate information. This was happening because they were using the tool to support their business processes and day-to-day work, but not in a way that supported the overall goal of the tool. Because of this structure, they made changes that made sense to them and didn’t focus on the metadata. The biggest issue in this scenario was it worked well for one group’s processes but was confusing for everyone else. If the tool was purchased just to support one particular team, then something like this example may work. But if it is meant to be an enterprise-wide tool, the content must resonate with the entire organization.
Capturing the Right Information
One essential piece to building the vision of a metadata management tool is to consider the end-user. The first step in capturing the correct information is determining who will use the information. For example, if you have 1,000 people using Collibra’s metadata management tool and want to customize or add metadata that applies to only five people, you have to evaluate if the metadata is creating clutter. Is it helpful to those five people but confusing to the other 995 people trying to access the tool?
Thinking about the end-user is one piece. The other is thinking about the tool as a whole. Organizations often assign different authors from different business areas in the tool. Because each author may have their idea of what they want to get out of the tool, having an overarching strategy will be key.
For example, two groups may want to capture information around the same business terms. One group may have one set of requirements for the metadata, and another may have a separate set of requirements. If both groups do things their way and don’t create a model that works for everyone, the result is a never-ending list of inconsistent metadata captured for the same asset types.
Similarly, I’ve seen scenarios where companies tried to capture too much metadata. In some cases, the tool has outdated information that users think is legitimate. From a usability standpoint, I’ve seen a client want to capture 85 different fields for a dataset. If a user wants to add a definition or an attribute and they see a list of too many options, it’s overwhelming. They don’t want to learn what all of these fields are. It creates a difficult experience both from an author’s perspective to add it and from an end-user’s perspective trying to weed out the pieces that apply to them.
Value of Use Cases
If you’re in the early stages of setting up a metadata management tool, start with one use case.
One common use case we see is business report lineage. It sounds simple enough to document reports, the fields within them and where the data is coming from. Oftentimes, it is quickly realized there is a lot of work involved in relating reports to their source and creating descriptions for the content. Then, once you expand this effort to other groups, there are commonly conflicting ideas of metrics or business concepts. You must overcome those challenges with good data governance policies and standards and ensure the right people are at the table.
When thinking about use cases, understand what you expect from the tool. You don’t want to boil the ocean — think incrementally. For this reason, we always suggest starting with a single simple use case to provide value with minimal effort and gradually expand to the next use case.
Starting Small and Thinking Scalability
Start with a small goal for the metadata management tool and make sure data changes will apply to other areas. Efforts within the tool should be repeatable. This can be done by starting as high level as you can and expanding from there once you start getting more users in the tool.
To ensure you’re capturing the correct metadata, consider the scalability of efforts across the tool. Determine who will add information and how often it will be updated. Decide on the upkeep of information and if it can be updated automatically.
But again, don’t try to boil the ocean. Thinking back to the report example I gave earlier, you can auto ingest reports and metadata from your data sources. Once users start to see the functionality, it’s easy to start ingesting a vast amount of metadata with no real game plan on what they will do with it. So, keeping the scope small to start (even if it is tempting to auto-ingest everything) helps to prove value incrementally. Then you can pull together a roadmap on which pieces to tackle next when you have the right people in place.
Here’s a side note about ingesting data: Something you hear from people with auto-ingested data, such as Tableau reports, is that it’s confusing for users. And that’s no surprise with auto-ingestion. People who want automation can still become frustrated when certain ingested metadata is garbage. (“Garbage in, garbage out,” as they say.)
If your metadata is bad or confusing to people, it reflects on the implementation of data — it’s not the fault of your metadata management tool. If you want clean, consistent metadata, get the policies and governance in place to ensure that happens. You can’t blame the metadata management tool for not presenting data in its ideal format if the originating data isn’t in an ideal format.
The People Side of the Tool
A metadata management system is only as effective as the support of the people behind the tool. If you don’t have people to maintain the information, it will fall apart. You want people to come into your metadata tool and find, understand and trust the data.
Make sure to have designated and repeatable resources for maintaining information in the tool before adding any content. For example, identify the role of the business steward. What are their responsibilities? How should they be adding and approving content? Whatever the strategy, it must work for all business stewards, not just one area.
Getting the Most from Your Metadata Management Tool
Implementing and getting the most out of a metadata management tool in your organization can seem daunting. Remember to start small, capture the right information and have use cases in mind and the right people to maintain it. Follow these tips to get the most value from your metadata management tool and avoid common mistakes.
