This chapter provides an overview from DAMA-DMBOK2 on metadata management (excerpted from pages 417-424), and then covers the additional metadata management responsibilities needed for blockchain to work well within our organizations.

Overview from DAMA-DMBOK2

The most common definition of metadata, “data about data,” is misleadingly simple. The kind of information that can be classified as metadata is wide-ranging. Metadata includes information about technical and business processes, data rules and constraints, and logical and physical data structures. It describes the data itself (e.g., databases, data elements, data models), the concepts the data represents (e.g., business processes, application systems, software code, technology infrastructure), and the connections (relationships) between the data and concepts.

Metadata helps an organization understand its data, its systems, and its workflows. It enables data quality assessment, and is integral to the management of databases and other applications. It enhances an organization’s ability to process, maintain, integrate, secure, audit, and govern other data.

To understand metadata’s vital role in data management, imagine a large library, with hundreds of thousands of books and magazines, but no online catalog. Without a catalog, readers might not even know how to start looking for a specific book or even a specific topic. The catalog not only provides the necessary information (e.g., which books and materials the library owns, and where they are shelved), it also allows patrons to find materials using different starting points (e.g., subject area, author, or title). Without a catalog, finding a specific book would be difficult if not impossible. An organization without metadata is like a library without a card catalog.

Metadata is often categorized into three types: business, technical, and operational.

Business metadata focuses largely on the content and condition of the data and includes details related to data governance. Examples of business metadata include:

• Definitions and descriptions of data sets, tables, and columns
• Business rules, transformation rules, calculations, and derivations
• Data models
• Data quality rules and measurement results
• Schedules by which data is updated

Technical metadata provides information about the technical details of data, the systems that store data, and the processes that move it within and between systems. Examples of technical metadata include:

• Physical database table and column names
• Column properties
• Database object properties
• Access permissions
• Physical data models, including data table names, keys, and indexes

Operational metadata describes details of the processing and accessing of data. For example:

• Logs of job execution for batch programs
• History of extracts and results
• Schedule anomalies
• Results of audit, balance, control measurements
• Error Logs

Additional responsibilities due to blockchain

Experts in metadata management must work closely with experts from the other data management disciplines to identify necessary metadata for blockchain applications, and to ensure that this metadata gets captured in the blockchain ledger.

Extending metadata standards

Industry standards will become more important (in some cases, critical) for blockchain applications that cross organizational boundaries. In this book, we’ve mentioned a number of these standards, such as NIEM and BiTA.

Metadata experts will need to work closely with data governance experts to agree on standards (and modify standards where needed) for a particular organization. In addition, metadata experts will need to work with data architects, data modelers, and DBAs to ensure these standards are adhered to in the blockchain protocol and ledger.

In addition, there are meta-standards that underlie these industry standards, such as meta-standards for building smart contracts and achieving recordkeeper consensus. Metadata experts should be involved in open-source collaboratives that focus in this area, such as Hyperledger. Hyperledger is an open-source collaborative effort created to advance cross-industry blockchain technologies. It is a global collaboration, hosted by The Linux Foundation, including leaders in finance, banking, Internet of Things, supply chain, manufacturing, and technology.33

Requiring additional metadata

Metadata experts will need to capture additional types of metadata for blockchain applications. They’ll also need to raise the importance of certain existing types of metadata, as certain metadata is more relevant to blockchain than others. Here are just a few of the specific types of metadata that are most important in blockchain.

Business metadata:

• Business rules, specific around smart contracts
• Public keys
• Blockchain addresses

Technical metadata:

• Recordkeeper total count
• Recordkeeper majority needed count
• Hashing and crypto-currency algorithms

Operational metadata:

• Purge criteria, that is when to deactivate data in the blockchain ledger as data cannot be deleted
• SLA requirements, especially around performance
• Archiving rules in the absence of “no deletes”

Storing metadata using blockchain

In traditional relational databases, metadata defines the buckets where the data will be stored. An attribute called “Customer Last Name” is the metadata for the actual last name of the customer. However, with non-relational databases like blockchain, the ledger stores text without the requirement of attribute metadata. For example, if only the number “5” is stored in the blockchain ledger, we need metadata to tell us whether this is “5 pounds,” “5 dollars,” or “5 people.”

The metadata experts will need to work closely with data governance and DBAs to ensure proper metadata is stored in the blockchain ledger.