technological solutions for a world of data
Spider8M
Spider8M - the eight legs of metadata
You request a download of a bank statement from your financial institution; it's a dataset for a selected point in time. You find a discrepancy on the statement. You phone your financial institution and request an explanation. Your bank is unable to trace the issue because it did not have the decoder of data (data about the data) of the transaction. This is metadata; the decoder of data and we rely on it to validate data. Any data asset should encompass a proven metadata solution. Otherwise, who should trust those numbers if they cannot be proven to be accurate?
One could wish metadata wasn't so useful. Take the NSA and their phone sniffing habits. They know the when/where/who of our phone calls. It is metadata that proves a case, verifies a transaction and affords us a trail of technological bread crumbs to root causes of issues.
More about the data…..
Metadata is in lockstep with coming of age of data in the technology industry. Open and available data is bountiful with no end in sight. Volumes of data are exchanged once unprecedented as advancements in the hardware & plumbing of technology has made this possible (memory is cheap and disk storage is the new competitive market). This ushers in the need to facilitate an efficient dissemination of data content activity. Until now, the multitude of technology stacks and data exchanges made one metadata management solution that could harvest data activity across your technology landscape, nearly impossible.
Introducing Spider8M
A metadata management solution was needed to sit above a technology stack, be data agnostic, flexibly harvest data activity and be customizable to serve industry specifics. Spider8M is system and data store agnostic metadata collection utility. With its own database, it systematically crawls through systems data stores and job schedulers to collect data activity. It is being developed with open source tools under the Creative Commons aid and credit to: Open Archives Initiative Protocol for Metadata Harvesting, and the books, “Provenance and Annotation of Data” and “Handbook of Metadata, Semantics and Ontologies”, And lastly, The Dublin Core Metadata Initiative (DCMI).