This presentation will describe a project recently completed to use Data Vault and Business Vault principles to consolidate, integrate and transform engineering data and documents from 15 source systems into a single repository for large O&G company. The enriched data in the Business Vault was then used to perform a migration to a new engineering data and document management system.
This agile project had to perform source system analysis and data loading/integration within tight timeframes; loading the raw data from 15 source systems into a raw DV2.0 SQL Server database. Using a custom built, data-driven rules engine; the raw data was consolidated into a single Business Vault representation via the sequential application of business rules.
As the processes needed to be in place for a year for business reasons – business rule application was triggered by data change in the Raw Data Vault or Rule change in the rules engine. To be placed in the Business Vault, raw data needed to be filtered, validated and potentially transformed or augmented via the business rules; in a repeatable, systematic way.
Application of business rules needed to be auditable, and the entire Business Vault needed to be able to be regenerated from raw data at any time. If rules did not change, ‘enriched’ data in the Business Vault would not change. All aspects of the Data Vault and Business Vault; as well as ETL procedures were generated from meta-data. This allowed for extreme flexibility as new source data/rules were encountered. The entire loading process and rules engine execution was repeatable, auto-generated and pattern-based.