Business Vault creation using a Rules Engine

Using a Rules Engine to create a Business Vault form a raw Data Vault

This presentation will describe a project recently completed to use Data Vault and Business Vault principles to consolidate, integrate and transform engineering data and documents from 15 source systems into a single repository for large O&G company. The enriched data in the Business Vault was then used to perform a migration to a new engineering data and document management system.

This agile project had to perform source system analysis and data loading/integration within tight timeframes; loading the raw data from 15 source systems into a raw DV2.0 SQL Server database. Using a custom built, data-driven rules engine; the raw data was consolidated into a single Business Vault representation via the sequential application of business rules.

As the processes needed to be in place for a year for business reasons – business rule application was triggered by data change in the Raw Data Vault or Rule change in the rules engine. To be placed in the Business Vault, raw data needed to be filtered, validated and potentially transformed or augmented via the business rules; in a repeatable, systematic way.

Application of business rules needed to be auditable, and the entire Business Vault needed to be able to be regenerated from raw data at any time. If rules did not change, ‘enriched’ data in the Business Vault would not change. All aspects of the Data Vault and Business Vault; as well as ETL procedures were generated from meta-data. This allowed for extreme flexibility as new source data/rules were encountered. The entire loading process and rules engine execution was repeatable, auto-generated and pattern-based.

  • An introduction to the concept of a business vault, in context of data consolidation, integration and standardization
  • Discussion of rules engine alternatives (Microsoft/Drools/Custom) and final design principles
  • Guiding principles of rules engine/processing for maximum flexibility and productivity using meta-data generation
  • Techniques for raw data vault loading using Change Data Capture as well as full table loads
  • Lessons learned and benefits of this approach over previous attempts to accomplish the same thing using traditional methods

Full Conference
Location: Great Hall Date: May 18, 2017 Time: 2:05 pm - 3:05 pm Bruce McCartney