Friday: Zero to Data Vault in a Day

A business lead and DevOps approach to Data Vault: Automate the creation and release of your data model with your own code

Conference Theme: How-To Build

Abstract

The Data Vault system can be used to create an agile data solution, but how do you engineer and deploy a raw vault with agility? Our talk explores the challenges faced in technology and internal processes at a large, 300 year old retail bank whilst implementing Data Vault 2.0 and how the successful implementation of a business-lead, well-engineered, automation framework that compliments the Data Vault system for business intelligence, overcame them.

Choosing technology within a strongly governed organisation is a minefield; picking the right tools for the job vs what’s available can make or break a project – we’re here to share the secret sauce on how to develop an abstraction layer framework upon popular infrastructure as code tooling which generates the raw vault through business descriptions and can be deployed automatically through continuous integration pipelines. There is no vendor lock-in, the business owns the code and the engineers can evolve and extend the system to any future requirements to support the raw and business vaults. A new source can be ingested, combined in the raw vault and released within minutes.

We’ll show you how to build, run and automate generating a raw vault in Snowflake, orchestrate automated deployments in production and how it fits in an event driven cloud data platform.

Agenda

  • Introductions
  • Our story: problem statement
  • Building Data Vault in Snowflake through code
  • Building an abstraction layer to generate a raw vault
  • Practical demonstration
  • Putting it into production
  • Q&A

Learning Objectives

  • How to build a raw vault model in a 300 year old retail bank
  • First attempts at automating the creation of a raw vault model in Snowflake
  • A business first, DevOps approach to Data Vault
  • Implementing Data Vault on Snowflake using dbt and dbtvault
  • Building an abstraction framework on dbtvault to automate Data Vault creation
  • Extending the framework to create source tables, pipelines and data quality testing
  • Orchestrating the automation of your end to end model
  • Building an event driven platform with AWS & Snowflake

Pre-Recorded
Location: On-line Date: May 27, 2022 Time: 5:01 pm - 6:01 pm Kiran Bhat Adam Dewberry