Friday: Operational excellence and best practices

Operational excellence and best practices for batch data vault automation

This talk will cover the following topics:

  • Business Layer Stabilization
  • How to insulate your customers from the back-end load process and provide 24/7 access to an active vault.
  • In one vault, our business layer supports web services and direct reads from downstream systems. In another vault, the business layer supports operational reporting.
  • We chose to implement an active/standby copy of the business layer for the data distribution hub. One copy is online while the other is off-line during the vault update.
  • This same architecture supports operational reporting use-cases and will also support a data mart analytics use-cases.
  • How did we do this?
  • Process Scheduling and Monitoring
  • How to integrate operational scheduling into your load process.
  • Whether you need to create your own delta feeds or run a re-load, fine grained run-time controls make it easy to manage the process.
  • Full or partial re-loads.
  • Performance monitoring and the importance of job logs
  • Exploiting pattern based ingestion with re-usable components
  • Finding the patterns – Identifying re-usable components and when enough is enough.
  • Embed intelligent operational process controls – how to re-load, re-start or re-run without worries
  • How to design for failures –
  • Table driven parameter based run-time controls

What you will learn:

  • How to insulate your customers from the back-end load process and provide 24/7 access to an active vault.
  • How to integrate operational scheduling into your load process.
  • How to identify re-usable components and when enough is enough.
  • How to embed intelligent operational controls in the load process.

Full Conference
Location: Pinnacle Room Date: May 24, 2019 Time: 8:00 am - 8:55 am Gregory Locke