Whether you batch, stream, melt or pivot, it all boils down to structured thinking and elbow grease.

All data ecosystems are built on frameworks and design patterns sharing the same fundamental principles: extract data, transform and store the final result for downstream analysis. Later on, this data will be consumed by business stakeholders through a visual or storytelling mechanism.

Another way to think about these tasks is to acquire data in its rawest form then cleanse, homogenise, curate, analyse and depict.

The challenge lies in doing all of this effectively and at scale. We want high availability, data freshness, correctness, data security and cost efficiency. One approach is the medallion design pattern, which structures data into guidelines of bronze, silver and gold zones, each with its own purpose. With proper orchestration (e.g a DAG), CI and CD workflows we can develop robust, fault tolerant and consistent solutions.

You can explore an implementation of the medallion design pattern at my GitHub repository