Robert Yi
Feb 15, 2022

Orchestration will likely need to be part of any data stack as it grows in maturity, but IMO you don't need to worry about it for a long time unless there's a very good use case for it - e.g. if ML is core to your business, if you absolutely need custom data pipelines, etc. dbt alone should replace core orchestration use cases for a long time.

But agreed that this is only half the picture. There are components to this I've purposely left out for now: orchestration, but also the metrics layer (too early to see how this'll develop, and dbt may solve this as well), monitoring/observability/CI for BI, the whole world of ML deployment/monitoring/retraining tools. But to me these are auxiliary value adds to the core immutable components that you need early on.

Robert Yi
Robert Yi

Written by Robert Yi

Chief Product Officer, Hyperquery (hyperquery.ai). Former ds @ Airbnb, Wayfair; Ph.D. @ MIT, physics @ Harvard. twitter.com/imrobertyi Also at think.ryi.me

No responses yet