Newsletters
Discover our implementation experience, where we improved a dbt workflow deployed on AWS MWAA.
While working with our client’s data warehouse, we noticed their dbt orchestration relied heavily on BashOperators inside Airflow. Over time, this setup introduced several issues: differences between local and production execution, inconsistent dependency handling and limited visibility when individual dbt models failed. The more the warehouse grew in volume, the more these orchestration gaps surfaced.
Introducing Cosmos gave us the opportunity to move away from Bash-based orchestration entirely. Cosmos allowed us to structure the pipeline using a clean Medallion-style flow built on DbtTaskGroups.
The Silver layer used tag-based selection, isolated resource pools and early data quality checks. The Gold layer is built on top, running aggregations with its own execution logic and batch testing. Cosmos managed dependency resolution, model grouping, retries and SLAs directly inside Airflow, without custom glue code.
For deployment, we configured dbt, Cosmos and MWAA using pinned versions, MWAA constraint files and Airflow Connections for credential management. Cosmos automatically detected the MWAA environment and generated a compatible dbt profile, removing all path and credential inconsistencies that previously required manual adjustments.
This transition resulted in a more predictable orchestration environment, consistent behavior across local and production stages, and a noticeable reduction in the maintenance overhead that comes with Bash-based orchestration.
MWAA is strict about paths and versions. The most reliable fixes involved version pinning, MWAA constraint files and Cosmos’s built-in path handling.
These issues were consistent and repeatable and Cosmos provided clear mechanisms for resolving them without additional code.
Turning on emit_datasets=True allowed each dbt model to register as an Airflow Dataset. This gave us cross-DAG visibility and made lineage tracking straightforward, especially when Silver and Gold models interacted with other workflows inside MWAA.
Cosmos supports dbt-style selection patterns across Airflow tasks. This works for tags, specific models or directory-level patterns.
Examples we used frequently:
select=["tag:silver"]
select=["model:customers"]Cosmos automatically resolved upstream and downstream dependencies inside the Airflow DAG.
Running Silver and Gold in separate resource pools prevents worker starvation during high-load windows. This was especially helpful during early morning ingestion windows for the client.
Clean configuration and MWAA consistency rules made the biggest difference.
If you are interested in following more of our hands-on dbt work, whether it’s learnings from client projects, architectural patterns we test or practical discoveries around Airflow, MWAA and the modern data stack, you can subscribe to stay connected. We’ll keep sharing the insights that prove useful in real implementations, not just theory.
Thanks for reading and see you in the next update.
We turn complex data into clear insights. Our comprehensive data analytics services, from strategy to execution, drive business growth and innovation.
.svg)
