Airflow
Dynamic TaskGroup in Airflow 2.0
TaskGroup feature in Airflow 2.0 - Dynamic creationIn this article we will uncover a way to use Airflow new feature called TaskGroup which allow you to manage your dependencies in a dynamic way. Many articles are showing you how to use them in a…
Dynamic TaskGroup Scalability in Airflow 2.0
Dynamic TaskGroup Scalability in Airflow 2.0 - Handle big DAGsIn the previous article I showed you how to instantiate TaskGroup in a Dynamic way. We will now see how we face the challenge of using it at a larger scale. This is not an easy task : you…
Apache Airflow 2.0 : How it works
Apache Airflow 2.0 & how it works : Scheduling and beyondAirflow became in the last recent years a major actor for scheduling a wide variety of actions. From running basic Python script to API calls up to acting as an ETL tools, it can address…