Scheduling

Dynamic TaskGroup in Airflow 2.0

TaskGroup feature in Airflow 2.0 - Dynamic creationIn this article we will uncover a way to use Airflow new feature called TaskGroup which allow you to manage your dependencies in a dynamic way. Many articles are showing you how to use them in a…


Dynamic TaskGroup Scalability in Airflow 2.0

Dynamic TaskGroup Scalability in Airflow 2.0 - Handle big DAGsIn the previous article I showed you how to instantiate TaskGroup in a Dynamic way. We will now see how we face the challenge of using it at a larger scale. This is not an easy task : you…


Apache Airflow 2.0 : How it works

Apache Airflow 2.0 & how it works : Scheduling and beyondAirflow became in the last recent years a major actor for scheduling a wide variety of actions. From running basic Python script to API calls up to acting as an ETL tools, it can address…