[Apache Airflow] Concurrency 설정

Concurrency 설정

• parallelism = 32/ AIRFLOW__CORE__PARALELISM
  • defines the maximum number of task instances that can run in Airflow per scheduler
  • by default, execution up to 32 tasks are possible at the same time.
  • if there are 2 schedulers: 64 tasks(2 x 32) are possible at the same time
  • the value here depends on the resources and the number of schedulers running.
• max_active_tasks_per_dag = 16 / AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG
  • defines the maximum number of task instances allowed to run concurrently in each DAG
  • By default, execution up to 16 tasks at the same time for a given DAG across all DAG Runs is possible. <!– * max_active_runs_per_dag = 16 / AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG
  • defines the maximum number of active DAG runs per DAG
  • by default, execution up to 16 DAG runs per DAG running at the same time is possible. –>
• worker_concurrency = 16
  • defines the number of task instances that a worker will take
  • size up your workers based on the resources on your worker box and the nature of the tasks