If you're running Airflow on Astronomer, you should modify these parameters with Astronomer environment variables. For more information, see Setting Configuration Options in the Apache Airflow documentation. To check current values for an existing Airflow environment, go to Admin > Configurations in the Airflow UI. Generally, all default values can be found in the Airflow Configuration Reference. They all have default values that can be overridden by setting the appropriate environment variable or modifying your airflow.cfg file. Environment-level settings Įnvironment-level settings are those that impact your entire Airflow environment (all DAGs).
Knowing the requirements of your use case before scaling Airflow will help you choose which parameters to modify. Airflow admins or DevOps engineers might tune scaling parameters at the environment level to ensure that their supporting infrastructure isn't overstressed, while DAG authors might tune scaling parameters at the DAG or task level to ensure that their pipelines don't overwhelm external systems. The reason Airflow allows so many adjustments is that, as an agnostic orchestrator, Airflow is used for a wide variety of use cases. Tuning these settings can impact DAG parsing and task scheduling performance, parallelism in your Airflow environment, and more. See Airflow executors explained.Īirflow has many parameters that impact its performance. To get the most out of this guide, you should have an understanding of: If you're using an earlier version of Airflow, some of the parameter names might be different. This guide references the parameters available in Airflow version 2.0 and later. you'll also learn how your choice of executor can impact scaling and how best to respond to common scaling issues. In this guide, you'll learn about the key parameters that you can use to modify Airflow performance. To make the most of Airflow, there are a few key settings that you should consider modifying as you scale up your data pipelines.Īirflow exposes a number of parameters that are closely related to DAG and task-level performance.
One of the biggest strengths of Apache Airflow is its ability to scale to meet the changing demands of your organization.
0 kommentar(er)
