![]() When you're learning about task execution, you'll want to be familiar with these somewhat confusing terms, all of which are called "environment variables." The terms themselves have changed a bit over Airflow versions, but this list is compatible with 1.10+.Įnvironment variables: A set of configurable values that allow you to dynamically fine tune your Airflow deployment. The difference between Executors comes down to the resources they have at hand and how they choose to utilize those resources to distribute work (or not distribute it at all). The executor works closely with the scheduler to determine what resources will actually complete those tasks (using a worker process or otherwise) as they're queued. The scheduler reads from the metadata database to check on the status of each task and decide what needs to get done and when. The Metadata Database keeps a record of all tasks within a DAG and their corresponding status ( queued, scheduled, running, success, failed, and so on) behind the scenes. See Introduction to Apache Airflow.Īfter a DAG is defined, the following needs to happen in order for the tasks within that DAG to execute and be completed: To get the most out of this guide, you should have an understanding of: Understand the purpose of the three most popular Executors: Local, Celery, and Kubernetes.Contextualize Executors with general Airflow fundamentals.Understand the core function of an executor.Even if you're a veteran user overseeing 20 or more DAGs, knowing what Executor best suits your use case at any given time isn't always easy - especially as the OSS project (and its utilities) continues to grow and develop. If you're new to Apache Airflow, the world of Executors is difficult to navigate.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |