For instance, if your customized pod template file name is new_pod_template. To configure a pod template for a specific task:Īdd a command to your Dockerfile that copies your customized pod template into your Docker image. In cases like this, you can deploy a pod template to a single task within a DAG. For instance, one task might require significantly more GPU than another task. Some tasks require a more specific pod configuration than other tasks. You should see the updates you made to the pod template in this specification.Ĭonfigure the Kubernetes executor for a specific task To confirm that the deploy was successful, launch the Airflow UI for your Deployment, click into any single task, and click K8s Pod Spec. In your terminal, run astro deploy -f to deploy your code and rebuild your Docker image. In this example, the file path would be /tmp/copied_pod_template.yaml. Its value should be the directory path for the pod template in your Docker image. In the Software UI, add the AIRFLOW_KUBERNETES_POD_TEMPLATE_FILE environment variable to your Deployment. Note: Depending on your configuration, you may also need to change your USER line to root in order to have the appropriate copy permissions. You'll specify this file in an Environment Variable in Step 2. This command uses new_pod_template.yaml to create copied_pod_template.yaml at build time as part of your Docker image. Run the following command to find the namespace (release name) of your Airflow Deployment:ĬOPY new_pod_template.yaml /tmp/copied_pod_template.yaml To configure a Deployment's Kubernetes executor, you need to modify the Deployment's pod template and reapply a custom template via environment variables. Configure the Kubernetes executor using pod templatesīy default, Airflow Deployments on Astronomer use a pod template to construct each pod. If you're unable to resolve your issue after reviewing the documentation, you can post a question on the Astronomer web forum or you can contact Astronomer support. To learn more about different executor types, see Airflow Executors Explained. To resolve an issue, Astronomer recommends reviewing the Astronomer documentation first. Astronomer Enterprise Edition customers get full access to all source code for the platform, and can therefore modify and extend it to suit their needs while keeping all data inside their own virtual private cloud (VPC). For more information on configuring an executor, see Configure a Deployment. Enterprise Edition provides the same feature-rich access to Airflow, but through a customer’s own Kubernetes cluster. You must have an Airflow Deployment on Astronomer running with the Kubernetes executor. For more information on configuring Pod template values, reference the Kubernetes documentation. To configure these resources for each Pod, you configure a Pod template. It also means you can configure the following for each individual Airflow task: This enables the executor to scale depending on how many Airflow tasks you're running at a given time. For each task that needs to run, the executor talks to the Kubernetes API to dynamically launch Pods which terminate when that task is completed. The Apache Airflow Kubernetes Executor relies on a fixed single Pod that dynamically delegates work and resources. Test_alerts_three frozen-ionization-5364 v0.7.Run and configure the Kubernetes executor on Astronomer. Paolas-MacBook-Pro:astro-trial paola$ astro deployment list Deployment can be accessed at the following URLs Here’s the output you can expect: Paolas-MacBook-Pro:astro-trial paola$ astro deployment create PAOLA Deployment ID (automatically generated)ĭeployment Name ( lunar-meteoroid-6200) and Deployment ID cannot be changed, but Name can be.Deployment Name (formerly Release Name - automatically generated).Name (Customizable in your deployment’s Configure page or in the initial CLI command). Ah yes, if you haven’t created a deployment yet you can create it via the CLI with that command and personalize the name at the onset.įor those following, a deployment has 3 attributes:
0 Comments
Leave a Reply. |