{"id":3454,"date":"2025-09-09T18:08:14","date_gmt":"2025-09-09T23:08:14","guid":{"rendered":"https:\/\/devsite.datacracy.co\/?p=3454"},"modified":"2025-09-10T08:55:17","modified_gmt":"2025-09-10T13:55:17","slug":"integrating-airflow-with-databricks","status":"publish","type":"post","link":"https:\/\/devsite.datacracy.co\/es\/integrating-airflow-with-databricks\/","title":{"rendered":"Integrating Airflow with Databricks"},"content":{"rendered":"\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

BLOG<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\n\t\t
\n\n\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t

\n\t\t\t\t\t\t\t\t\n\t\t\t\tIntegrating Airflow with Databricks\t\t\t\t\t- Overview of Airflow, Databricks, and dbt\t\t\t\t \n\t\t\t<\/h2 > \n\t\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t\n\t\t\t\t\t\t<\/span>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Before diving into the integration steps, let\u2019s quickly overview the technologies we\u2019ll be using.<\/p>

  • \u00a0**Apache Airflow**: An open-source platform used to programmatically author, schedule, and monitor workflows.<\/li>
  • **Databricks**: A unified data analytics platform that provides a Spark-based processing engine, allowing you to
    build scalable data pipelines.<\/li>
  • \u00a0**dbt (data build tool)**: A tool that enables data analysts and engineers to transform data in a warehouse via
    SQL, with the added benefit of validating the data quality through testing and documentation.<\/li><\/ul>

    By combining these tools, you can orchestrate a full end-to-end data pipeline in which data is processed in Databricks, and dbt is used to validate the quality of that data.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t

    Setting Up Airflow<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
    \n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    Start by installing and setting up Apache Airflow. If you already have it running, you can skip this section.<\/p>

    • Install Airflow via pip: pip install apache-airflow.<\/li>
    • Initialize the database and start the webserver:
      airflow db init
      airflow webserver
      airflow scheduler
      Next, configure Airflow to work with Databricks.
      Airflow Configuration for Databricks:
      \uf0b7 Install the Databricks integration package:
      pip install apache-airflow-providers-databricks
      \uf0b7 Configure the Databricks connection in Airflow\u2019s UI:
      \uf0b7 Go to the Airflow Admin tab \u2192 Connections \u2192 Add a new connection.
      \uf0b7 Set **Conn Type** to Databricks.
      \uf0b7 Add your **Databricks host URL** and **token**.<\/li><\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
      \n\t\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t

      Running a Databricks Job from Airflow<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t
      \n\t\t\t
      <\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
      \n\t\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t
      \n\t\t\t\t\t\t\t\t\t

      To orchestrate a Databricks job in Airflow, you can use the DatabricksSubmitRunOperator. Here\u2019s a simple example of a
      DAG that triggers a Databricks job.<\/p>


      from airflow import DAG
      from airflow.providers.databricks.operators.databricks import DatabricksSubmitRunOperator
      from airflow.utils.dates import days_ago
      # Define Databricks job parameters
      DATABRICKS_TASK = {
      \u00a0\u00a0\u00a0 &#39;new_cluster&#39;: {
      \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 &#39;spark_version&#39;: &#39;7.3.x-scala2.12&#39;,
      \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 &#39;node_type_id&#39;: &#39;i3.xlarge&#39;,
      \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 &#39;num_workers&#39;: 2,
      \u00a0\u00a0\u00a0 },
      \u00a0\u00a0\u00a0 &#39;notebook_task&#39;: {
      \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 &#39;notebook_path&#39;: &#39;\/Shared\/your_notebook_path&#39;,<\/p>

      \u00a0\u00a0\u00a0 },
      }
      default_args = {
      \u00a0\u00a0\u00a0 &#39;owner&#39;: &#39;airflow&#39;,
      \u00a0\u00a0\u00a0 &#39;start_date&#39;: days_ago(1),
      }
      # Define the DAG
      with DAG(&#39;databricks_job&#39;,
      \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 schedule_interval=&#39;@daily&#39;,
      \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 default_args=default_args,
      \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 catchup=False) as dag:
      \u00a0\u00a0\u00a0 # Submit Databricks job
      \u00a0\u00a0\u00a0 run_databricks_job = DatabricksSubmitRunOperator(
      \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 task_id=&#39;run_databricks_job&#39;,
      \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 json=DATABRICKS_TASK,
      \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 databricks_conn_id=&#39;databricks_default&#39;
      \u00a0\u00a0\u00a0 )
      \u00a0\u00a0\u00a0 run_databricks_job<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

      Integrating Airflow with Databricks: Adding Data Quality with dbt<\/p>\n","protected":false},"author":1,"featured_media":1816,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"elementor_canvas","format":"standard","meta":{"_eb_attr":"","site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[32],"tags":[],"class_list":["post-3454","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-dato"],"_links":{"self":[{"href":"https:\/\/devsite.datacracy.co\/es\/wp-json\/wp\/v2\/posts\/3454","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devsite.datacracy.co\/es\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devsite.datacracy.co\/es\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devsite.datacracy.co\/es\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/devsite.datacracy.co\/es\/wp-json\/wp\/v2\/comments?post=3454"}],"version-history":[{"count":7,"href":"https:\/\/devsite.datacracy.co\/es\/wp-json\/wp\/v2\/posts\/3454\/revisions"}],"predecessor-version":[{"id":3465,"href":"https:\/\/devsite.datacracy.co\/es\/wp-json\/wp\/v2\/posts\/3454\/revisions\/3465"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devsite.datacracy.co\/es\/wp-json\/wp\/v2\/media\/1816"}],"wp:attachment":[{"href":"https:\/\/devsite.datacracy.co\/es\/wp-json\/wp\/v2\/media?parent=3454"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devsite.datacracy.co\/es\/wp-json\/wp\/v2\/categories?post=3454"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devsite.datacracy.co\/es\/wp-json\/wp\/v2\/tags?post=3454"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}