Airflow

Installing Apache Airflow on Windows (Subsystem for Linux)

by Peter De Lange

I am working my way through Data Pipelines with Apache Airflow, Second Edition: Orchestration for data and AI and there is a point early in the book where it is explained how to run airflow locally in a python environment. This turns out not to work on a windows machine as some dependencies for Airflow are not available. Instead what you can do is run Ubuntu on the Windows Subsystem for Linux (WSL). I’ll provide the steps I followed here.

FYI, the next paragraph in the book actually recommends you to use docker. I think there will be a point where it’s to difficult to manage everything yourself and you’ll have to indeed go that direction.

Install Ubuntu

  1. Enable windows feature Windows-Subsystem-For-Linux
    1. PowerShell: wsl --install
  2. Reboot
  3. Install Ubuntu
    1. Powershell: wsl --install -d Ubuntu
  4. Reboot
  5. Open Ubuntu and setup user and password

Install virtual python environment

The below commands are to be run from the Ubuntu command line.

  1. Install git
    1. apt-get install git
  2. Install UV
    1. curl -LsSf https://astral.sh/uv/install.sh | sh
  3. Install virtual env
    1. uv tool install virtualenv

Test the setup

The below commands are to be run from the Ubuntu command line.

  1. Create your project:
    1. virtualenv myproject
  2. Activate it
    1. source myproject/bin/activate
  3. Verify python is running from inside environment:
    1. which python
  4. Install a package:
    1. pip install requests
  5. Deactivate the project:
    1. deactivate

Setup a new virtual python environment and install airflow

  1. Create virtual python environment for airflow
    1. virtualenv airflowtest
    2. source airflowtest/bin/activate
  2. Install apache airflow
    1. pip install apache-airflow
    2. The installation log will show you the admin password
  3. Run airflow
    1. airflow standalone
  4. Open airflow

Sources used