Install on Ubuntu

Table of Contents

Build Essentials

Install build essentials

sudo apt-get update && sudo apt-get upgrade -y
sudo apt-get install -y build-essential curl

Python

Ubuntu 20.04 default Python version is Python 3.8.x.

Ubuntu 22.04 default Python version is Python 3.10.x.

Ubuntu 23.10 default Python version is Python 3.11.x.

PyPI

Install PyPI

sudo apt-get update
sudo apt-get install -y python3-pip

Virtual Environment

Install Virtual Environment

sudo apt-get update
sudo apt-get install -y python3.8-venv

sudo apt-get update
sudo apt-get install -y python3.10-venv

sudo apt-get update
sudo apt-get install -y python3.11-venv

Activate Virtual Environment

cd ~
python3.8 -m venv <venv name>
source ~/<venv name>/bin/activate

cd ~
python3.10 -m venv <venv name>
source ~/<venv name>/bin/activate

cd ~
python3.11 -m venv <venv name>
source ~/<venv name>/bin/activate

Deactivate Virtual Environment

deactivate

Install Project needed Dependencies

# In venv
pip install -r requirements-py38.txt

# In venv
pip install -r requirements-py31x.txt

PostgreSQL

The version installed here is PostgreSQL 16.2.

Install PostgreSQL (Automatic)

Already included in the command make install, just execute it.

Install PostgreSQL (Manual)

make install-postgresql

Configure PostgreSQL

Connect to PostgreSQL Database using default user postgres.

sudo -u postgres psql

# In PostgreSQL shell
postgres=# \conninfo

Change password of default user postgres to postgres.

# In PostgreSQL shell
postgres=# \password

Create Database named airflow_db.

# In PostgreSQL shell
postgres=# CREATE DATABASE airflow_db;

List all databases.

# In PostgreSQL shell
postgres=# \l

Configure pgAdmin4

The email for pgAdmin4 is admin@admin.com.
The password for pgAdmin4 is postgres.

Open browser, go to http://127.0.0.1/pgadmin4. Click Add New Server.

Filled below parameters.

Category	Form	Value
General	Name	Airflow
Connection	Host name/address	localhost
	Port	5432
	Maintenance database	airflow_db
	Username	postgres
	Password	postgres

MinIO Object Storage for Linux

MinIO Server

The version installed here is latest.

Install MinIO Server (Automatic)

Already included in the command make install, just execute it.

Install MinIO Server (Manual)

make install-minio-server

Running MinIO Server

The username for WebUI default is minioadmin.
The password for WebUI default is minioadmin.

make run-minio-server

MinIO Client

The version installed here is latest.

Install MinIO Client (Automatic)

Already included in the command make install, just execute it.

Install MinIO Client (Manual)

make install-minio-client

Configure MinIO Client

Notices: Please configure MinIO Client after executing MinIO Server.

The MinIO Server host default is http://127.0.0.1:9000.
The access_key for MinIO Client API default is minioadmin.
The secret_key for MinIO Client API default is minioadmin.

make config-minio-client

Airflow

The version installed here is Airflow 2.8.3.

Install Airflow

Already installed via requirements.txt.

Running Airflow

The username for WebUI default is admin.
The password for WebUI can be seen in standalone_password.txt or in the terminal output log.

# In venv
export AIRFLOW_HOME=$(pwd)
airflow standalone

(Optional) Configure Airflow

Notices: Please configure Airflow after executing Airflow instance once.

Modify two parameters in the airflow.cfg file.

load_templates

Set False if you don't load airflow default templates. Default is True.

load_templates = False

Configure Airflow Database Connection

Notices: Please configure Airflow Database Connection after executing Airflow instance once.

Modify two parameters in the airflow.cfg file.

sql_alchemy_conn
executor

sql_alchemy_conn = postgresql+psycopg2://postgres:postgres@localhost:5432/airflow_db

executor = LocalExecutor

After the modification is completed, save the file and restart the airflow instance.

Last modified: 01 October 2024