Home · Volume 5 · Chapter 04

📖 Chapter 04 — Pipelines and Schedules

The orchestration of recurring data work. From "I run this every Monday" to "this runs every Monday, and I know when it fails."

Learning Objectives

Schedule recurring work with cron
Build a data pipeline with Python scripts
Use a pipeline orchestrator (Airflow or Prefect) for complex workflows
Monitor pipelines and alert on failure

Introduction

You have data. You have analysis. Now: when do you run the analysis? Once is exploration. Twice is a habit. Three times is a pattern that should be automated.

This chapter is about the patterns that turn "I run this manually" into "this runs on its own, and I know when it fails." Cron for simple cases, Airflow or Prefect for complex ones.

The simplest case: cron

For a single recurring task, cron is the right answer. Cron is the Unix scheduler: it runs commands at specified times.

Example: a script that pulls data from an API and saves it as a Parquet file every day at 6 AM.

0 6 * * * /usr/bin/python3 /mnt/tank/Lab/projects/etl/fetch.py

The format: minute hour day-of-month month day-of-week command. The example: at 6:00 AM every day, run the script.

Setting up cron

On the NAS, edit the crontab:

crontab -e

Add the line. Save and exit. Cron will run the command at the specified time.

For a Docker container, the cron runs inside the container. Add a cron job to the container's crontab. Or, more typically, run the container itself on a schedule (with a tool like ofelia or just a wrapper script).

What cron is good for

Single commands, run on a schedule
Scripts that don't depend on other scripts
Short-running tasks (a few minutes)

What cron is not good for

Complex dependencies (run B only after A succeeds, run C only if A or B fails)
Long-running tasks (hours) — cron doesn't track them well
Retries with backoff
Backfills (re-run for last month's dates)

For these, you need a pipeline orchestrator.

Pipeline orchestrators: Airflow and Prefect

A pipeline orchestrator is a tool for defining, scheduling, and monitoring multi-step workflows. The two popular open-source choices:

Apache Airflow: the standard for data engineering. Python-based DAGs, web UI for monitoring, large ecosystem. Heavier (more components to run).
Prefect: the modern alternative. Python-based flows, simpler UI, dynamic workflows. Lighter than Airflow, more modern.

For most home data science, the choice is between Prefect for simplicity and Airflow for ecosystem. If you're learning for the first time, Prefect is easier. If you're using tools that already integrate with Airflow (e.g., dbt, Great Expectations), Airflow is the standard.

A Prefect example

A simple pipeline: fetch data, clean it, save as Parquet, run a model, save the result.

from prefect import flow, task

@task
def fetch_data():
    import requests
    response = requests.get("https://api.example.com/data")
    return response.json()

@task
def clean_data(raw):
    import polars as pl
    df = pl.DataFrame(raw)
    return df.drop_nulls()

@task
def save_processed(df):
    df.write_parquet("/mnt/tank/Lab/data/processed/cleaned.parquet")

@task
def train_model(df):
    from sklearn.linear_model import LinearRegression
    model = LinearRegression()
    # ... fit the model
    return model

@flow
def daily_pipeline():
    raw = fetch_data()
    cleaned = clean_data(raw)
    save_processed(cleaned)
    model = train_model(cleaned)

daily_pipeline.serve(name="daily-pipeline", cron="0 6 * * *")

This is a Prefect flow. Each function decorated with @task is a step. The @flow function orchestrates the steps. The serve call schedules it.

When to use an orchestrator

Use an orchestrator when:

You have multi-step workflows with dependencies
You want retries, backfills, and monitoring out of the box
You want a web UI to see what's running and what failed
You have multiple pipelines that share infrastructure

Don't use an orchestrator for:

A single command that runs on a schedule (cron is enough)
A one-off analysis (just run the script)
Real-time stream processing (that's a different tool category)

The "I just need to know it ran" pattern

For most home data science, the right pattern is: simple cron + a script that emails you (or sends a notification) when it fails. The notification is the "I know it failed" piece; cron is the "it runs on schedule" piece.

Example:

#!/bin/bash
set -e  # exit on any error

# Run the pipeline
python3 /mnt/tank/Lab/projects/etl/fetch.py

# If we got here, it succeeded. Send a success notification.
curl -X POST https://ntfy.sh/my-topic-alerts -d "ETL job succeeded"

The script exits on any error. If it exits with a non-zero code, cron can email you (configure MAILTO in crontab). The curl at the end is a manual "it worked" notification.

For a more sophisticated setup, use a tool like healthchecks.io (self-hostable): the script pings a URL on success; if the URL isn't pinged, the system alerts you.

The "let me think about it" pattern for data

For exploratory work, the right pattern is: don't schedule. Run the analysis when you want to see it. Use cron or Prefect for the workflows that need to run regularly (data ingestion, regular reports, model retraining).

The discipline: if you've run an analysis three times, automate it. If you've only run it once or twice, leave it manual. The cost of automation is real; the benefit only shows up at scale.

Logging and monitoring

Every pipeline should log:

When it started
What it did (each step)
How long each step took
Any errors
When it finished (and whether it succeeded or failed)

For cron scripts, the log goes to a file. Configure the crontab line to redirect output:

0 6 * * * /path/to/script.sh >> /var/log/etl.log 2>&1

For orchestrators, the log is in the tool's UI. Airflow and Prefect both show you the log per task per run.

Failure modes

What to do when a pipeline fails:

The alert fires (email, Ntfy, etc.)
Check the log: what step failed, with what error
Diagnose: data changed, API changed, disk full, network down?
Fix the data, the API call, the disk, the network
Re-run the pipeline (manually or automatically via the orchestrator's retry)

The discipline: every pipeline has a runbook. When it fails, you follow the runbook. The runbook is in the project's docs/ directory.

Engineering Note

Scheduled work should be observable. A pipeline that runs every day but you don't know if it succeeded is a pipeline you don't trust. A pipeline that runs every day, with alerts on failure, with a runbook for common errors, with logs you can read — that's a pipeline you trust. The work of automation is not just running the code; it's making sure you know what's happening.

Summary

Cron for simple cases (one script, one schedule). Airflow or Prefect for complex cases (multi-step, dependencies, retries, monitoring). Each task is a Python function. The flow is a Python script that calls the tasks. Schedule with cron string. Log everything. Alert on failure. The discipline: automate the third time. Leave one-offs and explorations manual.

Checklist

⬜ Identify the workflows you run more than twice
⬜ For simple workflows, use cron + a log file + an alert on failure
⬜ For complex workflows, set up Prefect (or Airflow)
⬜ Document the runbook for each pipeline: what to do when it fails
⬜ Verify the alert fires when the pipeline fails (test it)

Looking Ahead

Chapter 05 is machine learning. The ML lifecycle on the NAS: training, tracking, deploying, monitoring. MLflow for experiment tracking. The chapter that turns "I trained a model" into "I have a reproducible model, with the code that trained it, the data it was trained on, and the metrics to know if it's still good."

Previous← Ch 03 · Data Storage and Format Next📖 Ch 05 · Machine Learning →

Ch 04 · v0.1 · drafted from the original ChatGPT conversation, July 2026