📖 Chapter 01 — The Data Scientist's NAS
What changes when the workload is analysis, not files.
Learning Objectives
- Identify the differences between a file server and a data analysis platform
- Set up the dataset hierarchy for analytical work
- Choose the right tools for the data layer (Jupyter, libraries, file formats)
Introduction
Volumes 1-4 built a NAS for files, family, creator, and lab. The workloads were: store files, serve them to the family, edit videos, run self-hosted services. This volume is a different workload: analyze data. The patterns from earlier volumes still apply; what's different is what "using the data" looks like.
This chapter sets up the data layer: where the datasets live, how to organize them, and what tools the data scientist's NAS has.
What changes when the workload is data
Three differences between a file-serving NAS and a data analysis NAS:
- Read patterns. The file server reads files once when the family opens them. The data analyst reads the same big file thousands of times (each query, each experiment). Sequential reads, not random.
- Write patterns. The file server writes new files occasionally (a new photo, a new document). The data analyst writes derived data: cleaned datasets, feature tables, model predictions, charts.
- Compute patterns. The file server's CPU is mostly idle. The data analyst's CPU and RAM are heavily used during analysis (and idle between sessions).
The data layer for a data scientist is about: lots of storage for raw and derived data, fast sequential reads (so the analysis doesn't wait), enough RAM to hold working datasets in memory, and the right tools (Jupyter, Python, R, SQL).
The dataset hierarchy
For analytical work, the data lives in a hierarchy:
The principle: raw data is sacred. The raw/ directory is the source of truth. Everything else can be regenerated from raw. This is the discipline that makes the work reproducible (Chapter 07).
The "raw is immutable" rule
Once a file lands in data/raw/, it's never edited. Why:
- Provenance. You can always trace back to the original. "Where did this number come from?" → the raw file.
- Reproducibility. If your analysis produces a surprising result, you can re-run it on the raw data and verify.
- Backup. Raw data is small (relatively). Backing up the raw/ directory is feasible. Backing up everything (including intermediate and processed) is wasteful.
The discipline: anything that touches the raw data is a script (in projects/<name>/src/). The script is what does the cleaning, the transforming, the analysis. The script is in version control. The raw data is not.
The tools
For a data scientist's home NAS, the standard tools:
- Python + Jupyter: the standard for interactive analysis. Pandas for data wrangling, scikit-learn for ML, matplotlib/plotly for visualization, polars for fast DataFrames.
- R + RStudio: the standard for statistical analysis. Tidyverse for data manipulation, ggplot2 for visualization.
- SQL + DuckDB: for querying data without loading it all into memory. DuckDB runs in-process, no server needed.
- DVC (Data Version Control): for versioning datasets. Like Git for data.
- MLflow: for tracking ML experiments. Records parameters, metrics, artifacts.
All of these are available as containers or VMs. The setup is: a Jupyter container with the relevant libraries, mounted to the projects/ directory on the NAS. The data is on the NAS; the analysis runs in the container; the results go back to the NAS.
The "where does the analysis run" question
For most home data science, the analysis runs in a container on the NAS. The NAS has enough CPU and RAM for the analysis. For heavier workloads (large dataset analysis, ML training), the analysis runs on the laptop (with the data mounted via SMB or NFS) or on a separate machine (Chapter 09 of Volume 4).
The conversation's principle: the data lives on the NAS; the analysis can run anywhere. The data scientist's NAS is the data layer. The compute can be on the NAS, the laptop, or a separate machine. The data doesn't move; the analysis moves to the data.
Network access for analytical tools
For analytical work, the data needs to be reachable from wherever the analysis runs. The options:
- SMB mount: the simplest. Mount the NAS share on the laptop (or the analysis container). The tools read files as if they were local.
- NFS mount: faster, more Linux-native. The NAS exports the dataset directories; the laptop (or container) mounts them. Slightly more setup but better performance.
- S3-compatible storage: if you use S3 (or MinIO) for data, the tools read from S3 directly. Useful for large datasets and cloud-style access.
For most home data science, SMB is fine. NFS is better for Linux-native workflows. S3 is for when you have lots of data and want cloud-style access.
The "what if my dataset is bigger than RAM" question
For most home data science, the working dataset fits in RAM. 32 GB of RAM is enough for datasets up to ~10-20 GB. For larger datasets, the tools that work without loading everything into memory:
- DuckDB: an in-process SQL engine that reads Parquet files without loading them all. Query 100 GB of data with 8 GB of RAM.
- Polars: a DataFrame library that uses lazy evaluation. Reads only what's needed for the query.
- Apache Arrow: the in-memory format. Zero-copy reads, efficient memory use.
For truly large datasets (1+ TB), the right answer is a distributed system (Spark, Dask) on a cluster. For most home data science, the right answer is DuckDB or Polars on a single machine.
Setting up the data layer
The minimum setup for a data scientist's NAS:
- Create the dataset hierarchy in
tank/Lab/data/andtank/Lab/projects/. - Set up a Jupyter container (Chapter 02) with the right libraries.
- Mount the projects directory in the Jupyter container.
- Start a new project:
mkdir -p projects/<name>/{notebooks,src,output}+ a README. - Put data in
data/raw/. - Start the analysis in
projects/<name>/notebooks/.
Engineering Note
The data is the asset; the analysis is disposable. The raw data is irreplaceable (in many cases). The analysis can be re-run. The trained model can be re-trained. The chart can be re-generated. The discipline of treating data as sacred and analysis as disposable is what makes the work reproducible. Don't put irreplaceable analysis in the raw/ directory. Put it in projects/ where it can be regenerated.
Summary
The data scientist's NAS is a data layer. The data lives in tank/Lab/data/ with raw, interim, processed, external, and models subdatasets. Raw is sacred and immutable. Projects live in tank/Lab/projects/ with their own notebooks, source, output, and README. The analysis runs in a Jupyter container on the NAS, on the laptop with SMB mount, or on a separate machine. For datasets bigger than RAM, DuckDB or Polars. The data doesn't move; the analysis moves to the data.
Checklist
- ⬜ Create the dataset hierarchy in
tank/Lab/data/ - ⬜ Create the projects hierarchy in
tank/Lab/projects/ - ⬜ Install Jupyter (Chapter 02)
- ⬜ For each new analysis, start a project with a README, notebooks, src, output
- ⬜ Never edit raw data. Always derive from raw via a script.
Looking Ahead
Chapter 02 is Jupyter: the interactive computing environment. JupyterHub for multi-user, JupyterLab for the modern UI, the patterns that make Jupyter a daily tool, not a one-off. The chapter that turns "I want to analyze some data" into "I'm in my notebook, the data is loaded, the analysis is running."