Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,11 @@ UC_TABLE_NAME=

# Lakehouse Analytics pipeline (optional - local mode does not need these)
# Used in: pipelines/lakehouse_analytics/
# Set GCP_PROJECT for BigLake mode, or AWS_S3_BUCKET for AWS Glue mode.
# Leave both unset for local SQLite Iceberg.
# Set GCP_PROJECT and GCS_BUCKET (both, your own project and bucket) for BigLake
# mode, or AWS_S3_BUCKET for AWS Glue mode. There are no defaults for these.
# Leave all unset for local SQLite Iceberg.
GCP_PROJECT=
GCS_BUCKET=daft-lakehouse
GCS_BUCKET=
AWS_S3_BUCKET=
AWS_REGION=us-east-1
LAKEHOUSE_NAMESPACE=analytics
Expand Down
11 changes: 8 additions & 3 deletions pipelines/catalog.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
Three catalog backends, selected by environment variables:

SQLite (default) — no env vars needed, writes to .lakehouse/
GCS + BigLake — set GCP_PROJECT
GCS + BigLake — set GCP_PROJECT and GCS_BUCKET (your own, no default)
AWS S3 + Glue — set AWS_S3_BUCKET

The S3 backend pairs naturally with AWS S3 Files: mount the same bucket
Expand All @@ -33,7 +33,7 @@
REPO_ROOT = Path(__file__).resolve().parent.parent
LAKEHOUSE_DIR = Path(os.environ.get("LAKEHOUSE_DIR") or REPO_ROOT / ".lakehouse")
GCP_PROJECT = os.environ.get("GCP_PROJECT") or None
GCS_BUCKET = os.environ.get("GCS_BUCKET") or "daft-lakehouse"
GCS_BUCKET = os.environ.get("GCS_BUCKET") or None
AWS_S3_BUCKET = os.environ.get("AWS_S3_BUCKET") or None
AWS_REGION = os.environ.get("AWS_REGION") or "us-east-1"
NAMESPACE = os.environ.get("LAKEHOUSE_NAMESPACE") or "analytics"
Expand All @@ -42,7 +42,12 @@
def get_session(namespace: str | None = None) -> Session:
"""Return a Session with an Iceberg catalog attached."""
namespace = namespace or NAMESPACE
if GCP_PROJECT:
if GCP_PROJECT or GCS_BUCKET:
if not (GCP_PROJECT and GCS_BUCKET):
raise RuntimeError(
"BigLake mode needs both GCP_PROJECT and GCS_BUCKET. "
"Set them to your own Google Cloud project and bucket, or unset both for local mode."
)
from pyiceberg.catalog import load_catalog

iceberg_catalog = load_catalog(
Expand Down
18 changes: 10 additions & 8 deletions pipelines/lakehouse_analytics/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Build a complete analytics lakehouse using Daft + Apache Iceberg. Three backends
| Backend | Env var | Catalog | Warehouse | Bonus |
|---------|---------|---------|-----------|-------|
| **Local** (default) | — | SQLite | `.lakehouse/` | Zero setup |
| **GCS + BigLake** | `GCP_PROJECT` | BigLake REST | GCS bucket | Auto-federated to BigQuery |
| **GCS + BigLake** | `GCP_PROJECT` + `GCS_BUCKET` | BigLake REST | GCS bucket | Auto-federated to BigQuery |
| **AWS S3 + Glue** | `AWS_S3_BUCKET` | AWS Glue | S3 bucket | Mount locally via [S3 Files](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-files.html) |

This pipeline demonstrates:
Expand Down Expand Up @@ -37,8 +37,8 @@ flowchart LR
# Local mode (SQLite Iceberg — no cloud needed)
uv run --extra lakehouse -m pipelines.lakehouse_analytics.ingest

# BigLake mode (GCS + BigQuery)
GCP_PROJECT=eventual-analytics uv run --extra lakehouse -m pipelines.lakehouse_analytics.ingest
# BigLake mode (GCS + BigQuery), your own project and bucket
GCP_PROJECT=my-project GCS_BUCKET=my-lakehouse uv run --extra lakehouse -m pipelines.lakehouse_analytics.ingest

# AWS mode (S3 + Glue)
AWS_S3_BUCKET=my-lakehouse uv run --extra lakehouse -m pipelines.lakehouse_analytics.ingest
Expand All @@ -53,7 +53,7 @@ uv run --extra lakehouse -m pipelines.lakehouse_analytics.backfill \

# Query and visualize
uv run --extra lakehouse -m pipelines.lakehouse_analytics.analyze
GCP_PROJECT=eventual-analytics uv run --extra lakehouse -m pipelines.lakehouse_analytics.analyze
GCP_PROJECT=my-project GCS_BUCKET=my-lakehouse uv run --extra lakehouse -m pipelines.lakehouse_analytics.analyze
AWS_S3_BUCKET=my-lakehouse uv run --extra lakehouse -m pipelines.lakehouse_analytics.analyze
```

Expand All @@ -67,8 +67,8 @@ cp .env.example .env

| Variable | Purpose | Default |
|----------|---------|---------|
| `GCP_PROJECT` | Enables BigLake mode and sets the Google Cloud project. | unset |
| `GCS_BUCKET` | GCS warehouse bucket for BigLake mode. | `daft-lakehouse` |
| `GCP_PROJECT` | Your Google Cloud project. Set together with `GCS_BUCKET` to enable BigLake mode. | unset |
| `GCS_BUCKET` | Your GCS warehouse bucket for BigLake mode. Required with `GCP_PROJECT`; no default. | unset |
| `AWS_S3_BUCKET` | Enables AWS Glue mode with this S3 bucket as warehouse. | unset |
| `AWS_REGION` | AWS region for the Glue catalog and S3 bucket. | `us-east-1` |
| `LAKEHOUSE_NAMESPACE` | Default Iceberg namespace for ingest and analyze. | `analytics` |
Expand All @@ -87,8 +87,8 @@ cp .env.example .env
```bash
# Query-based source
uv run --extra lakehouse -m pipelines.lakehouse_analytics.backfill \
--source-url bigquery://eventual-analytics \
--query "SELECT * FROM github_daft_stargazers.stargazers" \
--source-url bigquery://my-project \
--query "SELECT * FROM my_dataset.stargazers" \
--target-table github_stargazers \
--key _dlt_id \
--target-namespace analytics
Expand All @@ -105,6 +105,8 @@ The same values can be supplied through `BACKFILL_SOURCE_URL`, `BACKFILL_QUERY`,

## Setup (BigLake mode)

Use your own Google Cloud project and bucket. The pipeline writes tables to whatever bucket you configure, so never point it at a bucket you do not own. If you set only one of `GCP_PROJECT` / `GCS_BUCKET`, the pipeline stops with an error instead of falling back.

```bash
# 1. Create GCS bucket
gcloud storage buckets create gs://my-lakehouse --location=us-west1 --uniform-bucket-level-access
Expand Down
2 changes: 1 addition & 1 deletion pipelines/lakehouse_analytics/analyze.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

Usage:
uv run --extra lakehouse -m pipelines.lakehouse_analytics.analyze
GCP_PROJECT=eventual-analytics uv run --extra lakehouse -m pipelines.lakehouse_analytics.analyze
GCP_PROJECT=my-project GCS_BUCKET=my-lakehouse uv run --extra lakehouse -m pipelines.lakehouse_analytics.analyze
"""

from datetime import datetime
Expand Down
2 changes: 1 addition & 1 deletion pipelines/lakehouse_analytics/backfill.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
--target-namespace analytics

uv run --extra lakehouse -m pipelines.lakehouse_analytics.backfill \
--source-url bigquery://eventual-analytics \
--source-url bigquery://my-project \
--query "SELECT * FROM dataset.table" \
--target-table table \
--key id
Expand Down
4 changes: 2 additions & 2 deletions pipelines/lakehouse_analytics/ingest.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,11 @@
"""Ingest pipeline — fetch from APIs via custom DataSources, write to Iceberg.

Runs all sources and upserts into the lakehouse. Works in local mode (SQLite)
or BigLake mode (GCS + BigQuery) depending on GCP_PROJECT env var.
or BigLake mode (GCS + BigQuery) depending on whether GCP_PROJECT and GCS_BUCKET are set.

Usage:
uv run --extra lakehouse -m pipelines.lakehouse_analytics.ingest
GCP_PROJECT=eventual-analytics uv run --extra lakehouse -m pipelines.lakehouse_analytics.ingest
GCP_PROJECT=my-project GCS_BUCKET=my-lakehouse uv run --extra lakehouse -m pipelines.lakehouse_analytics.ingest
"""

import daft
Expand Down
Loading