A data pipeline needs a precise answer to a simple question: when is one input complete? Downloading a file is only part of that answer. Conversion can fail, publication can be interrupted, and a rerun can accidentally replace an output that was already correct.
I built a resumable bronze ingestion boundary for Inside Airbnb snapshots. Its unit of work is one city, snapshot date and table. DuckDB and Polars implement the same contract, S3 publication is protected by conditional writes, and a completion manifest records the evidence needed to accept or recover an output.
The recorded real-S3 run verified 194,912 Albany records across listings, calendar and reviews. That establishes the bronze ingestion boundary. It does not establish a finished Airbnb analytics platform.
Keep bronze faithful to the source
Inside Airbnb publishes compressed CSV inputs. The runner downloads the gzip bytes to a local work directory before conversion. It checks the response status, expected byte length when supplied, gzip signature and source checksum. Transient failures have bounded retries; an HTTP 403 is reported as a failure rather than silently skipped.
The conversion engines initially had a semantic disagreement: type inference could represent calendar availability as a boolean in one engine and a string in the other. Bronze now preserves CSV fields as UTF-8 strings in both engines. IDs, currencies and dates retain their source representation; typed business rules belong in the next layer.
DuckDB reads with all_varchar=true; Polars disables schema inference. Both produce Zstandard-compressed Parquet. A shared validator checks required columns, raw-string schema, nonempty row counts and identifier constraints before an S3 data write is accepted.
The contract makes changing the conversion engine an implementation choice. It does not create another dataset identity or require replacing a verified completion.
Treat publication as a per-dataset commit
Each selected input has a canonical bronze key and a corresponding completion manifest. The intended sequence is:
Download → convert → validate → publish Parquet → commit manifest
Conditional writes protect data and manifests from accidental replacement, including the multipart-upload path. The stored metadata and checksums allow interrupted publication to be reconciled. A manifest alone is not enough: verification checks that its referenced object still satisfies the completion contract.
This is deliberately independent for every city, snapshot and table. One inaccessible source can fail while previously completed datasets remain resumable. The CLI reports failed work and exits nonzero, rather than returning a successful-looking partial ingestion.
The current source configuration contains historical URLs that returned 403 during verification. A complete run needs those sources updated or an explicit reachable-city selection. Increasing retries will not repair a source that refuses access.
Orchestrate the same runner with Airflow
The Airflow 3 TaskFlow DAG uses the ingestion package rather than maintaining a second conversion implementation:
discover → ingest.expand(dataset=...) → verify
Discovery builds small dataset descriptions from configuration. Mapped ingestion tasks download, convert, validate and publish each dataset. The final task reconciles completion manifests with the stored outputs.
The DAG is manually triggered, defaults to Albany's three datasets, allows two concurrent mapped ingestion tasks and one active DAG run. Configuration and AWS credentials must be available to every worker. Bucket bootstrap is an explicit operator step before the DAG runs.
DAG imports and dependency checks passed in the recorded verification. A deployed scheduler execution has not been verified. I keep that distinction explicit because importing a DAG does not prove its workers can reach the source, obtain credentials and publish data successfully.
Verify a rerun changes nothing
The recorded ingestion evidence contains these round-trip counts:
| Dataset | Verified rows |
|---|---|
| Listings | 459 |
| Calendar | 167,535 |
| Reviews | 26,918 |
| Total | 194,912 |
The first run ingested all three datasets. A rerun using Polars skipped all three completed outputs, with no failures and no new S3 object versions. The recorded version count remained six before and after the rerun. Output checksums and independent Parquet reads reconciled with the manifests.
Bucket checks also recorded private access, versioning, encryption, ownership controls and incomplete-multipart cleanup. These are operational properties of the tested ingestion environment, not evidence that the unfinished analytical layers are correct.
The recorded ingestion evidence and ingestion guide are in the private repository; repository access is required to inspect them. The portfolio states their recorded results without presenting a private repository as publicly browsable.
Make the unfinished layers visible
Silver still has known correctness and runtime failures, including identifier and type-handling problems and unsafe multi-city overwrite behaviour identified by the diagnostic suite. The current repository reports twelve diagnostic failures. ClickHouse gold is not implemented, and no verified analytical dashboard is available.
The intended analysis includes neighbourhood prices, availability changes across snapshots and listing concentration among hosts. Those questions need verified transformations and carefully defined measures. Listed availability, for example, cannot simply be presented as observed occupancy.
I will repair and reconcile silver, implement gold, and verify a deployed Airflow run before presenting analytical findings. This project is separate from the Kafka/Flink event pipeline: its verified responsibility is bounded snapshot ingestion into S3 bronze.
Run the bounded ingestion workflow
With the repository environment installed and an AWS profile or role configured:
uv run airbnb-ingest plan --city albany
uv run airbnb-ingest bootstrap
uv run airbnb-ingest run --city albany
uv run airbnb-ingest verify --city albany
uv run airbnb-ingest run --city albany
The first command inspects selected inputs and output keys without network calls. The last command exercises the completed-dataset skip path. Failure, recovery and race tests complement that real-data evidence; they do not replace it.