A stream processor and a workflow scheduler solve different parts of a data-delivery problem. I built a local pipeline to make that boundary explicit: Flink continuously validates and classifies events; Kafka retains its outputs; Airflow performs finite publication work when data becomes available.
The source is a Java generator of synthetic Windows events over TCP. It is a controlled development input, rather than a claim that raw TCP logging is the default architecture for a modern organisation. The interesting engineering work is the separation of event-time computation, scheduling hints, durable output and acknowledgement.
Explore the streaming pipeline.
Give Flink an actual processing role
Flink validates required fields and types, normalizes UTC timestamps, and assigns severity and category fields. Valid curated events, security alerts and quarantine records go to separate versioned Kafka topics.
The security rule detects at least three distinct failed-login event identities for a computer in a 30-second event-time window. Five seconds of out-of-order tolerance are allowed. Repeated copies of one eventUid count once within that window. The rule counts event identities, not distinct Windows event codes: failed logins use the same numeric code.
Malformed input is preserved in quarantine. A valid failed-login event that arrives too late for a completed detection window still reaches the curated event stream and is also quarantined for the missed window. Alert identities are stable for a computer, window and rule.
The Java 17 job uses Flink 1.20.3, five-second checkpoints and at-least-once Kafka sinks. Those checkpoints do not make the TCP input replayable. Events lost before reaching retained Kafka records cannot be reconstructed by this pipeline.
Separate a notification from an acknowledgement
Airflow 3.3.2 uses the supported provider MessageQueueTrigger through an AssetWatcher. A notification callback passes only topic, partition and offset. Event payloads remain in Kafka, and XCom carries small batch references and summaries.
There are two consumer groups with different responsibilities:
| Group | Meaning of progress |
|---|---|
| Notification group | A scheduling hint has been constructed |
| Processing group | The corresponding output has been durably published and verified |
The provider commits notification progress before yielding its scheduling event. A crash in that gap can lose a hint, and notifications can advance ahead of unfinished output. The DAG therefore combines asset-triggered scheduling with a five-minute reconciliation schedule. Reconciliation reads the processing group's actual backlog, independently of notification offsets.
This preserves the benefit of message-driven scheduling without treating a hint as a durable work receipt. Airflow may coalesce asset updates, while one active DAG run serializes batch execution.
Keep the batch finite and the retry boundary stable
The DAG has three sequential TaskFlow tasks:
| Task | Responsibility | Behaviour on retry |
|---|---|---|
| Stage a Kafka batch | Read at most 1,000 records or 20 seconds and persist fixed offset ranges | Reuse the pending batch |
| Validate and publish | Check contracts, deduplicate and publish immutable Parquet with checksums | Reuse a completed publication |
| Commit and emit | Verify output, acknowledge next offsets and emit the output asset | Retry acknowledgements without rewriting output |
The stage boundary prevents a retry from quietly consuming a different batch. The publication boundary prevents partial files from appearing as completed output. The final acknowledgement records what has actually survived publication.
Airflow is doing bounded batch work here. Flink continues to process the stream independently; the scheduler is not being used as a continuously running per-message transformation engine.
Publish before advancing Kafka progress
Staging and publication use a shared local volume. The batch writes typed event, alert and quarantine Parquet files plus a checksum manifest outside the readable publication directory. Files are synced before a same-filesystem atomic directory rename makes the complete batch visible.
Only the published directory is a dataset. Empty typed output files are valid; partly written batches remain invisible. The protocol uses POSIX locking and atomic rename on one shared filesystem. It cannot be moved to object storage by simply changing a path.
Kafka acknowledgement happens after output verification. Each partition commits its next offset. If publication succeeds but acknowledgement fails, the retry recovers the pending batch and reuses the durable output. Existing committed progress is never moved backwards. A retention gap fails explicitly instead of silently skipping unavailable records.
Deduplicate transport replay and business replay
Replaying a Kafka record is different from republishing the same logical event at a new offset. The publication ledger handles both:
- Published offset ranges identify transport records already materialized, including replay with different batch boundaries or consumer groups.
- Immutable event, alert and quarantine identities identify logical records repeated at new offsets. A reused identity with a different payload fails the batch for investigation.
Every staged input is accounted for as published, quarantined or duplicate. Identity checks read the relevant identity and payload-hash columns. This is suitable for the local demonstration's volume; a larger archive would need compaction and a table-format catalog.
The resulting guarantee is at-least-once processing from retained Kafka records with idempotent Parquet output. It does not establish end-to-end exactly-once delivery across the non-replayable TCP segment.
What the recorded integration established
The 25 October verification report records a successful asset-triggered run with all three tasks complete. It observed 395 fresh events, a failed-login alert with four distinct identities in a 30-second window, checksum-valid output and acknowledged processing offsets.
Replaying existing Kafka records added zero event rows. Republishing a known event at a new offset retained exactly one output row for its identity. The checked batch published 256 checksum-verified rows. These are observations from that recorded run, not a live counter or a throughput benchmark.
The report records 27 passing Python tests in the Astro Runtime and five passing JUnit tests from the preceding Flink image build. Coverage includes failed publication, corrupted checksums, lost acknowledgement responses, pending recovery, retention gaps and separation of notification and processing groups.
Sustained throughput, a 24-hour endurance run and distributed storage transactions have not been established. Those are separate acceptance criteria, rather than conclusions inferred from one integration run.
Reproduce the local system
The combined repository contains the maintained Java source and all container builds; the original TCP connector does not need a separate checkout. Podman and Astro CLI run the services locally without a cloud account.
git clone https://github.com/vikramaditya-tatke/kafka-flink-airflow-pipeline.git
cd kafka-flink-airflow-pipeline
./scripts/pipeline.sh up
./scripts/pipeline.sh status
./scripts/pipeline.sh verify
The verifier requires an actual asset-triggered three-task run. It checks fresh events, security-window alerts, published checksums, processing acknowledgements and replay while normal scheduling continues. It does not substitute a manually triggered run for the scheduling behaviour under test.
The design notes explain the tradeoffs. The next work is sustained recovery testing and an explicit storage protocol for any distributed deployment.