Dataflow Engine
A lightweight data integration engine with YAML-defined pipelines, visual flow monitoring, and reconciliation checks — built as an n8n-adjacent demo.
Pipeline dashboard
Visual flow canvas with live run history
Why this exists
Understanding pipeline orchestration from first principles
Rather than starting with a framework, this spike builds a data integration engine from scratch — sources, transforms, sinks, scheduling, and reconciliation — to understand the core abstractions that tools like n8n are built on.
The web UI renders YAML pipeline definitions as visual node flows, with one-click execution and a persistent run history backed by SQLite.
What it demonstrates
1. Define a pipeline in YAML — source, transforms, sinks, schedule.
2. The engine loads and validates the definition at startup.
3. The web UI renders each pipeline as a connected node flow.
4. Run pipelines from the UI and inspect results in the history table.
Pipeline architecture
Connectors and transforms
What I learned
Building the engine first and the UI second made it clear which abstractions actually matter: the source/transform/sink pipeline model, reconciliation as a first-class concept, and YAML as a declarative interface all survived contact with real data. The visual flow was straightforward to layer on top because the domain model was already clean.
The reconciliation check — comparing records fetched to records written — is a small feature that punches above its weight. Most pipeline tools bolt this on later or leave it to the user. Making it declarative in the YAML definition keeps the pipeline author thinking about data quality from the start.