An auditable Excel-to-PostgreSQL import pipeline for a banking and finance counseling company, built around a dry run that shares its code path with the import.
Category
Data Engineering & Architecture
Stack
Financial data onboarding needs traceability before every write.
Spreadsheets arrive with drifting schemas, mixed date and number formats, and row-level errors that surface only after parsing, in a sector where each imported record must stay explainable.
Designed a five-step Angular 21 and DevExtreme wizard that selects the target entity, validates the upload, maps columns to database fields with reusable saved mappings, previews the transformed rows, and gates the write behind a human review.
On the server, a Clean Architecture .NET 10 backend pairs a streaming ExcelDataReader pipeline with a single-transaction binary COPY, runs the pre-flight dry run on a separate queue through the same parse and evaluation core, enforces per-entity validation rules, and writes a two-tier audit journal holding per-batch totals and per-row snapshots.
Upload handling rejects macros, inspects the OOXML package, caps declared sizes, and stores files under generated names outside the webroot.
Delivered during an internship at Axley Bankers, then continued independently with a batch-first journal, per-table concurrent imports, and distinct validating and copying phases.
Verified a 60 MB workbook of 1.1 million rows ingested end to end through the dev server.
Automated suites cover the domain layer, the application layer, functional paths against a real PostgreSQL, and browser specs in headless Edge, with every suite green.
The delivered V1 ships with French technical documentation and a user guide rendered to PDF with Quarto and Typst.