notna.io · EHC Business Intelligence Platform

Applications

Two tools, one pipeline

Each app is independently deployable and operable via CLI. Together they cover the full ETL orchestration lifecycle.

🔄

EHC.Bi.Db.Loader

EHC.Bi.Db.Loader.exe assembly load--object -a opale -b sa_opale -t PATIENT

Multicast ETL Anomaly-aware Incremental

Command lifecycle — how it all connects

Source via:

Step 1

create

Provision target schema

→

Step 2

profile

Build anomaly profile

→

Step 3

load

Stream data to target(s)

# Introspects source schema → creates target table(s) — no DDL scripting needed
EHC.Bi.Db.Loader.exe assembly create--object -a opale -b sa_opale -t PATIENT
✔ Target table created — ready for profiling and loading

A modular, cross-database ETL console application that transfers data from one source to one or more targets simultaneously. Designed to run autonomously — from a scheduled task, SSIS ExecuteProcess, or the Orchestrator — with zero user interaction once configured.

Multicast streaming — one source read, N target writes in parallel via in-memory channels. Loading into three targets costs no more than one.
Incremental loading — automatic watermark-based incremental loads. Configures once via a single row in dbo.incremental_watermark, no CLI changes needed.
Anomaly detection — two-phase statistical column profiling. Profiles built offline, applied per-batch at load time with zero overhead.
Plugin architecture — database drivers loaded at runtime via reflection. Extend to any engine without recompiling the core.
8 CLI commands — load--object, load--sql, create--object, create--sql, recover--load, profile--object, profile--sql, check.
Schema & truncate tokens — @schema=, @truncate, @postprocess inline in the CLI arg.

In development Anomaly detection

A two-phase statistical profiling engine. Phase 1: assembly profile--object scans a source column offline, builds a numeric or categorical signature (min/max/mean/stddev, coverage threshold, value distribution), and persists it to the Tools database. Phase 2: profiles are loaded once at startup and applied per batch during streaming — any row deviating from the signature triggers a warning without aborting the load. Zero extra DB round-trips during data movement.

Next milestone In-memory data transformation

Transformation rules applied inline to the batch already in memory during the streaming pass — type coercions, column renames, value mappings, derived columns — before the data is written to any target. No staging table, no second pass, no additional DB connection. Transformation cost is absorbed into the streaming time that already exists.

Supported engines

SQL Server

Oracle

MySQL

PostgreSQL

Progress OpenEdge

SQLite

Example — incremental load

# Insert once in Tools DB: INSERT INTO dbo.incremental_watermark (source, object_name, watermark_column, watermark_type) VALUES ('opale','PATIENT', 'LAST_MODIFIED,CREATED','DateTime'); # Then just run — no --where needed: EHC.Bi.Db.Loader.exe assembly load--object -a opale -b sa_opale -t PATIENT

⚡

EHC.Bi.Apps.Orchestrator

EHC.Bi.Apps.Orchestrator.exe orchestrator run --tasks opale_daily.json --batch {guid} --threads 4

Wall-clock optimiser LPT (Longest Processing Time) Retry-capable

A generic parallel task orchestrator whose primary goal is to minimise total wall-clock time across a batch of heterogeneous tasks. It dispatches any command-line process — fully decoupled from what the command does — and focuses entirely on scheduling its lifecycle for maximum throughput.

⏱

Wall-clock compression

A 24-task batch totalling 16 hours of sequential work completes in under 1 hour with 4 threads, automatically balanced by LPT (Longest Processing Time) bin-packing. The orchestrator computes the optimal thread count from the CV of task weights — no manual tuning needed.

LPT (Longest Processing Time) scheduling — tasks sorted by weight descending, distributed via greedy bin-packing. Task weights are median runtimes derived from 7 years of production history, making estimates reliable for known objects. New objects are seeded with the batch average and self-correct to a stable median after ~7 days of runs. Queue re-sorts dynamically when actual runtime deviates >50% from estimate.
Resource domain gating — a HardFail on any task immediately gates its entire resourceIdentifier group. Other domains continue uninterrupted.
Retry policy — configurable exit codes trigger retry with wait. Exit code -2 (process failed to start) always HardFails unconditionally.
Batch recovery — rerun with --recovery and the same GUID: completed tasks are skipped, failed tasks are re-queued.
Any runtime — .exe, python, node, php, any executable. The task file just needs a command string and a weight estimate.
--showconsole — opens a live console window per spawned process for interactive diagnostics, no recompile needed.

Task file — two-section format

{ "resources": [ { "resourceIdentifier": "multicast", "workingDirectory": "C:\\Tools\\Loader" } ], "tasks": [ { "objectCode": "pre", "projectCode": "OPALE", "resourceIdentifier": "multicast", "objectCommand": "Loader.exe assembly load--object -a opale -b sa_opale -t pre", "elapsedTime": 5011 } ] }

Exit code contract

0 → Success 1 (or --retrycodes) → Retry up to --maxretry other non-zero → HardFail, gate domain -2 → HardFail (process couldn't start)

ETL pipelines &
task orchestration,
re-imagined.

Three generations of production evidence

Two tools, one pipeline

How they work together

ETL pipelines & task orchestration, re-imagined.

Three generations of production evidence

Two tools, one pipeline

How they work together

ETL pipelines &
task orchestration,
re-imagined.