Project
Roadmap
Hermes ships a working foundation and builds the general-purpose core out progressively. The target architecture is specified in docs/architecture/hermes-core.md.
The journey at a glance
| Phase | Focus | ETA-ish |
|---|---|---|
| Phase 1 — Foundation | fetch, parse, normalize, validate, Dataset, schema system | Now → next |
| Phase 2 — Reliable infra | storage, snapshots, provenance, lineage, query interface | In progress |
| Phase 3 — Ecosystem | Hermes Finance, Defense, Healthcare, Trade, Energy, Climate… | Next |
| Phase 4 — Scale | object storage, distributed processing, continuous ingestion | Later |
| Phase 5 — Cloud | hosted datasets, APIs, catalogs, versioned data, team access | Future |
Today (v0.2.x)
| Area | Status |
|---|---|
| Connectors (10) | Working & tested |
| Feature engine (@feature, LineageGraph, TieredPlan) | Working |
| Country-risk features (5 groups) | Working; geopolitical stubbed, others partial |
| Financial features (technical / fundamental / crypto / filing) | Working |
| RawCache (parquet, TTLs, hit/miss) | Working |
| Asyncio scheduler (@schedule, cron/interval) | Working |
| Entities (countries, companies) | Working |
| Core lifecycle modules (parse/normalize/validate/schema/metadata/query/storage/api) | Scaffolded — being built out |
The core life-cycle specification
hermes-core.md describes the general-purpose data-lifecycle engine Hermes is building toward. Lifecycle subsystems:
- Acquisition — fetch, ingest, source, connect, read, stream
- Parsing — parse, detect_format, read_raw, decode (CSV/JSON/JSONL/XML/Parquet/Arrow/compressed)
- Data Contract / Schema — schema, infer_schema, validate_schema, migrate_schema
- Normalization — normalize, map, cast, standardize, convert_units, align_time
- Quality — validate, check, profile, deduplicate, detect_anomalies (ML-extensible)
- Transformation — transform, pipe, select, filter, join, aggregate (Polars/Arrow/DuckDB)
- Identity / Resolution — resolve, identify, match, link, entity (extension point)
- Storage — save, load, delete (FS/Parquet/Arrow/DuckDB/PostgreSQL)
- Query — query, sql with filter/project/join/aggregate/order/limit
- Versioning — version, snapshot, diff (immutable snapshots)
- Provenance & Lineage — lineage, provenance, trace
Cross-cutting design rules: a registry for every component, a defined error hierarchy, an extension architecture, and the dependency rule — Core never depends on a domain package. The public API target is hr.fetch/ingest/parse/normalize/validate/profile/inspect/transform/query/ save/load/export.
The phases
phases
Phase 1 — Foundation
fetch, ingest, parse, normalize, validate, profile, inspect,
transform, export, Dataset, connector system, schema system
Phase 2 — Reliable infrastructure
storage, versions, snapshots, provenance, lineage,
better validation & profiling, caching, query interface
Phase 3 — Ecosystem
Hermes Finance, Defense, Healthcare, Trade, Energy, Climate,
Geopolitics, Corporate, Entity, Features
Phase 4 — Scale
remote datasets, object storage, distributed processing,
continuous ingestion, large-dataset querying, cloud execution
Phase 5 — Hermes Cloud
hosted datasets, APIs, dataset catalogs, continuous pipelines,
versioned data, team access, usage controls, enterprise infraThe guiding philosophy: make high-quality data infrastructure accessible through one consistent developer experience. One engine, one ecosystem, any data.
Get involved
Hermes is open source. The repository ships GitHub labels (type / area / difficulty) and a contributing guide (
docs/engineering_team_guide.md) that defines the vertical-slice build order (e.g. World Bank: acquire → parse → normalize → validate → metadata → provenance → Dataset).