Overview
The Federator enables secure, policy-controlled Kafka-to-Kafka data sharing between organisations, ensuring only explicitly authorised, security-labelled data is exchanged.
Federator service architecture (cross-organisation view)
This diagram shows how multiple organisations exchange data using the Federator:
- Each organisation may run:
- One or more Federator Producers (server role) which read from approved Kafka topic streams
- One or more Federator Consumers (client role) which mutually authenticate to remote Producers, request authorised topics, and write consumed data from Producers
- Data is exchanged using gRPC over MTLS to guarantee strong authentication and encrypted transport.
- Authorisation, topic configuration, and release policies are centrally managed through a Management-Node, while short-lived identity tokens are issued by an Identity Provider.
- Offset tracking and replay determinism are handled via Redis, ensuring safe recovery and auditable message flow.
Together, these components enforce minimum-trust participation: explicit topic approval, policy-based message release, controlled ingress boundaries, replay safety, and cryptographic mutual authentication. The result is a federated data-sharing architecture that is secure, auditable, and aligned with NDTP minimum-trust principles.
Federator components context (runtime view)
Core runtime components:
Federator Producer
- Reads from source Kafka
- Filters messages using
securityLabel - Streams approved messages to Consumers over gRPC
Federator Consumer
- Authenticates to Producer (MTLS)
- Requests authorised topics
- Writes messages to local Kafka under
federated.*topics - Tracks offsets in Redis for deterministic replay
Redis
- Stores Consumer offset positionsE
- Enables deterministic replay and recovery
- Provides short-lived coordination/cache state
Management-Node
- Manages organisations, roles, and trust relationships
- Configures approved topics and release policies
- Distributes configuration to Producers and Consumers — configuration, roles, topics, filters
Identity Provider — issues short-lived tokens
- Issues short-lived identity tokens
- Supports federated trust between organisations
- Underpins MTLS and service authentication flows
Kafka
- Hosts source topics for Producers
- Receives federated data into federated.* topics
- Remains fully controlled within each organisation
Minimum trust guarantees (N2 participation)
The Federator enforces:
- Mutual authentication (MTLS)
- Explicit topic authorisation
- Policy-driven message release (
securityLabelfiltering) - Controlled ingress via
federated.*topics - Replay safety using Redis offsets
This ensures data sharing remains explicit, auditable, and aligned with NDTP minimum-trust principles.