Skip to content

Federator Overview

Purpose

This guide shows how to build, start, and validate a local Federator deployment. The Federator enables secure, policy-controlled data exchange between Integration Architecture (IA) Nodes for National Digital Twin Programme (NDTP) participation. It enforces minimum-trust (N2) requirements without shared infrastructure, credentials, or implicit trust between organisations.

For a conceptual overview of how the Federator works, see Concepts: Federator.
For the source developer documentation, see the Federator README on Github.

For a complete, runnable local setup (including configuration, identity, and end-to-end validation), start with:

➡ End-to-End Local Setup

This provides a single, step-by-step path to a working local deployment without trial and error.

Outcome

By completing this guide you will:

  • Build the Federator Java artefacts and Docker images
  • Start a local multi-client / multi-server federation topology
  • Configure Federator Producers and Consumers communicating over gRPC with MTLS
  • Validate secure Kafka-to-Kafka federation between organisational domains
  • Confirm policy-driven message filtering using securityLabel headers
  • Verify Redis-backed offset tracking for deterministic replay and resilience
  • Configure and authorise via the Management-Node
  • Produce federated RDF data ready for ingestion by downstream IA Nodes (e.g. federated.* topics)
  • Verify Kafka-to-Kafka federation integrity
  • Confirm compatibility with downstream IA Nodes consuming federated.* topics (Optional)

Guide Structure

The guide follows the same staged lifecycle as the IA Node documentation. Each section introduces and validates a distinct capability.

  1. End-to-End Local Setup (Recommended)
    Complete runnable setup covering configuration, identity, and validation.

  2. Environment Set Up
    Confirm platform requirements and tooling.

  3. Identity Set Up
    Understand MTLS, certificates, token validation, and authorisation configuration.

  4. Build Components
    Compile Java artefacts and build Docker images.

  5. Run & Validate
    Start a federation topology and validate Producers, Consumers, filtering, and Kafka-to-Kafka flow.

  6. Optional Integration
    Confirm downstream IA Node ingestion from federated.* topics.

  7. Reference
    Topologies, configuration reference, troubleshooting guidance, and operational checks.

Prerequisites

Platform
  • Linux, or Windows with WSL2 only
  • Run from the WSL filesystem (e.g. ~/src)
  • Do not run from /mnt/c (Docker volume semantics may cause permission and performance issues)
Required tools
  • Docker CLI or Docker Desktop
  • Java 21
  • Git
  • curl
Assumptions
  • Kafka is provided via Docker Compose
  • Management-Node and Identity Provider are preconfigured in the demo environment
  • Some configuration values (e.g. IDP and TLS settings) must be populated for local execution
  • This guide focuses on federation behaviour rather than IA Node internals

For the fastest path to a working system, start with:

➡ End-to-End Local Setup

Alternatively, follow the staged guide beginning with:

➡ Environment Set Up