Skip to content

End-to-End Local Setup

Purpose

This guide provides a complete, runnable local setup for the Federator with the Management-Node and supporting services. It is the recommended starting point for developers who want to run a working local deployment and validate federation end-to-end.

What you will do

Using this runbook you will walk through the full local flow. You will:

  • Start all required local services
  • Configure Federator runtime properties
  • Verify Management Node configuration
  • Start Federator (server + client)
  • Validate authentication and authorisation
  • Confirm Kafka-to-Kafka federation end-to-end

The steps below ensure the system can be started and validated without trial-and-error.

Pre-requisites

Before starting, confirm:

  • Docker is running
  • Java 21 is active
  • The Federator repository has been built successfully
  • You are working from a Linux filesystem (e.g. ~/src)

Unless otherwise stated, run all commands from the root of the Federator repository.

Step 1 — Set Federator configuration environment variables

Before starting the Federator, set the required configuration paths:

export FEDERATOR_SERVER_PROPERTIES=./src/configs/server.properties  
export FEDERATOR_CLIENT_PROPERTIES=./src/configs/client.properties

These variables are required for runtime configuration.

Verify

echo $FEDERATOR_SERVER_PROPERTIES  
echo $FEDERATOR_CLIENT_PROPERTIES  

ls -l $FEDERATOR_SERVER_PROPERTIES  
ls -l $FEDERATOR_CLIENT_PROPERTIES
Checkpoint
  • Both variables are set
  • Both referenced files exist
  • No empty output
  • No "No such file or directory" errors

Step 2 — Configure required local properties

The default configuration files may contain placeholder values and must be updated for local use.

Examples:

idp.jwks.url=${IDP_JWKS_URL}  
idp.client.id=${IDP_CLIENT_ID}  
idp.keystore.path=${IDP_KEYSTORE_PATH}

For local development, ensure required values are populated:

idp.jwks.url=https://localhost:8443/realms/management-node/protocol/openid-connect/certs  
idp.token.url=https://localhost:8443/realms/management-node/protocol/openid-connect/token  
idp.client.id=FEDERATOR_HEG  
idp.keystore.path=/path/to/client.p12  
idp.keystore.password=changeit  
idp.truststore.path=/path/to/truststore.jks  
idp.truststore.password=changeit

Verify configuration

grep -n "idp\." "$FEDERATOR_SERVER_PROPERTIES"  
grep -n "idp\." "$FEDERATOR_CLIENT_PROPERTIES"

Expected:

  • Required idp.* properties are present
  • Values are populated (not empty)

Check for unresolved placeholders:

grep -n '\${.*}' "$FEDERATOR_SERVER_PROPERTIES"  
grep -n '\${.*}' "$FEDERATOR_CLIENT_PROPERTIES"

Checkpoint

  • No unresolved ${...} placeholders for required settings

Step 3 — Start local dependencies

Start the required services:

  • Identity Provider (Keycloak)
  • Postgres
  • Management-Node
  • Kafka
  • Redis

If needed, pre-pull images:

docker pull redis:latest
docker pull confluentinc/cp-zookeeper:7.5.3
docker pull confluentinc/cp-kafka:7.5.3

Verify

docker compose ps

Checkpoint
  • All required services are running
  • No containers in "Exited" state

Step 4 — Verify Management-Node configuration

Before federation begins, check producer endpoints in the Management-Node database. They must contain:

  • Registered Producers
  • Approved topics
  • Authorisation rules linking Consumers to topics

Access the database (Docker):

Identify the Postgres container:

docker compose ps

Look for the Postgres service (e.g. postgres, mn-db, or similar).

Connect to the database:

docker exec -it <postgres-container-name> psql -U postgres

If a specific database is used:

docker exec -it <postgres-container-name> psql -U postgres -d <database-name>

Inspect producer endpoints

Run:

SELECT name, host, port
FROM mn.producer
ORDER BY name;
Checkpoint
  • Hosts are reachable locally (e.g. localhost)
  • Ports match your local deployment

Update for local development (if required)

If producer endpoints reference external hosts, update them:

UPDATE mn.producer
SET host='localhost', port=9001
WHERE name IN ('HEG-PRODUCER-1', 'BCC-PRODUCER-1');

Verify

Re-run:

SELECT name, host, port
FROM mn.producer
ORDER BY name;
Checkpoint
  • Producer endpoints point to local addresses
  • Values match the running local services

Step 5 — Confirm token and authentication behaviour

Federator validates tokens using the aud claim.

Configured value:

idp.client.id=FEDERATOR_HEG

Step 6 - Verify token contents (critical)

  • Inspect a token using a JWT decoder
  • Confirm:

aud contains FEDERATOR_HEG

If instead you see:

aud = management-node
azp = FEDERATOR_HEG

Then:

  • The token will be rejected by the Producer
  • Authentication will fail

Checkpoint

  • Token aud matches idp.client.id
  • Producer authentication succeeds

Step 7 — Set build metadata

export ARTIFACT_ID="federator"
export VERSION="1.1.0"

Verify

echo $VERSION

Checkpoint
  • Matches version in pom.xml

Step 8 — Start the Federator deployment

docker compose \
  --file docker/docker-compose-multiple-clients-multiple-server.yml \
  up --no-build --pull never

Verify

docker compose ps
docker compose ps -a

Checkpoint

  • All containers running
  • No restart loops

Step 9 — Verify Federator startup

Check logs:

docker logs federator-server --tail 80
docker logs federator-client --tail 80

Checkpoint

  • gRPC server initialised
  • MTLS handshake successful
  • Authorised topics discovered

Step 10 — Verify Redis offset tracking

docker exec -it <redis-container-name> redis-cli

Inside Redis:

keys *

Checkpoint
  • Offset keys present
  • Values update as messages are consumed

Step 11 — Verify Kafka message flow

Check source topic:

docker exec -it <kafka-source-container> \
  kafka-console-consumer \
  --bootstrap-server localhost:<source-port> \
  --topic <source-topic-name> \
  --from-beginning \
  --property print.headers=true

Checkpoint

Messages include a securityLabel header

  • Example: securityLabel=nationality=GBR

Check target topic:

docker exec -it <kafka-target-container> \
  kafka-console-consumer \
  --bootstrap-server localhost:<target-port> \
  --topic federated.<source-topic-name> \
  --from-beginning \
  --property print.headers=true

Checkpoint

  • Only messages with authorised securityLabel values appear
  • Non-matching labels are excluded
  • Message payloads remain unchanged

Step 12 — Verify end-to-end federation

Terminal 1:

docker exec -it kafka-target kafka-console-consumer \
  --bootstrap-server kafka-target:19092 \
  --topic federated-FederatorServer1-knowledge

Terminal 2:

echo "test-$(date +%s)" | docker exec -i kafka-src kafka-console-producer \
  --bootstrap-server kafka-src:19092 \
  --topic knowledge
Checkpoint
  • Message appears in Terminal 1
  • Full pipeline is working

Expected Outcome

A successful setup demonstrates:

  • Federator server and client startup
  • Management-Node integration
  • Secure authentication and MTLS
  • Topic discovery and authorisation
  • Kafka-to-Kafka federation
  • Policy-based filtering
  • Redis-backed replay safety

Troubleshooting

Check these first:

  • Missing FEDERATOR_*_PROPERTIES environment variables
  • Unresolved configuration placeholders
  • Non-local Management-Node producer endpoints
  • JWT aud mismatch
  • Token valid for one service but not another
  • Incorrect Kafka ports

Next Steps