Federator Troubleshooting

This section provides operational diagnostics and recovery steps for common Federator deployment issues.

Use this page when:

Containers fail to start
Federation does not begin
Kafka topics are missing
Access control configuration fails
Docker networking issues occur

This section assumes you have completed Build and Run & Validate.

Container Startup Issues

docker compose ps Shows Nothing Running

docker compose ps shows only running containers.

If it appears empty, check for exited containers:

docker compose ps -a

To inspect why a container exited:

docker logs <container-name>

Container Crash Loop

If a container repeatedly restarts:

docker inspect <container-name> --format 'RestartCount={{.RestartCount}}'

If RestartCount increases, the container is crash-looping.

Inspect logs to identify the root cause.

Container Name Already in Use

If you see:

container name "/zookeeper-src" is already in use

This indicates stale containers from a previous run.

Remove them:

docker rm -f zookeeper-src zookeeper-target kafka-src kafka-target redis

Or reset the stack:

docker compose down

Then restart the deployment.

Kafka Startup & Connectivity Issues

LEADER_NOT_AVAILABLE During Startup

You may see:

LEADER_NOT_AVAILABLE

This commonly appears during:

Broker initialisation
Topic creation
Leader election

It should resolve within 10–30 seconds.

If it persists:

Confirm correct Kafka ports
Check broker logs
Confirm topics exist

Incorrect Kafka Port

In this deployment:

kafka-src listens on 9092
kafka-target listens on 9093

Produce to source:

--bootstrap-server kafka-src:9092

Consume from target:

--bootstrap-server kafka-target:9093

If you use the wrong port, you may see:

Connection to node -1 could not be established
Broker may not be available

This usually indicates an incorrect port, not a broker crash.

Federated Topics Not Appearing

List topics on the target broker:

docker exec -it kafka-target \
  kafka-topics --bootstrap-server kafka-target:9093 --list

If no federated.* topics appear:

Confirm Consumers are running
Confirm Producers discovered source topics
Confirm authorisation rules permit access
Confirm filtering is not excluding all messages

Federator Server Configuration Errors

Missing server.accessMapValueFile

If the server exits with:

Missing property: 'server.accessMapValueFile'

Ensure server.properties contains:

server.accessMapValueFile=/config/<access-file>.json

The server will not start without this property.

Access Map Must Be JSON (Not JAR)

If you see:

JsonParseException: Unrecognized token 'PK'

You likely configured:

server.accessMapValueFile=/library/some-file.jar

.jar files are ZIP archives (begin with PK), not JSON.

Ensure:

The access map is a .json file
The JSON file exists
The path points to the JSON file

Correct example:

server.accessMapValueFile=/config/access.json

server.properties Formatting Errors

Ensure each property appears on its own line.

Incorrect:

javax.net.ssl.trustStorePassword=changeitserver.accessMapValueFile=/config/access.json

Correct:

javax.net.ssl.trustStorePassword=changeit
server.accessMapValueFile=/config/access.json

Improper formatting can prevent startup.

Access Map & Volume Mount Issues

Access Map File Not Found

If you see:

FileNotFoundException: /config/access.json

Verify the file is mounted into the container:

docker inspect federator-server --format '{{range .Mounts}}{{println .Source "->" .Destination}}{{end}}'

Confirm:

The JSON file appears in the mount list
The container path matches server.accessMapValueFile

If not mounted correctly:

Check Docker Compose volume configuration
Confirm the file exists on the host

Network Resource Still in Use

If you see:

Network core Resource is still in use

Check which containers are attached:

docker ps -a --filter network=core

Remove them:

docker rm -f <container-name>

Then remove the network:

docker network rm core

Diagnostic Commands (Quick Reference)

List running containers

docker compose ps
docker compose ps -a

View container logs

docker logs <container-name>

Check restart count

docker inspect <container-name> --format 'RestartCount={{.RestartCount}}'

List federated topics

docker exec -it kafka-target \
  kafka-topics --bootstrap-server kafka-target:9093 --list

When Federation Does Not Begin

If Producers and Consumers are running but no federation occurs, check:

Producer logs
Topics discovered
Authorisation rules loaded
Consumer logs
MTLS handshake successful
Topics authorised
Kafka
Source topics contain messages
Target broker reachable on correct port
Filtering
securityLabel headers present
Labels match authorisation policy

Federation fails closed — if any trust, policy, or identity validation fails, no data crosses the boundary.

Authentication & Configuration Issues

Missing Federator Configuration Environment Variables

If the Federator server or client fails to start, or exits immediately without clear logs, the required configuration environment variables may not be set.

Verify:

echo $FEDERATOR_SERVER_PROPERTIES
echo $FEDERATOR_CLIENT_PROPERTIES

If unset, define them:
``` sh
export FEDERATOR_SERVER_PROPERTIES=./src/configs/server.properties
export FEDERATOR_CLIENT_PROPERTIES=./src/configs/client.properties

JWT Audience Validation Failure

If you see:

UNAUTHENTICATED: Token audience does not include required client id

What this means

The Federator Producer validates tokens using the aud (audience) claim.

The token does not contain the expected client ID in the aud field.

This is commonly caused by a mismatch between Federator configuration and Identity Provider token structure.

Common cause

In default Keycloak configurations:

aud = management-node
azp = FEDERATOR_HEG'

Federator expects:

aud contains FEDERATOR_HEG'

As a result, the token is rejected even though it is otherwise valid.

How to verify

Decode the access token (e.g. using jwt.io or a CLI tool) and inspect:

aud (audience)
azp (authorised party)

Confirm whether the client ID appears in aud.

Resolution

One of the following must be implemented:

Configure the Identity Provider (Keycloak) to include the client ID in the aud claim
Issue tokens with an audience that matches idp.client.id
Modify Federator validation logic to accept the azp claim instead of (or in addition to) aud

Ensure alignment before restarting the Federator services.

Token Valid for One Service but Not Another

Symptom

Management-Node API calls succeed
Federator Producer authentication fails
Federation does not begin

What this means

The same access token is being used across multiple services with different audience (aud) expectations.

Cause

Different services validate different audience values.

For example:

Management-Node may accept: aud = management-node
Federator Producer may require: aud = FEDERATOR_HEG

A token valid for one service may be rejected by another.

Resolution

Ensure one of the following:

The token includes all required audiences for the services it is used with
OR
Separate tokens are issued per service, each with the correct audience

Verification

Decode the token and inspect the aud claim
Confirm it satisfies all target services
Re-test authentication after updating token configuration

Mental Model (Simplified)

Understanding the data flow helps isolate failures:

kafka-src (9092)
        ↓
federator-server (gRPC + filtering)
        ↓
federator-client
        ↓
kafka-target (9093, federated.* topics)

If messages are not appearing on kafka-target, the issue lies in one of these layers.

If issues persist:

Rebuild artefacts (./mvnw clean install)
Rebuild Docker images
Restart the stack
Review logs from startup