Run and Validate

Purpose

This section starts a local deployment and validate federation end-to-end.

Stage 4 - Run and Validate the Deployment

How to complete this stage

You will:

Insert a test user for access control validation
Deploy a test message producer
Send test data through Kafka
Verify federation behaviour
Inspect system components using tooling

4.1 Insert a test user

MONGODB_PASSWORD=$(kubectl get secret mongodb-access-ia-node-user -n org-a -o jsonpath='{.data.password}' | base64 -d)
kubectl exec -n org-a mongodb-0 -c mongod -- mongosh --quiet -u ia-node-user -p "$MONGODB_PASSWORD" --authenticationDatabase access access --eval '
db.users.deleteMany({email: "user@test.com"});
db.users.insertOne({
  externalId: "service-account-ianode",
  name: "Service Account IANode",
  userName: "service-account-ianode",
  email: "user@test.com",
  labels: [
    {
      name: "clearance",
      value: "TS",
      toString: "clearance=\"TS\"",
      toDataLabelString: "classification=\"TS\""
    },
    {
      name: "nationality",
      value: "GBR",
      toString: "nationality=\"GBR\"",
      toDataLabelString: "permitted_nationalities=\"GBR\""
    },
    {
      name: "deployed_organisation",
      value: "ExampleOrg",
      toString: "deployed_organisation=\"ExampleOrg\"",
      toDataLabelString: "permitted_organisations=\"ExampleOrg\""
    },
    {
      name: "personnel_type",
      value: "GOV",
      toString: "personnel_type=\"GOV\"",
      toDataLabelString: null
    }
  ],
  active: true,
  groups: [],
  userGroups: [],
  schemas: ["urn:ietf:params:scim:schemas:core:2.0:User"]
});
'

Confirm that the test user was inserted:

MONGODB_PASSWORD=$(kubectl get secret mongodb-access-ia-node-user -n org-a -o jsonpath='{.data.password}' | base64 -d)
kubectl exec -n org-a mongodb-0 -c mongod -- mongosh --quiet -u ia-node-user -p "$MONGODB_PASSWORD" --authenticationDatabase access access --eval 'db.users.find({email: "user@test.com"}, {userName: 1, email: 1, labels: 1, active: 1}).pretty()'

4.2 Deploy a test-message-pod

Deploy a test-message-pod to test data federation

Build the Docker image:

cd charts/test-message-pod
docker build -t test-message-pod:local .

Install the Helm chart:

helm install test-msg ./charts/test-message-pod -n org-a \
  --set kafka.bootstrapServer=kafka-cluster-kafka-bootstrap.org-a.svc.cluster.local:9092 \
  --set kafka.topic=knowledge \
  --set kafka.securityProtocol=SASL_PLAINTEXT \
  --set kafkaCredentialsSecret.name=kafka-auth-config \
  --set resources.requests.cpu=5m \
  --set resources.requests.memory=16Mi

4.3 Troubleshooting

Check values.yaml and ensure that the image is being pulled from the local image.

image:
  # For local development, build and use a local image:
  # docker build -t test-message-pod:local .
  # For Kind: kind load docker-image test-message-pod:local --name kind
  repository: test-message-pod
  pullPolicy: IfNotPresent
  tag: "local"

Check that the Dockerfile is using the current version of OpenJDK

# Install bash, OpenJDK 21, kcat, and debugging tools
RUN apk add --no-cache \
    bash \
    openjdk21-jre \
    kcat \
    wget \
    ca-certificates \
    openssl \
    jq \
    vim \
    nano \
    net-tools \
    bind-tools \
    iputils \
    tcpdump \
    sudo

4.4 Install the Kafka UI (optional)

Get the Kafka user password from the secret:

KAFKA_PASSWORD=$(kubectl get secret kafka-ia-node-user -n org-a -o jsonpath='{.data.password}' | base64 -d)

Install the Kafka UI:

helm repo add kafka-ui https://provectus.github.io/kafka-ui-charts

helm upgrade --install kafka-ui kafka-ui/kafka-ui \
  -n org-a \
  --reset-values \
  --set yamlApplicationConfig.kafka.clusters[0].name=kind-cluster \
  --set yamlApplicationConfig.kafka.clusters[0].bootstrapServers=kafka-cluster-kafka-bootstrap.org-a.svc.cluster.local:9092 \
  --set-string yamlApplicationConfig.kafka.clusters[0].properties.security\.protocol=SASL_PLAINTEXT \
  --set-string yamlApplicationConfig.kafka.clusters[0].properties.sasl\.mechanism=SCRAM-SHA-512 \
  --set-string yamlApplicationConfig.kafka.clusters[0].properties.sasl\.jaas\.config="org.apache.kafka.common.security.scram.ScramLoginModule required username=\"kafka-ia-node-user\" password=\"${KAFKA_PASSWORD}\";" \
  --set volumeMounts[0].name=kafka-auth-config \
  --set volumeMounts[0].mountPath=/kafka-auth \
  --set volumeMounts[0].readOnly=true \
  --set volumes[0].name=kafka-auth-config \
  --set volumes[0].secret.secretName=kafka-auth-config

4.5 Expose Federator's jobRunr dashboard

kubectl -n org-a port-forward deployment/federator-client 8085:8085

Execute a script to send a test message:

kubectl exec -n org-a test-msg-test-message-pod -- /opt/scripts/send-kafka-message.sh /tmp/test-data.trig

The script send-kafka-message.sh generates (a) a Content Type header, (b) a message, and (c) a command to add a port exclusion traffic.sidecar.istio.io/excludeOutboundPorts=9092 (using --set-string) so the Istio Envoy proxy sidecar does not intercept Kafka's plaintext listener. Without this exclusion, connections terminate during authentication.

kubectl exec -n org-a test-msg-test-message-pod -- /opt/scripts/send-kafka-message.sh /tmp/test-data.trig

4.6 Verify federation behaviour

Ensure that the test-pod is created from the local image:

image:
  # For local development, build and use a local image:
  # docker build -t test-message-pod:local .
  # For Kind: kind load docker-image test-message-pod:local --name kind
  repository: test-message-pod
  pullPolicy: IfNotPresent
  tag: "local"

Checkpoint

At the end of this stage, confirm:

Test user exists in MongoDB
Test message pod is running
Messages are successfully sent to Kafka
Federator components are active
Kafka topics contain expected data
No authentication or runtime errors are present

Your IA Node deployment is now running and validated.