Kafka, Schema Registry, JUnit and Test Containers — Part I

2023-11-13 584 words 3 minutes

Contents

Introduction

In this small series of articles, I will share some ideas on using JUnit and Test Containers to test a Kafka application. The use of test containers fits more into the integration test category than the unit test one, but I am not going to dive into that discussion. For me, tests help me develop software faster, be more focused and get faster feedback to know if I am going in the right direction or not. As a side note, yes, I believe and practice in Test Driven Development, but even if you write your tests after, I hope the ideas shared in this and the following articles are useful to you too.

Problem at hand

We are developing a Kafka based application, be it just using Kafka as a communication mechanism or a Kafka Streams Application. We want to be able to test our code quickly without having to manually set up a broker or a schema registry and then take it down every time you have a bug in the code that needs fixing. Test Containers provide a good mechanism to start containers inside our tests, so let’s leverage that project and make our lives simpler.

Using Test Containers

Testcontainers is an open source framework for providing throwaway, lightweight instances of databases, message brokers, web browsers, or just about anything that can run in a Docker container.

from Test Containers

We will be leveraging test containers already existing support for Kafka, but we will be extending it to have a schema registry.

The idea is to start two containers: a Kafka broker and a schema registry. Why only one Kafka broker? We are simply testing our application, not Kafka. Why a schema registry? We want to follow good practices and have a schema for our messages to be published and controlled.

Starting a Kafka container is already documented in the test container’s documentation. Starting a Kafka container with a schema registry requires some more work, as there is still no official support for a schema registry test container.

To get both containers communicating, we need to create a docker network and connect both containers to it. Luckily, the test containers project already has support for programmatically creating docker networks and adding containers to them.

Creating a docker network using the API from Test Containers is relatively simple:

        
@Testcontainers
class KafkaTest {

  private static final Network NETWORK = Network.newNetwork();

Now we can start a Kafka Broker test container on this network:

        
@Container
private static final KafkaContainer KAFKA_CONTAINER =
  new KafkaContainer(DockerImageName.parse("confluentinc/cp-kafka:7.5.2"))
    .withNetwork(NETWORK);

To get the schema registry to work, we will leverage Test Container’s Generic Container API:

        
        
        
    
@Container
private static final GenericContainer<?> SCHEMA_REGISTRY =
  new GenericContainer<>(DockerImageName.parse("confluentinc/cp-schema-registry:7.5.2"))
    .withNetwork(NETWORK)
    .withExposedPorts(8081)
    .withEnv("SCHEMA_REGISTRY_HOST_NAME", "schema-registry")
    .withEnv("SCHEMA_REGISTRY_LISTENERS", "http://0.0.0.0:8081")
    .withEnv("SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS",
      "PLAINTEXT://" + KAFKA_CONTAINER.getNetworkAliases().get(0) + ":9092")
    .waitingFor(Wait.forHttp("/subjects").forStatusCode(200));

Now that we have both containers running, we just need to set up the proper Kafka settings to use when starting our Kafka applications.

        
Properties kafkaSettings = new Properties();
kafkaSettings.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,
  KAFKA_CONTAINER.getBootstrapServers());
kafkaSettings.put("schema.registry.url",
  "http://" + SCHEMA_REGISTRY.getHost() +
    ":" + SCHEMA_REGISTRY.getFirstMappedPort());

// Remaining KAFKA Settings

Next Steps

This should give you the bare bones to get started with making integration tests with JUnit, Test Containers, Kafka and the Schema Registry. In the second part of this series, I will show how we can make a JUnit5 Extension that makes all this a lot simpler.

Articles in the series: