Testing Cloud Native Kafka Applications with Testcontainers

Testing in application development is very important. As a developer, you cannot write code without writing tests. You can but you should not.

You can skip writing tests and use debugging to see what you application does each time you write a new code. However, this is both wrong in terms of software engineering and might make your development process less reliable and slower.

There are many discussions around on web about these topics especially on when to debug your applications or when to write tests, but there is one simple constant that you must write tests.

Software development tests consists of many testing strategies that covers different aspects of a software system. These testing strategies are generally represented as a pyramid, which is called testing pyramid that shows the testing types in relation to time, integration, and cost.

In an efficient software development environment, you should start writing tests with small unit tests that covers the code you are writing and makes you validate your code. These kind of tests generally are focused on the low level algorithms and functionality without being related to any integration point such as another service or a middleware such as Apache Kafka (aka. Kafka).

Developers might want to write tests that covers the integration points more but they might avoid using real systems to integrate for testing because of the time and maintenance costs. So they generally mock the integration points and focus on the logic. This ends up with many integration kind of tests written, which are actually unit tests.

Mocking is good in many ways such as you can run your tests in a faster way. However, there are drawbacks. You don’t test the real integration, so there might be unpredictable issues with a message sending to the system. Taking our previous Apache Kafka example, you might want to test your code against a real Kafka instance, or you might be developing a Kafka admin application that interacts with the Kafka configurations. If you mock Apache Kafka, you are not able to apply these tests.

At this point, Testcontainers comes as a saviour.

What is Testcontainers?

Testcontainers is a testing library, which helps developers to use Docker or Podman containers in their integration or UI/Acceptance tests.

Testcontainers makes testing easier for developers for use cases such as:

• A data access layer that integrates with a database
• A middleware or an application service that integrates with your application
• UI or user acceptance tests along with libraries such as Selenium

Testcontainers supports many languages such as Java, .Net, Go, Python, Node.js, Rust, and Haskell.

In this tutorial, you will experience Testcontainers for Java along with the Kafka Containers module. You will:

• Learn how to configure Testcontainers for Apache Kafka in a pure Java and a Spring Boot application.
• Learn how to develop tests against Apache Kafka by using Testcontainers in a pure Java and a Spring Boot application.