A Golang project archetype for a Kafka Processor.
What's a Kafka Processor? It's a headless service that consumes messages from an Apache Kafka topic, processes the information in some way and then places the resulting message onto another Kafka topic. In this instance the input message contains a seed number, the processing is generation of a random number using the seed and the resulting message contains the random number. Combining services such as this together is a trait of a Reactive System.
Why Golang? It's richly-featured, relatively simple to learn, produces tiny binaries, and on average consumes far less memory than a JVM application. This final point, in particular, makes running a system of such services less resource intensive and if you're running in the cloud, easier on your bank account.
Golang makes dependency injection relatively straightforward out of the box. This is mainly due to the fact that structs do not have to declare they implement an interface, they simply have to have the correct function signatures that interface requires.
The project has a structure similar to that of a Spring Boot application,
but with the concrete implementations declared in application_context.go in the root of the project.
The application responds on a GET requests to /health with the following:
{
"status" : "ok"
}
This is a very simple health check endpoint which can be used by Docker or container orchestrators, such as Kubernetes to signal that the service is available.
The application sources configuration from a set of environment variables. This compliments running the application in a Docker container rather nicely. For the sake of the archetype, each value has a default value.
| Environment Variable | Purpose | Default Value |
|---|---|---|
KAFKA_PROCESSOR_ARCHETYPE_PORT |
The port to run the HTTP health-check on | 8080 |
KAFKA_PROCESSOR_ARCHETYPE_RANDOM_SEED |
The seed to use when generating a random number | 1 |
KAFKA_PROCESSOR_ARCHETYPE_KAFKA_BROKERS |
The comma separated list of Kafka brokers | localhost:9092 |
KAFKA_PROCESSOR_ARCHETYPE_SOURCE_TOPIC |
The topic to get messages from | source-topic |
KAFKA_PROCESSOR_ARCHETYPE_SINK_TOPIC |
The topic to place messages onto | sink-topic |
This project uses make for building the project. The following make tasks exist:
# Clean built artefacts
make clean
# Build the project
make build
# Run the tests
make test
# Package as a Docker container
make package
In addition to performing a go install, building the project also runs gofmt, govet, gocyclo, golint, ineffassign and misspell. The build does not fail if these tools report an error; it is up to the Developer if they take the advice. The vendor folder is excluded from the analysis.
The testing library of choice for this project is Testify. Testify is a sturdy, assert style framework.
In addition to usint tests, this project has integration tests that run against a real instance of Kafka by using the spotify/kafka Docker image. The kafka instance is spun up fresh for each integration test and then stopped after.
The application can be packaged as a Docker container by using the following command:
make package
The container is built on Alpine Linux and works out to be about 14MB. The container can be run using the following command:
docker run -it -p8080:8080 adbourne/go-archetype-kafa-processor
This project uses Govendor.