Version 2.9.1

Patch release fixing a bug for Kinesis collector running with SQS buffer, where in case of Kinesis issue, events would get sent to SQS forever, never switching back to Kinesis once available again.

PubSub collector got less verbose in the INFO logs.

CHANGELOG

• Run background check for Kinesis if it is made unhealthy and SQS buffer is activated (#315)
• PubSub: use debug for logging the inserts (#321)

Version 2.9.0

The primary reason for this release is to update the logic for /sink-health endpoint. It also comes with other nice improvements.

What's new ?

Improved logic for /sink-health endpoint (#276)

Until this release, the collector was starting with /sink-health endpoint healthy, until proven unhealthy (which was a problem in case of Kinesis outage for instance). The new logic is to start unhealthy until proven healthy, either by successfully sending events to the queue or by successfully checking the stream with AWS/GCP API, whichever comes first.

More details can be found on the related issue.

Kinesis and SQS: MaxRetries configurable (#295)

MaxRetries is used in 2 scenarios:

• In case of SQS collector or Kinesis collector without SQS buffer, /sink-health becomes unhealthy after this number of failed retries (even though it is retried forever).
• In case of Kinesis collector with SQS buffer, this is the number of retries to Kinesis before switching to SQS.

It was hard coded to 3 and is now configurable (here for Kinesis and here for SQS).

Pub/Sub: Pub/Sub emulator support (#270)

It is now possible to use Pub/Sub emulator (e.g. for testing).

Using it is as simple as running the Docker image and setting PUBSUB_EMULATOR_HOST.

Pub/Sub: second layer of retry (#304)

There is now a second layer of retry for inserting records into Pub/Sub, in addition to the built-in retry mechanism of Pub/Sub publisher. When a record can't be inserted after the built-in retries, it's added to a buffer and every retryInterval, the collector retries to insert the failed inserts.

MaxBytes configurable (#272)

In case a collector payload is too big, a size violation bad row is emitted.

The maximum size for each type of queue was hard coded (1 MB for Kinesis, 192 KB for SQS, 10 MB for Pub/Sub 1 MB for Kafka).

It is now configurable (Kinesis, SQS, Pub/Sub, Kafka).

CHANGELOG

• Bump protobuf-java to 3.21.7 (#308)
• PubSub: add second layer of retry (#304)
• Replace set-output in Github Actions (#305)
• Make MaxRetries configurable (#295)
• Update /sink-health endpoint logic (#276)
• Integration tests should work with both regular and distroless project (#301)
• Scan Docker images with Snyk container monitor in deploy.yml (#296)
• Add integration tests for Set-Cookie (#287)
• Add integration test for doNotTrackCookie (#289)
• Add integration test for X-Forwarded-For (#288)
• Add integration test for custom paths (#286)
• Add integration test for /health endpoint (#285)
• Kinesis: add integration tests (#283)
• Validate cookie.fallbackDomain config option on startup (#278)
• PubSub: add integration tests (#274)
• PubSub: make it possible to use PubSub emulator (#270)
• Put MaxBytes in default application.conf instead of hard coding it (#272)

Version 2.8.2

A maintenance release to bump the base docker image to a newer version

2.8.1

This patch release adds improvements for the collector's warmup feature which we added in the 2.7.0 release. The warmup feature is still considered experimental, and is off by default.

• Bump aws sdk to 1.12.327 (#266)
• Warmup process should iterate until success (#264)
• Health endpoint should return 503 during warmup phase (#263)

2.8.0

This release adds a new experimental asset that can write collector payloads to RabbitMQ.

Setup guide can be found here and configuration reference here.

Changelog

• Add RabbitMQ asset (#251)

2.7.1

This patch release is simply to generate new docker images that bring in sufficiently recent version of zlib1g, see here.

Changelog

• Ensure docker image has latest zlib1g version #254

2.7.0

This release adds an experimental feature to send some "warm-up" requests to the collector's own /health endpoint when the collector first starts up. We have found from experiment this can cut down the number of 502s returned from a load balancer in front of the collector in Kubernetes deployments. More details in #249.

The new feature is turned off by default, and can be enabled by setting collector.experimental.warmup.enable = true in the config file. See the example config files for a demonstration of where to put this configuration option.

Changelog

• Send warmup requests to self on startup (#249)

2.6.3

This patch release is simply to generate new docker images that bring in sufficiently recent version of libfreetype6, see here.

Changelog

• Ensure docker image has latest libfreetype6 version (#247)

2.6.2

A patch release to create new docker images, addressing a security vulnerability in open ssl

Changelog

• Ensure docker image has openssl version >= 1.1.1n-0+deb11u3 (#244)

2.6.1

A patch release to address security vulnerabilities in imported dependencies

Changelog

• Reduce number of error messages in kinesis collector logs (#242)
• Bump log4j to 2.17.2 (#241)
• Remove sbt-dependency-graph from plugins.sbt (#235)
• Bump jackson-databind to 2.12.7 (#240)
• Bump aws-java-sdk to 1.12.238 (#239)
• Bump google-cloud-pubsub to 1.119.1 (#238)
• Bump jnr-unixsocket to 0.38.17 (#237)
• Bump akka-http-metrics-datadog to 1.7.1 (#236)