Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix alerts duplication in HA cluster mode #4153

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

nip-was-here
Copy link

#4152

There is fix for duplicates in HA cluster mode & reduce for broadcasting several times exactly same state.

!cluster.OversizedMessage(b) is not necessary here:

On current release version (0.27.0)

To sum up: at step Choose way for send data in func there's same check with choosing way of sending data - TCP or UDP

For testing this fix, stand was created:

  • Several vmalert instances send alerts to 2 HA clusters of Alertmanager - with patch & without it
  • Each Alertmanager cluster know only about his own nodes
  • Both clusters of Alertmanager send to one self-developed receiver
  • Receiver calculates duplicates of alerts with delta < 1 minute and other stats

Results of tests with big by body incoming alerts:

{
  "main": {
    "alerts_last_minute": 2,
    "alerts_total": 10864,
    "duplicates_total": 2887
  },
  "patched": {
    "alerts_last_minute": 2,
    "alerts_total": 7579,
    "duplicates_total": 0
  }
}

@MarcWort
Copy link

I stumbled across this fix because we were sometimes duplicating notifications in Slack. Are there any plans to review this change?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants