Problems with notifications

[hell-g]

Hey guys,

I am a happy user of docker-volume-backup and the backups work very well. But unfortunately, I am struggling with the notifications. I have a local Gotify server running which receives the messages from several services. I just cannot figure out why it does not work reliably with docker-volume-backup. The strange thing is that on some days, I do receive a message on Gotify and on some days I do not. With other services, it is working flawlessly and I get all the messages. Whenever I try to debug it and run the backup command inside the Docker container shell, I am not successful in getting any messages on Gotify and get this output in the console:

time=2023-10-28T17:05:54.026+02:00 level=ERROR msg="Backup procedure ran successfully, but an error ocurred calling the registered hooks: runHooks: error running hook: notify: error notifying: sendNotification: error sending message: failed to send notification to Gotify: got HTTP 404 Not Found"

This is my ENV variable that I use to configure docker-volume-backup (I am using an Ansible role to start the Docker container):

NOTIFICATION_URLS: gotify://gotify.mydomain:myport/gotifytoken/?title=Docker-Volume-Backup-Test&priority=5

If there would be something wrong with the URL or the token, I would expect it to never work, but as written above, I do get the message on some days. I do have NOTIFICATION_LEVEL: "info" while figuring out this problem. In the logs of the Docker container I do not see any messages, just in the console, when I run the backup manually.

Do you have any idea what I am doing wrong or how I can debug this problem further?

Thanks a lot!

[m90]

Seeing that your error message contains got HTTP 404 Not Found, which means this is not networking, this can be one of two things:

• there is an issue in Gotify that intermittently makes the requested route 404
• there is an issue in shoutrrr (the library used for sending notifications) that under certain conditions tries to call a non-existent URL

Does Gotify log the requests made against it so you can compare what is being requested when successful, and compare it to what is being requested when failing?

[hell-g]

Thank you very much for your response and the hints to what it could be!

I think, I found the issue with Gotify while looking at the Gotify output and performing manual backups with docker-volume-backup. It looks to me like it is a timing issue. I added the label docker-volume-backup.stop-during-backup to the Gotify container. So the Gotify service gets stopped while performing the backup and then it can be that docker-volume-backup already sends the notification before Gotify is back up. It is behind Traefik and that service also gets stopped for the backup and always takes a moment to set up the routing.

So my ideas are now that either I would need a little timeout in docker-volume-backup before the notification is sent, or Gotify should not be stopped for performing the backup. I think, I prefer the first one, as stopping services to create a backup is good practice in my opinion.
Any suggestions from your side?

[m90]

That's a tricky situation. In an ideal world I would guess you deploy an additional Gotfiy server that sends notifications for backing up your Gotify server, but maybe that's not very practical.

Real world solutions I can think of right now:

• Add a sleep 60 as a post-archive command and hope it gives your Gotify server enough time to start up. This could still cause races of course
• Back up your Gotify database without stopping the container. I don't know how busy your install is, so I can't really tell if this is ok to do. Depending on your type of database you could also backup from a copy/dump, which you can create without interrupting the database availability
• Send notifications for this very backup run using a different notification channel, e.g. SMTP or similar

[hell-g]

Thanks again!
To start with, I would like to fix this with a sleep 60 as an archive-post command.
This brings me to a second issue that I have been having. I have been playing around with these custom commands you can run during the backup. I would like to send a ping to healthchecks.io towards the end of the backup to keep an eye on the regular execution and be notified, if it did not run for whatever reason. I have not managed to get this working yet, but the above notification problem with Gotify seemed more important. But now I am facing the same problems with my sleep command as with the ping to healthchecks.io.
I added this label to my docker-volume-backup container: docker-volume-backup.archive-post: sleep 20 (I used different versions of quotation to make sure it is not an Ansible problem I am introducing here.) I also tried docker-volume-backup.archive-post: /bin/sh -c 'sleep 20' with the same result. This leads to the following output while performing the backup:

time=2023-10-29T14:47:12.461+01:00 level=INFO msg="Running docker-volume-backup.archive-post command sleep 20 for container docker-volume-backup"
time=2023-10-29T14:47:12.462+01:00 level=ERROR msg="Fatal error running backup: runLabeledCommands: error from errgroup: runLabeledCommands: error executing command: exec: error demultiplexing output: %!w(<nil>)"

Also with this issue, I was not able to find a solution and was hoping, someone can have a look and can give me a hint where to look.

[m90]

Using healthchecks.io has been discussed here #159

[hell-g]

Thanks for the hint and the link. This is more or less exactly how I have been trying to do it. But I am unsuccessful with any custom commands so far.
It does not get much simpler than the sleep command but still the execution of the command triggers the above error. Do you have an idea where this demultiplexing error for the output is coming from?

[m90]

Never seen this, no. If you update to v2.33.1, you might see a more helpful error message though.