Support at least once delivery of webhook events

[andresuribe87]

Currently, webhooks are best effort. If delivery fails (because of a crash or because the receiving end is down), then we simply log a message and never try again. This limits the ways in which webhooks can be used.

To support a wider range of use cases, supporting at least once delivery would be useful.

Here's one way in which it can work:

• At the API level, add a deliveryOptions enum. By default, it's best effort, and can be set to at least once.
• When a webhook is configured with at least once, persist the payload that should be sent in a storage. Let this be webhook_messages_outgoing_stream. Each message should have a UUID.
  • Note that the storage interface might need to be extended to support queue (or streaming) operations.
• Have a pool of worker goroutines, which are started during main and periodically listen to webhook_messages_outgoing_stream.
• After a worker successfully delivers (success can be a response HTTP status code of 2xx), mark the webhook_message as sent. For debug purposes, the payload of the response received can be recorded. Take care to make things idempotent.