Extract broker syncer to it's own service

[mkolesnik]

Epic Description

We need a service dedicated to syncing the broker information from the local cluster to the broker, and the broker info to the local cluster.

We have seen on scale testing that having the broker syncer inside the gateway causes issues to connectivity when there's a high enough scale.

Additionally, multiple projects (Submariner, Lighthouse) are using a specific broker syncer to sync over their resources.
From an architectural POV it would be simpler if there was a "broker syncer" service which can sync resources to/from the remote broker, and the local services would only consume local resources.
This decentralizes the information further, and decouples the services allowing the local services to easily keep working even if the broker is temporarily down.
In addition, this opens up possibilities of easy broker HA since the data can be easily replicated.

This can also benefit Axon as it decouples the control plane information dissemination from Submariner, allowing alternative methods to be used.

Acceptance Criteria

• Submariner deployments work without any need for manual intervention
• Upgrade from older Submariner deployments works seamlessly without any need for manual intervention
• Lighthouse is supported
• Globalnet is supported

Definition of Done (Checklist)

• Code complete
• Relevant metrics added
• The acceptance criteria met
• Unit/e2e test added & pass
• CI jobs pass
• Deployed using cloud-prepare+subctl
• Deployed using ACM/OCM addon
• Deploy using Helm
• Deployed on supported platforms (for e.g kind, OCP on AWS, OCP on GCP)
• Run subctl verify, diagnose and gather
• Uninstall
• Troubleshooting (gather/diagnose) added
• Documentation added
• Release notes added

Work Items

• Add new submariner-brokersyncer repository in Quay.io
• Add new Dockerfile to build submariner-brokersyncer in submariner repository (Would admiral be a better place?)
• Implement generic broker syncer code (might need to break down to smaller tasks)
• Implement E2E tests for the new broker syncer service
• Add new broker syncer service to operator deployment logic
• Add new broker syncer service to subctl diagnose
• Add new broker syncer service to subctl gather
• Add new broker syncer service to subctl uninstall
• Remove broker syncing logic from submariner
• Remove broker syncing logic from lighthouse
• Add submariner-brokersyncer image to releases process
• Document the new service on the website

[stale]

This issue has been automatically marked as stale because it has not had activity for 60 days. It will be closed if no further activity occurs. Please make a comment if this issue/pr is still valid. Thank you for your contributions.

[tpantelis]

bump

[stale]

This issue has been automatically marked as stale because it has not had activity for 60 days. It will be closed if no further activity occurs. Please make a comment if this issue/pr is still valid. Thank you for your contributions.