Make some config adjustable during runtime

[lukerosser]

When the server is under load, as an example, it'd be preferable to be able to set some of the config values without restarting the process as restarting the process causes a load spike in itself while caches are rebuilt. It's also quite time consuming when trying to find the right values for configs to have to restart the whole process and wait for indexes and services to fully start across multiple orgs.

I know this won't be possible for all of the config but some of them seem isolated enough to allow it, for example Frontend.Resources.NotificationsPerSecond is only used by hunt managers so if this was updated the hunt managers could be restarted/updated.

It'd be great to hear your thoughts on if this is an option

[scudette]

I think you bring up some important issues. It's just actually impossible to modify the config during run time because we use it as read only object between multiple threads. So if we need to allow modification we need to lock it heavily and this is a huge undertaking for limited benefit.

The bigger question is, why do we need to modify the config at runtime? If there are reasons to change specific parameters we can make them dynamic in the data store.

The other issue you are raising is actually more interesting - it takes some time for the server to stabilise after startup while caches are filled etc.

This leads to a spike in load which is not desirable at startup. We can solve this by limiting the rate of caches filling up (for example limiting the rate of RSA operations).

[lukerosser]

If the load at startup was mitigated then being able to update the config during runtime would be more about convenience. Particularly when trying to tune performance as seeing the impact of a config change in realtime is a lot clearer than having to restart and wait for the load to return.

This leads to a spike in load which is not desirable at startup. We can solve this by limiting the rate of caches filling up (for example limiting the rate of RSA operations).

A configurable launch delay might be useful too. So client connections are blocked for X amount of time on launch to allow time for caches to build etc.

[scudette]

I think we can isolate those settings that make sense to vary at runtime and make them dynamic - it is not really possible to make the entire configuration dynamic (otherwise we need some fragile logic like if the frontend cert changes we need to know to restart the frontend service etc).

It doesn't really make sense to configure a startup delay because the clients are the ones filling in the caches - so without clients the caches are not filled in. But maybe by rate limiting some operations we can push back against clients connecting so they ramp up slowly.

[lukerosser]

That makes sense and both sound like good options, thanks!