However in most of the cases the Locale is not relevant (i.e. when only using String format specifier %s).

This is only true if you're converting objects whose toString methods are locale-insensitive, which may not be true in all cases. If you allow tinkering with selective options, you'll soon run into arguments about what to include or exclude, which is a waste of time. I'd say just take it all or add warning suppression annotation where you think your code is valid (or a method that redelegates).

Uwe may disagree but my opinion to this is pretty much consistent with google's java formatter - fewer configurable options lead to less bikeshedding over what to enable/disable, etc.

There are false positives and different approaches how to deal with those. I agree that going through each use case manually and individually mark as safe with the suppress annotation or just use the locale/timezone specific method is the safest approach.
However for large codebases this is often way too much effort. So you might be interested in first checking certain signatures which you know for sure cause issues. The other you can at least temporarily ignore with this option. I don't think that this adds too much complexity as this is opt-in, i.e. nothing changes when you don't configure it explicitly.

Regarding

This is only true if you're converting objects whose toString methods are locale-insensitive, which may not be true in all cases.

I would say that this needs to be fixed in the toString() method instead because even when using the locale specific String.format this is not passed to the called toString() method (as this never takes any arguments).