pull messages out of mempool rather than pushing messages into engine

[aditiharini]

Pull messages out of the mempool and run validations at that point so all messages fed to the engine are likely to be valid.

[sanjayprabhu]

Selecting directly here is dangerous, the mempool rx channel and overload the messages_request_rx channel, and make it so the request is never fulfilled if we get sufficient message traffic to the mempool.

It's better to loop using a tick and ensure that the messages request channel is processed first and always prioritized.

See

snapchain/src/consensus/proposer.rs

Lines 284 to 288 in 41f1e05

	let deadline = Instant::now() + timeout;
	loop {
	let timeout = time::sleep_until(deadline);
	select! {
	_ = poll_interval.tick() => {

for an example of how we're done this elsewhere.

[aditiharini]

I made a change to specify biased in the select macro. This causes the macro to preference branches in order of specification. I think this is what we want here?

In this case, I don't really think we need a timeout because we will just poll indefinitely.