Error: Order is not found/filled | Order is actually Filled | Repeat Multiple Orders | After dYdX Chain outage and subsequent Upgrade

[BradT7]

Hello, I experienced a major bug yesterday after a trade was triggered post dYdX Chain outage and subsequent dYdX Chain 4.0.2 (or 4.0.4) upgrade yesterday. https://status.dydx.trade/
I had multiple Error: Order is not found/filled. Retry again, count: counting all the way up to maximum in terminal.
The problem is all those orders were filled, and placed multiple times! This resulted in being extremely over leveraged from what TV signal size was. Thankfully I noticed it only a couple hours post trade. I checked the dYdX Trade History and confirmed all the same orders that supposedly Errored were actually placed successfully, but multiple times!
I think the logic used to check for a Filled trade has been broken since the Chain upgrade.
I've since modified dydx_v4 createOrder.ts line 34 maxTries to 1 and line 56 await _sleep to 60000. This way only one attempt will be made.
After making the code changes above, I've since restarted with Ctrl-C, and yarn start in terminal, so we wait and see.
I will check back after another signal to see if it shows it a successful dYdX v4 Order ID: xxxxxxxx-... in terminal, like it used to.
Let me know if you or anyone else has experienced this, and check your positions after any recent trades post the dYdX Chain outage.

[junta]

thank you for raising issue

Although I have not experienced this issue since yesterday, but I've modified await _sleep() to 60000 and pushed a new version of code, but keep maxTries 3.
Because I've seen that sometimes the order is not executed if we don't retry again.

I'll monitor if this is more stable configuration

[BradT7]

That restart must have helped, or maybe the validators needed some time to get up to speed/sync again, not sure. I still don't know what in the code would have failed and allowed or triggered all the duplicate orders that happened before, my suspicion is it was a validator failure outside the control of the code to check.

Here's my most recent results:
I tested all 3 validators after my code change for a test, all three triggered at the same time from the same signal (3 different ports and wallets tested)
restEndpoint: https://dydx-ops-rpc.kingnodes.com Error: Order is not found/filled. Retry again, count: 0 and then it worked after second attempt, 60s later, no duplicate orders made.
restEndpoint: https://dydx-rest.publicnode.com Failed no order placed at all, no transaction result in terminal
restEndpoint: https://dydx-mainnet-full-rpc.public.blastapi.io Success after first attempt.

I'm going to test with maxTries 2 and await _sleep 30000 since with 60 seconds it's just seemingly ignored, I don't think it has to do with waiting to get filled, it's like it was never sent at all.
The second attempt, after the 60s sleep, filled around 7 and 13 seconds, but usually it's closer to 4 seconds on average for the fill to take place.
I've also put all the validators back to https://dydx-mainnet-full-rpc.public.blastapi.io because I was having good results with that original one before this recent glitch. (I have no idea if https://dydx-ops-rpc.kingnodes.com is better, but it experienced one retry above. Let me know if you want me to compare both more side by side, or what your results are with both)

I'm going to wait for one more signal and follow up with results, to confirm it's no longer placing multiple trades.
That was a crazy glitch, I had maxTries set to at 8 or 9 before, so that's how many repeat trades it duplicated before, yikes!

[junta]

nice! thank you for the detailed report
I'll try with https://dydx-mainnet-full-rpc.public.blastapi.io again