[BUG] Sandbox order path blocks on synchronous position-feed WebSocket subscription

[SmartAlgo2024]

Bug: Sandbox order path blocks on synchronous position-feed WebSocket subscription

Type: Bug (performance / blocking)
Priority: High (order-path latency + failure coupling)
Component: sandbox/websocket_execution_engine.py
Version: 2.0.2.5

Summary

The sandbox's six position-feed hooks run the symbol subscribe / unsubscribe
WebSocket round-trip synchronously inline in the order-execution path. This
stalls the order ack while the gateway↔broker WS subscription completes, and
remains a latency/failure point even after the v2.0.2.2 client fix
(_run_on_loop in services/websocket_client.py).

Affected call sites (anchors on 2.0.2.5)

Hooks	Call made inline
notify_order_placed (~line 251)	self._subscribe_ws_symbols(subscribe_user, [(symbol, exchange)]) (~line 273)
notify_order_completed (~line 275)	self._unsubscribe_ws_symbols(unsubscribe_user, [(symbol, exchange)]) (~line 302)
notify_position_opened (~line 304)	self._subscribe_ws_symbols(user_id, [(symbol, exchange)]) (~line 322)
notify_position_closed (~line 324)	self._unsubscribe_ws_symbols(user_id, [(symbol, exchange)]) (~line 357)
notify_gtt_completed (~line 470)	self._unsubscribe_ws_symbols(unsubscribe_user, [(symbol, exchange)]) (~line 468)
notify_gtt_placed (~line 471)	self._subscribe_ws_symbols(subscribe_user, [(symbol, exchange)]) (~line 494)

_subscribe_ws_symbols / _unsubscribe_ws_symbols are defined at ~lines 663/695.

Why it blocks

These hooks fire on every order/position/GTT transition to maintain the live
MTM position feed. Each call awaits the client's subscribe round-trip (which
itself can wait on the broker WS). Running that inline means order placement /
completion is coupled to broker feed connectivity — a slow or dead socket
delays order acks well beyond the network round-trip (historically ~12 s before
the client fix): a latency and availability failure in the money path.

Expected behavior

Fire-and-forget position-feed subscription so notify_* returns immediately;
whether the MTM feed updates a moment later must not block the order.

Suggested fix

Add an offload helper and route the six call sites through it:

def _offload(self, target, *args):
    if eventlet is not None:
        eventlet.spawn_n(target, *args)          # worker hub greenlet
    else:
        threading.Thread(target=target, args=args, daemon=True).start()

> #### Example:
> self._offload(self._subscribe_ws_symbols, subscribe_user, [(symbol, exchange)])
> self._offload(self._unsubscribe_ws_symbols, unsubscribe_user, [(symbol, exchange)])
> 
> The subscription set is re-sent on (re)connect anyway, so no state is lost if an offloaded call races a disconnect.
> 
> Impact
> notify_order_placed / notify_order_completed block the sandbox order flow on broker feed I/O.
> Under gunicorn a stuck client wait can stall the worker the order handler runs on, delaying unrelated traffic on the same worker.
>