* fix(gateway): drain in-flight runs before closing checkpointer on shutdown
Chat runs execute in fire-and-forget background asyncio tasks that write
checkpoints through a shared checkpointer. On shutdown, langgraph_runtime's
AsyncExitStack tore down the checkpointer's postgres connection pool while
those run tasks were still mid-graph. langgraph's
AsyncPregelLoop._checkpointer_put_after_previous then ran its
`finally: await checkpointer.aput(...)` against the closed pool, raising
psycopg_pool.PoolClosed. Because that put runs in a langgraph-internal task
(not on run_agent's call stack), run_agent's try/except cannot catch it and it
surfaces as "unhandled exception during asyncio.run() shutdown".
Add RunManager.shutdown() to cancel and bounded-await all in-flight runs, and
call it from langgraph_runtime BEFORE the AsyncExitStack closes the
checkpointer, so the final checkpoint write lands while the pool is still open.
The drain is bounded by a timeout so a stuck run cannot hang worker shutdown,
and is shielded so a second shutdown signal cannot abandon it mid-drain and
reopen the race.
Closes#3373
* fix(gateway): address review — preserve completed-run status, bound drain persistence
Addresses Copilot review on #3381:
- RunManager.shutdown(): decide run status AFTER the drain. Under the lock it
now only requests cancellation; after asyncio.wait it marks/persists
`interrupted` only for runs still pending or ended cancelled. A run that
completes (e.g. `success`) during the drain window keeps its real terminal
status instead of being unconditionally overwritten.
- Bound the trailing status persistence within the timeout budget
(deadline = loop.time()+timeout; gather wrapped in asyncio.wait_for) so a slow
store backing off under DB pressure cannot push shutdown past the deadline.
- deps: use asyncio.create_task instead of asyncio.ensure_future.
- tests: wait deterministically for the run to be in-flight (poll the first
checkpoint) instead of a fixed sleep; init shutdown_calls explicitly in the
recovery test double; add regression test asserting a run completing during
the drain keeps its status (in memory and in the store).
* fix(gateway): address maintainer review — surface failed drain persists, clarify timeout constant
Addresses @WillemJiang review on #3381:
- shutdown(): inspect the gather result of the trailing interrupted-status
persistence. _persist_status is best-effort (it catches + logs its own
failure with exc_info and returns False, so it never raises out of the
gather), but the aggregate result was never checked — a partial failure had
no shutdown-level visibility. Now any escaped Exception is logged, and any
False (a persist that did not confirm) is logged with the run_id. Added
regression test test_shutdown_surfaces_failed_interrupted_persist.
- deps: clarify the _RUN_DRAIN_TIMEOUT_SECONDS comment — state the actual value
of _SHUTDOWN_HOOK_TIMEOUT_SECONDS (5.0s) and that both count toward the
lifespan shutdown window. Kept as two separate constants (independent teardown
steps that may diverge) rather than one shared "must match" value.
- Verified no other test fake needs the shutdown stub: _FakeRunManager in
test_worker_langfuse_metadata.py is a run_agent() argument (worker path),
never injected into langgraph_runtime, so it never receives shutdown().
268fdd6968