open
https://gitlab.synchro.net/main/sbbs/-/work_items/1289
## Symptom
The IRCd's CPU use climbs steadily for as long as it runs, until the single-threaded callback engine can no longer answer PINGs in time and every linked server drops it: a network-wide netsplit.
Seen on cvs.synchro.net on 2026-10-16: after about 5.5 days of uptime the `jsexec ircd.js` main thread was pinned at 100% of a core (about 72 hours of CPU over the process lifetime). Starting at 21:19:58 roughly 40 server links dropped within four minutes (`Ping Timeout (60 seconds)` and
`Connection reset by peer`), new clients got no reply at all within 25
seconds, and auto-reconnects kept failing because each netburst could not be processed in time. A restart cleared it immediately.
The IRCd never logged its own `Callback engine stalled` warning (the watchdog from #1245): its timers kept firing, it just could not keep up with socket traffic.
## Measurements
After the restart, sampled every 10 minutes:
| time | CPU ticks / 10 min | share of a core | RSS | inbound conns | |---|---|---|---|---|
| 21:46 (fresh) | 121 | 2% | 29 MB | 9 |
| 23:46 | 428 | 7% | 47 MB | 9 |
| 03:46 | 1047 | 17% | 91 MB | 16 |
| 08:16 | 1213 | 20% | 118 MB | 18 |
| 12:46 | 1423 | 24% | 148 MB | 18 |
CPU kept rising while the connection count held flat at 17-18. Linear extrapolation put the next collapse 4-5 days out, matching the previous one.
A gdb stack of the pinned process showed the main thread in `js_handle_events()` (js_internal.cpp), walking the event list. A gdb script walking the jsexec `cb` event list, about 2.5 hours apart:
```
13:13 events total: 127 (TIMEOUT 82, SOCKET_READABLE 23, INTERVAL 22) next_eid=31912043
15:48 events total: 138 (TIMEOUT 85, SOCKET_READABLE 27, INTERVAL 26) next_eid=39583917
```
The list stays short, but `next_eid` advanced about 7.7 million in 2h35m:
about 825 new events per second, up from about 550/s averaged over the first
16 hours. With only around 10 legitimate pending timeouts, the rest are
about 70 timers each re-arming every 100 ms.
## Cause
`Queue_Send()` (exec/load/ircd/core.js) re-arms itself with `js.setTimeout(Process_Sendq, 100, this)` while a socket won't accept more data. `User_Quit()` (exec/load/ircd/user.js) closes a local user's socket **without purging its send queue**. The server and unregistered-client quit paths both purge theirs.
`Socket.send()` on a closed socket fails without throwing (it returns `undefined`), so `sent > 0` is false and the drain re-arms the 100 ms retry again, forever. The queue never grows past its SendQ limit, because nothing
new is added to it.
So every local user who disconnects with output still queued leaves a
permanent 10 Hz timer behind. That is exactly the case for a ping timeout on
a client that stopped reading. Each orphan costs a full pass of the event
loop ten times a second, and they accumulate until the engine falls behind.
Other hubs on the network running this IRCd are likely accumulating the same orphans; several were broadcasting `Server stalled for N seconds` notices the same day, and eotlbbs was flapping (linking and ping-timing-out every few minutes) after cvs was restarted.
## Fix
- `Queue_Send()`: if the socket is no longer connected, purge the queue and
return instead of re-arming. This covers every path that closes a
connection with output pending.
- `User_Quit()`: purge the local user's recvq and sendq before closing, as
the server quit path already does.
- New unit test `exec/tests/ircd/sendq.js`: a stalled-but-connected socket
still gets the 100 ms back-off, a partial send continues immediately, and
output queued to (or left on) a closed socket is dropped without
re-arming. The two closed-socket cases fail without the fix.
Unrelated, noticed while running the suite: `exec/tests/ircd/connect.js`
"STATS l returns 211" fails because 39eda0071c (heater-29-magic, 2026-08-01) made STATS l oper-only; the test still expects it from a non-oper.
-- *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.38-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)