irkerd: Convert from threading to asyncio for juggling connections
This is a fairly extensive restructuring, but I think a
single-threaded, asynchronous framework is easier to debug than the
previous multi-threaded implementatation with it's extensive locking.
The new implementation uses locks for anti-flooding
(Channel._send_message) and new-connection channel joins
(Dispatcher._join_channels) which are in separate coroutines. This
allows out-of-band communication (e.g. PING/PONG exchanges) to occur
independently, so you won't time out on a PONG response because you're
locked sending some large, multi-line message.
Anti-flood protection sleeps use a per-connection (IRCProtcol) lock,
because the IRC server only cares about connection-level spamming, not
channel-level spamming. We need the outer channel-level lock to avoid
interleaving multiline messages within a single channel. Other than
this anti-flood locking, communication for separate channels is
independent, because Channel._send_message coroutines are always
launched in non-blocking Tasks from IRCProtcol.send_message (after
we've taken care of the synchronous _join error handling and such).
The parallel Channel._send_message coroutines function as an outgoing
message queue, with parallel contention for the per-channel
interleaving locks and per-connection anti-flood locks.
Channel._send_message coroutines are reaped by
Channel._reap_message_task, which catches and logs errors, re-queuing
the message for possible follow-up attempts. This allows us to avoid
sending messages after we've been kicked from a channel or had the
connection dropped, but we can keep the channel (and re-queued
messages) and try to resend them after we rejoin the channel. We
don't try to rejoin channels after we've been kicked though, because
that's just annoying.
I simplified the time-to-live calculations, with the following drops:
* XMIT_TTL, because we have per-channel timers
(IRCProtocol._transmit_ttl and Channel.last_tx), which closes quiet
channels in IRCProtocol._check_ttl (analagous to the old
CHANNEL_TTL). The IRCProtocol closes itself (from
_handle_channel_disconnect) if it no longer has any channels, so a
separate timeout at the IRCProtocol level isn't needed.
* PING_TTL, because it's a subset of the general case handled by
IRCProtocol._receive_ttl.
* DISCONNECT_TTL, because I could't reproduce delayed reconnect hangs.
Requesting a connection to a closed port raised:
localhost:6667: Multiple exceptions:
[Errno 111] Connect call failed ('::1', 6667, 0, 0),
[Errno 111] Connect call failed ('127.0.0.1', 6667)
which was caught without delay in Dispatcher._connection_created,
after which the failed connection was dropped. Connections that are
open but not responding will be caught by the handshake TTL.
FIXME: This will leave the target's dispatcher and channel queues in
memory, so cleaning up after some delay is probably a good idea.
* UNSEEN_TTL, because I could't reproduce invalid-servername hangs.
Requesting an invalid servername raised:
example.invalid:6667: [Errno -2] Name or service not known
which was caught without delay in Dispatcher._connection_created,
after which the failed connection was dropped.
I also removed ANTI_BUZZ_DELAY, because we're using scheduled
callbacks instead of polling for the timeout checks.
I dropped CONNECTION_MAX, because our limit is now the number of open
connection file descriptors, not thread memory usage. I don't expect
we'll bump into this limit, but it's easier to catch that exception
than to track global "connection count" state.