Commit Graph
14 Commits
Author SHA1 Message Date
Chris Marchesi 9f9acce972 [Merge to main] internal/servers/worker: Controller failure connection cleanup (#1330) (#1357)
This commit adds some rudimentary functionality for cleaning up
connections on controller failure, or more specifically, a failure to
communicate with the controllers.

In this scenario, when a status cannot be sent for any reason for longer
than 30 seconds, the worker will close all connections that are normally
not cleaned up (so for sessions that currently are *not* in the progress
of being canceled or terminated). Status requests now time out after 10
seconds if they are not made successfully.

Session status is not touched during this process, but natural cleanup
or connectionless sessions will still occur. This should be okay, as the
session should be added on re-connect if need be.

A few other changes are made to cancelConnections: first, error logging
has been moved to the function so that we can handle errors requesting
cancellation directly within the function; this entails simply just
locally marking these connections as closed with a warning. Second,
there was a minor logging bug where closed connections were not properly
being noted in the last trace log, this has been fixed.

Additional changes:

* Circle: Set acceptance test no_output_timeout to 30m
2021-06-28 17:32:13 -07:00
Jeff Mitchell 6f34da8923 Add cleanup of dead connections no longer reported by a worker (#1220) 2021-05-14 13:23:12 -04:00
Jeff Mitchell 717a3b52ee Add worker tagging (#862)
This allows workers to have tags assigned to them, which can then be filtered to control which targets they are allowed to serve.
2021-01-21 16:27:20 -05:00
Jeff Mitchell a6cca576a3 Attempt canceling sessions by the client (#831)
* Attempt canceling sessions by the client

In some circumstances (like a Ctrl-C), the client knows it's shutting
down and the TOFU token will be lost. In this case it makes sense to
cancel the session rather than have it time out. This introduces a
command enum to the handshake message which can be used for the client
to send commands to the worker -- in this case the only specified
command is to cancel the session. It doesn't occur if the session is
already timed out or if we can't authorize connections as this generally
means the session is already invalid.
2020-12-16 17:14:47 -05:00
Todd Knight 21f5cc274f Don't return nil, nil for API service methods. Add tests for updating w/ wrong version. (#396) 2020-09-21 15:09:30 -04:00
Jeff Mitchell fcdaf129a3 Add missing continue in close function 2020-09-21 02:44:53 -04:00
Jeff Mitchell 33b0021547 Add Sessions CLI command and add session cleanup to worker (#388) 2020-09-21 02:22:59 -04:00
Jeff Mitchell bb6ece69b6 Remove unused struct member 2020-09-21 00:11:37 -04:00
Jeff Mitchell 9cbc7b5c3d Add connection close call from worker to controller (#387) 2020-09-21 00:09:50 -04:00
Jeff Mitchell 7ff4b7f106 Send connected RPC to controller (#386) 2020-09-20 23:09:42 -04:00
Jeff Mitchell f7e48ec836 Plumb connection limit to proxy and output it (#384) 2020-09-20 16:28:00 -04:00
Jeff Mitchell 5214f14105 Work on connection authorization (#381) 2020-09-20 12:51:08 -04:00
Jeff Mitchell e002326293 Plumb timeouts to worker and set appropriate deadlines (#378) 2020-09-19 13:51:50 -04:00
Jeff Mitchell 07a7e9750a Tie together the database-driven session handling with the worker and add relevant CLI comands (#370) 2020-09-19 00:18:17 -04:00