This commit adds some rudimentary functionality for cleaning up
connections on controller failure, or more specifically, a failure to
communicate with the controllers.
In this scenario, when a status cannot be sent for any reason for longer
than 30 seconds, the worker will close all connections that are normally
not cleaned up (so for sessions that currently are *not* in the progress
of being canceled or terminated). Status requests now time out after 10
seconds if they are not made successfully.
Session status is not touched during this process, but natural cleanup
or connectionless sessions will still occur. This should be okay, as the
session should be added on re-connect if need be.
A few other changes are made to cancelConnections: first, error logging
has been moved to the function so that we can handle errors requesting
cancellation directly within the function; this entails simply just
locally marking these connections as closed with a warning. Second,
there was a minor logging bug where closed connections were not properly
being noted in the last trace log, this has been fixed.
Additional changes:
* Circle: Set acceptance test no_output_timeout to 30m
* Attempt canceling sessions by the client
In some circumstances (like a Ctrl-C), the client knows it's shutting
down and the TOFU token will be lost. In this case it makes sense to
cancel the session rather than have it time out. This introduces a
command enum to the handshake message which can be used for the client
to send commands to the worker -- in this case the only specified
command is to cancel the session. It doesn't occur if the session is
already timed out or if we can't authorize connections as this generally
means the session is already invalid.
* Possible api changes.
* Worker API update.
* Updating import alias name.
* Moving session down a level as a member of the GetSessionResponse.
* Marshal the auth flag into the GetSessionResponse proto instead of the session.
Jim
Jeff Mitchell
Chris Marchesi
Michael GaffneyandJeff Mitchell
Christian Frichot
Todd Knight