This backports some test work that is being added in #1340 to assert
that the worker is actually marking connections closed in its *local*
state on a failure to contact the controller. This was missed in #1357
during some late-stage refactoring.
Summary of the changes:
* If we can't actually contact the controller during connection closure
to mark the connections as closed, we actually construct a "fake"
response that just simply includes a closed state for all connections
requested. This is then used to update the local connection state.
* Added a 3 second timeout to the CloseConnection request to the
controller to ensure the call to close connections completed in a
timely manner and did not hang indefinitely.
Testing enhancements:
* Added a new helper in TestHelper called LookupSession that allows the
lookup of a session from the worker's internal session state.
* This is used in the new ExpectConnectionStateOnWorker check in the E2E
test, which is part of a larger amount of enhancements to the E2E test
that are coming in #1340.
* Added a ConnectionStatusFromProtoVal helper function to the session
package to allow the conversion of protobuf connection status values to
their higher-level counterparts; essentially it is a reverse of
ConnectionStatus.ProtoVal. This is used in some of our tests (with more
coming in #1340) to just help with keeping things more idiomatic.
Finally, there is some minor error refactoring, and we also change the
timeout on internal status and connection close requests to the
controller to 5 seconds.
This commit adds some rudimentary functionality for cleaning up
connections on controller failure, or more specifically, a failure to
communicate with the controllers.
In this scenario, when a status cannot be sent for any reason for longer
than 30 seconds, the worker will close all connections that are normally
not cleaned up (so for sessions that currently are *not* in the progress
of being canceled or terminated). Status requests now time out after 10
seconds if they are not made successfully.
Session status is not touched during this process, but natural cleanup
or connectionless sessions will still occur. This should be okay, as the
session should be added on re-connect if need be.
A few other changes are made to cancelConnections: first, error logging
has been moved to the function so that we can handle errors requesting
cancellation directly within the function; this entails simply just
locally marking these connections as closed with a warning. Second,
there was a minor logging bug where closed connections were not properly
being noted in the last trace log, this has been fixed.
Additional changes:
* Circle: Set acceptance test no_output_timeout to 30m
* Sessions support for read:self and cancel:self permissions. This is automatic with fallback; no API change is needed.
* Strips out values from listing if only self actions are allowed on the resource and the user ID is different. This prevents a default ACL of id=*;type=session;actions=read:self,cancel:self from causing all values to be visible when listing.
* Adds an unprivileged user to development/testing instances. This user has only default permissions plus the ability to authorize sessions in the default target.
Chris Marchesi
Jeff Mitchell
Louis Ruch
Jim