* Bump to Go 1.16.6, which fixes some important issues
* use linux/arm64 golang base image
- Discovered the correct digest using:
'docker manifest inspect golang:1.16.6 | jq -r '.manifests[] | select(.platform.os == "linux" and .platform.architecture == "amd64") | .digest'
Co-authored-by: Sam Salisbury <samsalisbury@gmail.com>
This fixes this test which was failing intermittently.
* Ups the grace period timeout to 2s.
* Re-positions the testify require.New() so that incorrect code path
warnings are not given.
This commit adds the support to do the following:
* Mark connections for non-reporting workers as closed. This is the
controller counterpart to the worker functionality (see #1330). This is
written as a scheduled job that does the work DB-side in a single atomic
query.
* Works to reconcile states if such a broken controller-worker
connection resumes and a worker reports a connection as connected that
should be disconnected. In this case, the controller will send an update
request, and the worker will honor it and terminate the connection.
* Further refinement of the grace period setting has been added here.
We have converged on the current server "liveness" setting as our
default here, which is half of the previous 30s (15 seconds, in other
words). Additionally, this is now configurable on the controller and
worker side, with the caveat that it's currently impossible to do so in
config as the setting has been untagged in HCL. This is exposed so that
we can run some sophisticated testing scenarios where we skew the grace
period to either the controller or worker to ensure the aforementioned
reconciliation works.
* Some repository functions have been added to support the new
functionality, in addition to some test code to the worker to allow
querying of session state while testing.
* Finally, we've added some add timestamp subtraction functions
as well, basically serving as the opposite of the addition functions.
This adds enumeration constraints to the server table, ensuring that no
servers can be added other than the types "controller" and "worker".
This should already be impossible to do in code, but this adds a check
on the database level to ensure it's the case as well.
This adds a return when aborting from closeConnections, which should
have happened as a part of #1369.
Note that it's currently inconsequential that it's not happening right
now, save some misleading log entries. setCloseTimeForResponse will
effectively no-op on empty or nil input maps.
This backports some test work that is being added in #1340 to assert
that the worker is actually marking connections closed in its *local*
state on a failure to contact the controller. This was missed in #1357
during some late-stage refactoring.
Summary of the changes:
* If we can't actually contact the controller during connection closure
to mark the connections as closed, we actually construct a "fake"
response that just simply includes a closed state for all connections
requested. This is then used to update the local connection state.
* Added a 3 second timeout to the CloseConnection request to the
controller to ensure the call to close connections completed in a
timely manner and did not hang indefinitely.
Testing enhancements:
* Added a new helper in TestHelper called LookupSession that allows the
lookup of a session from the worker's internal session state.
* This is used in the new ExpectConnectionStateOnWorker check in the E2E
test, which is part of a larger amount of enhancements to the E2E test
that are coming in #1340.
* Added a ConnectionStatusFromProtoVal helper function to the session
package to allow the conversion of protobuf connection status values to
their higher-level counterparts; essentially it is a reverse of
ConnectionStatus.ProtoVal. This is used in some of our tests (with more
coming in #1340) to just help with keeping things more idiomatic.
Finally, there is some minor error refactoring, and we also change the
timeout on internal status and connection close requests to the
controller to 5 seconds.
This commit adds some rudimentary functionality for cleaning up
connections on controller failure, or more specifically, a failure to
communicate with the controllers.
In this scenario, when a status cannot be sent for any reason for longer
than 30 seconds, the worker will close all connections that are normally
not cleaned up (so for sessions that currently are *not* in the progress
of being canceled or terminated). Status requests now time out after 10
seconds if they are not made successfully.
Session status is not touched during this process, but natural cleanup
or connectionless sessions will still occur. This should be okay, as the
session should be added on re-connect if need be.
A few other changes are made to cancelConnections: first, error logging
has been moved to the function so that we can handle errors requesting
cancellation directly within the function; this entails simply just
locally marking these connections as closed with a warning. Second,
there was a minor logging bug where closed connections were not properly
being noted in the last trace log, this has been fixed.
Additional changes:
* Circle: Set acceptance test no_output_timeout to 30m
Jim
Jeff MitchellandSam Salisbury
Chris Marchesi
Louis Ruch
Mike Green
Noel Quiles
Michael Gaffney
Carlos Alcaide Corvo
Joe Thompson
Pete Pacent
Susmitha Girumala
Todd Knight
Randall Morey
Kendall Strautman