* Randomize order of worker output. This uses the standard
rand.Shuffle() call; cribbed from
https://pkg.go.dev/math/rand#example-Shuffle
* Have some error/eventing in the worker status path use the base
context instead of the status-call-specific context to allow eventing
after a timeout event
* feat(workers): Add ability to read and reinitialize cert authority (#2312)
* feat(workers): Return Release Version on worker list and read (#2377)
* feat(cli): Read and reinitialize worker certificate authority (#2387)
On a SIGHUP, if the worker's `initial_upstreams` config option has
changed, it will use the new values as the address to send its status.
Any previously discovered upstream addresses from previous status
requests are dropped. Once a successful status request is made to the
new addresses, the worker will resume its normal behavior of using the
addresses from the response to the status request.
This allows a worker to be deployed in a more stable way in a dynamic
environment where controller IPs could change. In particular this helps
with a case where all controllers are replaced with new controllers
running on new IPs.
To illustrate with an example, suppose there is an instance group of
controllers instances. And there are worker instances that have their
config rendered from consul-template that sets `initial_upstreams` to
the IPs of the controller instances. If all controller instances are
restarted or replaced at the same time and assigned new IPs, a new
config will be rendered for the worker. Previously the only way for the
worker to use the new value would be for the worker process to be
restarted. Otherwise the worker would only know about the previous IPs
and would fail to connect to the new controllers. However, a restart
would mean any active sessions and session connections on the worker
instance would be terminated. With this update, consul-template can send
a SIGHUP to the worker, the worker will connect to the new controllers
and resume normal status reporting, allowing the sessions and session
connections to remain.
The same principles would apply for other examples of dynamic
environments such as nomad and kubernetes.
See:
https://github.com/hashicorp/consul-templatehttps://www.nomadproject.io/docs/job-specification/template
The intention of the code was to have one-hop-away workers list KMS
workers to get initial upstreams, and have other workers continue using
what was in their config file. However, nothing stops people from having
initial_upstreams but _also_ adding a cluster ID in their config file,
in which case we'd always run this check and replace the config-set
initial upstreams. So, ensure the config-set initial upstreams is also
len zero.
Right now there is a bit of a race condition in that a status call that
indirectly tells the controller that a worker is ready to accept
sessions can have its response unable to be processed in time for an
actual session to come in from a client. It's very unlikely but
technically there.
This changes where we check if we have ever successfully authenticated
from the function called by the ticker to the ticker itself, allowing us
to reset the timer to a very short time if we have not yet
authenticated. We already have the initial timer set to 0 for "as soon
as possible" behavior, but if that initial status failed we'd wait for
the normal interval. Now we will instead continue with the "as soon as
possible" behavior, and reduce the very unlikely race condition to
extremely unlikely.
(There are more permanent fixes but those are bigger and can come
later.)
* Discontinue support for old Server message in ServerCoordinationService. Split message into ServerWorkerStatus and UpstreamServer (which currently contains controller addresses). Remove unnecessary fields. Modify workerStatusServer to use new fields.
Co-authored-by: Todd <github@quaddi.com>
* Don't assume it's non-empty or that this is an issue
* Ensure it's empty if the response is coming back through the multihop
service so that it simply uses what is in the config file, at least
for now
* Move the server daemons into their own package
With this change the internal/server package now is dedicated to the server domain (workers and controllers) and not to the server daemons.
The hope is that this will make it easier to keep the daemons decoupled from the representation of the servers in the database.
Jeff Mitchell
Irena Rindos
Timothy Messier
Jim
Todd
irenarindos
HaotianandTodd