* oss: feat(proto): statistics rpc added to server coordination service
* oss: feat(session): add update statistics status service
* oss: feat(handlers): add Statistics method to worker service server
* oss: feat(worker): add statistics ticking
* oss: internal/proto: define the new RoutingInfo RPC
* oss: internal/server: repository changes for RoutingInfo RPC
* oss: internal/daemon: add RoutingInfo handler to worker service
The new RoutingInfo RPC is used to get information from the worker
that is used to make routing decisions on the controller. It also
handles the initial request from the worker.
* oss: feat(db): server_worker_session_info table
* oss: feat(proto): session info rpc for server coordination service
* oss: feat(server): worker session info request data type
* oss: feat(worker): session info ticker
* oss: feat(handler): server coordination handler for session info
* oss: feat(session): utilize session info request time for session cleanup
* oss: proto: fix some gotags comments
Some of these fields were missing comments, while others had
incorrectly formatted comments.
* oss: worker: send connection status to server
This is required for the correct tracking of orphaned connections
* oss: internal/daemon/worker: add function to wait for Statistics RPC
* oss: internal/daemon/worker: add routing info RPC
The new RoutingInfo RPC replaces part of the Status RPC,
namely the part which is used by the controller and worker
to perform decisions around the establishment of new sessions
and new worker connections.
* oss: internal/daemon/worker: remove unnecessary error check
* oss: internal/daemon/worker: lock access to config and upstreams
The new confLock protects access to the conf value and to the
addressReceivers slice.
* oss: worker: replace Status RPC with RoutingInfo
* oss: worker: deprecate the Status RPC and related types
The Status RPC and related messages have been replaced
by the RoutingInfo/SessionInfo/Statistics RPCs. The controller
will continue to support the Status RPC until two new minor
versions have been released.
* oss: internal/event: add default deny for new worker RPCs
* oss: internal/daemon/worker: allow RPC interval to be set
The new SessionInfo/RoutingInfo/Statistics RPCs run at
long intervals, which make some of our cluster tests take
a long time to complete. This allows their interval to be
configured in tests.
* oss: all: use TestWorkerRPCInterval in slow tests
This should speed up these tests significantly.
* oss: internal/tests/cluster: reset sys eventer in test
---------
Co-authored-by: Damian Debkowski <damian.debkowski@hashicorp.com>
At some point we migrated to grpc ConnectParams to control how often
retries were happening. One value that we didn't set was the
MinConnectTimeout. It turns out that if you don't set this, it won't use
the default; instead if will actually use a value of zero, which gRPC
internally ups to something that seems to be around 25ms to meet a
minimum threshold. This is not documented in the function comments.
This commit simply sets the value of MinConnectTimeout to the unexported
gRPC default.
This has been superceded by the nodeenrollment-based KMS worker auth
mechanism. This commit only removes the ability to connect with this
mechanism; some upgrade logic will remain intact for a bit and
can be removed later.
* fix: monitorUpstreamConnectionState CPU consumption
`monitorUpstreamConnectionState()` is a goroutine that listens for GRPC client connection changes using GRPC's `WaitForStateChange` method. `monitorUpstreamConnectionState()` was consuming the a lot of CPU. It was continuously running and not actually waiting for state changes. The `state` for `monitorUpstreamConnectionState()` was not getting correctly updated which caused the loop to continuously run instead of waiting for new state changes.
Update GRPC `WaitForStateChange` method to use the same `state` that gets updated.
With a suitably slow system -- say, if you're waiting on a Windows
Terminal to output text inside a virtual machine -- the expected order
of things can be thrown off. One specific way this can happen is that
the worker can attempt to dial and have its dial fail because of the
slowness causing the dial context to time out. If, because of the
slowness, we also haven't released the log gate yet, then the event with
the failure information will be queued, along with the context that was
used.
Unfortunately, in some of these error cases, the context that was used
was the dial context instead of the system base context. In many other
places in the function it was the system base context so this is just a
mismatch, probably from code written at different times.
Normally this wouldn't be a problem as we'd fall back to the underlying
logger, but when we release the log gate, things happen differently:
it's a synchronous function and on error it causes us to abandon
system startup entirely.
This commit fixes the issue with using the incorrect context. It's an
open question whether we should change the behavior around errors when
replaying queued events, falling back to the underlying logger instead
of erroring.
* feat: Worker Connectivity Health Check
Add worker upstream connection state. API calls to the worker's ops `/heath?worker_info=1` endpoint should now return a new filed called `upstream_connection_state`
```
{
"worker_process_info":{
"state":"active",
"active_session_count":0,
"upstream_connection_state":"READY"
}
}
```
The states returned will be GRPC connection states: https://grpc.github.io/grpc/core/md_doc_connectivity-semantics-and-api.html
There could be multiple upstream addresses but GRPC manual resolver determines the state of the connection.
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Adding explicit MPL license for sub-package
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Updating the license from MPL to Business Source License
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl.
* Update copyright file headers to BUS-1.1
* Rerun make gen
This will pick up the last of the license changes
* Revert "Rerun make gen"
This reverts commit dbe0968bffd92745d8663c6940debae098395a6b.
* Remove copywrite from generation script
The current version of copywrite is not sophisticated enough
to only apply the copywrite changes to certain subdirectories,
so to avoid accidentally relicensing our api and sdk modules,
we will stop running the copywrite script until such time
that it can do the right thing.
* License protobuf API as MPL
The protobuf API files are used to generate the SDK,
since we want the SDK to be MPL, the protobuf files
must also be MPL. This extends to all protobuf files
whose sources end up in the SDK directory, including:
- internal/proto/controller/api
- internal/proto/controller/custom_options
- internal/proto/plugin
It also includes the generated service directory in
internal/gen/controller/api/services, since they are
generated from the protobuf sources used for the SDK.
* Change website license file name
This makes the naming scheme consistent with the rest
of the repository.
* Add note on licensing to README
* Set copywrite license to BUSL
* Revert "Add note on licensing to README"
This reverts commit 4d865a91a7a4f11605f0dc4609355d7e3c3cfffe.
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
This adds a new mechanism for worker authentication that uses the
third-party arbitration of the original KMS flow by leveraging
recently-added capabilities in nodeenrollment to support a registration
wrapper. This allows for workers to be registered to the system with
just a name and a matching wrapper, while at the same time allowing
these workers to be used for private Vault access and multi-hop as they
contain per-worker encryption keys and support rotation.
Nodes using this mode will generate and rotate credentials in-memory
(using another recently added nodeenrollment capability) so will have a
new set of keys on every startup. This has the side effect of ensuring
that there is never a conflict or re-use of these credentials.
Note that the normal test worker flow was using the now-deprecated KMS
method; there is still a test that uses this (explicitly) but all other
tests using test workers that did not specify a specific PKI flow now
actually use this mechanism. This means there is fairly decent test
coverage in a general sense, plus the specific tests updated/added for
this feature.
This also adds a new KMS type that can be used to have
distinct KMSes for upstream authentication vs. accepting from
downstream.
When possible downstream workers will include their worker id information in
the connections they establish with the upstream using the nodeenrollment
library. The tracking listener now tracks connections that are connected in
this same way instead of just connections of type *tls.Conn.
* Fix target port handling
This fixes two issues that compounded on each other (see the Changelog
update for more information):
* The verification logic for hosts was not correct for update operations
(in multiple ways) which meant that a host could be updated after
creation to have a port. Targets had a previously fixed bug where they
did not require a default port, which meant that ports could be used
from hosts
* A recent (unreleased) change had prioritized any port coming from the
host over the default port, which would mean there was no way if a
host had a port specified to use it in multiple targets.
This fixes the update verification logic, and strikes a middle ground
between breaking things and not by allowing existing addresses with
ports to be used with targets but ignoring that port, instead requiring
targets to have default port set at authorize time (currently there is
backwards compat that does not require this due to the original optional
port bug).
* Update CHANGELOG.md
Co-authored-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
This adds an action to the API that allows pulling out an activation
token that can be given to a worker in its config (directly or via env
or file). The worker can pass this along with its normal request in
place of its normal nonce and the server will automatically accept it.
Co-authored-by: Irena Rindos <irenarindos@users.noreply.github.com>
This essentially worked for non-dev mode before but would create storage
directories that weren't actually used or needed, and it didn't work in
dev mode because it made assumptions about the presence of a worker auth
kms or not. This fixes those behaviors.
* Adds WorkerAuthStorage wrapper for worker credential encryption using nodeenrollment library.
* Adds AuthStoragePath field in worker config and dev mode to specify directory to store credentials.
* Move the server daemons into their own package
With this change the internal/server package now is dedicated to the server domain (workers and controllers) and not to the server daemons.
The hope is that this will make it easier to keep the daemons decoupled from the representation of the servers in the database.
Hugo
Johan Brandhorst-SatzkornandDamian Debkowski
daniandDamian Debkowski
Jeff Mitchell
Elim Tsiagbey
Irena Rindos
hashicorp-copywrite[bot]andJohan Brandhorst-Satzkorn
Todd
Jim
Haotian
irenarindos