* oss: feat(proto): statistics rpc added to server coordination service
* oss: feat(session): add update statistics status service
* oss: feat(handlers): add Statistics method to worker service server
* oss: feat(worker): add statistics ticking
* oss: internal/proto: define the new RoutingInfo RPC
* oss: internal/server: repository changes for RoutingInfo RPC
* oss: internal/daemon: add RoutingInfo handler to worker service
The new RoutingInfo RPC is used to get information from the worker
that is used to make routing decisions on the controller. It also
handles the initial request from the worker.
* oss: feat(db): server_worker_session_info table
* oss: feat(proto): session info rpc for server coordination service
* oss: feat(server): worker session info request data type
* oss: feat(worker): session info ticker
* oss: feat(handler): server coordination handler for session info
* oss: feat(session): utilize session info request time for session cleanup
* oss: proto: fix some gotags comments
Some of these fields were missing comments, while others had
incorrectly formatted comments.
* oss: worker: send connection status to server
This is required for the correct tracking of orphaned connections
* oss: internal/daemon/worker: add function to wait for Statistics RPC
* oss: internal/daemon/worker: add routing info RPC
The new RoutingInfo RPC replaces part of the Status RPC,
namely the part which is used by the controller and worker
to perform decisions around the establishment of new sessions
and new worker connections.
* oss: internal/daemon/worker: remove unnecessary error check
* oss: internal/daemon/worker: lock access to config and upstreams
The new confLock protects access to the conf value and to the
addressReceivers slice.
* oss: worker: replace Status RPC with RoutingInfo
* oss: worker: deprecate the Status RPC and related types
The Status RPC and related messages have been replaced
by the RoutingInfo/SessionInfo/Statistics RPCs. The controller
will continue to support the Status RPC until two new minor
versions have been released.
* oss: internal/event: add default deny for new worker RPCs
* oss: internal/daemon/worker: allow RPC interval to be set
The new SessionInfo/RoutingInfo/Statistics RPCs run at
long intervals, which make some of our cluster tests take
a long time to complete. This allows their interval to be
configured in tests.
* oss: all: use TestWorkerRPCInterval in slow tests
This should speed up these tests significantly.
* oss: internal/tests/cluster: reset sys eventer in test
---------
Co-authored-by: Damian Debkowski <damian.debkowski@hashicorp.com>
* feat(server): Refactor LookupWorkerByName to be a test method
* feat(server): Split worker tags table
* refact(worker): Export api and config tags
* refact(server): Return worker id from UpsertWorkerStatus
* refact(server): Create ListWorkerStorageBucketUpdates domain function
* refact(server): Use dedicated HCPb-worker list func
* refact(server): Create ConnectionRoute domain function
* refact(handlers): Move worker selection logic to internal/server package
This commit moves all the logic to select a worker to handle a session
to the internal/server package. Most functions were moved to this
package with no changes, however, there are a couple of notable
exceptions:
- Given that we don't use GRPC errors in repos, all errors were adjusted
to use the internal/errors package instead.
- StorageBucketFilterCredIdFn is now used to prevent the internal/server
package from importing internal/storage/plugin, which would cause a
dep cycle. Callers of SelectSessionWorkers must implement this func
which is responsible for returning a storage bucket's worker filter as
well as its associated credential id.
- authorizeSessionWithWorkerFilter now receives an
internal/server.(*Repository) object when called. This is transparent
to SelectSessionWorkers callers as it is handled internally.
* feat(server): Add FilteredWithFeatures fn and new server options
* refact(server): create new list and lookup worker queries
* test(server): Add WorkerList unit tests
* feat(server): Implement WorkerList randomness using crypto/rand pkg
* chore(db): add index to server_worker table
* refact(server): prefix test method with Test
* refact(server): rename Downstreamers interface to Graph
* chore(server): unexport and rename method
* chore: add doc string to WorkerAddress (#5420)
---------
Co-authored-by: Irena Rindos <irenarindos@users.noreply.github.com>
Co-authored-by: irenarindos <irena.rindos@gmail.com>
The scheduler monitors running jobs to make sure they are not hung. If
the status of a running job does not change after a duration (the
interruptThreshold in the scheduler), the scheduler kills the job. This
change passes the scheduler's interruptThreshold to the job when it
calls Run() on the job.
* refact: to use go-dbw CreateItems, DeleteItems
Latest version of go-dbw updated the signature of
CreateItems(...) and DeleteItems(...) on a happy
note this latest version of go-dbw actually does
batch inserts and updates for this functions.
* all: regenerate protos with new protoc-gen-go version
- add `LocalStorageState` to worker status reports
- update worker service handler to process `LocalStorageState`. If is empty for workers older workers, we set the `LocalStorageState` to `unknown`
- update field mask paths to support `LocalStorageState`
- Add new `local_storage_state` column to `server_worker` table which keeps track of the state of the local disk space.
- The valid values for `local_storage_state` are:
-- `available`: The worker local storage state is at an acceptable state
-- `lowStorage`: The worker is below the minimum threshold for local storage
-- `criticallyLowStorage`: The worker local storage state is below the critical minimum threshold for local storage
-- `outOfStorage`: The worker is out of local disk space
-- `notConfigured`: The worker does not have a local storage path configured
-- `unknown`: The default local storage state of a worker. Used when the local storage state of a worker is not yet known
* chore: update deps
* tests: update sqlmock tests for breaking chg
* tests (handlers): sort grants in checkEqualGrants(...)
sort the expected and got grants in
checkEqualGrants(...) to make the test more
deterministic.
* tests (asciicast): update backspace enc in TestEventMarshal(...)
* fix (auth/ldap): fix which DEK is used for oplog during authenticate
* chore: update go-kms-wrapping dep for new extras/kms changes
The latest version supports setting a table name prefix for a kms
which is needed to fix our oplog DEK problem
* chore: update extras/kms dep
This latest version supports an optional TableNamePrefix which is
required to fix our existing oplog DEK issue.
* fix (oplog): schema oplog DEK fix
These changes will allow use to store the oplog DEK in it's own
set of tables, so they won't get deleted when a scope is deleted
* fix (kms): move oplog DEK into its own internal kms
* tests: update tests to be compatible with oplog fix
* tests: remove migration tests which are incompatible with oplog fix
These tests no longer work because they are missing the required
schema changes for the oplog fix
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Adding explicit MPL license for sub-package
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Updating the license from MPL to Business Source License
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl.
* Update copyright file headers to BUS-1.1
* Rerun make gen
This will pick up the last of the license changes
* Revert "Rerun make gen"
This reverts commit dbe0968bffd92745d8663c6940debae098395a6b.
* Remove copywrite from generation script
The current version of copywrite is not sophisticated enough
to only apply the copywrite changes to certain subdirectories,
so to avoid accidentally relicensing our api and sdk modules,
we will stop running the copywrite script until such time
that it can do the right thing.
* License protobuf API as MPL
The protobuf API files are used to generate the SDK,
since we want the SDK to be MPL, the protobuf files
must also be MPL. This extends to all protobuf files
whose sources end up in the SDK directory, including:
- internal/proto/controller/api
- internal/proto/controller/custom_options
- internal/proto/plugin
It also includes the generated service directory in
internal/gen/controller/api/services, since they are
generated from the protobuf sources used for the SDK.
* Change website license file name
This makes the naming scheme consistent with the rest
of the repository.
* Add note on licensing to README
* Set copywrite license to BUSL
* Revert "Add note on licensing to README"
This reverts commit 4d865a91a7a4f11605f0dc4609355d7e3c3cfffe.
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
The old logic made it impossible for the caller to set
the release version with WithReleaseVersion. Only
set the release version when it's not explicitly set already.
refactor servers.(repository).ListWorkers(...) into just
servers.ListWorkers(...) used independently of the repo. The method
func is refactor to use the new bits as well.
* Add more logging to worker auth rotation and test flags
This makes it much easier to figure out when and why rotation is
happening and control it for tests/debugging.
* Update auth rotation logic
Because we were looking at validity period of client certs, which may
not have matched the actual rotation interval, it was possible (mostly
in test scenarios) for the rotation to happen multiple times on the
roots before we actually hit the worker rotation interval.
This changes how the interval is calculated by looking at the difference
between not-after values on the certs to understand the rotation
interval being used by the controller. We then use that as the starting
point for calculating a next-rotation time.
This makes it much easier to figure out when and why rotation is
happening and control it for tests/debugging.
Remove cert skew
Cherry-picked from release/0.12.x
Do not allow PKI-KMS workers to have names updated via API
This would cause their public IDs to change, which we do not want.
Descriptions are also disallowed, since those come from the config file.
This also adds logic to allow upgrading from old KMS method to new, with
a test.
Co-authored-by: Irena Rindos <irenarindos@users.noreply.github.com>
This adds a new mechanism for worker authentication that uses the
third-party arbitration of the original KMS flow by leveraging
recently-added capabilities in nodeenrollment to support a registration
wrapper. This allows for workers to be registered to the system with
just a name and a matching wrapper, while at the same time allowing
these workers to be used for private Vault access and multi-hop as they
contain per-worker encryption keys and support rotation.
Nodes using this mode will generate and rotate credentials in-memory
(using another recently added nodeenrollment capability) so will have a
new set of keys on every startup. This has the side effect of ensuring
that there is never a conflict or re-use of these credentials.
Note that the normal test worker flow was using the now-deprecated KMS
method; there is still a test that uses this (explicitly) but all other
tests using test workers that did not specify a specific PKI flow now
actually use this mechanism. This means there is fairly decent test
coverage in a general sense, plus the specific tests updated/added for
this feature.
This also adds a new KMS type that can be used to have
distinct KMSes for upstream authentication vs. accepting from
downstream.
When possible downstream workers will include their worker id information in
the connections they establish with the upstream using the nodeenrollment
library. The tracking listener now tracks connections that are connected in
this same way instead of just connections of type *tls.Conn.
Irena Rindos
Johan Brandhorst-SatzkornandDamian Debkowski
Michael Gaffney
Jim
Elim Tsiagbey
Michael Li
Jeff MitchellandDamian Debkowski
Lars Lehtonen
hashicorp-copywrite[bot]andJohan Brandhorst-Satzkorn
Todd