The SetupSuiteTargetFilters globally mutates the function used for logic
inside the target handler service. This means that we could get
inconsistent logical behavior in certain tests that were run in parallel,
since the logic can change at runtime. Ensure tests that make use
of this function are never run in parallel with other tests.
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Adding explicit MPL license for sub-package
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Updating the license from MPL to Business Source License
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl.
* Update copyright file headers to BUS-1.1
* Rerun make gen
This will pick up the last of the license changes
* Revert "Rerun make gen"
This reverts commit dbe0968bffd92745d8663c6940debae098395a6b.
* Remove copywrite from generation script
The current version of copywrite is not sophisticated enough
to only apply the copywrite changes to certain subdirectories,
so to avoid accidentally relicensing our api and sdk modules,
we will stop running the copywrite script until such time
that it can do the right thing.
* License protobuf API as MPL
The protobuf API files are used to generate the SDK,
since we want the SDK to be MPL, the protobuf files
must also be MPL. This extends to all protobuf files
whose sources end up in the SDK directory, including:
- internal/proto/controller/api
- internal/proto/controller/custom_options
- internal/proto/plugin
It also includes the generated service directory in
internal/gen/controller/api/services, since they are
generated from the protobuf sources used for the SDK.
* Change website license file name
This makes the naming scheme consistent with the rest
of the repository.
* Add note on licensing to README
* Set copywrite license to BUSL
* Revert "Add note on licensing to README"
This reverts commit 4d865a91a7a4f11605f0dc4609355d7e3c3cfffe.
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
This requires specifying `**` as the selector in the proto, which will
match any segments and simply requires that the verb is the last part of
the URL, which it already is.
* internal/tests/cluster: switch flaky test to polling
Also extend deadline from 10 seconds to 1 minute, hopefully
this should fix any flakiness in this test.
* fixup! internal/tests/cluster: switch flaky test to polling
Do not allow PKI-KMS workers to have names updated via API
This would cause their public IDs to change, which we do not want.
Descriptions are also disallowed, since those come from the config file.
This also adds logic to allow upgrading from old KMS method to new, with
a test.
Co-authored-by: Irena Rindos <irenarindos@users.noreply.github.com>
This adds a new mechanism for worker authentication that uses the
third-party arbitration of the original KMS flow by leveraging
recently-added capabilities in nodeenrollment to support a registration
wrapper. This allows for workers to be registered to the system with
just a name and a matching wrapper, while at the same time allowing
these workers to be used for private Vault access and multi-hop as they
contain per-worker encryption keys and support rotation.
Nodes using this mode will generate and rotate credentials in-memory
(using another recently added nodeenrollment capability) so will have a
new set of keys on every startup. This has the side effect of ensuring
that there is never a conflict or re-use of these credentials.
Note that the normal test worker flow was using the now-deprecated KMS
method; there is still a test that uses this (explicitly) but all other
tests using test workers that did not specify a specific PKI flow now
actually use this mechanism. This means there is fairly decent test
coverage in a general sense, plus the specific tests updated/added for
this feature.
This also adds a new KMS type that can be used to have
distinct KMSes for upstream authentication vs. accepting from
downstream.
With this commit, the Worker now sends BytesUp and BytesDown for each
of the connections it handles to the Controller in the Status request.
The Controller then persists these fields to the database. There is also
a new trigger to prevent bytes_up and bytes_down to change after a
connection has been closed, as the last update to a session_connection
row should be the one that is performed as part of the connection
closure process.
* Remove the server_id from session table, change it to worker_id on session_connection
This is step 1 in updating the worker table. The worker is becoming a public resource so it must have
a public id. We drop the server_id column on the session_connection table and add the worker_id column
so we can use the wt_public_key type. Since db migrations are applied while the controller is offline,
it should be safe to assume all session connections are not in a running state, thus allowing us to null
out all the worker ids in the session_connection table.
For now we don't have very good enforcement of the wt_public_id constraints, since the id is currently
defined in the worker config, but this will soon change and the worker's id will be generated by the
controller.
* Move the server daemons into their own package
With this change the internal/server package now is dedicated to the server domain (workers and controllers) and not to the server daemons.
The hope is that this will make it easier to keep the daemons decoupled from the representation of the servers in the database.
This reduces the chance that we'll hit a failure due to a mismatched
worker address, on the certificate, especially if a worker has multiple
addresses. We aren't relying on IP/DNS for security anyways, so let's
not have it be a hindrance.
This change introduces a `nonceFn` field that can be changed
by a test but at runtime is transparent to the code that calls it.
The current nonce-generating logic defaults to a test-only field which
opens us up to this value inadvertently being set.
With this change, the default behavior becomes using `base62` to
generate a random nonce, and any tests have to explicitly change that.
In general, these changes were introduced in the same vein of the
changes to controller `startListeners`.
Specifically, these changes also decouple the Controller and the Worker
shutdown processes. Currently, a call to `stopListeners` on an instance
that is running as both a Controller and a Worker would stop the
listeners for both.
Lastly, we also add unit/integration tests to the shutdown code.
This creates a domain service function `session.WorkerStatusReport`. The
service performs two functions:
1. Checks the status for all of the reported sessions, returning a
report of the sessions that are in a "non-active" state, i.e.
"cacneling" or "terminated".
2. Looks for any "orphaned" session connections, i.e. connections that
are active but were not reported by the worker. These connections are
marked as closed.
This also removes the "ShouldCloseConnectionsOnWorker" logic, as this
was determined to not be a possible state, and was unnecessary.
This stores worker connection nonces within the database to ensure
uniqueness of connections. Prior the connection required any attacker to
be able to decrypt the incoming encrypted authentication message in
order to replay (and if the attacker has access to the KMS you have
bigger problems), but with this change even if an attacker does have
access to the KMS key, they cannot reuse the same message. (If they're
able to encrypt against the KMS key, they can still act as a worker with
a new message, so this only really helps if they only have decryption
capabilities.)
The existing logic was to ignore Forbidden errors when recursively
listing in favor of looking for downstream scopes with appropriate
permissions. The problem is that when there is actually no
authentication token at all, what comes back is an Unauthenticated
error. Therefore if you didn't have permissions on the ingress scope
(e.g. the default of `global`) but did downstream for the anonymous
user, you'd get back an error.
I don't think there's any security issue with doing this; the user could
always have simply started the search at a downstream scope to begin
with so nothing more is being revealed, this just fixes behavior to
match expectations in this case.
A test was added for one of the services; the code is exactly the same
across them. The reason that the existing tests that were added with the
output fields testing didn't catch this is that even though they were
specifying `u_anon` as the user, the disabled test context skipped the
logic that would return a 401 in this case. It turns out that fixing
this is very complicated because we don't have a great way in most tests
of defining grants to put onto the generated scopes, so even if a true
anonymous context was added each test would require the anonymous user
to have list permissions on whatever resource was under test, requiring
a lot of role and grant and principal finagling. This is probably a good
longer-term TODO but not one that seemed worth it for this fix.
Fixes ICU-2301
This moves the generated pb files to the `sdk` module. This allows third parties to make use of them if desired (e.g. in upcoming plugins) and moves away from the `gen` nomenclature in the directory, since we are going to be adding some extra interface functions to the messages.
This adds functionality to ensure that connections are closed on a
worker when the server is shut down or reloaded.
Connections are closed, and then the shutdown process waits for the next
report to the controller to go through, waiting the grace period that
was added in #1330. The rest of shutdown is then proceeded with.
Listener shutdown has been moved to the start of the process to ensure
that new connections can't be made while we are shutting down.
Additionally, functionality has been added to force shut down servers
after a second interrupt has been received after the first one. This
takes the form of code in both the server and dev mode code paths that
just performs an os.Exit if another shutdown is received during the
normal shutdown process.
Additionally, the commit also includes some refactoring to tests so that
we could use some of the new E2E testing code to test session
connections during shutdown and reload.
Additionally, this fixes a panic in worker startup when there are no
controllers defined. Looks like the shutdown call was attempting to call
into the controller, not the worker.
Finally, this also removes the extra shutdown within the outer worker
startup.
Shutdown is actually handled within the inner StartWorker(), attempting
shutdown on errors was actually leading to potential use of an
uninitialized worker, which is leading to panics in testing.
This commit adds the support to do the following:
* Mark connections for non-reporting workers as closed. This is the
controller counterpart to the worker functionality (see #1330). This is
written as a scheduled job that does the work DB-side in a single atomic
query.
* Works to reconcile states if such a broken controller-worker
connection resumes and a worker reports a connection as connected that
should be disconnected. In this case, the controller will send an update
request, and the worker will honor it and terminate the connection.
* Further refinement of the grace period setting has been added here.
We have converged on the current server "liveness" setting as our
default here, which is half of the previous 30s (15 seconds, in other
words). Additionally, this is now configurable on the controller and
worker side, with the caveat that it's currently impossible to do so in
config as the setting has been untagged in HCL. This is exposed so that
we can run some sophisticated testing scenarios where we skew the grace
period to either the controller or worker to ensure the aforementioned
reconciliation works.
* Some repository functions have been added to support the new
functionality, in addition to some test code to the worker to allow
querying of session state while testing.
* Finally, we've added some add timestamp subtraction functions
as well, basically serving as the opposite of the addition functions.
This backports some test work that is being added in #1340 to assert
that the worker is actually marking connections closed in its *local*
state on a failure to contact the controller. This was missed in #1357
during some late-stage refactoring.
Summary of the changes:
* If we can't actually contact the controller during connection closure
to mark the connections as closed, we actually construct a "fake"
response that just simply includes a closed state for all connections
requested. This is then used to update the local connection state.
* Added a 3 second timeout to the CloseConnection request to the
controller to ensure the call to close connections completed in a
timely manner and did not hang indefinitely.
Testing enhancements:
* Added a new helper in TestHelper called LookupSession that allows the
lookup of a session from the worker's internal session state.
* This is used in the new ExpectConnectionStateOnWorker check in the E2E
test, which is part of a larger amount of enhancements to the E2E test
that are coming in #1340.
* Added a ConnectionStatusFromProtoVal helper function to the session
package to allow the conversion of protobuf connection status values to
their higher-level counterparts; essentially it is a reverse of
ConnectionStatus.ProtoVal. This is used in some of our tests (with more
coming in #1340) to just help with keeping things more idiomatic.
Finally, there is some minor error refactoring, and we also change the
timeout on internal status and connection close requests to the
controller to 5 seconds.
This commit adds some rudimentary functionality for cleaning up
connections on controller failure, or more specifically, a failure to
communicate with the controllers.
In this scenario, when a status cannot be sent for any reason for longer
than 30 seconds, the worker will close all connections that are normally
not cleaned up (so for sessions that currently are *not* in the progress
of being canceled or terminated). Status requests now time out after 10
seconds if they are not made successfully.
Session status is not touched during this process, but natural cleanup
or connectionless sessions will still occur. This should be okay, as the
session should be added on re-connect if need be.
A few other changes are made to cancelConnections: first, error logging
has been moved to the function so that we can handle errors requesting
cancellation directly within the function; this entails simply just
locally marking these connections as closed with a warning. Second,
there was a minor logging bug where closed connections were not properly
being noted in the last trace log, this has been fixed.
Additional changes:
* Circle: Set acceptance test no_output_timeout to 30m
* Sessions support for read:self and cancel:self permissions. This is automatic with fallback; no API change is needed.
* Strips out values from listing if only self actions are allowed on the resource and the user ID is different. This prevents a default ACL of id=*;type=session;actions=read:self,cancel:self from causing all values to be visible when listing.
* Adds an unprivileged user to development/testing instances. This user has only default permissions plus the ability to authorize sessions in the default target.
Jeff Mitchell
Irena Rindos
Johan Brandhorst-Satzkorn
hashicorp-copywrite[bot]andJohan Brandhorst-Satzkorn
Jim
Hugo Vieira
Damian Debkowski
Haotian
Timothy Messier
Irena Rindos
Chris Marchesi
Louis Ruch