Previously, the orphaned connection cleanup job would
not detect that a connection that was created before a
worker had submitted a successful session info was
orphaned. We now fall back to using the worker update
time as an indicator if the session info time is not set.
* oss: feat(proto): statistics rpc added to server coordination service
* oss: feat(session): add update statistics status service
* oss: feat(handlers): add Statistics method to worker service server
* oss: feat(worker): add statistics ticking
* oss: internal/proto: define the new RoutingInfo RPC
* oss: internal/server: repository changes for RoutingInfo RPC
* oss: internal/daemon: add RoutingInfo handler to worker service
The new RoutingInfo RPC is used to get information from the worker
that is used to make routing decisions on the controller. It also
handles the initial request from the worker.
* oss: feat(db): server_worker_session_info table
* oss: feat(proto): session info rpc for server coordination service
* oss: feat(server): worker session info request data type
* oss: feat(worker): session info ticker
* oss: feat(handler): server coordination handler for session info
* oss: feat(session): utilize session info request time for session cleanup
* oss: proto: fix some gotags comments
Some of these fields were missing comments, while others had
incorrectly formatted comments.
* oss: worker: send connection status to server
This is required for the correct tracking of orphaned connections
* oss: internal/daemon/worker: add function to wait for Statistics RPC
* oss: internal/daemon/worker: add routing info RPC
The new RoutingInfo RPC replaces part of the Status RPC,
namely the part which is used by the controller and worker
to perform decisions around the establishment of new sessions
and new worker connections.
* oss: internal/daemon/worker: remove unnecessary error check
* oss: internal/daemon/worker: lock access to config and upstreams
The new confLock protects access to the conf value and to the
addressReceivers slice.
* oss: worker: replace Status RPC with RoutingInfo
* oss: worker: deprecate the Status RPC and related types
The Status RPC and related messages have been replaced
by the RoutingInfo/SessionInfo/Statistics RPCs. The controller
will continue to support the Status RPC until two new minor
versions have been released.
* oss: internal/event: add default deny for new worker RPCs
* oss: internal/daemon/worker: allow RPC interval to be set
The new SessionInfo/RoutingInfo/Statistics RPCs run at
long intervals, which make some of our cluster tests take
a long time to complete. This allows their interval to be
configured in tests.
* oss: all: use TestWorkerRPCInterval in slow tests
This should speed up these tests significantly.
* oss: internal/tests/cluster: reset sys eventer in test
---------
Co-authored-by: Damian Debkowski <damian.debkowski@hashicorp.com>
* feat(server): Refactor LookupWorkerByName to be a test method
* feat(server): Split worker tags table
* refact(worker): Export api and config tags
* refact(server): Return worker id from UpsertWorkerStatus
* refact(server): Create ListWorkerStorageBucketUpdates domain function
* refact(server): Use dedicated HCPb-worker list func
* refact(server): Create ConnectionRoute domain function
* refact(handlers): Move worker selection logic to internal/server package
This commit moves all the logic to select a worker to handle a session
to the internal/server package. Most functions were moved to this
package with no changes, however, there are a couple of notable
exceptions:
- Given that we don't use GRPC errors in repos, all errors were adjusted
to use the internal/errors package instead.
- StorageBucketFilterCredIdFn is now used to prevent the internal/server
package from importing internal/storage/plugin, which would cause a
dep cycle. Callers of SelectSessionWorkers must implement this func
which is responsible for returning a storage bucket's worker filter as
well as its associated credential id.
- authorizeSessionWithWorkerFilter now receives an
internal/server.(*Repository) object when called. This is transparent
to SelectSessionWorkers callers as it is handled internally.
* feat(server): Add FilteredWithFeatures fn and new server options
* refact(server): create new list and lookup worker queries
* test(server): Add WorkerList unit tests
* feat(server): Implement WorkerList randomness using crypto/rand pkg
* chore(db): add index to server_worker table
* refact(server): prefix test method with Test
* refact(server): rename Downstreamers interface to Graph
* chore(server): unexport and rename method
* chore: add doc string to WorkerAddress (#5420)
---------
Co-authored-by: Irena Rindos <irenarindos@users.noreply.github.com>
Co-authored-by: irenarindos <irena.rindos@gmail.com>
This changes the periodic job that deletes terminated sessions to delete
the those sessions in batches. Prior to this, a single SQL DELETE
command was used to delete all qualifying terminated sessions. In
systems under heavy load, the statement could timeout leading to a
situation where no qualifying terminated sessions were ever being
deleted.
The scheduler monitors running jobs to make sure they are not hung. If
the status of a running job does not change after a duration (the
interruptThreshold in the scheduler), the scheduler kills the job. This
change passes the scheduler's interruptThreshold to the job when it
calls Run() on the job.
This replaces the start_time, end_time, and previous_end_time columns on
the session_state table with a time range column that records when the
state was active. This allowed for dropping several constraints on the
session_state table that were used to ensure that states for a session
did not overlap. Instead this can be ensured with a single exclusion
constraint. By reducing the number of constraints, it improves the
performance for a number of queries involved with session state. Most
notably when resolving delete cascades of resources related to sessions,
such as targets, where as part of the delete transaction large numbers
of sessions get canceled. Similarly, when sessions get deleted, and the
corresponding session_state rows are deleted, these constraints all must
be checked during the transaction.
The domain layer continues to expose StartTime and EndTime fields on
session.State. However it drops PreviousEndTime, which was never used by
anything calling the domain layer.
See:
https://www.postgresql.org/docs/current/ddl-constraints.html#DDL-CONSTRAINTS-EXCLUSIONhttps://www.postgresql.org/docs/current/rangetypes.html
This removes the Cartesian product in the DB for GrantsForUser in favor
of returning the actual grant scope information and dealing with it in
application code.
(cherry picked from commit 913cd536823d73994acc4f4d833208c1cc05f91b)
* internal/session: update variable name in test
This was missed in #5071
* internal/tests/cluster: skip failing tests
These tests will need to be fixed or removed. It is not
clear why they are failing, but they are always failing.
* Makefile: try upgrading golangci-lint
* internal/host/plugin: fix flaky test
Hosts contain IP addresses in string form, whose
order is not defined.
* refact: to use go-dbw CreateItems, DeleteItems
Latest version of go-dbw updated the signature of
CreateItems(...) and DeleteItems(...) on a happy
note this latest version of go-dbw actually does
batch inserts and updates for this functions.
* all: regenerate protos with new protoc-gen-go version
Several session.Repository methods need to encrypt or decrypt fields
when persisting or retrieving a Session. These encrypt/decrypt
operations where being performed while within a database transaction.
The first step in the encrypt/decrypt operation is to retrieve the
kms.Wrapper. While the wrappers get cached in memory, the retrieval
still results in a database query to check that the kms collection
version has not changed. This query happens with a separate db.Reader,
and therefore uses a separate database connection from the transaction
being used by the session.Repository.
If the controllers are configured with a relatively low
`max_open_connections`, and be under sufficient load interactive with
sessions, the controller can consume all of the available database
connections, and get stuck with open transactions while waiting for a
database connection to execute the kms related queries. This can be see
by checking pg_stat_activity and seeing transactions stuck in `idle in
transaction` with a wait_event of ClientRead.
This commit changes all of the Repository methods the perform any
encryption or decryption operations, including the retrieval of the
kms.Wrapper, outside of a database transaction. An alternative approach
would be to unsure that the kms.Wrapper retrieval happens within the
same database transaction. However, it seems preferable to avoid doing
these relatively expensive operations within a transaction when it is
not necessary.
As part of this fix, it also eliminates some logic that is no longer
needed. During previous would with KMS wrapping, the method of how
sessions were encrypted was changed, and some logic to lazily re-encrypt
sessions was introduced. However, given that terminated sessions are
deleted, and the KMS changes was introduced several versions ago
(0.13.0), this lazy re-encrypt logic should no longer be necessary.
Refs: edd323b73a
See:
https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-ACTIVITY-VIEWhttps://www.postgresql.org/docs/current/runtime-config-client.html#GUC-IDLE-IN-TRANSACTION-SESSION-TIMEOUThttps://developer.hashicorp.com/boundary/docs/configuration/controller#max_open_connections
When closing orphaned connections and checking if a session is not
active, the repository was opening a transaction, but then calling the
original reader instead of using the reader/writer from the transaction.
This would result in a transaction being opend, then a separate
transaction being used to run the query, and finally the original
transaction would commit without any statements being executed.
Co-authored-by: Sorawis Nilparuk (Bo) <sorawis.nilparuk@hashicorp.com>
* chore: update deps
* tests: update sqlmock tests for breaking chg
* tests (handlers): sort grants in checkEqualGrants(...)
sort the expected and got grants in
checkEqualGrants(...) to make the test more
deterministic.
* tests (asciicast): update backspace enc in TestEventMarshal(...)
This commit modifies a log line produced in the WorkerStatusReport function when closing
orphaned connections that appears to incorrectly label a worker identifier as a controller
identifier.
Signed-off-by: David Bond <davidsbond93@gmail.com>
The previous sort would run the risk of missing items which
collide on update or create times because of the compound
comparison using strictly less than for both the timestamp
and the public_id. Sorting both descending avoids thi issue.
* fix (events): convert multierror to stdlib error
multierror errors will be converted to a stdlib error (because
multierror doesn't support json marshaling).
* refact (errors): convert from multierror to stdlib errors.Join(...)
* refact: convert from pkg/errors to stdlib error wrapping
* fix (auth/ldap): fix which DEK is used for oplog during authenticate
* chore: update go-kms-wrapping dep for new extras/kms changes
The latest version supports setting a table name prefix for a kms
which is needed to fix our oplog DEK problem
* chore: update extras/kms dep
This latest version supports an optional TableNamePrefix which is
required to fix our existing oplog DEK issue.
* fix (oplog): schema oplog DEK fix
These changes will allow use to store the oplog DEK in it's own
set of tables, so they won't get deleted when a scope is deleted
* fix (kms): move oplog DEK into its own internal kms
* tests: update tests to be compatible with oplog fix
* tests: remove migration tests which are incompatible with oplog fix
These tests no longer work because they are missing the required
schema changes for the oplog fix
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Adding explicit MPL license for sub-package
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Updating the license from MPL to Business Source License
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl.
* Update copyright file headers to BUS-1.1
* Rerun make gen
This will pick up the last of the license changes
* Revert "Rerun make gen"
This reverts commit dbe0968bffd92745d8663c6940debae098395a6b.
* Remove copywrite from generation script
The current version of copywrite is not sophisticated enough
to only apply the copywrite changes to certain subdirectories,
so to avoid accidentally relicensing our api and sdk modules,
we will stop running the copywrite script until such time
that it can do the right thing.
* License protobuf API as MPL
The protobuf API files are used to generate the SDK,
since we want the SDK to be MPL, the protobuf files
must also be MPL. This extends to all protobuf files
whose sources end up in the SDK directory, including:
- internal/proto/controller/api
- internal/proto/controller/custom_options
- internal/proto/plugin
It also includes the generated service directory in
internal/gen/controller/api/services, since they are
generated from the protobuf sources used for the SDK.
* Change website license file name
This makes the naming scheme consistent with the rest
of the repository.
* Add note on licensing to README
* Set copywrite license to BUSL
* Revert "Add note on licensing to README"
This reverts commit 4d865a91a7a4f11605f0dc4609355d7e3c3cfffe.
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
This adds a new Job type that is for generic session state monitoring. Monitoring sessions does not include any information about specific connections.
- Refactor the Target domain repository functions so that they no longer return 4 different types.
`Target` struct now contains `HostSource` & `CredentialSources` with Getters & Setters
- Extending `Target` to contain `HostSource` & `CredentialSources` enables `funcs` to just return `Target` instead of multiple return types.
This also allows easy future refactors and extensions of `Target`
Hugo
Sorawis Nilparuk (Bo)
Ryan Derr
Johan Brandhorst-Satzkorn
daniandDamian Debkowski
Michael Gaffney
Timothy Messier
Jeff Mitchell
Jim
Louis Ruch
David Bond
Todd
Michael Milton
hashicorp-copywrite[bot]andJohan Brandhorst-Satzkorn
Elim Tsiagbey