This changes the periodic job that deletes terminated sessions to delete
the those sessions in batches. Prior to this, a single SQL DELETE
command was used to delete all qualifying terminated sessions. In
systems under heavy load, the statement could timeout leading to a
situation where no qualifying terminated sessions were ever being
deleted.
The scheduler monitors running jobs to make sure they are not hung. If
the status of a running job does not change after a duration (the
interruptThreshold in the scheduler), the scheduler kills the job. This
change passes the scheduler's interruptThreshold to the job when it
calls Run() on the job.
This replaces the start_time, end_time, and previous_end_time columns on
the session_state table with a time range column that records when the
state was active. This allowed for dropping several constraints on the
session_state table that were used to ensure that states for a session
did not overlap. Instead this can be ensured with a single exclusion
constraint. By reducing the number of constraints, it improves the
performance for a number of queries involved with session state. Most
notably when resolving delete cascades of resources related to sessions,
such as targets, where as part of the delete transaction large numbers
of sessions get canceled. Similarly, when sessions get deleted, and the
corresponding session_state rows are deleted, these constraints all must
be checked during the transaction.
The domain layer continues to expose StartTime and EndTime fields on
session.State. However it drops PreviousEndTime, which was never used by
anything calling the domain layer.
See:
https://www.postgresql.org/docs/current/ddl-constraints.html#DDL-CONSTRAINTS-EXCLUSIONhttps://www.postgresql.org/docs/current/rangetypes.html
This removes the Cartesian product in the DB for GrantsForUser in favor
of returning the actual grant scope information and dealing with it in
application code.
(cherry picked from commit 913cd536823d73994acc4f4d833208c1cc05f91b)
* internal/session: update variable name in test
This was missed in #5071
* internal/tests/cluster: skip failing tests
These tests will need to be fixed or removed. It is not
clear why they are failing, but they are always failing.
* Makefile: try upgrading golangci-lint
* internal/host/plugin: fix flaky test
Hosts contain IP addresses in string form, whose
order is not defined.
* refact: to use go-dbw CreateItems, DeleteItems
Latest version of go-dbw updated the signature of
CreateItems(...) and DeleteItems(...) on a happy
note this latest version of go-dbw actually does
batch inserts and updates for this functions.
* all: regenerate protos with new protoc-gen-go version
Several session.Repository methods need to encrypt or decrypt fields
when persisting or retrieving a Session. These encrypt/decrypt
operations where being performed while within a database transaction.
The first step in the encrypt/decrypt operation is to retrieve the
kms.Wrapper. While the wrappers get cached in memory, the retrieval
still results in a database query to check that the kms collection
version has not changed. This query happens with a separate db.Reader,
and therefore uses a separate database connection from the transaction
being used by the session.Repository.
If the controllers are configured with a relatively low
`max_open_connections`, and be under sufficient load interactive with
sessions, the controller can consume all of the available database
connections, and get stuck with open transactions while waiting for a
database connection to execute the kms related queries. This can be see
by checking pg_stat_activity and seeing transactions stuck in `idle in
transaction` with a wait_event of ClientRead.
This commit changes all of the Repository methods the perform any
encryption or decryption operations, including the retrieval of the
kms.Wrapper, outside of a database transaction. An alternative approach
would be to unsure that the kms.Wrapper retrieval happens within the
same database transaction. However, it seems preferable to avoid doing
these relatively expensive operations within a transaction when it is
not necessary.
As part of this fix, it also eliminates some logic that is no longer
needed. During previous would with KMS wrapping, the method of how
sessions were encrypted was changed, and some logic to lazily re-encrypt
sessions was introduced. However, given that terminated sessions are
deleted, and the KMS changes was introduced several versions ago
(0.13.0), this lazy re-encrypt logic should no longer be necessary.
Refs: edd323b73a
See:
https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-ACTIVITY-VIEWhttps://www.postgresql.org/docs/current/runtime-config-client.html#GUC-IDLE-IN-TRANSACTION-SESSION-TIMEOUThttps://developer.hashicorp.com/boundary/docs/configuration/controller#max_open_connections
When closing orphaned connections and checking if a session is not
active, the repository was opening a transaction, but then calling the
original reader instead of using the reader/writer from the transaction.
This would result in a transaction being opend, then a separate
transaction being used to run the query, and finally the original
transaction would commit without any statements being executed.
Co-authored-by: Sorawis Nilparuk (Bo) <sorawis.nilparuk@hashicorp.com>
* chore: update deps
* tests: update sqlmock tests for breaking chg
* tests (handlers): sort grants in checkEqualGrants(...)
sort the expected and got grants in
checkEqualGrants(...) to make the test more
deterministic.
* tests (asciicast): update backspace enc in TestEventMarshal(...)
This commit modifies a log line produced in the WorkerStatusReport function when closing
orphaned connections that appears to incorrectly label a worker identifier as a controller
identifier.
Signed-off-by: David Bond <davidsbond93@gmail.com>
The previous sort would run the risk of missing items which
collide on update or create times because of the compound
comparison using strictly less than for both the timestamp
and the public_id. Sorting both descending avoids thi issue.
* fix (events): convert multierror to stdlib error
multierror errors will be converted to a stdlib error (because
multierror doesn't support json marshaling).
* refact (errors): convert from multierror to stdlib errors.Join(...)
* refact: convert from pkg/errors to stdlib error wrapping
* fix (auth/ldap): fix which DEK is used for oplog during authenticate
* chore: update go-kms-wrapping dep for new extras/kms changes
The latest version supports setting a table name prefix for a kms
which is needed to fix our oplog DEK problem
* chore: update extras/kms dep
This latest version supports an optional TableNamePrefix which is
required to fix our existing oplog DEK issue.
* fix (oplog): schema oplog DEK fix
These changes will allow use to store the oplog DEK in it's own
set of tables, so they won't get deleted when a scope is deleted
* fix (kms): move oplog DEK into its own internal kms
* tests: update tests to be compatible with oplog fix
* tests: remove migration tests which are incompatible with oplog fix
These tests no longer work because they are missing the required
schema changes for the oplog fix
* Adding explicit MPL license for sub-package.
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Adding explicit MPL license for sub-package
This directory and its subdirectories (packages) contain files licensed with the MPLv2 `LICENSE` file in this directory and are intentionally licensed separately from the BSL `LICENSE` file at the root of this repository.
* Updating the license from MPL to Business Source License
Going forward, this project will be licensed under the Business Source License v1.1. Please see our blog post for more details at https://hashi.co/bsl-blog, FAQ at www.hashicorp.com/licensing-faq, and details of the license at www.hashicorp.com/bsl.
* Update copyright file headers to BUS-1.1
* Rerun make gen
This will pick up the last of the license changes
* Revert "Rerun make gen"
This reverts commit dbe0968bffd92745d8663c6940debae098395a6b.
* Remove copywrite from generation script
The current version of copywrite is not sophisticated enough
to only apply the copywrite changes to certain subdirectories,
so to avoid accidentally relicensing our api and sdk modules,
we will stop running the copywrite script until such time
that it can do the right thing.
* License protobuf API as MPL
The protobuf API files are used to generate the SDK,
since we want the SDK to be MPL, the protobuf files
must also be MPL. This extends to all protobuf files
whose sources end up in the SDK directory, including:
- internal/proto/controller/api
- internal/proto/controller/custom_options
- internal/proto/plugin
It also includes the generated service directory in
internal/gen/controller/api/services, since they are
generated from the protobuf sources used for the SDK.
* Change website license file name
This makes the naming scheme consistent with the rest
of the repository.
* Add note on licensing to README
* Set copywrite license to BUSL
* Revert "Add note on licensing to README"
This reverts commit 4d865a91a7a4f11605f0dc4609355d7e3c3cfffe.
---------
Co-authored-by: hashicorp-copywrite[bot] <110428419+hashicorp-copywrite[bot]@users.noreply.github.com>
Co-authored-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
This adds a new Job type that is for generic session state monitoring. Monitoring sessions does not include any information about specific connections.
- Refactor the Target domain repository functions so that they no longer return 4 different types.
`Target` struct now contains `HostSource` & `CredentialSources` with Getters & Setters
- Extending `Target` to contain `HostSource` & `CredentialSources` enables `funcs` to just return `Target` instead of multiple return types.
This also allows easy future refactors and extensions of `Target`
* refactor (kms): add GetWrapperer for RewrapFn parameter
* tests (kms): add MockWrapper and MockGetWrapperer
Make it a bit easier to write unhappy path tests which depend on
a Wrapper or GetWrapperer
daniandDamian Debkowski
Michael Gaffney
Timothy Messier
Jeff Mitchell
Johan Brandhorst-Satzkorn
Jim
Irena Rindos
Louis Ruch
David Bond
Todd
Michael Milton
hashicorp-copywrite[bot]andJohan Brandhorst-Satzkorn
Elim Tsiagbey
Danielle