The cluster listener supports a `max_request_duration` option, and
defaults to 90 seconds. However, this configuration was not being use
when processing grpc requests from the listener. This meant that
requests from worker to controller could hang indefinitely and
contributed to the recently fixed issue with database connections being
stuck in `idle in transaction`.
This commit adds a grpc interceptor that ensures each request has a
deadline set to the max request duration for the listener.
Refs: 29cd7bc9d6
Several session.Repository methods need to encrypt or decrypt fields
when persisting or retrieving a Session. These encrypt/decrypt
operations where being performed while within a database transaction.
The first step in the encrypt/decrypt operation is to retrieve the
kms.Wrapper. While the wrappers get cached in memory, the retrieval
still results in a database query to check that the kms collection
version has not changed. This query happens with a separate db.Reader,
and therefore uses a separate database connection from the transaction
being used by the session.Repository.
If the controllers are configured with a relatively low
`max_open_connections`, and be under sufficient load interactive with
sessions, the controller can consume all of the available database
connections, and get stuck with open transactions while waiting for a
database connection to execute the kms related queries. This can be see
by checking pg_stat_activity and seeing transactions stuck in `idle in
transaction` with a wait_event of ClientRead.
This commit changes all of the Repository methods the perform any
encryption or decryption operations, including the retrieval of the
kms.Wrapper, outside of a database transaction. An alternative approach
would be to unsure that the kms.Wrapper retrieval happens within the
same database transaction. However, it seems preferable to avoid doing
these relatively expensive operations within a transaction when it is
not necessary.
As part of this fix, it also eliminates some logic that is no longer
needed. During previous would with KMS wrapping, the method of how
sessions were encrypted was changed, and some logic to lazily re-encrypt
sessions was introduced. However, given that terminated sessions are
deleted, and the KMS changes was introduced several versions ago
(0.13.0), this lazy re-encrypt logic should no longer be necessary.
Refs: edd323b73a
See:
https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-ACTIVITY-VIEWhttps://www.postgresql.org/docs/current/runtime-config-client.html#GUC-IDLE-IN-TRANSACTION-SESSION-TIMEOUThttps://developer.hashicorp.com/boundary/docs/configuration/controller#max_open_connections
* Auto generate the permissions pages.
* Add add- set- remove- auto descriptions. Hide scope line when no scope provided
* Document the resource info struct
AuthToken selection involves checking to seeing if the one we are picking has been canceled or expired on the boundary controller. If we are only searching locally and not performing a refresh, there is no need to check if the tokens have expired or not.
* Add hidden ferry pause and resume functions
* add check for 200 status
* include pause/resume status in ferry status, fix output
* put error string back to previous state
When closing orphaned connections and checking if a session is not
active, the repository was opening a transaction, but then calling the
original reader instead of using the reader/writer from the transaction.
This would result in a transaction being opend, then a separate
transaction being used to run the query, and finally the original
transaction would commit without any statements being executed.
Co-authored-by: Sorawis Nilparuk (Bo) <sorawis.nilparuk@hashicorp.com>
* Add ListResolvableAliases to the user service
There are a few non common ways this method uses our ACL
and grant system. A user is able to list resolvable
aliases for another user if they are granted permission
to do so. That means we had to load the grants for
the user being listed for and not the requester when
determining if an alias is able to resolve to a destination
for which the requested user has permission.
Similarly, the grant hash used for the list pagination
is required to be that of the user being listed and not
the requester.
The worker_filter column was dropped from the
credential_vault_credential_private view in the
56/02_add_data_key_foreign_key_references.up.sql migration.
Any credential renewal/revocations that relied on the worker filter
would not successfully renew.
* internal/gen: better API docs
Add various pieces of metadata that can be used by the API docs renderer.
Improve the service labels by specifying explicit tags.
Specify error responses. Specify version at generation time.
* internal/gen: add better errors
The default error entry is now used to cover all errors, as
adding a new error for every possible status code was a bit much.
This also properly includes the real error type now!
* Add various pieces of metadata that can be used by the API docs renderer. Improve the service labels by specifying explicit tags.
* make auth method API docs
* make API docs better
* make auth methods API docs nicer
* internal/gen: better attribute examples
Replace Output Only comments with a field behavior annotation in accounts.
Mark some fields as required.
Add elaborate description for attributes.
* Make API docs better
* Make API docs better
* Make API docs better
* internal/proto: regenerate files
* internal/proto: finish off account service
* Make API docs better
* internal/proto: add better API docs for aliases
* Tidy up service descriptions
* Added description for Boundary controller API doc
* internal/gen: add better API intro
Copy parts of the API docs into the description
* sdk: add openapi dependency
* internal/gen: move global API doc to separate file
* internal/gen: remove required annotations
Unfortunately, while these help with the creating of resources, they still
appear required in contexts where they are not, such as when
updating a resource.
* Review comments
---------
Co-authored-by: jigar-hashicorp <jigar.shah@hashicorp.com>
Co-authored-by: Isaac Christian (he/him) <112639904+isaacchristian@users.noreply.github.com>
- add `LocalStorageState` to worker status reports
- update worker service handler to process `LocalStorageState`. If is empty for workers older workers, we set the `LocalStorageState` to `unknown`
- update field mask paths to support `LocalStorageState`
- Add new `local_storage_state` column to `server_worker` table which keeps track of the state of the local disk space.
- The valid values for `local_storage_state` are:
-- `available`: The worker local storage state is at an acceptable state
-- `lowStorage`: The worker is below the minimum threshold for local storage
-- `criticallyLowStorage`: The worker local storage state is below the critical minimum threshold for local storage
-- `outOfStorage`: The worker is out of local disk space
-- `notConfigured`: The worker does not have a local storage path configured
-- `unknown`: The default local storage state of a worker. Used when the local storage state of a worker is not yet known
Timothy Messier
Johan Brandhorst-Satzkorn
Todd
Irena Rindos
Michael Milton
Kerim Satirli
Stan Ryzhov
Cameron Perera
Lisbet Alvarez
Zhihe Li
Michael LiandElim Tsiagbey
Carlos Alcaide Corvo
dani
Louis Ruch
Damian Debkowski