The cluster listener supports a `max_request_duration` option, and
defaults to 90 seconds. However, this configuration was not being use
when processing grpc requests from the listener. This meant that
requests from worker to controller could hang indefinitely and
contributed to the recently fixed issue with database connections being
stuck in `idle in transaction`.
This commit adds a grpc interceptor that ensures each request has a
deadline set to the max request duration for the listener.
Refs: 29cd7bc9d6
Several session.Repository methods need to encrypt or decrypt fields
when persisting or retrieving a Session. These encrypt/decrypt
operations where being performed while within a database transaction.
The first step in the encrypt/decrypt operation is to retrieve the
kms.Wrapper. While the wrappers get cached in memory, the retrieval
still results in a database query to check that the kms collection
version has not changed. This query happens with a separate db.Reader,
and therefore uses a separate database connection from the transaction
being used by the session.Repository.
If the controllers are configured with a relatively low
`max_open_connections`, and be under sufficient load interactive with
sessions, the controller can consume all of the available database
connections, and get stuck with open transactions while waiting for a
database connection to execute the kms related queries. This can be see
by checking pg_stat_activity and seeing transactions stuck in `idle in
transaction` with a wait_event of ClientRead.
This commit changes all of the Repository methods the perform any
encryption or decryption operations, including the retrieval of the
kms.Wrapper, outside of a database transaction. An alternative approach
would be to unsure that the kms.Wrapper retrieval happens within the
same database transaction. However, it seems preferable to avoid doing
these relatively expensive operations within a transaction when it is
not necessary.
As part of this fix, it also eliminates some logic that is no longer
needed. During previous would with KMS wrapping, the method of how
sessions were encrypted was changed, and some logic to lazily re-encrypt
sessions was introduced. However, given that terminated sessions are
deleted, and the KMS changes was introduced several versions ago
(0.13.0), this lazy re-encrypt logic should no longer be necessary.
Refs: edd323b73a
See:
https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-ACTIVITY-VIEWhttps://www.postgresql.org/docs/current/runtime-config-client.html#GUC-IDLE-IN-TRANSACTION-SESSION-TIMEOUThttps://developer.hashicorp.com/boundary/docs/configuration/controller#max_open_connections
When closing orphaned connections and checking if a session is not
active, the repository was opening a transaction, but then calling the
original reader instead of using the reader/writer from the transaction.
This would result in a transaction being opend, then a separate
transaction being used to run the query, and finally the original
transaction would commit without any statements being executed.
Co-authored-by: Sorawis Nilparuk (Bo) <sorawis.nilparuk@hashicorp.com>
This adds support for tracking and reporting monthly active users for
the purpose of billing. It adds a new API endpoint,
`/v1/billing:monthly-active-users` and new cli command, `boundary
billing monthly-active-users` that can be used to view the monthly
active user counts.
This adds a new accumulating fact table to the data warehouse for auth
tokens. This fact table is used to calculate the number of active users
within a month, where an active user is a user that has at least one
issued auth token within the time range of the month.
This updates the billing repository to use timestmap.Timestamp when
constructing the queries to retrieve monthly active users. This allows
for specifying NegativeInfinity to easily retrieve all monthly active
user counts.
When converting a timestamp.Timestamp for use in SQL, this now correctly
converts the NegativeInfinity and PositiveInfinity constants into the
corresponding `-infinity` and `infinity` SQL constants.
These functions will ensure that UTC time is used for billing queries,
eliminating the need for the repository to set the timezone. This also
means the repository no longern needs a writer.
This adds two sql functions that can be used to access the views to
retrieve monthly active user counts. The functions ensure that any time
calculates are performed in UTC, so that the month bounds align with the
start of the month in UTC.
1. hcp_billing_monthly_active_users_last_2_months() will return monthly
active users for the current and previous month.
2. hcp_billing_monthly_active_users_all() will return monthly active
users for all months. This function can also be provided a start time
and or end time to filter the results to a specific range.
This test started failing now that the data warehouse records a auth
token fact when an auth token is issued and it requires that the auth
token has a user associated with it.
Since this test is testing the interaction with the database directly,
and not via the repository, it by-passes the application checks that
would prevent an auth token without an associated user being inserted.
This test is updated to meet this same pre-condition prior to attempting
to insert into the database.
Now that there is an auth token fact table in the data warehouse, it is
not possible to insert an auth_token with an old public_id even after
that auth_token was deleted, since it now causes a constraint violation
on the data warehouse table.
With the addition of an after insert trigger to the auth_token table,
entries are added to the wh_user_dimension table when auth token's are
inserted with the 'token issued' status. This caused a number of tests
to fail that assumed the user dimension table would be empty prior to
inserting sessions. This introduces a number of changes to the tests:
- In the test setup, auth tokens are not created with an explicit
expiration time and status. The expiration time ensures that the
tokens are 'valid' for the test. The status allows test to make
assertions on tokens in different statuses, and allow the status to
change during the test.
- Tests make the correct assertions on the number of entries in the user
dimension table after test setup.
- User dimension tests modify a user after the standard setup to ensure
that a new user dimension is added when a session is inserted.
This adds a function and after delete trigger to the auth_token table to
ensure that a corresponding auth token fact is updated with the deleted
time and sets the auth token valid time range. The data warehouse will
only be updated if the token that was deleted is in the 'token issued'
status.
This adds a function and update trigger to auth_token to ensure that:
1. If the auth token is updated to the issued state, a auth token fact
is added to the warehouse.
2. If the auth token's approximate last access time is updated, the
corresponding auth token fact is updated with the new time.
Previously the function used to upsert into the user dimension required
both an auth token id, and a user id. However, the user id can be
determined from the auth token, in fact it already was via the
whx_user_dimension_source view. This refactor removes the user id as an
argument, since it is not needed and will not be directly available when
this function is used when populating the new auth token accumulating
fact.
When running the sqltests, sometimes it is helpful to see the full
database container logs, especially when there is an error in a
migration or database setup. However, it also makes it difficult to see
the test results.
This adds a new variable to the makefile to easily toggle the output of
these logs on or off. To run the tests without printing the dtabase logs
use:
DB_LOGS=0 make test
To enable the logs use:
DB_LOGS=1 make test
By default the logs are not printed.
Some of the prepared statement identifies used in these tests exceeded
the identifier character limit. This results in postgres truncating the
name, which is fine for a test, but does result in some noise in the
test output. This change shortens the identifies to eliminate noise like
the following:
psql:/test/credential/vault/credential_vault_library_ssh_private_key_mapping_override.sql:49: NOTICE: identifier "insert_credential_vault_library_ssh_private_key_mapping_override" will be truncated to "insert_credential_vault_library_ssh_private_key_mapping_overrid"
The tests should no longer create notices like these.
After updating `boundary dev` to use postgres:12 by default, it was
noticed that the responsiveness of boundary was noticeably slower. This
is due to the introduction of JIT being enabled by default in
postgres:12. JIT seems to incur noticeable overhead without providing
benefits, at least for the instance used for `boundary dev`.
It is also worth noting that for postgres:14 and newer, JIT does not get
triggered for the same queries since they seem to result in a cheaper
query plan that does not exceed the default threshold for JIT.
See:
https://www.postgresql.org/docs/current/jit.html
ICU-12283
This adds a Access-Control-Expose-Headers response header to an Options
request to allow the rate limiting related headers to be accessible via
requests that are subject to CORS.
Normally the tests are run on a host machine with a docker container for
the database. The docker container should be accessible via 127.0.0.1.
However there might be cases where the test execution is run in a docker
container, or the postgres server is configured to use a different
docker network, in which case the container would be accessible via some
other dns name or ip. In these cases, users can now set the TEST_DB_HOST
environment variable so the tests can connect to the postgres server.
This increases the default rate limits to:
- 1,500 reqs / 30 seconds for list actions in total
- 1,500 reqs / 30 seconds for list actions per IP address
- 150 reqs / 30 seconds for list actions per auth token
And for all other actions:
- 30,000 reqs / 30 seconds for list actions in total
- 30,000 reqs / 30 seconds for list actions per IP address
- 3,000 reqs / 30 seconds for list actions per auth token
This seems to be a reasonable default for a small to medium size cluster
based on some scale testing.
(cherry picked from commit 237c8b6084f42655edaba49055e511163fdf1c53)
Go 1.21.5 fixes a few vulnerabilities. Most notably for boundary is a
fix in net/http to limit chunked data overhead.
See: https://groups.google.com/g/golang-announce/c/iLGK3x6yuNo
(cherry picked from commit 575cd9d63b1ac4787d048d86b0b209c61a7f3946)
Add support for rate limiting
This introduces rate limiting for the Controller API endpoints. Limits
can be set separately for each combination of resource and action. And
each combination has a limit per auth-token, IP address, and in total.
By default each list action for all resources has a limit of 120
requests per minute. All other actions for all resources have a limit of
3,000 requests per minute. This can be overridden by including
`api_rate_limit` stanzas in the controller configuration. For example to
set a limit on target list requests per auth token:
controller {
api_rate_limit {
resources = ["target"]
actions = ["list"]
per = "auth-token"
limit = 10
period = "1s"
}
// other controller configuration
}
Or to completely override all of the defaults:
controller {
// total limit for all resources and actions
api_rate_limit {
resources = ["*"]
actions = ["*"]
per = "total"
limit = 500
period = "1s"
}
// limit for ip addresses to all resources and actions
api_rate_limit {
resources = ["*"]
actions = ["*"]
per = "ip-address"
limit = 100
period = "1s"
}
// Limit for auth tokens to all resources and actions
api_rate_limit {
resources = ["*"]
actions = ["*"]
per = "auth-token"
limit = 50
period = "1s"
}
// other controller configuration
}
All rate limiting can be disabled by setting the
`api_rate_limit_disable` configuration variable:
controller {
api_rate_limit_disable = true
// other controller configuration
}
When rate limiting is enabled, boundary will create quotas to track the
number of requests used within a time period. Boundary will reserve a
set amount of memory for tracking quotas. This ensures that boundary
does not consume too much memory if there is a sudden burst of requests
from many different clients. However this does mean if boundary is
unable to store a quota, it will limit the request with a 503 HTTP
status code. The amount of memory can be controlled with the
`api_rate_limit_max_quotas` configuration variable. This can also be
monitored via two new metrics:
- `boundary_controller_api_ratelimiter_quota_storage_capacity`
- `boundary_controller_api_ratelimiter_quota_storage_usage`
Clients making requests to the controller API can inspect HTTP response
headers to understand the configured limits and current usage. Each
response will contain the following headers:
- RateLimit: Provides the current limit, number of remaining requests,
and the time the quota will reset for the limit that is closest to
being exhausted for the requested resource and action.
- RateLimit-Policy: Describes the limits for the requested resource and
action.
If a request is limited, the client will receive a 429 HTTP status code,
and the Retry-After header will be set to the number of seconds the
client should wait before attempting the request again.
Previously this would truncate to the nearest integer, which could
result in clients retrying before the quota has reset, and getting rate
limited again. This also includes an update from go-rate that will round
up when reporting the reset time for the RateLimit header.
Note that in the case of a 503, the Retry-After header was already being
set correctly.
See: https://github.com/hashicorp/go-rate/pull/8
If the api.Client receives a 429 or 503 along with a Retry-After header,
the client will automatically wait the amount of time specified, and
then re-try the request.
A sys event will be emitted when the controller is constructed, and if
the rate limits are changed via a SIGHUP. The event contains the details
of the rate limits for each combination of resource and action. This
event can be used by operators to confirm the limits are set as they
would expect based on the controller config.
This provides two prometheus metrics to monitor the rate limiter:
boundary_controller_api_ratelimiter_quota_storage_capacity:
This is a gague that shows the total number of quotas that can be stored
at a given moment by the rate limiter.
boundary_controller_api_ratelimiter_quota_storage_usage:
This is a gague that shows the number of quotas currently being stored
by the limiter. If this becomes equal to the capacity, requests that
would result in a new quota being created will get limited and receive a
429 status code.
This registers the ActionSets for each resource when the handler
packages are initialized. This then allows other packages to retrieve
the ActionSets for a given resource from the action package.
The initial consumer of this will be the ratelimit package so that it
can programmatically enumerate all of the valid combinations of a
resource and action.
This leaves a number of TODO to refactor the handler packages to also
retrieve the action sets from the registered actions instead of using a
package variable. This is left as a TODO since there is some
inconsistencies in how these package variables are used between the
handlers. For example a number of packages use a map of subtypes to
ActionSets which is then used withing the handler implementation. Others
use a map, but don't seem to use it at all. And others just use an
ActionSet.
This switches the implementation of an ActionSet from a slice to a map
to ensure the elements in an ActionSet are unique. This allows for other
packages to perform set operations such as a union or difference without
needing to account for duplicates.
The nhooyr/websocket library now has a release with the new default
behavior that removes the message size limit. This allows us to switch
off of the fork and back to the upstream.
In the same release that includes the fix we need for the boundary
client, the wspb helper package was removed from websocket. This package
contained some functions to read and write protobuf messages. However,
it was using the deprecated github.com/golang/protobuf/proto library. To
avoid this dependancy in the library they opted to just remove wspb.
Thus this provides a replacement implementation that uses
google.golang.org/protobuf/proto. Notably the newer proto library seems
to lack an equivalent of proto.NewBuffer so writes to the websocket and
not able to leverage a sync.Pool of byte.Buffers to reduce allocations.
Refs: 88bb253554
See: https://github.com/nhooyr/websocket/commit/ac385120c6e34fa6584f3856d5db949a21bbb65e
When generating the JSON for submitting the github comment, if the
message was too large, the workflow would fail with the following error:
/usr/bin/jq: Argument list too long
This fix switches to using the `--rawfile`, which allows jq to process
the large comment.
When the boundary client establishes a connection to the websocket used
to communicate with a worker, it was configured with a max websocket
message size of 32KiB. This was the default behavior for v1.8.7 of
nhooyr/websocket. However, there are cases where the worker can send
messages to the client that are larger than 32KiB. When this happens the
client returns an error and terminates the connection.
Notably this seems to only currently impact sessions made to SSH
Targets, although it could theoretically impact any target type
depending on the details of how the worker performs the writing of the
messages. The issue can be reproduced by attempting to scp a large
enough file from an SSH Target to the client. In this case the
connection gets terminated and results in a partial transfer of the
file.
This message size limit appears to be unnecessary, and the default
behavior is changing in v1.8.8 of nhooyr/websocket to make the message
size unlimited. Since this version is not yet released, this commit
switches the dependency to a fork with a version that is v1.8.7 plus the
patch to disable the limit. Once v1.8.8 is release this dependency can
be set back.
See:
https://github.com/nhooyr/websocket/pull/254https://github.com/nhooyr/websocket/pull/256
This updates the version make target to use the version/VERSION file in
the same manner as the build system. This ensures that the VERSION file
is the single source of truth for the version number.
This moves the contents of the readme used for hashicorp/boundary on
docker.io/registry.hub.docker.com to a separate readme file from the one that
documents how to build docker images. It also makes some minor fixups
and updates the docker.io readme, must notably updating the minimum
version of postgres that boundary requires.
This provides a new storage.FS implementation that expands the on
fstest.MemFS by allowing tests to mark the filesystem as out of space.
When this happens any write/create operation will start returning an
error indicating that the filesystem is out of space.
fix(session): List performance with large number of session connections
This includes the backported migration and a migration to ensure the next
release has the correct view.
As part of the performance improvements for listing sessions for v0.9.0,
the view used to retrieve sessions for listing was updated to no longer
include a join on session_connection. However, a later migration that
was adding additional columns to this view inadvertently re-added the
join and session_connection columns. This resulted in requests to list
sessions performing this join, and returning additional rows for each
connection in a session. The application would ignore this additional
data, since it is no longer returned via the API. This would be
particularly impactful in cases where there are a large number of
connections per session, as this would greatly increase the amount of
data returned by the database.
This fix removes the session_connection join from this view, which
should restore the original performance improvements even in cases where
there is a large number of session connections per session.
See: https://github.com/hashicorp/boundary/pull/2160
Ref: 32070678dc
Fixes: 1e3c941be1
(cherry picked from commit 7aef341682e65b42c1b919e03936bdf81e26e87c)
As part of the performance improvements for listing sessions for v0.9.0,
the view used to retrieve sessions for listing was updated to no longer
include a join on session_connection. However, a later migration that
was adding additional columns to this view inadvertently re-added the
join and session_connection columns. This resulted in requests to list
sessions performing this join, and returning additional rows for each
connection in a session. The application would ignore this additional
data, since it is no longer returned via the API. This would be
particularly impactful in cases where there are a large number of
connections per session, as this would greatly increase the amount of
data returned by the database.
This fix removes the session_connection join from this view, which
should restore the original performance improvements even in cases where
there is a large number of session connections per session.
See: https://github.com/hashicorp/boundary/pull/2160
Ref: 32070678dc
Fixes: 1e3c941be1
This removes the minimum on width and height, instead it will just used
the largest width, height seen as the initial size for the asciicast.
This also provides the ability to still set a minimum via options to
convert.ToAsciicast if the caller has a known minimum size. Also note
that in cases where there is no size information, ie if there was no pty
request, this will still pick a default size of 80x24.
For requests to the target authorize session endpoint we support passing
a target name instead of an id. This means that `req.GetId()` might not
be the target's public id. Since the first thing authorize session does
includes retrieving the target as part of the authorization checks, this
subtype check can leverage the public id of the target instead of using
the request.
Fixes: 6178dd516e
When an SSH channel is used for a file transfer, the summary will report
the detected direction of the file transfer as either an upload or
download, relative to the end client.
The session metadata is provided all at once when a new bsr is opened.
While the session recording metadata records metadata about a recording
while the recording is happening, notably it records each time a new
connection occurs.
The nested structure of the session metadata does not lend itself to the
append-only syntax of the recording meta file. In addition we do not
want to mix different kinds of metadata together. Thus the session
metadata is now written to a separate file in the BSR, and is written as
JSON which better handles the nested structure and allows for simpler
encoding/decoding.
Similarly the summary files are intended to be written to as containers
close and have been switched to JSON.
This was originally introduced to ensure that the meta files only
included ascii characters. However, this makes reading the protocol from
meta file less reliable since it does not exactly match the protocol
string. Instead protocol strings will be recorded in the meta files as
is. Generally protocol strings should include readable characters.
This also updates the ssh.Protocol to be 4 readable characters.
This adds another test storage.FS implemenation. This can be used to
write/read containers and files from a local filesystem via calls to the
os pacakge.
This changes the ToAscinema function to take a storage.TempFile instead
of a storage.FS. It was noticed that when this was being used a
storage.FS was being created just prior to calling this just so that it
could then create a temporary file. It seemed to make more sense to just
pass the temporary file in, instead of needing to expand the storage.FS
interface.
This also renames this method to align with the file format that the
returned io.Reader will contain.
This provides a new type for a temporary file and a method on a
RecordingStorage to create temporary files. Temporary files will get
removed when closed.
This updates the fstest implementation of a storage.FS to support
opening sub containers and files in read-only mode.
Now when calling `SubContainer`:
- If using `storage.WithCreateFile`, an error is returned if the
container already exists.
- If not using `storage.WithCreateFile` and error is returned if the
container does *not* exist.
When calling `OpenFile`:
- on a container that was created with `storage.ReadOnly`, an error is
returned if attempting to open the file wth `storage.ReadWrite`.
- with `storage.WithCreateFile`, an existing file is replaced with a new
empty file, (ie "truncated").
- without `storage.WithCreateFile`, an error is returned if the file
does not exist.
Timothy Messier
