When building the set of migrations to run as part of executing the
`database migrate` or `database init`, the migrations are collected and
combined to run within a database transaction. As such the `begin` and
`commit` statements in the migration files should get removed from the
final set of statements that get executed. However, when the migrations
files were updated to include copyright and license headers, this broke
the logic that would strip out the `begin` statement.
While this does not cause any issue with executing the statements in a
single transaction, it does result in noise in the database server's
logs, with log messages like:
BEGIN WARNING: 25001: there is already a transaction in progress
BEGIN LOCATION: BeginTransactionBlock, xact.c:3778
This commit fixes the logic to correctly strip the `begin` for files
that have the header lines.
The api.Client was first checking for the environment variable, and then
resetting the max retries to 2, thus overriding the environment variable
value. In addition, the cli would set max retries to zero if the
environment variable was not set. This would mean that by default the
max retries would be zero, and if the environment variable was set, the
max retries would be 2, regardless of the actual value for the
environment variable.
This fixes the order in which the environment variables are read, to
ensure they take precedence.
When a resource relating to a session is deleted,
notably a target, host, or auth_token, the corresponding sessions are
marked as canceled via a database trigger that calls the cancel_session
function. This function was performing a select as part of the insert
call to ensure that it would not attempt to insert a session_sate of
canceled if it was an invalid state transition. However, since this
function was first introduced, there has been additional constraints
added that would prevent an invalid state transition. In addition,
performing a select as part of an insert increases the chances for
contention with other queries that interact with the session_state
table, especially since a large number of sessions and session_state
rows can be included in a single transaction when things like
auth_tokens or targets get deleted.
Now, the cancel_function will perform the insert, and catch any of the
potential violations, allowing the query to continue without needing to
perform the select, thus reduction the potential for contention.
This replaces the start_time, end_time, and previous_end_time columns on
the session_state table with a time range column that records when the
state was active. This allowed for dropping several constraints on the
session_state table that were used to ensure that states for a session
did not overlap. Instead this can be ensured with a single exclusion
constraint. By reducing the number of constraints, it improves the
performance for a number of queries involved with session state. Most
notably when resolving delete cascades of resources related to sessions,
such as targets, where as part of the delete transaction large numbers
of sessions get canceled. Similarly, when sessions get deleted, and the
corresponding session_state rows are deleted, these constraints all must
be checked during the transaction.
The domain layer continues to expose StartTime and EndTime fields on
session.State. However it drops PreviousEndTime, which was never used by
anything calling the domain layer.
See:
https://www.postgresql.org/docs/current/ddl-constraints.html#DDL-CONSTRAINTS-EXCLUSIONhttps://www.postgresql.org/docs/current/rangetypes.html
Cache speed up and additional performance improvements
This includes several changes to improve the performance of the
client cache, including:
* Making the sqlite database for the cache persistent, so the cache does
not need to be rebuilt each time the boundary client is started.
* Limiting the number of results retrieved when listing resources to 250.
* Preventing redundant refreshes running at the same time for a given
client.
* Adding indexes to the cache database.
It also refactors the grants query that is run to fetch grants for a
user when processing a controller API request. This greatly reduces the
amount of data that is returned from the database, but required a number
of changes in the application logic to account for different kinds of
grant scopes.
Lastly this makes some performance improvements in the controller
database. It adds several indexes to help with the grants query and with
delete cascades for several resources. And it changes the delete trigger
for sessions to run per statement instead of per row to help with bulk
deleting of sessions.
This function was returning the set of deletion tables. A view seems
better suited for this task since it would allow for applying additional
filters of the result set. This was particularly necessary to easily
make changes to some sqltests due to switching the delete trigger for
the session table.
(cherry picked from commit 32d41639892950b21fafc39b50c10b997c6d5121)
When sessions are deleted, a trigger is used to insert records into the
session_deleted table. This table is utilized by the sessions list
endpoint when using a refresh token to inform a client of the sessions
that have been deleted since the last request. We delete sessions in
bulk via a controller job to delete sessions that have been terminated
over an hour ago, which results in the trigger running a large number of
separate insert statements while processing the delete statement.
This changes the trigger to run once for the delete statement, instead
of for each row, resulting in a single bulk insert statement to the
session_deleted table.
This new trigger function also avoids the use of `on conflict`. When
testing this function, while the single statement was still faster than
running multiple inserts, the `on conflict` still added significant
overhead, even when there were no conflicts. It should be safe to
perform the insert without the `on conflict`, since the same ID should
never be deleted more than once if it is successfully deleted.
(cherry picked from commit 3569b36c6eb2bee7a4e36e7584fcc6bbf1124f95)
When processing a controller API request, system uses a query to fetch
the grants for the requesting user. This query requires pulling together
information from several tables, and is currently performing many
sequential scans to do so. For a number of the tables involved, there
are indexes from multi-column primary keys, however, due to the order of
the columns in the index, the postgres planner would need to do a full
scan of the index, which can be less efficient than a sequential scan,
so instead it will perform a sequential scan of the table.
This recreates the primary keys while swapping the order of the columns
in the primary key definition, and thus the order in the index. By doing
so, the planner will not need to perform a full index scan, and will be
more likely to use the index when executing the grants query.
(cherry picked from commit 5577b3c358e9d45cd7ec02fc951e45d52e50595e)
This adds indexes for a few categories:
1. Foreign keys on the session table. These are set to `null` when the
referenced row is deleted. These indexes will help to more
efficiently set these `null` values in the case where a target is
deleted, or and auth token is deleted.
2. Foreign keys from other tables to the session table. These are either
set to `null` or cascade deleted when a session is deleted. These
indexes help with these update/deletes when a session is deleted.
3. A multi-column index on session_state. This helps with the query that
is used to delete all terminated sessions that have been terminated
for over an hour.
(cherry picked from commit 263a96e9ddd197cb36f2c4c74e91ac034112321e)
The target authorize session endpoint allows for alternative target
identifiers besides the public id. For this to function correctly, the
grpc gateway path definition for the endpoint deviates from the common
pattern used for other endpoints that include ids. When this change was
done, it was not accounted for in the logic used to capture the
prometheus metrics for api requests to this particular endpoint,
resulting in the metrics never incrementing for the authorize session
endpoint.
This fix updates the matching logic used for the metrics to ensure that
the metrics are incremented for this endpoint.
See: c149bc4a79
The cluster listener supports a `max_request_duration` option, and
defaults to 90 seconds. However, this configuration was not being use
when processing grpc requests from the listener. This meant that
requests from worker to controller could hang indefinitely and
contributed to the recently fixed issue with database connections being
stuck in `idle in transaction`.
This commit adds a grpc interceptor that ensures each request has a
deadline set to the max request duration for the listener.
Refs: 29cd7bc9d6
Several session.Repository methods need to encrypt or decrypt fields
when persisting or retrieving a Session. These encrypt/decrypt
operations where being performed while within a database transaction.
The first step in the encrypt/decrypt operation is to retrieve the
kms.Wrapper. While the wrappers get cached in memory, the retrieval
still results in a database query to check that the kms collection
version has not changed. This query happens with a separate db.Reader,
and therefore uses a separate database connection from the transaction
being used by the session.Repository.
If the controllers are configured with a relatively low
`max_open_connections`, and be under sufficient load interactive with
sessions, the controller can consume all of the available database
connections, and get stuck with open transactions while waiting for a
database connection to execute the kms related queries. This can be see
by checking pg_stat_activity and seeing transactions stuck in `idle in
transaction` with a wait_event of ClientRead.
This commit changes all of the Repository methods the perform any
encryption or decryption operations, including the retrieval of the
kms.Wrapper, outside of a database transaction. An alternative approach
would be to unsure that the kms.Wrapper retrieval happens within the
same database transaction. However, it seems preferable to avoid doing
these relatively expensive operations within a transaction when it is
not necessary.
As part of this fix, it also eliminates some logic that is no longer
needed. During previous would with KMS wrapping, the method of how
sessions were encrypted was changed, and some logic to lazily re-encrypt
sessions was introduced. However, given that terminated sessions are
deleted, and the KMS changes was introduced several versions ago
(0.13.0), this lazy re-encrypt logic should no longer be necessary.
Refs: edd323b73a
See:
https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-ACTIVITY-VIEWhttps://www.postgresql.org/docs/current/runtime-config-client.html#GUC-IDLE-IN-TRANSACTION-SESSION-TIMEOUThttps://developer.hashicorp.com/boundary/docs/configuration/controller#max_open_connections
When closing orphaned connections and checking if a session is not
active, the repository was opening a transaction, but then calling the
original reader instead of using the reader/writer from the transaction.
This would result in a transaction being opend, then a separate
transaction being used to run the query, and finally the original
transaction would commit without any statements being executed.
Co-authored-by: Sorawis Nilparuk (Bo) <sorawis.nilparuk@hashicorp.com>
This adds support for tracking and reporting monthly active users for
the purpose of billing. It adds a new API endpoint,
`/v1/billing:monthly-active-users` and new cli command, `boundary
billing monthly-active-users` that can be used to view the monthly
active user counts.
This adds a new accumulating fact table to the data warehouse for auth
tokens. This fact table is used to calculate the number of active users
within a month, where an active user is a user that has at least one
issued auth token within the time range of the month.
This updates the billing repository to use timestmap.Timestamp when
constructing the queries to retrieve monthly active users. This allows
for specifying NegativeInfinity to easily retrieve all monthly active
user counts.
When converting a timestamp.Timestamp for use in SQL, this now correctly
converts the NegativeInfinity and PositiveInfinity constants into the
corresponding `-infinity` and `infinity` SQL constants.
These functions will ensure that UTC time is used for billing queries,
eliminating the need for the repository to set the timezone. This also
means the repository no longern needs a writer.
This adds two sql functions that can be used to access the views to
retrieve monthly active user counts. The functions ensure that any time
calculates are performed in UTC, so that the month bounds align with the
start of the month in UTC.
1. hcp_billing_monthly_active_users_last_2_months() will return monthly
active users for the current and previous month.
2. hcp_billing_monthly_active_users_all() will return monthly active
users for all months. This function can also be provided a start time
and or end time to filter the results to a specific range.
This test started failing now that the data warehouse records a auth
token fact when an auth token is issued and it requires that the auth
token has a user associated with it.
Since this test is testing the interaction with the database directly,
and not via the repository, it by-passes the application checks that
would prevent an auth token without an associated user being inserted.
This test is updated to meet this same pre-condition prior to attempting
to insert into the database.
Now that there is an auth token fact table in the data warehouse, it is
not possible to insert an auth_token with an old public_id even after
that auth_token was deleted, since it now causes a constraint violation
on the data warehouse table.
With the addition of an after insert trigger to the auth_token table,
entries are added to the wh_user_dimension table when auth token's are
inserted with the 'token issued' status. This caused a number of tests
to fail that assumed the user dimension table would be empty prior to
inserting sessions. This introduces a number of changes to the tests:
- In the test setup, auth tokens are not created with an explicit
expiration time and status. The expiration time ensures that the
tokens are 'valid' for the test. The status allows test to make
assertions on tokens in different statuses, and allow the status to
change during the test.
- Tests make the correct assertions on the number of entries in the user
dimension table after test setup.
- User dimension tests modify a user after the standard setup to ensure
that a new user dimension is added when a session is inserted.
This adds a function and after delete trigger to the auth_token table to
ensure that a corresponding auth token fact is updated with the deleted
time and sets the auth token valid time range. The data warehouse will
only be updated if the token that was deleted is in the 'token issued'
status.
This adds a function and update trigger to auth_token to ensure that:
1. If the auth token is updated to the issued state, a auth token fact
is added to the warehouse.
2. If the auth token's approximate last access time is updated, the
corresponding auth token fact is updated with the new time.
Previously the function used to upsert into the user dimension required
both an auth token id, and a user id. However, the user id can be
determined from the auth token, in fact it already was via the
whx_user_dimension_source view. This refactor removes the user id as an
argument, since it is not needed and will not be directly available when
this function is used when populating the new auth token accumulating
fact.
When running the sqltests, sometimes it is helpful to see the full
database container logs, especially when there is an error in a
migration or database setup. However, it also makes it difficult to see
the test results.
This adds a new variable to the makefile to easily toggle the output of
these logs on or off. To run the tests without printing the dtabase logs
use:
DB_LOGS=0 make test
To enable the logs use:
DB_LOGS=1 make test
By default the logs are not printed.
Some of the prepared statement identifies used in these tests exceeded
the identifier character limit. This results in postgres truncating the
name, which is fine for a test, but does result in some noise in the
test output. This change shortens the identifies to eliminate noise like
the following:
psql:/test/credential/vault/credential_vault_library_ssh_private_key_mapping_override.sql:49: NOTICE: identifier "insert_credential_vault_library_ssh_private_key_mapping_override" will be truncated to "insert_credential_vault_library_ssh_private_key_mapping_overrid"
The tests should no longer create notices like these.
After updating `boundary dev` to use postgres:12 by default, it was
noticed that the responsiveness of boundary was noticeably slower. This
is due to the introduction of JIT being enabled by default in
postgres:12. JIT seems to incur noticeable overhead without providing
benefits, at least for the instance used for `boundary dev`.
It is also worth noting that for postgres:14 and newer, JIT does not get
triggered for the same queries since they seem to result in a cheaper
query plan that does not exceed the default threshold for JIT.
See:
https://www.postgresql.org/docs/current/jit.html
ICU-12283
This adds a Access-Control-Expose-Headers response header to an Options
request to allow the rate limiting related headers to be accessible via
requests that are subject to CORS.
Normally the tests are run on a host machine with a docker container for
the database. The docker container should be accessible via 127.0.0.1.
However there might be cases where the test execution is run in a docker
container, or the postgres server is configured to use a different
docker network, in which case the container would be accessible via some
other dns name or ip. In these cases, users can now set the TEST_DB_HOST
environment variable so the tests can connect to the postgres server.
This increases the default rate limits to:
- 1,500 reqs / 30 seconds for list actions in total
- 1,500 reqs / 30 seconds for list actions per IP address
- 150 reqs / 30 seconds for list actions per auth token
And for all other actions:
- 30,000 reqs / 30 seconds for list actions in total
- 30,000 reqs / 30 seconds for list actions per IP address
- 3,000 reqs / 30 seconds for list actions per auth token
This seems to be a reasonable default for a small to medium size cluster
based on some scale testing.
(cherry picked from commit 237c8b6084f42655edaba49055e511163fdf1c53)
Go 1.21.5 fixes a few vulnerabilities. Most notably for boundary is a
fix in net/http to limit chunked data overhead.
See: https://groups.google.com/g/golang-announce/c/iLGK3x6yuNo
(cherry picked from commit 575cd9d63b1ac4787d048d86b0b209c61a7f3946)
Add support for rate limiting
This introduces rate limiting for the Controller API endpoints. Limits
can be set separately for each combination of resource and action. And
each combination has a limit per auth-token, IP address, and in total.
By default each list action for all resources has a limit of 120
requests per minute. All other actions for all resources have a limit of
3,000 requests per minute. This can be overridden by including
`api_rate_limit` stanzas in the controller configuration. For example to
set a limit on target list requests per auth token:
controller {
api_rate_limit {
resources = ["target"]
actions = ["list"]
per = "auth-token"
limit = 10
period = "1s"
}
// other controller configuration
}
Or to completely override all of the defaults:
controller {
// total limit for all resources and actions
api_rate_limit {
resources = ["*"]
actions = ["*"]
per = "total"
limit = 500
period = "1s"
}
// limit for ip addresses to all resources and actions
api_rate_limit {
resources = ["*"]
actions = ["*"]
per = "ip-address"
limit = 100
period = "1s"
}
// Limit for auth tokens to all resources and actions
api_rate_limit {
resources = ["*"]
actions = ["*"]
per = "auth-token"
limit = 50
period = "1s"
}
// other controller configuration
}
All rate limiting can be disabled by setting the
`api_rate_limit_disable` configuration variable:
controller {
api_rate_limit_disable = true
// other controller configuration
}
When rate limiting is enabled, boundary will create quotas to track the
number of requests used within a time period. Boundary will reserve a
set amount of memory for tracking quotas. This ensures that boundary
does not consume too much memory if there is a sudden burst of requests
from many different clients. However this does mean if boundary is
unable to store a quota, it will limit the request with a 503 HTTP
status code. The amount of memory can be controlled with the
`api_rate_limit_max_quotas` configuration variable. This can also be
monitored via two new metrics:
- `boundary_controller_api_ratelimiter_quota_storage_capacity`
- `boundary_controller_api_ratelimiter_quota_storage_usage`
Clients making requests to the controller API can inspect HTTP response
headers to understand the configured limits and current usage. Each
response will contain the following headers:
- RateLimit: Provides the current limit, number of remaining requests,
and the time the quota will reset for the limit that is closest to
being exhausted for the requested resource and action.
- RateLimit-Policy: Describes the limits for the requested resource and
action.
If a request is limited, the client will receive a 429 HTTP status code,
and the Retry-After header will be set to the number of seconds the
client should wait before attempting the request again.
Previously this would truncate to the nearest integer, which could
result in clients retrying before the quota has reset, and getting rate
limited again. This also includes an update from go-rate that will round
up when reporting the reset time for the RateLimit header.
Note that in the case of a 503, the Retry-After header was already being
set correctly.
See: https://github.com/hashicorp/go-rate/pull/8
If the api.Client receives a 429 or 503 along with a Retry-After header,
the client will automatically wait the amount of time specified, and
then re-try the request.
A sys event will be emitted when the controller is constructed, and if
the rate limits are changed via a SIGHUP. The event contains the details
of the rate limits for each combination of resource and action. This
event can be used by operators to confirm the limits are set as they
would expect based on the controller config.
This provides two prometheus metrics to monitor the rate limiter:
boundary_controller_api_ratelimiter_quota_storage_capacity:
This is a gague that shows the total number of quotas that can be stored
at a given moment by the rate limiter.
boundary_controller_api_ratelimiter_quota_storage_usage:
This is a gague that shows the number of quotas currently being stored
by the limiter. If this becomes equal to the capacity, requests that
would result in a new quota being created will get limited and receive a
429 status code.
This registers the ActionSets for each resource when the handler
packages are initialized. This then allows other packages to retrieve
the ActionSets for a given resource from the action package.
The initial consumer of this will be the ratelimit package so that it
can programmatically enumerate all of the valid combinations of a
resource and action.
This leaves a number of TODO to refactor the handler packages to also
retrieve the action sets from the registered actions instead of using a
package variable. This is left as a TODO since there is some
inconsistencies in how these package variables are used between the
handlers. For example a number of packages use a map of subtypes to
ActionSets which is then used withing the handler implementation. Others
use a map, but don't seem to use it at all. And others just use an
ActionSet.
This switches the implementation of an ActionSet from a slice to a map
to ensure the elements in an ActionSet are unique. This allows for other
packages to perform set operations such as a union or difference without
needing to account for duplicates.
The nhooyr/websocket library now has a release with the new default
behavior that removes the message size limit. This allows us to switch
off of the fork and back to the upstream.
In the same release that includes the fix we need for the boundary
client, the wspb helper package was removed from websocket. This package
contained some functions to read and write protobuf messages. However,
it was using the deprecated github.com/golang/protobuf/proto library. To
avoid this dependancy in the library they opted to just remove wspb.
Thus this provides a replacement implementation that uses
google.golang.org/protobuf/proto. Notably the newer proto library seems
to lack an equivalent of proto.NewBuffer so writes to the websocket and
not able to leverage a sync.Pool of byte.Buffers to reduce allocations.
Refs: 88bb253554
See: https://github.com/nhooyr/websocket/commit/ac385120c6e34fa6584f3856d5db949a21bbb65e
When generating the JSON for submitting the github comment, if the
message was too large, the workflow would fail with the following error:
/usr/bin/jq: Argument list too long
This fix switches to using the `--rawfile`, which allows jq to process
the large comment.
When the boundary client establishes a connection to the websocket used
to communicate with a worker, it was configured with a max websocket
message size of 32KiB. This was the default behavior for v1.8.7 of
nhooyr/websocket. However, there are cases where the worker can send
messages to the client that are larger than 32KiB. When this happens the
client returns an error and terminates the connection.
Notably this seems to only currently impact sessions made to SSH
Targets, although it could theoretically impact any target type
depending on the details of how the worker performs the writing of the
messages. The issue can be reproduced by attempting to scp a large
enough file from an SSH Target to the client. In this case the
connection gets terminated and results in a partial transfer of the
file.
This message size limit appears to be unnecessary, and the default
behavior is changing in v1.8.8 of nhooyr/websocket to make the message
size unlimited. Since this version is not yet released, this commit
switches the dependency to a fork with a version that is v1.8.7 plus the
patch to disable the limit. Once v1.8.8 is release this dependency can
be set back.
See:
https://github.com/nhooyr/websocket/pull/254https://github.com/nhooyr/websocket/pull/256
This updates the version make target to use the version/VERSION file in
the same manner as the build system. This ensures that the VERSION file
is the single source of truth for the version number.
This moves the contents of the readme used for hashicorp/boundary on
docker.io/registry.hub.docker.com to a separate readme file from the one that
documents how to build docker images. It also makes some minor fixups
and updates the docker.io readme, must notably updating the minimum
version of postgres that boundary requires.
This provides a new storage.FS implementation that expands the on
fstest.MemFS by allowing tests to mark the filesystem as out of space.
When this happens any write/create operation will start returning an
error indicating that the filesystem is out of space.
fix(session): List performance with large number of session connections
This includes the backported migration and a migration to ensure the next
release has the correct view.
As part of the performance improvements for listing sessions for v0.9.0,
the view used to retrieve sessions for listing was updated to no longer
include a join on session_connection. However, a later migration that
was adding additional columns to this view inadvertently re-added the
join and session_connection columns. This resulted in requests to list
sessions performing this join, and returning additional rows for each
connection in a session. The application would ignore this additional
data, since it is no longer returned via the API. This would be
particularly impactful in cases where there are a large number of
connections per session, as this would greatly increase the amount of
data returned by the database.
This fix removes the session_connection join from this view, which
should restore the original performance improvements even in cases where
there is a large number of session connections per session.
See: https://github.com/hashicorp/boundary/pull/2160
Ref: 32070678dc
Fixes: 1e3c941be1
(cherry picked from commit 7aef341682e65b42c1b919e03936bdf81e26e87c)
As part of the performance improvements for listing sessions for v0.9.0,
the view used to retrieve sessions for listing was updated to no longer
include a join on session_connection. However, a later migration that
was adding additional columns to this view inadvertently re-added the
join and session_connection columns. This resulted in requests to list
sessions performing this join, and returning additional rows for each
connection in a session. The application would ignore this additional
data, since it is no longer returned via the API. This would be
particularly impactful in cases where there are a large number of
connections per session, as this would greatly increase the amount of
data returned by the database.
This fix removes the session_connection join from this view, which
should restore the original performance improvements even in cases where
there is a large number of session connections per session.
See: https://github.com/hashicorp/boundary/pull/2160
Ref: 32070678dc
Fixes: 1e3c941be1
This removes the minimum on width and height, instead it will just used
the largest width, height seen as the initial size for the asciicast.
This also provides the ability to still set a minimum via options to
convert.ToAsciicast if the caller has a known minimum size. Also note
that in cases where there is no size information, ie if there was no pty
request, this will still pick a default size of 80x24.
For requests to the target authorize session endpoint we support passing
a target name instead of an id. This means that `req.GetId()` might not
be the target's public id. Since the first thing authorize session does
includes retrieving the target as part of the authorization checks, this
subtype check can leverage the public id of the target instead of using
the request.
Fixes: 6178dd516e
Timothy Messier
