The script should be given a base branch to compare against, if not set
it defaults to main. The script will apply the schema migrations for
both the base branch and the current commit and generate dumps that are
then compared to create a diff. The dumps are left in a temporary
directory so that they can be compared using a users preferred diff tool.
For example, generate the dumps and diff against main
./scripts/schema-diff.sh main
Examine the diff to database functions using delta:
delta .schema-diff/funcs_$(git rev-parse main) .schema-diff/funcs_$(git rev-parse HEAD)
This also adds a github action that runs the script on pull-requests
that contain schema changes and reports the diffs as a comment on the
pull-request.
Co-authored-by: Hugo Vieira <hugoamvieira@users.noreply.github.com>
This runs the make targets:
- make test
- make test-api
- make test-sdk
The go tests in the main module are split by package and run in parallel
in a matrix build with GOMAXPROCS=1. This is done primarily to work
around an issue that is only seen on the github runners. When running
at a higher GOMAXPROCS tests that use vault instances in docker
containers start to intermitently fail with errors from vault, usually
either a 400, 403, or TLS error. It is not clear what the cause of this
is, since the same tests run fine on self hosted runners and on
CircleCI.
Co-authored-by: Sarah Thompson <sthompson@hashicorp.com>
Co-authored-by: Daniel Kimsey <daniel.kimsey@hashicorp.com>
The `boundary connect` subcommands attempt to parse the credentials to
use for brokering. This allows the subcommands to insert the credentials
automatically into the process called by the subcommand, i.e. `ssh` or
`psql`. There is some fallback logic to try a best attempt at parsing
the credential in cases where a credential type has not be set on the
credential source.
The bug was introduced with the recent change to the vault credential
library subtype being renamed from `vault` to `vault-generic`. This
fallback logic would only run if the credential source type was "vault"
or "static", so if the vault credential library was created without
specifying a credential-type, the credential would not be parsed and
automatically passed to the sub-process.
This fix removes the additional check on the credential source's type,
instead it will attempt to parse the credential, relying just on the
structure of the credential.
This also fixes a similar bug in the table output of secrets when using
`boundary connect` without a subcommand. The parsing of the credentials
for display in the table was only running for `vault` or `static`.
Blame: db42eafd7a
When using the json output of the cli (`-format=json`), a successful
call to the controller API would place the HTTP status code in the JSON
as `status_code`. However, if there was an error from the API the status
code would be placed in the JSON as `status`. In order to be consistent,
this adds a `status_code` field to the error response case. For
backwards compatibility it still populates `status` in the error case.
This inconsistency was missed partially due to a bug in the CLI tests.
The assertion on status code was not valid and would always be
successful. Thus this also fixes the assertions and tests.
This ensures that checks for features should work, and bails early if
that is not the case, since if this fails the binary is essentially
broken. This also simplifies code that performs these checks, ie instead
of:
binaryVersion, err := version.GetReleaseVersion()
if err != nil {
// handle error? panic?
}
if version.SupportsFeature(binaryVersion, version.FeatureA) {
// code for FeatureA
} else {
// without FeatureA
}
Implementations can do:
if version.SupportsFeature(version.Binary, version.FeatureA) {
// code for FeatureA
} else {
// without FeatureA
}
Add vault ssh certificate credential library
This adds a new credential library type, `vault-ssh-certificate`, which
can be used with vault's ssh secret engine to generate certificates to
use as credentials for ssh. This new credential library can be created
using:
boundary credential-libraries create vault-ssh-certificate
See the help output for a full list of the options supported.
This also renames the `vault` credential library subtype to
`vault-generic`. The previous subtype of `vault` will continue to work,
thus the following commands are equivalent:
boundary credential-libraries create vault-generic
boundary credential-libraries create vault
See: https://developer.hashicorp.com/vault/docs/secrets/ssh
When issuing credentials from vault credential libraries, only persist
the vault credential if it is revokable. A credential that is revokable
has a valid lease id that can be used to revoke the lease in vault. If
the credential is not revokable it is also not renewable since renewing
also requires a lease id.
See: https://developer.hashicorp.com/vault/docs/concepts/lease
The wh_upsert_credential_group function was incorrectly attempting to
update the session connection fact table. There are a couple problems
with this:
1. At the time that this runs, there would be no session connection facts
for this session.
2. This function is meant to update the session fact table, there is a
separate function that associates the credential group with the
session connection facts.
This also fixes another bug that was causing expired rows of the
credential dimension to be included in the group membership. And finally
it fixes a typo in the function name.
Trigger a downstream merge. This requires that the DOWNSTREAM_SLUG
environment variable is set in CircleCI. This will trigger downstream
merges for main and all release branches.
feat(config): Support multiple files
This updates the three subcommands that take a `-config` flag to
to support multiple config files via a repeated `-config` flag.
It updates the following subcommands:
- `server`
- `database init`
- `database migrate`
Note that the multiple file support only works if all config files are
HCL, not JSON. Also note that the config files are concatenated
together. There is no attempt to merge config files, so having the same
stanza in multiple files can yield unexpected results, with only one of
the stanzas being used.
When used with these limitations understood, this provides a way to take
different actions for different kinds of dynamic configuration changes.
An example would be when using consul-template to render the
configuration files for boundary. With the multi file support
consul-template can render one config file for the controller stanza,
and provide boundary with a dynamic database credential via vault. Then
when the credential is rotated, consul-template can send boundary a
SIGHUP to switch to the new credential without a restart. A separate
template could be used to render a config for a stanza that does not
support SIGHUP, such as the events stanza that gets its values via
consul's KV. If these values change, consul-template can restart
boundary so the new configuration is used.
Co-authored-by: Timothy Messier <tim.messier@gmail.com>
Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>
The admin ui assets are built via a github workflow and published as
github a artifact. The artifact can be downloaded, avoiding the need to
build the assets again locally, and eliminating the need for front-end
build tools like yarn and npm.
This updates the `build-ui` target to use a new script that will attempt
to download the pre-built assets. Note that this script requires that
the `gh` cli is installed since an authenticated request is necessary to
download the workflow artifacts. If the artifact cannot be downloaded
either due to the lack of `gh` or because the artifact cannot be found,
it will fallback to using `git clone` (and will require yarn to build
the assets).
See:
https://cli.github.com/https://docs.github.com/en/rest/actions/artifactshttps://github.com/actions/upload-artifact/issues/51
The schema changes introduced in 0.10.2 introduced a bug where an org
could not be deleted if there was a session for an org scoped user. It
would result in an error like:
scope.(Service).deleteFromRepo: unable to delete scope: \
iam.(Repository).DeleteScope: failed for o_VkNy46KwfS: \
iam.(Repository).delete: db.DoTx: iam.(Repository).delete: \
db.Delete: insert or update on table "session" violates foreign key constraint "target_fkey": integrity violation: error #1003
A worker around would be to first delete all projects in the org, then
finally delete the org.
This fix adds a regression test to the sqltests and updates several
constraints on the session table to be deferrable. Since the org delete
is a single transaction, this allows all of the necessary resources to
be deleted before attempting to enforce these constrains.
See: https://www.postgresql.org/docs/current/sql-set-constraints.html
Blame: 3a28ab614
Several of these tests are being skipped in the enos
run. This seems to be due to differences in the setup that is performed.
When these tests were run manually, `boundary dev` was used which
performs some setup of resources automatically. However since enos uses
a non-dev controller, a similar setup is performed via the enos modules.
Until the discrepancies between the setup can be investigated, this adds
a separate CI job to run the tests against `boundary dev`. These tests
are all passing, without any being skipped. This test suite can still be
useful even if the enos version is fixed, since this test suite will run
much faster, and therefore identify failures sooner.
When generating permissions from the grants, if a single grant was
limited to the set of "self" actions, the permission would be marked as
only self, even if additional grants were given for non-self actions.
This would result in the generated permissions being more limiting then
they should be based on the grants. This only impacts the sessions list
endpoint. It would result in users that have been granted access to list
other user's sessions to be unable to see these results in the list
results.
In cases where the server was started with both a controller and worker
stanza, the reload of the config was not properly setting the initial
upstream value on the new config. In this case the value is derived from
other config values and not necessarily set directly.
This now properly initializes the new config so that initial upstream is
set in the same manner as the first load.
Blame: 5d0cdf680f
Improve response time for list requests for targets and sessions. This
refactors the behavior between the target and session services and how
they interact with the domain layer. Now the service is able to evaluate
the set of permissions from the requesting user's grants, use these
permissions when constructing the corresponding repository, and retrieve
the list of the resource the user is authorized to list in a single
query.
This reduces the round trips to the database and greatly reduces the
amount of data received from the database.
This impacts commands (in particular when `-recursive` is used):
boundary targets list
boundary sessions list
And the corresponding API endpoints of:
GET /v1/targets
GET /v1/sessions
There is no change to the data returned.
See:
68568828e6bf263fbd7d
This unexported method was only used by a single test, and was taking
advantage of its implmentation to populate Connections rather than
Sessions. This repository should only interact with Sessions, since
there is a separate repository for Connections.
This adds a context to the session.RepositoryFactory, allowing
errors.NewDeprecated to be replaced with errors.New. It also removes the
common.SessionRepoFactory, instead packages just directly import from
the session package.
This builds on previous performance improvements to the list behavior by
reducing the number of queries against the database and the amount of
data transfered in order to retrieve a list of sessions. Previously, a
query was performed to retrieve a tuple of `(session_id, scope_id,
user_id)` for the requested scope, including all child scopes if the
request was recursive. These tuples were evaluated against the
requesting user's grants to identify the sessions that they were
authorized to list. In the case of clusters with a large number of
sessions, this would result in a large amount of data needing to be
transfered for sessions that would not be used. Then a second query
would be used to retrieve the full session information for the session
ids that were authorized.
This changes the list behavior to evaluate a set of permissions from
the requesting user's grants, use these permissions when constructing a
session repository, and retrieve the list of sessions the user is
authorized to list in a signle query.
Ref:
d74103482932070678dc
The session repository can now be passed an optional set of permissions.
The permissions are used to restrict results from the repository.
Currently this only impacts the List method. This enabled the sessions
service to use a single query to retrieve sessions and replaces the
previous behavior where the session service would first request all
sessions for a given set of scopes, evaluate the permissions, and then
request sessions based on a set of session ids.
Disabling the auth was resulting in some unrealistic tests setup and
expectations. It also makes it difficult to refactor any behavior in the
authn/authz flow since this option can bypass most of it.
Disabling the auth was resulting in some unrealistic tests setup and
expectations. It also makes it difficult to refactor any behavior in the
authn/authz flow since this option can bypass most of it.
On a SIGHUP, if the worker's `initial_upstreams` config option has
changed, it will use the new values as the address to send its status.
Any previously discovered upstream addresses from previous status
requests are dropped. Once a successful status request is made to the
new addresses, the worker will resume its normal behavior of using the
addresses from the response to the status request.
This allows a worker to be deployed in a more stable way in a dynamic
environment where controller IPs could change. In particular this helps
with a case where all controllers are replaced with new controllers
running on new IPs.
To illustrate with an example, suppose there is an instance group of
controllers instances. And there are worker instances that have their
config rendered from consul-template that sets `initial_upstreams` to
the IPs of the controller instances. If all controller instances are
restarted or replaced at the same time and assigned new IPs, a new
config will be rendered for the worker. Previously the only way for the
worker to use the new value would be for the worker process to be
restarted. Otherwise the worker would only know about the previous IPs
and would fail to connect to the new controllers. However, a restart
would mean any active sessions and session connections on the worker
instance would be terminated. With this update, consul-template can send
a SIGHUP to the worker, the worker will connect to the new controllers
and resume normal status reporting, allowing the sessions and session
connections to remain.
The same principles would apply for other examples of dynamic
environments such as nomad and kubernetes.
See:
https://github.com/hashicorp/consul-templatehttps://www.nomadproject.io/docs/job-specification/template
This adds the ability to define hooks that will run prior to a schema
migration. The hooks should only be used when a migration would fail due
to data being in an invalid state, likely due to a bug in the previous
schema definition. For example if a column was missing a constraint, and
a new migration is attempting to add the constraint, there might already
be values that violate the new constraint. In such cases, we would want
the user to make a decision to address the data, and need to provide
them with details about what data is problematic in case they want to
manually address the problem. To help the user, if there is a sane way
to address the data, we should provide an option to automatically
"repair" the data, allowing the migration to continue. However, this
still should be an explicit opt-in from the user, and we need to clearly
describe what the repair option would do to the user so they understand
the consequences of opting in.
To accomplish this each migration can be provided a hook that contains a
CheckFunc, RepairFunc, and RepairDescription. These functions must
operate at the point-in-time of the migration immediately preceding the
current migration, as as such they should not use and domain packages or
repositories, instead they should use direct sql with the sql library.
The description is used to explain what the RepairFunc will do.
Hooks should be defined in `internal/db/schema/migrations/oss/oss.go`
and registered with the schema edition.
See the example in `internal/db/schema/manager_example_test.go` for more
details.
This add the generated code and additional configuration to support
sending the ssh target subtype to the api via the sdk or cli.
The cli now supports an additional subcommand for the ssh target
subtype:
boundary targets create ssh
boundary targets update ssh
This adds two new config options for the `database` stanza:
- `max_idle_connections`
- `max_idle_time`
These correspond to settings for the underlying `sql.DB` instance. They
can be used to have finer control over the controller's database
connection pool. For example this can be used to mitigate the high
connection overhead when connecting to postgres by setting a high
`max_idle_connections` and either setting `max_idle_time` to zero or to
a high duration.
Alternatively, if using a connection pool like pgpool-II or pgbouncer,
`max_idle_connections` can be set to 0.
See:
https://pkg.go.dev/database/sql#DB.SetMaxIdleConnshttps://pkg.go.dev/database/sql#DB.SetConnMaxIdleTimehttps://www.pgpool.net/mediawiki/index.php/Main_Pagehttps://www.pgbouncer.org/
This introduces two changes to reduce the output of tests in CI and for
local developers:
- Silences gorm db logs
- Silences log indicating the lack of an eventer
This removes a lot of log clutter from normal test execution with `go
test -v`. There are a number of test that expect no results from the
database or other error cases from the database. The previous default
log level of dbw.Error would cause these cases to generate logs in the
success case, creating noise in the test output.
The new default for tests is dbw.Silent. This can be overriden in two
ways. A test can be updated to pass in the db.WithTestLogLevel option,
which allows for targeting specific tests to include additional output.
This may make sense for tests that normally should not cause errors, but
would report an error if it fails.
Alternatively a new environment variable can be set,
DEFAULT_TEST_LOG_LEVEL. This allows for adjusting the log level without
changing any test file. The environment variable should be set to the
string value of the log level, and is treated as case insensitive, ie:
export DEFAULT_TEST_LOG_LEVEL="error"
perf(session): Improve list response time
Notable changes include:
- Change to not include terminated sessions by default. Use the new
`include_terminated=true` query parameter to return terminated
sessions.
- Remove connections from list response. Connection information is still
returned for a read of an individual session.
- Creates a periodic job that will delete terminated sessions.
It was determined that it is a more common use case for users to request
non-terminated sessions (i.e. sessions in `pending`, `active`, or
`canceling` when requesting sessions. Notably, end users would need
non-terminated sessions to:
- Establish a new connection to a session.
- Terminate a session
So now by default a list request to `/v1/sessions` will *not* include
terminated sessions.
To still enable use cases where terminated sessions are needed, this
adds a new query parameter to include the terminated session:
`/v1/sessions?include_terminated=true`.
Note that this query parameter is considered experimental and subject to
change.
Including the nested connection information in the session list response
had a negative impact on performance as it created a significant
multiplier on the number of rows that needed to be fetched from the
database.
It was also determined that this information was used by neither the
admin ui nor the desktop client.
Note that nested connection information can still be retrieved via a
read request for a specific session.
There is a new periodic job that will delete terminated sessions that
are older than one hour. However, since until now terminated sessions
were not deleted, the first time the job runs there could be a large
number of sessions that need deleted. This could have a negative impact
on a running cluster. To mitigate the operational risk, this adds a
migration to perform an initial delete while the cluster is offline.
Since the Dockerfile is now at the repository root, when running
building a docker image, the entire repository was being added to the
context sent to the docker daemon. However, only the compiled binary and
a few configs in `.release/docker` are needed to build the docker
images.
This ignore file will only send necessary fields into the context. This
helps speed up the docker image building.
For example, when building the dev image, without the ignore file:
Sending build context to Docker daemon 2.043GB
And with this ignore file:
Sending build context to Docker daemon 130.7MB
See:
https://docs.docker.com/engine/reference/builder/#dockerignore-file
This test was using the third element in the result from a list request
to craft the input for the next assertion. However, one of the test
targets that is created does not have a default port set. Since the
target list call is not ordered, sometimes this element could be the
third element and a panic would result when default port interface was
converted to a float64.
This fixes the panic like:
--- FAIL: TestList (1.31s)
panic: interface conversion: interface {} is nil, not float64 [recovered]
panic: interface conversion: interface {} is nil, not float64
goroutine 1694 [running]:
testing.tRunner.func1.2({0x14b2320, 0xc002f486f0})
/prefix/usr/lib/go/src/testing/testing.go:1389 +0x24e
testing.tRunner.func1()
/prefix/usr/lib/go/src/testing/testing.go:1392 +0x39f
panic({0x14b2320, 0xc002f486f0})
/prefix/usr/lib/go/src/runtime/panic.go:838 +0x207
github.com/hashicorp/boundary/internal/tests/api/targets_test.TestList(0xc000f81380)
/go/src/github.com/hashicorp/boundary/internal/tests/api/targets/target_test.go:193 +0x174b
testing.tRunner(0xc000f81380, 0x2b933a8)
/prefix/usr/lib/go/src/testing/testing.go:1439 +0x102
created by testing.(*T).Run
/prefix/usr/lib/go/src/testing/testing.go:1486 +0x35f
FAIL github.com/hashicorp/boundary/internal/tests/api/targets 5.742s
FAIL
Refs: #2003
Listing Session and Targets Performance Improvements
Improve response time for `GET` requests to:
- `/v1/targets`
- `/v1/sessions`
These endpoints now perform authorization checks prior to retrieving the
full resources from the database.
This also includes a bug fix for retrieving sessions that could result
in incomplete results when there is a large number (10k+) of sessions.
When applying the limit and order by, the where clause also needs to be
applied to yield the correct results. This can be seen with an example.
Assuming a data set where there are 100 sessions, with 10 sessions
belonging to a user with id `u_user1` and 90 sessions belonging to
`u_user2`. If the constructed query is like:
select *
from
(select public_id from session limit 10) s,
session_list ss
where
s.public_id = ss.public_id
and s.user_id = 'u_user1'
order by create_time asc;
The initial sub-query with the limit could return sessions only
belonging to `u_user2`, then it would apply the where clause and exclude
all of the rows.
Instead if the query is:
with
session_ids as (
select public_id
from sessions as s
where
s.user_id = 'u_user1'
order by create_time asc
limit 10
)
select *
from session_list
where
session_list.public_id in (select * from session_ids)
order by create_time asc;
The CTE applies the where clause ensuring that only sessions for
`u_user1` are used for the final select from the view.
Note that the CTE is use primarily for readability.
This also adds an index on `session (create_time)` to aide the query
planner in performing the order by.
This updates the target resources to use a oneof for attributes. This
allows requests to the targets service to leverage the new
subtypes.AttributeTransformerInterceptor so that attributes are strongly
typed prior to the service handler.
The new `subtype.AttributeTransformerInterceptor` can return an error if
it fails to transform the attributes. This is mainly if invalid
attribute fields were included for a particular subtype. This error case
used to be checked in the grpc service handler functions. However, now
that it is happening in the interceptor, the `errorInterceptor` needs to
be moved up in the stack so that it will process the response of the new
transformation interceptor.
The `errorInterceptor` used to be after the `auditRequestInterceptor`
and prior to the `auditResponseInterceptor`, meaning that the
`auditResponseInterceptor` would have access to the result of the
`errorInterceptor`. But it seems the `auditRequestInterceptor` does not
use or need any of the headers set by the `errorInterceptor` so it
should be safe to move the `errorInterceptor`. This is important since
the `subtype.AttributeTransformerInterceptor` needs to be before the
audit interceptors in the stack to properly transform requests/responses
so they can be audited with the complete classification information.
This updates the credential store resources to use a oneof for
attributes. This allows requests to the credential store service to
leverage the new subtypes.AttributeTransformerInterceptor so that
attributes are strongly typed prior to the service handler.
This updates the credential library resources to use a oneof for
attributes. This allows requests to the credential library service to
leverage the new subtypes.AttributeTransformerInterceptor so that
attributes are strongly typed prior to the service handler.
This updates the credential store resources to use a oneof for
attributes. This allows requests to the credential store service to
leverage the new subtypes.AttributeTransformerInterceptor so that
attributes are strongly typed prior to the service handler.
This interceptor is responsible for maintaining compatability with the
JSON API with the change to the protobuf messages. The messages are
changing from a single attribue field to a oneof to explicitly associate
subtype attribute messages with the resource message. The interceptor
transforms the request prior to the auditReqeust interceptor, and
transforms the response after the auditResponse interceptor. This allows
the audit interceptors to use all the necessary classification
information when creating the audit events.
Timothy MessierandHugo Vieira
