This interceptor is responsible for maintaining compatability with the
JSON API with the change to the protobuf messages. The messages are
changing from a single attribue field to a oneof to explicitly associate
subtype attribute messages with the resource message. The interceptor
transforms the request prior to the auditReqeust interceptor, and
transforms the response after the auditResponse interceptor. This allows
the audit interceptors to use all the necessary classification
information when creating the audit events.
This allows for marking a field in a message as an id that can be used
to determine the message's subtype. When creating some resources, an id
from a related resource is used to indicate the resource's subtype,
since the resource must have the same subtype as the related resource.
This allows for associating all of the messages in a file with a domain.
This will be used by the global subtype registry to group subtypes
within a domain. When registering a subtype, it must be unique within a
domain. This allows for cases where two domains have the same subtypes,
such as two domains both having a "plugin" subtype.
These github actions run on Ubuntu based machine. Ubuntu by default
links `/bin/sh` to dash. So by invoking the plugin.sh script as `/bin/sh
./plugins.sh` the script would run under `dash` despite the shebang
line. A recent change that unified some of the build process, introduced
additional bashisms into this script which is caused the build to fail.
This updated the build system to just run the script directly, which
will honor the shebang line. This seems like the best approach since the
ubuntu machine has bash and the shebang line points to bash, so it is
natural for developers to assume bashisms will work.
See:
f370abc1dahttps://github.com/actions/virtual-environments/blob/ubuntu20/20220410.2/images/linux/Ubuntu2004-Readme.mdhttps://wiki.ubuntu.com/DashAsBinSh
This removes or reduces the time that is waited in a number of tests by
allowing the grace period time be a configuration option on the
connection repository.
This creates a domain service function `session.WorkerStatusReport`. The
service performs two functions:
1. Checks the status for all of the reported sessions, returning a
report of the sessions that are in a "non-active" state, i.e.
"cacneling" or "terminated".
2. Looks for any "orphaned" session connections, i.e. connections that
are active but were not reported by the worker. These connections are
marked as closed.
This also removes the "ShouldCloseConnectionsOnWorker" logic, as this
was determined to not be a possible state, and was unnecessary.
The query for CloseDeadConnectionsForWorker includes a time comparison
to exclude results that are too close to the current time. This is done
to account for a delay in reporting by the worker. However, since the
delay was hardcoded in the query, it made it difficult to test.
This moves the delay duration into a parameter that is set on the
repository. The amount of the delay can be set at repository creation
time using the new WithWorkerStateDelay option. This defaults to the
previously hardcoded amount.
This will allow tests to control the delay to exercise this
functionality fully.
When the make targets where updated to switch from `make dev` to
`make build`, the dependency for the `docker-build-dev` target was not
updated.
See: b53e4d4c3e
This adds a new makefile target to build and publish the test database
docker image for multiple architectures. It will create images for:
- linux/amd64
- linux/arm64
The target uses docker buildx to build the images. To build the images:
docker buildx create --driver docker-container --use
make docker-buildx
docker buildx rm
See: https://docs.docker.com/buildx/working-with-buildx/
The removeDups template function converts a slice into a map and
back into a slice in order to leverage the map keys to remove
duplicates. However, since the iteration order over a map is not
specified it would be possible to get different orders for multiple runs
of `make api`. While this would not have an impact on any functionality,
it is causing sporadic failures for the new make-gen-deltas CI job.
See: https://go.dev/ref/spec#For_statements
Ref: d57603022a
In CI were errors/warnings like:
./scripts/plugins.sh: 7: [: windowsx: unexpected operator
This comparison is compatible with more versions of bash and other
shells and should prevent these messages.
When moving to the new CRT build process, this was missed from the older
build scripts. When building the plugins for Windows, if the files do
not have the `.exe` suffix, boundary fails with errors like the
following:
Error initializing controller: error creating aws host plugin: error fetching host plugin rpc client: exec:
"C:\\Users\\alice\\AppData\\Local\\Temp\\441718622\\boundary-plugin-host-aws": file does not exist
Some OS build steps were not compiling the plugins before compiling the
boundary binary. This resulted in binaries that did not have plugins
included, and would result in an error when starting a boundary server.
This target is updated to set GOOS and GOARCH instead of XC_OSARCH since
the build script was updated to use these environment variables for
cross-compiling.
See: 1c8e0ed1e1
This allows users to set the type of credential that a credential
library will provide.
A User Password credential type and an Unspecified credential type are included
with the expectation that additional credential types will be added in the
future. This change also adds the credential type attribute to existing
credential libraries and sets it to Unspecified. The credential type can
only be set when a credential library is created.
The vault credential library includes functionality for converting a
Vault Secret into a User Password Credential. This mapping is enabled by
default, however, the user can override the attributes used for this
mapping.
This adds a new requirement onto subtypes to register a NewFunc.
Previously the subtypes where implementing this function anyway, but the
target package was not including them in the registry. This resulted in
packages that use the target package needing to import the subtypes
directly, which we want to avoid.
This refactor will require the subtypes to register their NewFunc which
can then be used by `target.New` so packages that need to create targets
can do so by only using the target package.
This also updates the tcp subpackage to register a NewFunc.
This adds new makefile targets and configuration to run the tests in the
api and sdk modules. These do not run by default since `go test ./...`
does not cross module boundaries.
Some phony targets where not included in the single .PHONY line, this
switches to marking targets as PHONY just before the target is defined,
which makes it easier to ensure the targets are correctly marked as
phony.
Trying to run the docker image built with `make docker-build-dev` would
result in an error at startup:
standard_init_linux.go:228: exec user process caused: no such file or directory
This was due to the dev docker image lacking `dumb-init` which is used
by `docker-entrypoint.sh`.
This commit updates the dev dockerfile to add dumb-init. It also adds
additional packages so that the dev image aligns with the release image.
This should help prevent issues like the one fixed in
f2ad4de554.
A version for a particular edition should only exist once, previously
which ever file was read last for a version would be included, which
would lead to some migrations not running.
This test started failing when the order of the list response changed.
Since the order does not matter, this test was updated to not depend on
the order.
Target subtypes must now implement another repository hook to validate
any CredentialLibraries that are being added/set. This allows for
subtypes to determine the valid set of CredentialLibraries.
For tcp.Targets only CredentialLibaries with
credential.ApplicationPurpose are allowed. If any other purpose is
provided an error is returned.
This change removes the hard-coded credential purpose for the Credential
Libraries. It also moves the logic of creating CredentialLibraries from
CredentialSources out of the repository and into the targets.Service.
This refactor will allow for additional credential purposes to be
supported in both the service and the repository.
For the target Repository's Create method, the options of
WithHostSources and WithCredentialSources where never used by the
targets.Service. As such the repository does not need to support these
options.
See: 51812ff779
Now that subtypes are registered via sub-packages, this TargetType is
not needed and should not be used. This removes it and updates the
target.Repository and targets.Service to use the updated code.
This was missed in eac433f7ff.
This moves all of the tcp target functionality into a tcp package. It
also establishes a Register function in the target package to allow
specific target types to be registered at initilazation time. When
registering, a target type needs to provide its Subtype, as well as some
RepositoryHooks that are used by the target Repository.
When creating or updating a target, the following fields should have
been marked as output only:
- host_set_ids
- host_source_ids
- application_credential_source_ids
- application_credential_library_ids
This updates the protobuf and generated files including the swagger file
used by the docs site to correctly mark these are output only.
Fixes#1588
This updates the docker image used for tests to support the new location
for sql migration files and the concept of editions. It continues to
process the old paths for now if the new directory does not exist, so
that the same image can be used during the transition.
This updates the docker image used for tests to support the new location
for sql migration files and the concept of editions. It continues to
process the old paths for now if the new directory does not exist, so
that the same image can be used during the transition.
This refactors the schema package and subpackages to support editions.
An Edition is a collection of sql migrations. A given boundary binary
can have multiple migration editions. The `schema.Manager` is updated to
apply the necessary migrations in the correct order, leveraging an
Edition's priority to determine the order in which the separate Editions
are applied.
The sql migration files are now nested under a directory for the
edition.
This also switches from the generated code to using embed, which
eliminates the need for the `make migrations` target and the generated
code. As such this commit removes the target and generated code.
See:
https://pkg.go.dev/embed
This adds new dimension table to record credential information. Since a
session can have multiple credentials, the wh_session_accumulating_fact
is is linked to the credential_dimensions via bridge tables,
wh_credential_group and wh_credential_group_membership. When a session
is inserted, the credental group and credential dimensions will be
upserted. If the group changed in any way, a new group is inserted along
with all of the corresponding credentials.
The docker package and its StartDbInDocker function should only be used
by the `boundary dev` command now. So moving it into an internal package
of `base` will prevent it from being use for tests accidentally, while
still allowing it to be used for `boundary dev`.
This updates the test-ci target to startup the test database container
first, wait for it to be ready, and then start running the tests.
It also creates some new targets for local devs. Prior to running any
tests either via `make test` or `go test`, the database should be
started with:
make test-database-up
It can take a few seconds for the database to fully initialize, the
status can be checked with:
docker logs -f boundary-sql-tests
This database can be stopped and destroyed with:
make test-database-down
In general it should only need to be restarted when new migrations are
added. But it might also be necessary to restart if some tests fail in
an unexpected way and fail to do cleanup steps.
This swaps the test initialization to use the dbtest package so each
test will get a fresh database created from a template database, instead
of a completely new docker container.
This is modeled after the internal/docker package to make it easy to use as a
replacement. Instead of creating a new docker container for each test,
it will create a new database in an existing database container, using a
template database as a source. This assumes that the template database
exists and already has the schema migrations run against it.
The default template is `boundary_template` which should already have
all migrations applied. However, some tests require an empty database to
test the migration code. In these cases the `template1` template
database should be used, ie:
dbtest.StartWithTemplate("postgres", dbtest.WithTemplate("template1"))
This images is built from the official postgres docker image, but with
updates to optimize the test execution:
- A config file for postgres to increase the performance of the
test server.
- An initialization script to create a template database with all
migrations run against it.
There are new make targets to build and publish this docker image.
The image can be pulled with:
docker pull hashicorpboundary/postgres:13-alpine
To use this image for tests start the container:
make -C internal/db/docker database-up
These fields are being refactored so they conform with the new
wh_credential_dimension. This will inform Business Intelligence tools
that these attributes conform across dimensions.
The tables in the operational part of the Boundary database (meaning
non-warehouse tables) suffix primary key columns and foreign key columns
with `_id`. Many of these columns appear in the warehouse dimension and
fact tables as normal attributes, not primary keys or foreign keys. To
disambiguate ID columns from operational tables, the primary key column
for dimension tables will be renamed from `id` to `key` and the foreign
key columns in the fact tables will be renamed to change the suffix from
`_id` to `_key`.
Now when initializing the test database for sql tests,
it will load the colors persona data into the database. This will
provide a data set that tests can use without needing to call
`wtt_load`.
The main motivation is to allow for a separation of setup and test
execution that requires crossing a transaction boundary. The pgTap tests
are required to exectue as a single transaction, so that the rollback
can reset the state, and each test can be isolated. However, can cause
issues with some test senarios, particuarly with the use of some times
releated functions like `current_timestamp` and `now` which return the
time that the current transaction started. This creates an issue if a
test relies on some setup that sets a time via `now`, then an update
that expects a new time via `now`.
See: https://www.postgresql.org/docs/13/functions-datetime.html#FUNCTIONS-DATETIME-CURRENT
This adds new columns to the wh_user_dimension to provide additional
data from OIDC that does not apply to Password Auth.
Note: Given the specifics of the data, it is possible these columns may
be used by other auth methods in the future, hence the lack of `oidc` in
the column names.
The specific columns are:
- auth_method_external_id
For OIDC this is the Issuer
- auth_account_external_id
For OIDC this is the Subject
- auth_account_full_name
For OIDC this is the account's Full Name, and can be `None`.
- auth_account_email
For OIDC this is the account's Email, and can be `None`.
For all of these new columns, they will be `Not Applicable` if the auth
method is Password.
This sets up the configuration for running a new suite of tests that are
written in sql, to test the sql logic implemented in plpgsql functions
and triggers. It is primarily designed to provides tests for the data
warehouse functionality that is implemented purely in sql.
The infrastructure to run these tests leverages docker. It creates one
docker container using the `library/postgres` image and provisions the
database with the boundary sql migrations and test helper functions. A
separate docker container is run that has the pgtap extension and
pg_prove cli to execute the tests.
As noted, pgtap is used for the tests. This postgresql extension
provides a number of helpful assertion functions to make writing sql
tests easier. It also, along with the pg_prove cli, provides readable
test output.
Along with the new testing infrastructure this commit establishes a
baseline of tests for the wh_user_dimension:
- Unit tests for individual views and functions:
- whx_user_dimension_source
- whx_user_dimension_target
- wh_upsert_user
- Integration tests for that verify that the wh_user_dimension gets
populated correctly when sessions are inserted or updated. This
ensures that:
- the trigger runs on session insert
- the trigger results in the correct changes to the
wh_user_dimension.
This commit provides a `Makefile` to run the tests locally that provides
support for:
- Setting the postgres docker image to easily test multiple versions.
- Passing through options to pg_prove
- Passing the test list to run individual tests
- Additional targets to separately startup the database image to quickly
re-run tests while developing new tests.
Lastly configures circleci to run this test suite as a matrix build
in parallel with the other tests. The matrix build tests multiple
versions of postgres:
- latest
- 13-alpine
- 12-alpine
- 11-alpine
See:
https://pgtap.org/
Timothy Messier