* feature (db): Add the ability to enable debug output for a connection
* refactor: Use scope.Global instead of "global"
* feature (db/migrations): Add kms audit keys
Add schema changes to support a new DEK for audit crypto operations.
* feature (kms/audit): Add audit key type
All the bits required to define a new DEK concrete type for
operating on audit entries.
* refactor (db): Remove deprecated migration unit tests.
We added a new audit DEK to kms.CreateKeysTx(...). Unfortunately,
unit tests like TestMigrations_UserDimension,
Test_PrimaryAuthMethodChanges and TestMigrations_CredentialDimension
have an indirect dependency on a specific historical set of DEKs via
an indirect call kms.CreateKeysTx. This of course fails because the
new DEK's tables is not a part of pinned migration versions that these
tests depend on.
This change removes the hard-coded credential purpose for the Credential
Libraries. It also moves the logic of creating CredentialLibraries from
CredentialSources out of the repository and into the targets.Service.
This refactor will allow for additional credential purposes to be
supported in both the service and the repository.
This moves all of the tcp target functionality into a tcp package. It
also establishes a Register function in the target package to allow
specific target types to be registered at initilazation time. When
registering, a target type needs to provide its Subtype, as well as some
RepositoryHooks that are used by the target Repository.
This refactors the schema package and subpackages to support editions.
An Edition is a collection of sql migrations. A given boundary binary
can have multiple migration editions. The `schema.Manager` is updated to
apply the necessary migrations in the correct order, leveraging an
Edition's priority to determine the order in which the separate Editions
are applied.
The sql migration files are now nested under a directory for the
edition.
This also switches from the generated code to using embed, which
eliminates the need for the `make migrations` target and the generated
code. As such this commit removes the target and generated code.
See:
https://pkg.go.dev/embed
* feat (events): Add status code to audit events
* feat (events): Add the resp proto msg to every audit event
* chore (events): Update eventlogger dep
This latest version will redact any map and/or map field which isn't
explicitly classified via a Taggable interface
* feat (events): Begin to add tagging to response proto msgs
Pass correct context to docker build for multiarch and dev build targets.
* fixed missing file errors; bumped boundary version
* downgraded alpine due to build issues
* fixing makefile
* bumping boundary and alpine versions
* fixed docker directory in makefile and dockerfile
* updating dev.dockerfile & makefile
This updates the test-ci target to startup the test database container
first, wait for it to be ready, and then start running the tests.
It also creates some new targets for local devs. Prior to running any
tests either via `make test` or `go test`, the database should be
started with:
make test-database-up
It can take a few seconds for the database to fully initialize, the
status can be checked with:
docker logs -f boundary-sql-tests
This database can be stopped and destroyed with:
make test-database-down
In general it should only need to be restarted when new migrations are
added. But it might also be necessary to restart if some tests fail in
an unexpected way and fail to do cleanup steps.
* feat(host): Add host plugin related schemas and tests
This adds the schema files for host plugins, plugin based catalogs and
plugin based sets. This also adds the protos and tests which utilize
gorm to ensure the schemas enforce the constraints we expect.
Along with this change was a migration which moved catalog names to the
base type table so we can ensure all catalogs in a single scope,
regardless of subtype, have a unique name.
* Updated plugin schemas to do the following:
- Add plugin_executable table
- plugin table now also has scope_id and name w/ uniqueness constraints
- host_plugin no longer has id_prefix column
- host_plugin separates name and plugin_name
- plugin_version table renamed version to symantic_version column.
* Fix typo in delete_plugin_subtype trigger. Add update test for host plugins.
* Update id prefixes. Fix public_id typos in protos and plugin_executable name in sql diagrams.
* Created internal/plugin and moved base plugin structs to it and moved host plugin subdirectory to it.
* Created internal/plugin and moved base plugin structs to it and moved host plugin subdirectory to it.
* Add first pass at enums for OS and Arch supported by plugin executables.
* Capitalizing the const enums for OS and Architectures.
* Add comments to sql files.
* Reviewer comments.
Remove unsupported OSes and Archs from SQL and go constants.
Add missing tests.
* Be more permissive in the values the db accepts.
The go logic can restrict which values it will allow getting populated into the db.
* Removing constraints on which values can be placed inside operating_system and architecture columns.
* Updating test helper functions to only create a single catalog or set.
* Move files to new migration version.
* Fixing job/plugin test.
* Update table names, named a number of constraints, remove plugin version and executable tables.
* Add constraints for plugin name and id prefix.
* Rename constraint, remove unhelpful comment block.
* Adding semantic version to plugin_host and set attributes to not null.
* Remove semantic versioning from all locations.
* Remove semantic_version from plugin_host table comment.
* Fix op copy/paste error. Don't use deprecated errors methods. Update constraint names. Add docs.
* Apply suggestions from code review
Fixing constraint names.
Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>
* feat (events): Add new options needed for AuditConfig
WithAuditWrapper and WithFilterOperations all for optional paramters
when creating AuditConfig types.
* refactor (event): Change EventerConfig.Sinks to a ptr
We need to modify the SinkConfig.AuditConfig during the NewEventer factory, so we have to use a ptr not a ref.
* feat (events): Add audit configuration types
* feat (events): add audit config to sink configuration.
* feat (events): add redact/encrypt node for audit events to Eventer
Integrate the encrypt.Filter node from the eventlogger into the
event.Eventer, so audit events can be redacted/encrypted based
on data classifications.
* test (events) Add test protobufs for events
Add test protobufs along with a test service for testing
the audit event encrypt node
This sets up the configuration for running a new suite of tests that are
written in sql, to test the sql logic implemented in plpgsql functions
and triggers. It is primarily designed to provides tests for the data
warehouse functionality that is implemented purely in sql.
The infrastructure to run these tests leverages docker. It creates one
docker container using the `library/postgres` image and provisions the
database with the boundary sql migrations and test helper functions. A
separate docker container is run that has the pgtap extension and
pg_prove cli to execute the tests.
As noted, pgtap is used for the tests. This postgresql extension
provides a number of helpful assertion functions to make writing sql
tests easier. It also, along with the pg_prove cli, provides readable
test output.
Along with the new testing infrastructure this commit establishes a
baseline of tests for the wh_user_dimension:
- Unit tests for individual views and functions:
- whx_user_dimension_source
- whx_user_dimension_target
- wh_upsert_user
- Integration tests for that verify that the wh_user_dimension gets
populated correctly when sessions are inserted or updated. This
ensures that:
- the trigger runs on session insert
- the trigger results in the correct changes to the
wh_user_dimension.
This commit provides a `Makefile` to run the tests locally that provides
support for:
- Setting the postgres docker image to easily test multiple versions.
- Passing through options to pg_prove
- Passing the test list to run individual tests
- Additional targets to separately startup the database image to quickly
re-run tests while developing new tests.
Lastly configures circleci to run this test suite as a matrix build
in parallel with the other tests. The matrix build tests multiple
versions of postgres:
- latest
- 13-alpine
- 12-alpine
- 11-alpine
See:
https://pgtap.org/
* updated dockerfile to support multi-arch; added new section in Make for multi-arch
* adding a line to changelog
* switching base image to hashicorp mirror
* Fix down migrations
* Udpates due to merge
* Sentinel values only have to be greater than 0
* Create credential related sdk structs and methods.
* Post merge interface updating.
* Removing scratch code that was not cleaned up.
* updates
* Convert client certificate and client certificate keys into pem blocks and validate.
* Add newline at end of file.
* Faster check for "s" at the end of a resource name when generating from templates.
* make fmt
* updates
* Add dynamic credentials to a session
* Refactor: move contents of file
* updates
* Updates
* Add InvalidDynamicCredential error
* Assign credentials
* Allow updating client certificates seperate from the client certificate key.
* Replace application purpose string with enum value
* PR feedback / fix go test .
* Delete dead code
* Use common view
* Refactor: extract interface and rename method
* Refactor: move code to eliminate privPurpLibrary
* Add comment to requestMap
* Refactor: do not export methods on an unexported struct
* Update internal/db/schema/migrations/postgres/10/03_vault_credential.up.sql
Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>
* Add comments to credential Purpose constants
* Reorder switch statements to handle errors first
* PR feedback
* update
* fix merge
* Organizing the scheduler code and extract the individual store creation test helper function.
* Run 'make fmt'
* Create certificate when creating store. Allow vault address to be passed in as option
* Refactor: rename database views and associated structs
This change renames two database views and associated go structs to use
a consistent naming pattern. Views ending in 'private' contain encrypted
credentials needed for connecting to Vault. Views ending in 'public'
mirror the 'private' views but do not include any encrypted values.
This change does not include any changes to functionality or behavior.
* Adding SAD target operations for credential libraries to the SDK.
* Add status field and enumeration table for credential status
* Refactor: rename Status to TokenStatus
* Add status to vault credentials
* Creating the template for the CLI and helper functions.
* Add private database view for vault credentials
* Add scope_id to view and cleanup TODO comment
* Reformat embedded sql queries
* Add vault client helpers
* updates
* white space
* Add LookupLease test
* CLI commands for Store work. Renamed fields in proto w/ vault prefix.
* Test issue credentials with client TLS
* Add additional checks to vault ping
* Add an error for vault token capabilities and create a VaultToken error kind
* Add support for working with Vault policies
* Use Vault 1.7.2 for testing
* Add testing helper for creating a vault client with a non-root token
* Fix Vault renew lease test
The Vault renew lease endpoint accepts an optional duration parameter
for requesting the number of seconds to extend the lease by. However,
Vault is not required to honor this request.
* Add a var for the required capabilities of a boundary Vault token
* Enhance the test helper for adding a policy to Vault
* Allow the test role in postgresql to revoke credentials
* Remove out of date comments
* Enhance Vault test helpers for the database secrets engine
* Reformat comments
* Add a method for getting the capabilities of the current Vault token
* Add revoke lease to internal Vault client
* Reformat: combine err return line with check for nil line
* Remove redundant checks in test code
Remove checks in test code that are redundant with checks in the test
harness.
* Refactor CreateToken test helper to return the secret and the token
* Fix build failures caused by signature change in vault.CreateToken
* Fix Vault version and Vault API link in comments
* ran `make gen` after merging from mgaffney-vault.
* Remove Vault prefix from attribute fields in API, create a base CLI option for adding field prefixes. format now uses that prefix for error reporting.
* Template now allows attaching attribute field prefixes.
* Credential store uses PrefixAttributeFieldErrorsWithSubactionPrefix.
* Fixing naming of vault flags and helper flags.
* Updating more references to generated api methods.
* Make the capabilities String method more readable
* Add comment outlining PrintApiError usage of Options.
* Refactor test to move declarations closer to first use
* Fix comment
* Implement revoke method on vault repository
* Tests now verify certs, CA, and tokens, addresses can be updated in a store.
* Added test for creating store with pk/cert in same field.
* Allo updating certificates seperate from private key. Only contact backing vault service when updating token.
* Fixing bad tests and removing unused code.
* Adding tests for various ways of updating tokens.
* Add private credential
* Add token revocation job and cred renewal/revocation job
* Regen after merge.
* Fixing field mask related merge error.
* Create Library CLI. Removes vault prefix from path. Adds a WithoutCleanup testOption for vault.
* Set all credentials to revoked when a token is revoked
* Fixing missed renaming update.
* fix copy/paste error.
* Fixing code generation errors after merge.
* make gen after merge.
* Update internal/cmd/commands/credentialstorescmd/vault_funcs.go
Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>
* Update internal/cmd/commands/credentialstorescmd/vault_funcs.go
Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>
* Update internal/cmd/commands/credentialstorescmd/vault_funcs.go
Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>
* Refactor: move migrations from 10 to 12
* Removing logic from transaction.
* wrapping client errors in the calling op for CredentialStore's client() method.
* Redact token
* redact client key
* updates
* Add trigger to revoke credentials when session is canceled or terminated
* Fixing some help text for some flags.
* Add ability to load values from files (file://) or environment variables (env://).
* Updating comments
* fix bad merge
* feedback
* Adding issued credentials to sessions.
* Error on capabilities
* updates
* Expose NewVaulTestServer
* Add godoc
* Adding generated SessionCredential struct for API.
* Progress on building handler test for Authorize Session.
* feedback
* Update policies
* Add 'revoke' as a status for Vault Tokens
* Add delete_time to vault credential store
* Create dynamic credentials at session creation time.
* Tests pass with credentials attached to AuthorizedSession result.
* Only call issue if there are credentials to issue.
* Fixing output only issues. Adding type to credential library.
* Fix test to clear un/pw generated. confirm generated un/pw changes from 1 authorize session call to another.
* Update lookupTokenError
* Add delete_time to private credential store
* Refactor: move private store to separate file
* Fixing tests for AuthorizeSession and verifying error cases.
* Only create the credential repo if there are credentials to issue.
* Rename Library field to CredentialLibrary.
* Fixing missing change of Library to CredentialLibrary.
* Removing Default SDK functions for fields which are required and update CLI to match.
* Replace resource specific id variables with "id" and the input parmater for paths to be just a single string and the resource name as it should show up in the path.
* Soft delete vault credential stores
* Prevent new SQL functions from silently overwriting existing functions
* Rename "Hmac" to "HMAC" in the human readable cli response table.
* Fix tests
* Add vault database cleanup jobs
* Add ability to set the token period in the vault test harness
* updates
* Add not_null_columns func test
* remove unused struct
* Up no output timeout
* Validate current vault token on addr update
* Fix client cert change test
* add cert test
* Update revocation job where
* Fix comment
* Use Vault's default port in CLI help strings
* Remove down migrations
* Increase migration file numbers by 1
* Run make migrations
* Move migrations from 12 to 10
* Increase Circle CI timeout for acceptance tests
* Fix spelling mistake in error messages
* ASD cmds for Target and returning credentials to AuthSession request (#1338)
* Add Add/Set/Remove commands for libraries on Targets. Return credentials on AuthorizeSession request.
Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>
Co-authored-by: Louis Ruch <louisruch@gmail.com>
Co-authored-by: Todd Knight <T.Alan.Knight@gmail.com>
Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>
* Adds job repo
* Adds scheduler
* Adds plugin table which is a place holder for future expansion of the scheduler. Currently all jobs that are created are linked to the pi_system plugin which are jobs that are held in a local registeredJobs map populated at controller startup.
Louis Ruch
Todd Knight
Jim
Jeff Mitchell
Timothy Messier
Jacob Mammoliti
Chris Marchesi
s-christoff