* chore: Add new dbw dependency
* refactor: Convert to use dbw.DB
* refactor (db): Change GetTicket(...) signature
* feature (db tests): Added TestDeleteWhere(...)
Adding this allows us to eliminate a leak of the gorm library
* refactor: Add ctx to ScanRows(...) signature
This is required to refactor all the deprecated errors pkg calls.
This refactor required adding a context to quite a few call chains
and in particular the job scheduler bits
* feature (db/tests): Add support for test opt: WithTestLogLevel(...)
This test only option allows callers to specify a log level for the
underlying db package (where supported/applicable) when calling
TestSetup(...)
* refactor (db): Simplify Db type
Only store the underlying *DB and create the *RW on demand when
needed.
* refactor (oplog): Port to using dbw pkg for db operations
* refactor (db): Remove gormDB(...) func
* tests (db): Add unit tests for TestDeleteWhere(...)
* chore: make fmt changes to formatting
This change adds a credential type attribute to credential library and
allows it to be set when creating a credential library.
Specifying the type of credential returned from a library allows for new
features and additional error checking to be added over time. A
User Password credential type and an Unspecified credential type are
included with the expectation that additional credential types will be
added in the future. This change also adds the credential type attribute
to existing credential libraries and sets it to Unspecified.
Known Limitations:
- Credential type is only supported on the repository create method.
Support for the other CRUD methods will be added in the future.
Currently host does not use a version, which means it's not protected
against race conditions of either itself or its value objects. This adds
and plumbs through version to ensure that if a host or its value objects
change that they are not going to be affected by a race condition.
This adds a parameter to host sets allowing the sync interval to be
user-specified on a per-host-set basis. A value of -1 disables syncing;
a value of null uses the current default.
* feature (db): Add the ability to enable debug output for a connection
* refactor: Use scope.Global instead of "global"
* feature (db/migrations): Add kms audit keys
Add schema changes to support a new DEK for audit crypto operations.
* feature (kms/audit): Add audit key type
All the bits required to define a new DEK concrete type for
operating on audit entries.
* refactor (db): Remove deprecated migration unit tests.
We added a new audit DEK to kms.CreateKeysTx(...). Unfortunately,
unit tests like TestMigrations_UserDimension,
Test_PrimaryAuthMethodChanges and TestMigrations_CredentialDimension
have an indirect dependency on a specific historical set of DEKs via
an indirect call kms.CreateKeysTx. This of course fails because the
new DEK's tables is not a part of pinned migration versions that these
tests depend on.
This change removes the hard-coded credential purpose for the Credential
Libraries. It also moves the logic of creating CredentialLibraries from
CredentialSources out of the repository and into the targets.Service.
This refactor will allow for additional credential purposes to be
supported in both the service and the repository.
* Add DB changes for host, set members, and host addresses for plugins.
* Add priority, address review feedback, update tests, naming, protos
Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>
Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>
This moves all of the tcp target functionality into a tcp package. It
also establishes a Register function in the target package to allow
specific target types to be registered at initilazation time. When
registering, a target type needs to provide its Subtype, as well as some
RepositoryHooks that are used by the target Repository.
Added a new Endpoint struct at the top level host package level to capture host, set, and address chosen.
Temporarily add a host_plugin_host table to capture the hosts returned by the plugins. This ensures the datawarehouse is able to query the table to decide what host was chosen. The view used to update the datawarehouse dimension table provides rows with host set ids and hosts which are not in that host set. This is ok since the function which updates the dimension table provides the host set and host id chosen so that relationship doesn't need to be captured in the view itself.
Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>
* Remove Prefix Id and Plugin Name fields. Creating catalogs now require either the plugin id or the name of the plugin resource (but not both). Host Catalogs and Host Sets API resources now have a read only plugin field populated with plugin info. Host Catalogs have a plugin id field as a top level field.
* Create minimalistic plugin map and test plugin.
* Pass secrets and attributes to the plugin when creating catalogs or sets. Add tests for CreateSet and CreateCatalog repo methods.
* make fmt and extract out getPrivateCatalog and getCatalog helper methods.
* Exporting TestPluginServer so it can be used in the handlers.
* Keep separate persisted data from host catalogs lookups which facilitate passing it onto the plugins.
* Add TODO to merge get Catalog and get Persisted data.
* Adding tests for unimplemented being returned by the plugin.
* Use GetFields() instead of AsMap() to confirm attributes should be populated. Make clone consistent between catalog and sets.
Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>
* feat(host): Add host plugin related schemas and tests
This adds the schema files for host plugins, plugin based catalogs and
plugin based sets. This also adds the protos and tests which utilize
gorm to ensure the schemas enforce the constraints we expect.
Along with this change was a migration which moved catalog names to the
base type table so we can ensure all catalogs in a single scope,
regardless of subtype, have a unique name.
* Updated plugin schemas to do the following:
- Add plugin_executable table
- plugin table now also has scope_id and name w/ uniqueness constraints
- host_plugin no longer has id_prefix column
- host_plugin separates name and plugin_name
- plugin_version table renamed version to symantic_version column.
* Fix typo in delete_plugin_subtype trigger. Add update test for host plugins.
* Update id prefixes. Fix public_id typos in protos and plugin_executable name in sql diagrams.
* Created internal/plugin and moved base plugin structs to it and moved host plugin subdirectory to it.
* Created internal/plugin and moved base plugin structs to it and moved host plugin subdirectory to it.
* Add first pass at enums for OS and Arch supported by plugin executables.
* Capitalizing the const enums for OS and Architectures.
* Add comments to sql files.
* Reviewer comments.
Remove unsupported OSes and Archs from SQL and go constants.
Add missing tests.
* Be more permissive in the values the db accepts.
The go logic can restrict which values it will allow getting populated into the db.
* Removing constraints on which values can be placed inside operating_system and architecture columns.
* Updating test helper functions to only create a single catalog or set.
* Move files to new migration version.
* Fixing job/plugin test.
* Update table names, named a number of constraints, remove plugin version and executable tables.
* Add constraints for plugin name and id prefix.
* Rename constraint, remove unhelpful comment block.
* Adding semantic version to plugin_host and set attributes to not null.
* Remove semantic versioning from all locations.
* Remove semantic_version from plugin_host table comment.
* Fix op copy/paste error. Don't use deprecated errors methods. Update constraint names. Add docs.
* Apply suggestions from code review
Fixing constraint names.
Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>
This is very useful when trying to write managed groups filters as you
can directly look at the structure of the claims for the accounts you're
trying to filter.
The output of the claims in text format on the CLI currently leaves
something to be desired. I started revamping some of the formatting
functions and it quickly got very fragile. Fundamentally at this point
we're likely better off -- now that we have a good idea what our output
format/standards are like -- rewriting the formatting code. I'll take
that as a tasker as I have a lot of ideas...
* Fix down migrations
* Udpates due to merge
* Sentinel values only have to be greater than 0
* Create credential related sdk structs and methods.
* Post merge interface updating.
* Removing scratch code that was not cleaned up.
* updates
* Convert client certificate and client certificate keys into pem blocks and validate.
* Add newline at end of file.
* Faster check for "s" at the end of a resource name when generating from templates.
* make fmt
* updates
* Add dynamic credentials to a session
* Refactor: move contents of file
* updates
* Updates
* Add InvalidDynamicCredential error
* Assign credentials
* Allow updating client certificates seperate from the client certificate key.
* Replace application purpose string with enum value
* PR feedback / fix go test .
* Delete dead code
* Use common view
* Refactor: extract interface and rename method
* Refactor: move code to eliminate privPurpLibrary
* Add comment to requestMap
* Refactor: do not export methods on an unexported struct
* Update internal/db/schema/migrations/postgres/10/03_vault_credential.up.sql
Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>
* Add comments to credential Purpose constants
* Reorder switch statements to handle errors first
* PR feedback
* update
* fix merge
* Organizing the scheduler code and extract the individual store creation test helper function.
* Run 'make fmt'
* Create certificate when creating store. Allow vault address to be passed in as option
* Refactor: rename database views and associated structs
This change renames two database views and associated go structs to use
a consistent naming pattern. Views ending in 'private' contain encrypted
credentials needed for connecting to Vault. Views ending in 'public'
mirror the 'private' views but do not include any encrypted values.
This change does not include any changes to functionality or behavior.
* Adding SAD target operations for credential libraries to the SDK.
* Add status field and enumeration table for credential status
* Refactor: rename Status to TokenStatus
* Add status to vault credentials
* Creating the template for the CLI and helper functions.
* Add private database view for vault credentials
* Add scope_id to view and cleanup TODO comment
* Reformat embedded sql queries
* Add vault client helpers
* updates
* white space
* Add LookupLease test
* CLI commands for Store work. Renamed fields in proto w/ vault prefix.
* Test issue credentials with client TLS
* Add additional checks to vault ping
* Add an error for vault token capabilities and create a VaultToken error kind
* Add support for working with Vault policies
* Use Vault 1.7.2 for testing
* Add testing helper for creating a vault client with a non-root token
* Fix Vault renew lease test
The Vault renew lease endpoint accepts an optional duration parameter
for requesting the number of seconds to extend the lease by. However,
Vault is not required to honor this request.
* Add a var for the required capabilities of a boundary Vault token
* Enhance the test helper for adding a policy to Vault
* Allow the test role in postgresql to revoke credentials
* Remove out of date comments
* Enhance Vault test helpers for the database secrets engine
* Reformat comments
* Add a method for getting the capabilities of the current Vault token
* Add revoke lease to internal Vault client
* Reformat: combine err return line with check for nil line
* Remove redundant checks in test code
Remove checks in test code that are redundant with checks in the test
harness.
* Refactor CreateToken test helper to return the secret and the token
* Fix build failures caused by signature change in vault.CreateToken
* Fix Vault version and Vault API link in comments
* ran `make gen` after merging from mgaffney-vault.
* Remove Vault prefix from attribute fields in API, create a base CLI option for adding field prefixes. format now uses that prefix for error reporting.
* Template now allows attaching attribute field prefixes.
* Credential store uses PrefixAttributeFieldErrorsWithSubactionPrefix.
* Fixing naming of vault flags and helper flags.
* Updating more references to generated api methods.
* Make the capabilities String method more readable
* Add comment outlining PrintApiError usage of Options.
* Refactor test to move declarations closer to first use
* Fix comment
* Implement revoke method on vault repository
* Tests now verify certs, CA, and tokens, addresses can be updated in a store.
* Added test for creating store with pk/cert in same field.
* Allo updating certificates seperate from private key. Only contact backing vault service when updating token.
* Fixing bad tests and removing unused code.
* Adding tests for various ways of updating tokens.
* Add private credential
* Add token revocation job and cred renewal/revocation job
* Regen after merge.
* Fixing field mask related merge error.
* Create Library CLI. Removes vault prefix from path. Adds a WithoutCleanup testOption for vault.
* Set all credentials to revoked when a token is revoked
* Fixing missed renaming update.
* fix copy/paste error.
* Fixing code generation errors after merge.
* make gen after merge.
* Update internal/cmd/commands/credentialstorescmd/vault_funcs.go
Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>
* Update internal/cmd/commands/credentialstorescmd/vault_funcs.go
Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>
* Update internal/cmd/commands/credentialstorescmd/vault_funcs.go
Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>
* Refactor: move migrations from 10 to 12
* Removing logic from transaction.
* wrapping client errors in the calling op for CredentialStore's client() method.
* Redact token
* redact client key
* updates
* Add trigger to revoke credentials when session is canceled or terminated
* Fixing some help text for some flags.
* Add ability to load values from files (file://) or environment variables (env://).
* Updating comments
* fix bad merge
* feedback
* Adding issued credentials to sessions.
* Error on capabilities
* updates
* Expose NewVaulTestServer
* Add godoc
* Adding generated SessionCredential struct for API.
* Progress on building handler test for Authorize Session.
* feedback
* Update policies
* Add 'revoke' as a status for Vault Tokens
* Add delete_time to vault credential store
* Create dynamic credentials at session creation time.
* Tests pass with credentials attached to AuthorizedSession result.
* Only call issue if there are credentials to issue.
* Fixing output only issues. Adding type to credential library.
* Fix test to clear un/pw generated. confirm generated un/pw changes from 1 authorize session call to another.
* Update lookupTokenError
* Add delete_time to private credential store
* Refactor: move private store to separate file
* Fixing tests for AuthorizeSession and verifying error cases.
* Only create the credential repo if there are credentials to issue.
* Rename Library field to CredentialLibrary.
* Fixing missing change of Library to CredentialLibrary.
* Removing Default SDK functions for fields which are required and update CLI to match.
* Replace resource specific id variables with "id" and the input parmater for paths to be just a single string and the resource name as it should show up in the path.
* Soft delete vault credential stores
* Prevent new SQL functions from silently overwriting existing functions
* Rename "Hmac" to "HMAC" in the human readable cli response table.
* Fix tests
* Add vault database cleanup jobs
* Add ability to set the token period in the vault test harness
* updates
* Add not_null_columns func test
* remove unused struct
* Up no output timeout
* Validate current vault token on addr update
* Fix client cert change test
* add cert test
* Update revocation job where
* Fix comment
* Use Vault's default port in CLI help strings
* Remove down migrations
* Increase migration file numbers by 1
* Run make migrations
* Move migrations from 12 to 10
* Increase Circle CI timeout for acceptance tests
* Fix spelling mistake in error messages
* ASD cmds for Target and returning credentials to AuthSession request (#1338)
* Add Add/Set/Remove commands for libraries on Targets. Return credentials on AuthorizeSession request.
Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>
Co-authored-by: Louis Ruch <louisruch@gmail.com>
Co-authored-by: Todd Knight <T.Alan.Knight@gmail.com>
Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>
* Adds job repo
* Adds scheduler
* Adds plugin table which is a place holder for future expansion of the scheduler. Currently all jobs that are created are linked to the pi_system plugin which are jobs that are held in a local registeredJobs map populated at controller startup.
* Update protos and gen
* Add new values to targets CLI command
* Lots more plumbing of new parameters through everything
* allow fields to be set to zero value.
* Disallow set but empty name/description strings in create/update verifiers
Co-authored-by: Jim Lambert <jlambert@hashicorp.com>
Jim
Michael Gaffney
Jeff MitchellandJim
Todd
Timothy Messier
