When applying the limit and order by, the where clause also needs to be
applied to yield the correct results. This can be seen with an example.
Assuming a data set where there are 100 sessions, with 10 sessions
belonging to a user with id `u_user1` and 90 sessions belonging to
`u_user2`. If the constructed query is like:
select *
from
(select public_id from session limit 10) s,
session_list ss
where
s.public_id = ss.public_id
and s.user_id = 'u_user1'
order by create_time asc;
The initial sub-query with the limit could return sessions only
belonging to `u_user2`, then it would apply the where clause and exclude
all of the rows.
Instead if the query is:
with
session_ids as (
select public_id
from sessions as s
where
s.user_id = 'u_user1'
order by create_time asc
limit 10
)
select *
from session_list
where
session_list.public_id in (select * from session_ids)
order by create_time asc;
The CTE applies the where clause ensuring that only sessions for
`u_user1` are used for the final select from the view.
Note that the CTE is use primarily for readability.
This also adds an index on `session (create_time)` to aide the query
planner in performing the order by.
Sometimes when creating a lot of sessions concurrently we
can run into a unique constraint error in the warehouse triggers
in the database. Simply retrying the call will allow the
session creation to succeed.
Our test helper packages were not possible to use
with benchmarks. This refactor switches to testing.TB
to allow both tests and benchmarks to use the helpers.
* Reorder session authz checks
This implements a mechanism for service handlers to pass a repo
satisfying an interface into the scope IDs listing function. The purpose
of this interface is to allow the same function to perform authorization
checking against resources rather than requiring service handlers to
fetch all resources in the determined set of scopes and then perform
this checking. While this could be done in each service handler, this
approach centralizes the logic, which has the nice benefit of removing
some boilerplate from service handler list functions that have adapted
to it.
Tests haven't changed, which is expected -- the function should return
exactly what was being returned before, simply faster.
This allows us to use SNI for actual host (e.g. routing) information. We
don't have the client cert yet so we can't look at that, so this
provides a way for us to convey the information needed to look up that
session's TLS stack.
Using SNI for hosts means we also run into the fact that we don't have
automatic agreement in terms of SANs. So when generating the certs we
now also pass worker address information to the function to be encoded
in the cert.
Finally, there is a change in how the websocket dialing happens, because
http.RoundTripper tries to be too clever for its own good and overwrites
NextProtos on a whim.
This removes or reduces the time that is waited in a number of tests by
allowing the grace period time be a configuration option on the
connection repository.
This creates a domain service function `session.WorkerStatusReport`. The
service performs two functions:
1. Checks the status for all of the reported sessions, returning a
report of the sessions that are in a "non-active" state, i.e.
"cacneling" or "terminated".
2. Looks for any "orphaned" session connections, i.e. connections that
are active but were not reported by the worker. These connections are
marked as closed.
This also removes the "ShouldCloseConnectionsOnWorker" logic, as this
was determined to not be a possible state, and was unnecessary.
The query for CloseDeadConnectionsForWorker includes a time comparison
to exclude results that are too close to the current time. This is done
to account for a delay in reporting by the worker. However, since the
delay was hardcoded in the query, it made it difficult to test.
This moves the delay duration into a parameter that is set on the
repository. The amount of the delay can be set at repository creation
time using the new WithWorkerStateDelay option. This defaults to the
previously hardcoded amount.
This will allow tests to control the delay to exercise this
functionality fully.
* chore: Add new dbw dependency
* refactor: Convert to use dbw.DB
* refactor (db): Change GetTicket(...) signature
* feature (db tests): Added TestDeleteWhere(...)
Adding this allows us to eliminate a leak of the gorm library
* refactor: Add ctx to ScanRows(...) signature
This is required to refactor all the deprecated errors pkg calls.
This refactor required adding a context to quite a few call chains
and in particular the job scheduler bits
* feature (db/tests): Add support for test opt: WithTestLogLevel(...)
This test only option allows callers to specify a log level for the
underlying db package (where supported/applicable) when calling
TestSetup(...)
* refactor (db): Simplify Db type
Only store the underlying *DB and create the *RW on demand when
needed.
* refactor (oplog): Port to using dbw pkg for db operations
* refactor (db): Remove gormDB(...) func
* tests (db): Add unit tests for TestDeleteWhere(...)
* chore: make fmt changes to formatting
Creates a new session_credential table and repository functions
to add and list credentials attached to a session. All credentials
are encrypted using the session scope id.
Target subtypes must now implement another repository hook to validate
any CredentialLibraries that are being added/set. This allows for
subtypes to determine the valid set of CredentialLibraries.
For tcp.Targets only CredentialLibaries with
credential.ApplicationPurpose are allowed. If any other purpose is
provided an error is returned.
This change removes the hard-coded credential purpose for the Credential
Libraries. It also moves the logic of creating CredentialLibraries from
CredentialSources out of the repository and into the targets.Service.
This refactor will allow for additional credential purposes to be
supported in both the service and the repository.
This moves all of the tcp target functionality into a tcp package. It
also establishes a Register function in the target package to allow
specific target types to be registered at initilazation time. When
registering, a target type needs to provide its Subtype, as well as some
RepositoryHooks that are used by the target Repository.
* refact: Changes required for gorm v2
* refact: Add db.DB wrapper and refact all test fixtures to use it
Stop leaking the gorm abstraction within our unit tests.
* refact: Migrate direct calls to pq to pgx driver
The refactor will remove the remaining direct dependencies on
the pq driver which has been deprecated in favor of the pgx
driver.
It includes the addition of common.SqlOpen which is a wrapper around
sql.Open, which is needed to deal with the driver name changing from
"postgres" to "pgx" but in our code base the dialect and driver are
synonyms.
This adds new dimension table to record credential information. Since a
session can have multiple credentials, the wh_session_accumulating_fact
is is linked to the credential_dimensions via bridge tables,
wh_credential_group and wh_credential_group_membership. When a session
is inserted, the credental group and credential dimensions will be
upserted. If the group changed in any way, a new group is inserted along
with all of the corresponding credentials.
* add new error funcs which take a ctx and deprecate existing funcs
* use context where available when writing error events
* remove circular internal/errors pkg dep
* suppress some chatty errors using errors.WithoutEvent()
* convert auth oidc funcs to take context where needed to write events
Co-authored-by: Jim <jlambert@hashicorp.com>
This is very useful when trying to write managed groups filters as you
can directly look at the structure of the claims for the accounts you're
trying to filter.
The output of the claims in text format on the CLI currently leaves
something to be desired. I started revamping some of the formatting
functions and it quickly got very fragile. Fundamentally at this point
we're likely better off -- now that we have a good idea what our output
format/standards are like -- rewriting the formatting code. I'll take
that as a tasker as I have a lot of ideas...
This commit adds the support to do the following:
* Mark connections for non-reporting workers as closed. This is the
controller counterpart to the worker functionality (see #1330). This is
written as a scheduled job that does the work DB-side in a single atomic
query.
* Works to reconcile states if such a broken controller-worker
connection resumes and a worker reports a connection as connected that
should be disconnected. In this case, the controller will send an update
request, and the worker will honor it and terminate the connection.
* Further refinement of the grace period setting has been added here.
We have converged on the current server "liveness" setting as our
default here, which is half of the previous 30s (15 seconds, in other
words). Additionally, this is now configurable on the controller and
worker side, with the caveat that it's currently impossible to do so in
config as the setting has been untagged in HCL. This is exposed so that
we can run some sophisticated testing scenarios where we skew the grace
period to either the controller or worker to ensure the aforementioned
reconciliation works.
* Some repository functions have been added to support the new
functionality, in addition to some test code to the worker to allow
querying of session state while testing.
* Finally, we've added some add timestamp subtraction functions
as well, basically serving as the opposite of the addition functions.
This backports some test work that is being added in #1340 to assert
that the worker is actually marking connections closed in its *local*
state on a failure to contact the controller. This was missed in #1357
during some late-stage refactoring.
Summary of the changes:
* If we can't actually contact the controller during connection closure
to mark the connections as closed, we actually construct a "fake"
response that just simply includes a closed state for all connections
requested. This is then used to update the local connection state.
* Added a 3 second timeout to the CloseConnection request to the
controller to ensure the call to close connections completed in a
timely manner and did not hang indefinitely.
Testing enhancements:
* Added a new helper in TestHelper called LookupSession that allows the
lookup of a session from the worker's internal session state.
* This is used in the new ExpectConnectionStateOnWorker check in the E2E
test, which is part of a larger amount of enhancements to the E2E test
that are coming in #1340.
* Added a ConnectionStatusFromProtoVal helper function to the session
package to allow the conversion of protobuf connection status values to
their higher-level counterparts; essentially it is a reverse of
ConnectionStatus.ProtoVal. This is used in some of our tests (with more
coming in #1340) to just help with keeping things more idiomatic.
Finally, there is some minor error refactoring, and we also change the
timeout on internal status and connection close requests to the
controller to 5 seconds.
* Fix down migrations
* Udpates due to merge
* Sentinel values only have to be greater than 0
* Create credential related sdk structs and methods.
* Post merge interface updating.
* Removing scratch code that was not cleaned up.
* updates
* Convert client certificate and client certificate keys into pem blocks and validate.
* Add newline at end of file.
* Faster check for "s" at the end of a resource name when generating from templates.
* make fmt
* updates
* Add dynamic credentials to a session
* Refactor: move contents of file
* updates
* Updates
* Add InvalidDynamicCredential error
* Assign credentials
* Allow updating client certificates seperate from the client certificate key.
* Replace application purpose string with enum value
* PR feedback / fix go test .
* Delete dead code
* Use common view
* Refactor: extract interface and rename method
* Refactor: move code to eliminate privPurpLibrary
* Add comment to requestMap
* Refactor: do not export methods on an unexported struct
* Update internal/db/schema/migrations/postgres/10/03_vault_credential.up.sql
Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>
* Add comments to credential Purpose constants
* Reorder switch statements to handle errors first
* PR feedback
* update
* fix merge
* Organizing the scheduler code and extract the individual store creation test helper function.
* Run 'make fmt'
* Create certificate when creating store. Allow vault address to be passed in as option
* Refactor: rename database views and associated structs
This change renames two database views and associated go structs to use
a consistent naming pattern. Views ending in 'private' contain encrypted
credentials needed for connecting to Vault. Views ending in 'public'
mirror the 'private' views but do not include any encrypted values.
This change does not include any changes to functionality or behavior.
* Adding SAD target operations for credential libraries to the SDK.
* Add status field and enumeration table for credential status
* Refactor: rename Status to TokenStatus
* Add status to vault credentials
* Creating the template for the CLI and helper functions.
* Add private database view for vault credentials
* Add scope_id to view and cleanup TODO comment
* Reformat embedded sql queries
* Add vault client helpers
* updates
* white space
* Add LookupLease test
* CLI commands for Store work. Renamed fields in proto w/ vault prefix.
* Test issue credentials with client TLS
* Add additional checks to vault ping
* Add an error for vault token capabilities and create a VaultToken error kind
* Add support for working with Vault policies
* Use Vault 1.7.2 for testing
* Add testing helper for creating a vault client with a non-root token
* Fix Vault renew lease test
The Vault renew lease endpoint accepts an optional duration parameter
for requesting the number of seconds to extend the lease by. However,
Vault is not required to honor this request.
* Add a var for the required capabilities of a boundary Vault token
* Enhance the test helper for adding a policy to Vault
* Allow the test role in postgresql to revoke credentials
* Remove out of date comments
* Enhance Vault test helpers for the database secrets engine
* Reformat comments
* Add a method for getting the capabilities of the current Vault token
* Add revoke lease to internal Vault client
* Reformat: combine err return line with check for nil line
* Remove redundant checks in test code
Remove checks in test code that are redundant with checks in the test
harness.
* Refactor CreateToken test helper to return the secret and the token
* Fix build failures caused by signature change in vault.CreateToken
* Fix Vault version and Vault API link in comments
* ran `make gen` after merging from mgaffney-vault.
* Remove Vault prefix from attribute fields in API, create a base CLI option for adding field prefixes. format now uses that prefix for error reporting.
* Template now allows attaching attribute field prefixes.
* Credential store uses PrefixAttributeFieldErrorsWithSubactionPrefix.
* Fixing naming of vault flags and helper flags.
* Updating more references to generated api methods.
* Make the capabilities String method more readable
* Add comment outlining PrintApiError usage of Options.
* Refactor test to move declarations closer to first use
* Fix comment
* Implement revoke method on vault repository
* Tests now verify certs, CA, and tokens, addresses can be updated in a store.
* Added test for creating store with pk/cert in same field.
* Allo updating certificates seperate from private key. Only contact backing vault service when updating token.
* Fixing bad tests and removing unused code.
* Adding tests for various ways of updating tokens.
* Add private credential
* Add token revocation job and cred renewal/revocation job
* Regen after merge.
* Fixing field mask related merge error.
* Create Library CLI. Removes vault prefix from path. Adds a WithoutCleanup testOption for vault.
* Set all credentials to revoked when a token is revoked
* Fixing missed renaming update.
* fix copy/paste error.
* Fixing code generation errors after merge.
* make gen after merge.
* Update internal/cmd/commands/credentialstorescmd/vault_funcs.go
Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>
* Update internal/cmd/commands/credentialstorescmd/vault_funcs.go
Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>
* Update internal/cmd/commands/credentialstorescmd/vault_funcs.go
Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>
* Refactor: move migrations from 10 to 12
* Removing logic from transaction.
* wrapping client errors in the calling op for CredentialStore's client() method.
* Redact token
* redact client key
* updates
* Add trigger to revoke credentials when session is canceled or terminated
* Fixing some help text for some flags.
* Add ability to load values from files (file://) or environment variables (env://).
* Updating comments
* fix bad merge
* feedback
* Adding issued credentials to sessions.
* Error on capabilities
* updates
* Expose NewVaulTestServer
* Add godoc
* Adding generated SessionCredential struct for API.
* Progress on building handler test for Authorize Session.
* feedback
* Update policies
* Add 'revoke' as a status for Vault Tokens
* Add delete_time to vault credential store
* Create dynamic credentials at session creation time.
* Tests pass with credentials attached to AuthorizedSession result.
* Only call issue if there are credentials to issue.
* Fixing output only issues. Adding type to credential library.
* Fix test to clear un/pw generated. confirm generated un/pw changes from 1 authorize session call to another.
* Update lookupTokenError
* Add delete_time to private credential store
* Refactor: move private store to separate file
* Fixing tests for AuthorizeSession and verifying error cases.
* Only create the credential repo if there are credentials to issue.
* Rename Library field to CredentialLibrary.
* Fixing missing change of Library to CredentialLibrary.
* Removing Default SDK functions for fields which are required and update CLI to match.
* Replace resource specific id variables with "id" and the input parmater for paths to be just a single string and the resource name as it should show up in the path.
* Soft delete vault credential stores
* Prevent new SQL functions from silently overwriting existing functions
* Rename "Hmac" to "HMAC" in the human readable cli response table.
* Fix tests
* Add vault database cleanup jobs
* Add ability to set the token period in the vault test harness
* updates
* Add not_null_columns func test
* remove unused struct
* Up no output timeout
* Validate current vault token on addr update
* Fix client cert change test
* add cert test
* Update revocation job where
* Fix comment
* Use Vault's default port in CLI help strings
* Remove down migrations
* Increase migration file numbers by 1
* Run make migrations
* Move migrations from 12 to 10
* Increase Circle CI timeout for acceptance tests
* Fix spelling mistake in error messages
* ASD cmds for Target and returning credentials to AuthSession request (#1338)
* Add Add/Set/Remove commands for libraries on Targets. Return credentials on AuthorizeSession request.
Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>
Co-authored-by: Louis Ruch <louisruch@gmail.com>
Co-authored-by: Todd Knight <T.Alan.Knight@gmail.com>
Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>
irenarindos
Timothy Messier
Jeff Mitchell
Johan Brandhorst-Satzkorn
Jim
Lars Lehtonen
Louis Ruch
Thor
s-christoffandJim
Chris MarchesiandMichael Gaffney
Michael Gaffney