Commit Graph
315 Commits
Author SHA1 Message Date
Jim 785241237e refactor (kms): adopt the go-kms-wrapping/extras/kms package (#2027)
* feature (db): add Db.UnderlyingDB()

This is needed to allow for integration with bits that need a *dbw.DB

* feature (kms): add new go-kms-wrapping/extras/kms dependency

* refactor (kms): schema changes required to adopt extras/kms pkg

* refactor (kms/iam): rename to DeprecateCreateKeysTx(...)

* refactor (kms): remove deprecated test

* refactor (kms): refactor Kms type

This refactor includes renaming the existing type to DeprecatedKms
and then refactoring the Kms type to become a wrapper of
go-kms-wrapping/extras/kms

* refactor (kms): delete all the unneeded bits

All this stuff is not needed now that we're using
go-kms-wrapping/extras/kms

* refactor (db): fixup tests after kms refactor

* refactor (kms/tests): use testing.TB for test helpers

Our test helper packages were not possible to use
with benchmarks. This refactor switches to testing.TB
to allow both tests and benchmarks to use the helpers.

* tests (session benchmarks): refactor to use new Kms type

* refactor (schema): rm test which relies on deprecated Kms code

* refactor (kms): post rebase on main changes required

* chore (kms refactor): re-add unit test from main

* refactor (kms): uses migrations that don't include KMS schema changes
2022-05-13 13:53:22 -04:00
irenarindos 54c30d2cad bug(session): null fkey trigger also checks for session terminated state 2022-05-12 14:59:22 -04:00
Todd b58dada40d Move the server daemons into their own package (#2061)
* Move the server daemons into their own package

With this change the internal/server package now is dedicated to the server domain (workers and controllers) and not to the server daemons.
The hope is that this will make it easier to keep the daemons decoupled from the representation of the servers in the database.
2022-05-09 12:45:10 -07:00
irenarindos c6764e85d9 fix(session): Update session state transition trigger 2022-05-02 15:58:41 -04:00
Timothy Messier f00af362dc fix(session): Include where clause with order and limit
When applying the limit and order by, the where clause also needs to be
applied to yield the correct results. This can be seen with an example.

Assuming a data set where there are 100 sessions, with 10 sessions
belonging to a user with id `u_user1` and 90 sessions belonging to
`u_user2`. If the constructed query is like:

    select *
    from
        (select public_id from session limit 10) s,
        session_list ss
    where
        s.public_id = ss.public_id
        and s.user_id = 'u_user1'
    order by create_time asc;

The initial sub-query with the limit could return sessions only
belonging to `u_user2`, then it would apply the where clause and exclude
all of the rows.

Instead if the query is:

    with
    session_ids as (
        select public_id
        from sessions as s
        where
            s.user_id = 'u_user1'
        order by create_time asc
        limit 10
    )
    select *
    from session_list
    where
        session_list.public_id in (select * from session_ids)
    order by create_time asc;

The CTE applies the where clause ensuring that only sessions for
`u_user1` are used for the final select from the view.
Note that the CTE is use primarily for readability.

This also adds an index on `session (create_time)` to aide the query
planner in performing the order by.
2022-04-29 16:33:44 -04:00
Johan Brandhorst-Satzkorn 7f9b294a7e refact(all): Use testing.TB for test helpers
Our test helper packages were not possible to use
with benchmarks. This refactor switches to testing.TB
to allow both tests and benchmarks to use the helpers.
2022-04-29 16:33:44 -04:00
Timothy Messier 4599e64cab feat(event): Remove BOUNDARY_DEVELOPER_ENABLE_EVENTS env var 2022-04-20 13:32:32 -04:00
Jim addbfee593 chore: upgrade gofumpt to v0.3.1 (#2028) 2022-04-19 11:42:54 -04:00
Louis Ruch e6b7b7f60a feat(db): Suppress not found events during db lookups (#1979) 2022-04-08 18:04:12 -07:00
Johan Brandhorst-Satzkorn b1d6a1da9a Switch to buf for protobuf generation (#1944)
* Install buf with go install

This reduces the impact of using buf on our dependency
closure, which was significant. It also avoids potential
issues with building buf given our own internal dependency
constraints. This has caused issues (including crashes!)
in the past.

See https://github.com/cockroachdb/cockroach/issues/67216#issuecomment-874176186
for more discussion on using tool dependencies generally, and
https://github.com/bufbuild/buf/issues/52
for an example of where this sort of thing caused an issue for a user.

* Upgrade buf version

We should migrate to v1 sooner rather than later, as it has some
breaking changes from v0.

* Change buf lint/breaking commands

These commands were moved up one level for v1

* Migrate to buf.yaml to v1

* Remove unused imports

These were uncovered by using "buf lint".

* Use buf v1.3.1

* Depend on googleapis and grpc-gateway via BSR

* Switch to buf for proto generation

* Remove third party dependency

* Remove unused tools and scripts

* Move local protos up one level

* Temporarily don't fail on breaking changes

* Generate CI config

* Fix generation of controller.json

* Replace use of deprecated flag

* Remove unused import, regenerate files

* Remove temporary skipping over breaking check
2022-04-04 13:55:42 -04:00
Jeff Mitchell 7eb29261b2 Update to go-kms-wrapping version 2, and plugin-based KMS (#1901) 2022-03-16 17:04:07 -04:00
irenarindos 32132d6bc1 refactor(session connections): Refactor connection closure 2022-03-15 12:11:31 -04:00
Irena Rindos 5a2c0db586 fix(session connections): Create session connection repository 2022-03-15 12:11:31 -04:00
Todd e4d1e501cf fix(warehouse): Handle duplicate addresses when migrating into wh_network_address_dimension (#1860) 2022-02-15 15:49:41 -07:00
Todd 11ce54f8fb Add DW network address dimension tables (#1855) 2022-02-14 16:30:18 -07:00
Haotian fa00a06bef feat(host): Add host set information to static host (#1828)
* feat(host): Add host set information to static host

Updates LookupHost and ListHost API to include sets a static host is part of, as like with plugin hosts.

* test: updated TestList_Plugin and TestCrud to be more consistent under race conditions
2022-02-01 10:06:13 -08:00
Irena Rindos edae7c899f bug(session): Allow only valid state transitions 2022-01-12 16:01:49 -05:00
Jim 591ec476e7 refactor: Update internal/db and oplog to use go-dbw package for database operations. (#1785)
* chore: Add new dbw dependency

* refactor: Convert to use dbw.DB

* refactor (db): Change GetTicket(...) signature

* feature (db tests): Added TestDeleteWhere(...)
Adding this allows us to eliminate a leak of the gorm library

* refactor: Add ctx to ScanRows(...) signature
This is required to refactor all the deprecated errors pkg calls.
This refactor required adding a context to quite a few call chains
and in particular the job scheduler bits

* feature (db/tests): Add support for test opt: WithTestLogLevel(...)
This test only option allows callers to specify a log level for the
underlying db package (where supported/applicable) when calling
TestSetup(...)

* refactor (db): Simplify Db type
Only store the underlying *DB and create the *RW on demand when
needed.

* refactor (oplog): Port to using dbw pkg for db operations

* refactor (db): Remove gormDB(...) func

* tests (db): Add unit tests for TestDeleteWhere(...)

* chore: make fmt changes to formatting
2022-01-05 14:38:38 -05:00
Louis Ruch 5fe23ab14d feat(session): Store session credentials
Creates a new session_credential table and repository functions
to add and list credentials attached to a session. All credentials
are encrypted using the session scope id.
2021-12-17 11:26:52 -08:00
Michael Gaffney ba8188e6d7 feat(credential): Add mapping override to credential library on lookup
This change returns any mapping override set on a credential library
when the library is retrieved from the repository using the lookup
method.
2021-12-16 09:52:18 -05:00
Michael Gaffney 0e637f47b5 feat(credential): Use credential type when issuing credentials
This change uses the credential type attribute when issuing credentials
including mapping overrides.
2021-12-16 09:52:16 -05:00
Michael Gaffney c317901db4 feat(credential): add credential type to credential library on create
This change adds a credential type attribute to credential library and
allows it to be set when creating a credential library.

Specifying the type of credential returned from a library allows for new
features and additional error checking to be added over time. A
User Password credential type and an Unspecified credential type are
included with the expectation that additional credential types will be
added in the future. This change also adds the credential type attribute
to existing credential libraries and sets it to Unspecified.

Known Limitations:

- Credential type is only supported on the repository create method.
  Support for the other CRUD methods will be added in the future.
2021-12-16 09:43:32 -05:00
Jim baa1d88f1f feature: Add client ip to inbound request information (#1678)
* feature: Add client ip to inbound request information
* feature (workers): Add the user's client IP to a session's connection information (#1749)
2021-12-09 20:41:08 -05:00
Michael Gaffney 72d4d6d205 fix(sql): Remove warning from create_time trigger (#1751)
Raising a warning in the default trigger used for setting the value in
create_time columns creates unnecessary noise in the PostgreSQL log
files.

Closes #1744
2021-12-01 11:05:19 -05:00
Jim 983f7b019c refactor (db): Remove unused gorm formatter (#1738) 2021-11-30 18:20:44 -05:00
Jeff Mitchell ad8ce50f69 Fix Windows (#1733)
The embed package always uses Unix-style forward slashes. filepath.Join
utilization looking for files would cause the slashes to be the wrong
direction on Windows and result in the files not being found.
2021-11-18 15:01:01 -05:00
Michael Gaffney 55c6c5872f feat(sqltest): Make container port configurable 2021-11-17 08:04:56 -05:00
Jeff MitchellandJim 866bfceefc Add secrets_hmac to host catalog output (#1712)
Co-authored-by: Jim <jlambert@hashicorp.com>
2021-11-16 18:54:38 -05:00
Thor 7fa2cd2526 rename migration from 01-09 after bad merge (#1709)
* rename migration from 01-09 after bad merge
2021-11-15 16:42:01 -06:00
Thor ad31b6d6c3 repo: Return connection informat in Session response object (#1690)
Add connections to the Session object

* db: creat new session_list view

* repo: session use new session_list view
2021-11-15 15:26:08 -06:00
Jeff Mitchell 8110e0eab3 Increment version of host during updates (#1707)
Currently host does not use a version, which means it's not protected
against race conditions of either itself or its value objects. This adds
and plumbs through version to ensure that if a host or its value objects
change that they are not going to be affected by a race condition.
2021-11-15 12:22:39 -05:00
Jim 1b6c9a25c8 Add more opts when using Create + WithOnConflict (#1706) 2021-11-14 09:52:18 -05:00
Jim c6d79bc911 feature (db): Add WithOnConflict support to CreateItems(...) (#1699) 2021-11-12 18:10:09 -05:00
Todd Knight 5850001a07 Rename file to match others in directory and remove TODO. 2021-11-12 09:53:56 -07:00
Todd Knight f41bceda83 Merge from main and make gen 2021-11-12 08:57:10 -07:00
Todd Knight 884e1f2d18 Merge branch 'main' into plugin-hostcatalogs
# Conflicts:
#	go.mod
#	sdk/go.mod
2021-11-12 08:55:17 -07:00
Jim 1ac151100e docs (internal/db): Add docs Update/Delete operations using WithWhere (#1687) 2021-11-11 16:13:44 -05:00
Todd 29b4537d3b Reference PreferredEndpoints by top level struct name (#1688)
* Reference PreferredEndpoints by top level struct name instead of embedded proto GetPreferredEndpoint method.

Reorganize the sql files a little.

* Make PreferredEndpoints replayable. Test refactoring to remove variables that persist across test calls. DeleteItems instead of trying to use WithWhere.
2021-11-11 13:51:52 -07:00
Jeff Mitchell 30e9f944a2 Add sync interval to host sets (#1680)
This adds a parameter to host sets allowing the sync interval to be
user-specified on a per-host-set basis. A value of -1 disables syncing;
a value of null uses the current default.
2021-11-11 11:28:08 -05:00
Jim c047c4d89d refactor (db): Remove the internal/db test tables from the migrations (#1675) 2021-11-09 11:58:57 -05:00
Jeff Mitchell 48ac8b8f0b Merge branch 'main' into plugin-hostcatalogs 2021-11-07 18:47:28 -05:00
Timothy Messier 242e353f18 fix(schema): Prevent duplicate versions per edition (#1670)
This should help prevent issues like the one fixed in
f2ad4de554.

A version for a particular edition should only exist once, previously
which ever file was read last for a version would be included, which
would lead to some migrations not running.
2021-11-05 14:57:10 -04:00
Jim b6ad76e4d2 fix (db): Clean up tests a bit (#1669) 2021-11-05 08:51:12 -04:00
Jim f2ad4de554 fix (migrations): Adjust migration numbers to remove dups (#1667) 2021-11-04 15:34:56 -04:00
Jim f63ca1313c feature (kms/audit): Add base audit key type (#1651)
* feature (db): Add the ability to enable debug output for a connection

* refactor: Use scope.Global instead of "global"

* feature (db/migrations): Add kms audit keys

Add schema changes to support a new DEK for audit crypto operations.

* feature (kms/audit): Add audit key type

All the bits required to define a new DEK concrete type for
operating on audit entries.

* refactor (db): Remove deprecated migration unit tests.

We added a new audit DEK to kms.CreateKeysTx(...).  Unfortunately,
unit tests like TestMigrations_UserDimension,
Test_PrimaryAuthMethodChanges and TestMigrations_CredentialDimension
have an indirect dependency on a specific historical set of DEKs via
an indirect call kms.CreateKeysTx.  This of course fails because the
new DEK's tables is not a part of pinned migration versions that these
tests depend on.
2021-11-04 09:34:01 -04:00
Jeff Mitchell 80d41b9044 Merge branch 'main' into plugin-hostcatalogs 2021-11-04 09:02:39 -04:00
Jim 4df3b40def feature (db): Add support for multi-column PKs (#1658) 2021-11-03 14:04:47 -04:00
Todd 1ff2968409 Warehouse contains "Unsupported" host_address for plugin based hosts (#1656)
* Populating the warehouse with the host_address from a plugin basd host is unsupported.

* Adding test for plugin hosts in the datawarehouse.
2021-11-03 07:24:53 -07:00
Timothy Messier 1a24a0fc2f feat(sqltest): Make container name configurable 2021-11-03 07:44:51 -04:00
Todd 1312628a1e Add plugin set sync and orphaned host cleanup jobs (#1647) 2021-11-01 09:07:56 -07:00