When creating the options for the preferencer, this used WithIpAddrs for both the host addresses and also DNS names. This causes errors when a host contains DNS names that are not IP addresses (ie: whenever dns names are present).
This change will:
- Replace WithIpAddrs with WithDNSNames option for DNS Names when constructing options for the preferencer.
- Add a test which includes a DNS name to prevent regressions
Fixes#1848
* feat(host): Add host set information to static host
Updates LookupHost and ListHost API to include sets a static host is part of, as like with plugin hosts.
* test: updated TestList_Plugin and TestCrud to be more consistent under race conditions
* chore: Add new dbw dependency
* refactor: Convert to use dbw.DB
* refactor (db): Change GetTicket(...) signature
* feature (db tests): Added TestDeleteWhere(...)
Adding this allows us to eliminate a leak of the gorm library
* refactor: Add ctx to ScanRows(...) signature
This is required to refactor all the deprecated errors pkg calls.
This refactor required adding a context to quite a few call chains
and in particular the job scheduler bits
* feature (db/tests): Add support for test opt: WithTestLogLevel(...)
This test only option allows callers to specify a log level for the
underlying db package (where supported/applicable) when calling
TestSetup(...)
* refactor (db): Simplify Db type
Only store the underlying *DB and create the *RW on demand when
needed.
* refactor (oplog): Port to using dbw pkg for db operations
* refactor (db): Remove gormDB(...) func
* tests (db): Add unit tests for TestDeleteWhere(...)
* chore: make fmt changes to formatting
* internal/host/plugin: remove custom queries for secrets
This removes the custom queries for host catalog secrets, now that #1658
has been merged.
It also does a lot of cleanup surrounding the related oplog entries,
using db.WithOplog natively.
* Simplify conflict actions by replacing SetColumnValues with SetColumns
* Remove unused oplog method
* internal/host/plugin: ensure UpdateCatalog triggers updates for sets
This ensures that attribute updates to a host catalog triggers an update
to any of its downstream host sets. This ensures that updates to
elements that affect a host set, such as the region a host catalog is
configured for, take effect immediately instead of requiring a wait of
long as 10m after an update.
* Add WithNextRunIn to test sync job
This is to ensure it doesn't run right away when added. This ensures
that we can accurately test *when* the job is set to run immediately.
* Update internal/host/plugin/repository_host_catalog.go
Co-authored-by: Todd <T.Alan.Knight@gmail.com>
* Change TestRepository_UpdateCatalog_SyncSets to check next run time, not wait for run
* Remove explicit version increment and instead rely on implicit increment
* Fix lanuage for rows updated check when flagging host sets for sync
* remove errant non-zero check when looping over the thing we're checking
* remove scheduler start line from TestRepository_UpdateCatalog_SyncSets
Co-authored-by: Todd <T.Alan.Knight@gmail.com>
Internally attributes are not allowed to be the null value. However, as
currently implemented, this then breaks our normal model of sending an
explicit `null` in the JSON to reset/clear a field. This is a problem
because the CLI doesn't have a way to say "set this but to empty"
without a lot of hoops, and causes some issues for API users and TF as
well.
This changes PatchBytes and PatchStruct to return an empty value (or
marshaled empty value in the case of PatchBytes) if the source is null
or empty. In the host catalog update function we then always set this
value if it's in the db mask (removing the nil check). Together this
means a null value ends up coming out as an empty value (and thus valid
for the DB) from PatchStruct/PatchBytes, and our API translation
function already returns nothing for attributes if the length of the
fields are zero.
Currently host does not use a version, which means it's not protected
against race conditions of either itself or its value objects. This adds
and plumbs through version to ensure that if a host or its value objects
change that they are not going to be affected by a race condition.
* Update sync interval seconds field.
* If the sync interval was updated, schedule the next sync job wakeup immediately.
* Calculate the next sync time for the scheduler should wake up if we only updated the sync interval.
* Reference PreferredEndpoints by top level struct name instead of embedded proto GetPreferredEndpoint method.
Reorganize the sql files a little.
* Make PreferredEndpoints replayable. Test refactoring to remove variables that persist across test calls. DeleteItems instead of trying to use WithWhere.
This adds a parameter to host sets allowing the sync interval to be
user-specified on a per-host-set basis. A value of -1 disables syncing;
a value of null uses the current default.
* internal/host/plugin: add UpdateSet
This adds the UpdateSet repository function.
UpdateSet runs the OnUpdateSet plugin call within the transaction as per
the changes in #1683.
The plugin call returns the updated set, the hosts in the set, the rows
updated, and any error.
* Ensure that attributes can be cleared by supplying a full null
* move version increment to the end and track perferred endpoint update separately
* move plugin call to end of transaction
* only clone returned set once, and move host fetch call out of transaction
* Request a sync when attributes are updated
* Add test to assert that old preferred endpoints are being removed correctly
* Only call the Plugin at create/update time after the db has been partially updated. This allows more constraints to be checked prior to calling the plugin.
* Create catalog service handler for updating plugin. Add tests for handler and run make fmt. Silently drop attributes where the value is nil when creating the catalog or set.
* internal/host/plugin: Add UpdateCatalog
This adds support for updating plugin-based host catalogs.
The workflow is:
* The existing catalog is fetched.
* Any attribute changes are patched against it, along with other changes
such as name and description.
* Both the existing catalog, and the state of the catalog as it will
exist post-update, along with any persisted secrets, are sent to the
plugin via an OnUpdateCatalog request.
Upon success, the following is done:
* The record is updated if non-secret updates need to be made.
* Secrets are updated if they still exist.
* If an empty secret set was sent back, secrets are instead deleted.
* If the secrets field was instead null, secret update is skipped
altogether.
There has also been a lot of helper code added to the tests to help make
the update tests more readable and reduce boilerplate. While the create
tests have not been updated to use these new helpers, it might be a good
idea to do so for consistency.
* move secret/persisted state closer to plugin call
* Update internal/host/plugin/repository_host_catalog.go
Co-authored-by: Todd <T.Alan.Knight@gmail.com>
* Update internal/host/plugin/repository_host_catalog_test.go
Co-authored-by: Todd <T.Alan.Knight@gmail.com>
* move all test helpers for UpdateCatalog inline
* WIP - transparent attributes struct/wire conversion
* Revert "WIP - transparent attributes struct/wire conversion"
This reverts commit 31af1b1b7e535119aa23bd0aa936da1a80eb5aa2.
* deal with multiple attribute masks
* Note TODO for upsertQuery for HostCatalogSecret
* add version check guard
* Update internal/host/plugin/host_catalog_secret.go
Co-authored-by: Todd <T.Alan.Knight@gmail.com>
* remove redundant JIRA reference
* don't separate number of updated catalogs and secrets
* increment version when secrets is the only thing being updated
* add another test for updating both catalog and secrets at once
* correct comment for mock plugin manager section
* add first-part path matching for secrets
* update plugin lookup error to return errors.Internal
* only add name/description to plugin request if they are non-empty
* update duplicate tests to use two project scopes
* remove unused test checks
* return plugin record along with host catalog in update
* move away from using passed in catalog after patching newCatalog
* use t.Cleanup() instead of defer
Co-authored-by: Todd <T.Alan.Knight@gmail.com>
* Add ability to read and list hosts. Tests to follow.
* Add tests for reading and listing as well as validation checks to not modify hosts.
* Add IpAddress and DnsNames to the api resource. Populate the plugin info.
* Adding go sdk tests for reading and listing hosts and ensuring update and deleting is disallowed.
* Add DB changes for host, set members, and host addresses for plugins.
* Add priority, address review feedback, update tests, naming, protos
Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>
Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>
* refact: Changes required for gorm v2
* refact: Add db.DB wrapper and refact all test fixtures to use it
Stop leaking the gorm abstraction within our unit tests.
* refact: Migrate direct calls to pq to pgx driver
The refactor will remove the remaining direct dependencies on
the pq driver which has been deprecated in favor of the pgx
driver.
It includes the addition of common.SqlOpen which is a wrapper around
sql.Open, which is needed to deal with the driver name changing from
"postgres" to "pgx" but in our code base the dialect and driver are
synonyms.
This adds a few command sketches but that's not the main point of the PR
and filling those out will come in a different PR. This adds a new flag
type that can combine inputs to a single target slice, retaining the
ability to distinguish where it came from but also keep ordering across
different flag names. It also implements a function that does the
heuristic parsing with fallback.
Added a new Endpoint struct at the top level host package level to capture host, set, and address chosen.
Temporarily add a host_plugin_host table to capture the hosts returned by the plugins. This ensures the datawarehouse is able to query the table to decide what host was chosen. The view used to update the datawarehouse dimension table provides rows with host set ids and hosts which are not in that host set. This is ok since the function which updates the dimension table provides the host set and host id chosen so that relationship doesn't need to be captured in the view itself.
Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>
Louis Ruch
Todd
Justen Walker
Haotian
Jim
Chris Marchesi
Jeff Mitchell
