Commit Graph
60 Commits
Author SHA1 Message Date
Todd Knight 95d29c3e95 Allow the TestController to be referenced externally (#65)
* Allow the TestController to be used outside of our go module.

* Adding package comment explaining this testing package is for internal use only.
2020-05-19 14:54:35 -07:00
7f0dfa2344 Fix Update Project Logic (#54)
* carve out scope into a separate branch for a PR

* refactor getOpts to be private

* narrowed to just the scope proto

* wrap in a transaction

* remove user from scopes PR

* make scope type immutable

* refactored with more referential integrity (thanks mike)

* refactor based on pair programming with Mike

* better VetForWrite that supports db.Options but that requires GetOpts and Options to be exported.

* stop printing out all the db logs

* refactor withScope to be private

* refactor the VetForWrite a bit based on Todd's PR comment

* fix typo

Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>

* fix ctx propagation and Scope returns

* fix ctx propagation and use a const for std retry count

* change Std to Crud

* fix ctx propagation

* make sure update with field masks doesn't update a field not in the mask and force a read after update.

* add description

* remove comment about future

* clean error strings

* clean up enum string rep

* stdMeta

* fix return value to be just nil

* fix returns and errors

* update test to match new error string

* change len of domain to 24

* provide common function for new public ids

* use common db.NewPublicId() and provide the correct prefix

* move list out of CrudActions and provide a CrudlActions shortcut as well

* some comments about the purpose of VetForWrite()

* move to random 10

* Adding project API interface.

* Creation of Project handler and Tests

* Move to using the iam objects.

* Setting up testify framework for mocking and assertions.

* Fix iam options typo references.

* Adding tests for CreateProject and UpdateProject.

This also adds some additional argument validation.

* Merging in PR 40

* Renamed fakeRepo to mockRepo

* Changing around TODOs to capture current plans with the code.

* Remove the repo interface and use a real DB for tests.

* Fixed a few tests.

* Adding checks for properly formatted identifiers.

* Updating code to account for recent merges.  Relaxing some restrictions on what an id can look like.

* Added small comment explaining why RegisterGrpcGatewayer interface exists.

* Move project service up a call level.

Long term I think it would make sense to define these services even higher up the call stack, potentially at the initialization of the controller server, near where the DB is initialized.

* Making these fields align with those in the resources under hosts/...

* Hard code the service initialization in the handler registration code.

* Regenerate project related code proto code.

* Removing type field for project.

* Adds basic project handling in CLI/SDK (#47)

*    Adds SDK for creating and reading projects
*    Reorganizes a bit
*    Adds CLI for creating and reading projects
*    Fixes some some generation bugs
*    Adds redirect support for default org ID

* Fix protobuf import references being moved.

* Update client test to look for "orgs" and "projects" instead of "org" and "project"

* Update expected segments test

* Fix setAddr handling when default organization is in use

* Set client in target after create or read

* Adding functionality and tests for update masks.

* Additional tests for update.  Disable update non existant project test.

* Make not providing an update mask be an invalid argument.

* Make empty update mask paths and no mutable fields in the path an InvalidArgument error.

* Fix get Project test.

* Adding back in the Update a non existant project test case.

* Adding seperate path structured tests.

* Attach an issue id to field mask validation TODO.

* Generate Project Update SDK code (#58)

* Created the UpdateFunction method.

* Regenerate protobuf derived code since pull #50 updated dependencies.

* Special case IDs to not be passed for updates.

* Updated the tools file since imports reordered it.

* Add spaces between each tools import/comment grouping.

Co-authored-by: Jim Lambert <jimlambrt@Jims-MBP-3.home>
Co-authored-by: Jim <jlambert@hashicorp.com>
Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>
Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>
2020-05-19 12:10:39 -07:00
Todd Knight ff705bf3b2 Revert "Allow the TestController to be used outside of our module."
This reverts commit e1010ba0
2020-05-18 16:26:25 -07:00
Todd Knight e1010ba055 Allow the TestController to be used outside of our module. 2020-05-18 16:24:49 -07:00
Jim 980a55a4e3 Added DeleteScope() with tests and TestVerifyOplog() for easier testing (#49) 2020-05-18 17:51:09 -04:00
Jim d1acc1437b fix multierror 2020-05-18 14:38:32 -04:00
Jim ee46266893 ErrRecordNotFound from LookupWhere() when no rows found (#59) 2020-05-18 12:57:45 -04:00
Michael Gaffney fad302a36d Run make migrations (#61) 2020-05-16 19:53:13 -04:00
Michael Gaffney 3d614dd732 Run migrations in a transaction (#60) 2020-05-16 13:26:57 -04:00
Jeff Mitchell f8b456f3a7 Add TestController (#50)
This can be used in tests to get access to a controller that behaves
just like a dev controller but doesn't require the normal Command bits
(UI, etc).

This can be enhanced over time as needed; for now it's pretty simple but
it's good enough to be used for API tests. A simple project create/read
API test is added as well, which serves to show the programmatic model
(as well as test those functions at the API level).
2020-05-15 10:19:55 -04:00
2aca5fff03 fix transaction rollbacks in DoTx (#55)
* fix transaction rollbacks in DoTx

* better assertion

* Update internal/db/read_writer_test.go

Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>

Co-authored-by: Jim Lambert <jimlambrt@Jims-MBP-3.home>
Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>
2020-05-15 08:38:36 -04:00
Jim 4a88d10ecc fix a minor defect in fakeFile Readdir() and unit tests (#53) 2020-05-14 13:30:45 -04:00
Michael Gaffney 477a47897e Add wt_timestamp type to database (#52)
* Add wt_timestamp type to database

* Swap the order of domains and oplog migrations

All migrations should use the new 'wt_timestamp' domain type for
timestamps.

* Remove dead code

* Add down migration for wt_timestamp
2020-05-13 10:40:57 -04:00
Jeff Mitchell f287a268e8 Update project service against changed repo 2020-05-12 21:02:51 -04:00
c0c6a9b0b8 Handler project (#44)
* Renamed fakeRepo to mockRepo

* Changing around TODOs to capture current plans with the code.

* Remove the repo interface and use a real DB for tests.

* Fixed a few tests.

* Adding checks for properly formatted identifiers.

* Updating code to account for recent merges.  Relaxing some restrictions on what an id can look like.

* Added small comment explaining why RegisterGrpcGatewayer interface exists.

* Move project service up a call level.

Long term I think it would make sense to define these services even higher up the call stack, potentially at the initialization of the controller server, near where the DB is initialized.

* Making these fields align with those in the resources under hosts/...

* Hard code the service initialization in the handler registration code.

* Regenerate project related code proto code.

* Removing type field for project.

* Adds basic project handling in CLI/SDK (#47)

*    Adds SDK for creating and reading projects
*    Reorganizes a bit
*    Adds CLI for creating and reading projects
*    Fixes some some generation bugs
*    Adds redirect support for default org ID

* Fix protobuf import references being moved.

* Update client test to look for "orgs" and "projects" instead of "org" and "project"

* Update expected segments test

* Fix setAddr handling when default organization is in use

* Set client in target after create or read

Co-authored-by: Jim Lambert <jimlambrt@Jims-MBP-3.home>
Co-authored-by: Jim <jlambert@hashicorp.com>
Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com>
Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>
2020-05-12 17:35:48 -07:00
Michael Gaffney 8d7f24268c Return url of test database (#48)
It is helpful to connect to a database when tests have failed.
2020-05-12 15:49:37 -04:00
Jeff Mitchell 7d8661be31 Remove migrations tag, not needed anymore 2020-05-12 12:31:40 -04:00
Jim c04c73453f refactor Update/Delete to return the number of rows affected. (#46) 2020-05-12 11:57:28 -04:00
Jim 61ca88dac9 refactor to use db.New() and clean up error msgs (#45) 2020-05-12 06:51:02 -04:00
Jeff Mitchell ad8d4e9c80 Rename Db's Tx to underlying 2020-05-11 21:05:11 -04:00
Jeff Mitchell 062ce6079f Create dev org in database (#41)
These changes create a dev org in the database on startup.

They also plumb through a dev-org-id flag that allows specifying a chosen org ID in dev mode.

The logic will allow us to let people specify a default org ID in the config file too, although currently that's not enabled.

Several other improvements were made as well:

*    Consolidation of some code from across the server commands
*    Plumbing shutdownch through so we can use it for canceling contexts during startup
2020-05-11 17:03:23 -04:00
Jeff Mitchell 670abaed88 Rename GormReadWriter -> Db 2020-05-11 16:54:05 -04:00
Jim ee4a0d3f45 refactor lookups so we can easily check for not found and do the "right" thing in the repos (#42) 2020-05-11 16:21:09 -04:00
Michael Gaffney 98cd37da97 Rename test files to match the name of the file they test (#43)
Rename test files to follow the convention of 'foo.go' and
'foo_test.go'.
2020-05-11 16:01:49 -04:00
Jim cd74d7f0ef refactor ClonableResource to just be a clonable interface (#40) 2020-05-11 14:11:21 -04:00
Jeff Mitchell 4d1b916348 Remove AllLoggers (#39)
This was copied from Vault code but shouldn't be necessary any more as a
change in hclog a while back meant that derived loggers share the log
level of the parent.
2020-05-11 13:30:38 -04:00
Jeff Mitchell 5a721f0ce3 Update deps (#38)
* Update deps

* Update against latest Vault dep
2020-05-11 13:27:54 -04:00
Jeff Mitchell 7f2aa5ca8f Add httpfs DB migrations and generate the input (#35) 2020-05-11 12:42:10 -04:00
Jim bd5ddea6e4 refactor clonable to not rely on the resource interface (#36) 2020-05-11 10:31:15 -04:00
Jim 0676ea51f1 refactor oplog to use db/internal migrations, remove InitTicket() for more reliability and simplify test step up and tear down (#37) 2020-05-11 10:30:44 -04:00
Jim c43e05be2e provide db.Open() and db.Migrate() as part of icu-64 (#34) 2020-05-08 17:21:22 -04:00
Jim 8a5197e39e implement schema, store and repository for IAM scopes of organization and project 2020-05-08 11:36:28 -04:00
Michael Gaffney 8824a6f7f9 database: add wt_public_id domain type (#30) 2020-05-06 08:29:13 -04:00
Todd Knight 05330db741 Add projectless paths for host-catalogs. (#31)
* Add projectless paths for host-catalogs.
2020-05-05 16:33:03 -07:00
Jim 725a777baf refactor friendly name to just be name (#29) 2020-05-05 16:35:24 -04:00
Jeff Mitchell 11f640a0f8 Fix compilation after Id changing 2020-05-05 14:13:08 -04:00
Jeff Mitchell f36f25a23b Add json_name fields to proto for complex names (#27)
This was generating fine before but it makes it explicit in the proto
and it avoids weirdness where the protobuf tag will say a name is
something like requestId where the json tag says it's request_id. This
way they line up.

One thing not honoring this properly is TraceId. I don't know why the
JSON tag being generated doesn't match even when explicitly set here.
For now I'm just manually fixing it.
2020-05-05 14:08:13 -04:00
Jim 27b4d7a034 basic internal db interfaces and Gorm implementation
* db reader and writer interfaces which should allow us to encapsulate the vast majority of database code in this package.

* GormReadWriter implementation of the Reader and Writer interfaces
2020-05-05 13:29:47 -04:00
Jeff Mitchell 1519a17f17 Rename catalog flag to catalog-id in hosts create command 2020-05-05 11:21:37 -04:00
Jeff Mitchell 1f3c52f495 Name and addr updates (#26)
* Rename FriendlyName to Name

* Standardize on connection parameters for the server being addr

This syncs up WATCHTOWER_ADDR and -addr (formerly -address) and frees up
-address to be used in resources (which it is).

* Add description fields
2020-05-05 11:19:42 -04:00
Jeff Mitchell e3c7ca3070 Add a hosts create command (#25) 2020-05-04 23:04:55 -04:00
Jeff Mitchell 4afb4f6be5 Minor proto updates (#22) 2020-05-04 22:57:22 -04:00
Jeff Mitchell 6970b14143 Update type value to wrapper 2020-05-04 14:58:18 -04:00
Jeff Mitchell 6254e18698 Fix compilation of command 2020-05-04 12:41:36 -04:00
Jeff Mitchell 2926cc1000 Add initial API generation bits (#20)
* Add initial API generation bits

* Revert accidental changes to oplog
2020-05-04 10:58:20 -04:00
Jeff Mitchell 7f03246c6f Add basic API client (#19) 2020-05-04 10:58:03 -04:00
Jim 2f0dd58f11 set default value in gorm tag, so database timestamp defaults will get set 2020-05-03 16:10:14 -04:00
Jim 73363d4c71 adapt to use structwrapping from go-kms-wrapping 2020-05-03 16:09:22 -04:00
Jeff Mitchell 13056e1f51 Migrate host resources to hosts package (#15)
Migrate host resources to hosts package and some other minor changes
2020-05-01 18:05:54 -04:00
Todd KnightandJeff Mitchell c7126ae69b Static Catalog Api definition (#12)
* Setup buf to see and process protos correctly.

This commit adds a proposed directory structure for protos.  Included is a rough draft of the Ec2Catalog protos.

* Add host and host_catalog service apis.

This also has for the first time the move away from holding individual ids for resources and instead using the full uri path as the api facing identifier.  There are some issues with the swagger generation stripping out the format of the path matching so currently only PATCH uses the approach I would like to move to.  In follow up commits I'll try to either get swagger to display the correct expected paths or I'll resolve if that is even needed by our clients.

* Move all protos and generated files to under the internal directory.

This also required updating the buf config to explicitly name the exluded linted proto files under oplog since previously /internal/ was listed as a root in buf.  Future commit will move oplog proto files to be with the others.

* Moved oplog related protos to under the proto directory and regenerated.

* Update the oplog protos to be compliant with buf lint.

* Clean up comments in tools.

* Merged in recent changes to oplog protos.

* Add resource result messages to contain the read only fields and separate them from the writable fields.

* Merge the ec2 catalog resources in with the more generic host_catalog resource.

* Cleaning the generated swagger.

This included adding summaries for each proto rpc method, Making the resources be the full fqn, removing the default error, and a few other tweaks.

* Make project plural in the paths.

* Adding the ability to add and remove hosts from a host set.

* Add the ignore result limit for host set request.

* Make size and host lists read only.

* Add documentation about what host references are allowed to be added.

* Remove ignore result limit when listing items.  These can be added in later.

* Add handlers and tests for unimplemented services.

* Add bootstrap target and update how tools are built (#13)

* Update tool installation method, add bootstrap target to makefile

* Changing org to orgs in all paths and org param to org_id.

* Adding tests for unimplemented service implementations.

Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>
2020-05-01 12:38:01 -07:00