7f0dfa2344
* carve out scope into a separate branch for a PR * refactor getOpts to be private * narrowed to just the scope proto * wrap in a transaction * remove user from scopes PR * make scope type immutable * refactored with more referential integrity (thanks mike) * refactor based on pair programming with Mike * better VetForWrite that supports db.Options but that requires GetOpts and Options to be exported. * stop printing out all the db logs * refactor withScope to be private * refactor the VetForWrite a bit based on Todd's PR comment * fix typo Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com> * fix ctx propagation and Scope returns * fix ctx propagation and use a const for std retry count * change Std to Crud * fix ctx propagation * make sure update with field masks doesn't update a field not in the mask and force a read after update. * add description * remove comment about future * clean error strings * clean up enum string rep * stdMeta * fix return value to be just nil * fix returns and errors * update test to match new error string * change len of domain to 24 * provide common function for new public ids * use common db.NewPublicId() and provide the correct prefix * move list out of CrudActions and provide a CrudlActions shortcut as well * some comments about the purpose of VetForWrite() * move to random 10 * Adding project API interface. * Creation of Project handler and Tests * Move to using the iam objects. * Setting up testify framework for mocking and assertions. * Fix iam options typo references. * Adding tests for CreateProject and UpdateProject. This also adds some additional argument validation. * Merging in PR 40 * Renamed fakeRepo to mockRepo * Changing around TODOs to capture current plans with the code. * Remove the repo interface and use a real DB for tests. * Fixed a few tests. * Adding checks for properly formatted identifiers. * Updating code to account for recent merges. Relaxing some restrictions on what an id can look like. * Added small comment explaining why RegisterGrpcGatewayer interface exists. * Move project service up a call level. Long term I think it would make sense to define these services even higher up the call stack, potentially at the initialization of the controller server, near where the DB is initialized. * Making these fields align with those in the resources under hosts/... * Hard code the service initialization in the handler registration code. * Regenerate project related code proto code. * Removing type field for project. * Adds basic project handling in CLI/SDK (#47) * Adds SDK for creating and reading projects * Reorganizes a bit * Adds CLI for creating and reading projects * Fixes some some generation bugs * Adds redirect support for default org ID * Fix protobuf import references being moved. * Update client test to look for "orgs" and "projects" instead of "org" and "project" * Update expected segments test * Fix setAddr handling when default organization is in use * Set client in target after create or read * Adding functionality and tests for update masks. * Additional tests for update. Disable update non existant project test. * Make not providing an update mask be an invalid argument. * Make empty update mask paths and no mutable fields in the path an InvalidArgument error. * Fix get Project test. * Adding back in the Update a non existant project test case. * Adding seperate path structured tests. * Attach an issue id to field mask validation TODO. * Generate Project Update SDK code (#58) * Created the UpdateFunction method. * Regenerate protobuf derived code since pull #50 updated dependencies. * Special case IDs to not be passed for updates. * Updated the tools file since imports reordered it. * Add spaces between each tools import/comment grouping. Co-authored-by: Jim Lambert <jimlambrt@Jims-MBP-3.home> Co-authored-by: Jim <jlambert@hashicorp.com> Co-authored-by: Michael Gaffney <mgaffney@users.noreply.github.com> Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>
oplog
oplog is a package for writing operation log (oplog) entries for the purpose of replication and verification of the data stored in the Watchtower RDBMS.
Usage
// you must init the ticket in its own transaction. You only need
// to init a ticket once in the database. It doesn't need to happen for
// every connection. Once it's persistent in the database, you can simply Get it.
initTx := db.Begin()
ticketer := &GormTicketer{Tx: initTx}
err = ticketer.InitTicket("users")
// if there's no error, then commit the initialized ticket
initTx.Commit()
userCreate := oplog_test.TestUser{
TestUser: oplog_test.TestUser{
Name: userName,
},
}
tx := db.Begin()
// write the user to the database
tx.Create(&userCreate)
ticketer = &GormTicketer{Tx: db}
// get a ticket for writing users to the oplog
ticket, err := ticketer.GetTicket("users")
// create an entry for the oplog with the entry's metadata (likely the entry's Scope)
newLogEntry := NewEntry(
"test-users",
[]Metadata{
Metadata{
Key: "deployment",
Value: "amex",
},
Metadata{
Key: "project",
Value: "central-info-systems",
},
},
cipherer, // wrapping.Wrapper
ticketer,
)
// write an entry with N messages (variadic parameter) in the order they were sent to the database
_, err = newLogEntry.WriteEntryWith(
context.Background(),
&GormWriter{tx},
ticket,
&Message{Message: &userCreate, TypeName: "user", OpType: OpType_CREATE_OP},
)
// if there's an error writing the oplog then roll EVERYTHING back
if err != nil {
tx.Rollback()
}
// no err means you can commit all the things.
tx.Commit()
TBD/TODO
We need to discuss and decide how Watchtower is going to handle the following oplog things:
- SQL migrations: you'll find the package's SQL migrations under: ./migrations/postgres We need to decide how Watchtower will manage migrations across the system and we will likely need to reference this package's migrations somehow.
oplog entry
Example Oplog Entry for the Target Aggregate
┌────────────────────────────────────────────────┐
│ FIFO with []byte buffer │
│ │
│┌─Msg 4────┐┌─Msg 3────┐┌─Msg 2────┐ │
││ ││ ││ │ │
││ Tags ││ Host ││ HostSet │ │
││┌────────┐││┌────────┐││┌────────┐│ │
│││ ││││ ││││ ││ │
│││ ││││ ││││ ││ │
│││ Tag ││││ Host ││││HostSet ││ │
│││protobuf││││protobuf││││protobuf││ │
│││ ││││ ││││ ││ │
│││ ││││ ││││ ││ │
││└────────┘││└────────┘││└────────┘│ │
││┌────────┐││┌────────┐││┌────────┐│ │
│││ ││││ ││││ ││ │
│││typeName││││typeName││││typeName││ │
│││ Tag ││││ Host ││││HostSet ││ │
││└────────┘││└────────┘││└────────┘│ │
││┌────────┐││┌────────┐││┌────────┐│ │
│││ ││││ ││││ ││ │
│││ OpType ││││ OpType ││││ OpType ││ │
│││ Create ││││ Create ││││ Create ││ │
││└────────┘││└────────┘││└────────┘│ │
│└──────────┘└──────────┘└──────────┘ │
└────────────────────────────────────────────────┘
oplog tables
oplog tables:
as the diagram shows, we can split the
oplog_entries into multiple tables if
needed for performance.
┌────────────────┐
│┌───────────────┴┐
││┌───────────────┴┐ ┌────────────────┐
│││ oplog_entries │ │ oplog_ticket │
││├────────────────┤ ├────────────────┤
│││id │╲ │id │
│││aggregate_name │──┼───────┼ │aggregate_name │
└┤│data │╱ │version │
└┤ │ │ │
└────────────────┘ └────────────────┘
┼
│
│
┼
╱│╲
┌────────────────┐
│ oplog_metadata │
├────────────────┤
│id │
│entry_id │
│key │
│value │
└────────────────┘
oplog optimistic locking using tickets
Alice's Database
transaction Bob's transaction
│ │ │
│─────────BEGIN──────────────────▶│ │
│ │◀───────────────BEGIN─────────────────┤
│ ┌───────────┴───────────┐ │
│ │ ticket version:1 │ │
│ └───────────┬───────────┘ │
│ Select oplog-ticket for │ │
├──────────────"Target"──────────▶│ Select oplog-ticket for │
│ │◀──────────────"Target"───────────────┤
│ │ │
│ Write to Tables in │ │
├───────Target Aggregate─────────▶│ │
│ │ Write to Tables in │
│ │◀─────────Target Aggregate────────────┤
│ │ │
│ │ │
│ Update Ticket │ │
├────Version = 2 where───────────▶│ │
│ Version = 1 ┌─────────┴─────────┐ │
│ │ ticket version: 2 │ │
│ └─────────┬─────────┘ │
│ │ Update Ticket │
│ │◀───────────Version = 2 where─────────┤
│ │ Version = 1 ┌────┴──────────┐
│ │ │ update failed │
├────────────Commit──────────────▶│ └────┬──────────┘
│ │ │
│ │ │
│ │ │
│ │◀────────────Rollback─────────────────┤
│ │ │
│ │ │
```