2020-04-08 14:55:40 -04:00
# oplog
2020-08-10 17:15:02 -04:00
oplog is a package for writing operation log (oplog) entries for the purpose of replication and verification of the data stored in the Boundary RDBMS.
2020-04-08 14:55:40 -04:00
- [oplog ](#oplog )
- [Usage ](#usage )
- [TBD/TODO ](#tbdtodo )
- [oplog entry ](#oplog-entry )
- [oplog tables ](#oplog-tables )
- [oplog optimistic locking using tickets ](#oplog-optimistic-locking-using-tickets )
## Usage
```go
// you must init the ticket in its own transaction. You only need
// to init a ticket once in the database. It doesn't need to happen for
// every connection. Once it's persistent in the database, you can simply Get it.
initTx := db . Begin ()
ticketer := & GormTicketer { Tx : initTx }
err = ticketer . InitTicket ( "users" )
// if there's no error, then commit the initialized ticket
initTx . Commit ()
userCreate := oplog_test . TestUser {
TestUser : oplog_test . TestUser {
2020-08-11 17:25:00 -04:00
Name : loginName ,
2020-04-08 14:55:40 -04:00
},
}
tx := db . Begin ()
// write the user to the database
tx . Create ( & userCreate )
ticketer = & GormTicketer { Tx : db }
// get a ticket for writing users to the oplog
ticket , err := ticketer . GetTicket ( "users" )
// create an entry for the oplog with the entry's metadata (likely the entry's Scope)
newLogEntry := NewEntry (
"test-users" ,
[] Metadata {
Metadata {
Key : "deployment" ,
Value : "amex" ,
},
Metadata {
Key : "project" ,
Value : "central-info-systems" ,
},
},
cipherer , // wrapping.Wrapper
ticketer ,
)
// write an entry with N messages (variadic parameter) in the order they were sent to the database
_ , err = newLogEntry . WriteEntryWith (
context . Background (),
& GormWriter { tx },
ticket ,
& Message { Message : & userCreate , TypeName : "user" , OpType : OpType_CREATE_OP },
)
// if there's an error writing the oplog then roll EVERYTHING back
if err != nil {
tx . Rollback ()
}
// no err means you can commit all the things.
tx . Commit ()
```
## TBD/TODO
2020-08-10 17:15:02 -04:00
We need to discuss and decide how Boundary is going to handle the following oplog things:
2020-04-08 14:55:40 -04:00
2021-01-25 14:43:50 -07:00
* SQL migrations: you'll find the package's SQL migrations under: ./db/schema We need to decide how Boundary will manage migrations across the system and we will likely need to reference this package's migrations somehow.
2020-04-08 14:55:40 -04:00
## oplog entry
```
Example Oplog Entry for the Target Aggregate
┌────────────────────────────────────────────────┐
│ FIFO with []byte buffer │
│ │
│┌─Msg 4────┐┌─Msg 3────┐┌─Msg 2────┐ │
││ ││ ││ │ │
││ Tags ││ Host ││ HostSet │ │
││┌────────┐││┌────────┐││┌────────┐│ │
│││ ││││ ││││ ││ │
│││ ││││ ││││ ││ │
│││ Tag ││││ Host ││││HostSet ││ │
│││protobuf││││protobuf││││protobuf││ │
│││ ││││ ││││ ││ │
│││ ││││ ││││ ││ │
││└────────┘││└────────┘││└────────┘│ │
││┌────────┐││┌────────┐││┌────────┐│ │
│││ ││││ ││││ ││ │
│││typeName││││typeName││││typeName││ │
│││ Tag ││││ Host ││││HostSet ││ │
││└────────┘││└────────┘││└────────┘│ │
││┌────────┐││┌────────┐││┌────────┐│ │
│││ ││││ ││││ ││ │
│││ OpType ││││ OpType ││││ OpType ││ │
│││ Create ││││ Create ││││ Create ││ │
││└────────┘││└────────┘││└────────┘│ │
│└──────────┘└──────────┘└──────────┘ │
└────────────────────────────────────────────────┘
` ``
## oplog tables
` ``
oplog tables:
as the diagram shows, we can split the
oplog_entries into multiple tables if
needed for performance.
┌────────────────┐
│┌───────────────┴┐
││┌───────────────┴┐ ┌────────────────┐
│││ oplog_entries │ │ oplog_ticket │
││├────────────────┤ ├────────────────┤
│││id │╲ │id │
│││aggregate_name │──┼───────┼ │aggregate_name │
└┤│data │╱ │version │
└┤ │ │ │
└────────────────┘ └────────────────┘
┼
│
│
┼
╱│╲
┌────────────────┐
│ oplog_metadata │
├────────────────┤
│id │
│entry_id │
│key │
│value │
└────────────────┘
` ``
## oplog optimistic locking using tickets
` ``
Alice's Database
transaction Bob's transaction
│ │ │
│─────────BEGIN──────────────────▶│ │
│ │◀───────────────BEGIN─────────────────┤
│ ┌───────────┴───────────┐ │
│ │ ticket version:1 │ │
│ └───────────┬───────────┘ │
│ Select oplog-ticket for │ │
├──────────────"Target"──────────▶│ Select oplog-ticket for │
│ │◀──────────────"Target"───────────────┤
│ │ │
│ Write to Tables in │ │
├───────Target Aggregate─────────▶│ │
│ │ Write to Tables in │
│ │◀─────────Target Aggregate────────────┤
│ │ │
│ │ │
│ Update Ticket │ │
├────Version = 2 where───────────▶│ │
│ Version = 1 ┌─────────┴─────────┐ │
│ │ ticket version: 2 │ │
│ └─────────┬─────────┘ │
│ │ Update Ticket │
│ │◀───────────Version = 2 where─────────┤
│ │ Version = 1 ┌────┴──────────┐
│ │ │ update failed │
├────────────Commit──────────────▶│ └────┬──────────┘
│ │ │
│ │ │
│ │ │
│ │◀────────────Rollback─────────────────┤
│ │ │
│ │ │
` ``
