Michael Gaffney
41bdfb4406
The hcp monthly billing views build a range of buckets using the
`generate_series` and the `now()` functions along with relative offsets
like `+ 1 month`. The calculated start and end times for these views can
vary depending on the time zone setting of the PostgreSQL server.
This adds 3 table functions that call their respective views with the
time zone set to UTC. Calling these functions instead of their
underlying views directly prevents the returned values from varying when
the server's time zone is not set to UTC.
This also drops support for PostgreSQL 11. The included tests use the
`date_trunc(field, source [, time_zone ])` function added in PostgreSQL
12.
Fixes: 41b1b5b4b7
oplog
oplog is a package for writing operation log (oplog) entries for the purpose of replication and verification of the data stored in the Boundary RDBMS.
Usage
// you must init the ticket in its own transaction. You only need
// to init a ticket once in the database. It doesn't need to happen for
// every connection. Once it's persistent in the database, you can simply Get it.
initTx := db.Begin()
ticketer := &GormTicketer{Tx: initTx}
err = ticketer.InitTicket("users")
// if there's no error, then commit the initialized ticket
initTx.Commit()
userCreate := oplog_test.TestUser{
TestUser: oplog_test.TestUser{
Name: loginName,
},
}
tx := db.Begin()
// write the user to the database
tx.Create(&userCreate)
ticketer = &GormTicketer{Tx: db}
// get a ticket for writing users to the oplog
ticket, err := ticketer.GetTicket("users")
// create an entry for the oplog with the entry's metadata (likely the entry's Scope)
newLogEntry := NewEntry(
"test-users",
[]Metadata{
Metadata{
Key: "deployment",
Value: "amex",
},
Metadata{
Key: "project",
Value: "central-info-systems",
},
},
cipherer, // wrapping.Wrapper
ticketer,
)
// write an entry with N messages (variadic parameter) in the order they were sent to the database
_, err = newLogEntry.WriteEntryWith(
context.Background(),
&GormWriter{tx},
ticket,
&Message{Message: &userCreate, TypeName: "user", OpType: OpType_CREATE_OP},
)
// if there's an error writing the oplog then roll EVERYTHING back
if err != nil {
tx.Rollback()
}
// no err means you can commit all the things.
tx.Commit()
TBD/TODO
We need to discuss and decide how Boundary is going to handle the following oplog things:
- SQL migrations: you'll find the package's SQL migrations under: ./db/schema We need to decide how Boundary will manage migrations across the system and we will likely need to reference this package's migrations somehow.
oplog entry
Example Oplog Entry for the Target Aggregate
┌────────────────────────────────────────────────┐
│ FIFO with []byte buffer │
│ │
│┌─Msg 4────┐┌─Msg 3────┐┌─Msg 2────┐ │
││ ││ ││ │ │
││ Tags ││ Host ││ HostSet │ │
││┌────────┐││┌────────┐││┌────────┐│ │
│││ ││││ ││││ ││ │
│││ ││││ ││││ ││ │
│││ Tag ││││ Host ││││HostSet ││ │
│││protobuf││││protobuf││││protobuf││ │
│││ ││││ ││││ ││ │
│││ ││││ ││││ ││ │
││└────────┘││└────────┘││└────────┘│ │
││┌────────┐││┌────────┐││┌────────┐│ │
│││ ││││ ││││ ││ │
│││typeName││││typeName││││typeName││ │
│││ Tag ││││ Host ││││HostSet ││ │
││└────────┘││└────────┘││└────────┘│ │
││┌────────┐││┌────────┐││┌────────┐│ │
│││ ││││ ││││ ││ │
│││ OpType ││││ OpType ││││ OpType ││ │
│││ Create ││││ Create ││││ Create ││ │
││└────────┘││└────────┘││└────────┘│ │
│└──────────┘└──────────┘└──────────┘ │
└────────────────────────────────────────────────┘
oplog tables
oplog tables:
as the diagram shows, we can split the
oplog_entries into multiple tables if
needed for performance.
┌────────────────┐
│┌───────────────┴┐
││┌───────────────┴┐ ┌────────────────┐
│││ oplog_entries │ │ oplog_ticket │
││├────────────────┤ ├────────────────┤
│││id │╲ │id │
│││aggregate_name │──┼───────┼ │aggregate_name │
└┤│data │╱ │version │
└┤ │ │ │
└────────────────┘ └────────────────┘
┼
│
│
┼
╱│╲
┌────────────────┐
│ oplog_metadata │
├────────────────┤
│id │
│entry_id │
│key │
│value │
└────────────────┘
oplog optimistic locking using tickets
Alice's Database
transaction Bob's transaction
│ │ │
│─────────BEGIN──────────────────▶│ │
│ │◀───────────────BEGIN─────────────────┤
│ ┌───────────┴───────────┐ │
│ │ ticket version:1 │ │
│ └───────────┬───────────┘ │
│ Select oplog-ticket for │ │
├──────────────"Target"──────────▶│ Select oplog-ticket for │
│ │◀──────────────"Target"───────────────┤
│ │ │
│ Write to Tables in │ │
├───────Target Aggregate─────────▶│ │
│ │ Write to Tables in │
│ │◀─────────Target Aggregate────────────┤
│ │ │
│ │ │
│ Update Ticket │ │
├────Version = 2 where───────────▶│ │
│ Version = 1 ┌─────────┴─────────┐ │
│ │ ticket version: 2 │ │
│ └─────────┬─────────┘ │
│ │ Update Ticket │
│ │◀───────────Version = 2 where─────────┤
│ │ Version = 1 ┌────┴──────────┐
│ │ │ update failed │
├────────────Commit──────────────▶│ └────┬──────────┘
│ │ │
│ │ │
│ │ │
│ │◀────────────Rollback─────────────────┤
│ │ │
│ │ │
```