Files
threshold/internal/db
Michael Gaffney a9e806b673 Data warehouse (#475)
* Define initial warehouse schema

* Add session id as the primary key to the session accumulating fact table

* Insert a row to use instead of nulls in the time and date dimensions

* Add missing attributes and a partial index

* Rename files

* Refactor: move fact table to a separate migration

* Add domain types for the warehouse

* Add session connection transaction fact table

* Change columns to match ingress data types

* Add views for host dimension ingress

* Create source and target views for the host and user dimensions

* Generate random ID for dimension table primary key

* Formatting changes

* Add current dimension id to target views

* Implement upsert function for host dimension

* Use domain types for upsert function parameters

* Fix indentation

* Add wh_public_id domain type

* Fix value of type columns when the other columns are null

* Implement upsert function for user dimension

* Add warehouse domain types for timestamps and text fields

* Move fact table migrations to a higher number

* Add functions for getting the current date and current time IDs

* Rename backend to endpoint in column names

* Rename fact tables

The fact tables are actually accumulating fact tables not transactional
fact tables.

* Add default value to connection_authorized_time column

* Update session fact table

* Add session_max_seconds, session_connection_limit, and default_port to wh_host_dimension

* Move session table above connection table

* Add after insert trigger to session table for warehouse ingress

* Add fk from connection warehouse table to session warehouse table

* Implement insert trigger for session connections

* Update warehouse for a user having multiple auth accounts

* Rename warehouse columns to be consistent with operational tables

* Add connection specific columns to the connection fact table

* Update session calculations in warehouse

* Rename functions and formatting changes

* Rename session columns to match names of session states

* Implement warehouse triggers for session and connection state changes

* Run 'make migrations'

* Use a static start and end date for the warehouse date dimension

* Document warehouse functions

* Remove outdated TODO comments

* Add TODO comments for additional dimensions and fact table columns
2020-09-30 14:09:40 -04:00
..
2020-08-10 17:15:02 -04:00
2020-08-10 17:15:02 -04:00
2020-09-30 14:09:40 -04:00
2020-08-10 17:15:02 -04:00
2020-08-19 16:47:18 -04:00
2020-08-18 16:50:26 -04:00
2020-06-29 08:18:57 -04:00
2020-06-29 08:18:57 -04:00
2020-08-19 16:47:18 -04:00
2020-08-19 16:47:18 -04:00
2020-05-11 16:54:05 -04:00
2020-08-10 17:15:02 -04:00

db package

Usage

Just some high-level usage highlights to get you started. Read the godocs for a complete list of capabilities and their documentation.

    conn, _ := gorm.Open("postgres", url)
    
    // Db implements both the Reader and Writer interfaces
    rw := Db{Tx: conn}
    
    // There are writer methods like: Create, Update and Delete
    // that will write Gorm struct to the db.  These writer methods
    // all support options for writing Oplog entries for the 
    // caller: WithOplog(yourWrapper, yourMetadata)
    // the caller is responsible for the transaction life cycle of the writer
    // and if an error is returned the caller must decide what to do with 
    // the transaction, which is almost always a rollback for the caller.
    err = rw.Create(context.Background(), user)
   
    // There are reader methods like: LookupByPublicId,  
    // LookupByName, SearchBy, LookupBy, etc
    // which will lookup resources for you and scan them into your Gorm struct
    err = rw.LookupByPublicId(context.Background(), foundUser)

    // There's reader ScanRows that facilitates scanning rows from 
    // a "raw" SQL query into your Gorm struct
    tx, err := rw.DB()
    where := "select * from test_users where name in ($1, $2)"
    rows, err := tx.Query(where, "alice", "bob")
	defer rows.Close()
	for rows.Next() {
        user := db_test.NewTestUser()
        // scan the row into your Gorm struct
		if err := rw.ScanRows(rows, &user); err != nil {
            return err
        }
        // Do something with the Gorm user struct
    }

    // DoTx is a writer function that wraps a TxHandler 
    // in a retryable transaction.  You simply implement a
    // TxHandler that does your writes and hand the handler
    // to DoTx, which will wrap the writes in a retryable 
    // transaction with the retry attempts and backoff
    // strategy you specify via options.
    _, err = w.DoTx(
        context.Background(),
        10,           // ten retries
        ExpBackoff{}, // exponential backoff
        func(w Writer) error {
            // the TxHandler updates the user's friendly name
            return w.Update(context.Background(), user, []string{"FriendlyName"},
                // write oplogs for this update
                WithOplog(
                    InitTestWrapper(t),
                    oplog.Metadata{
                        "key-only":   nil,
                        "deployment": []string{"amex"},
                        "project":    []string{"central-info-systems", "local-info-systems"},
                    },
                ),
            )
        })