Listing Session and Targets Performance Improvements
Improve response time for `GET` requests to:
- `/v1/targets`
- `/v1/sessions`
These endpoints now perform authorization checks prior to retrieving the
full resources from the database.
This also includes a bug fix for retrieving sessions that could result
in incomplete results when there is a large number (10k+) of sessions.
When applying the limit and order by, the where clause also needs to be
applied to yield the correct results. This can be seen with an example.
Assuming a data set where there are 100 sessions, with 10 sessions
belonging to a user with id `u_user1` and 90 sessions belonging to
`u_user2`. If the constructed query is like:
select *
from
(select public_id from session limit 10) s,
session_list ss
where
s.public_id = ss.public_id
and s.user_id = 'u_user1'
order by create_time asc;
The initial sub-query with the limit could return sessions only
belonging to `u_user2`, then it would apply the where clause and exclude
all of the rows.
Instead if the query is:
with
session_ids as (
select public_id
from sessions as s
where
s.user_id = 'u_user1'
order by create_time asc
limit 10
)
select *
from session_list
where
session_list.public_id in (select * from session_ids)
order by create_time asc;
The CTE applies the where clause ensuring that only sessions for
`u_user1` are used for the final select from the view.
Note that the CTE is use primarily for readability.
This also adds an index on `session (create_time)` to aide the query
planner in performing the order by.
The list benchmark tests uses pre-created database
dumps to run its benchmarks. This benchmark
will be used to guide optimizations around listing
of sessions.
Fixes https://hashicorp.atlassian.net/browse/ICU-4042
Mount the benchmark dump directory when running
the test database and create a new script for creating
templates from the dumps. This makes it easy to create
pre-populated databases for benchmark tests.
Fixes https://hashicorp.atlassian.net/browse/ICU-4040
Sometimes when creating a lot of sessions concurrently we
can run into a unique constraint error in the warehouse triggers
in the database. Simply retrying the call will allow the
session creation to succeed.
Our test helper packages were not possible to use
with benchmarks. This refactor switches to testing.TB
to allow both tests and benchmarks to use the helpers.
The new docker-load target uses docker buildx to build
a local image of each of the Dockerfiles, and load them
into the local docker socket image store.
* Reorder session authz checks
This implements a mechanism for service handlers to pass a repo
satisfying an interface into the scope IDs listing function. The purpose
of this interface is to allow the same function to perform authorization
checking against resources rather than requiring service handlers to
fetch all resources in the determined set of scopes and then perform
this checking. While this could be done in each service handler, this
approach centralizes the logic, which has the nice benefit of removing
some boilerplate from service handler list functions that have adapted
to it.
Tests haven't changed, which is expected -- the function should return
exactly what was being returned before, simply faster.
* refact(dbtest): Remove old migration code
This is no longer needed, so we can clean up this
script a bit.
* refact(dbtest): Change test image env var
The existing environment variable clashes with
the one used in the top level Makefile. The new
name prevents confusing behavior when setting
the env var and creating the test database from
the root of the repository.
Previously we were matching what boundary considers an id, but that is determined after requests are routed by grpc-gateway.
This changes the regexp to match any path segment to try to behave the same way that grpc-gateway would when assigning values
to the {id} variable declared in the protobuf.
The wrapper is sometimes unset at the time when the
audit config is validated, so it is impossible for us
to validate that a valid wrapper is configured. The wrapper
is configured by later setup steps because of a dependency
on the eventer in our KMS setup.
Classify fields for all api and cluster protobuf messages.
Before classification, audit events would be completely redacted.
Now events will contain unredacted values according to classification.
Audit events are now no longer considered a WIP.
This updates the target resources to use a oneof for attributes. This
allows requests to the targets service to leverage the new
subtypes.AttributeTransformerInterceptor so that attributes are strongly
typed prior to the service handler.
The new `subtype.AttributeTransformerInterceptor` can return an error if
it fails to transform the attributes. This is mainly if invalid
attribute fields were included for a particular subtype. This error case
used to be checked in the grpc service handler functions. However, now
that it is happening in the interceptor, the `errorInterceptor` needs to
be moved up in the stack so that it will process the response of the new
transformation interceptor.
The `errorInterceptor` used to be after the `auditRequestInterceptor`
and prior to the `auditResponseInterceptor`, meaning that the
`auditResponseInterceptor` would have access to the result of the
`errorInterceptor`. But it seems the `auditRequestInterceptor` does not
use or need any of the headers set by the `errorInterceptor` so it
should be safe to move the `errorInterceptor`. This is important since
the `subtype.AttributeTransformerInterceptor` needs to be before the
audit interceptors in the stack to properly transform requests/responses
so they can be audited with the complete classification information.
This updates the credential store resources to use a oneof for
attributes. This allows requests to the credential store service to
leverage the new subtypes.AttributeTransformerInterceptor so that
attributes are strongly typed prior to the service handler.
This updates the credential library resources to use a oneof for
attributes. This allows requests to the credential library service to
leverage the new subtypes.AttributeTransformerInterceptor so that
attributes are strongly typed prior to the service handler.
This updates the credential store resources to use a oneof for
attributes. This allows requests to the credential store service to
leverage the new subtypes.AttributeTransformerInterceptor so that
attributes are strongly typed prior to the service handler.
The original idea was for these methods to eventually become
entirely encapsulated by the subtype package, but as it turns
out we still have use for them in custom transformation functions,
so move the subtype package to use these shared functions
instead.
This allows the user to register a custom transformation callback
for a specific protobuf message, in cases where the transformation
is sufficiently complex for it to be cumbersome to formalise
the logic through protobuf options.
This return value is just a reference to the input, which
is being mutated. We may as well make that clear via
the signature, which will also simplify the new custom
transformation function registration logic.
It is not unreasonable for downstream users of the request and response
messages to mutate them, indeed we mutate the response
on the way out to get the desired external API. To avoid
inconsistent messages being audited, make a copy of the
entire message before returning from the interceptor.
This was discovered while testing the new subtype transformation
interceptor.
Irena Rindos
irenarindos
Timothy Messier
Jeff Mitchell
Johan Brandhorst-Satzkorn
Todd
Carlos Alcaide Corvo