Refactors ListCredentials and adds the new ListCredentialsRefresh
method to the Repository, supporting pagination
of credentials. Also adds the EstimateCount and
ListDeletedCredentialIds methods used during
pagination.
* internal/target: fix some target tests
We had dupes and missing test cases, this fixes them.
* internal/handlers: move listtoken parsing into helper
This significantly simplifies the parsing logic needed in each handler.
* testing/e2e: Use a separate config file for database init
The other config file contains templated parameters, which was causing
errors when loading the config file, even though the values were never
used.
* testing/e2e: add target pagination test
Exercises target pagination both over the CLI and the API. These tests
run in <10s on my machine, though they each involve creating 1000
targets (with current defaults).
---------
Co-authored-by: Michael Li <michael.li@hashicorp.com>
When using the new auto-pagination logic, we would previously return
the Body and Map of the last response, instead of one corresponding
to the returned result. Correct this by overwriting the Body and Map
after pagination has finished.
Validating that the update time of the last item is after the
create time of the token cannot account for items whose
update time are earlier than the original query because of
the added database transaction timeout buffer. Remove the
check.
The new design uses four separate service functions; List,
ListPage, ListRefresh and ListRefreshPage. The first two are
used during the initial pagination phase and the latter two
are used for any subsequent refresh phases.
The new design uses the base table for querying, so the base
table needs to have both create_time and update_time. In this
commit we add the update_time to the target base table. We
also add a number of new indexes to help with looking up targets.
The new design defines 4 distinct functions, List, ListPage, ListRefresh
and ListRefreshPage. The first two are used during the initial
pagination, and the other two are used during refreshing.
Updates all List endpoints to automatically paginate
through results to preserve backwards compatibility.
Adding support for user controllable pagination could
be considered at a later time.
The max_page_size variable lets operators control the size of the
pagination pages. This value configures both the maximum permitted
page size and the default page size when none is explicitly provided.
Operators can tweak this option up or down to increase or reduce
the load on the database during pagination.
The ListRefreshToken type is used, in its marshalled form,
for pagination requests. A new refresh token is returned
which each invocation of a list method (unless
there was no input refresh token and there were no results).
The pagination package provides generic functions for
iterating over a collection of items such that each incoming
request is filled even after filtering is applied.
The refreshtoken package encapsulates domain logic surrouding
the refresh token. A refresh token is used both for
paginating through a collection and for requesting any
updates to that collection.
The resource type method is used to automatically
create refresh tokens from a resource when that
resource is the last item returned in a page.
This also renames the iam domain ResourceType method
to GetResourceType to align with the new method.
Creates a new foo_deleted table for each subtype of each
resource that needs to support pagination. The deleted
tables are used to track deletions of resources and return
them to users of the new pagination API. We track only the
ID and delete_time of the resource, so there's no way to
filter out resources that the user could not see before it
was deleted. This is intentional to keep the design simple
and is not considered a security issue.
