Refactors ListCredentials and adds the new ListCredentialsRefresh
method to the Repository, supporting pagination
of credentials. Also adds the EstimateCount and
ListDeletedCredentialIds methods used during
pagination.
* internal/target: fix some target tests
We had dupes and missing test cases, this fixes them.
* internal/handlers: move listtoken parsing into helper
This significantly simplifies the parsing logic needed in each handler.
When using the new auto-pagination logic, we would previously return
the Body and Map of the last response, instead of one corresponding
to the returned result. Correct this by overwriting the Body and Map
after pagination has finished.
Validating that the update time of the last item is after the
create time of the token cannot account for items whose
update time are earlier than the original query because of
the added database transaction timeout buffer. Remove the
check.
The new design uses four separate service functions; List,
ListPage, ListRefresh and ListRefreshPage. The first two are
used during the initial pagination phase and the latter two
are used for any subsequent refresh phases.
The new design uses the base table for querying, so the base
table needs to have both create_time and update_time. In this
commit we add the update_time to the target base table. We
also add a number of new indexes to help with looking up targets.
The new design defines 4 distinct functions, List, ListPage, ListRefresh
and ListRefreshPage. The first two are used during the initial
pagination, and the other two are used during refreshing.
Updates all List endpoints to automatically paginate
through results to preserve backwards compatibility.
Adding support for user controllable pagination could
be considered at a later time.
The max_page_size variable lets operators control the size of the
pagination pages. This value configures both the maximum permitted
page size and the default page size when none is explicitly provided.
Operators can tweak this option up or down to increase or reduce
the load on the database during pagination.
The ListRefreshToken type is used, in its marshalled form,
for pagination requests. A new refresh token is returned
which each invocation of a list method (unless
there was no input refresh token and there were no results).
The pagination package provides generic functions for
iterating over a collection of items such that each incoming
request is filled even after filtering is applied.
The refreshtoken package encapsulates domain logic surrouding
the refresh token. A refresh token is used both for
paginating through a collection and for requesting any
updates to that collection.
The resource type method is used to automatically
create refresh tokens from a resource when that
resource is the last item returned in a page.
This also renames the iam domain ResourceType method
to GetResourceType to align with the new method.
Creates a new foo_deleted table for each subtype of each
resource that needs to support pagination. The deleted
tables are used to track deletions of resources and return
them to users of the new pagination API. We track only the
ID and delete_time of the resource, so there's no way to
filter out resources that the user could not see before it
was deleted. This is intentional to keep the design simple
and is not considered a security issue.
This increases the default rate limits to:
- 1,500 reqs / 30 seconds for list actions in total
- 1,500 reqs / 30 seconds for list actions per IP address
- 150 reqs / 30 seconds for list actions per auth token
And for all other actions:
- 30,000 reqs / 30 seconds for list actions in total
- 30,000 reqs / 30 seconds for list actions per IP address
- 3,000 reqs / 30 seconds for list actions per auth token
This seems to be a reasonable default for a small to medium size cluster
based on some scale testing.
(cherry picked from commit 237c8b6084f42655edaba49055e511163fdf1c53)
Previously this would truncate to the nearest integer, which could
result in clients retrying before the quota has reset, and getting rate
limited again. This also includes an update from go-rate that will round
up when reporting the reset time for the RateLimit header.
Note that in the case of a 503, the Retry-After header was already being
set correctly.
See: https://github.com/hashicorp/go-rate/pull/8
A sys event will be emitted when the controller is constructed, and if
the rate limits are changed via a SIGHUP. The event contains the details
of the rate limits for each combination of resource and action. This
event can be used by operators to confirm the limits are set as they
would expect based on the controller config.
