CyberCoveLLC/threshold
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: Update API docs (#4209)

Browse Source 
* docs: Update API docs

* docs: Update content

* docs: Update content

* docs: Update content

* docs: Fix typos and word choice

* docs: Revisions, add graphic

* Apply batch of suggestions from code review

Co-authored-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>

* docs: Deleting unnecessary content

* docs: Clarify deleted items

* docs: Add new limitation

* docs: Remove list tokens section

* docs: Update `list` command pages

* Update website/content/docs/api-clients/api/index.mdx

Co-authored-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>

* docs: Remove new options from worker doc

---------

Co-authored-by: Johan Brandhorst-Satzkorn <johan.brandhorst@gmail.com>
This commit is contained in:
Dan HeathandJohan Brandhorst-Satzkorn authored and GitHub committed 2024-01-30 20:18:14 -05:00
1 parent fa38d37ba7
commit cffcc30eda
23 files changed
+432 -192
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

website/content/docs/api-clients/api/index.mdx
+93
View File
@@ -0,0 +1,93 @@
---
layout: docs
page_title: API
description: |-
Boundary's HTTP API standards
---
# API
Boundary's API is a JSON-based HTTP API that adheres to a set of standards that are rigidly followed. At its core, it is a standards-compliant JSON API for both input and output.
Before reading this page, it is useful to understand Boundary's [domain model](/boundary/docs/concepts/domain-model) to be aware of the terminology used here.
Boundary's API is also described via OpenAPI v2; the version corresponding to any tag of Boundary's source code can be found in Boundary's [GitHub repository](https://github.com/hashicorp/boundary/blob/main/internal/gen/controller.swagger.json).
-> **NOTE:** A rendered version of this generated API definition can be found on the [API page](/boundary/api-docs).
Boundary's current API version is 1; all API paths begin with `/v1/`.
## Status codes
- `2XX`: Boundary returns a code between `200` and `299` on success. Generally this is `200`, but implementations should be prepared to accept any `2XX` status code as indicating success. If a call returns a `2XX` code that is not `200`, it will follow well-understood semantics for those status codes. (Starting with Boundary 0.2.1, `delete` actions return `204` on success.)
- `400`: Boundary returns `400` when a command cannot be completed due to invalid user input, except for a properly-formatted identifier that does not map to an existing resource, which will return a `404` as discussed below.
- `401`: Boundary returns `401` if no authentication token is provided or if the provided token is invalid. A valid token that simply does not have permission for a resource will return a `403` instead. A token that is invalid or missing, but where the anonymous user (`u_anon`) is able to successfully perform the action, will not return a `401` but instead will return the result of the action.
- `403`: Boundary returns `403` if a provided token was valid but does not have the grants required to perform the requested action.
- `404`: Boundary returns `404` if a resource cannot be found. Note that this happens _prior_ to authentication/authorization checking in nearly all cases as the resource information (such as its scope, available actions, etc.) is a required part of that check. As a result, an action against a resource that does not exist will return a `404` instead of a `401` or `403`. While this could be considered an information leak, since IDs are randomly generated and this only discloses whether an ID is valid, it's tolerable as it allows for far simpler and more robust client implementation.
- `405`: Boundary returns a `405` to indicate that the method (HTTP verb or custom action) is not implemented for the given resource.
- `429`: Boundary returns a `429` if any of the API rate limit quotas have been exhausted for the resource and action. It includes the `Retry-After` header so that the client knows how long to wait before making a new request.
- `500`: Boundary returns `500` if an error occurred that is not (directly) tied to invalid user input. If a `500` is generated, information about the error will be logged to Boundary's server log but is not generally provided to the client.
- `503`: Boundary returns a `503` if it is unable to store a quota due to the API rate limit being exceeded. It includes the `Retry-After` header so that the client knows how long to wait before making a new request.
## Path layout
Boundary follows a predictable path layout. There are two fundamental types of URL paths, each supporting a different set of operations.
### Collections
Collections of resources are top level paths with plural English names for the resource, e.g. `/roles` and `/hosts`. Collections support the following operations:
- Creation of new resources within that collection
- Listing of resources within that collection
All collection operations require supplying the enclosing resource. Depending on the collection type, this will be one of the following:
- A scope, indicating the scope in which an operation should take place. For instance, a POST to `/roles` would need to indicate whether the role should be created within the `global` scope or an org-level scope like `o_1234567890`.
- A parent resource of the appropriate type. For instance, hosts and host sets are child resources of host catalogs. When creating a new host set within a host catalog, a POST to `/host-sets` would need to indicate the host catalog ID with which the host-set should be associated.
### Resources
Resources themselves are defined by ID specifiers within a collection path, e.g. `/roles/r_1234567890`. Resources support the following operations:
- Reading a resource's properties
- Updating a resource's properties
- Deleting a resource
- Custom methods specific to a resource type
Depending on the resource type, various parameters may be available. Some are common across all resource types (e.g. `name` and `description`); others are available only for specific types. Further, some concrete-types of abstract resources include an opaque `attributes` JSON object with type-specific values.
For instance, an auth method is an abstract type; a `password` auth method is a concrete implementation of that type. When creating such an auth method, a `type` parameter will indicate that it should be the `password` type, while values specific to the `password` type auth method, such as minimum password length, will be contained within an `attributes` object.
## Methods
The following method conventions are used within Boundary's API:
### GET
`GET` is used for reading a resource or listing resources in a collection. The behavior depends on whether the `GET` is issued against a collection (`/roles`) or a singular resource (`/roles/r_1234567890`). In the former case it lists resources within the collection; in the latter it performs a read on that particular resource.
### POST
`POST` is used for creating a resource or performing custom actions against a resource. When creating a resource, `POST` is used against a collection (`/roles`). When performing a custom action, `POST` is used against a particular resource (`/roles/r_1234567890:set-principals`).
### PATCH
`PATCH` is used to update a resource's parameters. The following are behaviors to be aware of when using `PATCH`:
- In nearly all cases, a `version` parameter is required. This is used for check-and-set, to ensure that the update operation is being performed against a known resource. The version parameter is returned from a `GET` operation on the resource so the current version, along with the resource's other current values, can be looked up at any time.
- Passing a JSON `null` for a parameter has the effect of reverting that parameter to its default. For some parameters (e.g. `name`) this will simply clear the value (as the default `name` for a resource is empty); for other parameters this will revert to the current defaults within Boundary.
- All parameters specified as part of a `PATCH` operation will be considered to be parameters that should be updated.
### DELETE
`DELETE` is used for deleting a specific resource, and is only used against a particular resource path.
## Manage system resources
If your controllers try to process every API request, they may either run out of resources or overwhelm the database server.
Boundary features API rate limiting to help prevent system resources from being overwhlemed by too many simultaneous requests.
For more information, refer to [API rate limiting](/boundary/docs/api-clients/api/rate-limiting).
When you search for Boundary resources using the `list` operation, you could potentially receive a very large number of results.
Boundary uses an API pagination feature to enable you to search and filter large lists of search results without overwhelming your system resources.
Refer to [API list pagination](/boundary/docs/api-clients/api/pagination) for more information about pagination and using the local cache for searches.
website/content/docs/api-clients/api/pagination.mdx
+103
View File
@@ -0,0 +1,103 @@
---
layout: docs
page_title: API list pagination
description: Learn how the API pagination and cache works in Boundary to prevent system resources from being overwhlemed and to help you find resources.
---
# API list pagination
Boundary uses an API pagination feature to support searching and filtering large lists of results without overwhelming your system resources.
## Introduction
When you perform a `list` operation on a resource, the API request returns the first 1000 results, by default.
You can request the next page of results by repeating the `list` command and including a parameter with a token provided by Boundary.
## Initial pagination
When you run a `list` operation on a resource, the initial results are sorted in descending order by creation time and public ID (most recently created first).
Boundary returns the maximum number of responses along with the top-level fields `response_type` and `list_token`.
If the `response_type` is `delta`, it means there are more items available in the list.
List tokens contain general metadata associated with the ongoing pagination.
To request the next page of results, run the `list` operation again and include the `list_token` that Boundary provided in the last response.
Boundary returns the next page of results with new `response_type` and `list_token` values.
![API pagination workflow](/img/api-pagination/pagination-workflow.png)
You can continue viewing new pages of results using the `list_token` values until the `response_type` reads `complete`.
If you compose an invalid token, you receive an invalid token error.
## Request updated and deleted resources
When the `response_type` reads `complete`, you have viewed all the results that were available when you ran the initial `list` operation.
You can now request any resources that were updated or deleted since you started paginating.
To refresh the cached list results, make another `list` request using the `list_token` from the previous response.
Boundary returns a `response_type` and `list_token` as part of the response, similarly to when you ran the initial operation.
If the `response_type` is `delta`, it means there are more items available in the list.
You can use the `list_token` as a parameter to continue viewing new pages of results until the `response_type` reads `complete`.
Boundary also includes the top-level response `removed_ids` as part of the refresh operation.
The `removed_ids` response contains the IDs of any resources that were deleted since you began paginating the results of the previous `list` operation.
Any resources that have been deleted are included on the first page of results.
After 30 days, the `list_token` expires and you must make a new `list` request without the token to regenerate the full list of results.
If you make a `list` request using a token that has a `create_time` older than 30 days, Boundary treats it as an invalid token and returns an error.
## Response structure
All `list` endpoints support the following parameters:
- `list_token` (optional) - An opaque token that is returned from the previous `list` response.
If you do not supply a value, pagination starts from the beginning.
- `page_size` (optional) - An unsigned integer that indicates the number of items that should be included on the page of search results.
If you do not include a value, or if you include a value of `0`, Boundary uses the default page size of 1000 items.
If you configure it, the `page_size` value overrides the default page size.
Controller administrators can also configure a `max_page_size` option that constrains the maximum page size a user can request.
Boundary returns the following response parameters:
- `response_type` - Defines whether the response is a `delta` or `complete` listing.
The value is `delta` until the last page is returned, at which point it will be `complete`.
- `list_token` - An opaque token which is included in subsequent listing requests after the initial one, so that the client can continue to paginate the results.
- `sort_by` - The column name that the returned items should be sorted on.
This attribute is included in the response, so that the sort order for each endpoint is explicit.
- `sort_dir` - The direction in which the returned items are sorted.
Responses can include `desc` for descending, or `asc` for ascending.
- `items` - The list of items.
- `est_item_count` - An unsigned integer that indicates the total number of items that are available in the list of results.
This number may be an estimate, since the number of items may change during the pagination process.
### Example response
```json
{
"response_type": "<delta|complete>",
"list_token": "<opaque-token>",
"sort_by": "created_time",
"sort_dir": "desc",
"est_item_count": 1000,
"items": [
{
"id": "ttcp_1234567890",
"scope_id": "p_1234567890"
},
{
"id": "ttcp_2234567890",
"scope_id": "p_1234567890"
}
]
}
```
## Constraints and limitations
In scenarios where updates are so frequent that the results may change while the refresh operation runs, those updates may not appear at all in the refreshed list.
If you make frequent updates to a resource, or if you are updating data while you run a `list` operation, HashiCorp recommends that you generate the full list of results instead of performing a refresh.
Workers are not included in refresh operations because the updates they send to the controller are too frequent to be captured in a refresh.
To update a list of worker resources, you must regenerate the full list of results.
When you refresh the list of results, you may notice duplicate items in the list.
This is a fundamental tradeoff when you search across multiple database transactions.
website/content/docs/api-clients/api/rate-limiting.mdx
+199
View File
@@ -0,0 +1,199 @@
---
layout: docs
page_title: API rate limiting
description: Learn how API rate limiting lets you configure limits on the rates of API requests in Boundary to help manage resources and prevent them from being overwhelmed.
---
# Rate limiting
If your controllers try to process every API request, it can lead to situations when the controllers are overwhelmed with requests.
The controllers may either run out of resources or overwhelm the database server and exhaust its resources.
API rate limiting lets you configure limits on the rate of API requests to help manage your resources and prevent them from being overwhelmed.
## Quotas
Boundary creates quotas to track the number of requests in a given time period.
By default, Boundary tracks the number of requests by auth token, IP address, and the overall total.
Boundary reserves a portion of its memory for tracking quotas to ensure that it does not consume too much memory if there is a sudden burst of requests.
If Boundary is unable to store a quota, it limits the request with a 503 HTTP status code.
You can configure the maximum number of quotas Boundary allows using the `api_rate_limit_max_quotas` variable.
There are also two metrics that allow you to monitor quota tracking:
- `boundary_controller_api_ratelimiter_quota_storage_capacity`
- `boundary_controller_api_ratelimiter_quota_storage_usage`
## Default limits
API rate limiting is enforced on the controllers.
There are separate configurable limits for each combination of resource and action.
By default, the limits for `list` actions are:
- 150 requests per 30 seconds per auth token
- 1,500 requests per 30 seconds per IP address
- 1,500 requests per 30 seconds in total
The default limits for all other actions are:
- 3,000 requests per 30 seconds per auth token
- 30,000 requests per 30 seconds per IP address
- 30,000 requests per 30 seconds in total
You can override the default settings and configure other specific limitations using the `api_rate_limit` stanza in the controller configuration.
## HTTP headers
Clients that make requests to the controller API can inspect HTTP response headers to understand the configured limits and current usage.
Each response contains the following headers:
- `RateLimit` - Provides the current limit, number of remaining requests, and the time at which the quota will reset for the limit that is closest to being exhausted for the requested resource and action.
- `RateLimit-Policy` - Describes the limits for the requested resource and action.
If the request is limited, Boundary sends the client a 429 HTTP status code with a `Retry-After` header.
The `Retry-After` header contains the number of seconds the client should wait before it sends the request again.
## More information
Refer to the [controller stanza](/boundary/docs/configuration/controller#api_rate_limit) documentation for the specific `api_rate_limit` configuration options.
Some example configurations are listed below.
### Rate limiting configuration examples
The following example shows a simple configuration where the same limits are applied to all resources and examples:
```hcl
controller {
# Total limit for all resources and actions
api_rate_limit {
resources = ["*"]
actions = ["*"]
per = "total"
limit = 500
period = "1s"
}
# Limit for all IP addresses to all resources and actions to prevent a
# malicious host that is fabricating tokens or spamming unauthed endpoints
api_rate_limit {
resources = ["*"]
actions = ["*"]
per = "ip-address"
limit = 100
period = "1s"
}
# Limit for all authed requests to prevent one user
# from consuming all of the total limit
api_rate_limit {
resources = ["*"]
actions = ["*"]
per = "auth-token"
limit = 100
period = "1s"
}
}
```
The following example shows a configuration that disables rate limiting.
You may want to disable rate limiting if you already use an external system like a reverse proxy to apply rate limiting:
```hcl
controller {
api_rate_limit_disable = true
}
```
The following example uses the default settings for most endpoints, but configures a single override:
```hcl
controller {
api_rate_limit {
resources = ["target"]
actions = ["list"]
per = "auth-token"
limit = 10
period = "1s"
}
}
```
The following example is more complex.
Initially it sets some defaults to apply to all resources and actions.
Then it configures some specific endpoints with different limits.
```hcl
controller {
# Total limit for all resources and actions
api_rate_limit {
resources = ["*"]
actions = ["*"]
per = "total"
limit = 500
period = "1s"
}
# Total limit for all list operations,
# sets an overall cap on the list action since it is relatively expensive
api_rate_limit {
resources = ["*"]
actions = ["list"]
per = "total"
limit = 200
period = "1s"
}
# Limit for IP address to all resources and actions
# to prevent a malicious host that is fabricating tokens
# or spamming unauthed endpoints
api_rate_limit {
resources = ["*"]
actions = ["*"]
per = "ip-address"
limit = 50
period = "1s"
}
# Limit of all authed requests, this essentially sets a default
# for all authed requests to prevent one user from consuming
# all of the total list limit set above. Any limits that follow
# can override the default for specific sets of resources and actions
api_rate_limit {
resources = ["*"]
actions = ["*"]
per = "auth-token"
limit = 100
period = "1s"
}
# Limit of all list operations by auth token to set a sane default
# since they are generally more expensive
api_rate_limit {
resources = ["*"]
actions = ["list"]
per = "auth-token"
limit = 50
period = "1s"
}
# Limit for targets and sessions list by auth token. These are resources
# that clients will want to list frequently, but might be more expensive
# if there is a large number of them for each user. Having a lower limit
# encourages the use of refresh tokens and caching.
api_rate_limit {
resources = ["target", "session"]
actions = ["list"]
per = "auth-token"
limit = 20
period = "1s"
}
#Limit for authorize session by auth token
api_rate_limit {
resources = ["target"]
actions = ["authorize-session"]
per = "auth-token"
limit = 150
period = "1s"
}
}
```