Files
threshold-plugin-aws/plugin/service/host
Matt Romanelli f81151b58c bug(go) AuthFailure when performing static credential rotation
**Problem**
* While adding e2e tests for AssumeRole functionality, it was noticed that some of the static credential e2e tests were flaky.
* The tests would sometimes fail with an AuthFailure.

**Solution**
* It was discovered that this error happened sometimes after a static credential rotation and that AuthFailure occurs if EC2 is invoked with the wrong credentials.
* Due to IAM's eventual consistency model, it can take some time (up to several seconds) for changes, including rotated credentials, to get propagated to all AWS services, including EC2.
* The tests were intermittently failing because sometimes the credentials would be propagated in time before invoking EC2 and other times they were not propagated in time.
* The fix is to add a retry mechanism when calling DescribeInstances. A 30 second window is defined in which the retries can take place.
* The retries happen only for AuthFailures. Any other failure will not trigger a retry.

**Testing**
* After making this change, the intermittent failures were no longer seen for the e2e tests
* The e2e tests cover updating/creating catalogs and sets as well us updating the host list. All pass.
* New unit tests were added to test the retry mechanism. Support had to be added to mock a series of DescribeInstances calls vs a single call
* New and existing unit tests passed
2026-09-22 12:27:00 -05:00
..

Getting Started

Refer to Attributes and Secrets for more detail on configuration options for host catalogs and sets.

To create a host catalog (using default scope created by boundary dev):

boundary host-catalogs create plugin \
 -scope-id p_1234567890 \
 -name "Example Plugin-Based Host Catalog" \
 -description "Description for plugin-based host catalog" \
 -plugin-name aws \
 -attr region=REGION \
 -secret access_key_id='KEY' \
 -secret secret_access_key='SECRET'

To create a host set, filtering the host set based on tag keys foo or bar (either tag can be present), ensuring that any targets set to this host set only connect to external addresses in the 54.0.0.0/8 class A subnet:

boundary host-sets create plugin \
 -host-catalog-id HOST_CATALOG_ID \
 -name "Example Plugin-Based Host Set" \
 -description "Description for plugin-based host set" \
 -attr filters=tag-key=foo,bar \
 -preferred-endpoint "cidr:54.0.0.0/8"

As above, but instances must have both tags (both foo and bar must be present):

boundary host-sets create plugin \
 -host-catalog-id HOST_CATALOG_ID \
 -name "Example Plugin-Based Host Set" \
 -description "Description for plugin-based host set" \
 -attr filters=tag-key=foo \
 -attr filters=tag-key=bar \
 -preferred-endpoint "cidr:54.0.0.0/8"

As above, but matching on tag key named "development" with value "prod" or "dev" and launch date:

boundary host-sets create plugin \
 -host-catalog-id HOST_CATALOG_ID \
 -name "Example Plugin-Based Host Set" \
 -description "Description for plugin-based host set" \
 -attr filters=tag:development=prod,dev \
 -attr filters=launch-time=2022-01-04T* \
 -preferred-endpoint "cidr:54.0.0.0/8"

Required IAM Privileges

The following IAM privileges, at the very least, are required to be attached to a configured IAM user for this provider:

  • ec2:DescribeInstances, configured to * (DescribeInstances cannot be scoped to a resource ARN). Example policy:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "ec2:DescribeInstances"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}
  • iam:GetUser, iam:CreateAccessKey, and iam:DeleteAccessKey, configured to the IAM user to allow credential rotation. Example policy:
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "iam:DeleteAccessKey",
        "iam:GetUser",
        "iam:CreateAccessKey"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:iam::123456789012:user/JohnDoe"
    }
  ]
}

Attributes and Secrets

Host Catalog

The following attributes are valid on an AWS host catalog resource:

  • disable_credential_rotation (bool): If true, credential rotation will not be performed. See the Credential Rotation readme for more information.
  • region (string): The region to configure the host catalog for. All host sets in this catalog will be configured for this region.
  • role_arn (string): The AWS Role ARN used for AssumeRole authentication. If provided, disable_credential_rotation must be also set to true.
  • role_external_id (string): The external ID configured for the principal AssumeRole provider.
  • role_session_name (string): The session name configured for the principal AssumeRole provider.
  • role_tags (object): The key-value pair tags configured for the principal AssumeRole provider.
  • target_role_arn (string): The AWS role ARN of the target IAM role to assume for cross-account discovery. Required if any other target_* attributes are configured.
  • target_region (string): The region where the target hosts reside. Defaults to the principal account region if omitted.
  • target_role_external_id (string): The external ID configured for the target AssumeRole provider.
  • target_role_session_name (string): The session name configured for the target AssumeRole provider.
  • target_role_tags (object): The key-value pair tags configured for the target AssumeRole provider.

The following secrets are required on an AWS host catalog resource:

  • access_key_id (string): The access key ID for the IAM user to use with this host catalog.
  • secret_access_key (string): The secret access key for the IAM user to use with this host catalog.

See the Credential Rotation readme for more information.

Host Set

The following attributes are valid on an AWS host Set resource:

  • filters (array of strings): An array of string filters in the format key=val1,val2. The key corresponds to a filter option, and the value(s) are a comma-separated list. For a list of filter options, check out describe-instances in the AWS CLI reference.