**Problem**
* While adding e2e tests for AssumeRole functionality, it was noticed that some of the static credential e2e tests were flaky.
* The tests would sometimes fail with an AuthFailure.
**Solution**
* It was discovered that this error happened sometimes after a static credential rotation and that AuthFailure occurs if EC2 is invoked with the wrong credentials.
* Due to IAM's eventual consistency model, it can take some time (up to several seconds) for changes, including rotated credentials, to get propagated to all AWS services, including EC2.
* The tests were intermittently failing because sometimes the credentials would be propagated in time before invoking EC2 and other times they were not propagated in time.
* The fix is to add a retry mechanism when calling DescribeInstances. A 30 second window is defined in which the retries can take place.
* The retries happen only for AuthFailures. Any other failure will not trigger a retry.
**Testing**
* After making this change, the intermittent failures were no longer seen for the e2e tests
* The e2e tests cover updating/creating catalogs and sets as well us updating the host list. All pass.
* New unit tests were added to test the retry mechanism. Support had to be added to mock a series of DescribeInstances calls vs a single call
* New and existing unit tests passed
* feat: parse generic aws api errors
* feat: parse s3 api errors
* feat: return credential state for all status errors
* fix status codes on certain http error codes and re-enable two tests
* switch code to FailedPrecondition
* update certain error codes
* update go.mod for new sdk version
---------
Co-authored-by: Damian Debkowski <damian.debkowski@hashicorp.com>
This commit fixes a problem where the plugin/AWS SDK would not recognise
static AWS credentials set in the machine's environment variables (by
setting AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY)
The default `WithSharedCredentials` value is `true` in awsutil (v2) -
This essentially disables the ability to read credentials from the
environment because we set the `default` AWS profile options, making the
SDK go search for a profile that may not exist and ignoring the
environment in the way, so we need to set it to `false` (see
`awsutil@v2.(*CredentialsConfig).generateAwsConfigOptions`) for extra
details.
* fix(storage): Storage Assume Role Support
This commit introduces fixes to enable operators to use AWS dynamic
credentials on the storage side of this plugin, namely validation,
improved credential lifecycle management to allow for storage buckets to
be updated from static to dynamic credentials and vice versa, and a new
way to determine credential types.
It also introduces new tests for dynamic credential usage for the
various functions.
* fix(storage): Static rotated creds being deleted on plugin err
On Storage Bucket update, when we determine if the credentials are to be
updated to new ones, we call `credState.ReplaceCreds`.
This call, if the old credentials were managed by us (rotated at create
time), will delete the old credentials and set the new ones onto
credState.
We called this function without validating if the incoming credentials
are valid. This could get the bucket in a state where where, while we'd
return an update error if the incoming credentials weren't working with
S3 (err in the dry run process that happens at the end of the update),
we'd also silently delete the old credentials the bucket used, rendering
the bucket completely broken.
---------
Co-authored-by: Hugo <hugoamvieira@users.noreply.github.com>
* feat: assume role provider credentials
This commit updates the awsutil dependency to its latest version, which
supports the capability of using dynamic credentials. The storage service
was refactored to make static credentials optional. The credentials pkg
was refactored to support several attribute fields used for the assume
role provider. The host service still enforces the use of static
credentials.
Matt Romanelli
Ryan Flynn
aditya2548
hashicorp-copywrite[bot]
oss-core-libraries-dashboard[bot]andhashicorp-copywrite[bot]
Emilia
Damian Debkowski
Louis Ruch
Hugo
daniandDamian Debkowski
Hugo
