* temporary commit
* feat(host): replace dryRunValidation with checkHosts in catalog and set handlers
* refactor(host): clean up comments and fix typos in dry-run validation blocks
* refactor(host): address PR feedback on dry-run validation blocks
* refactor(host): remove dead dryRunValidation function and its tests
* refactor(host): extract dry run error message as a constant
* refactor(host): propagate BadRequestStatusf from buildFilters through buildDescribeInstancesInput
---------
Co-authored-by: ryanflynn9299 <ryan.flynn001@gmail.com>
This changeset implements multipart uploads. AWS caps PutObject calls at 5
GiB, effectively causing the plugin to error on any upload larger than that.
Multipart limit is 8 TiB, which should be plenty for most use cases.
Multipart implementation is size-gated: we only switch to multipart uploads once
the file size is greater than multipartThreshold, which is currently set to 8
MiB, and we set the part size to 8 MiB as well (this follows AWS CLI and boto3).
That gives us ~78 GiB max object before we auto-scale, due to 10,000 part cap on
AWS.
---------
Co-authored-by: Hugo <10965479+hugoghx@users.noreply.github.com>
This commit also introduces a new dry run validation step when creating
or updating a host catalog. Additionally, it introduces new tests
related to using AssumeRole credentials, as well as more error and
success state cases.
service/host/plugin.go now closely follows service/storage/plugin.go.
This is by design as to bring both "halves" of the AWS plugin into
parity with each other.
* fix: Append list error for permission validation
Instead of returning the error right away, append the error when `ListObject` is successful but the key expected if not found. Set the `Read` permission state as `Unknown` if the response from the list calls does not find the expected key.
This can be caused if the credentials do not have `s3:PutObject` but have `s3:ListBucket`. Without `s3:PutObject`, the object will not be written to S3 so other calls to find the object will not be successful even if they have valid permissions.
This prevents us from returning a `write` error on Boundary because the list call will always fail. Without this, the Permissions are not wrapped and the SBC states are all set to `unknown` even though we know certain permission failed.
* feat: parse generic aws api errors
* feat: parse s3 api errors
* feat: return credential state for all status errors
* fix status codes on certain http error codes and re-enable two tests
* switch code to FailedPrecondition
* update certain error codes
* update go.mod for new sdk version
---------
Co-authored-by: Damian Debkowski <damian.debkowski@hashicorp.com>
This commit fixes a problem where the plugin/AWS SDK would not recognise
static AWS credentials set in the machine's environment variables (by
setting AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY)
The default `WithSharedCredentials` value is `true` in awsutil (v2) -
This essentially disables the ability to read credentials from the
environment because we set the `default` AWS profile options, making the
SDK go search for a profile that may not exist and ignoring the
environment in the way, so we need to set it to `false` (see
`awsutil@v2.(*CredentialsConfig).generateAwsConfigOptions`) for extra
details.
* fix(storage): Storage Assume Role Support
This commit introduces fixes to enable operators to use AWS dynamic
credentials on the storage side of this plugin, namely validation,
improved credential lifecycle management to allow for storage buckets to
be updated from static to dynamic credentials and vice versa, and a new
way to determine credential types.
It also introduces new tests for dynamic credential usage for the
various functions.
* fix(storage): Static rotated creds being deleted on plugin err
On Storage Bucket update, when we determine if the credentials are to be
updated to new ones, we call `credState.ReplaceCreds`.
This call, if the old credentials were managed by us (rotated at create
time), will delete the old credentials and set the new ones onto
credState.
We called this function without validating if the incoming credentials
are valid. This could get the bucket in a state where where, while we'd
return an update error if the incoming credentials weren't working with
S3 (err in the dry run process that happens at the end of the update),
we'd also silently delete the old credentials the bucket used, rendering
the bucket completely broken.
---------
Co-authored-by: Hugo <hugoamvieira@users.noreply.github.com>
* feat: assume role provider credentials
This commit updates the awsutil dependency to its latest version, which
supports the capability of using dynamic credentials. The storage service
was refactored to make static credentials optional. The credentials pkg
was refactored to support several attribute fields used for the assume
role provider. The host service still enforces the use of static
credentials.
Delaine Rogers
Ryan Flynn
rebecca-yohandryanflynn9299
Matt Romanelli
richardtrinh1
dependabot[bot]
Susmitha Girumala
Piotr KazmierczakandHugo
Damian Debkowski
Louis Ruch
aditya2548
hashicorp-copywrite[bot]
oss-core-libraries-dashboard[bot]andhashicorp-copywrite[bot]
Emilia
Katrina Schwark
daniandDanielle Miu
Irena Rindos
Johan Brandhorst-Satzkorn
Elim Tsiagbey
Michael Li
Hugo