This commit extracts 3 separate pieces of functionality into their own
function/method:
- Extracts the database opening with the intention to have a function
that performs all that functionality but returns the created database
object rather than setting it to the Server structure.
- Extracts database validation logic, to provide a common code path for
upcoming functionality.
- Expose schema manager on the Command struct as well as providing
common functions to acquire shared lock/close.
In cases where the server was started with both a controller and worker
stanza, the reload of the config was not properly setting the initial
upstream value on the new config. In this case the value is derived from
other config values and not necessarily set directly.
This now properly initializes the new config so that initial upstream is
set in the same manner as the first load.
Blame: 5d0cdf680f
* adds CLI commands for add/set/removing worker tags, and respective tests
* adds StringSliceMapVar Flag to convert a string from the CLI to a map[string][]string value
This allows a check function to be set to determine whether it is
allowed to be set. This also ensures that if null is allowed it is
either the only value or not set.
For a flag type used for attributes/secrets, values have always been
able to be used with env:// or file:// format, but the error check to
see whether the value was successfully parsed was doing a simple string
equality rather than `errors.Is`. As a result a value containing a colon
would trip up this check and cause an error, rather than allowing it to
be used as a bare value.
* Revert "Remove `-token` flag (#2303)"
This reverts commit 65cff995d7.
* Change behavior of -token
* Add use of MustParsePath
* Update changelog
* Use errors.Is
* Have fallback eventing use json format
The server will initialize a logger via InitFallbackLogger(...)
after setting up the server's logging (and honoring the related
cmd flags). This fallback logger will be used if when no eventer
is initialized.
If no fallback logger is initialized, then a event.defaultLogger() will
be used if no eventer is defined.
Co-authored-by: Jim <jlambert@hashicorp.com>
This adds two new config options for the `database` stanza:
- `max_idle_connections`
- `max_idle_time`
These correspond to settings for the underlying `sql.DB` instance. They
can be used to have finer control over the controller's database
connection pool. For example this can be used to mitigate the high
connection overhead when connecting to postgres by setting a high
`max_idle_connections` and either setting `max_idle_time` to zero or to
a high duration.
Alternatively, if using a connection pool like pgpool-II or pgbouncer,
`max_idle_connections` can be set to 0.
See:
https://pkg.go.dev/database/sql#DB.SetMaxIdleConnshttps://pkg.go.dev/database/sql#DB.SetConnMaxIdleTimehttps://www.pgpool.net/mediawiki/index.php/Main_Pagehttps://www.pgbouncer.org/
This essentially worked for non-dev mode before but would create storage
directories that weren't actually used or needed, and it didn't work in
dev mode because it made assumptions about the presence of a worker auth
kms or not. This fixes those behaviors.
* fix: Propagate plugin options to CreatePlugin
Plugin options were previously only applied to the
BuildPluginMap call, resulting in the plugin
execution_dir variable being ineffective.
Also add a test to verify functionality
* fix: Support plugin execution_dir for all plugins
Previously, only external host plugins would respect
the plugin execution dir. Now, all uses of the embedded
plugins respect the configuration.
* Adds WorkerAuthStorage wrapper for worker credential encryption using nodeenrollment library.
* Adds AuthStoragePath field in worker config and dev mode to specify directory to store credentials.
* feature (db): add Db.UnderlyingDB()
This is needed to allow for integration with bits that need a *dbw.DB
* feature (kms): add new go-kms-wrapping/extras/kms dependency
* refactor (kms): schema changes required to adopt extras/kms pkg
* refactor (kms/iam): rename to DeprecateCreateKeysTx(...)
* refactor (kms): remove deprecated test
* refactor (kms): refactor Kms type
This refactor includes renaming the existing type to DeprecatedKms
and then refactoring the Kms type to become a wrapper of
go-kms-wrapping/extras/kms
* refactor (kms): delete all the unneeded bits
All this stuff is not needed now that we're using
go-kms-wrapping/extras/kms
* refactor (db): fixup tests after kms refactor
* refactor (kms/tests): use testing.TB for test helpers
Our test helper packages were not possible to use
with benchmarks. This refactor switches to testing.TB
to allow both tests and benchmarks to use the helpers.
* tests (session benchmarks): refactor to use new Kms type
* refactor (schema): rm test which relies on deprecated Kms code
* refactor (kms): post rebase on main changes required
* chore (kms refactor): re-add unit test from main
* refactor (kms): uses migrations that don't include KMS schema changes
* Move the server daemons into their own package
With this change the internal/server package now is dedicated to the server domain (workers and controllers) and not to the server daemons.
The hope is that this will make it easier to keep the daemons decoupled from the representation of the servers in the database.
* fix(cmd): Add profiling build imports
It seems like this file has been bitrotted since last used,
and it was missing some imports. This was discovered
by running goimports on the code.
* Add build-memprof Makefile target
The wrapper is sometimes unset at the time when the
audit config is validated, so it is impossible for us
to validate that a valid wrapper is configured. The wrapper
is configured by later setup steps because of a dependency
on the eventer in our KMS setup.
Since we call base.NewServer multiple times when building the Commands map this was returning
an error "duplicate metrics collector registration attempted". This PR only creates a single
server which is used for both the "server" command as well as "dev".
Adds a Prometheus gauge metric set to the value of 1, with labels for go version, git revision, and Boundary version.
Assigns the Prometheus DefaultRegisterer to the Server struct as a parameter to NewServer(), to avoid "duplicate metrics collector registration" panics from test cases.
We never ended up using this and it makes it much harder to reason about the listeners.
An explanation: for the moment I separated out all of the listeners in the struct, even though one should suffice, because when I did it that way first I was hitting some really weird and hard to find intermittent behavior where it seemed like the listeners were getting crossed. So I re-did it this way with each listener very explicitly referenced. I'm happy to make a follow-up PR that attempts to combine them all into a single listener without hitting the issues I did, if desired, but at least doing it via a separate PR means that it won't be something with removing the ALPN muxer that is at fault.
* feat(controller): New health service and server
This change also implements a new listener purpose, `ops`.
`ops` (short for operations) is a new listener purpose that is
responsible for serving Boundary's operations endpoints, like health
and possibly metrics.
At Startup: `ops` listeners get configured as late as possible to catch
any errors with the startup process;
At Shutdown: If the health endpoint has been set-up, we introduce a
configurable delay in the process. This is to enable the health endpoint
to report unhealthy for that amount of time. `ops` listeners are also
the last listeners to be shutdown.
This contains to related things:
1. An adapter allowing an hclog interface to take in information and send them to the eventer
2. A gate on the eventer, which is the analogue to the behavior prior to eventing where we'd ensure logs were not printed prior to the startup information. Without this, plugin logs using the hclog adapter will start showing up too early.
This adds a new requirement onto subtypes to register a NewFunc.
Previously the subtypes where implementing this function anyway, but the
target package was not including them in the registry. This resulted in
packages that use the target package needing to import the subtypes
directly, which we want to avoid.
This refactor will require the subtypes to register their NewFunc which
can then be used by `target.New` so packages that need to create targets
can do so by only using the target package.
This also updates the tcp subpackage to register a NewFunc.
Adds tests to `SetupControllerPublicClusterAddress` and
`SetupWorkerPublicAddress` ahead of changes in these functions to
ensure functionality is preserved.
The previous PR only initialized plugins if they were required, which
meant that there may not actually be a plugin manager running because a
second node would not actually create the dev database, which is where
the plugin was being added. The assumption that a plugin manager is
running is still a fine one at the moment, but we just need to make sure
it's running in _all_ cases.
* feature (db): Add the ability to enable debug output for a connection
* refactor: Use scope.Global instead of "global"
* feature (db/migrations): Add kms audit keys
Add schema changes to support a new DEK for audit crypto operations.
* feature (kms/audit): Add audit key type
All the bits required to define a new DEK concrete type for
operating on audit entries.
* refactor (db): Remove deprecated migration unit tests.
We added a new audit DEK to kms.CreateKeysTx(...). Unfortunately,
unit tests like TestMigrations_UserDimension,
Test_PrimaryAuthMethodChanges and TestMigrations_CredentialDimension
have an indirect dependency on a specific historical set of DEKs via
an indirect call kms.CreateKeysTx. This of course fails because the
new DEK's tables is not a part of pinned migration versions that these
tests depend on.
For the target Repository's Create method, the options of
WithHostSources and WithCredentialSources where never used by the
targets.Service. As such the repository does not need to support these
options.
See: 51812ff779
Hugo Vieira
Timothy Messier
Jeff Mitchell
Haotian
Johan Brandhorst-Satzkorn
Damian Debkowski
Todd
irenarindos