feat(config): Support multiple files
This updates the three subcommands that take a `-config` flag to
to support multiple config files via a repeated `-config` flag.
It updates the following subcommands:
- `server`
- `database init`
- `database migrate`
Note that the multiple file support only works if all config files are
HCL, not JSON. Also note that the config files are concatenated
together. There is no attempt to merge config files, so having the same
stanza in multiple files can yield unexpected results, with only one of
the stanzas being used.
When used with these limitations understood, this provides a way to take
different actions for different kinds of dynamic configuration changes.
An example would be when using consul-template to render the
configuration files for boundary. With the multi file support
consul-template can render one config file for the controller stanza,
and provide boundary with a dynamic database credential via vault. Then
when the credential is rotated, consul-template can send boundary a
SIGHUP to switch to the new credential without a restart. A separate
template could be used to render a config for a stanza that does not
support SIGHUP, such as the events stanza that gets its values via
consul's KV. If these values change, consul-template can restart
boundary so the new configuration is used.
Co-authored-by: Timothy Messier <tim.messier@gmail.com>
Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>
* Return worker process information in the health endpoint.
Experimental: The worker_info and returned worker_process_info fields are experimental and can change or be removed without notice.
When a user passes in "worker_info=1" or "worker_info=true" and a worker is running the returned json will be returned with a key named "worker_process_info" which value is a json object with information about the worker process.
If the database url is updated on Boundary's config and then a SIGHUP
triggered, we now connect to the new database, ensure it's OK to use
and use that one going forwards instead.
This allows a controller to be deployed in a more dynamic infrastructure
where, for example, database credentials could change, without having
to restart the controller. In this case only a database url config
change and a SIGHUP signal to all controllers would be required.
This uses an atomic.Pointer to store wrapped DB objects. This allows
atomically swapping to the new object. A timeout is introduced so that
if the object is swapped after the existing one has been looked up it
isn't immediately invalidated. The timeout is a maximum; a ticker once a
second that looks at stats to determine how many database connections
are still in use and then closes the driver as soon as none are in use.
This could be done with mutexes to avoid checking stats in a loop, but
that carries its own risk: if a write lock is attempted to be acquired
while a query is holding a read lock, the write lock will block, and any
subsequent read locks will also block since write locks take priority.
As a result, if a query is long-running or timing out, all other queries
in the system will become blocked. Our default user request timeout is
90 seconds and some jobs have no timeout at all, so this is not a purely
theoretical concern, even if it's unlikely.
Refs: #2435
Co-authored-by: Jeff Mitchell <jeffrey.mitchell@gmail.com>
This commit extracts 3 separate pieces of functionality into their own
function/method:
- Extracts the database opening with the intention to have a function
that performs all that functionality but returns the created database
object rather than setting it to the Server structure.
- Extracts database validation logic, to provide a common code path for
upcoming functionality.
- Expose schema manager on the Command struct as well as providing
common functions to acquire shared lock/close.
This commit introduces closing the Schema Manager's underlying driver
connection. We were only releasing the shared lock but not actually
cleaning up the db connection and relied on the fact that the process
would exit to do that for us.
In cases where the server was started with both a controller and worker
stanza, the reload of the config was not properly setting the initial
upstream value on the new config. In this case the value is derived from
other config values and not necessarily set directly.
This now properly initializes the new config so that initial upstream is
set in the same manner as the first load.
Blame: 5d0cdf680f
This adds an action to the API that allows pulling out an activation
token that can be given to a worker in its config (directly or via env
or file). The worker can pass this along with its normal request in
place of its normal nonce and the server will automatically accept it.
Co-authored-by: Irena Rindos <irenarindos@users.noreply.github.com>
On a SIGHUP, if the worker's `initial_upstreams` config option has
changed, it will use the new values as the address to send its status.
Any previously discovered upstream addresses from previous status
requests are dropped. Once a successful status request is made to the
new addresses, the worker will resume its normal behavior of using the
addresses from the response to the status request.
This allows a worker to be deployed in a more stable way in a dynamic
environment where controller IPs could change. In particular this helps
with a case where all controllers are replaced with new controllers
running on new IPs.
To illustrate with an example, suppose there is an instance group of
controllers instances. And there are worker instances that have their
config rendered from consul-template that sets `initial_upstreams` to
the IPs of the controller instances. If all controller instances are
restarted or replaced at the same time and assigned new IPs, a new
config will be rendered for the worker. Previously the only way for the
worker to use the new value would be for the worker process to be
restarted. Otherwise the worker would only know about the previous IPs
and would fail to connect to the new controllers. However, a restart
would mean any active sessions and session connections on the worker
instance would be terminated. With this update, consul-template can send
a SIGHUP to the worker, the worker will connect to the new controllers
and resume normal status reporting, allowing the sessions and session
connections to remain.
The same principles would apply for other examples of dynamic
environments such as nomad and kubernetes.
See:
https://github.com/hashicorp/consul-templatehttps://www.nomadproject.io/docs/job-specification/template
* Have fallback eventing use json format
The server will initialize a logger via InitFallbackLogger(...)
after setting up the server's logging (and honoring the related
cmd flags). This fallback logger will be used if when no eventer
is initialized.
If no fallback logger is initialized, then a event.defaultLogger() will
be used if no eventer is defined.
Co-authored-by: Jim <jlambert@hashicorp.com>
This adds two new config options for the `database` stanza:
- `max_idle_connections`
- `max_idle_time`
These correspond to settings for the underlying `sql.DB` instance. They
can be used to have finer control over the controller's database
connection pool. For example this can be used to mitigate the high
connection overhead when connecting to postgres by setting a high
`max_idle_connections` and either setting `max_idle_time` to zero or to
a high duration.
Alternatively, if using a connection pool like pgpool-II or pgbouncer,
`max_idle_connections` can be set to 0.
See:
https://pkg.go.dev/database/sql#DB.SetMaxIdleConnshttps://pkg.go.dev/database/sql#DB.SetConnMaxIdleTimehttps://www.pgpool.net/mediawiki/index.php/Main_Pagehttps://www.pgbouncer.org/
This essentially worked for non-dev mode before but would create storage
directories that weren't actually used or needed, and it didn't work in
dev mode because it made assumptions about the presence of a worker auth
kms or not. This fixes those behaviors.
* Don't automatically create worker names, as this is now invalid for
PKI workers
* Fix a check in dev that clears out name/description in exactly the
wrong cases
* Change eventing setup to use a hostname-based ID instead of the name
(which could change on every start anyways)
* Adds WorkerAuthStorage wrapper for worker credential encryption using nodeenrollment library.
* Adds AuthStoragePath field in worker config and dev mode to specify directory to store credentials.
* Remove the server_id from session table, change it to worker_id on session_connection
This is step 1 in updating the worker table. The worker is becoming a public resource so it must have
a public id. We drop the server_id column on the session_connection table and add the worker_id column
so we can use the wt_public_key type. Since db migrations are applied while the controller is offline,
it should be safe to assume all session connections are not in a running state, thus allowing us to null
out all the worker ids in the session_connection table.
For now we don't have very good enforcement of the wt_public_id contraints, since the id is currently
defined in the worker config, but this will soon change and the worker's id will be generated by the
controller.
* Updating server_id references to controller_id.
* Adding worker tag db writing test.
* Move the server table back to the top.
* Update comment for the worker object.
* Update comment for the worker table.
* Update the newWorkerId call to take a context and fix its error.
* Correct comment for the wt_network_address domain.
* Remove the server_id from session table, change it to worker_id on session_connection
This is step 1 in updating the worker table. The worker is becoming a public resource so it must have
a public id. We drop the server_id column on the session_connection table and add the worker_id column
so we can use the wt_public_key type. Since db migrations are applied while the controller is offline,
it should be safe to assume all session connections are not in a running state, thus allowing us to null
out all the worker ids in the session_connection table.
For now we don't have very good enforcement of the wt_public_id constraints, since the id is currently
defined in the worker config, but this will soon change and the worker's id will be generated by the
controller.
* feature (db): add Db.UnderlyingDB()
This is needed to allow for integration with bits that need a *dbw.DB
* feature (kms): add new go-kms-wrapping/extras/kms dependency
* refactor (kms): schema changes required to adopt extras/kms pkg
* refactor (kms/iam): rename to DeprecateCreateKeysTx(...)
* refactor (kms): remove deprecated test
* refactor (kms): refactor Kms type
This refactor includes renaming the existing type to DeprecatedKms
and then refactoring the Kms type to become a wrapper of
go-kms-wrapping/extras/kms
* refactor (kms): delete all the unneeded bits
All this stuff is not needed now that we're using
go-kms-wrapping/extras/kms
* refactor (db): fixup tests after kms refactor
* refactor (kms/tests): use testing.TB for test helpers
Our test helper packages were not possible to use
with benchmarks. This refactor switches to testing.TB
to allow both tests and benchmarks to use the helpers.
* tests (session benchmarks): refactor to use new Kms type
* refactor (schema): rm test which relies on deprecated Kms code
* refactor (kms): post rebase on main changes required
* chore (kms refactor): re-add unit test from main
* refactor (kms): uses migrations that don't include KMS schema changes
* Move the server daemons into their own package
With this change the internal/server package now is dedicated to the server domain (workers and controllers) and not to the server daemons.
The hope is that this will make it easier to keep the daemons decoupled from the representation of the servers in the database.
It is documented as illegal to use t.FailNow in a goroutine.
Uses of require assertions call t.FailNow. This may have led to
inconsistent test results as some assertions were subject to
undefined behavior. Note that it is still allowed to call
assert assertions in goroutines.
See: https://pkg.go.dev/testing#T.FailNow
* feat(controller): New health service and server
This change also implements a new listener purpose, `ops`.
`ops` (short for operations) is a new listener purpose that is
responsible for serving Boundary's operations endpoints, like health
and possibly metrics.
At Startup: `ops` listeners get configured as late as possible to catch
any errors with the startup process;
At Shutdown: If the health endpoint has been set-up, we introduce a
configurable delay in the process. This is to enable the health endpoint
to report unhealthy for that amount of time. `ops` listeners are also
the last listeners to be shutdown.
In general, these changes were introduced in the same vein of the
changes to controller `startListeners`.
Specifically, these changes also decouple the Controller and the Worker
shutdown processes. Currently, a call to `stopListeners` on an instance
that is running as both a Controller and a Worker would stop the
listeners for both.
Lastly, we also add unit/integration tests to the shutdown code.
This contains to related things:
1. An adapter allowing an hclog interface to take in information and send them to the eventer
2. A gate on the eventer, which is the analogue to the behavior prior to eventing where we'd ensure logs were not printed prior to the startup information. Without this, plugin logs using the hclog adapter will start showing up too early.
Timothy MessierandJeff Mitchell
Todd
Irena Rindos
Hugo VieiraandJeff Mitchell
Louis Ruch
Haotian
irenarindos
Johan Brandhorst-Satzkorn