Retry faster on initial status (#2243)

Right now there is a bit of a race condition in that a status call that
indirectly tells the controller that a worker is ready to accept
sessions can have its response unable to be processed in time for an
actual session to come in from a client. It's very unlikely but
technically there.

This changes where we check if we have ever successfully authenticated
from the function called by the ticker to the ticker itself, allowing us
to reset the timer to a very short time if we have not yet
authenticated. We already have the initial timer set to 0 for "as soon
as possible" behavior, but if that initial status failed we'd wait for
the normal interval. Now we will instead continue with the "as soon as
possible" behavior, and reduce the very unlikely race condition to
extremely unlikely.

(There are more permanent fixes but those are bigger and can come
later.)

internal/daemon/worker/status.go

@@ -47,6 +47,14 @@ func (w *Worker) startStatusTicking(cancelCtx context.Context, sessionManager se
 			return
 
 		case <-timer.C:
+			// If we've never managed to successfully authenticate then we won't have
+			// any session information anyways and this will produce a ton of noise in
+			// observability, so skip calling the function and retry in a short duration
+			if !w.everAuthenticated.Load() {
+				timer.Reset(10 * time.Millisecond)
+				continue
+			}
+
 			w.sendWorkerStatus(cancelCtx, sessionManager)
 			timer.Reset(getRandomInterval())
 		}
@@ -94,14 +102,6 @@ func (w *Worker) WaitForNextSuccessfulStatusUpdate() error {
 func (w *Worker) sendWorkerStatus(cancelCtx context.Context, sessionManager session.Manager) {
 	const op = "worker.(Worker).sendWorkerStatus"
 
-	// If we've never managed to successfully authenticate then we won't have
-	// any session information anyways and this will produce a ton of noise in
-	// observability, so suppress it
-	if !w.everAuthenticated.Load() {
-		event.WriteSysEvent(cancelCtx, op, "worker is not authenticated to an upstream, not sending status")
-		return
-	}
-
 	// First send info as-is. We'll perform cleanup duties after we
 	// get cancel/job change info back.
 	var activeJobs []*pbs.JobStatus