0%

0000000

0x00

Vault Secret Rotation Fails in the Application, Not Vault

Abdullah ChaudaryInfrastructure11 min read

Black card: Rotated on time. Read once. A gradient credential file feeds an empty grid beside the errors and 0 queries in 79 s. Looping GIF.

HashiCorp Vault secret rotation causes application downtime when a worker reads its credential once. Static roles do not overlap, and three fixes have measured costs in a Vault 2.1.2 sandbox.

At 16:20:03 UTC on 14 October 2026, my sandbox worker began answering every query with permission denied for table orders. The grants were untouched. Vault had rotated the credential on schedule, and a valid new one had been sitting in /secrets/db.json for about four seconds. The worker had read that file once, at startup. That is HashiCorp Vault secret rotation causing application downtime: Vault did its job and the consumer did not.

I run Vault Agent sidecars on self-hosted and cloud deployments, rotate all of the secrets, and have never had a rotation incident. A clean record does not tell me where rotation breaks, so I built a sandbox (Vault v2.1.2, PostgreSQL 17.11, Python 3.13.16 with psycopg_pool 3.3.3, Docker Compose) and broke it on purpose. Every figure below comes from that sandbox, none from client work (under NDA), and I have no production incident to report. Vault's place in a self-hosted stack is one line in the self-hosted security baseline that includes Vault; this post covers what happens after the secret changes.

TL;DR: HashiCorp Vault secret rotation causes application downtime when a process reads its credential once. Vault revoked on schedule and the worker had 0 successful queries for 79 s. Static roles do not overlap: the old password was refused 0.10 s after rotation. Make the worker re-read on change or on SIGHUP, then prove revocation with four checks.

Why Does HashiCorp Vault Secret Rotation Cause Application Downtime When Vault Rotated Correctly?

Vault controls how long a credential lives, not what the process holding it does next. In the sandbox the agent rendered a new credential at 16:19:59.394 and Vault revoked the old lease at 16:20:03.460, 4.07 s later. The worker had loaded the first credential at startup, so it failed from its next 2 s tick and never recovered.

The setup was one long-running Python worker and a Vault Agent sidecar sharing a volume. The agent renders database/creds/dyn to /secrets/db.json; the role has default_ttl=30s, max_ttl=60s and no revocation_statements. Every 2 s the worker runs three probes: a query through a ConnectionPool (max_lifetime=15), a query on one connection opened at startup, and a fresh psycopg.connect(). The worker log around the revoke:

2026-10-14T16:20:01.691Z [naive] pool-ok user=v-approle-dyn-cRCMiIUngA4nwhoZkhSe-1791994743 backend=3524
2026-10-14T16:20:03.764Z [naive] pool-FAIL err=InsufficientPrivilege msg='permission denied for table orders'
2026-10-14T16:20:18.976Z [naive] pool-FAIL err=PoolTimeout msg="couldn't get a connection after 5.00 sec"

One cause produced three different messages, and none of them says "credential revoked".

  • permission denied for table orders. Default revocation dropped the role but left its sessions open, so they lost the privileges held through app_read. The error reads as a grants bug, and the grants were fine.
  • FATAL: password authentication failed. New connections (the fresh probe, 28 ms later) were refused. The client message is identical for a wrong password and a dropped role. Only the Postgres server log says DETAIL: Role "v-approle-dyn-cRCMiIUngA4nwhoZkhSe-1791994743" does not exist.
  • PoolTimeout. For 15 s the pool kept handing out its old, privilege-less connections. When they hit max_lifetime the pool could not replace them, and the caller saw a timeout. The authentication failure appeared only in the pool library's WARNING log.

Measured against Vault's revoked lease line: first failure +0.30 s, first PoolTimeout +15.5 s, 15 of 15 pool probes failed, and 0 probes of any kind succeeded for the remaining 79 s. The shape matches a deploy that succeeded while the old version kept running: the platform reports done and the running process disagrees.

How Do You Find Every Consumer of a Secret Before You Rotate It?

Start from the database, because the database sees every client. In the sandbox, pg_stat_activity listed each session by role and backend pid, and vault list sys/leases/lookup/database/creds/dyn listed live leases. Neither names an application process, so the join is manual: lease to role, role to session, session to service.

My monitor.sh printed that join every 5 s. At 16:20:07.144, after the revoke, it showed one live lease and sessions=[<null>(usesysid=16423,pid=3245) <null>(usesysid=16423,pid=3524) <null>(usesysid=16423,pid=3547)]. The three <null> sessions were the worker's. A consumer that cached a credential stays in the session list long after its lease has left the lease list.

Two other things matter. The lease docs say "When a token is revoked, Vault will revoke all leases that were created using that token", so a consumer's auth token carries every credential it fetched. And a credential file counts as a consumer. My first two runs of the experiment started the worker before clearing the shared volume. It read the file the previous agent had left behind, 0.34 s before the new agent's first render, and that lease had been revoked 8.5 s earlier. The worker failed from its first second and logged 0 successful probes in 146 s. A credential file that outlives its agent is a credential with no lease behind it.

Audit devices, config search and scheduled jobs are other inventory sources; I did not exercise them and make no claim about them. Finding secrets that leaked into a repository is a scanner's job, and what a security gate should block is a separate question from rotating live ones.

Do Vault Static Roles Overlap the Old and New Password?

No. In the sandbox, the first login with the old password after rotate-role was refused 0.10 s after Vault recorded the rotation, and every attempt for the next 25 s was refused. Sessions opened before the rotation kept working: one ran 98 queries with 0 failures through it. New connections get no grace period.

A HashiCorp forum thread asked this on 2020-09-23 ("There is no 'buffer' time or not overlap between the old and new password timelines") and was re-asked on 2022-02-10. The page I read shows no answer. The database secrets engine page describes static roles as "a 1-to-1 mapping of Vault roles to usernames in a database".

The probe held one session open and tried a fresh login with the old password every 0.25 s; I forced the rotation six seconds in.

$ vault write -f database/rotate-role/static-blue
Success! Data written to: database/rotate-role/static-blue

The timeline, from the probe log and vault read database/static-creds/static-blue:

  1. 16:35:45.385: last successful login with the old password.
  2. 16:35:45.568: last_vault_rotation.
  3. 16:35:45.673: first refusal, FATAL: password authentication failed for user "app_static_blue".

The probe interval of 0.25 s is the resolution of this result, and it is one manual rotate-role, not a scheduled rotation_period rotation. I did not observe a scheduled one.

A second finding: creating a static role is itself a rotation. app_static_blue was created with the password initial-blue-pw. After vault write database/static-roles/static-blue ... that password was refused, and the role showed skip_import_rotation false. The API docs mark skip_import_rotation as an Enterprise parameter, and I did not try setting it to true.

Dynamic roles overlap, but the window is not a setting. The agent renewed each lease to the max_ttl cap and then fetched a new credential. Across 17 rotations in six runs, the time between "new credential rendered" and "old lease revoked" was 15.98 to 17.04 s in 14 cases and 4.02 to 5.11 s in 3. The short ones followed a second, capped renewal (TTL of "30s" exceeded the effective max_ttl of "4s"). A consumer gets 4 to 17 s, and nothing in the role configuration says which.

Does the Application Need a Restart or a Reload When a Secret Changes?

Reload if you can write the handler, restart if you cannot. In the sandbox, an agent SIGHUP and a file-change check each ran 222 probes with 0 failures. The process supervisor also showed 0, but it restarts the process on every rotation, and its 0 is a best case. I ran each fix against role dyn for 150 s.

  1. Template exec sends SIGHUP; the worker rebuilds its pool. 0 failed probes of 222. Render to rebuilt pool took 0.27 to 1.57 s across 4 renders, bounded by the 2 s tick. The rebuild itself took 22 to 32 ms. The cost is a reload handler in the application and a way to signal it (here a shared PID namespace). The template stanza gains:
   exec {
     command = ["sh", "-c", "... kill -HUP $(cat /secrets/worker.pid)"]
     timeout = "10s"
   }

The Agent docs say the exec block "executes a command when the template is rendered and the output has changed".

  1. The worker re-reads the file when its mtime changes. 0 failed probes of 222. Render to rebuilt pool took 1.29 to 1.93 s across 3 renders. The cost is application code only, with no signal and no agent change; the poll interval sets the delay. The check in the worker's loop:
   m = os.stat(CRED_FILE).st_mtime
   if m != S.mtime:
       S.build("file-changed")
  1. Vault Agent process supervisor with restart_on_secret_changes = "always". 0 failed probes, but every rotation is a full process restart. Render to SIGTERM took 2.00 to 2.02 s, a fixed delay seen on every rotation. SIGTERM to a new Python process took 0.49 to 0.53 s with no process serving, and render to first query on the new credential took 2.53 to 2.60 s. With max_ttl=60s that meant one restart every 43 s. My worker exits on SIGTERM in about 1 ms and was idle, so nothing was in flight. The docs give the child 30 s after SIGTERM before SIGKILL; I did not observe that. The supervisor "requires at least one env_template block and exactly one top level exec block" and "is incompatible with regular file template entries".

A fourth option, re-reading only on an authentication failure, was not enough alone. I ran vault lease revoke -prefix database/creds/dyn by hand at 16:38:20.366. The worker's re-read at 16:38:21.973 loaded the same revoked credential, because the agent had not noticed yet. The agent found out when its next renewal failed with lease not found, then rendered a new file. The gap between good queries was 4.3 s.

In What Order Should You Roll Out a Rotation?

Fix the consumers first, rotate second, retire the old credential last. Two measurements force that order: creating a static role changed the database password at once, and a dynamic credential's overlap window was 4 to 17 s. Neither leaves time to repair a consumer after rotation starts.

This is my reading of the sandbox numbers and HashiCorp's guidance, not a sequence I ran end to end:

  1. Move every consumer onto Vault-delivered credentials before creating a static role. Anything still using the user's old password breaks at the moment the role is created, as initial-blue-pw did.
  2. Ship the reload or re-read code with rotation still off.
  3. Force one rotation on one canary. vault write -f database/rotate-role/<name> for a static role, or a lease revoke for a dynamic one, with failed probes counted. The Well-Architected rotation page likewise says to update one service, confirm it connects, then continue.
  4. Enable the real rotation_period or TTL, then retire the old credential and run the revocation checks below.

How Do You Verify the Old Credential Is Really Revoked?

Run four checks and then try the credential. In the sandbox, the lease listing, the lease lookup and pg_roles all said "gone", and a login with the old credential was refused. Only a fourth query, for sessions with a NULL user name, found the connection still attached to the dropped role.

These are the commands from exp4-revocation.sh, with the docker compose -p b14demo exec -T prefixes shortened:

$ vault lease revoke database/creds/dyn/xiCMn0xxUWZIsQyxyG7cX5QN
All revocation operations queued successfully!

$ vault list sys/leases/lookup/database/creds/dyn
No value found at sys/leases/lookup/database/creds/dyn
(exit 2)

$ vault lease lookup database/creds/dyn/xiCMn0xxUWZIsQyxyG7cX5QN
error looking up lease id database/creds/dyn/xiCMn0xxUWZIsQyxyG7cX5QN: Error making API request.
...
Code: 400. Errors:
* invalid lease
(exit 2)

$ psql -c "select rolname from pg_roles where rolname = 'v-token-dyn-jp8N1shbpC7Vahqq1AYL-1791995772'"
(0 rows)

$ psql -c "select pid, usesysid, usename, backend_start, state from pg_stat_activity
           where backend_type = 'client backend' and datname = 'shop' and usename is null"
 pid  | usesysid | usename |         backend_start         | state
------+----------+---------+-------------------------------+-------
 8783 |    16461 |         | 2026-10-14 16:36:15.168997+00 | idle
(1 row)

login with the revoked credential:
psycopg.OperationalError: connection failed: connection to server at "172.21.0.2", port 5432 failed: FATAL:  password authentication failed for user "v-token-dyn-jp8N1shbpC7Vahqq1AYL-1791995772"

vault lease revoke prints "queued", not "done", so the listing and lookup after it are the real answer. The fourth query is the one that finds the leftover. A session with usename IS NULL is a connection whose role was dropped under it. With default revocation it stays connected until the client closes it, and inside it select current_user fails with invalid role OID: 16461.

To make the failure loud instead, add pg_terminate_backend to the role's revocation_statements. I ran a second role (dyn-term) with this:

SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE usename = '{{name}}'; DROP ROLE IF EXISTS "{{name}}";

Open connections were killed at the moment of revocation (FATAL: terminating connection due to administrator command) and pg_stat_activity showed sessions=[] from 16:22:35. The naive worker still failed: first error +1.36 s after the revoke, PoolTimeout from +8.4 s, 12 of 12 pool probes failed, 0 successes for 79 s. Terminating sessions makes the failure loud and immediate. It does not save a worker that never re-reads.

Limitations: What Does This Not Cover?

Nothing here is a production measurement. The sandbox compresses hours into minutes with default_ttl=30s and max_ttl=60s, so the mechanisms carry over and the windows scale. Versions: Vault v2.1.2 (built 2026-10-06), PostgreSQL 17.11, psycopg_pool 3.3.3.

  • Not tested: Kubernetes and KV v2. The Vault Agent injector, Vault Secrets Operator (rolloutRestartTargets), the CSI provider, KV v2 and static_secret_render_interval were out of scope. The sidecar here is Docker Compose with a shared volume.
  • Best case for restarts. My worker is single-threaded and idle between ticks. Requests in flight, a SIGTERM handler that drains, and the SIGKILL escalation were not tested, so "0 failed probes" in the supervisor run is a ceiling.
  • Not tested: two static roles alternating (blue/green) and scheduled rotation_period rotations. Static results come from one manual rotate-role.
  • Dev mode only. In-memory storage, no HA.
  • Revocation failures. A dynamic role that owns objects, so DROP ROLE fails, was not provoked. VALID UNTIL tracked the lease expiry in every sample.
  • One pool library. Only psycopg_pool 3.3.3. HikariCP, node-postgres and SQLAlchemy were not run.
  • One Enterprise label unchecked. The API docs mark skip_import_rotation as Enterprise. I never set it to true, so I can neither confirm nor refute that label.

The rest of the infrastructure field notes are in the category.

Vault can revoke a credential, but only the application can stop using it.

If you run Agent sidecars in front of connection pools, do you rebuild the pool on SIGHUP or on a file change, and what happens to a request that is in flight when the old credential dies? My worker was idle between ticks, so I have no number for that.

References

next up

n8n MCP Agent Workflow in Production: Where the Agent Stops