# Vault Secret Rotation Fails in the Application, Not Vault

- Author: Abdullah Chaudary
- Category: Infrastructure
- Published: 2026-10-14T21:37:02.449Z
- Updated: 2026-10-14T21:37:02.488Z
- Reading time: 11 min
- Tags: HashiCorp Vault, Secrets Management, Secret Rotation, Dynamic Secrets, DevSecOps

> HashiCorp Vault secret rotation causes application downtime when a worker reads its credential once. Static roles do not overlap, and three fixes have measured costs in a Vault 2.1.2 sandbox.

At 16:20:03 UTC on 14 October 2026, my sandbox worker began answering every query with `permission denied for table orders`. The grants were untouched. Vault had rotated the credential on schedule, and a valid new one had been sitting in `/secrets/db.json` for about four seconds. The worker had read that file once, at startup. That is HashiCorp Vault secret rotation causing application downtime: Vault did its job and the consumer did not.

I run Vault Agent sidecars on self-hosted and cloud deployments, rotate all of the secrets, and have never had a rotation incident. A clean record does not tell me where rotation breaks, so I built a sandbox (Vault v2.1.2, PostgreSQL 17.11, Python 3.13.16 with psycopg\_pool 3.3.3, Docker Compose) and broke it on purpose. Every figure below comes from that sandbox, none from client work (under NDA), and I have no production incident to report. Vault's place in a self-hosted stack is one line in [the self-hosted security baseline that includes Vault](/blog/aws-self-hosted-migration-cost); this post covers what happens after the secret changes.

**TL;DR:** HashiCorp Vault secret rotation causes application downtime when a process reads its credential once. Vault revoked on schedule and the worker had 0 successful queries for 79 s. Static roles do not overlap: the old password was refused 0.10 s after rotation. Make the worker re-read on change or on SIGHUP, then prove revocation with four checks.

## Why Does HashiCorp Vault Secret Rotation Cause Application Downtime When Vault Rotated Correctly?

Vault controls how long a credential lives, not what the process holding it does next. In the sandbox the agent rendered a new credential at 16:19:59.394 and Vault revoked the old lease at 16:20:03.460, 4.07 s later. The worker had loaded the first credential at startup, so it failed from its next 2 s tick and never recovered.

The setup was one long-running Python worker and a Vault Agent sidecar sharing a volume. The agent renders `database/creds/dyn` to `/secrets/db.json`; the role has `default_ttl=30s`, `max_ttl=60s` and no `revocation_statements`. Every 2 s the worker runs three probes: a query through a `ConnectionPool` (`max_lifetime=15`), a query on one connection opened at startup, and a fresh `psycopg.connect()`. The worker log around the revoke:

```text
2026-10-14T16:20:01.691Z [naive] pool-ok user=v-approle-dyn-cRCMiIUngA4nwhoZkhSe-1791994743 backend=3524
2026-10-14T16:20:03.764Z [naive] pool-FAIL err=InsufficientPrivilege msg='permission denied for table orders'
2026-10-14T16:20:18.976Z [naive] pool-FAIL err=PoolTimeout msg="couldn't get a connection after 5.00 sec"
```

One cause produced three different messages, and none of them says "credential revoked".

- **permission denied for table orders****.** Default revocation dropped the role but left its sessions open, so they lost the privileges held through `app_read`. The error reads as a grants bug, and the grants were fine.
- **FATAL: password authentication failed****.** New connections (the `fresh` probe, 28 ms later) were refused. The client message is identical for a wrong password and a dropped role. Only the Postgres server log says `DETAIL:  Role "v-approle-dyn-cRCMiIUngA4nwhoZkhSe-1791994743" does not exist.`
- **PoolTimeout****.** For 15 s the pool kept handing out its old, privilege-less connections. When they hit `max_lifetime` the pool could not replace them, and the caller saw a timeout. The authentication failure appeared only in the pool library's WARNING log.

Measured against Vault's `revoked lease` line: first failure +0.30 s, first `PoolTimeout` +15.5 s, 15 of 15 pool probes failed, and 0 probes of any kind succeeded for the remaining 79 s. The shape matches [a deploy that succeeded while the old version kept running](/blog/nextjs-stale-page-after-deploy-docker-layer-cache): the platform reports done and the running process disagrees.

## How Do You Find Every Consumer of a Secret Before You Rotate It?

Start from the database, because the database sees every client. In the sandbox, `pg_stat_activity` listed each session by role and backend pid, and `vault list sys/leases/lookup/database/creds/dyn` listed live leases. Neither names an application process, so the join is manual: lease to role, role to session, session to service.

My `monitor.sh` printed that join every 5 s. At 16:20:07.144, after the revoke, it showed one live lease and `sessions=[<null>(usesysid=16423,pid=3245) <null>(usesysid=16423,pid=3524) <null>(usesysid=16423,pid=3547)]`. The three `<null>` sessions were the worker's. A consumer that cached a credential stays in the session list long after its lease has left the lease list.

Two other things matter. The lease docs say "When a token is revoked, Vault will revoke all leases that were created using that token", so a consumer's auth token carries every credential it fetched. And a credential file counts as a consumer. My first two runs of the experiment started the worker before clearing the shared volume. It read the file the previous agent had left behind, 0.34 s before the new agent's first render, and that lease had been revoked 8.5 s earlier. The worker failed from its first second and logged 0 successful probes in 146 s. A credential file that outlives its agent is a credential with no lease behind it.

Audit devices, config search and scheduled jobs are other inventory sources; I did not exercise them and make no claim about them. Finding secrets that leaked into a repository is a scanner's job, and [what a security gate should block](/blog/trivy-snyk-gitlab-ci-security-gate) is a separate question from rotating live ones.

## Do Vault Static Roles Overlap the Old and New Password?

No. In the sandbox, the first login with the old password after `rotate-role` was refused 0.10 s after Vault recorded the rotation, and every attempt for the next 25 s was refused. Sessions opened before the rotation kept working: one ran 98 queries with 0 failures through it. New connections get no grace period.

A HashiCorp forum thread asked this on 2020-09-23 ("There is no 'buffer' time or not overlap between the old and new password timelines") and was re-asked on 2022-02-10. The page I read shows no answer. The database secrets engine page describes static roles as "a 1-to-1 mapping of Vault roles to usernames in a database".

The probe held one session open and tried a fresh login with the old password every 0.25 s; I forced the rotation six seconds in.

```text
$ vault write -f database/rotate-role/static-blue
Success! Data written to: database/rotate-role/static-blue
```

The timeline, from the probe log and `vault read database/static-creds/static-blue`:

1. 16:35:45.385: last successful login with the old password.
2. 16:35:45.568: `last_vault_rotation`.
3. 16:35:45.673: first refusal, `FATAL: password authentication failed for user "app_static_blue"`.

The probe interval of 0.25 s is the resolution of this result, and it is one manual `rotate-role`, not a scheduled `rotation_period` rotation. I did not observe a scheduled one.

A second finding: creating a static role is itself a rotation. `app_static_blue` was created with the password `initial-blue-pw`. After `vault write database/static-roles/static-blue ...` that password was refused, and the role showed `skip_import_rotation false`. The API docs mark `skip_import_rotation` as an Enterprise parameter, and I did not try setting it to `true`.

Dynamic roles overlap, but the window is not a setting. The agent renewed each lease to the `max_ttl` cap and then fetched a new credential. Across 17 rotations in six runs, the time between "new credential rendered" and "old lease revoked" was 15.98 to 17.04 s in 14 cases and 4.02 to 5.11 s in 3. The short ones followed a second, capped renewal (`TTL of "30s" exceeded the effective max_ttl of "4s"`). A consumer gets 4 to 17 s, and nothing in the role configuration says which.

## Does the Application Need a Restart or a Reload When a Secret Changes?

Reload if you can write the handler, restart if you cannot. In the sandbox, an agent SIGHUP and a file-change check each ran 222 probes with 0 failures. The process supervisor also showed 0, but it restarts the process on every rotation, and its 0 is a best case. I ran each fix against role `dyn` for 150 s.

1. **Template ****exec**** sends SIGHUP; the worker rebuilds its pool.** 0 failed probes of 222. Render to rebuilt pool took 0.27 to 1.57 s across 4 renders, bounded by the 2 s tick. The rebuild itself took 22 to 32 ms. The cost is a reload handler in the application and a way to signal it (here a shared PID namespace). The template stanza gains:

```hcl
   exec {
     command = ["sh", "-c", "... kill -HUP $(cat /secrets/worker.pid)"]
     timeout = "10s"
   }
   
```

The Agent docs say the exec block "executes a command when the template is rendered and the output has changed".

1. **The worker re-reads the file when its mtime changes.** 0 failed probes of 222. Render to rebuilt pool took 1.29 to 1.93 s across 3 renders. The cost is application code only, with no signal and no agent change; the poll interval sets the delay. The check in the worker's loop:

```python
   m = os.stat(CRED_FILE).st_mtime
   if m != S.mtime:
       S.build("file-changed")
   
```

1. **Vault Agent process supervisor with ****restart\_on\_secret\_changes = "always"****.** 0 failed probes, but every rotation is a full process restart. Render to SIGTERM took 2.00 to 2.02 s, a fixed delay seen on every rotation. SIGTERM to a new Python process took 0.49 to 0.53 s with no process serving, and render to first query on the new credential took 2.53 to 2.60 s. With `max_ttl=60s` that meant one restart every 43 s. My worker exits on SIGTERM in about 1 ms and was idle, so nothing was in flight. The docs give the child 30 s after SIGTERM before SIGKILL; I did not observe that. The supervisor "requires at least one `env_template` block and exactly one top level `exec` block" and "is incompatible with regular file `template` entries".

A fourth option, re-reading only on an authentication failure, was not enough alone. I ran `vault lease revoke -prefix database/creds/dyn` by hand at 16:38:20.366. The worker's re-read at 16:38:21.973 loaded the same revoked credential, because the agent had not noticed yet. The agent found out when its next renewal failed with `lease not found`, then rendered a new file. The gap between good queries was 4.3 s.

## In What Order Should You Roll Out a Rotation?

Fix the consumers first, rotate second, retire the old credential last. Two measurements force that order: creating a static role changed the database password at once, and a dynamic credential's overlap window was 4 to 17 s. Neither leaves time to repair a consumer after rotation starts.

This is my reading of the sandbox numbers and HashiCorp's guidance, not a sequence I ran end to end:

1. **Move every consumer onto Vault-delivered credentials before creating a static role.** Anything still using the user's old password breaks at the moment the role is created, as `initial-blue-pw` did.
2. **Ship the reload or re-read code with rotation still off.**
3. **Force one rotation on one canary.** `vault write -f database/rotate-role/<name>` for a static role, or a lease revoke for a dynamic one, with failed probes counted. The Well-Architected rotation page likewise says to update one service, confirm it connects, then continue.
4. **Enable the real ****rotation\_period**** or TTL, then retire the old credential and run the revocation checks below.**

## How Do You Verify the Old Credential Is Really Revoked?

Run four checks and then try the credential. In the sandbox, the lease listing, the lease lookup and `pg_roles` all said "gone", and a login with the old credential was refused. Only a fourth query, for sessions with a NULL user name, found the connection still attached to the dropped role.

These are the commands from `exp4-revocation.sh`, with the `docker compose -p b14demo exec -T` prefixes shortened:

```text
$ vault lease revoke database/creds/dyn/xiCMn0xxUWZIsQyxyG7cX5QN
All revocation operations queued successfully!

$ vault list sys/leases/lookup/database/creds/dyn
No value found at sys/leases/lookup/database/creds/dyn
(exit 2)

$ vault lease lookup database/creds/dyn/xiCMn0xxUWZIsQyxyG7cX5QN
error looking up lease id database/creds/dyn/xiCMn0xxUWZIsQyxyG7cX5QN: Error making API request.
...
Code: 400. Errors:
* invalid lease
(exit 2)

$ psql -c "select rolname from pg_roles where rolname = 'v-token-dyn-jp8N1shbpC7Vahqq1AYL-1791995772'"
(0 rows)

$ psql -c "select pid, usesysid, usename, backend_start, state from pg_stat_activity
           where backend_type = 'client backend' and datname = 'shop' and usename is null"
 pid  | usesysid | usename |         backend_start         | state
------+----------+---------+-------------------------------+-------
 8783 |    16461 |         | 2026-10-14 16:36:15.168997+00 | idle
(1 row)

login with the revoked credential:
psycopg.OperationalError: connection failed: connection to server at "172.21.0.2", port 5432 failed: FATAL:  password authentication failed for user "v-token-dyn-jp8N1shbpC7Vahqq1AYL-1791995772"
```

`vault lease revoke` prints "queued", not "done", so the listing and lookup after it are the real answer. The fourth query is the one that finds the leftover. A session with `usename IS NULL` is a connection whose role was dropped under it. With default revocation it stays connected until the client closes it, and inside it `select current_user` fails with `invalid role OID: 16461`.

To make the failure loud instead, add `pg_terminate_backend` to the role's `revocation_statements`. I ran a second role (`dyn-term`) with this:

```text
SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE usename = '{{name}}'; DROP ROLE IF EXISTS "{{name}}";
```

Open connections were killed at the moment of revocation (`FATAL:  terminating connection due to administrator command`) and `pg_stat_activity` showed `sessions=[]` from 16:22:35. The naive worker still failed: first error +1.36 s after the revoke, `PoolTimeout` from +8.4 s, 12 of 12 pool probes failed, 0 successes for 79 s. Terminating sessions makes the failure loud and immediate. It does not save a worker that never re-reads.

## Limitations: What Does This Not Cover?

Nothing here is a production measurement. The sandbox compresses hours into minutes with `default_ttl=30s` and `max_ttl=60s`, so the mechanisms carry over and the windows scale. Versions: Vault v2.1.2 (built 2026-10-06), PostgreSQL 17.11, psycopg\_pool 3.3.3.

- **Not tested: Kubernetes and KV v2.** The Vault Agent injector, Vault Secrets Operator (`rolloutRestartTargets`), the CSI provider, KV v2 and `static_secret_render_interval` were out of scope. The sidecar here is Docker Compose with a shared volume.
- **Best case for restarts.** My worker is single-threaded and idle between ticks. Requests in flight, a SIGTERM handler that drains, and the SIGKILL escalation were not tested, so "0 failed probes" in the supervisor run is a ceiling.
- **Not tested: two static roles alternating** (blue/green) and scheduled `rotation_period` rotations. Static results come from one manual `rotate-role`.
- **Dev mode only.** In-memory storage, no HA.
- **Revocation failures.** A dynamic role that owns objects, so `DROP ROLE` fails, was not provoked. `VALID UNTIL` tracked the lease expiry in every sample.
- **One pool library.** Only psycopg\_pool 3.3.3. HikariCP, node-postgres and SQLAlchemy were not run.
- **One Enterprise label unchecked.** The API docs mark `skip_import_rotation` as Enterprise. I never set it to `true`, so I can neither confirm nor refute that label.

[The rest of the infrastructure field notes](/blog/category/infrastructure) are in the category.

Vault can revoke a credential, but only the application can stop using it.

If you run Agent sidecars in front of connection pools, do you rebuild the pool on SIGHUP or on a file change, and what happens to a request that is in flight when the old credential dies? My worker was idle between ticks, so I have no number for that.

## References

- HashiCorp Vault docs, v2.x (latest), read 2026-10-14: [database secrets engine](https://developer.hashicorp.com/vault/docs/secrets/databases), [database API](https://developer.hashicorp.com/vault/api-docs/secret/databases), [leases](https://developer.hashicorp.com/vault/docs/concepts/lease).
- Vault Agent, read 2026-10-14: [template](https://developer.hashicorp.com/vault/docs/agent-and-proxy/agent/template) and [process supervisor](https://developer.hashicorp.com/vault/docs/agent-and-proxy/agent/process-supervisor).
- HashiCorp Well-Architected Framework, [rotate secrets](https://developer.hashicorp.com/well-architected-framework/secure-systems/secrets/rotate-secrets), read 2026-10-14.
- HashiCorp Discuss, [Static role credentials rotation without breaking the running clients](https://discuss.hashicorp.com/t/static-role-credentials-rotation-without-breaking-the-running-clients/14567), posted 2020-09-23 and 2022-02-10, read 2026-10-14. Cited as a report, not as my evidence.
- The sandbox (Compose file, worker, Agent configs, scripts, `measure.py` and every log) from 14 October 2026: the source for every measured figure above.