Run Vault Enterprise with many namespaces

Use namespaces to create isolated environments within Vault Enterprise. By default, Vault limits the number and depth of namespaces based on your storage configuration. The information below provides guidance on how to modify your namespace limits and what expect when operating a Vault cluster with 7000+ namespaces.

Default namespace limits

The entire list of namespaces must fit in a single storage entry. However, the effective limit is generally much smaller because each namespace must have at least two secret engine mounts (for sys and identity), one local secret engine (cubbyhole) and one auth engine mount (token).

The maximum nesting depth calculation assumes a cost of 40 bytes per namespace path element. 160 nested paths = 160 namespaces ranging from 40 bytes to 6400 bytes.

You can monitor the number of namespaces by querying sys/namespaces.

To estimate the number of namespaces that can be created, divide the mount point limit by the larger of the number of auth mounts per namespace (including ns_token) and the number of secret mounts per namespace (including identity and sys.)

How to modify your namespace limit

The maximum size of an object written to a storage backend is determined by that backend.

The default entry size limit for the integrated storage backend, is 1 MiB. You can configure the allowable entry size with the max_entry_size paramter in your the storage stanza. Vault automatically chunks any storage entry that is larger than 512 KiB but smaller than max_entry_size into smaller pieces before writing the entry to Raft.

For Vault deployments using the Consul storage backend, the default entry size limit is 512 KiB. The default size in enforced by Consol rather than Vault. You can configure the entry size limit with the kv_max_value_size Consul parameter.

However, Consul does not chunk storage entries the way Vault does. Consul stores the entry as a single, large write. Even small changes may result in large read-modify-write cycles for storage entries, which can degrade Vault performance. Larger writes may also destabilize your Consul cluster by delaying heartbeats, which can lead to cluster leadership instability.

Performance considerations

Running Vault with thousands of namespaces can have operational impacts on a cluster. Below are some performance considerations to take into account before using thousands of namespaces.

It is not recommended to use thousands of namespaces with any version of Vault lower than 1.13.9, 1.14.5, or 1.15.0. Improvements were released in those versions which can improve the reliability of Raft heartbeats when using many namespaces.

Unseal times

Vault sets up and initializes every mount after an unseal event. At minimum, the initialization process includes the default mounts for all active namespaces (sys, identity, cubbyhole, and token).

The more namespaces and custom mounts in the deployment, the longer the post-unseal initialization takes. As a result, even with auto-unseal, Vault will be unresponsive during initialization for deployments with many namespaces.

Post-unseal times observed during testing:

Cluster leadership transfer times

Vault high availability clusters have a leader (also known as an active node) which is the server that accepts writes to the cluster and replicates the written data to the follower nodes. If the leader crashes or needs to be removed from the cluster, one of the follower nodes must take over leadership. This is known as a leadership transfer.

Whenever a leadership transfer happens, the new active node must go through all of the mounts in the cluster and set them up before the node can be ready to be the leader. Because every namespace has at least 4 mounts (sys, identity, cubbyhole, and token), the time for a leadership transfer to complete will increase with the number of namespaces.

Leadership transfer times observed for the vault operator step-down command:

System requirements

Minimum memory requirements

Each namespace requires at least 435 KB of memory to store information about the paths available within the namespace. Given N namespaces, your Vault deployment must include at least (435 x N) KB memory for namespace support to avoid degraded performance.

Rollback and rotation worker requirements

Sometimes, Vault secret and auth engines need to clean up data after a request is canceled or a request fails halfway through. Vault issues rollback operations every minute to each mount in order to periodically trigger the clean up process.

By default, Vault uses 256 workers to perform rollback operations. Mounts with a large number of namespaces can become bottlenecks that slow down the overall rollback process. The effects of the slowdown vary based on the particular mounts. At minimum, your Vault deployment will take longer to fully purge stale data and periodic rotations may happen less frequently than intended.

You can tell whether the number of rollback workers is sufficient by monitoring the following metrics:

Identity secret engine warnings

When using OIDC with many namespaces, you may see warnings in your Vault logs from the identity secret mount under the root namespace. For example:

2023-10-24T15:47:56.594Z [WARN]  secrets.identity.identity_51eb2411: error expiring OIDC public keys: err="context deadline exceeded"
2023-10-24T15:47:56.594Z [WARN]  secrets.identity.identity_51eb2411: error rotating OIDC keys: err="context deadline exceeded"

The secrets.identity warnings occur because the root namespace is responsible for rotating the OIDC keys of all other namespaces.