How to Configure NFS Storage for CloudBees CI Managed Controllers HA on OpenShift

CloudBees CI HA on OpenShift starts with one requirement that can stop the deployment before it even begins: shared storage with ReadWriteMany (RWX) access. ODF is a common choice, but if your OpenShift cluster does not have ODF, you do not need to deploy an entire storage platform just to provide RWX storage. NFS is a supported option for on premises environments, with CloudBees recommending NFS 4.1.

The real challenge is getting NFS right for CloudBees CI. Storage performance, SSD or NVMe selection, network speed, NFS configuration, filesystem settings, mount options, OpenShift integration, and failure handling all matter. This guide walks through the complete setup, from building and sizing the NFS server to configuring OpenShift and enabling HA for CloudBees CI Managed Controllers, with a focus on the settings that matter in production.

Configure NFS Storage for CloudBees CI Managed Controllers HA on OpenShift

How CloudBees CI HA actually Works

Before touching any YAML files, it is worth understanding what CloudBees CI High Availability (HA) is actually doing under the hood. Once you understand how the replicas share state, workloads, and persistent data, the NFS requirements make a lot more sense.

CloudBees CI HA runs multiple replicas of the same Managed Controller as one logical controller. The replicas work together to provide failover, workload distribution, and rolling operations:

• Controller failover: If a replica fails, running builds can be adopted by another replica, allowing the controller to continue operating without the failed replica.
• Horizontal scaling: A single logical controller can spread its workload across multiple replicas. CloudBees CI also provides explicit build load balancing, which can direct new builds toward the replica with the least workload.
• Rolling restarts and upgrades: Replicas can be restarted or upgraded incrementally, allowing the controller to remain available during supported rolling operations.
• Live state synchronization: CloudBees CI replicas use the embedded Hazelcast distributed computation library to keep their live state synchronized. No separate Hazelcast deployment or storage configuration is required.
• Shared filesystem: This is the part that matters most for this guide. The replicas need access to shared persistent controller content. That means the storage backing the controller must support simultaneous read and write access from multiple pods, which is why ReadWriteMany (RWX) storage is required for a multi-replica HA controller.

This is fundamentally different from a standard non-HA Managed Controller, where a single controller pod can use storage with ReadWriteOnce (RWO) access. When you move that controller to HA with multiple replicas, the storage architecture changes with it. The replicas must be able to access the same shared filesystem, which is where an RWX-capable backend such as NFS becomes important for on-premises OpenShift environments. CloudBees’ current platform guidance specifically supports NFS 4.1 and higher, and its on-premises planning guidance strongly recommends NFS for the storage class.

The architecture at a glance

CloudBees CI HA combines multiple Managed Controller replicas into a single logical controller. The replicas share workloads, synchronize live state through embedded Hazelcast, and access the same shared filesystem for persistent controller data. For resilience, CloudBees recommends placing HA replicas on different worker nodes and, where possible, across different availability zones.

 		  ┌───────────────────────────────────────────┐
                  │            OpenShift Cluster              │
                  │                                           │
                  │    CloudBees CI HA Managed Controller     │
                  │                                           │
                  │      ┌───────────┐     ┌───────────┐      │
                  │      │ Replica 1 │     │ Replica 2 │      │
                  │      └─────┬─────┘     └─────┬─────┘      │
                  │            │                 │            │
                  │            │   Hazelcast     │            │
                  │            │ synchronization │            │
                  │            └────────┬────────┘            │
                  │                     │                     │
                  │           ┌─────────▼─────────┐           │
                  │           │  Shared RWX PVC   │           │
                  │           │ NFS StorageClass  │           │
                  │           └─────────┬─────────┘           │
                  └─────────────────────┼─────────────────────┘
                                        │
                                        │ NFS 4.1+
                                        │ over TCP
                          ┌─────────────▼─────────────┐
                          │        NFS Server         │
                          │       RHEL / Linux        │
                          │       SSD / NVMe          │
                          │       High-speed NIC      │
                          └───────────────────────────┘

There are two separate paths to understand:

• Hazelcast handles live state synchronization between the controller replicas, while
• the shared RWX filesystem provides persistent controller storage.

CloudBees requires shared storage between HA replicas and recommends NFS 4.1 for on premises deployments.

Every Managed Controller replica mounts the same NFS-backed RWX volume concurrently. This makes the NFS layer a critical dependency for the controller’s persistent filesystem. Its storage media, network capacity, latency, NFS version, filesystem configuration, and failure behavior therefore become important parts of the overall CloudBees CI HA architecture.

NFS does not provide the HA functionality itself. It provides the shared storage layer that HA requires. The complete HA architecture combines multiple controller replicas, shared RWX storage, Hazelcast-based state synchronization, and the OpenShift scheduling and networking infrastructure that connects them.

Part 1: Sizing and Building the NFS Server

Hardware and Storage Type

CloudBees CI’s own reference architecture is clear about the storage requirement:

• Pipelines typically read and write many files during execution, so CloudBees CI requires high speed storage.
• CloudBees also notes that the underlying storage provider must provide sufficient IOPS, throughput, and low latency. For on premises deployments, CloudBees recommends NFS 4.1.

That matters even more when the storage is backing HA Managed Controllers. Multiple controller replicas can access the same persistent filesystem concurrently, so a slow NFS backend can quickly become the bottleneck for the entire controller.

For a production NFS server backing CloudBees CI, this is the baseline I would use:

Component	Recommendation	Why it matters
Disk type	Enterprise SSD or NVMe, preferably with redundant storage such as RAID 10	CloudBees CI generates frequent filesystem activity, including metadata operations, configuration updates, logs, plugin access, and build data. SSD or NVMe provides substantially lower latency and higher IOPS than spinning disks.
Filesystem	XFS for a Linux NFS server, or the filesystem managed by your NAS platform	XFS is a strong default for a high concurrency Linux NFS server. Ext4 is also viable when it better matches your existing storage standards.
RAID controller cache	Battery backed or flash backed write cache, where supported	A protected write cache can reduce write latency while preserving data integrity during a power or controller failure. Do not enable unprotected write back caching for controller data.
CPU	4 or more cores as a starting point	NFS request processing and filesystem activity consume CPU. Increase CPU when monitoring shows the server becoming CPU bound.
RAM	16 to 32 GB as a starting point	Linux uses available memory for the filesystem page cache, which can significantly reduce repeated reads from the underlying storage. Larger environments may benefit from more memory.
IOPS	Size from measured workload rather than a fixed CloudBees number	CloudBees does not publish a universal IOPS requirement. The correct target depends on controller count, concurrent builds, filesystem activity, and workload characteristics.

The important point is that the table above is an engineering baseline, not a CloudBees minimum specification. CloudBees requires sufficient IOPS, throughput, and low latency, but there is no single IOPS figure that is correct for every installation.

If you are virtualizing the NFS server, pay just as much attention to the storage underneath the VM. A 16 core NFS VM with plenty of RAM will still perform badly if its virtual disks sit on an oversubscribed datastore. This is one of those problems that often looks like a CloudBees CI performance issue until you look at the storage latency underneath the NFS server.

For production, I would also avoid putting the CloudBees CI export on the same storage pool as unrelated workloads such as backup repositories, large file shares, or bulk archival jobs. CI workloads can produce sudden bursts of filesystem activity, and competing workloads can turn otherwise healthy storage into a latency problem.

Network Requirements

Because every HA controller replica accesses the shared $JENKINS_HOME over the network, NFS network performance needs to be treated as a core part of the storage design.

There is no single network speed that CloudBees defines as a requirement for NFS. The right size depends on the number of controllers, workload, concurrency, and storage performance. As a practical starting point:

Deployment size	Minimum starting point	Recommended production target	Target latency
Small / development	1 GbE	10 GbE	< 1 ms
Standard production	10 GbE	10 GbE or higher	< 1 ms
Large / multi team HA cluster	10 GbE	25 GbE or higher	< 1 ms

These are infrastructure design targets, not CloudBees product requirements. Measure the actual network path between the OpenShift worker nodes and the NFS server before deciding that additional bandwidth is necessary.

A few practical rules make a significant difference:

• Keep NFS traffic isolated where practical. A dedicated NIC, VLAN, or storage network prevents storage traffic from competing with unrelated high volume traffic such as backups, image transfers, or application ingress.
• Keep the NFS server close to the OpenShift worker nodes. Avoid putting the shared controller filesystem across a WAN, across regions, or across a high latency network path.
• Watch retransmissions and packet loss. A link can have plenty of available bandwidth and still deliver poor NFS performance when packets are being retransmitted.
• Do not enable jumbo frames blindly. MTU 9000 can reduce network overhead for large transfers, but it must be configured consistently across the entire path. If the NFS server, OpenShift nodes, switches, VLAN interfaces, or routers disagree about the MTU, troubleshooting becomes much harder.
• Benchmark with concurrent clients. A network that looks excellent with one NFS client can behave very differently when several controller replicas and other workloads are accessing the server simultaneously.

For an HA deployment, I would establish a baseline for latency, throughput, packet loss, TCP retransmissions, and peak interface utilization before putting production controllers on the storage.

NFS Server software

Use NFS 4.1 or later for this deployment. CloudBees supports NFS v3.x and NFS v4.1 and higher, but does not support NFS 4.0 because of known performance issues. For CloudBees CI on premises, CloudBees specifically recommends NFS 4.1.

If you are running a Linux based NFS server, install and maintain the NFS server packages provided by your distribution’s supported repositories. For example, RHEL uses nfs-utils. Keep the operating system and NFS components patched and within the vendor’s supported lifecycle.

For NFS 4.1, do not blindly copy older NFSv3 configuration guides. NFSv4 does not require rpcbind, so an NFS 4.1 only server does not need it for NFS communication. Install and enable the NFS server components required by your operating system, then verify that the server is actually exporting the filesystem over NFS 4.1.

If you are using NFSv4 identity mapping, make sure the UID and GID mapping behavior is consistent between the NFS server and clients. With OpenShift, however, do not assume that containers will run as root. OpenShift requires containers to run as non root users and commonly assigns arbitrary UIDs, so permissions need to be designed accordingly.

Example /etc/exports:

cat /etc/exports

# CloudBees CI NFS export
/path/to/cloudbees-ci 10.0.0.0/16(rw,sync,no_subtree_check)

Replace both the path and network subnet with the ones from your environment:

• /path/to/cloudbees-ci is the directory or filesystem you created for CloudBees CI on the NFS server. It could be something such as /srv/nfs/cloudbees-ci, /export/cloudbees-ci, or a mount point backed by a dedicated storage volume.
• 10.0.0.0/16 is the IP network containing the OpenShift worker nodes that need to mount the export. Replace it with your actual OpenShift node subnet or, where appropriate, a specific list of allowed client networks.

A few export options deserve particular attention:

• rw allows the OpenShift clients to read and write the CloudBees CI filesystem.
• sync is the safer choice for persistent controller data. The NFS server does not acknowledge writes before the changes are committed to stable storage.
• no_subtree_check avoids unnecessary subtree checking when exporting a directory from a larger filesystem.
• Do not add no_root_squash simply because an NFS example includes it. root_squash is the default and prevents root on an NFS client from having root privileges on the exported filesystem. Disabling it weakens the security boundary and does not solve the underlying OpenShift UID problem.
• Restrict the export to the OpenShift nodes or networks that actually require access. The NFS export rules should be backed by host firewall and network controls rather than being treated as the only security boundary.

The OpenShift UID problem, and why no_root_squash does not solve it

Generic Kubernetes NFS guides frequently reach for no_root_squash because, on a default Kubernetes cluster, the provisioner container runs as UID 0 and root_squash maps it to an anonymous identity that cannot create directories on the export.

OpenShift does not behave that way, and CloudBees documents this in its OpenShift pre-installation requirements: containers must run as non-root and non-privileged, and OpenShift assigns arbitrary UIDs. Under the restricted-v2 SCC, pods receive an arbitrary UID from the project’s allocated range and never run as root. The provisioner therefore never presents itself as root, so root_squash has nothing to squash and disabling it changes nothing.

The real requirement is different. The provisioner presents a high, arbitrary UID that matches no identity on the NFS server, and it needs write access to the export directory as that unmatched identity. If the export is owned by root with default permissions, directory creation fails and PVCs remain Pending, or pods encounter permission errors on mount. Both symptoms are easily misread as a network or StorageClass fault.

Solve this on the NFS server by granting write access to the export directory for the identity the provisioner actually presents. Options include group ownership with a shared GID, or all_squash with anonuid and anongid mapped to a dedicated account that owns the export. Choose the approach that fits your organisation’s identity model, and resolve ownership and write access through UID, GID, group permissions, security context, and provisioner configuration rather than making the entire export universally writable.

Note also that the provisioner chart applies its own ownership and mode to each subdirectory it creates, controlled by nfs.defaultMode, nfs.defaultUid, and nfs.defaultGid, which default to 777, 0, and 0. These govern the directories the provisioner creates, not the identity it creates them as, so they do not substitute for correct permissions on the export directory itself.

The functional validation in Part 2 exercises exactly this path. If the test PVC reaches Bound and both test pods can write, the permission model on your export is correct. If either step fails with permission errors, resolve it here rather than by relaxing the export. Copying the common no_root_squash example from a generic Jenkins NFS tutorial may make a permissions problem disappear during testing, but it is not a good production security model

Tuning the NFS Server Daemon

Do not start by throwing a collection of NFS kernel parameters at the server or setting RPCNFSDCOUNT=16 because a tuning guide says 16 is a good number.

Modern Linux NFS servers can handle client demand dynamically, so there is no universal nfsd thread count that is optimal for every CloudBees CI environment. The right level of concurrency depends on the NFS server’s CPU capacity, storage performance, network capacity, number of connected clients, and workload. Start with the operating system defaults, measure performance under realistic CloudBees CI load, and tune only when the measurements show that NFS request processing is actually limiting performance.

Start with the default configuration and establish a baseline under realistic CloudBees CI load.

On the NFS server, use nfsstat to check server-side NFS activity and retransmissions:

sudo nfsstat -s

Use iostat to check storage latency, utilization, and queue depth:

iostat -xz 1

Use vmstat to get a quick view of CPU, memory, and I/O pressure:

vmstat 1

Use sar to check network traffic and interface utilization:

sar -n DEV 1

On an NFS client, you can also inspect client side NFS statistics:

nfsstat -c

Watch for:

• High disk latency
• Increasing storage queue depth
• CPU saturation
• Network saturation
• NFS retransmissions
• Memory pressure
• Increasing NFS operation latency

If the server is CPU bound while the storage and network still have capacity, increasing NFS concurrency may be worth testing. If the disks are already saturated, increasing NFS threads will not make the storage faster. It will simply put more requests into the queue.

That distinction matters. Tune the bottleneck, not the NFS server by folklore.

If the NFS server is also being used as a backup target, registry storage, general file server, or anything else with heavy I/O, isolate the CloudBees CI storage whenever possible. A build storm combined with a large backup job is exactly the kind of workload that can turn a healthy NFS server into a controller performance problem.

The goal is not to find a magic NFS configuration. The goal is to give CloudBees CI a fast, predictable, low latency shared filesystem with enough IOPS, throughput, and headroom to remain stable during peak workload. That is the foundation the HA configuration depends on.

If you are looking at guides on how to install and setup NFS server, refer to our guides.

Part 2: Wiring NFS into OpenShift as a StorageClass

Deploy the NFS Client Provisioner

At this point, you have an NFS server with a directory exported to your OpenShift nodes. The next step is to make that storage available to Kubernetes/OpenShift as a StorageClass.

A StorageClass allows OpenShift to create Persistent Volumes automatically when CloudBees CI requests storage. Instead of manually creating a PV for every controller, the NFS provisioner creates a directory on the NFS export for each dynamically provisioned volume.

For this guide, we will use the NFS Subdir External Provisioner, which dynamically creates Persistent Volumes from an existing NFS export.

We will install the provisioner using Helm. Helm is the package manager commonly used to install and manage Kubernetes applications. In this case, Helm installs the NFS provisioner and creates the StorageClass that CloudBees CI will use later.

Before starting, have these two values from your NFS server configuration:

• NFS server address: for example, 10.185.10.199
• NFS export path: for example, /data/cloudbees

The export path is the path you configured in /etc/exports. It is not necessarily /data/cloudbees, so use the actual path from your environment.

First, add the NFS provisioner’s Helm repository:

helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner/

Update the local Helm repository information:

helm repo update

Create a dedicated namespace for the provisioner:

oc create namespace nfs-provisioner

Now install the provisioner. This is where the NFS server address and export path are actually used. Important: the namespace used for the provisioner is independent of the NFS export path and the CloudBees CI namespace. You can use a dedicated namespace such as nfs-provisioner; you do not need to place the provisioner in openshift-storage.

helm install nfs-provisioner \
  nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
  --set nfs.server=10.185.10.199 \
  --set nfs.path=/data/cloudbees \
  --set storageClass.name=managed-nfs-storage \
  --set storageClass.accessModes=ReadWriteMany \
  -n nfs-provisioner

Replace:

10.185.10.199

with the IP address or resolvable hostname of your NFS server, and:

/data/cloudbees

with the actual NFS export path you configured earlier.

The important part is that nfs.server and nfs.path point to the same NFS server and export you configured in Part 1. The provisioner uses these values to know where to create the directories for dynamically provisioned Persistent Volumes.

The storageClass.name value, managed-nfs-storage in this example, is the name OpenShift will use when requesting storage. You can choose a different name, but keep it consistent when you configure CloudBees CI.

The ReadWriteMany setting is critical for this deployment because CloudBees CI HA requires the controller replicas to access the shared filesystem concurrently.

After installation, verify that the provisioner is running:

oc get pods -n nfs-provisioner

You should see the NFS provisioner pod in a Running state.

Sample output;

NAME                                                          READY   STATUS    RESTARTS   AGE
nfs-subdir-nfs-subdir-external-provisioner-7c78c69788-x9k2p   1/1     Running   0          12s

Then verify that the StorageClass was created:

oc get storageclass

You should see your storage class, managed-nfs-storage (or whatever name you called it), listed.

At this point, the path is:

NFS Server
    │
    └── /data/cloudbees
            │
            ▼
    NFS Subdir External Provisioner
            │
            ▼
    managed-nfs-storage (or your SC name)
            │
            ▼
    CloudBees CI PVC
            │
            ▼
    RWX Persistent Volume

The next step is to prove that dynamic provisioning actually works by creating a small RWX test PVC before involving CloudBees CI. This catches NFS permissions, connectivity, StorageClass, and OpenShift security issues early, instead of discovering them later when the Managed Controller fails to start.

Create the StorageClass

Now that the NFS provisioner is running, create the StorageClass that CloudBees CI will use for its persistent storage.

The important settings here are the NFS version, the mount behavior, and the access mode. CloudBees recommends NFS 4.1 for on-premises deployments, so explicitly specifying NFS 4.1 avoids accidentally mounting the export with a different protocol version.

The provisioner field must match the provisioner name registered by the Helm release, which is derived from the release name. Read the actual value before writing the manifest:

oc get deployment -n nfs-provisioner -o jsonpath='{.items[0].spec.template.spec.containers[0].env[?(@.name=="PROVISIONER_NAME")].value}'

For a release named nfs-provisioner, this returns cluster.local/nfs-provisioner-nfs-subdir-external-provisioner. Use the returned value below rather than the example.

Create a file named managed-nfs-storage.yaml:

cat > managed-nfs-storage.yaml << 'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfs-storage
provisioner: cluster.local/nfs-provisioner
parameters:
  archiveOnDelete: "false"
reclaimPolicy: Retain
volumeBindingMode: Immediate
mountOptions:
  - hard
  - nfsvers=4.1
EOF

The two mount options above are the important starting point:

• hard keeps retrying an NFS operation if the server becomes temporarily unavailable instead of immediately returning an I/O error to the application. This is appropriate for persistent controller data.
• nfsvers=4.1 explicitly selects NFS 4.1, which is the version CloudBees recommends for on-premises deployments.

What about rsize, wsize, actimeo, and nconnect?

You will find many NFS tuning guides recommending values such as rsize=262144, wsize=262144, actimeo=60, or nconnect=4. Do not treat those values as universal CloudBees CI requirements.

Modern NFS clients can negotiate appropriate read and write sizes, and the best values for attribute caching or multiple connections depend on the kernel, NFS server, network, and workload. Changing these settings without measuring the result can improve one workload while making another worse.

Start with the documented NFS 4.1 configuration and establish a performance baseline. If testing shows that the NFS client is the bottleneck, tune these options based on measurements from your specific environment.

For example, nconnect can be useful on high-bandwidth networks by allowing multiple connections to an NFS server, but support and behavior depend on the client kernel and NFS server. Treat it as an optimization to test, not a requirement for CloudBees CI HA.

Apply the manifest to create the storageclass:

oc apply -f managed-nfs-storage.yaml

Verify the StorageClass:

oc get storageclass managed-nfs-storage

Decide: Cluster Default or Explicit StorageClass

Once the NFS StorageClass has been created and verified, CloudBees CI can use it in two ways:

1. Make the NFS StorageClass the OpenShift cluster default.
2. Keep the existing OpenShift default unchanged and configure CloudBees CI to use the NFS StorageClass explicitly.

Both approaches are supported by CloudBees. For a shared production OpenShift cluster, leaving the existing cluster default unchanged and explicitly configuring the CloudBees CI storage class is generally the more controlled approach.

The NFS StorageClass name used in this guide, managed-nfs-storage, is an example. Replace it with the actual name of the NFS StorageClass created in your OpenShift cluster.

Option 1: Make the NFS StorageClass the cluster default

A PVC that does not specify a storageClassName uses the cluster’s default StorageClass. Therefore, making the NFS StorageClass the OpenShift default allows CloudBees CI to use NFS without specifying the StorageClass separately.

For example:

oc patch storageclass managed-nfs-storage \
  -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

If another StorageClass is currently marked as the default, remove that designation:

oc patch storageclass <existing-default-storage-class> \
  -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

Verify the result:

oc get storageclass

The NFS StorageClass should now be marked as the default.

This approach is straightforward, but the change is cluster-wide. The default StorageClass is an OpenShift/Kubernetes cluster setting, not a CloudBees CI setting. Any PVC that does not explicitly specify a StorageClass can therefore use the new default.

For example, if the existing default StorageClass provides block storage and is used by other applications in the cluster, changing the default to NFS can affect those applications as well.

For that reason, changing the cluster default is generally not appropriate on a shared OpenShift platform unless NFS is intentionally intended to become the default storage backend for the cluster.

If the OpenShift cluster is dedicated to CloudBees CI and there is a deliberate requirement for NFS to be the cluster-wide default, this approach is valid.

For the deployment described in this guide, the existing OpenShift default remains unchanged and the NFS StorageClass is configured explicitly for CloudBees CI.

Option 2: Configure CloudBees CI to use the NFS StorageClass

The CloudBees CI Helm chart provides the Persistence.StorageClass value for selecting the StorageClass used for CloudBees CI persistent storage.

The chart defines this value at the top level:

Persistence:
  StorageClass: null

The default value is null. This does not mean that CloudBees CI uses NFS. When Persistence.StorageClass is unset, CloudBees CI uses the default StorageClass configured in the Kubernetes/OpenShift cluster.

You can verify the current chart value with:

helm show values cloudbees/cloudbees-core | grep -iC 5 storageclass

The relevant chart definition is:

Persistence:
...
  StorageClass: null
...

For current CloudBees CI versions, CloudBees documents that setting Persistence.StorageClass changes the StorageClass used by both the Operations Center and Managed Controllers.

This distinction is important because CloudBees CI has separate persistent storage for the Operations Center and for each Managed Controller. The Operations Center has its own $JENKINS_HOME, and each Managed Controller has its own $JENKINS_HOME. When new Managed Controllers are provisioned, the Operations Center creates the Kubernetes resources and persistent storage required for them.

The correct configuration therefore depends on when the NFS StorageClass is introduced.

Scenario 1: NFS is available before CloudBees CI is installed

For a new CloudBees CI installation, create and verify the NFS StorageClass first, then decide how widely it should apply.

Persistence.StorageClass applies to both the Operations Center and Managed Controllers. Setting it places every CloudBees CI component on that StorageClass, including components that do not require RWX. There are two valid configurations.

Option A: NFS for everything

Set Persistence.StorageClass at install:

helm install cloudbees-core cloudbees/cloudbees-core \
  --set OperationsCenter.HostName='<hostname>' \
  --set OperationsCenter.Route.tls.Enable=true \
  --set Subdomain=true \
  --set Agents.SeparateNamespace.Enabled=true \
  --set Agents.SeparateNamespace.Create=true \
  --set CasCBundleService.enabled=true \
  --set Persistence.StorageClass='<nfs-sc-name>' \
  -n cloudbees-ci

Replace <nfs-sc-name> with the actual name of the NFS StorageClass in your cluster.

This is simple and uniform, with a single storage dependency for the whole installation. The cost is that the Operations Center and non-HA controllers run on NFS rather than block storage, which is slower for the small-file I/O Jenkins generates.

Option B: NFS only where HA requires it

Omit Persistence.StorageClass so the Operations Center and non-HA controllers use the cluster default block storage:

helm install cloudbees-core cloudbees/cloudbees-core \
  --set OperationsCenter.HostName='<hostname>' \
  --set OperationsCenter.Route.tls.Enable=true \
  --set Subdomain=true \
  --set Agents.SeparateNamespace.Enabled=true \
  --set Agents.SeparateNamespace.Create=true \
  --set CasCBundleService.enabled=true \
  -n cloudbees-ci

Then select the NFS StorageClass per controller using the Storage Class Name field when provisioning an HA Managed Controller, as described under “HA Managed Controllers” below.

This guide follows Option B, and the summary diagram at the end reflects it. RWX is applied where HA requires it and nowhere else.

Under either option, establishing the storage backend before CloudBees CI creates any persistent storage avoids a migration later. Changing Persistence.StorageClass after installation does not move existing data, as covered in Scenario 2.

Scenario 2: CloudBees CI is already installed and NFS is added later

NFS can also be introduced after CloudBees CI has already been installed.

First create and verify the NFS StorageClass. Then update the existing CloudBees CI Helm release so that Persistence.StorageClass points to the NFS StorageClass.

Because the CloudBees CI release already exists, use helm upgrade rather than running helm install again.

For example:

helm upgrade cloudbees-core cloudbees/cloudbees-core \
  --reuse-values \
  --set Persistence.StorageClass='managed-nfs-storage' \
  -n cloudbees-ci

The --reuse-values option is useful when the original installation was performed with --set parameters and no separate values file. It tells Helm to retain the values already stored for the release while applying the new Persistence.StorageClass value.

The existing release values can be inspected first with:

helm get values cloudbees-core -n cloudbees-ci

The important point is that changing Persistence.StorageClass does not migrate an existing PVC.

If the Operations Center already has a PVC backed by another StorageClass, changing the Helm value does not change that PVC’s StorageClass or move its data to NFS. CloudBees explicitly documents that StorageClass changes do not apply to an existing PV/PVC. Moving existing data to a different StorageClass requires provisioning new storage and migrating the data.

Therefore, after introducing NFS into an existing installation:

• The existing Operations Center PVC remains on its current storage.
• The existing Managed Controller PVCs remain on their current storage.
• Persistence.StorageClass becomes the configured StorageClass for subsequent CloudBees CI storage provisioning.
• Existing CloudBees CI data is not automatically copied to NFS.
• If existing data must actually be moved to NFS, perform a separate storage migration.

Do not delete an existing CloudBees CI PVC simply to force the new StorageClass to be used. The PVC contains the persistent $JENKINS_HOME data. CloudBees provides separate procedures for recreating or migrating persistent storage when a StorageClass change is required.

Managed Controllers Provisioned After NFS Is Available

The Operations Center provisions Managed Controllers and creates the Kubernetes resources and persistent storage required for them.

If Persistence.StorageClass has been configured to use the NFS StorageClass, that configuration applies to new Managed Controllers as well as the Operations Center in current CloudBees CI versions.

Therefore, a new ordinary Managed Controller does not require a separate Helm StorageClass configuration simply because it is being created after NFS was introduced.

CloudBees also provides a Storage Class Name setting in Managed Controller provisioning. This can be used when a particular controller should use a specific StorageClass instead of the default configured for the Kubernetes cluster or provisioning configuration. Current CloudBees documentation states that when provisioning a controller, the administrator can set the storage class name if the cluster default should not be used.

For example:

Storage Class Name: <nfs-storage-class>

Use the actual NFS StorageClass name from the cluster.

This is especially relevant in environments with multiple Kubernetes clusters or controller provisioning endpoints, where storage configuration may need to be selected for a particular provisioning target.

HA Managed Controllers

HA introduces an additional storage requirement.

A Managed Controller running in HA mode has multiple replicas that must access the same shared filesystem. The StorageClass used for that filesystem must therefore provide ReadWriteMany.

CloudBees’ current HA documentation provides a Storage Class Name field specifically for the replica’s shared filesystem. If the field is left empty, CloudBees uses the default StorageClass defined in the Kubernetes cluster.

Therefore, when the NFS StorageClass is not the OpenShift cluster default, explicitly select the NFS StorageClass when configuring the HA Managed Controller:

Storage Class Name: <nfs-storage-class>

For example:

Storage Class Name: managed-nfs-storage

The name above is only the example used in this guide. Use the actual NFS StorageClass name in your environment.

The cluster can therefore retain its existing default StorageClass while CloudBees CI uses NFS for the HA shared filesystem.

For example:

OpenShift default StorageClass:  <existing-default>
NFS StorageClass:                <nfs-storage-class>

There is no requirement to make the NFS StorageClass the OpenShift cluster default simply to provision an HA Managed Controller.

The important requirement is that the StorageClass selected for the HA replica shared filesystem supports ReadWriteMany. CloudBees explicitly requires RWX-capable shared storage for HA.

Existing Managed Controllers and HA

An existing Managed Controller cannot be converted to HA merely by changing the StorageClass selection.

If its existing $JENKINS_HOME is backed by a StorageClass that does not support ReadWriteMany, the controller must first be migrated to RWX-capable storage.

CloudBees provides a specific migration procedure for this situation. The procedure creates a new volume using the RWX StorageClass, copies the existing controller data to the new volume, updates the controller configuration to use the new storage, and then enables HA.

This is necessary because changing a StorageClass configuration does not change the storage backend of an existing PVC.

For example:

Existing Managed Controller
        |
        +-- Existing PVC
              |
              +-- RWO StorageClass

cannot simply become:

Existing Managed Controller
        |
        +-- Existing PVC
              |
              +-- RWX NFS StorageClass

The existing PVC remains associated with its original storage. A new RWX volume must be provisioned and the controller’s persistent data migrated.

For a new Managed Controller, no such migration is required. The controller can be provisioned using the NFS StorageClass from the beginning.

For a shared on-premises OpenShift cluster, the recommended approach is:

1. Create and verify the NFS StorageClass.
2. Keep the existing OpenShift cluster default unchanged unless NFS is intentionally required as the default storage backend for the entire cluster.
3. For a new CloudBees CI installation, set Persistence.StorageClass to the NFS StorageClass during the Helm installation.
4. This causes the configured CloudBees CI persistent storage to use the NFS StorageClass for both the Operations Center and Managed Controllers in current CloudBees CI versions.
5. If CloudBees CI is already installed, update the existing Helm release with helm upgrade and add Persistence.StorageClass. When the original installation used --set parameters rather than a values file, --reuse-values can be used to retain the existing release values.
6. Understand that changing Persistence.StorageClass does not migrate existing PVCs. Existing Operations Center and Managed Controller data remains on its existing storage until a separate migration is performed.
7. When provisioning a new Managed Controller, the configured CloudBees CI StorageClass applies by default. A specific Storage Class Name can also be selected in the Managed Controller provisioning configuration when required.
8. When creating an HA Managed Controller, explicitly select the NFS StorageClass in the Storage Class Name field for the replicas’ shared filesystem when NFS is not the cluster default.
9. Verify that the selected NFS StorageClass provides ReadWriteMany.
10. If an existing Managed Controller must be converted to HA and its current storage is not RWX-capable, migrate its $JENKINS_HOME to RWX storage before enabling HA.

The key distinction is between three different storage decisions:

• OpenShift cluster default StorageClass: a cluster-wide Kubernetes/OpenShift setting that affects PVCs that do not specify another StorageClass.
• CloudBees CI Persistence.StorageClass: the CloudBees CI Helm configuration that, in current versions, establishes the StorageClass used for persistent storage by both the Operations Center and Managed Controllers.
• HA Managed Controller Storage Class Name: the StorageClass specifically selected for the shared filesystem used by the HA replicas. If left empty, the cluster default is used.

For this reason, there is no need to change the OpenShift cluster default simply to introduce NFS for CloudBees CI HA. Keep the cluster default appropriate for the wider OpenShift environment, configure CloudBees CI’s persistent storage explicitly, and select the RWX-capable NFS StorageClass for the HA shared filesystem.

Verify RWX dynamic provisioning

Do not proceed to performance benchmarking or a production CloudBees CI deployment until the storage path has been validated end-to-end. The objective of this step is not to benchmark NFS; it is to establish that the configured StorageClass can dynamically provision a volume, that the resulting PVC binds successfully, and that the volume can be mounted read-write by multiple pods.

This validation should use a disposable PVC and test pods rather than a CloudBees CI Operations Center or Managed Controller $JENKINS_HOME. A successful PVC bind alone is not sufficient evidence that the storage path is usable by the workload.

The path being validated is:

• NFS export
• nfs-subdir-external-provisioner
• StorageClass
• PVC
• dynamically provisioned PV
• RWX mount in pods
• read/write verification

Each stage must be verified independently. A failure to provision, bind, mount, or write must be diagnosed before continuing. Do not infer that the entire path is working from a single successful Kubernetes object or from documentation describing a different version or configuration.

1. Confirm the StorageClass and provisioner configuration

Before creating the test PVC, verify that the StorageClass actually exists and that its provisioner matches the provisioner installed in the cluster:

oc get storageclass <storage-class-name> -o yaml

oc get pods -A | grep -i nfs

Confirm the following before continuing:

• <storage-class-name> is the StorageClass you intend to test.
• Its provisioner value matches the provisioner configuration.
• The provisioner is running and healthy.
• The provisioner is configured with the intended NFS server and export path.

For nfs-subdir-external-provisioner, the upstream project explicitly describes dynamic provisioning as creating per-PVC subdirectories on an existing NFS export and recommends testing by creating a PVC and pod, writing a file, and checking the NFS server.

Do not substitute a StorageClass name or provisioner configuration from an older installation guide. Verify the values against the version actually deployed in the cluster.

2. Create a disposable RWX test PVC

Assuming your storageClass name is managed-nfs-storage, create a small test claim using the StorageClass under test:

oc new-project cbci-nfs-demo

oc apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-rwx-test-pvc
  namespace: cbci-nfs-demo
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: managed-nfs-storage
  resources:
    requests:
      storage: 1Gi
EOF

Apply it and watch the claim:

oc get pvc nfs-rwx-test-pvc -n cbci-nfs-demo

The expected result is:

STATUS: Bound

OpenShift defines ReadWriteMany (RWX) as a volume that can be mounted read-write by many nodes, and NFS supports RWX.

If the PVC remains Pending, stop here. Inspect the claim and the provisioner rather than proceeding to pod-level testing:

oc describe pvc nfs-rwx-test-pvc -n cbci-nfs-demo

oc get events -n cbci-nfs-demo --sort-by='.lastTimestamp'

Then inspect the provisioner’s logs using the actual provisioner pod name:

oc logs -n <provisioner-namespace> <provisioner-pod>

Do not assume that a Pending claim is an NFS network problem. The failure may be in StorageClass selection, provisioner configuration, RBAC, provisioning, or access to the configured export.

3. Mount the claim from a test pod and perform a write/read test

Once the PVC is Bound, mount it from a disposable pod:

oc apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: nfs-rwx-test-pod
  namespace: cbci-nfs-demo
spec:
  containers:
    - name: test
      image: registry.access.redhat.com/ubi9/ubi-minimal:latest
      command: ["sleep", "3600"]
      volumeMounts:
        - name: nfs-test-vol
          mountPath: /mnt/nfs-test
  volumes:
    - name: nfs-test-vol
      persistentVolumeClaim:
        claimName: nfs-rwx-test-pvc
EOF

Apply the pod and wait for it to become ready:

oc wait --for=condition=Ready pod/nfs-rwx-test-pod -n cbci-nfs-demo --timeout=60s

Then perform an explicit write and read:

oc exec -n cbci-nfs-demo nfs-rwx-test-pod -- sh -c \
  'echo "rwx-check-$(date +%s)" > /mnt/nfs-test/hello.txt && cat /mnt/nfs-test/hello.txt'

The command must successfully create and read the file.

Sample output;

rwx-check-1786379962

If it fails with Permission denied, diagnose the effective UID/GID, the pod security context, the NFS export permissions, and the provisioned directory ownership/mode. Do not treat no_root_squash as a generic fix or weaken NFS export security merely to make the test pass.

The nfs-subdir-external-provisioner documentation specifically notes that provisioned directories may have configurable UID, GID, and mode, including per-PVC annotations. Validate the permission model used by the version and deployment configuration in your environment rather than applying a permission workaround from an unrelated example.

4. Verify shared visibility from a second pod

A successful write/read from one pod proves that that pod can use the mounted volume. It does not by itself demonstrate that the same PVC can be mounted and accessed concurrently by another pod.

Create a second disposable pod using the same PVC:

oc run nfs-rwx-test-pod-2 \
  -n cbci-nfs-demo \
  --image=registry.access.redhat.com/ubi9/ubi-minimal:latest \
  --restart=Never \
  --overrides='{
    "spec": {
      "containers": [{
        "name": "nfs-rwx-test-pod-2",
        "image": "registry.access.redhat.com/ubi9/ubi-minimal:latest",
        "command": ["sleep", "3600"],
        "volumeMounts": [{
          "name": "v",
          "mountPath": "/mnt/nfs-test"
        }]
      }],
      "volumes": [{
        "name": "v",
        "persistentVolumeClaim": {
          "claimName": "nfs-rwx-test-pvc"
        }
      }]
    }
  }'

Wait for the second pod:

oc wait --for=condition=Ready pod/nfs-rwx-test-pod-2 -n cbci-nfs-demo --timeout=60s

Then read the file created by the first pod:

oc exec -n cbci-nfs-demo nfs-rwx-test-pod-2 -- cat /mnt/nfs-test/hello.txt

The second pod must be able to see the file created by the first pod.

Sample output;

rwx-check-1786379962

This verifies the important workload behavior: the same PVC can be mounted read-write by multiple pods and changes made through one mount are visible through another. OpenShift’s definition of RWX is specifically read-write mounting by many nodes; the two-pod test provides an additional practical verification of the shared filesystem behavior.

5. Verify the provisioned directory on the NFS server

If you have administrative access to the NFS server, independently verify that the test file exists in the export.

For example:

On the NFS server

find <NFS_EXPORT_PATH> -name hello.txt -exec cat {} \;

Where:

<NFS_EXPORT_PATH>

should be replaced with the actual NFS export, for example:

find /srv/nfs/cloudbees-ci -name hello.txt -exec cat {} \;

The file should contain the value written by the test pod.

This provides an independent check that the write reached the NFS export rather than relying solely on the Kubernetes-side view. The upstream provisioner documentation likewise recommends checking the NFS server after performing its test write.

6. Record the validation result

Before proceeding, confirm all of the following:

• The intended StorageClass exists.
• The StorageClass references the intended provisioner.
• The provisioner is running and configured for the intended NFS export.
• The test PVC reaches Bound.
• The resulting PV is associated with the test PVC.
• A pod can mount the PVC successfully.
• The pod can create and read a file.
• A second pod can mount the same PVC and read the first pod’s file.
• The file is visible on the NFS export when server-side verification is available.
• No permission or security workaround was introduced solely to make the test pass.

If any item fails, stop and resolve the underlying storage configuration before continuing.

7. Clean up

After the validation is complete:

oc delete pod nfs-rwx-test-pod nfs-rwx-test-pod-2 -n cbci-nfs-demo

oc delete pvc nfs-rwx-test-pvc -n cbci-nfs-demo

oc delete project cbci-nfs-demo

If the StorageClass or provisioner is configured to retain provisioned directories, also verify the resulting directory on the NFS server and remove the test data according to the retention policy you actually configured.

8. Proceed to performance testing only after functional validation

Only after the end-to-end functional test succeeds should you proceed to the fio benchmarking described later in this guide.

Do not use benchmark results as evidence that the storage configuration is correct. Functional validation and performance validation are separate tests. First establish that the storage provisions, binds, mounts, and provides the required shared read/write behavior. Then measure whether its performance is suitable for the intended CloudBees CI workload.

Likewise, do not point a production Operations Center or Managed Controller at the StorageClass until this disposable validation has succeeded.

For CloudBees CI specifically, verify the storage configuration against the CloudBees documentation and the exact CloudBees CI/chart version being deployed. Current CloudBees documentation identifies Persistence.StorageClass as the Helm value used to configure the storage class, and notes that storage-class changes do not modify an existing PV/PVC; a new volume and data migration are required.

The governing rule for this guide is simple: current configuration must be researched and validated before it is presented as fact. Do not carry forward a key path, command, Helm value, or behavior from an older KB or installation guide without confirming that it applies to the versions currently being deployed.

Part 3: Installing CloudBees CI with HA Managed Controllers on NFS

This section distinguishes two separate requirements that must not be conflated:

• The Operations Center requires persistent storage for its own $JENKINS_HOME.
• A Managed Controller running in HA mode requires shared ReadWriteMany storage so that its replicas can access the same $JENKINS_HOME.

CloudBees CI on modern cloud platforms provides built-in fault tolerance for the Operations Center, which automatically restarts it when unhealthy. CloudBees documents HA (active/active), shared-storage requirements, replicas, and load balancing as features of Managed Controllers, not as an HA mode that must be enabled on the Operations Center.

Therefore, do not configure the Operations Center’s storage as RWX/NFS merely because the Managed Controllers will run in HA mode. The NFS-backed RWX StorageClass is required for the HA Managed Controllers.

Pre-installation checklist for OpenShift

CloudBees’ current OpenShift requirements specify:

• A currently supported OpenShift version.
• Network access to the required container images, either through Docker Hub or a private registry.
• An OpenShift project with permission to create Role and RoleBinding objects.
• DNS and TLS requirements prepared for the CloudBees CI deployment.
• A defined default StorageClass for persistent storage.
• Containers running as non-root and non-privileged on OpenShift.

For HA Managed Controllers, CloudBees additionally requires a StorageClass supporting ReadWriteMany.

The resulting storage requirement is therefore:

• Operations Center: persistent storage; a suitable default StorageClass is sufficient.
• Non-HA Managed Controller: persistent storage; RWX is not inherently required.
• HA Managed Controller: shared persistent storage; the StorageClass must support RWX.

Do not make RWX/NFS a blanket requirement for the entire CloudBees CI installation. It is specifically required for the shared filesystem used by HA Managed Controller replicas.

Install the Operations Center

Install the Operations Center using the standard CloudBees CI OpenShift installation procedure.

The Operations Center still requires persistent storage. CloudBees documents that the Operations Center has its own $JENKINS_HOME, and that CloudBees CI dynamically provisions persistent storage for the Operations Center and for Managed Controllers.

However, the Operations Center does not need RWX storage simply because the Managed Controllers will later run in HA mode.

If the cluster’s default StorageClass provides suitable persistent storage for the Operations Center, allow CloudBees CI to use the default rather than unnecessarily forcing the Operations Center onto the NFS StorageClass.

For example, the Helm configuration does not need to override the Operations Center storage class solely to support controller HA:

OperationsCenter:
  HostName: cjoc.example.com

The important distinction is that omitting an explicit storage-class override does not mean that the Operations Center has no storage. It means that the configured/default StorageClass is used for its persistent storage.

CloudBees’ current OpenShift requirements explicitly require a defined default StorageClass, while the additional RWX requirement is listed separately under “Storage requirements for High Availability on OpenShift.”

Do not point the Operations Center at NFS without a reason

If the existing default StorageClass is appropriate for the Operations Center, there is no HA-related reason to change it to NFS storageclass.

Use the NFS-backed RWX StorageClass where it is actually required: the $JENKINS_HOME of a Managed Controller whose replicas will run in HA mode.

This also avoids unnecessarily making the Operations Center dependent on the NFS infrastructure simply because the Managed Controllers use it.

Enable HA on a Managed Controller

HA is configured on the Managed Controller.

CloudBees’ current HA documentation describes HA as multiple replicas of a single logical Managed Controller sharing storage. When HA is enabled, the Operations Center dashboard exposes a Pod status column showing the number of controller replicas and how many are available.

When configuring a Managed Controller for HA, configure:

• High Availability: enabled
• Managed controller replicas: desired number of replicas
• Maximum number of replicas: only when using autoscaling
• CPU threshold: when autoscaling is enabled
• Storage Class Name: the StorageClass providing the shared RWX filesystem

CloudBees specifically documents the Storage Class Name as the storage class for the replicas’ shared filesystem. If it is left empty, the cluster’s default StorageClass is used.

For this guide, the HA controller should therefore use:

Storage Class Name: managed-nfs-storage
High Availability: enabled
Managed controller replicas: 3

The critical relationship is:

HA replicas
     ↓
shared $JENKINS_HOME
     ↓
ReadWriteMany
     ↓
managed-nfs-storage

Do not interpret this as requiring the Operations Center to use the same StorageClass.

Deployment rather than StatefulSet

When HA is enabled, CloudBees CI manages the controller pods with a Deployment rather than a StatefulSet. CloudBees’ migration documentation states that existing YAML customizations must be adjusted to replace StatefulSet with Deployment.

This matters more than it appears, because the failure mode is silent rather than loud. Advanced YAML supplied with a kind that does not match the workload CloudBees actually generates is not rejected with an error. It is accepted and ignored, so the controller starts successfully while the customization never takes effect.

After supplying Advanced YAML to an HA Managed Controller, verify that the setting reached the live object rather than assuming it applied:

oc get deployment <controller-name> -n <namespace> -o yaml | grep -A5 <expected-field>

If the field is absent, the override was ignored. Check the kind first.

Configuration as Code

If Managed Controllers are being provisioned through CasC, use the current CloudBees CI items.yaml schema for the exact CloudBees CI version being deployed.

Do not copy a replication example from an older release and assume that the schema is unchanged. The HA settings are version-sensitive and should be validated against the current items.yaml reference before deployment.

At a conceptual level, the configuration must express:

items:
  - kind: managedController
    name: "controller-alpha"
    configuration:
      kubernetes:
        replication:
          config:
            replicas: 3
            maxReplicas: 5
            targetCPUUtilizationPercentage: 80

The exact supported fields and defaults must be checked against the items.yaml reference for the CloudBees CI release being installed.

If autoscaling is not required, configure only the replica count supported by that release rather than assuming that omitted fields have a particular default.

Install CloudBees CI

Once the storage requirements have been validated, install CloudBees CI using the current OpenShift installation procedure

Deploying CloudBees CI on OpenShift: Architecture, Installation, Security, and Production Best Practices

The CloudBees OpenShift installation documentation should be treated as the authoritative installation procedure for the release being deployed.

Before installation, verify the actual chart values for the release rather than assuming that values from an older CloudBees CI version remain valid.

Migrating an Existing non-HA Managed Controller to HA

Do not treat this as a simple StatefulSet edit.

CloudBees’ current migration documentation explicitly states that an HA Managed Controller requires a StorageClass supporting ReadWriteMany. If the existing controller does not already use RWX storage, CloudBees provides a migration procedure to move its $JENKINS_HOME to a new RWX-backed volume.

The current migration process is broadly:

1. Create and test the new RWX storage.
2. Create a new volume using the RWX StorageClass.
3. Synchronize the existing controller data to the new volume.
4. Stop the Managed Controller during the final synchronization window.
5. Synchronize the final changes.
6. Rename/rebind the PVCs as required by the CloudBees migration procedure.
7. Update the Managed Controller configuration to enable HA and use the new storage.
8. Start the Managed Controller on the new volume.
9. Verify the controller and all replicas.
10. Remove the old storage only after the migration has been fully validated.

CloudBees specifically notes that the controller’s $JENKINS_HOME can contain very large numbers of small files and that migration can therefore be I/O-intensive and time-consuming.

Do not manually edit a generated StatefulSet as the primary migration procedure. The Managed Controller is provisioned and managed by the Operations Center, and the current CloudBees migration procedure is based on changing the persistent volume/PVC configuration and then enabling HA through the Managed Controller configuration.

If the controller already uses a volume with RWX capabilities, CloudBees states that migration can be simpler because there is no need to move the data to another storage system; the storage configuration still needs to satisfy the HA requirements.

Always perform this migration against a non-production controller first.

Validate NFS Performance before Production

Functional validation and performance validation are separate activities. The previous section verified that managed-nfs-storage can provision an RWX PVC and that multiple pods can access the same volume. This section measures the performance of that same storage path.

Verify the documented NFS requirements

Complete these before benchmarking. Measuring a misconfigured export produces numbers that describe the misconfiguration rather than the storage.

Requirement	Check	Expected
NFS version 4.1 or higher (4.0 is not supported)	mount | grep <nfs-path> on a node	vers=4.1 or higher
Export options	grep <export> /etc/exports on the NFS server	rw,sync,no_subtree_check. async improves write performance, but the server acknowledges writes before they reach stable storage. Only consider async with a protected write cache, and record the trade-off.
nfsd thread count recorded	grep threads /etc/nfs.conf (RHEL 8 and 9) or grep RPCNFSDCOUNT /etc/sysconfig/nfs (RHEL 7)	Record the current value as part of the baseline. Do not raise it before measuring; see “Tuning the NFS Server Daemon” above.
SSD-backed export	Confirm with the storage owner	SSD recommended

For reference, the CloudBees NFS knowledge base article uses the following client mount options: rw,bg,hard,intr,rsize=32768,wsize=32768,vers=4.1,proto=tcp,timeo=600,retrans=2,noatime,nodiratime,async

Treat this as an example rather than a target configuration. As noted in the StorageClass section above, modern NFS clients negotiate read and write sizes, and 32768 is conservative by current standards. Note also that intr has had no effect since Linux kernel 2.6.25 and is retained only for historical compatibility. The async mount option here is a client-side setting and is unrelated to the async export option discussed above.

Create the temporary project

oc new-project nfs-performance-test

All resources created below are placed in the current nfs-performance-test project.

Create the test PVC

Use the same StorageClass that will be used by the HA Managed Controllers:

oc apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-fio-test-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: managed-nfs-storage
  resources:
    requests:
      storage: 5Gi
EOF

Wait for the PVC to become Bound:

oc get pvc nfs-fio-test-pvc

Do not continue until the PVC is Bound.

Create two FIO pods on the same PVC. The second is required for the concurrency test below.

for i in 1 2; do
oc apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: nfs-fio-test-${i}
spec:
  restartPolicy: Never
  containers:
    - name: fio
      image: quay.io/cloud-bulldozer/fio:latest
      command: ["/bin/sh", "-c", "sleep 3600"]
      volumeMounts:
        - name: nfs-test
          mountPath: /mnt/test-nfs
  volumes:
    - name: nfs-test
      persistentVolumeClaim:
        claimName: nfs-fio-test-pvc
EOF
done

oc wait --for=condition=Ready pod/nfs-fio-test-1 pod/nfs-fio-test-2 --timeout=120s

The cloud-bulldozer/fio image is used here because it ships fio preinstalled. Under the restricted-v2 SCC, OpenShift assigns an arbitrary UID to the container, so confirm the pod can write to the mount before running the benchmarks:

oc exec nfs-fio-test-1 -- sh -c 'id; touch /mnt/test-nfs/.write-check && echo OK'

If this fails, the benchmark will fail for permission reasons rather than performance reasons. Resolve it using the same UID and ownership approach described in Part 1

Confirm the mount and the negotiated NFS version:

oc exec nfs-fio-test-1 -- sh -c 'df -h /mnt/test-nfs; mount | grep test-nfs'

Establish the single-client baseline

The small-block workload models JENKINS_HOME config and build-record I/O, which determines controller responsiveness. The large-block workload models artifact and log handling.

oc exec nfs-fio-test-1 -- \
  fio --name=metadata --directory=/mnt/test-nfs --rw=randrw --bs=4k \
      --size=512M --numjobs=8 --iodepth=16 --runtime=60 --time_based \
      --group_reporting

oc exec nfs-fio-test-1 -- \
  fio --name=throughput --directory=/mnt/test-nfs --rw=readwrite --bs=1M \
      --size=2G --numjobs=4 --iodepth=8 --runtime=60 --time_based \
      --group_reporting

Record four values per workload, for read and write separately: IOPS, BW, clat p99, and clat p99.99. Averages conceal stalls; percentiles do not.

While each run is active, capture server-side data on the NFS host. Without it, a poor result cannot be attributed to disk, network, or NFS configuration.

iostat -x 1        # %util near 100 indicates disk saturation
nfsstat -s         # rising badcalls or retransmits indicate server or network fault
nfsiostat 1        # per-operation NFS latency

On an OpenShift worker node running one of the test pods:

nfsstat -c         # client-side NFS call and retransmission counts
nfsiostat 1        # per-operation NFS latency, read from client mountstats

nfsiostat reads /proc/self/mountstats on the NFS client, so it must be run on the OpenShift node, not on the NFS server.

Test HA-level concurrency

HA controllers hold the same RWX volume simultaneously. This result carries more weight than the single-client baseline. Both clients must run at the same time; sequential runs measure nothing new.

oc exec nfs-fio-test-1 -- \
  fio --name=metadata --directory=/mnt/test-nfs --rw=randrw --bs=4k \
      --size=512M --numjobs=8 --iodepth=16 --runtime=60 --time_based \
      --group_reporting > pod1-concurrent.log 2>&1 &

oc exec nfs-fio-test-2 -- \
  fio --name=metadata --directory=/mnt/test-nfs --rw=randrw --bs=4k \
      --size=512M --numjobs=8 --iodepth=16 --runtime=60 --time_based \
      --group_reporting > pod2-concurrent.log 2>&1 &

wait

Interpret the results

CloudBees defines no numeric pass/fail threshold, so these ranges are used to detect misconfiguration, not to gate deployment. They describe normal behaviour for SSD-backed NFS over a LAN.

Metric	Normal	Investigate
4k random IOPS	500 to 5,000+	Under 100
4k p99 latency	Under approximately 20ms	Sustained hundreds of ms
1M sequential bandwidth	100 to 400+ MB/s, link dependent	Under 20 to 30 MB/s
Read vs write symmetry	Within approximately 2x	Write an order of magnitude below read

Two rules govern the deployment decision.

• Rule 1: Single-client results below the normal range are a configuration fault, not a hardware ceiling. The usual causes are a sync export, NFS 4.0, an undersized RPCNFSDCOUNT, or rsize/wsize negotiated lower than requested. HA will not improve a poor single-client result.
• Rule 2: Concurrency must degrade proportionally, not collapse. Sum the two pods’ IOPS from above. A combined total near the single-client baseline, with latency rising by a modest multiple, confirms the storage shares capacity correctly and HA is viable. A combined total well below the baseline, or p99 latency rising by an order of magnitude, indicates the storage path saturates under concurrency. Resolve this through NFS server tuning, a dedicated export, or an alternative RWX backend before deploying HA.

Record the single-client and concurrent figures side by side. That comparison is the deliverable of this section and becomes the environment baseline for future capacity planning.

Clean up

oc delete project nfs-performance-test

If the namespace stalls in Terminating:

oc get namespace nfs-performance-test -o json \
  | jq 'del(.spec.finalizers)' \
  | oc replace --raw "/api/v1/namespaces/nfs-performance-test/finalize" -f -

Confirm the outcome on the NFS server. With archiveOnDelete: "true", the provisioner renames the directory with an archived- prefix rather than removing it. Delete the archived benchmark directory manually once the results have been recorded.

Common Pitfalls

Using RWX for the Operations Center simply because controllers use HA

• This is unnecessary.
• The Operations Center requires persistent storage, but CloudBees’ OpenShift requirements distinguish the normal default StorageClass requirement from the additional RWX requirement for HA.
• Use the default StorageClass for the Operations Center unless there is an explicit architectural reason to use NFS.

Assuming “persistent storage” means NFS

• CloudBees requires dynamically provisioned persistent storage for the Operations Center and Managed Controllers. HA adds the requirement that the Managed Controller’s storage support RWX.

Configuring HA without RWX

• An HA Managed Controller’s replicas must share the same filesystem. CloudBees explicitly requires a ReadWriteMany StorageClass for HA.
• Do not enable HA against an RWO-only storage class.

Assuming a Bound PVC proves that HA storage works

• Bound proves the provisioner works. It proves nothing about concurrent access, throughput, or latency.
• Validate in three stages before production: provisioning, multi-pod concurrent access, then measured performance.
• CloudBees states that the storage must provide sufficient IOPS, throughput, and low latency, but publishes no numeric threshold. Establishing the environment baseline is the operator’s responsibility.
• This staged validation is engineering practice rather than a documented CloudBees procedure.

Manually modifying generated controller resources

• The Operations Center manages Managed Controller Kubernetes resources. Configure the Managed Controller through the supported CloudBees configuration mechanisms rather than treating its generated Deployment/StatefulSet as an independently managed workload.

Using soft mounts to avoid hangs.

• Trades data integrity for the appearance of resilience; soft returns I/O errors on timeout and can corrupt $JENKINS_HOME. Use hard and fix the capacity problem underneath.

Summary

What the storage architecture looks like:

                         CloudBees CI
                              │
        ┌─────────────────────┼─────────────────────┐
        │                     │                     │
 Operations Center    Managed Controller    Managed Controller
                         (non-HA)                 (HA, 3 replicas)
        │                     │                     │
   $JENKINS_HOME        $JENKINS_HOME          $JENKINS_HOME
        │                     │                     │
   RWO block storage    RWO block storage      RWX storage class
    [recommended]        [recommended]         managed-nfs-storage
                                                 [REQUIRED]
                                                     │
                                              NFS shared filesystem
                                                     │
                                    ┌────────────────┼────────────────┐
                                 Replica 1        Replica 2        Replica 3

Read the labels carefully, because they aren’t equally binding.

[REQUIRED] is documented. CloudBees states that to install HA on OpenShift you must set up a storage class with ReadWriteMany access mode, and that an existing controller not already on an RWX class must be migrated before it can run HA. That arrow is not negotiable.

[recommended] is engineering judgement, mine, not a CloudBees requirement. Nothing in the documentation says the operations center or a non-HA controller must avoid shared storage. The reasoning is that RWX buys you nothing without HA and NFS is slower than block storage for the small-file, metadata-heavy I/O Jenkins generates, so paying the shared-storage cost where it isn’t needed is waste.

There is a legitimate case against that recommendation, and it’s CloudBees’ own. In its EKS guidance, CloudBees notes that putting both the operations center and managed controllers on EFS (shared, RWX storage) lets pods be rescheduled onto healthy nodes during an availability-zone outage. If your OpenShift cluster spans failure domains and you want that rescheduling property for the operations center too, putting the OC on NFS is a deliberate availability trade, slower I/O in exchange for not being pinned to one zone’s block storage. Decide it on your own topology; don’t treat either arrangement as the mandated one.

So the one-line version: RWX where HA forces it, everything else your call, with a real trade-off to weigh if you’re multi-zone.

Further Reading

• CloudBees CI: Install HA on modern cloud platforms
• CloudBees CI: High Availability fundamentals
• CloudBees CI: Migrate an existing Managed Controller to HA