An InfiniBand link can be active while the application is still unable to reach its peers. For a shared AI cluster, the useful question is not simply “is the fabric up?” It is “which workload ports should communicate, through which partition, and can we prove the unwanted paths are blocked?” This guide provides a partition inventory worksheet, a worked 256-GPU example and a deployment test matrix.
Scope: this is a hypothetical design and an unexecuted fabric configuration example, not a benchmark or a validated production deployment. The worksheet arithmetic was executed locally. Configuration behavior is grounded in the version-pinned NVIDIA MLNX_OFED documentation and the UFM documentation linked below; confirm the corresponding behavior on your installed release.
The problem: tenant isolation without breaking collective traffic
Imagine two research teams sharing GPU infrastructure. Each team needs communication between its own workers, and both need a shared storage service. Giving everybody the same broad membership is easy to operate but does not express that separation. Giving every worker limited membership in its training partition can break the very worker-to-worker communication that distributed jobs require: limited members cannot communicate with other limited members of the same partition.[3]
The proposed design therefore separates training peer connectivity from shared-service connectivity. Within a team's training partition, workers receive full membership. Within a separate service partition, workers receive limited membership and approved service ports receive full membership. Full-to-full and full-to-limited communication are permitted by the partition membership model; limited-to-limited communication is not.[3]
Treat this as a fabric policy, not a complete security architecture. Our deployment checklist additionally requires workload authorization, host hardening, management-plane protection and storage permissions. Do not assume a partition number is a credential or that network reachability authorizes access to somebody else's checkpoint.
For the broader bring-up sequence, see the earlier 256-GPU acceptance checklist. This installment concentrates on membership correctness rather than repeating performance commissioning.
Understand three different identifiers before editing anything
OpenSM partition definitions use the low 15 bits of the P_Key value to identify the partition.[2] Membership is represented by the most-significant bit, so the same partition can appear with different full and limited values in a port's table.[3] The worksheet below computes those representations rather than treating them as separate tenants.
A P_Key value is also different from a P_Key table index. NVIDIA documents the indx0 option as requesting insertion in block 0, index 0.[2] Do not infer that a partition value such as 0x0101 belongs in table index 257. Record both value and observed index when diagnosing application selection. This guide deliberately avoids prescribing an application-specific index environment variable: verify that interface against the exact communication library and release you deploy.
Finally, inventory the port GUID, not merely the server name or a device label. The configuration format identifies member end ports using port GUIDs.[2] Our recommended inventory joins server, tenant, adapter, physical port, rail, subnet and port GUID. This makes an adapter replacement or cable move a reviewable policy change rather than a mysterious application outage.
Worked design: 256 GPUs, two tenants and separate rails
All quantities here are design assumptions, not hardware specifications. Assume 32 servers with eight GPUs each. Allocate 16 servers to tenant A and 16 to tenant B. Assume each server exposes eight fabric end ports, one in each of eight independent rail subnets. This is a selected topology for the example, not a claim that every eight-GPU server has that adapter layout.
On each rail, define three application partitions: tenant A training, tenant B training and shared service. Each compute end port joins its tenant partition as full and the service partition as limited. Each rail also has two hypothetical service end ports, both full members of the service partition. The default partition is handled separately and must be included in the implementation review.
Source: original Network freak diagram of this hypothetical design; no customer topology or device identifiers used.
Executed inventory worksheet
| Quantity | Calculation | Result |
|---|---|---|
| GPUs | 32 servers × 8 GPUs | 256 |
| Compute end ports across rails | 32 servers × 8 ports | 256 |
| Compute end ports per rail | 16 tenant A + 16 tenant B | 32 |
| Application partition instances | 3 partitions × 8 subnets | 24 |
| Compute application memberships | 256 ports × 2 partitions | 512 |
| Service application memberships | 2 service ports × 8 rails × 1 partition | 16 |
| Application membership assignments | 512 + 16 | 528 |
| Membership assignments per rail | 16 + 16 + 32 + 2 | 66 |
These are membership assignments, not switch ports, bandwidth, or the total hardware P_Key table requirement. The count excludes the default partition, subnet-manager endpoints and any vendor-required memberships. Check the supported table capacity on each actual device against its complete effective configuration. In particular, the busiest shared-service port may acquire more memberships if you choose separate per-tenant service partitions later.
Reusing a numeric P_Key across independent subnets does not make a shared control plane. NVIDIA states that OpenSM attaches to a specific local IB port and configures only the fabric connected to that port.[2] Our operational requirement is therefore to validate all eight rail subnets independently, even when policy is generated from one inventory template.
Reusable Python calculator
This worksheet was executed locally; it does not contact hardware or validate reachability. Edit the inputs for your topology. Do not use GPU count as a substitute for an actual adapter-port inventory.
servers = 32
gpus_per_server = 8
rails = 8
compute_ports_per_server = 8
service_ports_per_rail = 2
assert compute_ports_per_server == rails # assumption for this model
compute_ports = servers * compute_ports_per_server
compute_memberships = compute_ports * 2
service_memberships = service_ports_per_rail * rails
print('GPUs:', servers * gpus_per_server)
print('Application partition instances:', 3 * rails)
print('Compute end ports:', compute_ports)
print('Application memberships:', compute_memberships + service_memberships)
for base in (0x0101, 0x0102, 0x0200):
print(f'base={base:#06x} limited={base:#06x} full={base | 0x8000:#06x}')
The resulting membership representations are 0x0101/0x8101, 0x0102/0x8102 and 0x0200/0x8200 for limited/full respectively. The model's most important scaling variable is memberships per endpoint, not the size of the numeric partition space.
Implementation: generate policy, do not improvise it
1. Establish configuration ownership
Determine whether this fabric uses standalone OpenSM or a UFM-managed configuration. The cited UFM documentation says its generated partitions.conf cannot be manually modified and describes partitions.conf.user_ext as the supported extension mechanism, merged through UFM synchronization.[1] A hand-edited generated file is therefore the wrong source of truth for that documented UFM workflow.
For standalone OpenSM, the documented default configuration path is /etc/opensm/partitions.conf, with --Pconfig or -P available to select another file.[2] Inspect the actual running instance and service configuration rather than assuming that editing the default path changes the active manager.
Save the effective pre-change policy, software version, active and standby manager identities, and an endpoint membership snapshot. In our proposed process, a reviewed inventory generates the candidate policy and a second reviewer checks the diff. Keep recovery access on a path that does not depend on the partition being changed.
2. Review the default partition explicitly
The default partition cannot simply be removed according to the MLNX_OFED documentation.[2] More importantly, published defaults need careful reading: the UFM format reference describes all end ports as full members when the partition file is missing or inaccessible, and limited when an accessible file omits a Default rule.[1] The MLNX_OFED version cited here describes other end ports as partial members in its summary.[2]
Do not resolve that documentation difference by guessing which behavior applies to your deployment. Inspect your manager's effective policy and actual endpoint tables. Our acceptance condition is that no unintended alternate common full-membership path defeats the tenant separation. Test default-partition behavior in an isolated lab; deliberately removing a live configuration file is not an acceptable production experiment.
3. Create a small explicit policy example
The following is a partial, unexecuted standalone OpenSM syntax example for one rail. Every GUID is invented for illustration. Replace it with verified port inventory, and retain an explicitly reviewed default-partition policy outside this fragment. It is not a complete replacement file.
# Hypothetical rail: two sample workers per tenant, one service endpoint.
TenantA=0x0101,defmember=full :
0x0200000000000001, 0x0200000000000002;
TenantB=0x0102,defmember=full :
0x0200000000000011, 0x0200000000000012;
SharedService=0x0200,defmember=limited :
0x0200000000000001, 0x0200000000000002,
0x0200000000000011, 0x0200000000000012,
0x02000000000000f1=full;
The format supports explicit P_Keys, a default membership for each list, and per-port membership overrides.[2] Full membership is intentional for training peers; limited membership is intentional for clients of the shared service. Never add ALL=full as a troubleshooting shortcut without recognizing that you are changing the intended isolation model.
This fragment intentionally omits ipoib. NVIDIA documents that the ipoib flag requests creation of the IPoIB broadcast group.[2] If the deployment requires IP over InfiniBand, design and validate that separately, including host interface selection and multicast properties. Do not treat this RDMA-oriented policy fragment as a ready-to-run IPoIB setup.
4. Apply through the supported manager workflow
In the cited MLNX_OFED release, new partitions and port membership changes can be applied by signaling the manager with HUP, whereas changes to existing partition IPoIB and multicast-group flags require an SM restart.[2] That is a release-scoped distinction, not a universal instruction to signal every OpenSM process on a host.
Identify the exact instance and follow its service-management procedure. For UFM, use the documented configuration ownership and synchronization path instead.[1] Schedule the change, drain affected workloads when the rollback plan requires it, and keep performance and security acceptance separate. Successful parsing only proves that the manager accepted a configuration, not that every expected endpoint received it.
Verification: prove the permitted path and the forbidden path
A useful acceptance record identifies source port, destination port, selected P_Key, selected transport, expected outcome and observed outcome. A failed application run alone does not prove isolation: the application might simply be misconfigured. Conversely, a successful test over an unrelated Ethernet interface does not prove the InfiniBand policy works.
Use the same test program and parameters for positive and negative controls where possible. Record transport selection from diagnostic output, then correlate the result with effective endpoint membership. Do not fabricate a throughput target from a nominal port speed; establish the performance baseline separately using the commissioning workflow.
Source: original Network freak acceptance workflow; proposed checks, not measured test results.
| Test | Proposed expected result | Evidence required |
|---|---|---|
| A worker to A worker, tenant A partition | Allowed | Selected partition and transport, completed transfer |
| B worker to B worker, tenant B partition | Allowed | Equivalent evidence for tenant B |
| A worker to B worker, attempted tenant path | Denied | Negative control with functioning endpoints |
| Worker to approved full service member | Allowed | Selected service partition and application authorization |
| Worker to worker using limited service membership | Denied | Same service partition selected at both endpoints |
| Unassigned spare compute port | No tenant application access | Effective memberships plus negative probe |
| Replacement adapter | Access only after inventory approval | Old membership removed, new GUID reviewed |
| Manager failover | Reviewed policy remains effective | Repeat positive and negative controls after takeover |
| Each independent rail | Same intended policy, correct local GUIDs | Separate evidence set per subnet |
The membership expectations in the first five rows follow the full/limited communication model.[3] The remaining rows are our recommended operational controls, not vendor-certified test results. Run negative tests only against endpoints you own and have authorized for the maintenance exercise.
For the example, testing every unordered compute-port pair on every rail would require 3,968 pair selections, before multiple P_Keys, direction-specific checks or service endpoints. That figure is an executed calculation of 8 × 32 × 31 / 2, not a recommendation to launch thousands of simultaneous RDMA jobs. Use exhaustive low-impact policy comparison and a staged traffic matrix; expand active coverage according to risk and maintenance capacity.
Troubleshooting matrix: active links but broken jobs
| Symptom | First discriminating check | Safer next action |
|---|---|---|
| Training peers cannot communicate | Are both limited in the selected partition? | Compare effective tables with intended tenant full membership |
| Storage works but collective startup fails | Does the tenant partition exist on every job port? | Test worker pairs explicitly, not just service reachability |
| Only one rail fails | Correct GUIDs and active manager instance for that subnet? | Compare that rail with its own approved inventory |
| Membership disappears after a change | Is UFM regenerating the file you edited? | Restore supported configuration ownership |
| Policy works until manager takeover | Does standby load the same reviewed policy? | Repair configuration distribution before retrying failover |
| Tenant crossover still succeeds | Is another common partition or transport being used? | Inspect actual transport and all effective memberships |
| Configuration change appears ignored | Did you alter membership or multicast properties? | Check release-specific apply/restart requirements |
For manager-level diagnosis, NVIDIA documents /var/log/opensm.log as the default log and SUBNET UP as the message indicating successful subnet setup.[2] Use that as one observation, not as the entire application acceptance test. Capture timestamps so manager sweeps, endpoint changes and job failures can be correlated.
A rollback should restore a known reviewed policy, reapply it using the supported method, and rerun both positive and negative controls. “Everything connects again” is not an adequate rollback result for an isolation change. Preserve the failed candidate and observations for diagnosis without leaving broad emergency memberships active.
Alternatives, bottlenecks and failure domains
Shared-service partition with limited clients: this is our example. It makes the intended client-to-service pattern clear but requires explicit application partition selection and careful service endpoint protection. Limited membership is not suitable for a training peer group that needs all-to-all reachability.[3]
Separate service partitions per tenant: consider this when operational ownership or service endpoint separation matters more than minimizing memberships. Recalculate service endpoint table use and maintain a per-tenant acceptance matrix. Do not assume extra P_Keys automatically produce application isolation inside a shared storage server.
Dedicated fabrics or dedicated service endpoints: consider these when your threat model or operating model demands a simpler physical boundary. Compare acquisition cost, spare capacity, management overhead and the consequences of a failed shared component. This is an architecture decision, not a promise that one option is universally secure.
Our design does not reserve bandwidth merely by assigning partitions. Require a separate congestion, quality-of-service and workload-placement review before promising tenant performance. Shared service endpoints, rail switches, manager configuration and the inventory generator are all failure domains worth documenting. A correct P_Key table will not repair an overloaded storage target or an incorrectly placed distributed job.
Use the Slurm placement checklist alongside these checks when scheduler allocations and fabric membership disagree. Browse the AI Infrastructure hub and Data Center hub for the surrounding design topics.
Deployment sign-off checklist
- Every compute and service port has a verified GUID, subnet, owner and intended membership.
- The complete policy includes a reviewed default partition and device-capacity check.
- Training peer groups use intentional membership rather than inherited defaults.
- UFM or standalone OpenSM ownership is documented; generated files are not hand-maintained.
- The exact software release and apply procedure are recorded.
- Positive and negative tests identify the actual transport and selected partition.
- Adapter replacement and spare-port onboarding require inventory approval.
- Standby takeover and rollback preserve isolation, not merely connectivity.
- Storage authorization and management-plane controls are reviewed separately.
- Performance baselines and security acceptance have separate pass criteria.
Practical takeaway: begin with a port-level policy inventory, calculate the membership footprint, then prove both connectivity and denial on every rail. For this hypothetical 256-GPU design, the useful planning output is 528 application membership assignments and a repeatable test matrix—not a claim that a green link or one successful collective validates tenant isolation.
Post a Comment