NCP-AII Sample Questions Answers

The correct command is show asic-version. In NVIDIA Spectrum and Mellanox/NVIDIA switch environments, system software version alone does not prove that all switch modules or ASIC-related firmware components are aligned. NVIDIA documentation states that after firmware updates, administrators should run show asic-version; this command lists switch modules with their firmware versions and should be checked to ensure versions match the expected default or approved firmware baseline. show version is useful for the operating system or software image level, but it does not provide the same module-level ASIC firmware validation. show interfaces status confirms port state, but ports can be up even when firmware is inconsistent or unsupported. show inventory is helpful for asset identification and serial numbers, but it is not the primary firmware-compliance command. During AI cluster bring-up, switch firmware consistency matters because inconsistent module firmware can cause unpredictable link behavior, degraded fabric performance, or supportability issues before NCCL, RDMA, or Spectrum-X workload validation.

The correct best practice is to install only the GPU and DOCA driver versions that are validated for the current operating system, kernel, hardware platform, and NVIDIA software release. In NVIDIA AI infrastructure, driver compatibility is critical because GPU drivers, CUDA libraries, DCGM, Fabric Manager, DOCA-OFED, network adapters, DPUs, and kernel modules must align. Installing the newest driver without checking compatibility can cause driver/library mismatches, failed nvidia-smi, broken RDMA, or unsupported DOCA behavior. NVIDIA DGX documentation notes that DGX OS updates include coordinated OS, kernel, GPU driver, CUDA toolkit, and DCGM updates, and NVIDIA AI Enterprise provides support matrices to confirm supported platform combinations. DOCA-OFED is also tied to the operating system and kernel environment, so version selection must be deliberate. Legacy drivers may lack required fixes or hardware support, and default distribution drivers are usually not sufficient for validated DGX or AI Enterprise deployments. After installation, administrators should reboot if required, validate GPU visibility, confirm networking driver status, and run health checks before returning the node to production.

Updating an NVIDIA DGX system (like the H100) is a multi-layered process because the system contains numerous programmable logic devices, including CPLDs, FPGAs, and the EROT (Electrically Resilient Root of Trust) modules. Many of these low-level hardware components cannot be updated via a simple operating system reboot. NVIDIA ' s official firmware update procedure requires a specific sequence to " commit " the new images to the hardware. First, the update utility (like nvfwupd) writes the images to the flash memory. To activate them, a " Cold Power Cycle " (removing and restoring power) is necessary to force the hardware to reload from the newly written flash blocks. Furthermore, because the BMC (Baseboard Management Controller) orchestrates the power-on sequence and monitors the EROT, it must be reset (Option D) to synchronize its state with the new component versions. Finally, an " AC Power Cycle " ensures that even the standby-power components, such as the power delivery controllers and CPLDs, undergo a full hardware reset. Skipping these steps can result in " Incomplete " or " Mismatched " firmware versions, where the OS reports one version while the hardware continues to run old, potentially buggy code in the background.

When a DGX system exhibits high fan speeds and performance degradation, it is typically engaging in Thermal Throttling. High-performance GPUs like the A100 or H100 will automatically reduce their clock speeds (and thus performance) if they exceed safe temperature thresholds. The first and most critical diagnostic step is to run nvidia-smi. This utility provides immediate, real-time telemetry on GPU temperatures, power draw, and " Clocks Throttle Reasons. " By reviewing the output, an administrator can see if " Thermal " is listed as the reason for reduced clocks. This identifies whether the issue is environmental (blocked airflow/hot aisle temperature) or hardware-specific (a failed GPU thermal interface or a dead internal fan). Running more workloads (Option A) would exacerbate the heat, while a power drain (Option C) is a " last resort " that doesn ' t provide diagnostic data. nvidia-smi provides the evidentiary data needed to determine if an RMA (Return Merchandise Authorization) is required for the GPU tray.

NVIDIA Multi-Instance GPU (MIG) technology, introduced with the Ampere architecture (A100) and enhanced in Hopper (H100), allows a single physical GPU to be partitioned into multiple isolated instances. To manage these instances and their scheduling behavior (such as moving from a default time-sliced scheduler to a fixed-share or " MIG " partitioned mode), the nvidia-smi utility is used. The command nvidia-smi -i 0 -mig 1 enables MIG mode on the first GPU (index 0). Once MIG is enabled, the administrator can create specific GPU instances with dedicated compute and memory resources. This " Fixed Share " approach ensures that one tenant ' s workload does not impact the performance or latency of another, which is critical for deterministic AI inference and multi-tenant cloud environments. Option A and B refer to VMware ESXi specific commands which are not the primary method for raw hardware configuration in standard AI infrastructure, and Option D is for network adapter configuration.

Firmware consistency is a pillar of stable InfiniBand fabric performance. When a cluster is expanded, new transceivers or cables may arrive with newer or older firmware than the existing base, leading to " FW Version Mismatch " alerts in management consoles like UFM (Unified Fabric Manager). The flint tool (or mstflint) is the correct utility for querying the specific firmware levels embedded within the transceivers. While iblinkinfo provides data on link speeds and port states, it does not provide the deep hardware-level firmware telemetry required for version validation. flint allows the administrator to query the device, compare the current burn version against the target image, and perform the necessary updates to bring the cluster into a uniform state. In NVIDIA AI infrastructure, maintaining uniform firmware across the fabric ensures that features like Adaptive Routing and Congestion Control operate predictably. Without version parity, inconsistent behavior in Forward Error Correction (FEC) or link-up negotiation can lead to intermittent performance drops that are difficult to diagnose at the application (NCCL) level.

NVIDIA NeMo and large language model (LLM) training workloads are extremely demanding on HBM (High Bandwidth Memory). Out-of-Memory (OOM) errors often occur not because the total dataset is too large, but because memory fragmentation or sudden spikes in allocation (spikes during data shuffling or batch loading) exceed the available GPU memory. To mitigate this during intensive burn-in tests, engineers utilize the RMM (RAPIDS Memory Manager) library, which provides an asynchronous allocator. Enabling RMM asynchronous allocation allows the system to pre-allocate a pool of memory and manage it more efficiently than the standard CUDA allocator, reducing the overhead of constant allocations and deallocations. Furthermore, setting a specific blocksize (e.g., 1GB) for data loading ensures that the data ingestion pipeline reads data in manageable, deterministic chunks. This prevents the system from attempting to load massive files entirely into memory at once, which is the primary cause of OOMs in question-answering tasks involving large Parquet or JSON datasets. Switching to FP32 (Option B) would actually double the memory footprint and increase the likelihood of an OOM error.

Managing a large-scale AI fabric requires centralized visibility into the physical layer. The NVIDIA Unified Fabric Manager (UFM) provides a comprehensive Dashboard for InfiniBand networks. To check transceiver firmware—which is critical for ensuring feature parity and stability across the fabric—the administrator can use the UFM Enterprise GUI. By navigating to the " Devices " section and selecting a specific switch, the " Cables " tab will aggregate telemetry for every occupied port. This view displays the manufacturer, part number, and the specific firmware version of the transceivers (LinkX) or Active Optical Cables (AOC). This consolidated view is far more efficient than manual CLI queries (Option C) for 200+ ports. Maintaining uniform firmware across transceivers ensures that optimizations like Adaptive Routing and Congestion Control perform consistently across the entire 400G or 200G fabric.

GPUDirect Storage is the most effective solution because it reduces unnecessary CPU involvement in the data path between storage and GPU memory. In a traditional AI data pipeline, checkpoint data may move from storage into system memory, be processed by the CPU, and then be copied into GPU memory. This creates CPU overhead and can leave GPUs idle while waiting for data. GPUDirect Storage allows compatible storage systems, file systems, drivers, and NVIDIA GPUs to transfer data more directly into GPU memory. This improves checkpoint reload performance, reduces CPU copy overhead, and helps keep GPUs saturated during training. Increasing batch size may reduce the frequency of storage access, but it does not solve the checkpoint transfer bottleneck. SATA SSDs in RAID 0 may improve local sequential throughput, but they are not ideal for distributed training or shared cluster data access. Adding more GPUs can worsen the issue if the storage path is already the bottleneck. In NVIDIA AI infrastructure, storage-to-GPU optimization is critical for large model training, checkpointing, recovery, and maintaining high accelerator utilization.

The NVIDIA System Management (NVSM) tool is the definitive CLI utility for monitoring the health of DGX platforms. While replacing a PSU (Power Supply Unit) is a common maintenance task, verifying that the new component is correctly integrated into the system’s health model is mandatory. While nvsm show power would provide specific data regarding wattage and voltage for the PSU, the most comprehensive way to ensure the replacement hasn ' t caused secondary issues or that the system hasn ' t remained in a " Degraded " state is to run nvsm show health. This command performs a global check across all subsystems: GPUs, NVLink switches, storage, fans, and power. If the PSU replacement was successful and the system is back to full redundancy, nvsm show health will return a " Healthy " status. In an AI factory setting, where DGX H100 nodes pull significant power, ensuring that all 6 PSUs (in an N+N or N+1 configuration) are not only physically green but logically acknowledged by the Baseboard Management Controller (BMC) is critical for preventing unexpected shutdowns during high-load training iterations.

The correct option is export HPL_OOC_SAFE_SIZE=4.0. NVIDIA HPL out-of-core mode allows matrix data that exceeds GPU memory capacity to be placed in host memory, but the GPU still needs reserved memory for drivers and runtime overhead. NVIDIA documents HPL_OOC_SAFE_SIZE as the amount of GPU memory, in GiB, reserved for the driver and not used by HPL out-of-core mode; increasing it is recommended when GPU out-of-memory errors occur. HPL_OOC_MODE=0 disables out-of-core mode, which would not help run a larger 40B matrix. HPL_OOC_NUM_STREAMS=8 changes the number of CUDA streams used for out-of-core operations, but it does not reserve driver memory. HPL_OOC_MAX_GPU_MEM=90 limits total GPU memory use, but the specific variable intended to leave safe driver space is HPL_OOC_SAFE_SIZE. During cluster burn-in, this setting helps preserve test validity while avoiding false failures caused by memory reservation issues rather than actual hardware instability.

The primary purpose of an NCCL burn-in test is to stress GPU communication links and expose hardware or interconnect problems before the cluster is released to production. NCCL tests exercise collective communication patterns such as all-reduce, broadcast, reduce-scatter, and all-gather. These operations are central to distributed AI training, where GPUs across multiple servers must exchange gradients and model data efficiently. A burn-in test runs communication repeatedly over time, helping reveal unstable cables, weak links, switch issues, HCA problems, GPUDirect RDMA faults, driver mismatches, topology issues, or intermittent errors that may not appear in a short validation. GPU detection and driver visibility are normally checked with nvidia-smi and related health commands, not an NCCL burn-in. NCCL does not automatically tune deep learning frameworks, and it is not intended to replace application-level benchmarking with real user training scripts. In NVIDIA AI infrastructure, NCCL burn-in is a pre-production confidence test that validates the fabric under sustained GPU communication load, reducing the risk of failed large-scale training jobs.

In a large-scale Spectrum-X Ethernet fabric, " East-West " traffic refers to the cross-rack communication between compute nodes. To validate the " Bisectional Bandwidth " (the throughput between two halves of the cluster), administrators use NCCL tests with specific environment variables to control traffic patterns. The NCCL_TESTS_SPLIT variable is used to partition the GPUs into distinct groups for the benchmark. Setting NCCL_TESTS_SPLIT= " DIV 8 " is a standard configuration for multi-node testing on 8-GPU systems. It effectively divides the total number of GPUs by the node count, creating a test environment where each GPU communicates with its corresponding rank on other nodes. By combining this with -g 1 (one GPU per process) across multiple nodes, the engineer can force data to travel across the leaf-and-spine switches rather than staying within the NVLink fabric of a single node. This isolates the physical network performance from the internal GPU-to-GPU bandwidth, providing a true measurement of the fabric ' s ability to handle high-speed AI traffic.

The result should be interpreted as optimal in the context of the provided ClusterKit NCCL bandwidth test. ClusterKit is designed for high-performance cluster validation and includes GPU communication tests such as GPU-GPU latency, GPU-GPU bandwidth, GPU-host latency, and NCCL bandwidth and latency. A reported 350 GB/s value in this type of question normally represents aggregate NCCL communication bandwidth across GPUs, not the raw line rate of a single 400G network port. If ClusterKit reports this result as part of a healthy NCCL bandwidth run, it indicates that the GPU communication path, RDMA stack, and fabric are performing as expected. Option A is incorrect because it incorrectly references HDR and assumes a single-link expectation above 390 GB/s. Option B incorrectly jumps to FEC tuning without evidence of link errors, retransmissions, or degraded counters. Option D is also incorrect because CPU stress testing does not validate NCCL GPU-to-GPU fabric bandwidth. In production validation, this result should still be considered alongside consistency, error counters, NCCL logs, topology, and reference values for the exact DGX, GPU, HCA, and switch configuration.

NVIDIA Base Command Manager (powered by Bright) relies on a High Availability (HA) pair of head nodes to ensure the AI factory remains operational if the primary controller fails. The cmsh (Bright Shell) is the definitive management interface for this stack. Running the status command within cmsh provides a real-time summary of the cluster’s health, specifically identifying which head node currently holds the " Active " role and which is in " Standby. " It also monitors the status of the heartbeat and the synchronization of the database (MariaDB/MySQL). While nvsm (Option B) is essential for DGX hardware health, it does not manage the logic of the Bright Cluster Manager software. Checking connectivity via ping (Option D) only confirms network reachability but does not verify that the management services (CMDaemon) are actually ready to perform a stateful failover. Thus, cmsh status is the only verified way to confirm that the control plane ' s redundancy logic is active.

The correct answer is mig-parted. NVIDIA MIG Partition Editor, commonly called mig-parted, is designed to automate and manage Multi-Instance GPU partitioning profiles, especially in environments where repeatable MIG layouts are required. An H100 GPU can be divided into multiple isolated GPU instances, and a seven-slice configuration is a common profile pattern for maximizing the number of smaller MIG instances on a single GPU. While nvidia-smi can enable MIG mode and manually create or destroy GPU instances, mig-parted is better suited when an administrator needs to apply defined MIG profiles consistently across HGX systems or multiple nodes. nvcc is the CUDA compiler and has no role in configuring MIG partitions. nvlink-config is not the correct utility for MIG profile configuration. In NVIDIA AI infrastructure, MIG configuration must be controlled carefully because Kubernetes device plugins, GPU Operator, Slurm GRES configuration, and tenant scheduling depend on consistent GPU instance layouts. Using mig-parted helps reduce manual errors and supports repeatable server bring-up for shared inference or multi-tenant AI environments.

Interpreting NCCL (NVIDIA Collective Communications Library) results requires understanding the difference between theoretical link rate and effective bus bandwidth. A 400G (NDR) InfiniBand link has a theoretical unidirectional capacity of 50 GB/s. In an 8-GPU DGX system, where each GPU is typically mapped to its own 400G HCA, the aggregate theoretical bandwidth is 400 GB/s. However, NCCL performance benchmarks report Bus Bandwidth, which accounts for the mathematical overhead of collective operations like all_reduce. Achieving 350 GB/s (Option A) is widely considered optimal and representative of a healthy, well-configured fabric. This indicates that the system is effectively utilizing GPUDirect RDMA to bypass host memory and that the network switches are routing traffic without significant congestion or packet drops. Expecting > 390 GB/s (Option C) is unrealistic once protocol headers and algorithmic overhead are subtracted from the raw bit rate. Rerunning with CPU stress (Option D) would not provide insights into the InfiniBand/GPU fabric health.

The critical step is to query the device PSID using mstflint -d < PCI_ID > query before selecting or installing the firmware image. The PSID identifies the exact NVIDIA networking device variant, board configuration, and supported firmware branch. In a BlueField-3 or ConnectX-based AI infrastructure environment, using firmware built for the wrong PSID can cause installation failure, unsupported behavior, or device instability. This is especially important in cluster operations because firmware consistency must be maintained across nodes, DPUs, SuperNICs, and adapters. Checking BMC reachability may help with platform management but does not confirm firmware-device compatibility. Firmware file size is not a reliable compatibility indicator. Hash verification can confirm file integrity, but it does not prove the image matches the device PSID. NVIDIA MFT and mstflint workflows are designed for querying device identity and firmware details before burn or update operations, and NVIDIA documentation references PSID-based firmware querying in MFT workflows.

The correct answer is PM, which refers to the Performance Manager or performance-management function used for collecting performance counter information from the InfiniBand fabric. In NVIDIA AI infrastructure, cluster counters are essential for troubleshooting and optimization because they expose link-level and port-level behavior across switches, HCAs, and fabric paths. These counters can include transmitted and received data, symbol errors, link errors, congestion indicators, XmitWait, packet drops, and other telemetry used to detect performance degradation. SM, or Subnet Manager, is responsible for fabric discovery, LID assignment, routing, and maintaining the logical InfiniBand subnet, but it is not primarily the performance-counter collection function. GM and FM are not the standard answers for collecting cluster counters in this context. NVIDIA UFM Telemetry and diagnostic tooling collect and monitor InfiniBand port statistics such as bandwidth, congestion, errors, and latency, while UFM diagnostic output also includes PM counter dumps. In AI clusters, these counters help identify cabling faults, congestion, degraded links, and issues that can reduce NCCL and RDMA performance.

The best strategy is to ensure compatible storage protocols and APIs across the on-premises and cloud environments. AI workflows often move through multiple stages, including data ingestion, preprocessing, training, checkpointing, validation, inference, and analytics. If each environment uses incompatible storage interfaces, applications may require custom integration work, data copies, or workflow redesign. Using common protocols and APIs such as NFS for file-based access or S3-compatible APIs for object access improves portability and allows AI tools, data pipelines, and orchestration systems to operate more consistently across platforms. Standardizing on one vendor may reduce management complexity, but it can limit flexibility and create lock-in. Using only native cloud storage with middleware can work, but it may add complexity and inconsistent performance. Increasing network bandwidth helps data movement, but it does not solve protocol compatibility or application integration. In NVIDIA AI infrastructure, storage design must support high-throughput data access while also preserving operational flexibility across DGX, HGX, Kubernetes, Slurm, and cloud-connected AI environments.

The nvidia-ctk runtime configure command is a crucial step that modifies the Docker daemon configuration file (/etc/docker/daemon.json) to register the nvidia runtime. However, the Docker daemon only reads this configuration file during its initialization phase. Even though the toolkit is installed and the configuration file is updated, Docker will not be able to spawn GPU-accelerated containers until the service is refreshed. Executing sudo systemctl restart docker (or the equivalent for your container engine) is the mandatory final step. This forces Docker to reload its settings and recognize the NVIDIA Container Runtime as a valid option. Without this restart, attempting to run a container with the --gpus all flag will result in an error stating that the " nvidia " runtime is not found or is unconfigured. This is a common point of failure in automated AI infrastructure deployments where the configuration script finishes, but the service state remains stale.

The correct approach is to create a unique, strong administrative user and avoid default or weak credentials. NVIDIA DGX first-boot guidance states that the setup process creates an administrative user account for the system, BMC, and GRUB boot loader. DGX OS guidance also notes that the default BMC credentials stop working after new credentials are created, and earlier DGX OS guidance specifies that the username must use lower-case letters and a strong password. This matters because the BMC provides powerful out-of-band management functions, including remote console, power operations, firmware updates, and hardware monitoring. GRUB protection is also important because unauthorized boot-loader access can weaken system security or bypass controls. Creating separate usernames is not aligned with the documented first-boot flow where the administrative identity is reused for BMC and GRUB. Retaining defaults is insecure, and using a generic username with a simple password is unacceptable for enterprise production. In DGX infrastructure, secure first-boot credential setup is part of system hardening and operational readiness.

The two most likely causes are network packet loss and thermal throttling. A performance drop after 48 hours of HPL burn-in is less likely to be a simple launch-time MPI configuration issue, because MPI affinity errors usually appear from the beginning of the run as consistently poor performance. A delayed degradation suggests the system changed state under sustained load. Thermal throttling is a common cause: after many hours, rack cooling imbalance, blocked airflow, high inlet temperature, or fan behavior can cause GPU clocks to drop. nvidia-smi dmon helps monitor GPU temperature, power, utilization, and clocks over time. Network packet loss is also likely in multi-node HPL because HPL depends on heavy communication across the InfiniBand fabric. Link errors, symbol errors, retransmissions, degraded cables, or congestion can reduce sustained performance. ibdiagnet is the correct fabric-level diagnostic tool to collect and analyze InfiniBand health, topology, counters, and link issues. Rebooting and reducing matrix size would hide the symptom rather than diagnose it. Correct burn-in practice is to preserve evidence, inspect thermal telemetry, review network diagnostics, and compare the affected node against healthy peers.

In a Bright Cluster Manager HA setup, the database (MySQL/MariaDB) must remain perfectly synchronized between the active and standby head nodes to allow for a seamless transition. This synchronization typically occurs over a dedicated management or heartbeat network. If cmsh status shows the database service as [FAIL] on the secondary node, it almost always points to a communication breakdown. Without a stable network path, the secondary node cannot receive the binary logs from the primary node to keep its local copy up to date. While licensing (Option A) is important, a license failure usually disables management capabilities entirely rather than just the MySQL sync. Furthermore, head nodes are management servers and do not require GPU drivers (Option C) for their primary function. Ensuring low-latency, reliable connectivity between the two head nodes is the primary troubleshooting step for resolving " MySQL FAIL " states in BCM.

The best answer is data masking combined with role-based access control. In healthcare AI environments, patient data must be protected throughout ingestion, preprocessing, training, inference, analytics, and operational access. Data masking helps reduce exposure of sensitive patient identifiers or protected health information when full raw values are not required for a user or processing stage. RBAC ensures that only authorized users, services, or roles can access specific datasets, models, logs, or administrative functions. This is especially important in AI infrastructure where multiple teams may share GPUs, storage, notebooks, Kubernetes namespaces, or model registries. Tokenization and MFA are also valuable, but the option does not include fine-grained authorization over data access. Symmetric encryption at rest is important, but relying solely on passwords is not sufficient for regulated environments. Asymmetric encryption for communications is useful, but deduplication is a storage efficiency feature, not a primary security control. In NVIDIA AI infrastructure, healthcare deployments should combine RBAC, encryption, audit logging, data minimization, masking or de-identification, secure container access, and controlled administrative privileges.

The first step is to review the firmware update logs and run nvsm show health. After a DGX H100 firmware upgrade, a missing GPU can result from incomplete firmware activation, failed component update, PCIe enumeration failure, GPU tray communication issues, BMC inventory mismatch, or an actual hardware fault. NVSM is the correct DGX platform-level health tool because it checks hardware state across GPUs, NVSwitch components, PCIe devices, storage, power, cooling, and system sensors. Firmware logs are equally important because they show whether each update completed successfully and whether a reboot, cold power cycle, BMC reset, or AC power cycle is still required. Replacing the GPU immediately is premature and may cause unnecessary downtime. Ignoring the issue is unsafe because production AI workloads expect all GPUs to be visible and healthy. Re-running firmware across all components without diagnosis can hide the original failure or introduce more risk. Proper bring-up practice is to collect evidence, verify hardware health, confirm firmware activation state, and then decide whether reseating, power cycling, reapplying firmware, or service escalation is required.

Stable operation of high-density AI infrastructure like the DGX H100 requires strict adherence to power and thermal specifications. A single DGX H100 system can draw up to 10.2kW under peak load. Therefore, the most essential validation step is ensuring the electrical " infrastructure-to-server " handoff is healthy. This involves verifying that the system is connected to redundant PDUs (Power Distribution Units) capable of handling the amperage requirements without tripping breakers. Using NVSM (NVIDIA System Management), an administrator must check that all six power supplies (PSUs) are functional and receiving nominal input voltage (typically 200V-240V). If a PSU reports sub-optimal input or a " Loss of Redundancy, " the system may throttle performance or shut down unexpectedly during a heavy training run. Fans running at 100% (Option A) at all times would actually indicate an inefficient or failed cooling policy, as fans should dynamically scale based on thermals. Overclocking (Option B) is not supported or recommended for enterprise DGX systems, as they are already factory-tuned for the highest stable performance.

The output provided is a result of the NVIDIA System Management (NVSM) tool, specifically the nvsm show healthinfo command. NVSM is an essential diagnostic framework for NVIDIA DGX systems that monitors hardware health, identifies faults, and helps ensure the system remains within its validated operational state.

In this specific diagnostic trace, the system reports that the " Verify installed GPU ' s " check has returned a status of Unhealthy. To provide a root cause, NVSM cross-references the live hardware enumeration from the lspci command against the system ' s known " Golden Configuration " (the hardware manifest defined in the firmware). The explicit error message, " Missing GPU at PCI address ' 07:00.0 ' " , indicates that the system expects a GPU module to be present at that specific PCIe bus address, but the hardware is not responding or visible to the bus.

This insight allows a system administrator to conclude that a GPU is missing from the logical perspective of the system. This is a critical hardware fault rather than a software or driver issue. In a DGX H100 or A100 system, this could be caused by a physical module failure, a power delivery issue to that specific segment of the GPU baseboard, or a failure in the PCIe switch fabric. Because the DGX relies on a full set of 8 GPUs for high-speed collective communications (NCCL), a single missing GPU will prevent the node from participating in large-scale training jobs, requiring physical inspection or a GPU tray replacement (RMA).

The correct answer is D because adding GPUs to an existing server requires validation of physical, electrical, thermal, and platform compatibility requirements before installation. The server must have available PCIe slot allocation with the correct mechanical size, electrical lane support, and platform topology so the GPU can operate at the expected bandwidth. It must also have compatible hardware, including supported risers, power cables, firmware, BIOS settings, chassis airflow design, and driver support. Adequate rack power is mandatory because modern NVIDIA data center GPUs can significantly increase node and rack power draw under AI workloads. NVIDIA DGX SuperPOD data center guidance emphasizes that DGX rack density must fit within available power and cooling capacity, and NVIDIA-Certified Systems guidance notes that optimal PCIe server configuration depends on workload and system design. Cooling is equally critical because insufficient airflow or data center cooling can cause thermal throttling, instability, or hardware shutdowns. Storage and networking are important for workload design, but they are not the core server installation requirements for physically adding GPUs.

A floating Virtual IP address is required because users, compute nodes, and management services need a consistent endpoint for the active BCM head node. NVIDIA DGX BasePOD HA documentation states that the external VIP is the IP address that should always be used to access the active head node after HA configuration is complete. The same HA workflow also verifies failover behavior after setup by checking HA status from both directions and confirming which head node is active. Therefore, a controlled failover validation is necessary to prove that the secondary head node can assume service responsibility. Compute nodes do not need to be powered on and running workloads during HA setup; NVIDIA documentation actually shows cluster nodes being powered off before HA configuration. Independent static IP addresses alone do not provide transparent service continuity because clients would need to reconnect manually. Synchronization occurs as part of the HA setup workflow, but the best two answers for automatic failover and verified readiness are the floating VIP and failover validation.

The correct command is sudo nvsm stress-test --force. NVIDIA recommends using NVSM for the DGX pre-flight stress test before putting a DGX H100 system into production or after servicing. The documented NVSM stress test can run checks across supported components, including GPUs, CPU, memory, and storage, and the recommended command for all supported components is sudo nvsm stress-test --force. nvidia-smi -q provides detailed GPU information, but it does not execute a full platform stress test. The Linux stress command can load CPU and I/O subsystems, but it is generic and does not validate the DGX platform using NVIDIA’s health model. gpu_burn may stress GPUs, but it does not cover CPU, system memory, storage, and DGX-specific platform checks in the recommended way. During server bring-up, NVSM is preferred because it understands DGX hardware components and can identify platform health issues before the node is released to production workloads.

The best strategy is to use workload-specific benchmarks such as MLPerf-style AI benchmarks to understand energy efficiency in real-world scenarios. NVIDIA has argued that traditional PUE is not enough for modern AI data centers because PUE measures facility overhead relative to IT power, but it does not measure useful computational output. For AI infrastructure, the important question is not only how much power the facility consumes, but how much useful AI work is completed per unit of energy. NVIDIA’s discussion of next-generation efficiency metrics emphasizes useful work per energy and the need to account for real applications. Kilowatt-hours are useful for measuring energy consumed, but they do not by themselves capture productive AI output. Watts-used is only instantaneous power and does not reflect completed work. PUE remains useful for facilities management, but relying on it as the primary metric misses the performance and efficiency characteristics of accelerated computing. Workload-specific benchmarks allow teams to compare training, inference, and system performance against energy consumed in practical AI operations.

The correct command is nvidia-smi. Multi-Instance GPU, or MIG, is managed and verified through NVIDIA System Management Interface utilities. On supported GPUs such as NVIDIA A100 and H100, nvidia-smi can be used to check whether MIG mode is enabled, list available GPU instances, and confirm how the GPU is partitioned. For example, administrators commonly use commands such as nvidia-smi, nvidia-smi -L, or nvidia-smi -i < GPU_ID > -q to inspect GPU state and MIG information. The options nvidia-mode, nvidia-mig, and nvidia-enable are not standard NVIDIA commands for verifying MIG mode. This is an important server bring-up step because MIG profiles must be confirmed before workloads are scheduled through Kubernetes, Slurm, NVIDIA GPU Operator, or other cluster management tools. If MIG is not enabled or profiles are not created correctly, users may not see the expected GPU slices, and AI inference or multi-tenant workloads can fail placement or run with incorrect resource isolation.