Skip to main content
Guide, 9 minute read

On-Premises AI Server Buying Guide: Sizing, Power, and Networking

When an on-premises AI server makes sense, how to size GPU memory for inference and fine-tuning, what NVIDIA-Certified Systems covers, published power and cooling per server, and networking.

Planning guidesIT directors, infrastructure managers, and operations leaders at mid-size organizations planning an on-premises AI or GPU serverBy , Founder and CEO, Days Dynamics. Published November 10, 2026.

When an on-premises AI server makes sense, and how to size one

An on-premises AI server usually makes sense when your data has to stay on hardware you control, or when GPUs will be busy most hours of most days. For short or bursty projects, cloud GPUs are usually the simpler start. Size the server from the model: how much GPU memory it needs, whether you only run it (inference) or also train it (fine-tuning), and the power, cooling, and network your room can supply.

When on-premises beats cloud for AI

As a planning rule of thumb, on-premises GPU servers fit best under two conditions: the data cannot leave your control, and the GPUs will stay busy. If neither is true, cloud GPU capacity is usually easier to start with, because you can pay by the hour and the provider runs the hardware.

Data control

Patient records, student data, contracts, or source code may be bound by policy or contract to stay on systems you manage. An on-premises server can keep the model, the prompts, and the outputs inside your own network and access controls.

Steady utilization

The purchase cost of a server is fixed whether it runs one hour a day or all day. A server that runs inference most of the day spreads that cost across more work than one used for occasional tests.

Local systems and large files

Workloads that read from on-site systems, such as document scanners, cameras, or a local enterprise resource planning (ERP) database, avoid sending large files over the internet when the model runs in the same building.

When cloud fits better

Pilots, experiments, and seasonal spikes fit cloud GPUs better. You can test several models and GPU types before committing to hardware, power work, and a support contract.

See the Replace the server or move to Azure.

How much GPU memory a model needs

GPU memory is the first sizing number. In NVIDIA's worked example, model weights take the parameter count times the bytes per parameter. At 16-bit precision (FP16 or BF16), a 7 billion parameter model takes roughly 14 GB for its weights alone. The KV cache, which holds the context for each request, comes on top and grows with batch size and sequence length.

WorkloadWhat uses the GPU memoryWhat NVIDIA documents
Inference at 16-bitModel weights plus the KV cache for active requestsA 7 billion parameter model needs roughly 14 GB for weights; the KV cache grows with batch size and sequence length
Quantized inferenceWeights stored in 8 bits or fewerMost deep learning models can be represented with eight or even fewer bits per value, so the same GPU fits a larger model
Parameter-efficient fine-tuning (PEFT), such as LoRAThe base model plus small adapter layers and their gradientsPEFT reduces trainable parameters to less than 1% of the original model, and gradients are computed only for the adapters
Full fine-tuning (SFT)Every parameter is updatedUse SFT when you need maximum accuracy and have sufficient compute

Server options, from one PCIe GPU server to an 8-GPU system

The main choice is between a standard rack server that takes PCIe GPU cards and an 8-GPU system whose GPUs connect over NVIDIA NVLink. The examples below come from each vendor's own spec page. They illustrate the categories and are not recommendations.

OptionExample, as the vendor lists itUsually fits when
PCIe GPU serverHPE ProLiant Compute DL380a Gen12 (4U): up to 10 double-wide L40S, H100 NVL, or H200 GPUs, or 8 double-wide RTX PRO 6000 Blackwell Server Edition GPUsInference, PEFT fine-tuning, and models that fit on one or a few GPUs
8-GPU SXM serverDell PowerEdge XE9680 (6U) or Supermicro SYS-821GE-TNHR (8U), each listing NVIDIA HGX H100 and H200 8-GPU optionsModels too large for a few cards, full fine-tuning, or many users at once
NVIDIA DGX systemNVIDIA DGX B200 (10U): 8 B200 GPUs with 1,440 GB total GPU memoryYou want one integrated 8-GPU system specified end to end by NVIDIA
Cloud GPU instancesRented from a cloud provider by the hour or on a longer termDemand is uneven, or you are still choosing a model and GPU type

See the Technology procurement and the Hardware and software refresh.

Power and cooling per server, as vendors publish them

Power and heat usually decide where an AI server can go. NVIDIA lists 10.2 kW maximum for a DGX H100/H200 and 14.3 kW maximum for a DGX B200, Compare that with the power available to the rack. The figures below come from vendor pages, checked November 2026. NVIDIA labels its H200 figures preliminary and subject to change.

System or GPUGPU memoryPower, as publishedCooling, as published
NVIDIA L40S (PCIe card)48GB GDDR6 with ECC350W max power consumptionPassive, cooled by the server's airflow
NVIDIA H200 NVL (PCIe card)141GBUp to 600W (configurable)Dual-slot, air-cooled
NVIDIA H200 SXM141GBUp to 700W (configurable)Not listed (SXM module)
NVIDIA DGX H100/H200 (8U)640 GB (H100) or 1,128 GB (H200) total10.2 kW max; six 3.3 kW supplies1105 CFM front to back at 80% fan PWM; 5°C to 30°C operating
NVIDIA DGX B200 (10U)1,440 GB total14.3 kW max; six 3.3 kW supplies1,550 CFM airflow; 10°C to 35°C operating
Dell PowerEdge XE9680 (6U)8 x H100 80GB or 8 x H200 141GB SXM52800W Titanium hot-swap redundant supplies, among other optionsAir cooling
Supermicro SYS-821GE-TNHR (8U)HGX H100 8-GPU (80GB) or HGX H200 8-GPU (141GB)6 x 3000W redundant (4 + 2) Titanium suppliesUp to 10 heavy duty fans
HPE ProLiant Compute DL380a Gen12 (4U)Depends on the GPUs chosenThree power domains, two with six power supplies dedicated to GPUsAir-cooled or Direct Liquid Cooling (DLC)

Networking for one server or a cluster

An AI server usually needs two kinds of network. NVIDIA's enterprise reference architecture separates north-south networking for storage, uplink, and management traffic from a full east-west fabric for distributed workloads, which carries GPU traffic between servers.

  • East-west fabric. You need it when one job spans several servers. NVIDIA's DGX H100/H200 and DGX B200 each carry eight ConnectX-7 adapters for cluster traffic, InfiniBand by default at up to 400Gb/s.
  • Bandwidth per GPU. NVIDIA names configurations in the pattern CPU-GPU-Network Adaptor-Average GPU Network Bandwidth. Its RTX PRO AI Factory 2-8-5-200 design has 2 CPUs, 8 GPUs, 5 network adapters, and 200 GbE of east-west bandwidth per GPU.
  • North-south links. Size the uplinks to storage and to users. Training data, model files, and documents the model searches all cross this network.
  • One server first. If one server runs the whole workload, an east-west fabric can wait. Leave switch ports and rack space for it if a second server is likely.
  • Management network. Keep each server's remote management port on a separate, restricted network.

See the Network and Wi-Fi.

What NVIDIA-Certified Systems tells you

NVIDIA-Certified Systems is NVIDIA's program for servers, workstations, and edge systems that NVIDIA has validated with its GPUs. NVIDIA lists certification per configuration, so check the exact GPU model, GPU count, and network adapter you plan to buy.

What is tested

NVIDIA says systems undergo functional, security, and performance tests, including single-GPU, multi-GPU, and multi-node tests. Server testing also covers network and storage offload, security features such as the Trusted Platform Module (TPM), and remote management.

Configuration matters

NVIDIA's list names each configuration as CPU-GPU-Network Adaptor-Average GPU Network Bandwidth. A number in parentheses after a GPU name is the maximum number of that GPU certified for the system.

Cluster testing

Systems intended for cluster deployment are tested as single nodes and in a two-node configuration. Testing includes network bandwidth tests of remote direct memory access (RDMA).

Who supports the hardware

NVIDIA states that hardware support for NVIDIA-Certified Systems is available from the system vendors. Plan the hardware support contract with the server's manufacturer.

Planning checklist for an on-premises AI server

  • Workload. Name the model, whether you will run it or also fine-tune it, how many people will use it, and how fast answers must come back.
  • GPU memory. Use the model's published minimum GPU memory per precision and GPU count, or parameters times bytes per parameter, then add room for the KV cache.
  • Data rules. List which data must stay on site and who may see the prompts and outputs.
  • Utilization. Estimate how many hours a day the GPUs will be busy. If they would sit idle most of the week, price cloud GPUs first.
  • Rack power. Compare each candidate's published maximum power with the circuits available per rack before you choose a model.
  • Cooling. Confirm the room can remove that heat and hold the vendor's operating temperature range. Ask the vendor what facilities work direct liquid cooling requires.
  • Floor and rack. Check rack depth, rack units, and weight. NVIDIA lists the DGX H100/H200 at up to 287.6 lbs (130.45 kg).
  • Networks. Decide whether you need an east-west fabric now, and size the north-south uplinks to storage and users.
  • Certification and support. Confirm the exact configuration appears on the NVIDIA-Certified Systems list, and who supports the hardware and the software.
  • Compare like for like. Price configurations from different manufacturers against the same requirements.

See the Enterprise infrastructure modernization.

Frequently asked questions

When does an on-premises AI server make more sense than cloud GPUs?

When the data must stay on systems you control, or when the GPUs will be busy most of the day. For pilots, experiments, and uneven demand, cloud GPUs are usually the easier start.

How much GPU memory does a model need?

Check the model's published deployment profiles first. NVIDIA's NIM documentation lists, for many model profiles, a minimum GPU memory per GPU for each precision and tensor-parallel size, which is the number of GPUs the model is split across. Without a profile, multiply parameters by bytes per parameter and add room for the KV cache.

Can one GPU run several smaller models?

Some can. NVIDIA lists the H200 with up to 7 Multi-Instance GPU (MIG) instances, at 18GB each on H200 SXM and 16.5GB each on H200 NVL. NVIDIA lists the L40S without MIG support.

What does NVIDIA-Certified mean for a server?

The system configuration has been validated by NVIDIA through functional, security, and performance testing. It applies to the configuration listed, and hardware support comes from the system vendor.

How much power does an 8-GPU AI server draw?

It depends on the GPUs. NVIDIA lists 10.2 kW maximum for a DGX H100/H200 and 14.3 kW maximum for a DGX B200. Use the vendor's published maximum for the exact configuration when you plan circuits and cooling.

Size an on-premises AI server against your room and your data

Send us the models you plan to run, the data rules they must follow, and the power and rack space you have. We will compare configurations from different manufacturers against the same requirements and flag power, cooling, and network gaps. A planning call is planning support, not a quote, an audit, or a commitment on licensing, availability, or delivery dates.

Sources

Primary sources checked when this guide was written. Dates and product status can change, so confirm on the source before you rely on them.

Planning this project now?

Turn the checklist into a plan

The 2-minute IT quiz turns this checklist into ranked priorities for your situation, before any contact details.