Instructor-led training / ai-leadership

Building AI Infrastructure

Learn to evaluate AI workloads, deployment models, accelerator platforms, economics, and operational requirements to develop an evidence-based AI infrastructure strategy.

Request private training Talk to us

Instructor-led

ai-leadership

Duration: 2 Days
Format: Live Online
Access: Multiple Timezones

Overview

About this course.

AI infrastructure in 2026 extends far beyond GPUs, racks, power, and cooling. Organizations must determine which AI capabilities to consume as services, which workloads can run locally, which should use shared or hosted infrastructure, and when cost, security, data control, performance, or scale justify operating private AI infrastructure.

This two-day course prepares technical leaders and architects to set that direction. Participants examine the workloads driving modern AI infrastructure and how each creates different requirements for compute, memory, storage, networking, identity, security, and operations.

The course covers the full deployment spectrum, from managed model APIs and powerful AI workstations to departmental inference servers, hosted accelerators, private AI platforms, and large-scale GPU infrastructure. Participants compare NVIDIA and AMD accelerator options, CUDA and ROCm software ecosystems, local and server-class inference, and the infrastructure required as workloads move from a workstation to shared enterprise platforms.

Particular attention is given to AI economics, including API and token consumption, hardware utilization, capacity, power, cooling, staffing, and lifecycle costs. Directional knowledge checks help participants evaluate and refine their organizations' AI infrastructure assumptions throughout the course.

By the end of the course, participants will be able to identify what AI infrastructure their organization needs, what it should consume versus own, and where local, cloud, hosted, hybrid, or private infrastructure makes technical and economic sense.

Objectives

What you'll learn

Identify infrastructure requirements created by different AI workloads
Compare managed, local, hosted, cloud, hybrid, and private AI deployment models
Evaluate API and token costs against local and private inference economics
Understand workstation, server, and data-center accelerator options
Compare NVIDIA CUDA and AMD ROCm accelerator ecosystems
Size compute and accelerator resources for inference and model adaptation
Explain how model size, memory, quantization, throughput, and latency affect infrastructure
Identify the compute, storage, networking, and software layers of an AI platform
Evaluate networking, power, cooling, and capacity requirements
Identify security, identity, governance, and operational requirements
Develop an evidence-based AI infrastructure direction

Audience

Who it's for

IT and infrastructure leaders
Enterprise and solution architects
Cloud and platform architects
Data center, network, and storage architects
AI/ML platform leaders
Engineering leaders responsible for AI infrastructure
Technical decision-makers setting an organizational AI direction

Prerequisites

Before you start

No prior AI/ML engineering experience is required
Participants should be comfortable with enterprise infrastructure concepts such as servers, networking, storage, cloud platforms, virtualization, or containers

Outline

Course outline.

Download PDF

1. AI Workloads Define the Infrastructure

Lecture: The AI Infrastructure Landscape
What constitutes AI infrastructure in 2026
Models, applications, data, agents, platforms, and physical infrastructure
AI infrastructure vs. traditional application infrastructure
Why workload requirements precede infrastructure decisions
Lecture: AI Workloads and Their Infrastructure
Enterprise AI SaaS
API-driven AI applications
Retrieval-augmented generation (RAG)
Agentic AI systems
Local model inference
Shared and enterprise inference
Model adaptation and fine-tuning
Model training
Knowledge Check: Starting Direction
Assess where organizational AI inference is expected to run
Identify the workload expected to create the largest infrastructure requirement
Evaluate expectations for owning GPU infrastructure
Identify the strongest drivers of current AI infrastructure decisions

2. Consume, Run Locally, Rent, or Own

Lecture: The Deployment Spectrum
Managed model APIs and Model-as-a-Service
Local inference
Professional AI workstations
Departmental inference servers
Hosted GPU infrastructure
Public cloud AI infrastructure
Private and on-premises AI platforms
Hybrid AI
Edge AI
Build vs. buy vs. rent
Lecture: AI Infrastructure Economics
API and token consumption
Workstation economics
GPU rental and hosted infrastructure
Capital cost of private infrastructure
Utilization and idle capacity
Capacity planning
Data movement and storage
Operations and staffing
Total cost of ownership
Knowledge Check: Sourcing Direction
Identify workloads suited to managed APIs
Identify workloads suited to local or private inference
Evaluate when recurring API and token expenses justify owning the runtime
Compare vendor dependency with infrastructure ownership
Reassess the expected deployment model

3. Inference Architecture

Lecture: What Happens When You Run a Model
Model parameters and memory requirements
Tokens and inference compute
Context and KV cache
VRAM and HBM
Memory capacity vs. memory bandwidth
Quantization
CPU, GPU, and NPU inference
Throughput vs. latency
Lecture: From Workstation GPUs to Data-Center Accelerators
Consumer GPUs
Professional AI workstation GPUs
NVIDIA RTX PRO
AMD Radeon and Radeon PRO
NVIDIA data-center accelerators
AMD Instinct accelerators
GDDR vs. HBM
PCIe accelerator architectures
Single-GPU and multi-GPU inference
When workstation hardware is sufficient
When server and data-center accelerators become necessary
Lecture: Model Serving
Local inference runtimes
Inference servers
vLLM, TGI, and Triton
Model APIs and API compatibility
Batching and caching
Model routing and gateways
Autoscaling and high availability
Knowledge Check: Inference Direction
Identify workloads that can run on a workstation
Identify workloads that require shared infrastructure
Evaluate model fit, throughput, concurrency, latency, and availability requirements
Reassess where inference is expected to run

4. RAG and Agentic Infrastructure

Lecture: Infrastructure for RAG
Document ingestion
Embedding generation
Vector storage and search
Object and file storage
Data pipelines
Data locality and movement
Lecture: Infrastructure for Agentic Systems
Agent runtimes and harnesses
Tool connectivity
MCP and tool interfaces
State and persistence
Execution sandboxes
Workload identity
Secrets and delegated credentials
Agent authority boundaries
Human approval points
Local agents accessing remote systems
Knowledge Check: Application Direction
Evaluate where enterprise data should meet the model
Identify data that may leave organizational control
Assess whether local inference changes the viability of agentic systems
Compare infrastructure concerns created by models with those created by surrounding systems and data

5. Model Adaptation and Training

Lecture: From Prompting to Training
Prompting vs. RAG vs. fine-tuning vs. training
PEFT and LoRA
Fine-tuning infrastructure
Pre-training infrastructure
Dataset preparation
Checkpointing
Experiment tracking
Artifact and model storage
Lecture: Scaling Accelerated Compute
Single-node vs. distributed workloads
Scale-up vs. scale-out
GPU-to-GPU communication
Distributed training communication
Ethernet, InfiniBand, and RDMA
When specialized AI fabrics become necessary
Knowledge Check: Training Direction
Determine whether the organization needs to train models
Identify requirements that can be satisfied through prompting or RAG
Identify requirements that justify fine-tuning
Evaluate training as an internal capability or externally consumed service

6. The AI Platform Software Stack

Lecture: Accelerator Software Ecosystems
NVIDIA CUDA
AMD ROCm
Drivers, runtimes, libraries, and toolchains
PyTorch, JAX, and TensorFlow
Hardware and framework compatibility
Software portability and accelerator dependency
Hardware and software as a combined architecture decision
Lecture: From One GPU to a Shared AI Platform
Linux
Containers and registries
Kubernetes and alternative orchestration approaches
GPU scheduling and device management
GPU sharing and partitioning
Distributed computing frameworks
Model registries
AI development environments
Managed AI platforms
Knowledge Check: Platform Direction
Compare NVIDIA/CUDA, AMD/ROCm, managed abstraction, and an intentionally undecided approach
Evaluate the importance of accelerator portability
Determine when centralized scheduling becomes worthwhile
Identify which platform components the organization should own and operate

7. Data, Storage, Networking, Power, and Cooling

Lecture: Data and Storage Infrastructure
Structured and unstructured AI data
Object vs. file vs. block storage
Dataset, model, and checkpoint storage
Storage throughput
Data locality and movement
Data lifecycle and lineage
Lecture: Networking for AI
AI application networking
East-west vs. north-south traffic
GPU and cluster interconnects
Ethernet, InfiniBand, and RDMA
Storage networking
Distributed workload requirements
Lecture: Physical Infrastructure
Workstation and server power requirements
GPU server and rack density
Rack-level power
Air vs. liquid cooling
Facility capacity
Data center constraints
Scaling from workstation to server to rack to cluster
Knowledge Check: Physical Infrastructure Direction
Evaluate whether existing facilities can support the envisioned AI infrastructure
Determine whether the expected workload requires a GPU cluster
Assess whether power, cooling, networking, or facility constraints favor hosted infrastructure
Determine where owning infrastructure stops making sense

8. Security, Governance, and AI Operations

Lecture: AI Infrastructure Security
Network segmentation and private connectivity
IAM and workload identity
Secrets management
Encryption and data access controls
Secure execution and isolation
Model and dependency vulnerability management
Agent and tool authorization
Lecture: Governance
Data governance
Data residency and sovereignty
Model governance
Model access policies
Model provenance and AI asset inventory
Auditability and compliance
Lecture: AI Operations / LLMOps
Infrastructure and accelerator monitoring
Model-serving monitoring
Application observability
Logging and tracing
Token and cost monitoring
Model evaluation
Capacity and utilization management
Model lifecycle and versioning
Availability, disaster recovery, and incident response
Final Knowledge Check: Your AI Infrastructure Direction
Reassess where organizational AI inference is expected to run
Reassess expectations for owning GPU infrastructure
Identify which workloads belong locally, centrally, or externally
Identify the strongest architectural driver
Identify the first infrastructure capability the organization should investigate
Reflect on how technical and economic information changed infrastructure assumptions

For teams

Training a whole team?

Request private training for Building AI Infrastructure and we'll follow up with a quote for your group.

Request a quote Browse the catalog