Nvidia Nim

NVIDIA NIM packages AI models with CUDA/TensorRT optimization into Docker containers that expose an OpenAI-compatible REST API — meaning any tool that works with OpenAI works with NIM by changing the base URL. For agents: deploy NIM containers on NVIDIA GPU infrastructure (self-hosted or NGC cloud), point existing OpenAI client code at the NIM endpoint, and get optimized inference with TensorRT-LLM throughput improvements (often 2-5x vs. unoptimized serving). NIM catalog covers LLMs (Llama, Mistral, Mixtral), vision models, speech, and embedding models. Data sovereignty: everything runs on your infrastructure. NIM API catalog at api.nvcf.nvidia.com provides cloud-hosted access for evaluation without GPU setup. Confidence is docs-derived.

OpenAI-compatible API: POST /v1/chat/completions, POST /v1/completions, POST /v1/embeddings — same request/response schema as OpenAI API. Change base_url to NIM endpoint; keep same OpenAI client library. Example: client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-used-for-local"). Container startup: docker run --gpus all -p 8000:8000 nvcr.io/nim/meta/llama-3.1-8b-instruct:latest. NIM exposes /v1/models (list models), /v1/health/ready (health check), /metrics (Prometheus). NGC API catalog (api.nvcf.nvidia.com): same OpenAI-compatible interface with NGC key auth for cloud-hosted evaluation.

Self-hosted NIM: API key optional (no auth by default; add NGINX/reverse proxy for auth). NGC catalog (cloud-hosted): Authorization: Bearer {ngc_api_key} header. NGC API keys from NGC portal (catalog.ngc.nvidia.com) → API Key. Docker image pull: docker login nvcr.io (NGC credentials required). TLS: operator-managed for self-hosted (typically reverse proxy). Enterprise: NIM available via NVIDIA AI Enterprise subscription — adds support SLA and additional models. Data sovereignty: self-hosted NIMs process no external network calls for inference.

OpenAI-compatible responses — same error format as OpenAI (type, code, message). Health endpoint: GET /v1/health/ready returns 200 when model loaded (useful for startup probe). Model loading time: LLMs take 30s–5 minutes to load depending on model size and GPU. Out-of-memory: GPU OOM errors if model too large for available VRAM — check NIM model card for GPU requirements. Throughput: TensorRT-LLM optimized; concurrent request handling via continuous batching. Self-hosted: operator-responsible for GPU availability and container orchestration. NVIDIA support SLA available via AI Enterprise subscription.

docs.nvidia.com/nim covers NIM architecture, container deployment guide, model catalog, GPU requirements per model, and API reference. Getting started: NGC API key + NVIDIA GPU (Ampere+ recommended) + Docker → pull NIM image → docker run → OpenAI-compatible API available. NIM quickstart scripts in NVIDIA GitHub repos. NIM supported models listed in NGC catalog (catalog.ngc.nvidia.com/ai-foundation-models). Community via NVIDIA Developer forums, NGC forums, and NVIDIA Discord. Enterprise support via NVIDIA AI Enterprise subscription.