Get Started With NVIDIA Triton

NVIDIA Triton Inference Server, or Triton for short, is an open-source inference serving software. It lets teams deploy, run, and scale AI models from any framework (TensorFlow, NVIDIA TensorRT™, PyTorch, ONNX, XGBoost, Python, custom, and more) on any GPU- or CPU-based infrastructure (cloud, data center, or edge). For more information, please visit the Triton webpage.

Triton Management Service is for automated and resource-efficient orchestration of models for inference at scale. It automates deployment of Triton instances, model loading on demand, unloading when not in use, and more. Triton Management Service is available exclusively with NVIDIA AI Enterprise, an enterprise-ready AI software platform.

Triton Model Analyzer is an offline tool for optimizing inference deployment configurations (batch size, number of model instances, etc.) for throughput, latency, and/or memory constraints on the target GPU or CPU. It supports analysis of a single model, model ensembles, and multiple concurrent models.

Triton is included with NVIDIA AI Enterprise, an end-to-end AI software platform that offers enterprise-grade support, security stability, and manageability for the entire software stack across data center and cloud.

NVIDIA AI Enterprise includes bsiness-standard support. There are additional support and services available, including business-critical support, access to a technical account manager, training, and professional services. For more information, please visit the Enterprise Support and Services User Guide.

Yes, there are several labs that use Triton in NVIDIA Launchpad.

Yes, Triton is the top ecosystem choice for AI inference and model deployment. Triton is available in AWS, Microsoft Azure, and Google Cloud marketplaces with NVIDIA AI Enterprise. It’s also available in Alibaba Cloud, Amazon Elastic Kubernetes Service (EKS), Amazon Elastic Container Service (ECS), Amazon SageMaker, Google Kubernetes Engine (GKE), Google Vertex AI, HPE Ezmeral, Microsoft Azure Kubernetes Service (AKS), Azure Machine Learning, and Oracle Cloud Infrastructure Data Science Platform.