Video analytics AI agents are AI-powered assistants that can see, reason, and act on live or recorded video streams. They use vision language models and large language models to help search, summarize, and understand video through natural language.
Workloads
Computer Vision / Video Analytics
Industries
Manufacturing
Smart Cities/Spaces
Retail/ Consumer Packaged Goods
Media and Entertainment
Healthcare and Life Sciences
Business Goal
Return on Investment
Innovation
Overview
Traditional video analytics applications and their development workflows are typically built on fixed-function, limited models that are designed to see and identify only a select set of predefined objects. With generative AI and foundation models, you can now build applications with fewer models that have incredibly complex and broad perception and rich contextual understanding. This new generation of vision language models (VLMs), such as NVIDIA Cosmos™, is giving rise to smart, powerful video analytics AI agents.
A video analytics AI agent can see, reason, and act by combining vision and language modalities to understand a broad range of natural language questions or prompts applied to a recorded or live video stream. This deeper understanding of video content enables more accurate and meaningful interpretations, improving the functionality of video analytics applications and analysis of real-world scenarios. These agents promise to unlock entirely new insights and possibilities for automation.
Highly perceptive, accurate, and interactive video analytics AI agents will be deployed throughout factories, warehouses, retail stores, airports, traffic intersections, and more. This will have a tremendous impact on operations teams looking to make safer spaces and better decisions using richer insights generated from natural interactions. Managers and operations teams will also communicate with these agents in natural language, all powered by generative AI and VLMs with NVIDIA NIM™ microservices at their core.
Quick Links
Technical Implementation
The brain inside every video analytics AI agent is a VLM that can see and reason. Two common VLMs are NVIDIA Cosmos 3 and Cosmos Embed. Both can be used to augment current computer vision applications with rich metadata and content summaries.
NVIDIA NIM is a set of accelerated inference microservices that are optimized for NVIDIA GPUs and include industry standard APIs, domain-specific code, optimized inference engines, and enterprise runtime. It delivers a combination of VLMs, large language models (LLMs), and retrieval-augmented generation (RAG) for building your video analytics AI agent that can process live or archived images or videos to extract actionable insights using natural language. We’ve created a reference workflow of a video analytics AI agent that you can try out to accelerate your development process.
Quick Links
The NVIDIA Metropolis Blueprint for video search and summarization (VSS) makes it easy to build and customize video analytics AI agents using generative AI, VLMs, LLMs, RAG, and NVIDIA NIM. The video analytics AI agents are given tasks through natural language and can analyze, interpret, and process vast amounts of video data to provide critical insights that help a range of industries optimize processes, improve safety, and cut costs.
VSS delivers modularized components that enable high flexibility, accelerated microservices that support real-time video intelligence, agentic fusion search across diverse embeddings, and comprehensive report generation capabilities. It also provides agent skills and tools that let developers build video analytics AI agents with simple natural language prompts and coding agents.
VSS also enables seamless integration of generative AI into existing computer vision pipelines—enhancing inspection, search, and analytics with multimodal understanding and zero-shot reasoning. VSS is easily deployed from the edge to the cloud on platforms including NVIDIA RTX™ 4500, NVIDIA RTX PRO™ 6000, NVIDIA DGX Spark™, and NVIDIA® Jetson Thor™.
Quick Links
Traditional approaches to customizing models for video analytics AI agents were linear and slow—collect video, label frames, train, evaluate, repeat—with a human in the loop at every step and months to reach acceptable accuracy. Modern approaches break this cycle by enabling coding agents to iteratively improve VLM and vision foundation model performance based on target goals.
Fine-tune vision language models with NVIDIA TAO agent skills.
NVIDIA TAO is a suite of agent skills and tools for fine-tuning vision AI models with natural language prompts. Coding agents use these tools and skills to autonomously hit model accuracy targets by iteratively evaluating model accuracy, determining the precise training data needed, and then mining existing data or synthetically generating needed data.
Solve training data challenge with agent skills for synthetic data generation.
When training data is limited, developers can quickly generate synthetic defect data for visual inspection or augment videos for different scenarios such as weather, lighting, and more.
Quick Links
FAQs
Yes, you can now build video analytics AI agents faster from simple natural language prompts using VSS skills with coding agents like Codex and Claude. Explore a suite of VSS skills in github.
A NIM is a set of easy-to-use microservices designed for secure, reliable deployment of high-performance AI model inferencing across the cloud, data center, and workstations. It supports a wide range of AI models, including open source community and NVIDIA AI foundation models to ensure seamless, scalable AI inferencing — on premises or in the cloud — using industry-standard APIs. All NIM microservices and associated preview APIs can be found at build.nvidia.com.
Visit build.nvidia.com to start exploring the NVIDIA Metropolis VSS Blueprint and available NIM microservices such as Cosmos 3 Reasoner NIM.
All users can get started for free with the preview APIs on build.nvidia.com. Each new account can receive up to 5,000 credits to try out the APIs. To continue development after credits run out, you can deploy the downloadable NIM microservices locally to your hardware or to a cloud instance. Developers can also access NIM via the NVIDIA Developer Program. See details in this FAQ.
NVIDIA NIMs is free for developers to try out. To go to production, downloadable NIM microservices require an NVIDIA AI Enterprise License. To learn more, visit this page.
The NIM developer forum is the best place to ask questions and engage with our developer community. You can access the forums here.
Explore the reference workflow, powered by multiple visual language models, to easily build your video analytics AI agent.
Tap into the power of the VSS blueprint to deploy AI agents seamlessly from edge to cloud, with scalable performance across a diverse range of GPUs.