AI Agents

AI agents are advanced AI systems designed to autonomously reason, plan, and execute complex tasks based on high-level goals.

AI agents are the new digital workforce—working for and with us. They represent the next evolution in artificial intelligence, transitioning from simple automation to autonomous systems capable of managing complex workflows. These agents not only automate repetitive and time-consuming tasks but also empower individuals and organizations to operate more efficiently by acting as intelligent personal assistants. 

Unlike traditional generative AI models that follow a basic “request-and-respond” framework, AI agents go beyond by orchestrating resources, collaborating with other agents and utilizing tools such as large language models (LLMs), retrieval augmented generation (RAG), vector databases, APIs, frameworks, and high-level programming languages like Python.

Often referred to as “agentic AI” or “LLM agents,” these systems stand out for their ability to achieve goals through iterative planning and decision-making. For example, an AI agent tasked with building a website could autonomously manage tasks like layout design, writing HTML and CSS code, connecting backend processes, generating content, and debugging—all while requiring minimal human input.

To understand how AI agents operate, it’s crucial to examine their core components. These components work in tandem to enable agents to reason, plan, and execute tasks effectively:

• LLM: The “brain” of the AI agent, a large language model (LLM) is responsible for coordinating decision-making. It reasons through tasks, plans actions, selects appropriate tools, and manages access to necessary data to achieve objectives. The agent core is where the agent’s overall goals and objectives are defined and orchestrated.
• Memory Modules: AI agents rely on memory to maintain context and adapt to ongoing or historical tasks:
  • Short-Term Memory: Tracks the agent’s “train of thought” and recent actions, ensuring context is preserved throughout the current workflow.
  • Long-Term Memory: Retains historical interactions and relevant information, allowing for deeper contextual understanding and improved decision-making over time.
• Planning Modules: Planning modules enable AI agents to break down complex tasks into actionable steps:
  • Without Feedback: Uses structured techniques like “Chain of Thought” or “Tree of Thought” to decompose tasks into manageable steps.
  • With Feedback: Incorporates iterative improvement methods like ReAct, Reflexion, or human-in-the-loop feedback for refined strategies and outcomes.
• Tools: AI agents can serve as tools themselves, but they also extend their capabilities by integrating with external systems such as:
  • APIs: Access real-time data or executing actions programmatically.
  • Databases and RAG pipelines: Retrieve relevant information and ensure accurate knowledge bases.
  • Other AI Models: Collaborate with additional models for specialized tasks.

AI agents seamlessly combine their core components to tackle complex tasks. Below is an example illustrating how these components work together in response to a specific user request.

Example Prompt: Analyze our latest quarterly sales data and provide a graph.

Step 1. User or Machine Request 

A user, or even another agent or system, initiates the agent’s workflow by requesting an analysis of sales data and a visual representation. The agent processes this input and decomposes it into actionable steps.

Step 2. The LLM: Understanding the Task

The LLM acts as the brain of the AI agent. It interprets the user’s prompt to understand the task requirements, such as:

• Retrieving data from database.
• Performing data analysis.
• Creating a visual graph.

The LLM determines:

• What information it already has.
• What additional data or tools it needs.
• A step-by-step plan to fulfill the task.

Step 3. Planning Module: Task Breakdown

The planning module divides the task into specific actions:

• Fetch: Retrieve the latest sales data from the company database.
• Analyze: Apply appropriate algorithms to identify trends and insights.
• Visualize: Generate a graph to present the results.

Step 4. Memory Module: Providing Context

The memory module ensures context is preserved for efficient task execution:

• Short-term memory: Tracks the context of the current workflow, such as similar tasks requested last quarter, to streamline the process.
• Long-term memory: Retains historical knowledge, like the database location or preferred analysis methods, enabling deeper contextual understanding.

Step 5. Tool Integration: Performing the Task

The agent core orchestrates external tools to complete each step: 

• APIs: Retrieve raw sales data.
• Machine learning algorithms: Analyze data for trends and patterns.
• Code interpreter: Generates the graph based on the analysis results.

Step 6. Reasoning and Reflection: Improving Outcomes

Throughout the process, the agent applies reasoning to refine its workflow and enhance accuracy. This includes:

• Evaluating the effectiveness of each action.
• Ensuring efficient use of tools and resources.
• Learning from user feedback to enhance future tasks.

For example, if the generated graph needs refinement, the agent adapts its approach to deliver better results in subsequent workflows.

The reasoning layer is a defining feature of agentic AI, enabling agents to think about how they achieve their goals. By combining LLM capabilities with tools like APIs, orchestration software, and contextual memory,  reasoning empowers agents to navigate complex environments with precision and efficacy. This adaptability makes AI agents invaluable for automating and optimizing intricate workflows.

AI agents can be written directly in Python, especially for simple workflows and experimentation. When moving to more complex workflows or production environments, telemetry, logging, and evaluation become important, and agent frameworks become helpful. AI agent frameworks are specialized development platforms or libraries designed to simplify the process of building, deploying, and managing AI agents. These frameworks abstract much of the underlying complexity of creating agentic systems, allowing developers to focus on specific applications and agent behaviors rather than the technical details of implementation.

When choosing an AI agent framework, it’s important to consider factors such as:

• Multi-agent collaboration: Does the project require multiple agents working together?
• Project complexity: Is the framework suitable for simple tasks or complex workflows?
• Data handling: Does the framework support necessary data integration and retrieval?
• Customization needs: How much flexibility is needed for tailoring the agent’s behavior?
• LLM emphasis: Does the framework prioritize working with large language models?

Depending on these requirements, a range of frameworks exists to suit different use cases and levels of complexity.

There are many ways to implement AI agents—for example, bring your own Python, LangChain, and Llama Stack.

AI agents can be classified based on their complexity, decision-making processes, and adaptability to their environment. Below are the key types of AI agents, ranging from simple systems to highly intelligent and adaptive frameworks:

AI agents and AI assistants differ significantly in their capabilities, autonomy, and the complexity of tasks they can handle.

AI assistants are an evolution of traditional AI chatbots. They use natural language processing (NLP) to understand user queries in the form of text or voice and perform tasks based on direct human instruction. These systems, such as Apple’s Siri, Amazon’s Alexa, or the Google Assistant, excel at handling predefined tasks or responding to specific commands. 

AI agents represent a more advanced form of AI that extends beyond the capabilities of assistants. They leverage planning, reasoning, and contextual memory to tackle complex, open-ended tasks autonomously. AI agents can perform iterative workflows, utilize a broad set of tools, and adapt based on feedback and prior interactions.

The potential use cases of AI agents could be basically infinite. Deploying AI agents will be a matter of imagination and expertise, spanning from simpler use cases like generating and distributing content to complex use cases like orchestrating enterprise software and database functionality.

A task execution agent, which could also be called an “API agent” or an “execution agent,” can carry out a task requested by a user by using a set of predefined executive functions.

Example: “Write me a social media post to market our latest product and be sure to mention it’s on sale and now comes in green.”

AI agents for specific applications can help streamline how efficient a human is at using that tool. For example, AI co-pilots can help a user understand all the features of an application and automate how those features are used or suggest how a person can best use that tool.

Example: Optimize data center performance with a swarm of agents and an OODA loop strategy.

Data analysis can be performed by multi-agent systems designed to extract data and make sense of it. Think of it as an “extract and execute” strategy where one set of agents works to gather the data from short- or long-term memory, or even PDF, and then another set of execution agents that call on APIs to trigger the data analysis tools.

Example: “In how many quarters of this year did the company have a positive cash flow?”

AI agents can provide 24-hour support while understanding natural language queries in both text and voice forms, resolving complex issues by taking action on behalf of the customer.

Example: A call center operator or chatbot can automate workflow tasks such as connecting to internal systems like the CRM, checking to see if a customer request qualifies for a refund, or inputting data needed to start a return.

AI agents can function as coding assistants for software developers, helping to provide code suggestions, point out errors and offer one-click fixes, provide pull request summaries, and generate code. 

Example: One of the most popular AI agents in use today is the GitHub Copilot, which operates as an assistant to developers, generating and suggesting code, managing documentation, and fixing errors.

A multi-agent system or “swarm” of agents can help optimize the supply chain by analyzing data in real time, monitoring and adjusting inventory levels based on demand, and even help source raw materials by keeping an eye on market fluctuations.

Example: A hierarchical agent system can have tiers of agents that look after different aspects of the supply chain, reporting up to an orchestrating agent that makes decisions based on the data.

NVIDIA offers tools and software to ease the development and deployment of agentic AI at scale.

1. NVIDIA Blueprints provide a starting point for developers creating AI applications that use one or more AI agents. They include sample applications built with NVIDIA AI and NVIDIA Omniverse™ libraries, SDKs, and microservices and provide a foundation for custom AI solutions. Each blueprint includes reference code for constructing workflows, tools, and documentation for deployment and customization and a reference architecture outlining API definitions and microservice interoperability.
2. Developers have access to the newest AI models within the NVIDIA API catalog to build and deploy their own agentic AI applications.