In this article, you will learn how LangChain, LlamaIndex, and raw API calls each solve a different layer of the LLM application stack, and how to choose among them based on what your project actually requires.
Topics we will cover include:
What each option is designed to do, stated plainly without marketing spin.
How the three approaches compare on performance, token overhead, debugging clarity, and code volume.
A practical decision framework for picking the right level of abstraction before you build — and before that choice becomes expensive to undo.
Let’s not waste any more time.
Introduction
You have a working prompt. The model is giving good answers. Then the next requirement lands. Maybe it is memory; the model needs to remember what was said three messages ago. Maybe it is retrieval — the model needs to answer questions about documents it was not trained on. Maybe it is tool use; the model needs to check a database, run a calculation, or call an external API before it can respond. Suddenly, a single client.chat.completions.create() call is not enough, and you are standing at the first real architectural decision in your LLM project.
Three paths exist from that moment: reach for LangChain, reach for LlamaIndex, or build a thin layer on top of the raw SDK yourself. Getting this choice wrong does not break the prototype. It breaks the production system six months later, when you are debugging stack traces 40 frames deep, paying 2.7x what you should be on token costs, or spending a sprint migrating away from breaking API changes.
LLM API spend doubled from \$3.5 billion to \$8.4 billion between late 2024 and mid-2025. These are real production budgets. The framework layer — the code that sits between your application and the model — directly determines how much of that spend is doing useful work versus paying for abstraction you did not need.
This article gives you an honest comparison: what each option actually is, where it genuinely wins, where it costs you, and a decision framework you can use tomorrow.
The Landscape in Plain English
Before comparing trade-offs, it helps to understand what each option actually is — not what its marketing says, but what problem it was built to solve.
LangChain started in October 2022 as a general-purpose framework for chaining LLM operations together. Its core idea was that building real applications required composing multiple steps — prompt templates, model calls, output parsers, memory, tools — and there should be a standard way to do that. It has grown into the largest LLM framework by adoption: 119K GitHub stars, 500+ integrations, and a sprawling ecosystem. The LangChain team now builds LangGraph, a separate package for stateful, graph-based agent workflows, as the recommended way to build production agents within the ecosystem.
LlamaIndex (launched as GPT Index in November 2022) was built to solve a different problem: getting LLMs to reason over your own data. Its design is organized around data ingestion, chunking, embedding, indexing, and retrieval. Where LangChain is about orchestrating what happens between steps, LlamaIndex is about making the retrieval step itself as accurate and efficient as possible. It sits at 44K GitHub stars with 300+ data connectors through LlamaHub, covering sources like Notion, Google Drive, Slack, PDFs, and databases.
Raw API calls means using the OpenAI Python SDK, the Anthropic SDK, or any model provider’s client directly — no orchestration layer, no abstractions beyond what the provider ships. You write the prompt, call the model, and handle the response yourself. This is not the primitive fallback it is sometimes presented as; it is the approach production teams are increasingly migrating back to for workloads where the framework’s complexity stopped paying for itself.
The critical thing to understand before reading any comparison is that these three options are not competing on the same dimension. LangChain is an orchestration toolkit. LlamaIndex is a retrieval toolkit. Raw API calls are a stance on how much abstraction you need. Many production systems use two of them together. The question is always: given what I am actually building, which layer of abstraction earns its cost?
LangChain: The Orchestration Layer
LangChain’s strength is assembling complexity. If your application involves multiple steps, multiple tools, conditional routing, memory across turns, or agents that reason before acting, LangChain provides the building blocks for all of it, with connectors to 500+ services and a community large enough that someone has already solved most of the edge cases you will encounter.
LangGraph, built by the same team and stable at v1.0 since October 2025, is where the serious agent work lives now. It models agent workflows as directed graphs, where nodes are Python functions, edges are state transitions, and a central typed state object flows through the entire execution. It has built-in persistence via checkpointers to SQLite, PostgreSQL, or Redis, which means agents can pause mid-workflow, persist their state, and resume hours later. That is genuinely hard to build yourself and is one of LangChain’s clearest justifications in a production context.
The honest trade-offs are worth naming directly. LangChain adds ~10ms framework overhead per step, and LangGraph adds ~14ms. For most human-facing applications that make LLM calls taking 1–3 seconds each, this is irrelevant. For high-throughput pipelines processing thousands of requests per minute, it compounds. Stack traces from LangChain production errors routinely span 15 to 40 frames of internal framework code; finding the actual source of a bug is slower than in a system you wrote yourself. And for simple use cases, one documented comparison found LangChain incurring 2.7x higher costs than a native implementation for a basic RAG pipeline — the abstraction overhead consumed tokens that did not need to be consumed.
LangChain v1.0 (October 2025) committed to API stability after a turbulent v0.1 through v0.3 period that forced multiple breaking migrations. That history is worth knowing. For new projects, the stability concern is largely resolved. For teams running v0.x code in production, the migration cost to v1.0 is real.
Here is a working LangChain LCEL chain — the modern way to compose LangChain operations.
Prerequisites:
pip install langchain langchain-openai python-dotenv
pip install langchain langchain-openai python-dotenv
How to run: Save as langchain_chain.py, add OPENAI_API_KEY to your .env, run python langchain_chain.py
# langchain_chain.py
# A LangChain LCEL chain: prompt template → model → output parser
# Prerequisites: pip install langchain langchain-openai python-dotenv
# How to run: python langchain_chain.py
import os
from dotenv import load_dotenv
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_openai import ChatOpenAI
load_dotenv()
# ── MODEL ─────────────────────────────────────────────────────────────────────
# ChatOpenAI wraps OpenAI’s chat models. Swap the model string to switch
# to gpt-4o-mini (cheaper) or claude-3-5-sonnet (via langchain-anthropic) –
# the chain code below stays identical either way. This model portability
# is one of LangChain’s genuine advantages over raw API calls.
llm = ChatOpenAI(
model=”gpt-4o”,
temperature=0.2,
api_key=os.getenv(“OPENAI_API_KEY”)
)
# ── PROMPT TEMPLATE ───────────────────────────────────────────────────────────
# ChatPromptTemplate defines the message structure with named variables.
# {topic} gets filled in at runtime — templates are reusable and versionable.
prompt = ChatPromptTemplate.from_messages((
(“system”, “You are a concise technical explainer. Keep answers under 100 words.”),
(“human”, “Explain {topic} in simple terms.”)
))
# ── OUTPUT PARSER ─────────────────────────────────────────────────────────────
# StrOutputParser extracts the text content from the model’s AIMessage response.
# Without it you get back an AIMessage object rather than a plain string.
parser = StrOutputParser()
# ── CHAIN (LCEL) ──────────────────────────────────────────────────────────────
# The pipe operator (|) builds a sequential chain: prompt → llm → parser.
# LCEL (LangChain Expression Language) makes the composition readable and
# supports streaming, batching, and async execution with the same interface.
chain = prompt | llm | parser
if __name__ == “__main__”:
# invoke() runs the full chain synchronously
result = chain.invoke({“topic”: “vector embeddings”})
print(result)
# stream() yields tokens as they arrive — no code changes needed for streaming
print(“\n— Streaming response —“)
for chunk in chain.stream({“topic”: “RAG pipelines”}):
print(chunk, end=””, flush=True)
print()
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
# langchain_chain.py
# A LangChain LCEL chain: prompt template → model → output parser
# Prerequisites: pip install langchain langchain-openai python-dotenv
# How to run: python langchain_chain.py
import os
from dotenv import load_dotenv
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_openai import ChatOpenAI
load_dotenv()
# ── MODEL ─────────────────────────────────────────────────────────────────────
# ChatOpenAI wraps OpenAI’s chat models. Swap the model string to switch
# to gpt-4o-mini (cheaper) or claude-3-5-sonnet (via langchain-anthropic) —
# the chain code below stays identical either way. This model portability
# is one of LangChain’s genuine advantages over raw API calls.
llm = ChatOpenAI(
model=”gpt-4o”,
temperature=0.2,
api_key=os.getenv(“OPENAI_API_KEY”)
)
# ── PROMPT TEMPLATE ───────────────────────────────────────────────────────────
# ChatPromptTemplate defines the message structure with named variables.
# {topic} gets filled in at runtime — templates are reusable and versionable.
prompt = ChatPromptTemplate.from_messages((
(“system”, “You are a concise technical explainer. Keep answers under 100 words.”),
(“human”, “Explain {topic} in simple terms.”)
))
# ── OUTPUT PARSER ─────────────────────────────────────────────────────────────
# StrOutputParser extracts the text content from the model’s AIMessage response.
# Without it you get back an AIMessage object rather than a plain string.
parser = StrOutputParser()
# ── CHAIN (LCEL) ──────────────────────────────────────────────────────────────
# The pipe operator (|) builds a sequential chain: prompt → llm → parser.
# LCEL (LangChain Expression Language) makes the composition readable and
# supports streaming, batching, and async execution with the same interface.
chain = prompt | llm | parser
if __name__ == “__main__”:
# invoke() runs the full chain synchronously
result = chain.invoke({“topic”: “vector embeddings”})
print(result)
# stream() yields tokens as they arrive — no code changes needed for streaming
print(“\n— Streaming response —“)
for chunk in chain.stream({“topic”: “RAG pipelines”}):
print(chunk, end=””, flush=True)
print()
What this does: Three objects — prompt, llm, parser — are connected with the | operator. LangChain’s LCEL executes them in order: the template fills in {topic}, passes a formatted message to the model, and the parser extracts a plain string from the response. The same chain supports .invoke(), .stream(), .batch(), and .ainvoke() without any changes to the chain definition itself. That interface consistency is the clearest argument for LangChain on projects that need multiple execution patterns.
Here is the same foundation extended to a tool-using agent with LangGraph.
Prerequisites:
pip install langchain langchain-openai langgraph langchain-community python-dotenv
pip install langchain langchain-openai langgraph langchain-community python-dotenv
How to run: Save as langchain_agent.py and run python langchain_agent.py
# langchain_agent.py
# A LangGraph ReAct agent with two tools: web search and a calculator
# Prerequisites: pip install langchain langchain-openai langgraph langchain-community python-dotenv
# How to run: python langchain_agent.py
import os
from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
from langchain.tools import tool
from langchain_community.tools import DuckDuckGoSearchRun
from langchain_core.messages import HumanMessage
from langgraph.prebuilt import create_react_agent
load_dotenv()
llm = ChatOpenAI(model=”gpt-4o”, temperature=0, api_key=os.getenv(“OPENAI_API_KEY”))
# Web search — no API key required
search = DuckDuckGoSearchRun()
@tool
def calculate(expression: str) -> str:
“””
Evaluate a safe mathematical expression. Use for arithmetic or percentage calculations.
Input: a Python math expression string (e.g., ‘1500 * 0.08’).
“””
try:
result = eval(expression, {“__builtins__”: {}}, {})
return f”Result: {result}”
except Exception as e:
return f”Error: {str(e)}”
tools = (search, calculate)
# create_react_agent wires together the LLM, tools, and a built-in ReAct loop.
# The agent thinks, calls a tool, reads the result, and continues until done.
agent = create_react_agent(llm, tools)
if __name__ == “__main__”:
result = agent.invoke({
“messages”: (HumanMessage(content=”What is 15% of 2400?”))
})
print(result(“messages”)(-1).content)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
# langchain_agent.py
# A LangGraph ReAct agent with two tools: web search and a calculator
# Prerequisites: pip install langchain langchain-openai langgraph langchain-community python-dotenv
# How to run: python langchain_agent.py
import os
from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
from langchain.tools import tool
from langchain_community.tools import DuckDuckGoSearchRun
from langchain_core.messages import HumanMessage
from langgraph.prebuilt import create_react_agent
load_dotenv()
llm = ChatOpenAI(model=”gpt-4o”, temperature=0, api_key=os.getenv(“OPENAI_API_KEY”))
# Web search — no API key required
search = DuckDuckGoSearchRun()
@tool
def calculate(expression: str) -> str:
“””
Evaluate a safe mathematical expression. Use for arithmetic or percentage calculations.
Input: a Python math expression string (e.g., ‘1500 * 0.08’).
“””
try:
result = eval(expression, {“__builtins__”: {}}, {})
return f”Result: {result}”
except Exception as e:
return f”Error: {str(e)}”
tools = (search, calculate)
# create_react_agent wires together the LLM, tools, and a built-in ReAct loop.
# The agent thinks, calls a tool, reads the result, and continues until done.
agent = create_react_agent(llm, tools)
if __name__ == “__main__”:
result = agent.invoke({
“messages”: (HumanMessage(content=”What is 15% of 2400?”))
})
print(result(“messages”)(-1).content)
What this does: create_react_agent abstracts the full reasoning loop. The model decides whether to use a tool, LangGraph executes the selected tool, feeds the result back into the message history, and repeats until the model has a final answer. What would take 50+ lines in a raw implementation is four lines here. That abstraction is appropriate when you need it. The question the next section addresses is: when do you not?
LlamaIndex: The Retrieval Layer
LlamaIndex was designed from the ground up for one job: helping LLMs reason over external data. That focus is both its biggest strength and the clearest signal for when to use it. If your application’s central challenge is “how do I get the model to answer accurately from my documents,” LlamaIndex is the right starting point.
The performance numbers reflect that specialization. LlamaIndex indexes documents 2.5x faster than LangChain and hits sub-200ms query latency for 10,000 documents. Its framework overhead of ~6ms compares favorably to LangChain’s ~10ms and LangGraph’s ~14ms. At the token level, LlamaIndex uses ~1.6K tokens per query versus LangChain’s ~2.4K — a 33% difference that adds up quickly at scale.
The architectural reason for those differences is that LlamaIndex treats retrieval as a first-class primitive, not a composable component. Its five core abstractions — data connectors, node parsers, indices, query engines, and workflows — are designed to work together out of the box. Hierarchical chunking preserves parent-child relationships between document sections. Auto-merging retrieval recombines related chunks at query time. Sub-question decomposition breaks complex queries into simpler ones and merges the results. You get all of this with less code: LangChain requires 30–40% more code than LlamaIndex for equivalent RAG pipelines.
Where LlamaIndex is weaker is on the agent side. Its Workflows system handles async, event-driven pipelines well, but stateful multi-turn agents with built-in persistence require more manual implementation than LangGraph. LangGraph’s checkpointing — where an agent pauses, persists its full state, and resumes later — is something LlamaIndex Workflows can achieve but does not provide out of the box. For document Q&A and knowledge retrieval, this rarely matters. For long-running agentic workflows with human-in-the-loop requirements, it matters a great deal.
Here is a complete LlamaIndex RAG pipeline, from document ingestion to query.
Prerequisites:
pip install llama-index llama-index-llms-openai llama-index-embeddings-openai python-dotenv
pip install llama-index llama-index-llms-openai llama-index-embeddings-openai python-dotenv
How to run: Save as llamaindex_rag.py and run python llamaindex_rag.py
# llamaindex_rag.py
# Complete LlamaIndex RAG pipeline: ingest documents → index → query
# Prerequisites: pip install llama-index llama-index-llms-openai
# llama-index-embeddings-openai python-dotenv
# How to run: python llamaindex_rag.py
import os
from dotenv import load_dotenv
from llama_index.core import VectorStoreIndex, Document, Settings
from llama_index.llms.openai import OpenAI as LlamaOpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
load_dotenv()
# ── GLOBAL SETTINGS ───────────────────────────────────────────────────────────
# LlamaIndex v0.10+ uses a global Settings object instead of ServiceContext.
# Configure your LLM and embedding model once here — all pipeline components
# pick them up automatically. Swap models here to change the whole pipeline.
Settings.llm = LlamaOpenAI(
model=”gpt-4o”,
temperature=0,
api_key=os.getenv(“OPENAI_API_KEY”)
)
Settings.embed_model = OpenAIEmbedding(
model=”text-embedding-3-small”, # Fast and cost-effective for most RAG tasks
api_key=os.getenv(“OPENAI_API_KEY”)
)
# ── DOCUMENTS ─────────────────────────────────────────────────────────────────
# In production, replace with: SimpleDirectoryReader(“./docs”).load_data()
# LlamaHub provides 300+ connectors for Notion, Google Drive, PDFs, databases.
# Documents created inline here to keep the example fully self-contained.
documents = (
Document(
text=(
“LlamaIndex is a data framework for LLM applications. “
“It specializes in document ingestion, chunking, embedding, and retrieval. “
“Core abstractions: data connectors, node parsers, indices, query engines, “
“and workflows. LlamaHub provides 300+ pre-built data connectors.”
),
metadata={“source”: “llamaindex_overview”}
),
Document(
text=(
“LangChain is a general-purpose LLM orchestration framework. “
“It excels at chaining operations, multi-step agents, tool use, and memory. “
“LangGraph — the recommended way to build stateful agents in the LangChain “
“ecosystem — stabilized at v1.0 in October 2025.”
),
metadata={“source”: “langchain_overview”}
),
Document(
text=(
“Raw API calls use the OpenAI or Anthropic SDK directly with no framework. “
“This approach has the lowest latency and highest transparency. “
“Best for simple, one-off tasks where framework abstraction adds no value. “
“As complexity grows, a thin internal wrapper is usually preferable to “
“adopting a full orchestration framework.”
),
metadata={“source”: “raw_api_overview”}
),
)
# ── INDEX ─────────────────────────────────────────────────────────────────────
# from_documents() handles the full pipeline: chunk → embed → store.
# By default, vectors are stored in memory. For production, pass a vector store:
# index = VectorStoreIndex.from_documents(docs, storage_context=storage_context)
# where storage_context points to Pinecone, Weaviate, Chroma, etc.
index = VectorStoreIndex.from_documents(documents)
# ── QUERY ENGINE ──────────────────────────────────────────────────────────────
# as_query_engine() creates a retrieval + generation pipeline in one call.
# similarity_top_k=2 retrieves the 2 most relevant chunks per query.
# response_mode=”compact” merges retrieved chunks before passing to the LLM –
# reduces token usage compared to “default” mode, which sends each chunk separately.
query_engine = index.as_query_engine(
similarity_top_k=2,
response_mode=”compact”
)
if __name__ == “__main__”:
questions = (
“What is LlamaIndex best suited for?”,
“How does LangChain differ from LlamaIndex?”,
“When should I use raw API calls instead of a framework?”,
)
for q in questions:
print(f”Q: {q}”)
response = query_engine.query(q)
print(f”A: {response}\n”)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
# llamaindex_rag.py
# Complete LlamaIndex RAG pipeline: ingest documents → index → query
# Prerequisites: pip install llama-index llama-index-llms-openai
# llama-index-embeddings-openai python-dotenv
# How to run: python llamaindex_rag.py
import os
from dotenv import load_dotenv
from llama_index.core import VectorStoreIndex, Document, Settings
from llama_index.llms.openai import OpenAI as LlamaOpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
load_dotenv()
# ── GLOBAL SETTINGS ───────────────────────────────────────────────────────────
# LlamaIndex v0.10+ uses a global Settings object instead of ServiceContext.
# Configure your LLM and embedding model once here — all pipeline components
# pick them up automatically. Swap models here to change the whole pipeline.
Settings.llm = LlamaOpenAI(
model=”gpt-4o”,
temperature=0,
api_key=os.getenv(“OPENAI_API_KEY”)
)
Settings.embed_model = OpenAIEmbedding(
model=”text-embedding-3-small”, # Fast and cost-effective for most RAG tasks
api_key=os.getenv(“OPENAI_API_KEY”)
)
# ── DOCUMENTS ─────────────────────────────────────────────────────────────────
# In production, replace with: SimpleDirectoryReader(“./docs”).load_data()
# LlamaHub provides 300+ connectors for Notion, Google Drive, PDFs, databases.
# Documents created inline here to keep the example fully self-contained.
documents = (
Document(
text=(
“LlamaIndex is a data framework for LLM applications. “
“It specializes in document ingestion, chunking, embedding, and retrieval. “
“Core abstractions: data connectors, node parsers, indices, query engines, “
“and workflows. LlamaHub provides 300+ pre-built data connectors.”
),
metadata={“source”: “llamaindex_overview”}
),
Document(
text=(
“LangChain is a general-purpose LLM orchestration framework. “
“It excels at chaining operations, multi-step agents, tool use, and memory. “
“LangGraph — the recommended way to build stateful agents in the LangChain “
“ecosystem — stabilized at v1.0 in October 2025.”
),
metadata={“source”: “langchain_overview”}
),
Document(
text=(
“Raw API calls use the OpenAI or Anthropic SDK directly with no framework. “
“This approach has the lowest latency and highest transparency. “
“Best for simple, one-off tasks where framework abstraction adds no value. “
“As complexity grows, a thin internal wrapper is usually preferable to “
“adopting a full orchestration framework.”
),
metadata={“source”: “raw_api_overview”}
),
)
# ── INDEX ─────────────────────────────────────────────────────────────────────
# from_documents() handles the full pipeline: chunk → embed → store.
# By default, vectors are stored in memory. For production, pass a vector store:
# index = VectorStoreIndex.from_documents(docs, storage_context=storage_context)
# where storage_context points to Pinecone, Weaviate, Chroma, etc.
index = VectorStoreIndex.from_documents(documents)
# ── QUERY ENGINE ──────────────────────────────────────────────────────────────
# as_query_engine() creates a retrieval + generation pipeline in one call.
# similarity_top_k=2 retrieves the 2 most relevant chunks per query.
# response_mode=”compact” merges retrieved chunks before passing to the LLM —
# reduces token usage compared to “default” mode, which sends each chunk separately.
query_engine = index.as_query_engine(
similarity_top_k=2,
response_mode=”compact”
)
if __name__ == “__main__”:
questions = (
“What is LlamaIndex best suited for?”,
“How does LangChain differ from LlamaIndex?”,
“When should I use raw API calls instead of a framework?”,
)
for q in questions:
print(f”Q: {q}”)
response = query_engine.query(q)
print(f”A: {response}\n”)
What this does: Settings.llm and Settings.embed_model configure the entire pipeline once. VectorStoreIndex.from_documents() handles chunking, embedding, and indexing in a single call — a process that takes 30–40% more code in LangChain. as_query_engine() then creates a retrieval + generation pipeline with two lines. The similarity_top_k and response_mode parameters give you control over the retrieval behavior without requiring you to assemble the retrieval components yourself. That is the LlamaIndex value proposition in concrete form: less assembly, more retrieval quality.
A two-column architecture diagram comparing LlamaIndex and LangChain RAG pipelines side by side (click to enlarge)
Raw API Calls: The Minimal Path
The default assumption in most LLM developer communities is that you start with raw API calls and graduate to a framework as your project grows. The pattern worth examining in 2026 is the reverse: teams that started with LangChain and are quietly rewriting to raw SDKs.
The OpenAI Agents SDK, released in March 2025 with 26,900 GitHub stars and 10.3 million monthly downloads, provides tool use, multi-agent handoffs, built-in tracing, and guardrails in a minimal package. Its overhead per tool call is 2–5ms versus LangChain’s 10–30ms. Teams migrating from LangChain to raw SDKs typically see a 40–60% reduction in code volume and a 70–90% reduction in monthly framework maintenance burden.
The argument for the raw path is not that frameworks are bad. It is that the value of an abstraction layer depends entirely on whether it is hiding complexity you actually face. In 2022, building prompt chains and handling tool calls reliably required framework support because vendor APIs were inconsistent. By 2026, OpenAI and Anthropic have absorbed tool calling, streaming, function schemas, and multi-turn memory into their native SDKs. The framework’s abstractions no longer hide meaningful differences. They hide clarity.
Raw API is consistently the fastest option, with no framework overhead and no extra LLM calls for orchestration. Frameworks add 100–500ms of Python overhead per agent step. For latency-sensitive workloads — real-time customer support, voice agents, and high-throughput pipelines — that overhead is real and worth avoiding.
Here is a complete tool-using agent built on the raw OpenAI SDK in under 80 lines.
Prerequisites:
pip install openai python-dotenv
pip install openai python-dotenv
How to run: Save as raw_api_agent.py and run python raw_api_agent.py
# raw_api_agent.py
# A complete tool-using agent on the raw OpenAI SDK — no framework.
# This is ~75 lines including comments. Compare it to the LangChain equivalent.
# Prerequisites: pip install openai python-dotenv
# How to run: python raw_api_agent.py
import os
import json
from dotenv import load_dotenv
from openai import OpenAI
load_dotenv()
client = OpenAI(api_key=os.getenv(“OPENAI_API_KEY”))
# ── TOOL DEFINITIONS ──────────────────────────────────────────────────────────
# The model reads these descriptions to decide when and how to call each tool.
# Clear, specific descriptions are more important here than in any framework –
# there is no wrapper to fill in gaps.
TOOLS = (
{
“type”: “function”,
“function”: {
“name”: “calculate”,
“description”: (
“Evaluate a mathematical expression. Use for arithmetic, “
“percentages, or numerical computation. “
“Input: a Python math expression as a string.”
),
“parameters”: {
“type”: “object”,
“properties”: {
“expression”: {
“type”: “string”,
“description”: “A Python math expression, e.g. ‘1500 * 0.08′”
}
},
“required”: (“expression”)
}
}
},
{
“type”: “function”,
“function”: {
“name”: “get_word_count”,
“description”: “Count the number of words in a given string of text.”,
“parameters”: {
“type”: “object”,
“properties”: {
“text”: {“type”: “string”, “description”: “The text to count.”}
},
“required”: (“text”)
}
}
}
)
# ── TOOL IMPLEMENTATIONS ──────────────────────────────────────────────────────
def calculate(expression: str) -> str:
try:
result = eval(expression, {“__builtins__”: {}}, {})
return str(result)
except Exception as e:
return f”Error: {e}”
def get_word_count(text: str) -> str:
return str(len(text.split()))
# Maps tool name → Python function for dynamic dispatch in the loop below
TOOL_DISPATCH = {“calculate”: calculate, “get_word_count”: get_word_count}
# ── AGENT LOOP ────────────────────────────────────────────────────────────────
def run_agent(user_message: str) -> str:
“””
A complete ReAct-style agent loop using raw OpenAI tool calls.
The model decides whether to call a tool or return a final answer.
The loop continues until the model stops requesting tool calls.
Every step is visible — no framework wrapping, no hidden logic.
“””
messages = (
{“role”: “system”, “content”: “You are a helpful assistant.”},
{“role”: “user”, “content”: user_message},
)
while True:
response = client.chat.completions.create(
model=”gpt-4o”,
messages=messages,
tools=TOOLS,
tool_choice=”auto”, # Model decides: call a tool or respond directly
temperature=0,
)
message = response.choices(0).message
messages.append(message) # Always add the assistant message to history
# No tool calls = the model has its final answer
if not message.tool_calls:
return message.content
# Execute each tool call the model requested
for tool_call in message.tool_calls:
name = tool_call.function.name
args = json.loads(tool_call.function.arguments)
fn = TOOL_DISPATCH.get(name)
result = fn(**args) if fn else f”Unknown tool: {name}”
# Tool result goes back into the message history.
# The model reads this on the next iteration to decide what to do next.
messages.append({
“role”: “tool”,
“tool_call_id”: tool_call.id,
“content”: result,
})
# Loop — the model now processes the tool results
if __name__ == “__main__”:
queries = (
“What is 18% of 3500?”,
“How many words are in: The quick brown fox jumps over the lazy dog?”,
“Split 240 items into groups of 16. How many groups?”,
)
for q in queries:
print(f”Q: {q}\nA: {run_agent(q)}\n”)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
# raw_api_agent.py
# A complete tool-using agent on the raw OpenAI SDK — no framework.
# This is ~75 lines including comments. Compare it to the LangChain equivalent.
# Prerequisites: pip install openai python-dotenv
# How to run: python raw_api_agent.py
import os
import json
from dotenv import load_dotenv
from openai import OpenAI
load_dotenv()
client = OpenAI(api_key=os.getenv(“OPENAI_API_KEY”))
# ── TOOL DEFINITIONS ──────────────────────────────────────────────────────────
# The model reads these descriptions to decide when and how to call each tool.
# Clear, specific descriptions are more important here than in any framework —
# there is no wrapper to fill in gaps.
TOOLS = (
{
“type”: “function”,
“function”: {
“name”: “calculate”,
“description”: (
“Evaluate a mathematical expression. Use for arithmetic, “
“percentages, or numerical computation. “
“Input: a Python math expression as a string.”
),
“parameters”: {
“type”: “object”,
“properties”: {
“expression”: {
“type”: “string”,
“description”: “A Python math expression, e.g. ‘1500 * 0.08′”
}
},
“required”: (“expression”)
}
}
},
{
“type”: “function”,
“function”: {
“name”: “get_word_count”,
“description”: “Count the number of words in a given string of text.”,
“parameters”: {
“type”: “object”,
“properties”: {
“text”: {“type”: “string”, “description”: “The text to count.”}
},
“required”: (“text”)
}
}
}
)
# ── TOOL IMPLEMENTATIONS ──────────────────────────────────────────────────────
def calculate(expression: str) -> str:
try:
result = eval(expression, {“__builtins__”: {}}, {})
return str(result)
except Exception as e:
return f”Error: {e}”
def get_word_count(text: str) -> str:
return str(len(text.split()))
# Maps tool name → Python function for dynamic dispatch in the loop below
TOOL_DISPATCH = {“calculate”: calculate, “get_word_count”: get_word_count}
# ── AGENT LOOP ────────────────────────────────────────────────────────────────
def run_agent(user_message: str) -> str:
“””
A complete ReAct-style agent loop using raw OpenAI tool calls.
The model decides whether to call a tool or return a final answer.
The loop continues until the model stops requesting tool calls.
Every step is visible — no framework wrapping, no hidden logic.
“””
messages = (
{“role”: “system”, “content”: “You are a helpful assistant.”},
{“role”: “user”, “content”: user_message},
)
while True:
response = client.chat.completions.create(
model=”gpt-4o”,
messages=messages,
tools=TOOLS,
tool_choice=”auto”, # Model decides: call a tool or respond directly
temperature=0,
)
message = response.choices(0).message
messages.append(message) # Always add the assistant message to history
# No tool calls = the model has its final answer
if not message.tool_calls:
return message.content
# Execute each tool call the model requested
for tool_call in message.tool_calls:
name = tool_call.function.name
args = json.loads(tool_call.function.arguments)
fn = TOOL_DISPATCH.get(name)
result = fn(**args) if fn else f”Unknown tool: {name}”
# Tool result goes back into the message history.
# The model reads this on the next iteration to decide what to do next.
messages.append({
“role”: “tool”,
“tool_call_id”: tool_call.id,
“content”: result,
})
# Loop — the model now processes the tool results
if __name__ == “__main__”:
queries = (
“What is 18% of 3500?”,
“How many words are in: The quick brown fox jumps over the lazy dog?”,
“Split 240 items into groups of 16. How many groups?”,
)
for q in queries:
print(f”Q: {q}\nA: {run_agent(q)}\n”)
What this does: The agent loop is fully transparent. There is no framework between you and the model’s response. The while True loop runs until message.tool_calls is empty, which happens when the model decides it has enough information to answer directly. Every message — system, user, assistant, and tool result — is in a plain Python list you can inspect, log, or modify at any point. That transparency is the raw path’s core advantage: when something breaks, you know exactly where to look.
Head-to-Head Comparison
The same task was evaluated across three dimensions. All measurements reflect current benchmarks from independent analysis cited throughout this article.
Framework Overhead and Performance
Metric
Raw API
LlamaIndex
LangChain (LCEL)
LangGraph
Framework overhead
~0ms
~6ms
~10ms
~14ms
Token overhead (per query)
0
~1.6K
~2.4K
~2.0K
Per tool call latency
2–5ms
N/A
10–30ms
10–30ms
Stack trace depth on error
2–5 frames
5–10 frames
15–40 frames
15–40 frames
Debug transparency
High
Medium
Low
Low
Code Volume: Same RAG Task, Three Ways
This is the most concrete way to feel the trade-off. All three implementations below answer the same question from the same context document:
Implementation
Lines of code
Framework install size
Debugging clarity
Raw OpenAI SDK
~20 lines
openai only
Full visibility
LlamaIndex
~15 lines
llama-index + plugins
Medium
LangChain LCEL
~18 lines
langchain + langchain-openai
Low–medium
For a basic one-document Q&A, the difference is marginal. Where LlamaIndex’s code advantage compounds is when you add chunking strategies, multiple documents, re-ranking, metadata filtering, and hybrid search — each of which requires more assembly in LangChain than in LlamaIndex.
When Each One Breaks
Knowing when each approach fails is as useful as knowing when it succeeds.
Failure mode
Raw API
LlamaIndex
LangChain
Retrieval accuracy degrades
You built it, you fix it
Tune chunking/index strategy
Tune each pipeline component separately
Agent loops indefinitely
Add max_iterations manually
Workflow timeout
max_iterations parameter
Prompt changes break output
Immediate, obvious
Immediate, obvious
May propagate through chain silently
Model API changes
Update SDK
Update llama-index package
Update langchain-openai + retest
Debugging a production error
Direct, small stack
Moderate
Deep stack traces, hard to isolate
Scaling to high throughput
Optimal
Good
Framework overhead compounds
Full Working Example
The same document Q&A task implemented three ways. Same input document, same question, different path through the stack. Read these side by side and the trade-offs become concrete.
Prerequisites:
pip install openai langchain langchain-openai llama-index \
llama-index-llms-openai llama-index-embeddings-openai python-dotenv
pip install openai langchain langchain-openai llama-index \
llama-index-llms-openai llama-index-embeddings-openai python-dotenv
How to run: Save as three_ways.py and run python three_ways.py
# three_ways.py
# The same document Q&A task implemented three ways:
# Raw OpenAI SDK, LlamaIndex, and LangChain LCEL.
# Same input. Same output. Different path through the stack.
# Prerequisites: pip install openai langchain langchain-openai llama-index
# llama-index-llms-openai llama-index-embeddings-openai python-dotenv
# How to run: python three_ways.py
import os
import time
from dotenv import load_dotenv
load_dotenv()
QUESTION = “What is retrieval-augmented generation and why does it matter?”
CONTEXT_DOC = “””
Retrieval-Augmented Generation (RAG) is a technique that improves LLM responses
by fetching relevant context from an external knowledge base before generating
an answer. Instead of relying solely on training data, RAG retrieves the most
relevant document chunks and includes them in the prompt. This reduces
hallucinations, keeps answers grounded in your actual data, and allows the model
to answer questions about information it was never trained on.
“””
# ─────────────────────────────────────────────────────────────────────────────
# APPROACH 1: RAW OPENAI SDK
# When to use: simple, one-off calls where full visibility matters most
# ─────────────────────────────────────────────────────────────────────────────
def raw_api_answer(question: str, context: str) -> str:
“””Answer a question using context, via raw OpenAI SDK — no framework.”””
from openai import OpenAI
client = OpenAI(api_key=os.getenv(“OPENAI_API_KEY”))
# Everything is explicit: the system prompt, the context injection,
# the message structure. Nothing is hidden in a framework abstraction.
response = client.chat.completions.create(
model=”gpt-4o”,
temperature=0,
messages=(
{
“role”: “system”,
“content”: (
“Answer questions using only the provided context. “
“If the answer is not in the context, say so clearly.”
)
},
{
“role”: “user”,
“content”: f”Context:\n{context}\n\nQuestion: {question}”
}
)
)
return response.choices(0).message.content
# ─────────────────────────────────────────────────────────────────────────────
# APPROACH 2: LLAMAINDEX
# When to use: document-heavy retrieval where you want optimized RAG out of the box
# ─────────────────────────────────────────────────────────────────────────────
def llamaindex_answer(question: str, context: str) -> str:
“””Answer a question using LlamaIndex — purpose-built retrieval pipeline.”””
from llama_index.core import VectorStoreIndex, Document, Settings
from llama_index.llms.openai import OpenAI as LlamaOpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
# Configure once — all pipeline components pick it up
Settings.llm = LlamaOpenAI(
model=”gpt-4o”, temperature=0,
api_key=os.getenv(“OPENAI_API_KEY”)
)
Settings.embed_model = OpenAIEmbedding(
model=”text-embedding-3-small”,
api_key=os.getenv(“OPENAI_API_KEY”)
)
# from_documents() = chunk + embed + index in one call
# For multiple documents, pass a list: from_documents((doc1, doc2, doc3))
index = VectorStoreIndex.from_documents((Document(text=context)))
# as_query_engine() = retriever + generator, wired together automatically
query_engine = index.as_query_engine(similarity_top_k=1)
return str(query_engine.query(question))
# ─────────────────────────────────────────────────────────────────────────────
# APPROACH 3: LANGCHAIN LCEL
# When to use: workflows that will grow to include agents, memory, or routing
# ─────────────────────────────────────────────────────────────────────────────
def langchain_answer(question: str, context: str) -> str:
“””Answer a question using a LangChain LCEL chain.”””
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
model=”gpt-4o”, temperature=0,
api_key=os.getenv(“OPENAI_API_KEY”)
)
prompt = ChatPromptTemplate.from_messages((
(“system”,
“Answer using only the provided context. “
“If the answer is not in the context, say so.\n\nContext:\n{context}”),
(“human”, “{question}”)
))
# The same chain supports .stream(), .batch(), .ainvoke() — no code changes needed
chain = prompt | llm | StrOutputParser()
return chain.invoke({“context”: context, “question”: question})
# ─────────────────────────────────────────────────────────────────────────────
# RUN ALL THREE AND COMPARE
# ─────────────────────────────────────────────────────────────────────────────
if __name__ == “__main__”:
approaches = (
(“Raw OpenAI SDK”, raw_api_answer),
(“LlamaIndex”, llamaindex_answer),
(“LangChain LCEL”, langchain_answer),
)
for name, fn in approaches:
print(f”\n{‘=’*60}”)
print(f”Approach: {name}”)
print(f”{‘=’*60}”)
start = time.perf_counter()
answer = fn(QUESTION, CONTEXT_DOC)
elapsed = time.perf_counter() – start
print(f”Answer: {answer}”)
print(f”Time (excluding LLM): visible in wall clock”)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
# three_ways.py
# The same document Q&A task implemented three ways:
# Raw OpenAI SDK, LlamaIndex, and LangChain LCEL.
# Same input. Same output. Different path through the stack.
# Prerequisites: pip install openai langchain langchain-openai llama-index
# llama-index-llms-openai llama-index-embeddings-openai python-dotenv
# How to run: python three_ways.py
import os
import time
from dotenv import load_dotenv
load_dotenv()
QUESTION = “What is retrieval-augmented generation and why does it matter?”
CONTEXT_DOC = “””
Retrieval-Augmented Generation (RAG) is a technique that improves LLM responses
by fetching relevant context from an external knowledge base before generating
an answer. Instead of relying solely on training data, RAG retrieves the most
relevant document chunks and includes them in the prompt. This reduces
hallucinations, keeps answers grounded in your actual data, and allows the model
to answer questions about information it was never trained on.
“””
# ─────────────────────────────────────────────────────────────────────────────
# APPROACH 1: RAW OPENAI SDK
# When to use: simple, one-off calls where full visibility matters most
# ─────────────────────────────────────────────────────────────────────────────
def raw_api_answer(question: str, context: str) -> str:
“””Answer a question using context, via raw OpenAI SDK — no framework.”””
from openai import OpenAI
client = OpenAI(api_key=os.getenv(“OPENAI_API_KEY”))
# Everything is explicit: the system prompt, the context injection,
# the message structure. Nothing is hidden in a framework abstraction.
response = client.chat.completions.create(
model=”gpt-4o”,
temperature=0,
messages=(
{
“role”: “system”,
“content”: (
“Answer questions using only the provided context. “
“If the answer is not in the context, say so clearly.”
)
},
{
“role”: “user”,
“content”: f”Context:\n{context}\n\nQuestion: {question}”
}
)
)
return response.choices(0).message.content
# ─────────────────────────────────────────────────────────────────────────────
# APPROACH 2: LLAMAINDEX
# When to use: document-heavy retrieval where you want optimized RAG out of the box
# ─────────────────────────────────────────────────────────────────────────────
def llamaindex_answer(question: str, context: str) -> str:
“””Answer a question using LlamaIndex — purpose-built retrieval pipeline.”””
from llama_index.core import VectorStoreIndex, Document, Settings
from llama_index.llms.openai import OpenAI as LlamaOpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
# Configure once — all pipeline components pick it up
Settings.llm = LlamaOpenAI(
model=”gpt-4o”, temperature=0,
api_key=os.getenv(“OPENAI_API_KEY”)
)
Settings.embed_model = OpenAIEmbedding(
model=”text-embedding-3-small”,
api_key=os.getenv(“OPENAI_API_KEY”)
)
# from_documents() = chunk + embed + index in one call
# For multiple documents, pass a list: from_documents((doc1, doc2, doc3))
index = VectorStoreIndex.from_documents((Document(text=context)))
# as_query_engine() = retriever + generator, wired together automatically
query_engine = index.as_query_engine(similarity_top_k=1)
return str(query_engine.query(question))
# ─────────────────────────────────────────────────────────────────────────────
# APPROACH 3: LANGCHAIN LCEL
# When to use: workflows that will grow to include agents, memory, or routing
# ─────────────────────────────────────────────────────────────────────────────
def langchain_answer(question: str, context: str) -> str:
“””Answer a question using a LangChain LCEL chain.”””
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
model=”gpt-4o”, temperature=0,
api_key=os.getenv(“OPENAI_API_KEY”)
)
prompt = ChatPromptTemplate.from_messages((
(“system”,
“Answer using only the provided context. “
“If the answer is not in the context, say so.\n\nContext:\n{context}”),
(“human”, “{question}”)
))
# The same chain supports .stream(), .batch(), .ainvoke() — no code changes needed
chain = prompt | llm | StrOutputParser()
return chain.invoke({“context”: context, “question”: question})
# ─────────────────────────────────────────────────────────────────────────────
# RUN ALL THREE AND COMPARE
# ─────────────────────────────────────────────────────────────────────────────
if __name__ == “__main__”:
approaches = (
(“Raw OpenAI SDK”, raw_api_answer),
(“LlamaIndex”, llamaindex_answer),
(“LangChain LCEL”, langchain_answer),
)
for name, fn in approaches:
print(f”\n{‘=’*60}”)
print(f”Approach: {name}”)
print(f”{‘=’*60}”)
start = time.perf_counter()
answer = fn(QUESTION, CONTEXT_DOC)
elapsed = time.perf_counter() – start
print(f”Answer: {answer}”)
print(f”Time (excluding LLM): visible in wall clock”)
What this does: All three functions receive the same QUESTION and CONTEXT_DOC and return a string answer. The raw API version manually constructs the message list and extracts the response. The LlamaIndex version uses from_documents() and as_query_engine() to handle the pipeline. The LangChain version assembles a prompt, model, and parser with the | operator. At this scale — one document, one question — the differences are minimal. Feed this function 500 documents and a complex query, and the gap between LlamaIndex’s purpose-built retrieval and the other two approaches opens up significantly.
Wrapping Up
The framework decision is not about which option has the most GitHub stars or the most features. It is about matching the abstraction level of your tool to the actual complexity of your problem.
For simple, one-shot tasks, raw API calls are faster to write, faster to run, and easier to debug than any framework. For document retrieval at any meaningful scale, LlamaIndex earns its dependency through better chunking, faster indexing, and less code. For stateful agents with memory, tools, and multi-step reasoning, LangGraph’s persistence and graph-based control flow are genuinely hard to replicate cleanly with a hand-rolled loop.
The pattern that most production teams converge on by mid-2026 is not a single framework but a layered stack: raw SDK for the simple calls, LlamaIndex for the retrieval layer, LangGraph for the agent loop, and LangSmith for tracing across everything. None of those choices locks you out of the others. They compose.
The practical rule is this: start with the minimal option that handles your current requirements, and add a framework when you hit a problem the framework was built to solve — not before. A retrieval problem you encounter is a reason to add LlamaIndex. A state management problem you encounter is a reason to add LangGraph. Adding either before you feel the pain they address means adding maintenance overhead for a future problem that may not arrive in the shape you expected.
GIPHY App Key not set. Please check settings