Is it possible to use create_sql_agent with ollama models?

[cboettig]

The docs describe how to create an SQL agent using OpenAI as an example but implying that the approach is generic.

I am able to use create_sql_query_chain just fine against either an OpenAI LLM or an Ollama LLM (examples below). However, when I try to use create_sql_agent on the Ollama LLM, the agent throws an error on any agent.run() call, while the identical setup works fine with the OpenAI agent.

I have seen the ollama-sql template, but this seems to create a 'chain' (a la create_sql_query_chain?) and not an agent? I don't quite see how to use it. I'd like to use create_sql_agent() with a ollama agent -- the documentation seems to imply this should be possible.

Setup:

## My database connection
from sqlalchemy import create_engine
from langchain.sql_database import SQLDatabase
db_uri = "duckdb:///database.duckdb"
engine = create_engine(db_uri)
db = SQLDatabase(engine, view_support=True)

create_sql_query_chain works just fine:

from langchain_community.llms import Ollama
llm = Ollama(model="duckdb-nsql", temperature=0)


## A SQL Chain
from langchain.chains import create_sql_query_chain
chain = create_sql_query_chain(llm, db)

# we can do stuff with the chain
chain.get_prompts()[0].pretty_print()
response = chain.invoke({"question": "How many total rows are in the fee table?"})
response

But creating an agent does not:

# An agent -- does not translate to non OpenAI??
from langchain_community.agent_toolkits import create_sql_agent
agent = create_sql_agent(llm, db=db, verbose=True)
agent.run("How many total rows are in the fee table?")

ERROR:

ValueError: An output parsing error occurred. In order to pass this error back to the agent and have it try again, pass `handle_parsing_errors=True` to the AgentExecutor. This is the error: Could not parse LLM output: ` SELECT COUNT(*) FROM fee;`

The same code executes fine with an ChatOpenAI LLM

## ChatGPT Connection
import os
import streamlit as st
from langchain_openai import ChatOpenAI
os.environ["OPENAI_API_KEY"] = st.secrets["OPENAI_API_KEY"]
chatgpt = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)

# An agent can execute queries
from langchain_community.agent_toolkits import create_sql_agent
chatgpt_agent = create_sql_agent(chatgpt, db=db, agent_type="openai-tools", verbose=True)
chatgpt_agent.run("How many total rows are in the fee table?")

[agenthunt007]

were you able to make it work? I am running into similar issue

[maximeperrindev]

Hi,

create_sql_agent uses tools under the hood to communicate with your database (https://python.langchain.com/v0.1/docs/use_cases/sql/agents/). That means the LLM model should support tool calling (e.g being fine tuned to use tools and correct format).
Ollama models do not support tool calling natively.

You can check this answer : #21907 (comment)

[IP050]

Since some models available within ollama now allow toolcalling you can indeed use an ollama model such as llama3-groq-tool-use, either with create_sql_agent or as post above stated using the tools that it uses under the hood directly.

If you want to use create_sql_agent, you must specify the toolkit over db, it uses get_tools() itself so no need for that:

from langchain_core.messages import HumanMessage
from langchain_ollama import ChatOllama
from langchain_community.agent_toolkits import SQLDatabaseToolkit
from langchain_community.agent_toolkits import create_sql_agent

llm = ChatOllama(model="llama3-groq-tool-use:latest", base_url="{url}") 
toolkit = SQLDatabaseToolkit(db=db, llm=llm) 
agent_executor = create_sql_agent(llm, toolkit=toolkit, agent_type="tool-calling")
agent_executor.invoke([HumanMessage(content="A question about the database?")])

Should help you on the way.

I do highly recommend looking under the hood and using either the langgraph.prebuilt create_react_agent or stepping away from usage of prebuilt/community components for a while and diving deeper in it's core to gain more understanding of both langchain/langgraph, it can save you lots of time and makes you more flexible versus changes within langchain, langgraph, models or stuff like streamlit etc.

[Apeto13]

I used it locally but it does not seem to interact with my data ?

It is hallucinating making up table that do not exist within my database.