Utilizing Scikit-LLM with Open-Supply LLMs

In this article, you will learn how to use locally hosted language models through Ollama to perform text classification tasks, all without spending a cent on API calls.

Topics we will cover include:

How to install Ollama and pull open-source models like Llama 3, Mistral, and Gemma to run locally on your machine.
How to configure the Scikit-LLM library to route requests to a local Ollama endpoint instead of a paid cloud API.
How to build a zero-shot text classifier using a local large language model and scikit-LLM in a familiar scikit-learn-style workflow.

Using Scikit-LLM with Open-Source LLMs

Introduction

This article will teach you how to perform a language task like text classification by integrating locally hosted large language models (LLMs) of manageable size, like Mistral, Gemma, and Llama 3: all for free thanks to Ollama — a free repository for local LLMs — and the Scikit-LLM Python library.

Pre-requisite: Installing Ollama

It is recommended to use an IDE to run this tutorial, as we will need to interact with your locally installed version of Ollama from there. New to Ollama? Then I recommend you check this article out first. Nonetheless, here is a summary of what to do in the local command line terminal to download a local LLM after installing To be on your computer.

# Pulling Llama 3 (one of Ollama’s most popular downloadable models)
ollama run llama3

# Or alternatively, try pulling Mistral
ollama run mistral

# Or, if you feel picky today, just pull Google’s Gemma
ollama run gemma

# Pulling Llama 3 (one of Ollama’s most popular downloadable models)

ollama run llama3

# Or alternatively, try pulling Mistral

ollama run mistral

# Or, if you feel picky today, just pull Google’s Gemma

ollama run gemma

Once you see the model interaction window in the terminal, you can type “/bye” to keep it running in the background, waiting for API calls. Meanwhile, in a newly created project in your Python IDE, you will need to have the following libraries installed:

pip install scikit-learn pandas scikit-llm

If you encounter a “Module not found” error when executing the Python code, try installing the above dependencies one by one.

Okay! Time to fill in our Python code file (name it as you wish!), step by step. First, of course, come the imports. One of them is the class ZeroShotGPTClassifier. Similar to classical scikit-learn, this is a dedicated class for training and using a model for zero-shot classification: concretely, an LLM from Ollama.

import pandas as pd
from sklearn.model_selection import train_test_split
from skllm.config import SKLLMConfig
from skllm.models.gpt.classification.zero_shot import ZeroShotGPTClassifier

import pandas as pd

from sklearn.model_selection import train_test_split

from skllm.config import SKLLMConfig

from skllm.models.gpt.classification.zero_shot import ZeroShotGPTClassifier

Next, we need to apply a couple of specific configurations to be able to communicate with Ollama.

# Use this to tell Scikit-LLM to route cloud requests towards your default local Ollama port
SKLLMConfig.set_gpt_url(“http://localhost:11434/v1”)

# Scikit-LLM needs, by default, a key to pass internal validation checks.
# But because Ollama is local and free, this string will be ignored in practice.
SKLLMConfig.set_openai_key(“local-ollama-is-free”)

# Use this to tell Scikit-LLM to route cloud requests towards your default local Ollama port

SKLLMConfig.set_gpt_url(“http://localhost:11434/v1”)

# Scikit-LLM needs, by default, a key to pass internal validation checks.

# But because Ollama is local and free, this string will be ignored in practice.

SKLLMConfig.set_openai_key(“local-ollama-is-free”)

After that, we create a small dataset and prepare it for classification. Since we are not going to evaluate the model’s classification performance in this tutorial — our main goal is to learn how to use Scikit-LLM locally with open-source models like those available through Ollama — we do not need a large number of data examples.

data = {
“review”: (
“The new macOS update is fantastic and runs smoothly.”,
“My battery is draining incredibly fast after the patch.”,
“I need help resetting my account password.”,
“The display on this monitor is breathtakingly crisp.”,
“Customer support hung up on me, very disappointing.”
),
“category”: (
“Positive Feedback”,
“Technical Issue”,
“Support Request”,
“Positive Feedback”,
“Negative Feedback”
)
}

df = pd.DataFrame(data)
X = df(“review”)
y = df(“category”)

# Splitting data into train/test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42)

data = {

“review”: (

“The new macOS update is fantastic and runs smoothly.”,

“My battery is draining incredibly fast after the patch.”,

“I need help resetting my account password.”,

“The display on this monitor is breathtakingly crisp.”,

“Customer support hung up on me, very disappointing.”

“category”: (

“Positive Feedback”,

“Technical Issue”,

“Support Request”,

“Positive Feedback”,

“Negative Feedback”

)

}

df = pd.DataFrame(data)

X = df(“review”)

y = df(“category”)

# Splitting data into train/test sets

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42)

The dataset contains user reviews and their corresponding categories, e.g. types of customer inquiries or feedback. We also made a training/test split as usual with machine learning modeling.

In the next part of the code, we add the necessary instructions for initializing and running our classifier, which will be at its core a task-adapted running instance of one of our installed Ollama models, such as Llama 3:

print(“Initializing ZeroShotGPTClassifier with local Llama 3…”)

# Using the ‘custom_url::’ prefix to tell the system to use your “set_gpt_url” endpoint (see above)
clf = ZeroShotGPTClassifier(model=”custom_url::llama3″)

# Fitting the model
clf.fit(X_train, y_train)

print(“Sending data to Ollama for local inference…\n”)
predictions = clf.predict(X_test)

print(“Initializing ZeroShotGPTClassifier with local Llama 3…”)

# Using the ‘custom_url::’ prefix to tell the system to use your “set_gpt_url” endpoint (see above)

clf = ZeroShotGPTClassifier(model=”custom_url::llama3″)

# Fitting the model

clf.fit(X_train, y_train)

print(“Sending data to Ollama for local inference…\n”)

predictions = clf.predict(X_test)

To finish up, we print some outputs consisting of a couple of model inference results (classification predictions) on the two examples contained in the test set. This is a very small dataset, but the aim here is to show how we managed to link Scikit-LLM with a local, free Ollama model to elegantly use an LLM for a specific task at no cost!

for review, prediction in zip(X_test, predictions):
print(f”Review Text: ‘{review}'”)
print(f”Predicted Tag: {prediction}”)
print(“-” * 50)

for review, prediction in zip(X_test, predictions):

print(f”Review Text: ‘{review}'”)

print(f”Predicted Tag: {prediction}”)

print(“-” * 50)

The result (it may vary depending on your test examples):

Sending data to Ollama for local inference…

100%|███████████████████████████████████████████████████████████| 2/2 (00:12<00:00, 6.36s/it) Review Text: 'My battery is draining incredibly fast after the patch.' Predicted Tag: Support Request -------------------------------------------------- Review Text: 'Customer support hung up on me, very disappointing.' Predicted Tag: Support Request --------------------------------------------------

Sending data to Ollama for local inference…

100%|███████████████████████████████████████████████████████████| 2/2 (00:12<00:00, 6.36s/it)

Review Text: ‘My battery is draining incredibly fast after the patch.’

Predicted Tag: Support Request

————————————————–

Review Text: ‘Customer support hung up on me, very disappointing.’

Predicted Tag: Support Request

————————————————–

Alternatively, you could run your Python script from your terminal. For example, if you named it local_classification.py, execute this command:

python local_classification.py

Either way, if you followed all the steps, you should have it working. Well done!

Wrapping Up

This article illustrated how to swap in free, locally run models served through Ollama, such as Llama, Mistral, or Gemma — all for free, and in a few easy steps — thanks to Python’s Scikit-LLM library, which enables the use of cutting-edge LLMs within a familiar classical machine learning workflow.

Source link