Guide · Open source
How to use CLIP: install it and classify images with text labels
Updated · written with AI from the project's own documentation and checked against our catalogue data
CLIP is an open-source neural network from OpenAI that scores how well a short text snippet fits an image. This guide is for Python users who want to try it, and it walks through installation, a working prediction example, and the main API calls. By the end you will be able to load a CLIP model, pass in an image and a list of plain-language labels, and read back which label fits best.
CLIPopenai/CLIP · ★ 34.6k · MIT Install options & downloads Quick start
- 1Check prerequisites. Make sure you have Python, PyTorch 1.7.1 or later, and torchvision. Note whether your machine has a CUDA GPU.
- 2Install dependencies. Install PyTorch and torchvision with conda, then install ftfy, regex and tqdm with pip.
- 3Install CLIP. Run the pip install command that points to the openai/CLIP GitHub repository.
- 4Load a model. Call clip.load with a model name such as ViT-B/32 and choose cuda or cpu as the device.
- 5Prepare image and labels. Preprocess your image with the returned transform and tokenize your plain-language labels with clip.tokenize.
- 6Read the scores. Run the model under torch.no_grad(), apply softmax to the image logits, and pick the label with the highest probability.
What CLIP does and who it suits
CLIP (Contrastive Language-Image Pre-Training) is trained on many (image, text) pairs. You can instruct it in natural language to predict the most relevant text snippet for an image. According to the README, it can do this without being directly optimized for the task, and it matches the original ResNet50 on ImageNet zero-shot without using the 1.28M labeled ImageNet examples.
- Python developers experimenting with image classification using text labels
- Researchers evaluating zero-shot image recognition
- Students learning how vision-language models work
- Teams that want to classify images without collecting a labeled dataset first
Install PyTorch, torchvision and CLIP
You need Python and PyTorch 1.7.1 or later, plus torchvision. The README also lists small extra dependencies, ftfy, regex and tqdm, and then installs this repo as a Python package. The commands below are for a CUDA GPU machine.
Replace cudatoolkit=11.0 with the CUDA version that matches your machine, or use cpuonly on a machine without a GPU. The README gives the PyTorch install page as the reference for other setups.
$ conda install --yes -c pytorch pytorch=1.7.1 torchvision cudatoolkit=11.0
$ pip install ftfy regex tqdm
$ pip install git+https://github.com/openai/CLIP.gitRun zero-shot prediction on an image
The core workflow is: load the model with clip.load, preprocess your image, tokenize your labels with clip.tokenize, then compare them. The device defaults to CUDA if available, otherwise CPU. The README example uses the ViT-B/32 model and a set of three labels.
The model returns logits_per_image, which are cosine similarities times 100. Applying softmax turns them into probabilities that sum to one across your labels. The README's example prints a probability of about 0.99 for "a diagram" on its own sample image, and the exact numbers may vary by device.
import torch
import clip
from PIL import Image
device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)
image = preprocess(Image.open("CLIP.png")).unsqueeze(0).to(device)
text = clip.tokenize(["a diagram", "a dog", "a cat"]).to(device)
with torch.no_grad():
image_features = model.encode_image(image)
text_features = model.encode_text(text)
logits_per_image, logits_per_text = model(image, text)
probs = logits_per_image.softmax(dim=-1).cpu().numpy()
print("Label probs:", probs)Use the main API calls
The clip module provides a small set of functions. clip.available_models() lists model names. clip.load(name, device=..., jit=False) returns the model and its TorchVision transform, and downloads the model if needed. The name can also be a path to a local checkpoint.
clip.tokenize(text, context_length=77) turns one string or a list of strings into tokens. The model exposes encode_image, encode_text, and a combined forward call that takes an image batch and a text batch.
- clip.available_models(): returns the names of the available models
- clip.load(name, device=..., jit=False): returns model and preprocess transform
- clip.tokenize(text, context_length=77): returns tokenized text
- model.encode_image(image): returns image features
- model.encode_text(text): returns text features
- model(image, text): returns logit scores as cosine similarity times 100
Try a larger label set
The README's zero-shot example uses the CIFAR-100 dataset and tests an image against its 100 class names, using the prompt "a photo of a {class}". It normalizes the features, computes similarity, and prints the top 5 labels. The README notes that the sample output will look similar but the exact numbers may differ by device.
For the linear-probe example, the README uses scikit-learn logistic regression on CLIP image features. It also notes that the C value should be chosen with a hyperparameter sweep using a validation split, not copied blindly.
- Prompt wording such as "a photo of a {class}" is used in the README's CIFAR-100 example
- Linear probe requires scikit-learn and a labeled training set
What people use it for
- Pick the best caption from a short list of labels for a photo
- Test whether a model can recognize categories without training on them first
- Compare image features with text features in a learning or research project
- Run zero-shot classification on sample datasets such as CIFAR-100 as shown in the README
Frequently asked questions
Is CLIP free to use?
The repository is released under the MIT licence, which the directory lists for openai/CLIP. Check the licence file in the repo for the full terms before using it in a product.
What do I need to install CLIP?
The README asks for Python, PyTorch 1.7.1 or later, torchvision, ftfy, regex and tqdm. It then installs the repo from GitHub with pip. A CUDA GPU is optional, since the README also covers CPU-only installs.
How do I install CLIP on a CPU-only machine?
The README says to replace cudatoolkit=11.0 with cpuonly when installing on a machine without a GPU. The rest of the install steps stay the same.
Can CLIP classify images without a app or no-code tool?
The README only shows Python usage. CLIP is a library you call from code, so you need some Python skill to run it.
What is an alternative to openai/CLIP?
The README points to OpenCLIP, which includes larger and independently trained CLIP models up to ViT-G/14, and to the Hugging Face implementation of CLIP for easier integration with that ecosystem. ALIGN is also listed as a comparable project in the directory.
Is CLIP accurate on specialized images?
The README reports only general benchmark results, such as ImageNet zero-shot performance. It does not give accuracy for specialized domains, so test on your own data before relying on it.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
CLIP suits Python developers, researchers and students who want to classify images with plain-language labels and no labeled training set. It is a poor fit for people who need a ready-made app, guaranteed accuracy in a specialized field, or the larger model families, where OpenCLIP is the README's suggested route.
Install options & downloads