Skip to content
Get packs

CLIP

A neural network that matches images with text descriptions, usable without task-specific training.

GitHub Download source
Stars
34.6k
Forks
4.1k
Licence
MIT
Language
Jupyter Notebook
Last update
8 months ago
Type
AI model / agent

What it is

CLIP is a neural network trained on many image and text pairs so it can judge how well a text snippet fits an image. You give it an image and a list of possible labels written in plain language, and it scores which label is most relevant. The README says it can do this without being trained on the specific task, and reports that it matches the original ResNet50 on ImageNet zero-shot without using ImageNet's 1.28M labeled examples.

  • Classify an image against a custom list of text labels using clip.load and model(image, text)
  • Run zero-shot prediction on the CIFAR-100 dataset as shown in the README
  • Extract image features with model.encode_image for downstream tasks such as logistic regression
  • Compare text and image embeddings via cosine similarity scores

Great for

  • Python developers experimenting with image classification using text labels
  • Researchers evaluating zero-shot image recognition
  • Students learning how vision-language models work
  • Teams that want to classify images without collecting a labeled dataset first

Not ideal for

  • People who need a ready-made app with no coding, since the README only shows Python usage
  • Projects needing guaranteed accuracy on specialized domains, because the README reports only general benchmark results
  • Users needing a newer or larger model family, since the README points to OpenCLIP for larger variants
Step-by-step guide: how to use CLIPInstall, first run, examples, tips and FAQs

How to get it

  1. 1

    Install with a package manager

    You'll need: Python · PyTorch 1.7.1 or later · torchvision · ftfy · regex

    Install PyTorch and torchvision (CUDA GPU machine)

    conda install --yes -c pytorch pytorch=1.7.1 torchvision cudatoolkit=11.0

    Install small additional dependencies

    pip install ftfy regex tqdm

    Install CLIP from GitHub

    pip install git+https://github.com/openai/CLIP.git
  2. 2

    Run it

    • Install PyTorch, torchvision and the extra packages as described in the README
    • Run the Python usage example with an image such as CLIP.png
    • Pass a list of candidate text labels to clip.tokenize
    • Read the printed label probabilities from the softmax output
  3. 3

    Get the most out of it

    • Use the zero-shot CIFAR-100 example to test prompts on a labeled dataset
    • Use encode_image and encode_text to extract features for downstream models such as logistic regression
    • Replace cudatoolkit=11.0 with the CUDA version that matches your machine, or use cpuonly without a GPU
    • Try clip.available_models() to see which model variants are provided
    • Check the Colab notebook linked in the README for interactive use

Downloads

Source code (.zip)main branch · straight from GitHub

Strengths

  • Works with plain-language labels rather than a fixed set of trained classes
  • The README includes runnable examples, a Colab notebook, a paper, and a model card
  • Released under the MIT license

Trade-offs

  • Requires Python, PyTorch and a few extra packages, so setup is non-trivial for beginners
  • The README is truncated and offers limited detail on training data and limitations beyond the linked model card
  • Output quality depends on how labels are worded, as the example uses the prompt 'a photo of a {c}'

How it compares

ProjectStarsTypeLevelUpdatedBest for
CLIP (this)34.6kAI model / agentintermediate8 months agoPython developers experimenting with image classification using text labels
machine-learning-for-trading21.7kLearning resourceintermediate8 days agoReaders of the Machine Learning for Trading book who want to run its code
LaTeX-OCR16.6kAI model / agentintermediate1.8 years agoStudents who want to turn a handwritten or printed formula photo into editable LaTeX
albumentations15.3kLibrarybeginner1.4 years agoMachine learning engineers training image classification or segmentation models with limited data
imgaug14.7kLibraryintermediate2.3 years agoMachine learning engineers training image models on a limited dataset
fashion-mnist12.9kLearning resourcebeginner4.4 years agoMachine learning students comparing classifiers on a familiar format

Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.