Skip to content
Get packs

Latent Diffusion

Research code for generating and editing high-resolution images from text and masks using latent diffusion.

GitHub Download source
Stars
14.2k
Forks
1.7k
Licence
MIT
Language
Jupyter Notebook
Last update
2.7 years ago
Type
AI model / agent

No commits in over a year — check for forks or alternatives before adopting.

What it is

This is the official research code for Latent Diffusion Models, a method for generating high-resolution images. It runs diffusion in a compressed latent space, which the README says makes generation more efficient than working on full-size pixels. The repository includes scripts for text-to-image generation, inpainting, class-conditional ImageNet sampling, and retrieval-augmented models, along with pretrained checkpoints.

  • Generate images from a text prompt using scripts/txt2img.py
  • Fill in masked regions of an image with scripts/inpaint.py
  • Sample class-conditional ImageNet images through the provided Colab notebook
  • Try retrieval-augmented generation with the ArtBench or OpenImages databases

Great for

  • Researchers reproducing or building on the latent diffusion paper
  • Developers who want to run text-to-image sampling locally with the provided checkpoints
  • People experimenting with inpainting using their own image and mask pairs
  • Users comfortable with conda, Python, and command-line tools

Not ideal for

  • Non-technical users wanting a polished app, since the README only describes command-line scripts and no GUI
  • Anyone needing a supported production service, since the README presents this as research code with preliminary checkpoints for some models
  • Users with limited disk space, since the README lists several downloads of 5.7GB, about 11GB, and about 21GB for an index
Step-by-step guide: how to use Latent DiffusionInstall, first run, examples, tips and FAQs

How to get it

  1. 1

    Clone and build from source

    You'll need: conda · Python

    Create the conda environment

    conda env create -f environment.yaml

    Activate the environment

    conda activate ldm
  2. 2

    Run it

    • Clone the repository and create the ldm conda environment
    • Activate the ldm environment
    • Download the text-to-image checkpoint into models/ldm/text2img-large/
    • Run scripts/txt2img.py with a prompt, for example 'a virus monster is playing guitar, oil on canvas'
    • Check outputs/txt2img-samples for the generated images
  3. 3

    Get the most out of it

    • Raise scale for stronger prompt adherence, noting the README says this reduces diversity
    • Increase ddim_steps for quality, though the README says returns diminish above 250
    • Use --plms together with --ddim_eta 0.0 for faster sampling, per the README
    • Try H and W arguments to generate non-square sizes, noting controllability drops compared to 256x256

Downloads

Source code (.zip)main branch · straight from GitHub

Strengths

  • Official implementation from the paper authors, with an arXiv reference and BibTeX citation
  • Clear sampling controls documented in the README, such as scale, ddim_steps, and ddim_eta
  • Released under the MIT license, which the metadata lists

Trade-offs

  • Setup relies on conda and large downloads, and the README does not describe hardware requirements
  • Several README sections are labeled preliminary or 'Coming Soon', and the last push in the metadata is from February 2024
  • Some README text is incomplete or contains typos, such as the truncated ImageNet section, so parts of setup may need trial and error

Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.