Guide · Agent skill

How to use the Auto Review Loop (Generic LLM) agent skill

Updated 2026-10-29 · written with AI from the project's own documentation and checked against our catalogue data

The Auto Review Loop (Generic LLM) skill is an agent skill that makes an AI coding agent repeatedly review a research project with an external language model, apply fixes, and review again. It is written for researchers and students who want a second automated critical read of a paper or experiment plan before they submit or continue. After this guide you will know what the skill does, how to install it, how to trigger it with a plain request, and what it needs and risks before you run it.

Auto Review Loop (Generic LLM)wanshuiyin/Auto-claude-code-research-in-sleep · MIT Download the skill

Quick start

1Download and unzip the skill. Download the ZIP from this page and unzip it into ~/.claude/skills/ or your project's .claude/skills/ folder. Alternatively, upload it under Settings → Capabilities → Skills in the Claude apps.
2Get an API key. Create an API key with an OpenAI-compatible provider such as OpenAI, DeepSeek or Moonshot.
3Configure llm-chat. Add the llm-chat MCP server to ~/.claude/settings.json with LLM_API_KEY, LLM_BASE_URL and LLM_MODEL, and make sure Python 3 is available.
4Start the review. Ask the agent with a request such as "auto review loop llm" or "llm review", optionally naming the scope.
5Check the log after each round. Read review-stage/AUTO_REVIEW.md to see the score, verdict, criticisms and the changes the agent made.
6Confirm the outcome. Stop when the state file shows completed, or when four rounds have run without a passing verdict, and review the final summary.

What is an agent skill and what does this one do?

An agent skill is a folder containing a SKILL.md file that gives an AI agent a set of instructions for a specific task. This skill is named auto-review-loop-llm and is the file SKILL.md in the skills/auto-review-loop-llm folder of the wanshuiyin/Auto-claude-code-research-in-sleep repository.

Once the skill is active, the agent runs a review-fix-re-review cycle on your research project. It sends the project's claims, methods and results to an external LLM through any OpenAI-compatible API, asks for a score from 1 to 10 and a verdict, and plans the fixes. It then implements fixes and repeats, for up to 4 rounds or until the reviewer says the work is ready.

Each round is appended to review-stage/AUTO_REVIEW.md, including the full raw reviewer response.
Progress is saved to review-stage/REVIEW_STATE.json after each round, so the loop can resume after an interruption.
The loop stops early only when the score is at least 6 and the verdict is "ready" or "almost".

Install the auto-review-loop-llm skill

Download the ZIP for this skill from this page and unzip it. You can install it in one of two places: your personal skills folder, which applies to all your projects, or a single project's .claude/skills/ folder.

Alternatively, in the Claude apps you can upload the skill under Settings → Capabilities → Skills.

Personal install: unzip into ~/.claude/skills/ so the auto-review-loop-llm folder sits there.
Project install: unzip into .claude/skills/ inside your project folder.
Claude apps: go to Settings → Capabilities → Skills and upload the skill.

Configure the LLM provider

The skill calls any OpenAI-compatible API. The recommended setup is the llm-chat MCP server, configured in ~/.claude/settings.json. You provide an API key, a base URL and a model name. The source lists these providers with their base URLs and example models: OpenAI, DeepSeek, MiniMax, Kimi (Moonshot), ZhiPu (GLM), SiliconFlow, 阿里云百炼 (Dashscope) and 零一万物 (Lingyi Wanwu).

The llm-chat server defaults to gpt-6.1-sol. If LLM_FALLBACK_MODEL is not set, the 504-timeout fallback uses LLM_MODEL. Calls to api.openai.com use max_completion_tokens; other OpenAI-compatible endpoints use max_tokens. You can override this with LLM_MAX_TOKENS_FIELD.

LLM_API_KEY: your provider's API key.
LLM_BASE_URL: the provider's base URL, for example https://api.deepseek.com/v1.
LLM_MODEL: the model name, for example deepseek-chat.
The llm-chat server is a Python 3 script, so the command in the example points to /usr/bin/python3. Adjust the path to match your system and where the server file is stored.

{
  "mcpServers": {
    "llm-chat": {
      "command": "/usr/bin/python3",
      "args": ["/Users/yourname/.claude/mcp-servers/llm-chat/server.py"],
      "env": {
        "LLM_API_KEY": "your-api-key",
        "LLM_BASE_URL": "https://api.deepseek.com/v1",
        "LLM_MODEL": "deepseek-chat"
      }
    }
  }
}

Trigger the skill with a natural request

The skill is triggered by the phrases "auto review loop llm" or "llm review". You can add a topic or scope after the request, since the skill accepts an optional argument. Phrase it as you would to a colleague: name the project or the part you want reviewed.

The agent then reads the project context and any prior reviews, checks for a saved state file to resume from, and starts round 1.

"Run an llm review on my draft in this folder."
"auto review loop llm on the experiment results in review-stage."
"Do an llm review of my method section and tell me the top weaknesses."
"Resume the llm review from where it stopped."

Understand what happens in each round

Each round has five phases. The agent first sends the review prompt to the external model, asking it to score the work, rank the remaining weaknesses by severity, give the minimum fix for each, and state whether the work is ready. The full raw response is saved before anything else.

The agent then parses the score and verdict, implements fixes in order of priority (metric additions first, then reframing, then new experiments), waits for any remote experiments, and documents the round. Fixes are implemented before the next re-review.

Phase A: Review, sent to the LLM.
Phase B: Parse the assessment and check the stop condition.
Phase C: Implement fixes.
Phase D: Wait for results from remote experiments.
Phase E: Document the round and write the state file.

Read the review log and state file

The cumulative log is review-stage/AUTO_REVIEW.md. Each round section holds a summary with the score, verdict and key criticisms, the complete reviewer response inside a collapsible block, the actions taken and the results. If the review-stage folder does not exist in a legacy project, the skill falls back to ./AUTO_REVIEW.md.

The state file records the round number, status (in_progress or completed), last score, last verdict, pending experiments and a timestamp. Set status to completed when the loop finishes.

What people use it for

Getting a ranked list of weaknesses in a paper draft before you send it to a conference or journal.
Testing whether an experiment plan's claims are supported by the planned metrics.
Comparing how different LLM providers rate the same project by changing LLM_BASE_URL and LLM_MODEL.
Recovering a review that was interrupted, using the saved state file.

Frequently asked questions

Is the auto review loop llm skill free?

The skill is published under the MIT licence. You still pay the API provider for each model call, and the source does not list prices for any provider.

What do I need to run it?

You need an API key for an OpenAI-compatible provider and the llm-chat MCP server configured in ~/.claude/settings.json, which is recommended. The fallback path uses curl instead, and the llm-chat server needs Python 3.

Can I use it without the llm-chat MCP server?

Yes. The skill has a curl fallback that calls the chat completions endpoint at LLM_BASE_URL with your API key and model. Use max_tokens for third-party endpoints and max_completion_tokens for direct OpenAI calls.

Which LLM providers work with this skill?

Any OpenAI-compatible API should work. The source lists OpenAI, DeepSeek, MiniMax, Kimi (Moonshot), ZhiPu (GLM), SiliconFlow, 阿里云百炼 and 零一万物, each with its base URL and example models.

Is the review as good as a human peer review?

No. The scores come from another language model and can be overconfident or wrong. Use the output as a structured second read, not a substitute for expert peer review.

Why did the loop stop before four rounds?

It stops early only when the score is at least 6 and the verdict is exactly "ready" or "almost". A verdict of "not ready" never stops the loop early, so the loop runs until the work passes or the 4-round limit is reached.

Should I run it on unpublished work?

Be careful. Project content is sent to the third-party API you configure and is governed by that provider's terms. Avoid sensitive or unpublished work unless you accept that.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Use this skill if you are preparing a paper or drafts and want a repeatable, automated critical read with ranked fixes and a log of every round. Pick something else if you need a human-quality peer review, if your work is sensitive and you cannot send it to a third-party API, or if you do not want an agent running shell commands and editing files in your project.

Download the skill