How to use the Embedding anisotropy check agent skill

What the skill makes the agent do

Embeddings are lists of numbers that represent text, images or hidden states inside a model. A cosine similarity score says how alike two of them are. Raw cosine is rarely centred on zero, because most sentence encoders and hidden-state layers put many vectors near one shared direction. Two unrelated inputs can then score 0.6, 0.9 or 0.99.

The skill makes the agent measure a floor for each comparison. It takes unrelated pairs from the same comparison, measures the anisotropy of each side, centres the vectors by their mean, and reports the matched score beside that floor, both before and after centring. The agent does not create the vectors itself; it checks the number it is about to report.

Install the skill for Claude

Download the ZIP for this skill from this page. Then choose one of two routes. For a personal install, unzip it into ~/.claude/skills/. For a single project, unzip it into the project's .claude/skills/ folder.

You can also upload the skill in the Claude apps. Open Settings, go to Capabilities, then Skills, and upload the ZIP there.

• Personal install: ~/.claude/skills/
• Project install: .claude/skills/ inside your project
• Claude apps: Settings → Capabilities → Skills

Ask for it with natural requests

You do not need to name the skill. The skill applies when the agent is about to report a cosine number, whoever produced the vectors. Three situations trigger it: the agent wrote an evaluation script that prints a mean cosine, saved embedding or hidden-state arrays already exist in your workspace, or the agent is drafting a README, PR description or results table that calls a similarity high or two conditions equal.

Requests that fit well include asking whether two layers encode the same thing, checking that a quantised export still matches the original, or asking for a retrieval evaluation that reports a baseline.

• Check that the 4-bit export of this encoder still matches bf16 before we say it is fine.
• Are these two layers encoding the same thing? Load the saved arrays and check the similarity against a floor.
• Write a retrieval eval for our index and report the random-pair baseline next to the matched score.
• Before we write in the README that the two conditions agree, check the similarity against its floor.

Run the check yourself on two arrays

Save the compared rows as two .npy arrays of shape (n, d), where row i of A pairs with row i of B. If you can, also save a train split from the same domain, model and modality, and use it to fit the means. A mean fitted on a different domain can leave part of the shared direction in place or add a direction of its own.

Run the bundled script with the command below. Pass --shared when A and B come from one model and one population, so both sides are centred by one mean. Without the --fit flags, the means come from the compared rows themselves, and the output's means field says which rows each mean came from.

How to read the output

The anisotropy value is the mean cosine between different rows of one side. Values above about 0.3 mean rows cluster strongly around one direction, and the script prints a warning. The raw.floor field is what an unrelated pair scores in this comparison, and a matched cosine only means something through its distance from that floor.

The centred.gap_over_floor_sd and centred.cohens_d fields show whether matched pairs separate from unrelated ones at all. If unrelated pairs all score the same, gap_over_floor_sd is null and you should read the plain gap instead. partner_ranked_first is the retrieval view of the same pairs; centring can move a ranking either way, so report it both ways.

• Report side by side: anisotropy of each side, matched versus floor before centring, matched versus floor after centring, and partner_ranked_first against its chance rate 1/n.
• Average the signed difference across seeds or items, then take the absolute value. Averaging absolute differences biases the result upward.
• Give a gap with its spread and the smallest effect the test could have detected. An underpowered comparison is not a refutation.

Safety, requirements and what needs the network

The skill runs Python scripts and needs Python 3.9 or later with numpy. The core check needs nothing else. The bundled example, scripts/example_scifact_halves.py, also needs datasets and sentence-transformers, plus torch and network access to download the pinned model and dataset.

Review the scripts before running them on sensitive data, since the skill runs scripts and the example uses the network. The skill's own rules say the check does not apply to two independently trained models without an explicit mapping, because the cosine is then meaningless.

• Runs scripts: yes (scripts/anisotropy_check.py, scripts/example_scifact_halves.py)
• Uses network: yes, for the bundled example only, to download the pinned model and dataset
• Licence: MIT

A worked example and tips

The skill's example uses bge-small-en-v1.5 on the first 100 documents of mteb/scifact's corpus, splitting each abstract into halves at the middle sentence boundary. The halves of one abstract are the matched pair, and one mean is shared by both sides. Rounded to four decimal places, the raw matched cosine is 0.8636 against a floor of 0.6024, while the centred matched cosine is 0.6487 against a floor of -0.0131.

Here an unrelated half already scores about 70% of what the matching half scores before centring. Both spaces still separate matched from unrelated halves, and the partner ranks first for 97 and 98 of 100 abstracts. Centring removes the baseline offset.

• Put the floor from the same comparison beside every result. A floor from another run, model or domain does not correct this one.
• Do not use this for two independently trained models unless you first fit an explicit map, such as orthogonal Procrustes on paired training rows.
• Do not use it for quick lookups where no similarity number is reported.