skill-creator by anthropics/skills
npx skills add https://github.com/anthropics/skills --skill skill-creator一个用于创建新技能并迭代改进它们的技能。
从高层次来看，创建技能的过程大致如下：
eval-viewer/generate_review.py 脚本向用户展示结果供其查看，同时也让他们查看定量指标使用此技能时，你的工作是弄清楚用户处于这个过程的哪个阶段，然后介入并帮助他们推进这些阶段。例如，用户可能说"我想为 X 创建一个技能"。你可以帮助明确他们的意图，撰写草稿，编写测试用例，确定他们希望如何评估，运行所有提示，并重复这个过程。
另一方面，用户可能已经有了技能的草稿。在这种情况下，你可以直接进入评估/迭代循环。
当然，你应该始终保持灵活性，如果用户说"我不需要运行一堆评估，只需和我一起探讨"，你也可以那样做。
技能完成后（再次强调，顺序是灵活的），你还可以运行技能描述优化器，我们有一个单独的脚本来优化技能的触发。
明白了吗？好的。
技能创建者可能会被不同编程术语熟悉程度的人使用。如果你没听说过（你怎么会知道呢，这只是最近才开始的事情），现在有一种趋势，Claude 的强大能力正在激励水管工打开他们的终端，父母和祖父母去谷歌搜索"如何安装 npm"。另一方面，大多数用户可能对计算机相当熟悉。
因此，请注意上下文线索，以理解如何措辞你的沟通！在默认情况下，给你一些参考：
如果你不确定，可以简要解释术语；如果你不确定用户是否能理解，可以随时用一个简短的定义来澄清术语。
首先理解用户的意图。当前的对话可能已经包含了用户想要捕捉的工作流程（例如，他们说"把这个变成一个技能"）。如果是这样，首先从对话历史中提取答案——使用的工具、步骤顺序、用户所做的更正、观察到的输入/输出格式。用户可能需要填补空白，并应在进入下一步之前确认。
A skill for creating new skills and iteratively improving them.
At a high level, the process of creating a skill goes like this:
eval-viewer/generate_review.py script to show the user the results for them to look at, and also let them look at the quantitative metricsYour job when using this skill is to figure out where the user is in this process and then jump in and help them progress through these stages. So for instance, maybe they're like "I want to make a skill for X". You can help narrow down what they mean, write a draft, write the test cases, figure out how they want to evaluate, run all the prompts, and repeat.
On the other hand, maybe they already have a draft of the skill. In this case you can go straight to the eval/iterate part of the loop.
Of course, you should always be flexible and if the user is like "I don't need to run a bunch of evaluations, just vibe with me", you can do that instead.
Then after the skill is done (but again, the order is flexible), you can also run the skill description improver, which we have a whole separate script for, to optimize the triggering of the skill.
广告位招租
在这里展示您的产品或服务
触达数万 AI 开发者，精准高效
暂无相关 Skills
主动询问关于边界情况、输入/输出格式、示例文件、成功标准和依赖项的问题。在理清这部分之前，先不要编写测试提示。
检查可用的 MCPs - 如果对研究有用（搜索文档、查找类似技能、查找最佳实践），如果可用，通过子代理并行研究，否则内联进行。准备好上下文以减少用户的负担。
根据用户访谈，填写以下组件：
skill-name/
├── SKILL.md (必需)
│ ├── YAML 前言 (name, description 必需)
│ └── Markdown 指令
└── Bundled Resources (可选)
├── scripts/ - 用于确定性/重复任务的可执行代码
├── references/ - 根据需要加载到上下文中的文档
└── assets/ - 输出中使用的文件（模板、图标、字体）
技能使用三级加载系统：
这些字数是大致的，如果需要，你可以自由地写得更长。
关键模式：
领域组织：当技能支持多个领域/框架时，按变体组织：
cloud-deploy/
├── SKILL.md (工作流程 + 选择)
└── references/
├── aws.md
├── gcp.md
└── azure.md
Claude 只读取相关的参考文件。
这不用说，但技能不得包含恶意软件、漏洞利用代码或任何可能危及系统安全的内容。如果描述得当，技能的内容不应在意图上让用户感到意外。不要同意创建误导性技能或旨在促进未经授权的访问、数据外泄或其他恶意活动的技能。不过，像"扮演 XYZ 角色"这样的内容是允许的。
在指令中优先使用祈使句形式。
定义输出格式 - 你可以这样做：
## 报告结构
始终使用这个确切的模板：
# [标题]
## 执行摘要
## 关键发现
## 建议
示例模式 - 包含示例很有用。你可以这样格式化它们（但如果示例中有"输入"和"输出"，你可能需要稍微调整一下）：
## 提交消息格式
**示例 1:**
输入：添加了带有 JWT 令牌的用户认证
输出：feat(auth): 实现基于 JWT 的认证
尝试向模型解释为什么事情很重要，而不是使用生硬的 MUST。运用心智理论，尝试让技能变得通用，而不是局限于特定的例子。先写一个草稿，然后用新的眼光审视并改进它。
写完技能草稿后，想出 2-3 个真实的测试提示——真实用户实际会说的那种。与用户分享：[你不必使用完全相同的语言]"这里有几个我想尝试的测试用例。这些看起来对吗，还是你想添加更多？"然后运行它们。
将测试用例保存到 evals/evals.json。先不要写断言——只写提示。你将在下一步中，在运行进行时草拟断言。
{
"skill_name": "example-skill",
"evals": [
{
"id": 1,
"prompt": "用户的任务提示",
"expected_output": "预期结果的描述",
"files": []
}
]
}
完整的模式（包括你稍后将添加的 assertions 字段）请参见 references/schemas.md。
本节是一个连续的序列——不要中途停止。不要使用 /skill-test 或任何其他测试技能。
将结果放在 <skill-name>-workspace/ 中，作为技能目录的同级目录。在工作区内，按迭代组织结果（iteration-1/、iteration-2/ 等），在每个迭代内，每个测试用例都有一个目录（eval-0/、eval-1/ 等）。不要预先创建所有这些——只需在过程中创建目录。
对于每个测试用例，在同一轮中生成两个子代理——一个带技能，一个不带。这很重要：不要先生成带技能的运行，然后再回来做基线。同时启动所有任务，以便它们大约在同一时间完成。
带技能运行：
Execute this task:
- Skill path: <path-to-skill>
- Task: <eval prompt>
- Input files: <eval files if any, or "none">
- Save outputs to: <workspace>/iteration-<N>/eval-<ID>/with_skill/outputs/
- Outputs to save: <what the user cares about — e.g., "the .docx file", "the final CSV">
基线运行（相同的提示，但基线取决于上下文）：
without_skill/outputs/。cp -r <skill-path> <workspace>/skill-snapshot/），然后将基线子代理指向快照。保存到 old_skill/outputs/。为每个测试用例编写一个 eval_metadata.json（断言现在可以为空）。根据测试内容给每个评估一个描述性名称——不仅仅是"eval-0"。目录也使用这个名称。如果此迭代使用新的或修改过的评估提示，为每个新的评估目录创建这些文件——不要假设它们从之前的迭代中延续下来。
{
"eval_id": 0,
"eval_name": "descriptive-name-here",
"prompt": "用户的任务提示",
"assertions": []
}
不要只是等待运行完成——你可以有效地利用这段时间。为每个测试用例草拟定量断言，并向用户解释。如果 evals/evals.json 中已经存在断言，请查看并解释它们检查什么。
好的断言是客观可验证的，并且具有描述性名称——它们应该在基准测试查看器中清晰可读，以便有人浏览结果时能立即理解每个断言检查什么。主观技能（写作风格、设计质量）最好进行定性评估——不要将断言强加于需要人类判断的事物。
草拟断言后，更新 eval_metadata.json 文件和 evals/evals.json。同时向用户解释他们将在查看器中看到什么——定性的输出和定量的基准测试。
当每个子代理任务完成时，你会收到一个包含 total_tokens 和 duration_ms 的通知。立即将此数据保存到运行目录中的 timing.json：
{
"total_tokens": 84852,
"duration_ms": 23332,
"total_duration_seconds": 23.3
}
这是捕获此数据的唯一机会——它通过任务通知传来，不会在其他地方持久化。每个通知到达时立即处理，而不是尝试批量处理。
所有运行完成后：
对每次运行评分 - 生成一个评分子代理（或内联评分），该代理读取 agents/grader.md 并根据输出评估每个断言。将结果保存到每个运行目录中的 grading.json。grading.json 中的 expectations 数组必须使用字段 text、passed 和 evidence（而不是 name/met/details 或其他变体）——查看器依赖于这些确切的字段名。对于可以通过编程方式检查的断言，编写并运行一个脚本，而不是目测检查——脚本更快、更可靠，并且可以在迭代中重复使用。
汇总到基准测试 - 从 skill-creator 目录运行聚合脚本：
python -m scripts.aggregate_benchmark <workspace>/iteration-N --skill-name <name>
这将生成 benchmark.json 和 benchmark.md，其中包含每个配置的 pass_rate、time 和 tokens，以及均值 ± 标准差和差值。如果手动生成 benchmark.json，请参见 references/schemas.md 了解查看器期望的确切模式。将每个 with_skill 版本放在其基线对应项之前。
进行分析 - 阅读基准测试数据，并揭示汇总统计可能隐藏的模式。参见 agents/analyzer.md（"分析基准测试结果"部分）了解需要关注的内容——例如，无论技能如何总是通过的断言（无区分度）、高方差评估（可能不稳定）以及时间/令牌权衡。
启动查看器，包含定性输出和定量数据：
nohup python <skill-creator-path>/eval-viewer/generate_review.py \
<workspace>/iteration-N \
--skill-name "my-skill" \
--benchmark <workspace>/iteration-N/benchmark.json \
> /dev/null 2>&1 &
VIEWER_PID=$!
对于第 2 次及以后的迭代，还要传递 --previous-workspace <workspace>/iteration-<N-1>。
Cowork / 无头环境： 如果 webbrowser.open() 不可用或环境没有显示器，请使用 --static <output_path> 写入一个独立的 HTML 文件，而不是启动服务器。当用户点击"提交所有评论"时，反馈将作为 feedback.json 文件下载。下载后，将 feedback.json 复制到工作区目录中，以便下一次迭代拾取。
注意：请使用 generate_review.py 创建查看器；无需编写自定义 HTML。
"输出"标签页一次显示一个测试用例：
"基准测试"标签页显示统计摘要：每个配置的通过率、时间和令牌使用情况，以及每个评估的细分和分析观察结果。
通过上一个/下一个按钮或箭头键导航。完成后，他们点击"提交所有评论"，将所有反馈保存到 feedback.json。
当用户告诉你他们完成后，阅读 feedback.json：
{
"reviews": [
{"run_id": "eval-0-with_skill", "feedback": "图表缺少轴标签", "timestamp": "..."},
{"run_id": "eval-1-with_skill", "feedback": "", "timestamp": "..."},
{"run_id": "eval-2-with_skill", "feedback": "完美，太棒了", "timestamp": "..."}
],
"status": "complete"
}
空反馈意味着用户认为没问题。将改进重点放在用户有具体抱怨的测试用例上。
完成后关闭查看器服务器：
kill $VIEWER_PID 2>/dev/null
这是循环的核心。你已经运行了测试用例，用户已经审查了结果，现在你需要根据他们的反馈使技能变得更好。
从反馈中归纳。 这里发生的大背景是，我们试图创建可以被使用一百万次（也许是真的，也许更多谁知道呢）的技能，跨越许多不同的提示。在这里，你和用户只是在少数几个例子上反复迭代，因为这有助于更快地推进。用户对这些例子了如指掌，他们可以快速评估新的输出。但是，如果你和用户共同开发的技能只适用于那些例子，那就没用了。与其加入琐碎的过拟合更改，或压迫性的 MUST，如果存在一些顽固的问题，你可以尝试拓展思路，使用不同的隐喻，或推荐不同的工作模式。尝试的成本相对较低，也许你会找到很棒的方法。
保持提示简洁。 删除那些没有发挥作用的冗余部分。一定要阅读对话记录，而不仅仅是最终输出——如果看起来技能让模型浪费了大量时间做无益的事情，你可以尝试去掉导致这种情况的技能部分，看看会发生什么。
解释原因。 努力解释你要求模型做的一切事情的原因。今天的 LLM 很聪明。它们具有良好的心智理论，当给予良好的框架时，可以超越死板的指令，真正让事情发生。即使用户的反馈简短或带有挫败感，也要尝试真正理解任务以及用户为什么写他们写的内容，他们实际写了什么，然后将这种理解转化为指令。如果你发现自己用全大写写 ALWAYS 或 NEVER，或者使用超级僵化的结构，那是一个警告信号——如果可能，重新表述并解释推理，以便模型理解你要求的事情为什么重要。这是一种更人性化、更强大、更有效的方法。
寻找跨测试用例的重复工作。 阅读测试运行的对话记录，注意子代理是否都独立编写了类似的辅助脚本，或者对某些事情采取了相同的多步骤方法。如果所有 3 个测试用例都导致子代理编写了 create_docx.py 或 build_chart.py，这是一个强烈的信号，表明技能应该捆绑该脚本。编写一次，放在 scripts/ 中，并告诉技能使用它。这可以避免未来的每次调用都重新发明轮子。
这个任务非常重要（我们正试图在这里创造每年数十亿美元的经济价值！），你的思考时间不是瓶颈；花点时间，真正深思熟虑。我建议先写一个修订草稿，然后用新的眼光审视并进行改进。尽最大努力理解用户的想法，理解他们想要和需要什么。
改进技能后：
iteration-<N+1>/ 目录中，包括基线运行。如果你正在创建一个新技能，基线始终是 without_skill（无技能）——这在迭代中保持不变。如果你正在改进现有技能，根据什么有意义作为基线来判断：用户最初带来的原始版本，或上一次迭代。--previous-workspace 指向上一次迭代继续直到：
对于想要更严格地比较两个技能版本的情况（例如，用户问"新版本真的更好吗？"），有一个盲比较系统。阅读 agents/comparator.md 和 agents/analyzer.md 了解细节。基本思想是：将两个输出交给一个独立的代理，不告诉它哪个是哪个，让它判断质量。然后分析获胜者为什么获胜。
这是可选的，需要子代理，大多数用户不需要。人类审查循环通常就足够了。
SKILL.md 前言中的 description 字段是决定 Claude 是否调用技能的主要机制。创建或改进技能后，提供优化描述以提高触发准确性。
创建 20 个评估查询——混合应该触发和不应该触发的。保存为 JSON：
[
{"query": "用户提示", "should_trigger": true},
{"query": "另一个提示", "should_trigger": false}
]
查询必须是真实的，是 Claude Code 或 Claude.ai 用户实际会输入的内容。不是抽象的请求，而是具体、详细、有大量细节的请求。例如，文件路径、关于用户工作或情况的个人背景、列名和值、公司名称、URL。带一点背景故事。有些可能是小写，或包含缩写、拼写错误或随意口语。使用不同长度的混合，并专注于边界情况，而不是让它们清晰明了（用户将有机会签署同意）。
不好的例子："格式化这些数据"、"从 PDF 中提取文本"、"创建一个图表"
好的例子："好的，我的老板刚刚给我发了这个 xlsx 文件（在我的下载文件夹里，文件名类似'Q4 sales final FINAL v2.xlsx'），她想让我添加一列显示利润率百分比。收入在 C 列，成本在 D 列，我想"
对于应该触发的查询（8-10 个），考虑覆盖率。你需要相同意图的不同措辞——有些正式，有些随意。包括用户没有明确命名技能或文件类型但显然需要它的情况。加入一些不常见的用例，以及此技能与另一个竞争但应该获胜的情况。
对于不应该触发的查询（8-10 个），最有价值的是接近的误报——与技能共享关键词或概念但实际上需要其他东西的查询。考虑相邻领域、模糊措辞（天真的关键词匹配会触发但不应该），以及查询涉及技能所做的某些事情但在另一个工具更合适的上下文中的情况。
要避免的关键点：不要让不应该触发的查询明显无关。对于 PDF 技能，"写一个斐波那契函数"作为负面测试太简单了——它没有测试任何东西。负面案例应该是真正棘手的。
使用 HTML 模板向用户展示评估集以供审查：
assets/eval_review.html 读取模板__EVAL_DATA_PLACEHOLDER__ → 评估项的 JSON 数组（不要加引号——它是一个 JS 变量赋值）__SKILL_NAME_PLACEHOLDER__ → 技能的名称__SKILL_DESCRIPTION_PLACEHOLDER__ → 技能的当前描述/tmp/eval_review_<skill-name>.html）并打开：open /tmp/eval_review_<skill-name>.html~/Downloads/eval_set.json —— 检查 Downloads 文件夹中的最新版本，以防有多个（例如 eval_set (1).json）这一步很重要——糟糕的评估查询会导致糟糕的描述。
告诉用户："这需要一些时间——我将在后台运行优化循环，并定期检查进度。"
将评估集保存到工作区，然后在后台运行：
python -m scripts.run_loop \
--eval-set <path-to-trigger-eval.json> \
--skill-path <path-to-skill> \
--model <model-id-powering-this-session> \
--max-iterations 5 \
--verbose
使用你的系统提示中的模型 ID（为当前会话提供动力的那个），以便触发测试与用户实际体验的相匹配。
运行时，定期跟踪输出，向用户更新它处于哪个迭代以及分数看起来如何。
这会自动处理完整的优化循环。它将评估集分为 60% 的训练集和 40% 的保留测试集，评估当前描述（每个查询运行 3 次以获得可靠的触发率），然后调用 Claude 根据失败的情况提出改进建议。它在训练集和测试集上重新评估每个新描述，最多迭代 5 次。完成后，它在浏览器中打开一个 HTML 报告，显示每次迭代的结果，并返回包含 best_description 的 JSON——根据测试分数而不是训练分数选择，以避免过拟合。
理解触发机制有助于设计更好的评估查询。技能以其名称 + 描述出现在 Claude 的 available_skills 列表中，Claude 根据该描述决定是否咨询某个技能。需要知道的重要一点是，Claude 只会在无法自行轻松处理的任务上咨询技能——简单的、一步式的查询，如"读取此 PDF"，即使描述完全匹配，也可能不会触发技能，因为 Claude 可以用基本工具直接处理它们。复杂的、多步骤的或专门的查询，当描述匹配时，会可靠地触发技能。
这意味着你的评估查询应该有足够的实质性内容，以便 Claude 实际上会受益于咨询技能。像"读取文件 X"这样的简单查询是糟糕的测试用例——无论描述质量如何，它们都不会触发技能。
从 JSON 输出中获取 best_description 并更新技能的 SKILL.md 前言。向用户展示前后对比并报告分数。
present_files 工具可用时）检查你是否可以访问 present_files 工具。如果没有，请跳过此步骤。如果有，打包技能并向用户呈现 .skill 文件：
python -m scripts.package_skill <path/to/skill-folder>
打包后，引导用户到生成的 .skill 文件路径，以便他们可以安装它。
在 Claude.ai 中，核心工作流程是相同的（草稿 → 测试 → 审查 → 改进 → 重复），但由于 Claude.ai 没有子代理，一些机制会改变。以下是需要调整的内容：
运行测试用例：没有子代理意味着没有并行执行。对于每个测试用例，读取技能的 SKILL.md，然后按照其指令自己完成测试提示。一次一个地进行。这不如独立的子代理严格（你编写了技能，并且也在运行它，所以你拥有完整的上下文），但这是一个有用的完整性检查——人类审查步骤可以弥补这一点。跳过基线运行——只需使用技能按要求完成任务。
审查结果：如果你无法打开浏览器（例如，Claude.ai 的 VM 没有显示器，或者你在远程服务器上），请完全跳过浏览器查看器。相反，直接在对话中呈现结果。对于每个测试用例，显示提示和输出。如果输出是用户需要查看的文件（如 .docx 或 .xlsx），请将其保存到文件系统并告诉他们文件在哪里，以便他们可以下载和检查。内联请求反馈："这个看起来怎么样？有什么想改的吗？"
基准测试：跳过定量基准测试——它依赖于基线比较，而没有子代理，这些比较就没有意义。专注于用户的定性反馈。
迭代循环：和之前一样——改进技能，重新运行测试用例，请求反馈——只是中间没有浏览器查看器。如果你有文件系统，仍然可以将结果组织到迭代目录中。
描述优化：此部分需要 claude CLI 工具（特别是 claude -p），该工具仅在 Claude Code 中可用。如果你在 Claude.ai 上，请跳过它。
盲比较：需要子代理。跳过它。
打包：package_skill.py 脚本在任何有 Python 和文件系统的地方都可以工作。在 Claude.ai 上，你可以运行它，用户可以下载生成的 .skill 文件。
更新现有技能：用户可能要求你更新现有技能，而不是创建新技能。在这种情况下：
name 前言字段——保持不变地使用它们。例如，如果已安装的技能是 research-helper，则输出 research-helper.skill（而不是 research-helper-v2）。/tmp/skill-name/，在那里编辑，并从副本打包。/tmp/中暂存，然后复制到输出目录——直接写入可能因权限问题而失败。如果你在 Cowork 中，主要需要知道的是：
--static <output_path> 写入一个独立的 HTML 文件，而不是启动服务器。然后提供一个链接，用户可以点击在浏览器中打开 HTML。generate_review.py（而不是编写你自己的定制 html 代码）。提前抱歉，但我要在这里用全大写强调：在你自己评估输入之前生成评估查看器。你希望尽快将它们呈现在人类面前！feedback.json 文件。然后你可以从那里读取它（你可能需要先请求访问权限）。package_skill.py 只需要 Python 和文件系统。run_loop.py / run_eval.py）在 Cowork 中应该可以正常工作，因为它通过子进程使用 claude -p，而不是浏览器，但请等到你完全完成技能制作并且用户同意它状态良好后再进行。agents/ 目录包含专门子代理的指令。当你需要生成相关子代理时阅读它们。
agents/grader.md — 如何根据输出评估断言agents/comparator.md — 如何对两个输出进行盲 A/B 比较agents/analyzer.md — 如何分析一个版本为什么击败了另一个references/ 目录有额外的文档：
references/schemas.md — evals.json、grading.json 等的 JSON 结构再次强调核心循环以突出重点：
eval-viewer/generate_review.py 帮助用户审查它们请将步骤添加到你的 TodoList（如果你有的话），以确保你不会忘记。如果你在 Cowork 中，请特别将"创建 evals JSON 并运行 eval-viewer/generate_review.py 以便人类可以审查测试用例"放入你的 TodoList，以确保它发生。
祝你好运！
每周安装量
68.9K
仓库
GitHub Stars
87.3K
首次出现
Jan 19, 2026
安全审计
安装于
opencode54.1K
gemini-cli50.0K
codex49.5K
github-copilot45.7K
claude-code41.8K
cursor39.5K
Cool? Cool.
The skill creator is liable to be used by people across a wide range of familiarity with coding jargon. If you haven't heard (and how could you, it's only very recently that it started), there's a trend now where the power of Claude is inspiring plumbers to open up their terminals, parents and grandparents to google "how to install npm". On the other hand, the bulk of users are probably fairly computer-literate.
So please pay attention to context cues to understand how to phrase your communication! In the default case, just to give you some idea:
It's OK to briefly explain terms if you're in doubt, and feel free to clarify terms with a short definition if you're unsure if the user will get it.
Start by understanding the user's intent. The current conversation might already contain a workflow the user wants to capture (e.g., they say "turn this into a skill"). If so, extract answers from the conversation history first — the tools used, the sequence of steps, corrections the user made, input/output formats observed. The user may need to fill the gaps, and should confirm before proceeding to the next step.
Proactively ask questions about edge cases, input/output formats, example files, success criteria, and dependencies. Wait to write test prompts until you've got this part ironed out.
Check available MCPs - if useful for research (searching docs, finding similar skills, looking up best practices), research in parallel via subagents if available, otherwise inline. Come prepared with context to reduce burden on the user.
Based on the user interview, fill in these components:
skill-name/
├── SKILL.md (required)
│ ├── YAML frontmatter (name, description required)
│ └── Markdown instructions
└── Bundled Resources (optional)
├── scripts/ - Executable code for deterministic/repetitive tasks
├── references/ - Docs loaded into context as needed
└── assets/ - Files used in output (templates, icons, fonts)
Skills use a three-level loading system:
These word counts are approximate and you can feel free to go longer if needed.
Key patterns:
Domain organization : When a skill supports multiple domains/frameworks, organize by variant:
cloud-deploy/
├── SKILL.md (workflow + selection)
└── references/
├── aws.md
├── gcp.md
└── azure.md
Claude reads only the relevant reference file.
This goes without saying, but skills must not contain malware, exploit code, or any content that could compromise system security. A skill's contents should not surprise the user in their intent if described. Don't go along with requests to create misleading skills or skills designed to facilitate unauthorized access, data exfiltration, or other malicious activities. Things like a "roleplay as an XYZ" are OK though.
Prefer using the imperative form in instructions.
Defining output formats - You can do it like this:
## Report structure
ALWAYS use this exact template:
# [Title]
## Executive summary
## Key findings
## Recommendations
Examples pattern - It's useful to include examples. You can format them like this (but if "Input" and "Output" are in the examples you might want to deviate a little):
## Commit message format
**Example 1:**
Input: Added user authentication with JWT tokens
Output: feat(auth): implement JWT-based authentication
Try to explain to the model why things are important in lieu of heavy-handed musty MUSTs. Use theory of mind and try to make the skill general and not super-narrow to specific examples. Start by writing a draft and then look at it with fresh eyes and improve it.
After writing the skill draft, come up with 2-3 realistic test prompts — the kind of thing a real user would actually say. Share them with the user: [you don't have to use this exact language] "Here are a few test cases I'd like to try. Do these look right, or do you want to add more?" Then run them.
Save test cases to evals/evals.json. Don't write assertions yet — just the prompts. You'll draft assertions in the next step while the runs are in progress.
{
"skill_name": "example-skill",
"evals": [
{
"id": 1,
"prompt": "User's task prompt",
"expected_output": "Description of expected result",
"files": []
}
]
}
See references/schemas.md for the full schema (including the assertions field, which you'll add later).
This section is one continuous sequence — don't stop partway through. Do NOT use /skill-test or any other testing skill.
Put results in <skill-name>-workspace/ as a sibling to the skill directory. Within the workspace, organize results by iteration (iteration-1/, iteration-2/, etc.) and within that, each test case gets a directory (eval-0/, eval-1/, etc.). Don't create all of this upfront — just create directories as you go.
For each test case, spawn two subagents in the same turn — one with the skill, one without. This is important: don't spawn the with-skill runs first and then come back for baselines later. Launch everything at once so it all finishes around the same time.
With-skill run:
Execute this task:
- Skill path: <path-to-skill>
- Task: <eval prompt>
- Input files: <eval files if any, or "none">
- Save outputs to: <workspace>/iteration-<N>/eval-<ID>/with_skill/outputs/
- Outputs to save: <what the user cares about — e.g., "the .docx file", "the final CSV">
Baseline run (same prompt, but the baseline depends on context):
without_skill/outputs/.cp -r <skill-path> <workspace>/skill-snapshot/), then point the baseline subagent at the snapshot. Save to old_skill/outputs/.Write an eval_metadata.json for each test case (assertions can be empty for now). Give each eval a descriptive name based on what it's testing — not just "eval-0". Use this name for the directory too. If this iteration uses new or modified eval prompts, create these files for each new eval directory — don't assume they carry over from previous iterations.
{
"eval_id": 0,
"eval_name": "descriptive-name-here",
"prompt": "The user's task prompt",
"assertions": []
}
Don't just wait for the runs to finish — you can use this time productively. Draft quantitative assertions for each test case and explain them to the user. If assertions already exist in evals/evals.json, review them and explain what they check.
Good assertions are objectively verifiable and have descriptive names — they should read clearly in the benchmark viewer so someone glancing at the results immediately understands what each one checks. Subjective skills (writing style, design quality) are better evaluated qualitatively — don't force assertions onto things that need human judgment.
Update the eval_metadata.json files and evals/evals.json with the assertions once drafted. Also explain to the user what they'll see in the viewer — both the qualitative outputs and the quantitative benchmark.
When each subagent task completes, you receive a notification containing total_tokens and duration_ms. Save this data immediately to timing.json in the run directory:
{
"total_tokens": 84852,
"duration_ms": 23332,
"total_duration_seconds": 23.3
}
This is the only opportunity to capture this data — it comes through the task notification and isn't persisted elsewhere. Process each notification as it arrives rather than trying to batch them.
Once all runs are done:
Grade each run — spawn a grader subagent (or grade inline) that reads agents/grader.md and evaluates each assertion against the outputs. Save results to grading.json in each run directory. The grading.json expectations array must use the fields text, passed, and evidence (not name/met/details or other variants) — the viewer depends on these exact field names. For assertions that can be checked programmatically, write and run a script rather than eyeballing it — scripts are faster, more reliable, and can be reused across iterations.
Aggregate into benchmark — run the aggregation script from the skill-creator directory:
python -m scripts.aggregate_benchmark <workspace>/iteration-N --skill-name <name>
This produces benchmark.json and benchmark.md with pass_rate, time, and tokens for each configuration, with mean ± stddev and the delta. If generating benchmark.json manually, see references/schemas.md for the exact schema the viewer expects. Put each with_skill version before its baseline counterpart.
Do an analyst pass — read the benchmark data and surface patterns the aggregate stats might hide. See agents/analyzer.md (the "Analyzing Benchmark Results" section) for what to look for — things like assertions that always pass regardless of skill (non-discriminating), high-variance evals (possibly flaky), and time/token tradeoffs.
Launch the viewer with both qualitative outputs and quantitative data:
nohup python <skill-creator-path>/eval-viewer/generate_review.py \
<workspace>/iteration-N \
--skill-name "my-skill" \
--benchmark <workspace>/iteration-N/benchmark.json \
> /dev/null 2>&1 &
VIEWER_PID=$!
For iteration 2+, also pass --previous-workspace <workspace>/iteration-<N-1>.
Cowork / headless environments: If webbrowser.open() is not available or the environment has no display, use --static <output_path> to write a standalone HTML file instead of starting a server. Feedback will be downloaded as a feedback.json file when the user clicks "Submit All Reviews". After download, copy feedback.json into the workspace directory for the next iteration to pick up.
Note: please use generate_review.py to create the viewer; there's no need to write custom HTML.
The "Outputs" tab shows one test case at a time:
The "Benchmark" tab shows the stats summary: pass rates, timing, and token usage for each configuration, with per-eval breakdowns and analyst observations.
Navigation is via prev/next buttons or arrow keys. When done, they click "Submit All Reviews" which saves all feedback to feedback.json.
When the user tells you they're done, read feedback.json:
{
"reviews": [
{"run_id": "eval-0-with_skill", "feedback": "the chart is missing axis labels", "timestamp": "..."},
{"run_id": "eval-1-with_skill", "feedback": "", "timestamp": "..."},
{"run_id": "eval-2-with_skill", "feedback": "perfect, love this", "timestamp": "..."}
],
"status": "complete"
}
Empty feedback means the user thought it was fine. Focus your improvements on the test cases where the user had specific complaints.
Kill the viewer server when you're done with it:
kill $VIEWER_PID 2>/dev/null
This is the heart of the loop. You've run the test cases, the user has reviewed the results, and now you need to make the skill better based on their feedback.
Generalize from the feedback. The big picture thing that's happening here is that we're trying to create skills that can be used a million times (maybe literally, maybe even more who knows) across many different prompts. Here you and the user are iterating on only a few examples over and over again because it helps move faster. The user knows these examples in and out and it's quick for them to assess new outputs. But if the skill you and the user are codeveloping works only for those examples, it's useless. Rather than put in fiddly overfitty changes, or oppressively constrictive MUSTs, if there's some stubborn issue, you might try branching out and using different metaphors, or recommending different patterns of working. It's relatively cheap to try and maybe you'll land on something great.
Keep the prompt lean. Remove things that aren't pulling their weight. Make sure to read the transcripts, not just the final outputs — if it looks like the skill is making the model waste a bunch of time doing things that are unproductive, you can try getting rid of the parts of the skill that are making it do that and seeing what happens.
Explain the why. Try hard to explain the why behind everything you're asking the model to do. Today's LLMs are smart. They have good theory of mind and when given a good harness can go beyond rote instructions and really make things happen. Even if the feedback from the user is terse or frustrated, try to actually understand the task and why the user is writing what they wrote, and what they actually wrote, and then transmit this understanding into the instructions. If you find yourself writing ALWAYS or NEVER in all caps, or using super rigid structures, that's a yellow flag — if possible, reframe and explain the reasoning so that the model understands why the thing you're asking for is important. That's a more humane, powerful, and effective approach.
Look for repeated work across test cases. Read the transcripts from the test runs and notice if the subagents all independently wrote similar helper scripts or took the same multi-step approach to something. If all 3 test cases resulted in the subagent writing a create_docx.py or a build_chart.py, that's a strong signal the skill should bundle that script. Write it once, put it in scripts/, and tell the skill to use it. This saves every future invocation from reinventing the wheel.
This task is pretty important (we are trying to create billions a year in economic value here!) and your thinking time is not the blocker; take your time and really mull things over. I'd suggest writing a draft revision and then looking at it anew and making improvements. Really do your best to get into the head of the user and understand what they want and need.
After improving the skill:
iteration-<N+1>/ directory, including baseline runs. If you're creating a new skill, the baseline is always without_skill (no skill) — that stays the same across iterations. If you're improving an existing skill, use your judgment on what makes sense as the baseline: the original version the user came in with, or the previous iteration.--previous-workspace pointing at the previous iterationKeep going until:
For situations where you want a more rigorous comparison between two versions of a skill (e.g., the user asks "is the new version actually better?"), there's a blind comparison system. Read agents/comparator.md and agents/analyzer.md for the details. The basic idea is: give two outputs to an independent agent without telling it which is which, and let it judge quality. Then analyze why the winner won.
This is optional, requires subagents, and most users won't need it. The human review loop is usually sufficient.
The description field in SKILL.md frontmatter is the primary mechanism that determines whether Claude invokes a skill. After creating or improving a skill, offer to optimize the description for better triggering accuracy.
Create 20 eval queries — a mix of should-trigger and should-not-trigger. Save as JSON:
[
{"query": "the user prompt", "should_trigger": true},
{"query": "another prompt", "should_trigger": false}
]
The queries must be realistic and something a Claude Code or Claude.ai user would actually type. Not abstract requests, but requests that are concrete and specific and have a good amount of detail. For instance, file paths, personal context about the user's job or situation, column names and values, company names, URLs. A little bit of backstory. Some might be in lowercase or contain abbreviations or typos or casual speech. Use a mix of different lengths, and focus on edge cases rather than making them clear-cut (the user will get a chance to sign off on them).
Bad: "Format this data", "Extract text from PDF", "Create a chart"
Good: "ok so my boss just sent me this xlsx file (its in my downloads, called something like 'Q4 sales final FINAL v2.xlsx') and she wants me to add a column that shows the profit margin as a percentage. The revenue is in column C and costs are in column D i think"
For the should-trigger queries (8-10), think about coverage. You want different phrasings of the same intent — some formal, some casual. Include cases where the user doesn't explicitly name the skill or file type but clearly needs it. Throw in some uncommon use cases and cases where this skill competes with another but should win.
For the should-not-trigger queries (8-10), the most valuable ones are the near-misses — queries that share keywords or concepts with the skill but actually need something different. Think adjacent domains, ambiguous phrasing where a naive keyword match would trigger but shouldn't, and cases where the query touches on something the skill does but in a context where another tool is more appropriate.
The key thing to avoid: don't make should-not-trigger queries obviously irrelevant. "Write a fibonacci function" as a negative test for a PDF skill is too easy — it doesn't test anything. The negative cases should be genuinely tricky.
Present the eval set to the user for review using the HTML template:
assets/eval_review.html__EVAL_DATA_PLACEHOLDER__ → the JSON array of eval items (no quotes around it — it's a JS variable assignment)__SKILL_NAME_PLACEHOLDER__ → the skill's name__SKILL_DESCRIPTION_PLACEHOLDER__ → the skill's current description/tmp/eval_review_<skill-name>.html) and open it: open /tmp/eval_review_<skill-name>.html~/Downloads/eval_set.json — check the Downloads folder for the most recent version in case there are multiple (e.g., eval_set (1).json)This step matters — bad eval queries lead to bad descriptions.
Tell the user: "This will take some time — I'll run the optimization loop in the background and check on it periodically."
Save the eval set to the workspace, then run in the background:
python -m scripts.run_loop \
--eval-set <path-to-trigger-eval.json> \
--skill-path <path-to-skill> \
--model <model-id-powering-this-session> \
--max-iterations 5 \
--verbose
Use the model ID from your system prompt (the one powering the current session) so the triggering test matches what the user actually experiences.
While it runs, periodically tail the output to give the user updates on which iteration it's on and what the scores look like.
This handles the full optimization loop automatically. It splits the eval set into 60% train and 40% held-out test, evaluates the current description (running each query 3 times to get a reliable trigger rate), then calls Claude to propose improvements based on what failed. It re-evaluates each new description on both train and test, iterating up to 5 times. When it's done, it opens an HTML report in the browser showing the results per iteration and returns JSON with best_description — selected by test score rather than train score to avoid overfitting.
Understanding the triggering mechanism helps design better eval queries. Skills appear in Claude's available_skills list with their name + description, and Claude decides whether to consult a skill based on that description. The important thing to know is that Claude only consults skills for tasks it can't easily handle on its own — simple, one-step queries like "read this PDF" may not trigger a skill even if the description matches perfectly, because Claude can handle them directly with basic tools. Complex, multi-step, or specialized queries reliably trigger skills when the description matches.
This means your eval queries should be substantive enough that Claude would actually benefit from consulting a skill. Simple queries like "read file X" are poor test cases — they won't trigger skills regardless of description quality.
Take best_description from the JSON output and update the skill's SKILL.md frontmatter. Show the user before/after and report the scores.
present_files tool is available)Check whether you have access to the present_files tool. If you don't, skip this step. If you do, package the skill and present the .skill file to the user:
python -m scripts.package_skill <path/to/skill-folder>
After packaging, direct the user to the resulting .skill file path so they can install it.
In Claude.ai, the core workflow is the same (draft → test → review → improve → repeat), but because Claude.ai doesn't have subagents, some mechanics change. Here's what to adapt:
Running test cases : No subagents means no parallel execution. For each test case, read the skill's SKILL.md, then follow its instructions to accomplish the test prompt yourself. Do them one at a time. This is less rigorous than independent subagents (you wrote the skill and you're also running it, so you have full context), but it's a useful sanity check — and the human review step compensates. Skip the baseline runs — just use the skill to complete the task as requested.
Reviewing results : If you can't open a browser (e.g., Claude.ai's VM has no display, or you're on a remote server), skip the browser reviewer entirely. Instead, present results directly in the conversation. For each test case, show the prompt and the output. If the output is a file the user needs to see (like a .docx or .xlsx), save it to the filesystem and tell them where it is so they can download and inspect it. Ask for feedback inline: "How does this look? Anything you'd change?"
Benchmarking : Skip the quantitative benchmarking — it relies on baseline comparisons which aren't meaningful without subagents. Focus on qualitative feedback from the user.
The iteration loop : Same as before — improve the skill, rerun the test cases, ask for feedback — just without the browser reviewer in the middle. You can still organize results into iteration directories on the filesystem if you have one.
Description optimization : This section requires the claude CLI tool (specifically claude -p) which is only available in Claude Code. Skip it if you're on Claude.ai.
Blind comparison : Requires subagents. Skip it.
Packaging : The package_skill.py script works anywhere with Python and a filesystem. On Claude.ai, you can run it and the user can download the resulting .skill file.
Updating an existing skill : The user might be asking you to update an existing skill, not create a new one. In this case:
name frontmatter field -- use them unchanged. E.g., if the installed skill is research-helper, output research-helper.skill (not research-helper-v2)./tmp/skill-name/, edit there, and package from the copy./tmp/ first, then copy to the output directory -- direct writes may fail due to permissions.If you're in Cowork, the main things to know are:
--static <output_path> to write a standalone HTML file instead of starting a server. Then proffer a link that the user can click to open the HTML in their browser.generate_review.py (not writing your own boutique html code). Sorry in advance but I'm gonna go all caps here: GENERATE THE EVAL VIEWER BEFORE evaluating inputs yourself. You want to get them in front of the human ASAP!feedback.json as a file. You can then read it from there (you may have to request access first).package_skill.py just needs Python and a filesystem.run_loop.py / run_eval.py) should work in Cowork just fine since it uses claude -p via subprocess, not a browser, but please save it until you've fully finished making the skill and the user agrees it's in good shape.The agents/ directory contains instructions for specialized subagents. Read them when you need to spawn the relevant subagent.
agents/grader.md — How to evaluate assertions against outputsagents/comparator.md — How to do blind A/B comparison between two outputsagents/analyzer.md — How to analyze why one version beat anotherThe references/ directory has additional documentation:
references/schemas.md — JSON structures for evals.json, grading.json, etc.Repeating one more time the core loop here for emphasis:
eval-viewer/generate_review.py to help the user review themPlease add steps to your TodoList, if you have such a thing, to make sure you don't forget. If you're in Cowork, please specifically put "Create evals JSON and run eval-viewer/generate_review.py so human can review test cases" in your TodoList to make sure it happens.
Good luck!
Weekly Installs
68.9K
Repository
GitHub Stars
87.3K
First Seen
Jan 19, 2026
Security Audits
Installed on
opencode54.1K
gemini-cli50.0K
codex49.5K
github-copilot45.7K
claude-code41.8K
cursor39.5K