triage-ci-flake by payloadcms/payload
npx skills add https://github.com/payloadcms/payload --skill triage-ci-flake用于在 CI 中排查和修复测试失败的系统化工作流程,特别是那些在本地通过但在 CI 中失败的“不稳定”测试。那些能进入 main 分支的测试通常由于时序、打包或环境差异而变得不稳定。
关键规则:在提出任何修复方案之前,你必须运行复现工作流程。没有例外。
main 分支上的 CI 测试失败你必须执行这些命令。阅读代码或分析日志不算作复现。
pnpm dev $SUITE_NAME(使用 run_in_background=true)pnpm prepare-run-test-against-prod广告位招租
在这里展示您的产品或服务
触达数万 AI 开发者,精准高效
pnpm dev:prod $SUITE_NAME 并再次运行测试只有在执行了这些命令并看到它们的输出之后,你才能继续进行分析和修复。
“从日志分析”不是复现。你必须运行这些命令。
digraph triage_ci {
"CI failure reported" [shape=box];
"Extract details from CI logs" [shape=box];
"Identify suite and test name" [shape=box];
"Run dev server: pnpm dev $SUITE" [shape=box];
"Run specific test by name" [shape=box];
"Did test fail?" [shape=diamond];
"Debug with dev code" [shape=box];
"Run prepare-run-test-against-prod" [shape=box];
"Run: pnpm dev:prod $SUITE" [shape=box];
"Run specific test again" [shape=box];
"Did test fail now?" [shape=diamond];
"Debug bundling issue" [shape=box];
"Unable to reproduce - check logs" [shape=box];
"Fix and verify" [shape=box];
"CI failure reported" -> "Extract details from CI logs";
"Extract details from CI logs" -> "Identify suite and test name";
"Identify suite and test name" -> "Run dev server: pnpm dev $SUITE";
"Run dev server: pnpm dev $SUITE" -> "Run specific test by name";
"Run specific test by name" -> "Did test fail?";
"Did test fail?" -> "Debug with dev code" [label="yes"];
"Did test fail?" -> "Run prepare-run-test-against-prod" [label="no"];
"Run prepare-run-test-against-prod" -> "Run: pnpm dev:prod $SUITE";
"Run: pnpm dev:prod $SUITE" -> "Run specific test again";
"Run specific test again" -> "Did test fail now?";
"Did test fail now?" -> "Debug bundling issue" [label="yes"];
"Did test fail now?" -> "Unable to reproduce - check logs" [label="no"];
"Debug with dev code" -> "Fix and verify";
"Debug bundling issue" -> "Fix and verify";
}
从 CI 日志或 GitHub Actions URL 中识别:
i18n、fields、lexical)test/i18n/e2e.spec.ts)关键:始终按名称运行特定测试,而不是整个套件。
服务器管理规则:
# ========================================
# 步骤 2A:停止所有服务器
# ========================================
lsof -ti:3000 | xargs kill -9 2>/dev/null || echo "Port 3000 clear"
# ========================================
# 步骤 2B:启动开发服务器
# ========================================
# 使用套件启动开发服务器(在后台使用 run_in_background=true)
pnpm dev $SUITE_NAME
# ========================================
# 步骤 2C:等待服务器就绪
# ========================================
# 等待服务器准备就绪(必需 - 不要跳过)
until curl -s http://localhost:3000/admin > /dev/null 2>&1; do sleep 1; done && echo "Server ready"
# ========================================
# 步骤 2D:运行特定测试
# ========================================
# 直接使用 Playwright 运行特定的失败测试
# 对于端到端测试(不要使用 pnpm test:e2e,因为它会启动自己的服务器):
pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts -g "exact test name"
# 对于集成测试:
pnpm test:int $SUITE_NAME -t "exact test name"
测试失败了吗?
如果测试在开发代码下通过,问题很可能出在打包/生产代码中。
重要:在启动生产服务器之前,你必须停止开发服务器。
# ========================================
# 步骤 3A:停止所有服务器(包括步骤 2 中的开发服务器)
# ========================================
lsof -ti:3000 | xargs kill -9 2>/dev/null || echo "Port 3000 clear"
# ========================================
# 步骤 3B:为生产环境构建和打包
# ========================================
# 构建所有包并打包它们(这需要时间 - 请耐心等待)
pnpm prepare-run-test-against-prod
# ========================================
# 步骤 3C:启动生产服务器
# ========================================
# 启动生产开发服务器(在后台使用 run_in_background=true)
pnpm dev:prod $SUITE_NAME
# ========================================
# 步骤 3D:等待服务器就绪
# ========================================
# 等待服务器准备就绪(必需 - 不要跳过)
until curl -s http://localhost:3000/admin > /dev/null 2>&1; do sleep 1; done && echo "Server ready"
# ========================================
# 步骤 3E:运行特定测试
# ========================================
# 再次直接使用 Playwright 运行特定测试
pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts -g "exact test name"
# 或者对于集成测试:
pnpm test:int $SUITE_NAME -t "exact test name"
现在测试失败了吗?
如果在两次尝试后仍无法在本地复现:
for i in {1..10}; do pnpm test:e2e...; done)修复模式:
toBeVisible()、toHaveText())waitForFunction()afterEach 中缺少清理修复模式:
afterEach 中清理deleteAllsetTimeout/sleep 而不是基于条件的等待修复模式:
waitForPageStability() 辅助函数修复端到端测试时,请注意这些 eslint 规则:
playwright/no-networkidle - 避免使用 waitForLoadState('networkidle')(改用基于条件的等待)payload/no-wait-function - 避免使用自定义的 wait() 函数(使用 Playwright 内置的等待)payload/no-flaky-assertions - 避免使用不可重试的断言playwright/prefer-web-first-assertions - 使用内置的 Playwright 断言现有代码可能违反这些规则 - 在添加新代码时,即使现有代码不遵守,也要遵循这些规则。
修复后:
# 确保开发服务器在端口 3000 上运行
# 多次运行测试以确认稳定性
for i in {1..10}; do
pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts -g "exact test name" || break
done
# 运行完整套件
pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts
# 如果你修改了打包代码,使用生产构建进行测试
lsof -ti:3000 | xargs kill -9 2>/dev/null
pnpm prepare-run-test-against-prod
pnpm dev:prod $SUITE_NAME
until curl -s http://localhost:3000/admin > /dev/null; do sleep 1; done
pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts
没有复现,就没有修复
如果你在完成工作流程的步骤 1-3 之前就提出修复方案,你就违反了这项技能。
即使出现以下情况,这也适用:
没有例外。首先运行复现工作流程。
每一个跳过复现的借口,以及为什么它是错误的:
| 合理化理由 | 现实情况 |
|---|---|
| “日志显示了确切的错误” | 日志显示的是症状,而不是根本原因。进行复现。 |
| “我能在代码中看到问题” | 你是在猜测。进行复现以确认。 |
| “这显然是一个竞态条件” | 也许吧。进行复现以确定。 |
| “我以前见过这个错误” | 这次可能不同。进行复现。 |
| “堆栈跟踪很清晰” | 堆栈跟踪显示的是位置,而不是原因。进行复现。 |
| “时间紧迫 - 需要快速修复” | 复现本身就是快速的。猜测会浪费时间。 |
| “测试文件显示了问题” | 阅读 ≠ 运行。执行命令。 |
| “我先分析一下代码” | 代码分析应在复现之后进行。 |
| “让我调查一下根本原因” | 复现本身就是调查。 |
| “我需要理解这个错误” | 理解来自于看到它失败。 |
这些想法意味着你即将违反这项技能:
所有这些都意味着:立即停止。执行复现命令。没有执行就没有分析。
正确的方法 - 首先执行命令:
User: CI failed on i18n suite - https://github.com/.../actions/runs/XXX
Agent:
1. Extracted details from CI logs:
- Suite: i18n
- Test: "ensure translations update correctly when switching language"
- File: test/i18n/e2e.spec.ts:124
- Error: "Error loading language: en" TypeError: network error
2. Executing reproduction workflow:
[Uses Bash tool to kill ports]
$ lsof -ti:3000 | xargs kill -9 2>/dev/null || echo "Port 3000 clear"
[Uses Bash tool with run_in_background=true]
$ pnpm dev i18n
[Waits for server to start with curl check]
$ until curl -s http://localhost:3000/admin > /dev/null; do sleep 1; done && echo "Server ready"
[Uses Bash tool to run Playwright directly]
$ pnpm exec playwright test test/i18n/e2e.spec.ts -g "ensure translations update correctly when switching language"
Output shows: Test PASSED locally
3. Test passed with dev code, trying bundled code:
[Kills ports again]
$ lsof -ti:3000 | xargs kill -9 2>/dev/null
[Uses Bash tool]
$ pnpm prepare-run-test-against-prod
[Uses Bash tool with run_in_background=true]
$ pnpm dev:prod i18n
[Waits for server]
$ until curl -s http://localhost:3000/admin > /dev/null; do sleep 1; done
[Uses Bash tool]
$ pnpm exec playwright test test/i18n/e2e.spec.ts -g "ensure translations update correctly when switching language"
Output shows: Test FAILED - reproduced!
4. Now that I've reproduced it, analyzing the failure...
[Code analysis, root cause identification]
5. Implementing fix: [specific changes]
6. Verification:
[Uses Bash tool]
$ for i in {1..10}; do pnpm test:e2e i18n -g "exact test"; done
All 10 runs passed!
错误的方法 - 不执行就进行分析:
User: CI failed on i18n suite
Agent: Looking at the error logs, I can see this is a race condition...
[Analyzes code, proposes fix without running any commands]
❌ WRONG - This violates the skill. Must execute reproduction commands first.
| 错误 | 修复方法 |
|---|---|
| 首先运行完整测试套件 | 按名称运行特定测试 |
| 跳过开发代码复现 | 始终先尝试开发代码 |
| 不使用打包代码测试 | 如果开发代码通过,使用 prepare-run-test-against-prod 测试 |
| 未复现就提出修复方案 | 遵循工作流程 - 先复现 |
在新代码中使用 networkidle | 使用带有 waitForFunction() 的基于条件的等待 |
添加任意的 wait() 调用 | 使用 Playwright 内置的断言和等待 |
在你完成以下步骤之后:
你必须提示用户创建 PR:
The fix has been verified and is ready for review. Would you like me to create a PR with these changes?
Summary of changes:
- [List files modified]
- [Brief description of the fix]
- [Verification results]
重要:
这确保了用户对提交审查的内容具有可见性和控制权。
每周安装量
114
仓库
GitHub 星标数
41.4K
首次出现
2026年1月21日
安全审计
安装于
codex102
opencode101
gemini-cli98
github-copilot94
cursor93
claude-code92
Systematic workflow for triaging and fixing test failures in CI, especially flaky tests that pass locally but fail in CI. Tests that made it to main are usually flaky due to timing, bundling, or environment differences.
CRITICAL RULE: You MUST run the reproduction workflow before proposing any fixes. No exceptions.
main branch after PR was mergedYOU MUST EXECUTE THESE COMMANDS. Reading code or analyzing logs does NOT count as reproduction.
pnpm dev $SUITE_NAME (use run_in_background=true)pnpm prepare-run-test-against-prodpnpm dev:prod $SUITE_NAME and run test againOnly after EXECUTING these commands and seeing their output can you proceed to analysis and fixes.
"Analysis from logs" is NOT reproduction. You must RUN the commands.
digraph triage_ci {
"CI failure reported" [shape=box];
"Extract details from CI logs" [shape=box];
"Identify suite and test name" [shape=box];
"Run dev server: pnpm dev $SUITE" [shape=box];
"Run specific test by name" [shape=box];
"Did test fail?" [shape=diamond];
"Debug with dev code" [shape=box];
"Run prepare-run-test-against-prod" [shape=box];
"Run: pnpm dev:prod $SUITE" [shape=box];
"Run specific test again" [shape=box];
"Did test fail now?" [shape=diamond];
"Debug bundling issue" [shape=box];
"Unable to reproduce - check logs" [shape=box];
"Fix and verify" [shape=box];
"CI failure reported" -> "Extract details from CI logs";
"Extract details from CI logs" -> "Identify suite and test name";
"Identify suite and test name" -> "Run dev server: pnpm dev $SUITE";
"Run dev server: pnpm dev $SUITE" -> "Run specific test by name";
"Run specific test by name" -> "Did test fail?";
"Did test fail?" -> "Debug with dev code" [label="yes"];
"Did test fail?" -> "Run prepare-run-test-against-prod" [label="no"];
"Run prepare-run-test-against-prod" -> "Run: pnpm dev:prod $SUITE";
"Run: pnpm dev:prod $SUITE" -> "Run specific test again";
"Run specific test again" -> "Did test fail now?";
"Did test fail now?" -> "Debug bundling issue" [label="yes"];
"Did test fail now?" -> "Unable to reproduce - check logs" [label="no"];
"Debug with dev code" -> "Fix and verify";
"Debug bundling issue" -> "Fix and verify";
}
From CI logs or GitHub Actions URL, identify:
i18n, fields, lexical)test/i18n/e2e.spec.ts)CRITICAL: Always run the specific test by name, not the full suite.
SERVER MANAGEMENT RULES:
# ========================================
# STEP 2A: STOP ALL SERVERS
# ========================================
lsof -ti:3000 | xargs kill -9 2>/dev/null || echo "Port 3000 clear"
# ========================================
# STEP 2B: START DEV SERVER
# ========================================
# Start dev server with the suite (in background with run_in_background=true)
pnpm dev $SUITE_NAME
# ========================================
# STEP 2C: WAIT FOR SERVER READY
# ========================================
# Wait for server to be ready (REQUIRED - do not skip)
until curl -s http://localhost:3000/admin > /dev/null 2>&1; do sleep 1; done && echo "Server ready"
# ========================================
# STEP 2D: RUN SPECIFIC TEST
# ========================================
# Run ONLY the specific failing test using Playwright directly
# For E2E tests (DO NOT use pnpm test:e2e as it spawns its own server):
pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts -g "exact test name"
# For integration tests:
pnpm test:int $SUITE_NAME -t "exact test name"
Did the test fail?
If test passed with dev code, the issue is likely in bundled/production code.
IMPORTANT: You MUST stop the dev server before starting prod server.
# ========================================
# STEP 3A: STOP ALL SERVERS (INCLUDING DEV SERVER FROM STEP 2)
# ========================================
lsof -ti:3000 | xargs kill -9 2>/dev/null || echo "Port 3000 clear"
# ========================================
# STEP 3B: BUILD AND PACK FOR PROD
# ========================================
# Build all packages and pack them (this takes time - be patient)
pnpm prepare-run-test-against-prod
# ========================================
# STEP 3C: START PROD SERVER
# ========================================
# Start prod dev server (in background with run_in_background=true)
pnpm dev:prod $SUITE_NAME
# ========================================
# STEP 3D: WAIT FOR SERVER READY
# ========================================
# Wait for server to be ready (REQUIRED - do not skip)
until curl -s http://localhost:3000/admin > /dev/null 2>&1; do sleep 1; done && echo "Server ready"
# ========================================
# STEP 3E: RUN SPECIFIC TEST
# ========================================
# Run the specific test again using Playwright directly
pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts -g "exact test name"
# OR for integration tests:
pnpm test:int $SUITE_NAME -t "exact test name"
Did the test fail now?
If you cannot reproduce locally after both attempts:
for i in {1..10}; do pnpm test:e2e...; done)Fix patterns:
toBeVisible(), toHaveText())waitForFunction() with condition checksafterEachFix patterns:
afterEachdeleteAll that affects other testssetTimeout/sleep instead of condition-based waitingFix patterns:
waitForPageStability() helperWhen fixing e2e tests, be aware of these eslint rules:
playwright/no-networkidle - Avoid waitForLoadState('networkidle') (use condition-based waiting instead)payload/no-wait-function - Avoid custom wait() functions (use Playwright's built-in waits)payload/no-flaky-assertions - Avoid non-retryable assertionsplaywright/prefer-web-first-assertions - Use built-in Playwright assertionsExisting code may violate these rules - when adding new code, follow the rules even if existing code doesn't.
After fixing:
# Ensure dev server is running on port 3000
# Run test multiple times to confirm stability
for i in {1..10}; do
pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts -g "exact test name" || break
done
# Run full suite
pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts
# If you modified bundled code, test with prod build
lsof -ti:3000 | xargs kill -9 2>/dev/null
pnpm prepare-run-test-against-prod
pnpm dev:prod $SUITE_NAME
until curl -s http://localhost:3000/admin > /dev/null; do sleep 1; done
pnpm exec playwright test test/$SUITE_NAME/e2e.spec.ts
NO FIX WITHOUT REPRODUCTION FIRST
If you propose a fix before completing steps 1-3 of the workflow, you've violated this skill.
This applies even when:
No exceptions. Run the reproduction workflow first.
Every excuse for skipping reproduction, and why it's wrong:
| Rationalization | Reality |
|---|---|
| "The logs show the exact error" | Logs show symptoms, not root cause. Reproduce. |
| "I can see the problem in the code" | You're guessing. Reproduce to confirm. |
| "This is obviously a race condition" | Maybe. Reproduce to be sure. |
| "I've seen this error before" | This might be different. Reproduce. |
| "The stack trace is clear" | Stack trace shows where, not why. Reproduce. |
| "Time pressure - need to fix fast" | Reproducing IS fast. Guessing wastes time. |
| "The test file shows the issue" | Reading ≠ running. Execute the commands. |
| "I'll analyze the code first" | Code analysis comes AFTER reproduction. |
| "Let me investigate the root cause" | Reproduction IS the investigation. |
| "I need to understand the error" | Understanding comes from seeing it fail. |
These thoughts mean you're about to violate the skill:
All of these mean: Stop immediately. Execute the reproduction commands. No analysis without execution.
CORRECT approach - executes commands first:
User: CI failed on i18n suite - https://github.com/.../actions/runs/XXX
Agent:
1. Extracted details from CI logs:
- Suite: i18n
- Test: "ensure translations update correctly when switching language"
- File: test/i18n/e2e.spec.ts:124
- Error: "Error loading language: en" TypeError: network error
2. Executing reproduction workflow:
[Uses Bash tool to kill ports]
$ lsof -ti:3000 | xargs kill -9 2>/dev/null || echo "Port 3000 clear"
[Uses Bash tool with run_in_background=true]
$ pnpm dev i18n
[Waits for server to start with curl check]
$ until curl -s http://localhost:3000/admin > /dev/null; do sleep 1; done && echo "Server ready"
[Uses Bash tool to run Playwright directly]
$ pnpm exec playwright test test/i18n/e2e.spec.ts -g "ensure translations update correctly when switching language"
Output shows: Test PASSED locally
3. Test passed with dev code, trying bundled code:
[Kills ports again]
$ lsof -ti:3000 | xargs kill -9 2>/dev/null
[Uses Bash tool]
$ pnpm prepare-run-test-against-prod
[Uses Bash tool with run_in_background=true]
$ pnpm dev:prod i18n
[Waits for server]
$ until curl -s http://localhost:3000/admin > /dev/null; do sleep 1; done
[Uses Bash tool]
$ pnpm exec playwright test test/i18n/e2e.spec.ts -g "ensure translations update correctly when switching language"
Output shows: Test FAILED - reproduced!
4. Now that I've reproduced it, analyzing the failure...
[Code analysis, root cause identification]
5. Implementing fix: [specific changes]
6. Verification:
[Uses Bash tool]
$ for i in {1..10}; do pnpm test:e2e i18n -g "exact test"; done
All 10 runs passed!
INCORRECT approach - analyzes without executing:
User: CI failed on i18n suite
Agent: Looking at the error logs, I can see this is a race condition...
[Analyzes code, proposes fix without running any commands]
❌ WRONG - This violates the skill. Must execute reproduction commands first.
| Mistake | Fix |
|---|---|
| Running full test suite first | Run specific test by name |
| Skipping dev code reproduction | Always try dev code first |
| Not testing with bundled code | If dev passes, test with prepare-run-test-against-prod |
| Proposing fix without reproducing | Follow the workflow - reproduce first |
Using networkidle in new code | Use condition-based waiting with waitForFunction() |
Adding arbitrary wait() calls | Use Playwright's built-in assertions and waits |
After you have:
You MUST prompt the user to create a PR:
The fix has been verified and is ready for review. Would you like me to create a PR with these changes?
Summary of changes:
- [List files modified]
- [Brief description of the fix]
- [Verification results]
IMPORTANT:
This ensures the user has visibility and control over what gets submitted for review.
Weekly Installs
114
Repository
GitHub Stars
41.4K
First Seen
Jan 21, 2026
Security Audits
Gen Agent Trust HubPassSocketPassSnykPass
Installed on
codex102
opencode101
gemini-cli98
github-copilot94
cursor93
claude-code92
MCP-CLI:命令行访问MCP服务器的工具,支持GitHub、文件系统、数据库交互
8,200 周安装
智能体评估模式详解:agentic-eval 迭代优化与自我提升框架
8,300 周安装
Nano Banana Pro OpenRouter:基于OpenRouter的AI图像生成与编辑工具,支持Gemini-3-Pro模型
8,300 周安装
.NET/C# 设计模式代码审查工具 - 自动化分析命令、工厂、仓储等模式实现
8,300 周安装
AI就绪规范创建工具 - 结构化文档生成器,提升AI协作效率
8,400 周安装
Playwright自动化表单填写教程 - 使用MCP实现高效Web表单自动化测试
8,500 周安装