可变文件与不可变文件

使用说明

步骤 1：安装先决条件

# Install uv (fast Python package manager)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Clone the repository
git clone https://github.com/karpathy/autoresearch
cd autoresearch

# Install locked dependencies
uv sync

步骤 2：准备数据（一次性，约 2 分钟）

# Downloads FineWeb-Edu parquet shards, trains BPE tokenizer
# Last shard is reserved for validation — never seen during training
uv run prepare.py

对于受限硬件，在运行前编辑 prepare.py：

# Lower MAX_SEQ_LEN for GPUs with limited VRAM
MAX_SEQ_LEN = 256   # default: 2048

步骤 3：运行基线实验

# Single 5-minute experiment to verify setup
uv run train.py > run.log 2>&1

# Extract key metrics
grep "^val_bpb:\|^peak_vram_mb:" run.log

预期输出：

val_bpb: 0.9979
peak_vram_mb: 38420

步骤 4：编写 program.md

program.md 是人类编写的研究章程，代理在每次循环迭代开始时都会读取。将其编写为精确的 Markdown 指令：

# Research Program

## Goal
Minimize val_bpb on the FineWeb-Edu validation set within the 300-second budget.

## Current Baseline
val_bpb: 0.9979 (depth-12 GPT, Muon + AdamW optimizer)

## Directions to Explore
1. Attention variants: MLA, GQA, sliding window, local-global hybrid
2. Layer types: MoE FFN layers, SwiGLU activations
3. Optimizer tuning: Muon momentum, AdamW β values, learning rate schedule
4. Architectural depth/width tradeoffs within VRAM budget

## Constraints
- Must complete within 300 seconds
- Peak VRAM must stay under 39GB
- No new packages (use only what is in pyproject.toml)
- Do not modify prepare.py or constants.py

## Notes from Previous Runs
- Depth-12 improvements transfer to depth-24 (scale-invariant gains)
- RoPE positional encoding outperformed learned embeddings (+0.008 val_bpb)

有效的 program.md 原则：

• 明确说明要探索的内容——模糊的指令会浪费实验次数
• 记录已经尝试过的内容（防止重复实验）
• 明确注明硬件限制
• 使用当前最佳的 val_bpb 作为参考点

步骤 5：运行自主代理循环

将你的 AI 代理（Claude Code、Codex 等）指向仓库，并将 program.md 作为其研究上下文。代理将：

1. 读取 program.md + 当前的 train.py
2. 假设一个改进方案
3. 修改 train.py + 提交
4. 执行 uv run train.py（300 秒）
5. 提取 val_bpb；通过 git 保留或回滚
6. 追加到 results.tsv
7. 重复

使用 Claude Code (OMC)：

# From inside autoresearch/
# Give Claude the context: "Run the autoresearch loop following program.md"

直接使用 Claude Code CLI：

claude "Follow program.md. Run autonomous research loop on train.py.
Execute: uv run train.py, extract val_bpb, keep improvements, revert failures.
Log everything to results.tsv. Do not stop until I say so."

步骤 6：监控结果

# Live monitoring during a run
watch -n 30 "tail -20 results.tsv"

# Count kept vs. discarded
awk -F'\t' '{print $4}' results.tsv | sort | uniq -c

# Find the best experiment
sort -t$'\t' -k2 -n results.tsv | head -5

# Check current best val_bpb
git log --oneline -5

步骤 7：解读 results.tsv

commit    val_bpb    memory_gb    status     description
a3f2c91   0.9697     37.2         keep       SwiGLU activation + depth-12
b8e1d04   0.9821     38.1         discard    MoE 4-expert: marginal gain
c1a5f30   crash      —            crash      OOM: sequence length 4096

示例

示例 1：夜间运行摘要

Session summary: 126 experiments, 18 improvements
Best val_bpb: 0.9697 (started: 0.9979)
Top improvements:
- SwiGLU activation: -0.012 val_bpb
- GQA with 4 KV heads: -0.009 val_bpb
- Muon momentum 0.92→0.95: -0.006 val_bpb

示例 2：低 VRAM 配置（6GB GPU）

# In prepare.py — edit before uv run prepare.py
MAX_SEQ_LEN = 256       # was 2048
EVAL_TOKENS = 2_097_152  # was 20_971_520 (scale down proportionally)

示例 3：按类别提取实验

# Find all attention-related experiments
grep -i "attention\|GQA\|MLA\|MHA" results.tsv

# List only improvements sorted by gain
awk -F'\t' '$4=="keep"' results.tsv | sort -t$'\t' -k2 -n

可用脚本

从 autoresearch 仓库目录内运行：

# Typical overnight session
bash scripts/check-hardware.sh
bash scripts/setup.sh --seq-len 512     # adjust for your VRAM
# Edit program.md with your research directives
bash scripts/run-loop.sh --max 100 --desc "session-1"
bash scripts/show-results.sh --kept-only

参考资料

references/ 目录中的详细文档：

最佳实践

1. 在运行前编写 program.md —— 代理的好坏取决于其指令；模糊的程序会浪费算力
2. 首先运行基线实验 —— 在启动循环之前，始终手动运行 uv run train.py 以确认设置有效
3. 保持 prepare.py 中的 MAX_SEQ_LEN 一致 —— 在运行中途更改会使 val_bpb 比较无效
4. 切勿修改 prepare.py 或 constants.py —— 评估框架必须保持固定，结果才有意义
5. 在提交前扩展改进 —— 在将深度 12 的改进视为根本性增益之前，测试其在深度 24 是否也成立
6. 提交 program.md 更新 —— 将你的研究指令与 results.tsv 一起进行版本控制，以确保可复现性
7. 监控 VRAM —— 在 program.md 中为你的 GPU 余量添加 peak_vram_mb 限制
8. 无新依赖项 —— 代理不能 pip install；它只能使用 pyproject.toml 中的内容

硬件要求

关键指标参考

参考链接

• GitHub — karpathy/autoresearch
• nanochat — 底层 LLM 训练框架
• Karpathy 的原始公告 (X/Twitter)
• DeepWiki — autoresearch 架构
• MIT 许可证

每周安装次数

1

仓库

akillness/oh-my-gods

首次出现

1 天前

安全审计

Gen Agent Trust HubPassSocketPassSnykFail

安装于

mcpjam1

claude-code1

replit1

junie1

windsurf1

zencoder1

Mutable vs. Immutable Files

Instructions

Step 1: Install Prerequisites

# Install uv (fast Python package manager)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Clone the repository
git clone https://github.com/karpathy/autoresearch
cd autoresearch

# Install locked dependencies
uv sync

Step 2: Prepare Data (One-Time, ~2 Minutes)

# Downloads FineWeb-Edu parquet shards, trains BPE tokenizer
# Last shard is reserved for validation — never seen during training
uv run prepare.py

For constrained hardware, edit prepare.py before running:

# Lower MAX_SEQ_LEN for GPUs with limited VRAM
MAX_SEQ_LEN = 256   # default: 2048

Step 3: Run a Baseline Experiment

# Single 5-minute experiment to verify setup
uv run train.py > run.log 2>&1

# Extract key metrics
grep "^val_bpb:\|^peak_vram_mb:" run.log

Expected output:

val_bpb: 0.9979
peak_vram_mb: 38420

Step 4: Author program.md

program.md is the human-written research charter the agent reads at the start of every loop iteration. Write it as precise Markdown instructions:

# Research Program

## Goal
Minimize val_bpb on the FineWeb-Edu validation set within the 300-second budget.

## Current Baseline
val_bpb: 0.9979 (depth-12 GPT, Muon + AdamW optimizer)

## Directions to Explore
1. Attention variants: MLA, GQA, sliding window, local-global hybrid
2. Layer types: MoE FFN layers, SwiGLU activations
3. Optimizer tuning: Muon momentum, AdamW β values, learning rate schedule
4. Architectural depth/width tradeoffs within VRAM budget

## Constraints
- Must complete within 300 seconds
- Peak VRAM must stay under 39GB
- No new packages (use only what is in pyproject.toml)
- Do not modify prepare.py or constants.py

## Notes from Previous Runs
- Depth-12 improvements transfer to depth-24 (scale-invariant gains)
- RoPE positional encoding outperformed learned embeddings (+0.008 val_bpb)

Effective program.md principles:

• Be specific about what to explore — vague directives waste experiments
• Record what has already been tried (prevents redundant experiments)
• Note hardware constraints explicitly
• Use the current best val_bpb as a reference point

Step 5: Run the Autonomous Agent Loop

Point your AI agent (Claude Code, Codex, etc.) at the repository with program.md as its research context. The agent will:

1. Read program.md + current train.py
2. Hypothesize an improvement
3. Modify train.py + commit
4. Execute uv run train.py (300 seconds)
5. Extract val_bpb; keep or revert via git
6. Append to results.tsv
7. Repeat

With Claude Code (OMC):

# From inside autoresearch/
# Give Claude the context: "Run the autoresearch loop following program.md"

With Claude Code CLI directly:

claude "Follow program.md. Run autonomous research loop on train.py.
Execute: uv run train.py, extract val_bpb, keep improvements, revert failures.
Log everything to results.tsv. Do not stop until I say so."

Step 6: Monitor Results

# Live monitoring during a run
watch -n 30 "tail -20 results.tsv"

# Count kept vs. discarded
awk -F'\t' '{print $4}' results.tsv | sort | uniq -c

# Find the best experiment
sort -t$'\t' -k2 -n results.tsv | head -5

# Check current best val_bpb
git log --oneline -5

Step 7: Interpret results.tsv

commit    val_bpb    memory_gb    status     description
a3f2c91   0.9697     37.2         keep       SwiGLU activation + depth-12
b8e1d04   0.9821     38.1         discard    MoE 4-expert: marginal gain
c1a5f30   crash      —            crash      OOM: sequence length 4096

Examples

Example 1: Overnight Run Summary

Session summary: 126 experiments, 18 improvements
Best val_bpb: 0.9697 (started: 0.9979)
Top improvements:
- SwiGLU activation: -0.012 val_bpb
- GQA with 4 KV heads: -0.009 val_bpb
- Muon momentum 0.92→0.95: -0.006 val_bpb

Example 2: Low-VRAM Configuration (6GB GPU)

# In prepare.py — edit before uv run prepare.py
MAX_SEQ_LEN = 256       # was 2048
EVAL_TOKENS = 2_097_152  # was 20_971_520 (scale down proportionally)

Example 3: Extract Experiments by Category

# Find all attention-related experiments
grep -i "attention\|GQA\|MLA\|MHA" results.tsv

# List only improvements sorted by gain
awk -F'\t' '$4=="keep"' results.tsv | sort -t$'\t' -k2 -n

Available scripts

Run from inside the autoresearch repository directory:

# Typical overnight session
bash scripts/check-hardware.sh
bash scripts/setup.sh --seq-len 512     # adjust for your VRAM
# Edit program.md with your research directives
bash scripts/run-loop.sh --max 100 --desc "session-1"
bash scripts/show-results.sh --kept-only

References

Detailed documentation in references/:

Best practices

1. Write program.md before running — the agent is only as good as its directives; vague programs waste compute
2. Start with the baseline first — always uv run train.py manually before launching the loop to confirm the setup works
3. KeepMAX_SEQ_LEN in prepare.py consistent — changing it mid-run invalidates val_bpb comparisons
4. Never modifyprepare.py or constants.py — the evaluation harness must stay fixed for results to be meaningful
5. Scale improvements before committing — test that a depth-12 improvement also holds at depth-24 before treating it as a fundamental gain
6. Commitprogram.md updates — version-control your research directives alongside results.tsv for reproducibility
7. Monitor VRAM — add peak_vram_mb constraints in program.md for your GPU's headroom
8. No new dependencies — the agent cannot pip install; it can only use what is in pyproject.toml

Hardware Requirements

Key Metrics Reference

References

• GitHub — karpathy/autoresearch
• nanochat — the underlying LLM training framework
• Karpathy's original announcement (X/Twitter)
• DeepWiki — autoresearch architecture
• MIT License

Weekly Installs

1

Repository

akillness/oh-my-gods

First Seen

1 day ago

Security Audits

Gen Agent Trust HubPassSocketPassSnykFail

Installed on

mcpjam1

claude-code1

replit1

junie1

windsurf1

zencoder1