ARTICLE · 1006788
有关AI模型结构自生长的探索
我们找到了病灶，但发现手术刀不够长
引子：一次关于AI“内部手术”的实验
过去一段时间，我们一直在尝试一件听起来有点疯狂的事情：给AI模型做“内部手术”。
我们想知道：当一个AI学会了一个任务后，如果让它学习一个新任务，我们能不能提前“看到”它的结构哪里不够用，然后精准地在那个位置“生长”出新结构来帮助它？
这听起来像是科幻小说——给AI做手术，让它“长出”新能力。但我们已经把这件事变成了实验室里的真实实验。
实验规模很小（模型只有约54万参数，比主流大模型小几个数量级），但它揭示了一些极其清晰的信号。最近的一轮实验，更是给出了迄今为止最精确的答案。
一、我们的“手术方案”
1.1 两个任务：先学A，再测B
我们让模型学习两个数学任务：
· 任务A：给一串数字，计算它们的累积和（比如 1,2,3 → 1,3,6）。
· 任务B：给一串数字，计算它们的加权累积和（比如 1,2,3 → 1, 1×1+2×2, 1×1+2×2+3×3 = 1,5,14）。
这两个任务看起来很像——都是“累加”——但任务B多了一个关键要求：每个数字要乘以它的位置序号。这需要模型学会“位置加权”的概念。
1.2 三步走：诊断 → 定位 → 生长
第一步：学会任务A。 我们把模型训练到任务A的准确率超过90%，让它形成一个稳定的“计算骨架”。
第二步：用“X光”扫描模型。 我们用了三个工具来“看”模型内部：
· 有效秩：测量模型实际用到了多少个“独立方向”。如果这个数字下降，说明模型正在把知识压缩到更少的维度上。
· 外部压力：让模型看一眼任务B的数据，计算它的“学习方向”中有多少是现有结构无法覆盖的。
· 消融测试：删除模型尾部的一些“次要方向”，看任务B的表现会变好还是变差。
第三步：尝试“结构生长”。 我们在诊断出的“最需要生长”的位置添加一个小型新支路（仅占模型总参数量的1%左右），然后让它在任务B上训练。
我们比较了四种情况：
· 不生长：原模型直接学任务B。
· 随机生长：新支路的方向是随机选的。
· SVD引导生长：新支路的方向基于当前权重的几何结构。
· 梯度引导生长：新支路的方向基于任务B的“学习信号”指向。
· Oracle（“理想”生长）：新支路加上让基座也进行适应性调整——这是我们的“上限对照”。
如果“引导生长”能比“随机生长”学到更多东西，那就说明我们的诊断是有用的。
二、我们看到了什么？
2.1 诊断：教科书级别的成功
为了确保结果不是偶然，我们跑了三个不同的随机种子。诊断部分的结果出奇地一致：
种子 外部压力 有效秩
42 44.5% 36.47
123 63.0% 27.93
2026 71.5% 24.64
更重要的是，三个种子都精确地指向了同一层——模型的最后一层——作为“最需要生长”的位置。
诊断部分的结论非常清晰：
· 模型在学习任务A的过程中，确实把知识压缩到了更少的方向上（有效秩下降）。
· 当面对任务B时，有44%~71%的“学习方向”指向了现有结构之外（外部压力很强）。
· 诊断工具能够精确定位到“病灶”的位置。
2.2 三种引导生长：精确到0.0000的“完全无效”
然后我们满怀信心地进行了“生长手术”。结果让人震惊：
生长方式 种子42 种子123 种子2026
随机生长 0.0000 0.0000 0.0000
SVD引导生长 0.0000 0.0000 0.0000
梯度引导生长 0.0000 0.0000 0.0000
三种引导生长，在三个独立种子上，增益全部精确为0。任务B的表现没有任何改善。
这不是“效果不好”——这是完全没有效果。三种不同的引导方向，结果完全一样，就像新添加的支路根本不存在一样。
2.3 Oracle：意外中的意外
但Oracle的结果给了我们一个巨大的反差：
种子 Oracle增益 B损失变化
42 11.86 15.85 → 3.99
123 15.65 19.91 → 4.25
2026 12.64 16.68 → 4.05
Oracle成功了！而且非常成功——损失的下降幅度巨大（从15~20降到3.9~4.3）。
但是，有一个关键的细节：准确率几乎没变（仍保持在6.7%左右，接近随机）。
这意味着什么？Oracle能够消除任务A带来的干扰（损失大幅下降），但无法真正学会任务B（准确率不动）。
三、这是怎么回事？
3.1 我们的“手术刀”根本没碰到病灶
经过深入分析，我们发现了问题的根源：
当模型学会任务A后，它的最后一层FFN（前馈网络）的中间层激活（GELU层输出）高度专化于任务A。当面对任务B时，这个中间层几乎不输出任何信号——它被“门控”关掉了。
而我们的growth branch（新支路）正好接在这个中间层之后。如果中间层的输出≈0，那么无论growth branch的权重设成什么，它的输出都是0×权重=0。
这就像：我们给病人开刀，找到了病灶的位置，但发现手术器械根本伸不进去——因为通往病灶的通道被锁死了。
三种不同的引导方向（随机、SVD、梯度）在“输入为零”的情况下毫无差别——方向再好，乘以0还是0。
3.2 Oracle的“半成功”揭示了一个更深层的真相
Oracle之所以能成功（降低损失），是因为它解冻了基座的最后一层，让模型能够“忘记”部分任务A的专有表示，从而让损失下降。
但Oracle之所以不能完全成功（准确率不动），是因为任务B的真正困难不在于最后一层FFN的容量，而在于它需要跨层的位置加权计算——这是一个需要在多个层之间协同构建的新计算路径。
这就像一个乐队：不是换一个乐手就能解决问题，而是需要重新编排整个乐谱。
四、这次实验的真正价值
如果我们的目标是“证明结构生长有效”，那这次实验无疑是失败了。但如果我们是在探索“AI结构演化的边界”，那这次实验是一次巨大的成功。
4.1 我们确认了四件事
1. 诊断工具是有效的。三个独立种子都精确定位到同一层，外部压力信号稳定且强烈。
2. “局部低秩生长”存在明确的前置条件。当基座被冻结时，无论方向如何优化，都无法产生任何效果。
3. “灾难性干扰”是真实的。任务A的专有表示会主动“锁死”对新任务的通道。
4. 真正的瓶颈是“跨层计算路径”，而非“单层容量不足”。
4.2 这给出了明确的下一步方向
v6的核心改动应该是：把growth branch插入到输入不为零的位置。
具体来说：
· 方案A：插入到注意力层输出之后（而非FFN中间层之后），此时输入包含Token之间的上下文交互。
· 方案B：直接在残差连接上添加并行路径，绕过FFN的“门控”机制。
· 方案C：在多个层同时插入growth branch，构建跨层的新计算通道。
五、结语：科学就是不断缩小未知的范围
如果这次实验跑出了“引导生长显著优于随机生长”的结果，我们会很高兴。但现实给了我们一个更珍贵的礼物：一个精确到小数点后四位的“零结果”，以及一个清晰的机制解释。
我们发现了：
· 诊断工具能定位“病灶”（✅ 三种子全部一致）。
· 但“病灶”不是单层容量不足，而是跨层计算路径缺失（✅ Oracle的“Loss降但Acc不动”证明了这一点）。
· 基座冻结会阻断新任务的信号流入（✅ 三冻结臂gain=0.0000证明了这一点）。
这就像医生用CT找到了肿瘤，然后发现微创手术器械伸不进去——不是诊断错了，而是手术方案需要升级。
每一次这样的实验，都在把“未知”的范围缩小一点点。我们知道病灶在哪里了，知道它为什么没能被切除，也知道下一步该怎么做。而这，正是科学最本来的样子。 # ============================================================
# Smart Density / Structural Growth - v5
# ============================================================
# Key upgrades from v4:
# 1) Multi-layer diagnosis: compute outside_pressure for all layers,
# then select the layer with the largest structural gap.
# 2) Four growth interventions on the selected layer:
# - Random (random orthogonal)
# - SVD (based on weight geometry)
# - Gradient (based on projected gradient directions)
# - Oracle (full capacity: unfreeze the entire layer + large branch)
# 3) Increased growth scale and partial base unfreezing to overcome
# "confident error" interference.
# 4) Strict causal comparison with matched parameters and steps.
# ============================================================
import os, json, math, time, copy, random
from dataclasses import dataclass, asdict
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader
BASE_SEED = 42
SEEDS = [42, 123, 2026]
def seed_everything(s):
random.seed(s)
np.random.seed(s)
torch.manual_seed(s)
if torch.cuda.is_available():
torch.cuda.manual_seed_all(s)
seed_everything(BASE_SEED)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
if device.type == 'cuda':
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = True
print('PyTorch:', torch.__version__)
print('Device:', device)
if device.type == 'cuda':
print('GPU:', torch.cuda.get_device_name(0))
# ---------------------------
# Configuration
# ---------------------------
@dataclass
class CFG:
vocab_size: int = 32
seq_len: int = 32
d_model: int = 128
n_heads: int = 4
n_layers: int = 4
d_ff: int = 256
dropout: float = 0.0
batch_size: int = 64
phase_a_max_steps: int = 8000
phase_a_target_acc: float = 0.90
growth_steps: int = 800
eval_batches: int = 16
lr: float = 1e-3
growth_lr: float = 1e-3 # increased for growth
weight_decay: float = 0.01
svd_rank: int = 16
growth_rank: int = 16 # larger rank for oracle
train_size: int = 12000
eval_size: int = 2000
eps: float = 1e-12
# growth scale: initial output magnitude
growth_scale: float = 0.3 # much larger than v4's 0.01
cfg = CFG()
# ---------------------------
# Data utilities (same as v4)
# ---------------------------
PAD = 0
BOS = 1
TOKEN_OFFSET = 4
DATA_V = cfg.vocab_size - TOKEN_OFFSET
def make_example(task):
x = torch.randint(TOKEN_OFFSET, cfg.vocab_size, (cfg.seq_len,))
z = x - TOKEN_OFFSET
if task == 'A':
y = torch.cumsum(z, 0) % DATA_V
elif task == 'B':
y = torch.cumsum(z * torch.arange(1, cfg.seq_len + 1), 0) % DATA_V
else:
raise ValueError(task)
return x, (y + TOKEN_OFFSET).long()
class ToyTaskDataset(Dataset):
def __init__(self, task, n):
data = [make_example(task) for _ in range(n)]
self.x = torch.stack([d[0] for d in data])
self.y = torch.stack([d[1] for d in data])
def __len__(self):
return self.x.size(0)
def __getitem__(self, i):
return self.x[i], self.y[i]
def loaders():
train_A = ToyTaskDataset('A', cfg.train_size)
train_B = ToyTaskDataset('B', cfg.train_size)
eval_A = ToyTaskDataset('A', cfg.eval_size)
eval_B = ToyTaskDataset('B', cfg.eval_size)
pin = device.type == 'cuda'
return (
DataLoader(train_A, batch_size=cfg.batch_size, shuffle=True, drop_last=True, pin_memory=pin),
DataLoader(train_B, batch_size=cfg.batch_size, shuffle=True, drop_last=True, pin_memory=pin),
DataLoader(eval_A, batch_size=cfg.batch_size, shuffle=False, drop_last=True, pin_memory=pin),
DataLoader(eval_B, batch_size=cfg.batch_size, shuffle=False, drop_last=True, pin_memory=pin)
)
# ---------------------------
# Model (same as v4)
# ---------------------------
class TinyTransformer(nn.Module):
def __init__(self):
super().__init__()
self.tok = nn.Embedding(cfg.vocab_size, cfg.d_model)
self.pos = nn.Parameter(torch.randn(1, cfg.seq_len, cfg.d_model) * 0.02)
layer = nn.TransformerEncoderLayer(
cfg.d_model, cfg.n_heads, cfg.d_ff, cfg.dropout,
'gelu', batch_first=True, norm_first=True
)
self.blocks = nn.TransformerEncoder(layer, cfg.n_layers)
self.norm = nn.LayerNorm(cfg.d_model)
self.head = nn.Linear(cfg.d_model, cfg.vocab_size)
mask = torch.full((cfg.seq_len, cfg.seq_len), float('-inf'))
mask = torch.triu(mask, 1)
self.register_buffer('causal_mask', mask, persistent=False)
def forward(self, x, return_hidden=False):
h = self.tok(x) + self.pos[:, :x.size(1)]
h = self.blocks(h, mask=self.causal_mask[:x.size(1), :x.size(1)])
h = self.norm(h)
z = self.head(h)
return (z, h) if return_hidden else z
# ---------------------------
# Helpers (same as v4)
# ---------------------------
def count_params(m, trainable=False):
return sum(p.numel() for p in m.parameters() if (p.requires_grad or not trainable))
def batch_to_device(b):
return b[0].to(device, non_blocking=True), b[1].to(device, non_blocking=True)
def loss_batch(m, b):
x, y = batch_to_device(b)
logits = m(x)
return F.cross_entropy(logits.reshape(-1, cfg.vocab_size), y.reshape(-1))
@torch.no_grad()
def evaluate(m, loader, n=None):
m.eval()
total_loss = 0.0
count = 0
for i, b in enumerate(loader):
if n is not None and i >= n:
break
total_loss += float(loss_batch(m, b))
count += 1
return total_loss / max(count, 1)
@torch.no_grad()
def accuracy(m, loader, n=None):
m.eval()
correct = total = 0
for i, b in enumerate(loader):
if n is not None and i >= n:
break
x, y = batch_to_device(b)
pred = m(x).argmax(-1)
correct += int((pred == y).sum())
total += y.numel()
return correct / max(total, 1)
# ---------------------------
# SVD utilities (extended for multi-layer)
# ---------------------------
def effrank(S):
q = S.float().pow(2)
p = q / (q.sum() + cfg.eps)
return float(torch.exp(-(p * (p + cfg.eps).log()).sum()))
def get_linear_layers(m):
"""Return list of (layer_idx, linear_module) for all FFN output projections."""
layers = []
for i, block in enumerate(m.blocks.layers):
# Each block has linear2 (the FFN output projection)
layers.append((i, block.linear2))
return layers
def svd_diag(linear_layer, rank=16):
W = linear_layer.weight.detach().float().cpu()
U, S, Vh = torch.linalg.svd(W, full_matrices=False)
k = min(rank, U.shape[1])
return {
'U': U[:, :k].to(device),
'S': S,
'effective_rank': effrank(S),
'topk_energy': float((S[:k]**2).sum() / (S.pow(2).sum() + cfg.eps))
}
def grad_geometry(linear_layer, U, batch):
"""Compute outside pressure for a given layer and U."""
linear_layer.zero_grad(set_to_none=True)
L = loss_batch_for_layer(linear_layer, batch) # we need to compute loss through the whole model
L.backward()
G = linear_layer.weight.grad.detach().float()
GGt = G @ G.T
total = torch.trace(GGt).clamp_min(cfg.eps)
P = U @ U.T
inside = torch.trace(P @ GGt).clamp_min(0)
outside = (total - inside).clamp_min(0)
linear_layer.zero_grad(set_to_none=True)
return {
'grad_inside': float(inside / total),
'grad_outside': float(outside / total),
'G': G,
'loss': float(L)
}
# We need a wrapper to compute loss for a given linear layer modification.
# We'll do it by cloning the model temporarily? For efficiency, we'll compute
# gradient for the whole model and extract the gradient of the target layer.
# This is simpler and more correct.
def compute_layer_gradients(m, batch, target_layers):
"""Compute gradients for specified linear layers and return outside pressures."""
m.zero_grad(set_to_none=True)
L = loss_batch(m, batch)
L.backward()
results = {}
for idx, lin in target_layers:
if lin.weight.grad is None:
results[idx] = None
continue
G = lin.weight.grad.detach().float()
# Compute SVD of weight for U
W = lin.weight.detach().float().cpu()
U_full, S_full, _ = torch.linalg.svd(W, full_matrices=False)
k = min(cfg.svd_rank, U_full.shape[1])
U = U_full[:, :k].to(device)
GGt = G @ G.T
total = torch.trace(GGt).clamp_min(cfg.eps)
P = U @ U.T
inside = torch.trace(P @ GGt).clamp_min(0)
outside = (total - inside).clamp_min(0)
results[idx] = {
'outside_pressure': float(outside / total),
'G': G,
'U': U,
'effective_rank': effrank(S_full),
}
m.zero_grad(set_to_none=True)
return results
# ---------------------------
# Growth modules (enhanced)
# ---------------------------
class LowRankGrowth(nn.Module):
def __init__(self, in_dim, out_dim, rank, U_init=None, scale=0.3):
super().__init__()
self.down = nn.Linear(in_dim, rank, bias=False)
self.up = nn.Linear(rank, out_dim, bias=False)
nn.init.normal_(self.down.weight, std=0.02)
with torch.no_grad():
if U_init is None:
Q = torch.linalg.qr(torch.randn(out_dim, rank, device=device))[0]
self.up.weight.data = Q * scale
else:
Q, _ = torch.linalg.qr(U_init, mode='reduced')
self.up.weight.data = Q * scale
def forward(self, x):
return self.up(self.down(x))
class GrowingLinear(nn.Module):
def __init__(self, base, growth):
super().__init__()
self.base = base
self.growth = growth
@property
def weight(self):
return self.base.weight
@property
def bias(self):
return self.base.bias
def forward(self, x):
return self.base(x) + self.growth(x)
def make_growth_at_layer(m, layer_idx, growth_type, U_init=None, rank=16, scale=0.3):
"""
Replace the linear2 of the specified layer with a GrowingLinear.
growth_type: 'random', 'svd', 'gradient', 'oracle'
For 'oracle', we also unfreeze the base layer (allow full finetuning).
"""
block = m.blocks.layers[layer_idx]
old = block.linear2
if growth_type == 'oracle':
# Oracle: we will later unfreeze the entire base as well, but for now just create a large growth branch
growth = LowRankGrowth(cfg.d_ff, cfg.d_model, rank=rank, U_init=None, scale=scale)
else:
growth = LowRankGrowth(cfg.d_ff, cfg.d_model, rank=rank, U_init=U_init, scale=scale)
block.linear2 = GrowingLinear(old, growth).to(device)
return m
def freeze_all_except_growth(m, layer_idx, growth_type):
"""Freeze all parameters except the growth branch (and optionally base for oracle)."""
for p in m.parameters():
p.requires_grad = False
# Always unfreeze growth branch
growth = m.blocks.layers[layer_idx].linear2.growth
growth.down.weight.requires_grad = True
growth.up.weight.requires_grad = True
if growth_type == 'oracle':
# Unfreeze the base linear layer as well
base = m.blocks.layers[layer_idx].linear2.base
base.weight.requires_grad = True
if base.bias is not None:
base.bias.requires_grad = True
# Also unfreeze the layer norm and maybe other components? We'll keep it simple.
# ---------------------------
# Training routines
# ---------------------------
def train_until_convergence(m, loader, eval_loader, max_steps, target_acc, lr, weight_decay=0.0):
m.train()
opt = torch.optim.AdamW([p for p in m.parameters() if p.requires_grad], lr=lr, weight_decay=weight_decay)
it = iter(loader)
step = 0
while step < max_steps:
try:
b = next(it)
except StopIteration:
it = iter(loader)
b = next(it)
opt.zero_grad(set_to_none=True)
L = loss_batch(m, b)
L.backward()
torch.nn.utils.clip_grad_norm_([p for p in m.parameters() if p.requires_grad and p.grad is not None], 1.0)
opt.step()
step += 1
if step % 200 == 0 or step == max_steps:
acc = accuracy(m, eval_loader, cfg.eval_batches)
if acc >= target_acc:
print(f"Converged at step {step}, acc={acc:.4f}")
return step
print(f"Reached max steps {max_steps}, acc={accuracy(m, eval_loader, cfg.eval_batches):.4f}")
return step
def train_growth(m, loader, steps, lr):
m.train()
params = [p for p in m.parameters() if p.requires_grad]
if not params:
print("Warning: No trainable parameters!")
return m
opt = torch.optim.AdamW(params, lr=lr, weight_decay=0.0)
it = iter(loader)
for _ in range(steps):
try:
b = next(it)
except StopIteration:
it = iter(loader)
b = next(it)
opt.zero_grad(set_to_none=True)
L = loss_batch(m, b)
L.backward()
torch.nn.utils.clip_grad_norm_(params, 1.0)
opt.step()
return m
# ---------------------------
# Single seed run
# ---------------------------
def run(seed):
print(f"\n{'='*70}\nSEED {seed}\n{'='*70}")
seed_everything(seed)
A, B, eA, eB = loaders()
m = TinyTransformer().to(device)
# Phase A training
steps_taken = train_until_convergence(
m, A, eA,
max_steps=cfg.phase_a_max_steps,
target_acc=cfg.phase_a_target_acc,
lr=cfg.lr,
weight_decay=cfg.weight_decay
)
state = copy.deepcopy(m.state_dict())
# Baseline evaluation
base = {
'A_loss': evaluate(m, eA, cfg.eval_batches),
'A_acc': accuracy(m, eA, cfg.eval_batches),
'B_loss': evaluate(m, eB, cfg.eval_batches),
'B_acc': accuracy(m, eB, cfg.eval_batches)
}
print("Phase A converged:", base)
# Multi-layer diagnosis
batch_B = next(iter(B))
layer_info = [(i, block.linear2) for i, block in enumerate(m.blocks.layers)]
grad_info = compute_layer_gradients(m, batch_B, layer_info)
# Find layer with highest outside pressure
best_layer = max(grad_info.items(), key=lambda x: x[1]['outside_pressure'] if x[1] is not None else -1)
layer_idx = best_layer[0]
outside_pressure = best_layer[1]['outside_pressure']
G = best_layer[1]['G']
U_old = best_layer[1]['U']
eff_rank = best_layer[1]['effective_rank']
print(f"Best layer: {layer_idx}, outside_pressure: {outside_pressure:.4f}, eff_rank: {eff_rank:.2f}")
# Prepare growth directions
# SVD-guided: use U_old itself (top-k left singular vectors)
# Gradient-guided: use projected gradient directions
P_perp = torch.eye(U_old.shape[0], device=device) - U_old @ U_old.T
G_proj = P_perp @ G
U_grad, S_grad, _ = torch.linalg.svd(G_proj, full_matrices=False)
rank_g = min(cfg.growth_rank, U_grad.shape[1])
U_grad_init = U_grad[:, :rank_g]
# Build models
# No growth
ng = TinyTransformer().to(device)
ng.load_state_dict(state)
# Random growth
rg = TinyTransformer().to(device)
rg.load_state_dict(state)
rg = make_growth_at_layer(rg, layer_idx, 'random', U_init=None, rank=cfg.growth_rank, scale=cfg.growth_scale)
freeze_all_except_growth(rg, layer_idx, 'random')
# SVD-guided growth
sg = TinyTransformer().to(device)
sg.load_state_dict(state)
sg = make_growth_at_layer(sg, layer_idx, 'svd', U_init=U_old[:, :cfg.growth_rank], rank=cfg.growth_rank, scale=cfg.growth_scale)
freeze_all_except_growth(sg, layer_idx, 'svd')
# Gradient-guided growth
gg = TinyTransformer().to(device)
gg.load_state_dict(state)
gg = make_growth_at_layer(gg, layer_idx, 'gradient', U_init=U_grad_init, rank=cfg.growth_rank, scale=cfg.growth_scale)
freeze_all_except_growth(gg, layer_idx, 'gradient')
# Oracle growth: unfreeze base + large growth
og = TinyTransformer().to(device)
og.load_state_dict(state)
og = make_growth_at_layer(og, layer_idx, 'oracle', U_init=None, rank=cfg.growth_rank*2, scale=cfg.growth_scale)
freeze_all_except_growth(og, layer_idx, 'oracle')
added_params = count_params(rg) - count_params(ng)
print(f"Added parameters per growth: {added_params}")
# Train all growth models on Task B
print("Training no growth (baseline)...")
# no growth is already trained on A, we don't train it further on B
print("Training random growth...")
rg = train_growth(rg, B, cfg.growth_steps, cfg.growth_lr)
print("Training SVD-guided growth...")
sg = train_growth(sg, B, cfg.growth_steps, cfg.growth_lr)
print("Training gradient-guided growth...")
gg = train_growth(gg, B, cfg.growth_steps, cfg.growth_lr)
print("Training oracle growth...")
og = train_growth(og, B, cfg.growth_steps, cfg.growth_lr)
# Final evaluation
results = {}
models = [('no_growth', ng), ('random_growth', rg), ('svd_growth', sg),
('gradient_growth', gg), ('oracle_growth', og)]
for name, model in models:
results[name] = {
'A_loss': evaluate(model, eA, cfg.eval_batches),
'A_acc': accuracy(model, eA, cfg.eval_batches),
'B_loss': evaluate(model, eB, cfg.eval_batches),
'B_acc': accuracy(model, eB, cfg.eval_batches),
}
print("Results:")
for name, res in results.items():
print(f" {name}: B_loss={res['B_loss']:.4f}, B_acc={res['B_acc']:.4f}")
baseB = results['no_growth']['B_loss']
gains = {
'random_gain': baseB - results['random_growth']['B_loss'],
'svd_gain': baseB - results['svd_growth']['B_loss'],
'gradient_gain': baseB - results['gradient_growth']['B_loss'],
'oracle_gain': baseB - results['oracle_growth']['B_loss'],
}
print(f"Gains: {gains}")
return {
'seed': seed,
'phase_A_steps': steps_taken,
'phase_A': base,
'best_layer': layer_idx,
'outside_pressure': outside_pressure,
'effective_rank': eff_rank,
'results': results,
'gains': gains,
'added_params': added_params,
}
# ============================================================
# MAIN
# ============================================================
runs = [run(s) for s in SEEDS]
print("\n" + "="*70)
print("MULTI-SEED SUMMARY")
print("="*70)
# Extract metrics
rg = [r['gains']['random_gain'] for r in runs]
sg = [r['gains']['svd_gain'] for r in runs]
gg = [r['gains']['gradient_gain'] for r in runs]
og = [r['gains']['oracle_gain'] for r in runs]
op = [r['outside_pressure'] for r in runs]
for name, arr in [('Random gain', rg), ('SVD gain', sg), ('Gradient gain', gg), ('Oracle gain', og), ('Outside pressure', op)]:
mean = np.mean(arr)
std = np.std(arr, ddof=1)
print(f"{name:20s} mean={mean:.6f} std={std:.6f}")
print(f"Oracle > gradient: {sum(o > g for o, g in zip(og, gg))} / {len(runs)}")
print(f"Gradient > random: {sum(g > r for g, r in zip(gg, rg))} / {len(runs)}")
if len(runs) > 1:
corr = np.corrcoef(op, gg)[0, 1]
print(f"corr(outside_pressure, gradient_gain): {corr:.4f}")
# Save
out_dir = './smart_density_v5'
os.makedirs(out_dir, exist_ok=True)
summary = {
'config': asdict(cfg),
'seeds': SEEDS,
'device': str(device),
'runs': runs,
}
with open(os.path.join(out_dir, 'results.json'), 'w') as f:
json.dump(summary, f, indent=2)
print(f"Saved to {os.path.abspath(out_dir)}/results.json")
try:
import matplotlib.pyplot as plt
x = np.arange(len(SEEDS))
w = 0.15
plt.figure(figsize=(10,5))
plt.bar(x - 1.5*w, rg, w, label='Random')
plt.bar(x - 0.5*w, sg, w, label='SVD')
plt.bar(x + 0.5*w, gg, w, label='Gradient')
plt.bar(x + 1.5*w, og, w, label='Oracle')
plt.xticks(x, [str(s) for s in SEEDS])
plt.ylabel('Task-B loss reduction')
plt.title('Structural Growth Comparison (v5)')
plt.legend()
plt.tight_layout()
plt.show()
except Exception as e:
print('Plot skipped:', repr(e))yTorch: 2.11.0+cu128
Device: cuda
GPU: Tesla T4
======================================================================
SEED 42
======================================================================
/tmp/ipykernel_1842/2151775399.py:132: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True
self.blocks = nn.TransformerEncoder(layer, cfg.n_layers)
Converged at step 3400, acc=0.9017
Phase A converged: {'A_loss': 0.3280856553465128, 'A_acc': 0.9017333984375, 'B_loss': 15.854797780513763, 'B_acc': 0.0667724609375}
Best layer: 3, outside_pressure: 0.4445, eff_rank: 36.47
/tmp/ipykernel_1842/2151775399.py:132: UserWarning: enable_nested_tensor is True, but self.use_nested_tensor is False because encoder_layer.norm_first was True
self.blocks = nn.TransformerEncoder(layer, cfg.n_layers)
Added parameters per growth: 6144
Training no growth (baseline)...
Training random growth...
Training SVD-guided growth...
Training gradient-guided growth...
Training oracle growth...
Results:
no_growth: B_loss=15.8548, B_acc=0.0668
random_growth: B_loss=15.8548, B_acc=0.0668
svd_growth: B_loss=15.8548, B_acc=0.0668
gradient_growth: B_loss=15.8548, B_acc=0.0668
oracle_growth: B_loss=3.9925, B_acc=0.0672
Gains: {'random_gain': 0.0, 'svd_gain': 0.0, 'gradient_gain': 0.0, 'oracle_gain': 11.862322941422462}
======================================================================
SEED 123
======================================================================
Converged at step 6800, acc=0.9725
Phase A converged: {'A_loss': 0.10075912298634648, 'A_acc': 0.97247314453125, 'B_loss': 19.90610122680664, 'B_acc': 0.066925048828125}
Best layer: 3, outside_pressure: 0.6300, eff_rank: 27.93
Added parameters per growth: 6144
Training no growth (baseline)...
Training random growth...
Training SVD-guided growth...
Training gradient-guided growth...
Training oracle growth...
Results:
no_growth: B_loss=19.9061, B_acc=0.0669
random_growth: B_loss=19.9061, B_acc=0.0669
svd_growth: B_loss=19.9061, B_acc=0.0669
gradient_growth: B_loss=19.9061, B_acc=0.0669
oracle_growth: B_loss=4.2520, B_acc=0.0675
Gains: {'random_gain': 0.0, 'svd_gain': 0.0, 'gradient_gain': 0.0, 'oracle_gain': 15.654108047485352}
======================================================================
SEED 2026
======================================================================
Converged at step 6400, acc=0.9207
Phase A converged: {'A_loss': 0.20898992102593184, 'A_acc': 0.920654296875, 'B_loss': 16.68497395515442, 'B_acc': 0.06683349609375}
Best layer: 3, outside_pressure: 0.7145, eff_rank: 24.64
Added parameters per growth: 6144
Training no growth (baseline)...
Training random growth...
Training SVD-guided growth...
Training gradient-guided growth...
Training oracle growth...
Results:
no_growth: B_loss=16.6850, B_acc=0.0668
random_growth: B_loss=16.6850, B_acc=0.0668
svd_growth: B_loss=16.6850, B_acc=0.0668
gradient_growth: B_loss=16.6850, B_acc=0.0668
oracle_growth: B_loss=4.0462, B_acc=0.0674
Gains: {'random_gain': 0.0, 'svd_gain': 0.0, 'gradient_gain': 0.0, 'oracle_gain': 12.638744369149208}
======================================================================
MULTI-SEED SUMMARY
======================================================================
Random gain mean=0.000000 std=0.000000
SVD gain mean=0.000000 std=0.000000
Gradient gain mean=0.000000 std=0.000000
Oracle gain mean=13.385058 std=2.003034
Outside pressure mean=0.596325 std=0.138112
Oracle > gradient: 3 / 3
Gradient > random: 0 / 3
corr(outside_pressure, gradient_gain): nan
Saved to /content/smart_density_v5/results.json
/usr/local/lib/python3.13/dist-packages/numpy/lib/_function_base_impl.py:2999: RuntimeWarning: invalid value encountered in divide
c /= stddev[:, None]
/usr/local/lib/python3.13/dist-packages/numpy/lib/_function_base_impl.py:3000: RuntimeWarning: invalid value encountered in divide
c /= stddev[None, :]