企业官网建设流程全解析

【Bug已解决】Pytorch: Weight in cross entropy loss 解决方案

问题描述

在 PyTorch 中使用交叉熵损失函数（CrossEntropyLoss）时，weight参数是一个强大但常被误解的功能。它用于处理类别不平衡问题——当某些类别的样本数量远少于其他类别时，模型倾向于偏向多数类，导致少数类的识别性能很差。

weight参数允许你为每个类别分配不同的权重，使得少数类的错误获得更大的惩罚。然而，开发者在使用weight时经常遇到以下问题：

1. 权重维度不匹配：不知道weight应该是一维张量，长度等于类别数。
2. 权重归一化困惑：不清楚是否需要手动归一化权重，以及 PyTorch 内部如何处理。
3. 与reduction参数的交互：使用weight后，mean和sum的行为发生变化。
4. 梯度缩放问题：权重导致梯度幅度变化，影响训练稳定性。
5. 多标签分类误用：将weight用于多标签分类场景（应该用BCEWithLogitsLoss）。

典型场景包括：医学图像分割中前景/背景极度不平衡、欺诈检测中正常/异常样本比例悬殊、罕见疾病分类、文本分类中的长尾分布。

错误复现

import torch import torch.nn as nn import torch.nn.functional as F num_classes = 5 batch_size = 8 # 模拟模型输出和标签 logits = torch.randn(batch_size, num_classes) labels = torch.randint(0, num_classes, (batch_size,)) # ============================== # 错误1: weight 维度错误 # ============================== try: # 错误: 给了标量权重 criterion = nn.CrossEntropyLoss(weight=torch.tensor(2.0)) loss = criterion(logits, labels) except Exception as e: print(f"错误1: {e}") # RuntimeError: weight tensor should be defined as a 1D tensor # ============================== # 错误2: weight 长度不匹配 # ============================== try: # 错误: weight 长度不等于类别数 criterion = nn.CrossEntropyLoss(weight=torch.tensor([1.0, 2.0, 3.0])) # 只有3个，需要5个 loss = criterion(logits, labels) except Exception as e: print(f"错误2: {e}") # IndexError 或 RuntimeError # ============================== # 错误3: weight dtype 错误 # ============================== try: # 错误: 使用整数类型 criterion = nn.CrossEntropyLoss(weight=torch.tensor([1, 2, 3, 4, 5], dtype=torch.long)) loss = criterion(logits, labels) except Exception as e: print(f"错误3: {e}") # 可能不报错但行为不正确，或 dtype 警告 # ============================== # 错误4: 不理解 reduction='mean' 与 weight 的交互 # ============================== # 当使用 weight 且 reduction='mean' 时，损失是加权平均 # 而不是简单的平均 weights = torch.tensor([1.0, 1.0, 1.0, 1.0, 10.0]) # 类别4权重很高 criterion_mean = nn.CrossEntropyLoss(weight=weights, reduction='mean') criterion_sum = nn.CrossEntropyLoss(weight=weights, reduction='sum') loss_mean = criterion_mean(logits, labels) loss_sum = criterion_sum(logits, labels) # 手动计算验证 manual_loss = F.cross_entropy(logits, labels, weight=weights, reduction='sum') manual_mean = manual_loss / weights[labels].sum() print(f"\nloss_mean: {loss_mean.item():.4f}") print(f"manual_mean: {manual_mean.item():.4f}") print(f"loss_sum: {loss_sum.item():.4f}") print(f"manual_loss: {manual_loss.item():.4f}") # loss_mean != loss_sum / batch_size (关键区别!)

根因分析

1. CrossEntropyLoss 的内部计算

PyTorch 的CrossEntropyLoss内部结合了LogSoftmax和NLLLoss。带权重的计算公式为：

loss(x, target) = weight[target] * (-log(softmax(x)[target]))

当reduction='mean'时：

loss = sum(weight[target_i] * (-log(softmax(x_i)[target_i]))) / sum(weight[target_i])

当reduction='sum'时：

loss = sum(weight[target_i] * (-log(softmax(x_i)[target_i])))

关键区别：带权重时，reduction='mean'的分母是sum(weight[target_i])，而不是batch_size。这意味着权重会归一化，损失值不会因为权重大小而无限增大。

2. 权重归一化的内部处理

PyTorch 内部不会自动归一化权重，但通过reduction='mean'的除法实现了等效的归一化效果：

# 假设权重为 [1, 1, 1, 1, 10] # 如果 batch 中有 2 个类别4的样本和 6 个其他类别的样本 # reduction='mean' 的分母 = 6*1 + 2*10 = 26 # 而不是 batch_size = 8 # 这意味着即使权重很大，损失值也不会爆炸 # 但梯度方向会偏向高权重类别

3. 权重选择策略

常见的权重选择方法：

# 方法1: 逆频率 # 如果类别0有1000个样本，类别1有100个样本 # weight = [100/1100, 1000/1100] 或 [1/1000, 1/100] # 方法2: 有效样本数 (Effective Number of Samples) # beta = 0.999 # effective_num = 1.0 - beta**samples_per_class # weights = (1.0 - beta) / effective_num # 方法3: 中位数频率 # median_freq = median(samples_per_class) # weights = median_freq / samples_per_class # 方法4: 自定义手动调整 weights = torch.tensor([1.0, 5.0, 10.0, 2.0, 8.0])

4. 常见误解

# 误解1: 认为 weight 需要归一化到 [0, 1] # 实际上不需要，PyTorch 的 reduction='mean' 会自动处理 # 误解2: 认为 weight 是每个样本的权重 # 实际上 weight 是每个类别的权重 # 误解3: 认为 reduction='mean' 的分母是 batch_size # 实际上分母是 sum(weight[target_i]) # 误解4: 在多标签分类中使用 CrossEntropyLoss + weight # 多标签应该用 BCEWithLogitsLoss + pos_weight

解决方案

方案一：逆频率加权（推荐）

根据训练集中各类别的样本数量计算权重：

import torch import torch.nn as nn from collections import Counter # 假设这是训练集中所有标签 all_labels = [0] * 1000 + [1] * 500 + [2] * 100 + [3] * 50 + [4] * 10 label_counts = Counter(all_labels) num_classes = 5 # 计算逆频率权重 total_samples = len(all_labels) class_weights = [] for c in range(num_classes): count = label_counts.get(c, 1) weight = total_samples / (num_classes * count) class_weights.append(weight) weights = torch.tensor(class_weights, dtype=torch.float32) print(f"类别权重: {weights}") # tensor([0.316, 0.632, 3.16, 6.32, 31.6]) criterion = nn.CrossEntropyLoss(weight=weights)

方案二：有效样本数加权（高级）

import numpy as np # 有效样本数方法 (Cui et al., 2019) samples_per_class = np.array([1000, 500, 100, 50, 10]) beta = 0.999 effective_num = 1.0 - np.power(beta, samples_per_class) weights = (1.0 - beta) / np.array(effective_num) weights = torch.tensor(weights, dtype=torch.float32) weights = weights / weights.sum() * num_classes # 归一化使均值为1 print(f"有效样本数权重: {weights}") criterion = nn.CrossEntropyLoss(weight=weights)

方案三：手动指定权重

# 根据领域知识手动设置 weights = torch.tensor([1.0, 2.0, 5.0, 10.0, 20.0], dtype=torch.float32) criterion = nn.CrossEntropyLoss(weight=weights) # 如果使用 GPU device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') weights = weights.to(device) criterion = nn.CrossEntropyLoss(weight=weights)

方案四：动态权重调整

在训练过程中根据验证集表现动态调整权重：

class DynamicWeightedCELoss: """动态调整权重的交叉熵损失""" def __init__(self, num_classes, initial_weights=None, device='cpu'): self.num_classes = num_classes if initial_weights is not None: self.weights = torch.tensor(initial_weights, dtype=torch.float32, device=device) else: self.weights = torch.ones(num_classes, dtype=torch.float32, device=device) self.device = device def update_weights(self, class_accuracies, smoothing=0.1): """根据类别准确率更新权重""" # 准确率低的类别获得更高权重 acc_tensor = torch.tensor(class_accuracies, dtype=torch.float32, device=self.device) # 使用 (1 - acc) 作为权重，加上平滑项避免极端值 new_weights = (1.0 - acc_tensor + smoothing) * self.num_classes # 指数移动平均 self.weights = 0.9 * self.weights + 0.1 * new_weights print(f"更新权重: {self.weights}") def __call__(self, logits, targets): return F.cross_entropy(logits, targets, weight=self.weights) # 使用示例 dynamic_loss = DynamicWeightedCELoss(num_classes=5, initial_weights=[1, 1, 1, 1, 1]) # 训练过程中根据验证结果更新 # dynamic_loss.update_weights([0.95, 0.90, 0.60, 0.30, 0.10])

完整修复代码

import torch import torch.nn as nn import torch.nn.functional as F import numpy as np from collections import Counter from sklearn.metrics import classification_report def demonstrate_weighted_cross_entropy(): """完整演示带权重的交叉熵损失""" print("=" * 60) print("CrossEntropyLoss weight 参数完整示例") print("=" * 60) num_classes = 5 batch_size = 16 device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') # ============================== # 1. 模拟不平衡数据 # ============================== print("\n--- 1. 模拟不平衡数据 ---") # 类别分布: 类别0最多，类别4最少 class_distribution = {0: 1000, 1: 500, 2: 200, 3: 50, 4: 10} total = sum(class_distribution.values()) print("类别分布:") for c, count in sorted(class_distribution.items()): print(f" 类别{c}: {count} 样本 ({count/total*100:.1f}%)") # ============================== # 2. 计算权重 # ============================== print("\n--- 2. 计算权重 ---") # 方法A: 逆频率 weights_inv_freq = [] for c in range(num_classes): count = class_distribution.get(c, 1) weight = total / (num_classes * count) weights_inv_freq.append(weight) weights_inv_freq = torch.tensor(weights_inv_freq, dtype=torch.float32, device=device) print(f"逆频率权重: {weights_inv_freq}") # 方法B: 有效样本数 samples = np.array([class_distribution[c] for c in range(num_classes)]) beta = 0.999 effective_num = 1.0 - np.power(beta, samples) weights_ens = (1.0 - beta) / effective_num weights_ens = torch.tensor(weights_ens, dtype=torch.float32, device=device) weights_ens = weights_ens / weights_ens.sum() * num_classes print(f"有效样本数权重: {weights_ens}") # 方法C: 均匀权重（无加权） weights_uniform = torch.ones(num_classes, dtype=torch.float32, device=device) # ============================== # 3. 对比不同权重的损失 # ============================== print("\n--- 3. 对比不同权重的损失 ---") torch.manual_seed(42) logits = torch.randn(batch_size, num_classes, device=device) # 创建一个包含稀有类别的标签批次 labels = torch.tensor([0, 0, 0, 0, 1, 1, 1, 2, 2, 3, 0, 1, 4, 0, 1, 2], device=device, dtype=torch.long) # 计算不同权重下的损失 for name, weights in [("无权重", weights_uniform), ("逆频率", weights_inv_freq), ("有效样本数", weights_ens)]: criterion = nn.CrossEntropyLoss(weight=weights, reduction='mean') loss = criterion(logits, labels) # 也计算 sum 版本 criterion_sum = nn.CrossEntropyLoss(weight=weights, reduction='sum') loss_sum = criterion_sum(logits, labels) # 手动验证 mean 的分母 batch_weights = weights[labels] manual_mean = loss_sum / batch_weights.sum() print(f"\n{name}:") print(f" loss(mean): {loss.item():.4f}") print(f" loss(sum): {loss_sum.item():.4f}") print(f" 手动mean: {manual_mean.item():.4f}") print(f" 权重和: {batch_weights.sum().item():.4f}") print(f" batch_size: {batch_size}") # ============================== # 4. 训练对比实验 # ============================== print("\n--- 4. 训练对比实验 ---") # 创建简单模型 class SimpleModel(nn.Module): def __init__(self, input_dim, num_classes): super().__init__() self.fc = nn.Sequential( nn.Linear(input_dim, 128), nn.ReLU(), nn.Dropout(0.2), nn.Linear(128, 64), nn.ReLU(), nn.Linear(64, num_classes) ) def forward(self, x): return self.fc(x) input_dim = 20 # 生成不平衡训练数据 def generate_imbalanced_data(n_samples, class_probs, input_dim, num_classes): labels = np.random.choice(num_classes, n_samples, p=class_probs) features = np.random.randn(n_samples, input_dim).astype(np.float32) # 让特征与标签有某种关系 for i in range(num_classes): features[labels == i] += i * 0.5 return torch.tensor(features), torch.tensor(labels, dtype=torch.long) class_probs = np.array([1000, 500, 200, 50, 10], dtype=np.float64) class_probs /= class_probs.sum() X_train, y_train = generate_imbalanced_data(2000, class_probs, input_dim, num_classes) X_train, y_train = X_train.to(device), y_train.to(device) # 训练函数 def train_model(weights, name, epochs=50): model = SimpleModel(input_dim, num_classes).to(device) optimizer = torch.optim.Adam(model.parameters(), lr=0.001) criterion = nn.CrossEntropyLoss(weight=weights) for epoch in range(epochs): model.train() optimizer.zero_grad() logits = model(X_train) loss = criterion(logits, y_train) loss.backward() optimizer.step() # 评估 model.eval() with torch.no_grad(): logits = model(X_train) preds = logits.argmax(dim=1) # 计算每个类别的准确率 per_class_acc = [] for c in range(num_classes): mask = y_train == c if mask.sum() > 0: acc = (preds[mask] == c).float().mean().item() else: acc = 0.0 per_class_acc.append(acc) overall_acc = (preds == y_train).float().mean().item() print(f"\n{name}:") print(f" 总体准确率: {overall_acc:.4f}") for c in range(num_classes): print(f" 类别{c} 准确率: {per_class_acc[c]:.4f}") return per_class_acc print("训练无权重模型:") acc_no_weight = train_model(weights_uniform, "无权重") print("\n训练逆频率权重模型:") acc_inv_freq = train_model(weights_inv_freq, "逆频率权重") print("\n训练有效样本数权重模型:") acc_ens = train_model(weights_ens, "有效样本数权重") # ============================== # 5. reduction 参数详解 # ============================== print("\n--- 5. reduction 参数详解 ---") weights = torch.tensor([1.0, 2.0, 3.0, 4.0, 5.0], device=device) logits = torch.randn(4, 5, device=device) labels = torch.tensor([0, 2, 4, 1], device=device) # reduction='none': 返回每个样本的损失 loss_none = F.cross_entropy(logits, labels, weight=weights, reduction='none') print(f"reduction='none': {loss_none}") # reduction='sum': 加权求和 loss_sum = F.cross_entropy(logits, labels, weight=weights, reduction='sum') print(f"reduction='sum': {loss_sum.item():.4f}") print(f" 验证 (none求和): {loss_none.sum().item():.4f}") # reduction='mean': 加权平均 (除以权重和，不是batch_size) loss_mean = F.cross_entropy(logits, labels, weight=weights, reduction='mean') batch_weight_sum = weights[labels].sum() print(f"reduction='mean': {loss_mean.item():.4f}") print(f" 验证 (sum/权重和): {(loss_sum / batch_weight_sum).item():.4f}") print(f" 注意: sum/batch_size = {(loss_sum / 4).item():.4f} (不等于mean!)") # ============================== # 6. label_smoothing 与 weight 结合 # ============================== print("\n--- 6. label_smoothing 与 weight 结合 ---") criterion_smooth = nn.CrossEntropyLoss( weight=weights, label_smoothing=0.1, reduction='mean' ) loss_smooth = criterion_smooth(logits, labels) print(f"带 label_smoothing 的损失: {loss_smooth.item():.4f}") # ============================== # 7. ignore_index 与 weight 结合 # ============================== print("\n--- 7. ignore_index 与 weight 结合 ---") # 添加一个忽略标签 (255) labels_with_ignore = torch.tensor([0, 2, 255, 1], device=device) criterion_ignore = nn.CrossEntropyLoss( weight=weights, ignore_index=255, reduction='mean' ) loss_ignore = criterion_ignore(logits, labels_with_ignore) print(f"带 ignore_index 的损失: {loss_ignore.item():.4f}") print(" (标签255的样本被忽略，不参与损失计算和权重归一化)") # ============================== # 8. 安全使用工具函数 # ============================== print("\n--- 8. 安全使用工具函数 ---") def create_weighted_loss(num_classes, class_counts, method='inv_freq', device='cpu'): """ 创建带权重的交叉熵损失 参数: num_classes: 类别数 class_counts: 每个类别的样本数 (list 或 dict) method: 权重计算方法 ('inv_freq', 'ens', 'median') device: 计算设备 返回: nn.CrossEntropyLoss 实例 """ if isinstance(class_counts, dict): counts = np.array([class_counts.get(c, 1) for c in range(num_classes)], dtype=np.float64) else: counts = np.array(class_counts, dtype=np.float64) if method == 'inv_freq': total = counts.sum() weights = total / (num_classes * counts) elif method == 'ens': beta = 0.999 effective_num = 1.0 - np.power(beta, counts) weights = (1.0 - beta) / effective_num weights = weights / weights.sum() * num_classes elif method == 'median': median = np.median(counts) weights = median / counts else: raise ValueError(f"未知方法: {method}") weights_tensor = torch.tensor(weights, dtype=torch.float32, device=device) print(f" 权重 ({method}): {weights_tensor}") return nn.CrossEntropyLoss(weight=weights_tensor) # 测试 counts = {0: 1000, 1: 500, 2: 200, 3: 50, 4: 10} loss_inv = create_weighted_loss(5, counts, 'inv_freq', device) loss_ens = create_weighted_loss(5, counts, 'ens', device) loss_med = create_weighted_loss(5, counts, 'median', device) print("\n" + "=" * 60) print("所有示例完成!") print("=" * 60) if __name__ == "__main__": demonstrate_weighted_cross_entropy()

常见陷阱与注意事项

1. weight 必须在正确设备上

# 错误: weight 在 CPU，模型输出在 GPU weights = torch.tensor([1.0, 2.0, 3.0]) # CPU logits = model(input) # GPU # F.cross_entropy(logits, labels, weight=weights) # 设备不匹配错误! # 正确: 将 weight 移到正确设备 weights = weights.to(logits.device)

2. reduction='mean' 的分母不是 batch_size

# 当使用 weight 时: # reduction='mean' 的分母 = sum(weight[target_i]) # 而不是 batch_size # 这意味着: # 1. 如果所有权重都是1，则等价于除以 batch_size # 2. 如果权重不同，分母会变化 # 3. 不能简单地将 'sum' 结果除以 batch_size 来得到 'mean'

3. 不要混淆样本权重和类别权重

# CrossEntropyLoss 的 weight 是类别权重 (per-class) # 如果需要样本权重 (per-sample)，需要手动实现: def weighted_cross_entropy(logits, labels, class_weights, sample_weights): """带类别权重和样本权重的交叉熵""" # 先计算每个样本的损失 (无 reduction) losses = F.cross_entropy(logits, labels, weight=class_weights, reduction='none') # 再用样本权重加权 return (losses * sample_weights).sum() / sample_weights.sum()

4. 多标签分类不要用 CrossEntropyLoss

# 多标签分类 (每个样本可以属于多个类别) 应该用 BCEWithLogitsLoss # 而不是 CrossEntropyLoss # 错误: 多标签用 CrossEntropyLoss # criterion = nn.CrossEntropyLoss(weight=weights) # 正确: 多标签用 BCEWithLogitsLoss + pos_weight pos_weight = torch.tensor([0.1, 0.5, 2.0, 5.0, 20.0]) # 逆频率 criterion = nn.BCEWithLogitsLoss(pos_weight=pos_weight)

5. 权重过大导致梯度爆炸

# 如果权重差异过大 (如 [1, 1000])，可能导致梯度爆炸 # 解决方案: 限制权重范围或使用梯度裁剪 weights = torch.tensor([1.0, 1000.0]) criterion = nn.CrossEntropyLoss(weight=weights) # 训练时使用梯度裁剪 torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

6. label_smoothing 与 weight 的交互

# label_smoothing 会改变目标分布，与 weight 一起使用时 # 权重仍然只应用于真实类别 criterion = nn.CrossEntropyLoss( weight=weights, label_smoothing=0.1 ) # 损失计算中，weight 只乘以真实类别的 -log(p) 部分

7. 验证集上不要使用 weight

# 训练时使用 weight 处理不平衡 # 但验证/测试时应该使用无权重的指标来反映真实表现 # 训练 train_criterion = nn.CrossEntropyLoss(weight=class_weights) # 验证 (不使用 weight) val_criterion = nn.CrossEntropyLoss() # 无权重 # 或直接计算准确率、F1 等指标

8. 权重的梯度传播

# weight 参数不参与梯度计算 (requires_grad=False) # 如果需要可学习的权重，需要自定义实现 # 标准用法: weight 是固定的 weights = torch.tensor([1.0, 2.0, 3.0], requires_grad=False) criterion = nn.CrossEntropyLoss(weight=weights) # 可学习权重 (高级用法，需要自定义) class LearnableWeightedLoss(nn.Module): def __init__(self, num_classes): super().__init__() self.logits_weights = nn.Parameter(torch.zeros(num_classes)) def forward(self, logits, targets): weights = torch.softmax(self.logits_weights, dim=0) * num_classes return F.cross_entropy(logits, targets, weight=weights)

总结

CrossEntropyLoss的weight参数是处理类别不平衡问题的重要工具。核心要点如下：

核心要点：

1. 理解计算公式：带权重时reduction='mean'的分母是sum(weight[target_i])，不是batch_size。
2. 权重是类别级：weight为每个类别分配权重，不是每个样本。
3. 设备一致性：weight张量必须与模型输出在同一设备上。
4. 验证不加权：训练用加权损失，验证用无加权指标反映真实表现。
5. 多标签用 BCE：多标签分类应使用BCEWithLogitsLoss+pos_weight，而非CrossEntropyLoss。
6. 权重范围控制：避免权重差异过大导致梯度爆炸，必要时使用梯度裁剪。
7. 与 label_smoothing 兼容：两者可以同时使用，权重只应用于真实类别部分。

通过正确使用weight参数，你可以有效缓解类别不平衡问题，提升模型在少数类上的识别能力，同时保持整体分类性能的稳定。