APEX
如何安装
git clone https://github.com/NVIDIA/apex.git
cd apex
pip3 install --no-cache-dir --global-option="--pyprof" --global-option="--cpp_ext" --global-option="--cuda_ext"
google colab install apex amp
try:
import apex
except Exception:
! git clone https://github.com/NVIDIA/apex.git
% cd apex
!pip install --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" .
% cd ..
code
import apex # OK
from apex import amp # error
error
ImportError: cannot import name 'amp' from 'apex'
add --user
!pip install --no-cache-dir --global-option="--pyprof" --global-option="--cpp_ext" --global-option="--cuda_ext"
still wrong......
如何使用
1) 训练、保存模型
2) 测试
Pytorch使用
1) 训练
from torch.cuda.amp import autocast as autocast
# 创建model,默认是torch.FloatTensor
model = Net().cuda()
optimizer = optim.SGD(model.parameters(), ...)
# 在训练最开始之前实例化一个GradScaler对象
scaler = GradScaler()
for epoch in epochs:
for input, target in data:
optimizer.zero_grad()
# 前向过程(model + loss)开启 autocast
with autocast():
output = model(input)
loss = loss_fn(output, target)
# Scales loss. 为了梯度放大.
scaler.scale(loss).backward()
# scaler.step() 首先把梯度的值unscale回来.
# 如果梯度的值不是 infs 或者 NaNs, 那么调用optimizer.step()来更新权重,
# 否则,忽略step调用,从而保证权重不更新(不被破坏)
scaler.step(optimizer)
# 准备着,看是否要增大scaler
View Code
2) 测试
参考
1. 【PyTorch】唯快不破:基于Apex的混合精度加速;
https://zhuanlan.zhihu.com/p/79887894
判断你的GPU是否支持FP16:支持的有拥有Tensor Core的GPU(2080Ti、Titan、Tesla等),不支持的(Pascal系列,1080Ti)就不建议折腾了。
https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html#training_pytorch
https://pytorch.org/docs/stable/notes/amp_examples.html
如何使用 PyTorch 进行半精度训练
https://featurize.cn/notebooks/368cbc81-2b27-4036-98a1-d77589b1f0c4
【深度学习训练小技巧】1080ti与2080ti区别、apex与梯度累加、torch.no_grad
https://zhuanlan.zhihu.com/p/150860679
MIXED PRECISION TRAINING
https://on-demand.gputechconf.com/gtc-taiwan/2018/pdf/5-1_Internal%20Speaker_Michael%20Carilli_PDF%20For%20Sharing.pdf
https://github.com/NVIDIA/apex/blob/master/apex/amp/lists/functional_overrides.py
https://github.com/NVIDIA/apex
https://nvidia.github.io/apex/index.html
PyTorch的自动混合精度(AMP)
https://colab.research.google.com/github/pytorchbearer/torchbearer/blob/master/docs/_static/notebooks/apex_torchbearer.ipynb#scrollTo=kaCrlsfk-UDw
https://developer.nvidia.com/automatic-mixed-precision
完