onnx标准 & onnxRuntime加速推理引擎
文章目录
参考博客：
- ONNX运行时：跨平台、高性能ML推断和训练加速器
- python关于onnx模型的一些基本操作
- ONNX 與 Azure Machine Learning：建立並加速 ML 模型
- 👋超有用ONNX参考资料整理
- 模型部署之 ONNX ONNXRuntime
参考官方文档：
- onnx training-start
- Get started with ORT for Python
- onnxruntime API
- MPI: Parallel Programming in Python with Message Passing Interface (mpi4py)
一、onnx简介
通常我们在训练模型时可以使用很多不同的框架，比如有的同学喜欢用 Pytorch，有的同学喜欢使用 TensorFLow，也有的喜欢 MXNet，以及深度学习最开始流行的 Caffe等等，这样不同的训练框架就导致了产生不同的模型结果包，在模型进行部署推理时就需要不同的依赖库，而且同一个框架比如tensorflow 不同的版本之间的差异较大， 为了解决这个混乱问题，LF AI 这个组织联合 Facebook, MicroSoft等公司制定了机器学习模型的标准，这个标准叫做ONNX, Open Neural Network Exchage，所有其他框架产生的模型包 (.pth, .pb) 都可以转换成这个标准格式，转换成这个标准格式后，就可以使用统一的 ONNX Runtime等工具进行统一部署。（和Java生成的中间文件可以在JVM上运行一样，onnx runtime引擎为生成的onnx模型文件提供推理功能）
onnx主页: onnxruntime.ai
开发文档和教程: onnxruntime.ai/docs
Companion sample repositories:
- 基于ONNX Runtime 的推理: microsoft/onnxruntime-inference-examples
- 基于ONNX Runtime 的训练: microsoft/onnxruntime-training-examples
onnx支持的算子：https://github.com/onnx/onnx/blob/main/docs/Operators.md
onnx支持哪些平台、为哪些语言提供了API、支持的指令集架构、支持哪些硬件的加速：
ONNX源码阅读
二、pytorch转onnx
参考
- (OPTIONAL) EXPORTING A MODEL FROM PYTORCH TO ONNX AND RUNNING IT USING ONNX RUNTIME
- 将 PyTorch 模型转换为 ONNX
在模型的推理阶段中，通过传入模型和输入的Tensor，利用torch.onnx即可实现pth文件到onnx文件的转化
#Function to Convert to ONNX
def Convert_ONNX(model,input):
# set the model to inference mode
model.eval()
# Let's create a dummy input tensor
dummy_input = input
# Export the model
torch.onnx.export(model, # model being run
dummy_input, # model input (or a tuple for multiple inputs)
"ImageClassifier.onnx", # where to save the model
export_params=True, # store the trained parameter weights inside the model file
opset_version=13, # the ONNX version to export the model to
do_constant_folding=True, # whether to execute constant folding for optimization
input_names = ['modelInput'], # the model's input names
output_names = ['modelOutput'], # the model's output names
dynamic_axes={'modelInput' : {0 : 'batch_size'}, # variable length axes
'modelOutput' : {0 : 'batch_size'}})
print(" ")
print('Model has been converted to ONNX')
if __name__ == '__main__':
checkpoints = torch.load("./pth/mobileNetV3_SMART_CE.pth",map_location="cpu") #全部权重加载到模型中
model.load_state_dict(checkpoints)
# device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
device = torch.device("cpu")
model = model.to(device)
Convert_ONNX(model, sample)
但是在使用MobileNetV3的pth文件转onnx时，存在问题：
RuntimeError: Exporting the operator relu6 to ONNX opset version 11 is not supported. Please feel free to request support or submit a pull request on PyTorch GitHub.
原因是relu6（也就是hardSigmoid）只在MobileNetV3中实现，onnx中不存在该算子，参考https://github.com/pytorch/vision/issues/3463，https://github.com/onnx/onnx/blob/main/docs/Operators.md。
解决方法：
三、tf1.0 / tf2.0 ckpt转onnx
参考将tensorflow 1.x & 2.x转化成onnx文件
四、python onnx的使用
- Get started with ORT for Python
- onnxruntime API
1、环境安装
If using pip, run pip install --upgrade pip prior to downloading.
| Artifact | Description | Supported Platforms |
|---|---|---|
| onnxruntime | CPU (Release 稳定版) | Windows (x64), Linux (x64, ARM64), Mac (X64), |
| ort-nightly | CPU (Dev 测试版) | Same as above |
| onnxruntime-gpu | GPU (Release) | Windows (x64), Linux (x64, ARM64) |
| ort-nightly-gpu | GPU (Dev) | Same as above |
pip命令如下：
pip install onnxruntime
2、获得onnx模型权重参数（可视化）
-
https://netron.app/
-
模型部署之 ONNX ONNXRuntime
这里以SCRFD模型为例
import cv2
import onnx
from onnx import helper
import onnxruntime
import numpy as np
'''一、获取onnx模型的输出层,输出层和模型中的节点数'''
if __name__ == '__main__':
# 参考 <https://blog.csdn.net/CFH1021/article/details/108732114>, https://onnxruntime.ai/docs/get-started/with-python.html
# 加载模型
# model = onnx.load('./weights/mbv2_ID_recognition.onnx') # Load the onnx model with onnx.load
model = onnx.load('./weights/scrfd_500m_kps.onnx')
# model = onnx.load('./weights/mbv3_fire_classifier.onnx')
# 检查模型格式是否完整及正确
onnx.checker.check_model(model)
'''
ref: https://github.com/onnx/onnx/blob/main/docs/IR.md
Graphs have the following properties:
name: string The name of the model graph.
node: Node[] A list of nodes, forming a partially ordered computation graph based on input/output data dependencies. It is in topological order.
initializer: Tensor[] A list of named tensor values. When an initializer has the same name as a graph input, it specifies a default value for that input. When an initializer has a name different from all graph inputs, it specifies a constant value. The order of the list is unspecified.
doc_string: string Human-readable documentation for this model. Markdown is allowed.
input: ValueInfo[] The input parameters of the graph, possibly initialized by a default value found in ‘initializer.’
output: ValueInfo[] The output parameters of the graph. Once all output parameters have been written to by a graph execution, the execution is complete.
value_info: ValueInfo[] Used to store the type and shape information of values that are not inputs or outputs.
'''
input = model.graph.input # 获取输入层，包含层名称、维度信息
output = model.graph.output # 获取输出层，包含层名称、维度信息
depth = len(model.graph.node) # 获取节点数
doc_string = model.graph.doc_string # 获取关于onnx模型的相关文档，是在哪里转换的
print(f"input = {input}")
print(f"output = {output}")
print(f"depth = {depth}")
print(f"doc_string = {doc_string}")
# 参考 https://www.jianshu.com/p/476478c17b8e
# Print a human readable representation of the graph
print(onnx.helper.printable_graph(model.graph))
输出结果：
input = [name: "images"
type {
tensor_type {
elem_type: 1
shape {
dim {
dim_value: 1
}
dim {
dim_value: 3
}
dim {
dim_value: 640
}
dim {
dim_value: 640
}
}
}
}
]
output = [name: "out0"
type {
tensor_type {
elem_type: 1
shape {
dim {
dim_value: 1
}
dim {
dim_value: 12800
}
dim {
dim_value: 1
}
}
}
}
, name: "out1"
type {
tensor_type {
elem_type: 1
shape {
dim {
dim_value: 1
}
dim {
dim_value: 3200
}
dim {
dim_value: 1
}
}
}
}
, name: "out2"
type {
tensor_type {
elem_type: 1
shape {
dim {
dim_value: 1
}
dim {
dim_value: 800
}
dim {
dim_value: 1
}
}
}
}
, name: "out3"
type {
tensor_type {
elem_type: 1
shape {
dim {
dim_value: 1
}
dim {
dim_value: 12800
}
dim {
dim_value: 4
}
}
}
}
, name: "out4"
type {
tensor_type {
elem_type: 1
shape {
dim {
dim_value: 1
}
dim {
dim_value: 3200
}
dim {
dim_value: 4
}
}
}
}
, name: "out5"
type {
tensor_type {
elem_type: 1
shape {
dim {
dim_value: 1
}
dim {
dim_value: 800
}
dim {
dim_value: 4
}
}
}
}
, name: "out6"
type {
tensor_type {
elem_type: 1
shape {
dim {
dim_value: 1
}
dim {
dim_value: 12800
}
dim {
dim_value: 10
}
}
}
}
, name: "out7"
type {
tensor_type {
elem_type: 1
shape {
dim {
dim_value: 1
}
dim {
dim_value: 3200
}
dim {
dim_value: 10
}
}
}
}
, name: "out8"
type {
tensor_type {
elem_type: 1
shape {
dim {
dim_value: 1
}
dim {
dim_value: 800
}
dim {
dim_value: 10
}
}
}
}
]
depth = 191
doc_string =
graph torch-jit-export (
%images[FLOAT, 1x3x640x640]
) initializers (
%neck.lateral_convs.0.conv.weight[FLOAT, 16x72x1x1]
%neck.lateral_convs.0.conv.bias[FLOAT, 16]
%neck.lateral_convs.1.conv.weight[FLOAT, 16x152x1x1]
%neck.lateral_convs.1.conv.bias[FLOAT, 16]
%neck.lateral_convs.2.conv.weight[FLOAT, 16x288x1x1]
%neck.lateral_convs.2.conv.bias[FLOAT, 16]
%neck.fpn_convs.0.conv.weight[FLOAT, 16x16x3x3]
%neck.fpn_convs.0.conv.bias[FLOAT, 16]
%neck.fpn_convs.1.conv.weight[FLOAT, 16x16x3x3]
%neck.fpn_convs.1.conv.bias[FLOAT, 16]
%neck.fpn_convs.2.conv.weight[FLOAT, 16x16x3x3]
%neck.fpn_convs.2.conv.bias[FLOAT, 16]
%neck.downsample_convs.0.conv.weight[FLOAT, 16x16x3x3]
%neck.downsample_convs.0.conv.bias[FLOAT, 16]
%neck.downsample_convs.1.conv.weight[FLOAT, 16x16x3x3]
%neck.downsample_convs.1.conv.bias[FLOAT, 16]
%neck.pafpn_convs.0.conv.weight[FLOAT, 16x16x3x3]
%neck.pafpn_convs.0.conv.bias[FLOAT, 16]
%neck.pafpn_convs.1.conv.weight[FLOAT, 16x16x3x3]
%neck.pafpn_convs.1.conv.bias[FLOAT, 16]
%bbox_head.stride_cls.(8, 8).weight[FLOAT, 2x64x3x3]
%bbox_head.stride_cls.(8, 8).bias[FLOAT, 2]
%bbox_head.stride_cls.(16, 16).weight[FLOAT, 2x64x3x3]
%bbox_head.stride_cls.(16, 16).bias[FLOAT, 2]
%bbox_head.stride_cls.(32, 32).weight[FLOAT, 2x64x3x3]
%bbox_head.stride_cls.(32, 32).bias[FLOAT, 2]
%bbox_head.stride_reg.(8, 8).weight[FLOAT, 8x64x3x3]
%bbox_head.stride_reg.(8, 8).bias[FLOAT, 8]
%bbox_head.stride_reg.(16, 16).weight[FLOAT, 8x64x3x3]
%bbox_head.stride_reg.(16, 16).bias[FLOAT, 8]
%bbox_head.stride_reg.(32, 32).weight[FLOAT, 8x64x3x3]
%bbox_head.stride_reg.(32, 32).bias[FLOAT, 8]
%bbox_head.stride_kps.(8, 8).weight[FLOAT, 20x64x3x3]
%bbox_head.stride_kps.(8, 8).bias[FLOAT, 20]
%bbox_head.stride_kps.(16, 16).weight[FLOAT, 20x64x3x3]
%bbox_head.stride_kps.(16, 16).bias[FLOAT, 20]
%bbox_head.stride_kps.(32, 32).weight[FLOAT, 20x64x3x3]
%bbox_head.stride_kps.(32, 32).bias[FLOAT, 20]
%555[FLOAT, 16x3x3x3]
%556[FLOAT, 16]
%558[FLOAT, 16x1x3x3]
%559[FLOAT, 16]
%561[FLOAT, 16x16x1x1]
%562[FLOAT, 16]
%564[FLOAT, 16x1x3x3]
%565[FLOAT, 16]
%567[FLOAT, 40x16x1x1]
%568[FLOAT, 40]
%570[FLOAT, 40x1x3x3]
%571[FLOAT, 40]
%573[FLOAT, 40x40x1x1]
%574[FLOAT, 40]
%576[FLOAT, 40x1x3x3]
%577[FLOAT, 40]
%579[FLOAT, 72x40x1x1]
%580[FLOAT, 72]
%582[FLOAT, 72x1x3x3]
%583[FLOAT, 72]
%585[FLOAT, 72x72x1x1]
%586[FLOAT, 72]
%588[FLOAT, 72x1x3x3]
%589[FLOAT, 72]
%591[FLOAT, 72x72x1x1]
%592[FLOAT, 72]
%594[FLOAT, 72x1x3x3]
%595[FLOAT, 72]
%597[FLOAT, 152x72x1x1]
%598[FLOAT, 152]
%600[FLOAT, 152x1x3x3]
%601[FLOAT, 152]
%603[FLOAT, 152x152x1x1]
%604[FLOAT, 152]
%606[FLOAT, 152x1x3x3]
%607[FLOAT, 152]
%609[FLOAT, 288x152x1x1]
%610[FLOAT, 288]
%612[FLOAT, 288x1x3x3]
%613[FLOAT, 288]
%615[FLOAT, 288x288x1x1]
%616[FLOAT, 288]
%618[FLOAT, 288x1x3x3]
%619[FLOAT, 288]
%621[FLOAT, 288x288x1x1]
%622[FLOAT, 288]
%624[FLOAT, 288x1x3x3]
%625[FLOAT, 288]
%627[FLOAT, 288x288x1x1]
%628[FLOAT, 288]
%630[FLOAT, 288x1x3x3]
%631[FLOAT, 288]
%633[FLOAT, 288x288x1x1]
%634[FLOAT, 288]
%636[FLOAT, 288x1x3x3]
%637[FLOAT, 288]
%639[FLOAT, 288x288x1x1]
%640[FLOAT, 288]
%642[FLOAT, 16x1x3x3]
%643[FLOAT, 16]
%645[FLOAT, 64x16x1x1]
%646[FLOAT, 64]
%648[FLOAT, 64x1x3x3]
%649[FLOAT, 64]
%651[FLOAT, 64x64x1x1]
%652[FLOAT, 64]
%654[FLOAT, 16x1x3x3]
%655[FLOAT, 16]
%657[FLOAT, 64x16x1x1]
%658[FLOAT, 64]
%660[FLOAT, 64x1x3x3]
%661[FLOAT, 64]
%663[FLOAT, 64x64x1x1]
%664[FLOAT, 64]
%666[FLOAT, 16x1x3x3]
%667[FLOAT, 16]
%669[FLOAT, 64x16x1x1]
%670[FLOAT, 64]
%672[FLOAT, 64x1x3x3]
%673[FLOAT, 64]
%675[FLOAT, 64x64x1x1]
%676[FLOAT, 64]
%677[INT64, 1]
%678[INT64, 1]
%679[INT64, 1]
%680[INT64, 1]
%681[INT64, 1]
%682[INT64, 1]
%683[INT64, 1]
%684[INT64, 1]
%685[INT64, 1]
%686[INT64, 1]
%687[INT64, 1]
%688[INT64, 1]
%689[INT64, 1]
%690[INT64, 1]
%691[INT64, 1]
%692[INT64, 1]
%693[INT64, 1]
%694[INT64, 1]
) {
%554 = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [2, 2]](%images, %555, %556)
%288 = Relu(%554)
%557 = Conv[dilations = [1, 1], group = 16, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%288, %558, %559)
%291 = Relu(%557)
%560 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%291, %561, %562)
%294 = Relu(%560)
%563 = Conv[dilations = [1, 1], group = 16, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [2, 2]](%294, %564, %565)
%297 = Relu(%563)
%566 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%297, %567, %568)
%300 = Relu(%566)
%569 = Conv[dilations = [1, 1], group = 40, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%300, %570, %571)
%303 = Relu(%569)
%572 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%303, %573, %574)
%306 = Relu(%572)
%575 = Conv[dilations = [1, 1], group = 40, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [2, 2]](%306, %576, %577)
%309 = Relu(%575)
%578 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%309, %579, %580)
%312 = Relu(%578)
%581 = Conv[dilations = [1, 1], group = 72, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%312, %582, %583)
%315 = Relu(%581)
%584 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%315, %585, %586)
%318 = Relu(%584)
%587 = Conv[dilations = [1, 1], group = 72, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%318, %588, %589)
%321 = Relu(%587)
%590 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%321, %591, %592)
%324 = Relu(%590)
%593 = Conv[dilations = [1, 1], group = 72, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [2, 2]](%324, %594, %595)
%327 = Relu(%593)
%596 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%327, %597, %598)
%330 = Relu(%596)
%599 = Conv[dilations = [1, 1], group = 152, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%330, %600, %601)
%333 = Relu(%599)
%602 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%333, %603, %604)
%336 = Relu(%602)
%605 = Conv[dilations = [1, 1], group = 152, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [2, 2]](%336, %606, %607)
%339 = Relu(%605)
%608 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%339, %609, %610)
%342 = Relu(%608)
%611 = Conv[dilations = [1, 1], group = 288, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%342, %612, %613)
%345 = Relu(%611)
%614 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%345, %615, %616)
%348 = Relu(%614)
%617 = Conv[dilations = [1, 1], group = 288, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%348, %618, %619)
%351 = Relu(%617)
%620 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%351, %621, %622)
%354 = Relu(%620)
%623 = Conv[dilations = [1, 1], group = 288, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%354, %624, %625)
%357 = Relu(%623)
%626 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%357, %627, %628)
%360 = Relu(%626)
%629 = Conv[dilations = [1, 1], group = 288, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%360, %630, %631)
%363 = Relu(%629)
%632 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%363, %633, %634)
%366 = Relu(%632)
%635 = Conv[dilations = [1, 1], group = 288, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%366, %636, %637)
%369 = Relu(%635)
%638 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%369, %639, %640)
%372 = Relu(%638)
%373 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%324, %neck.lateral_convs.0.conv.weight, %neck.lateral_convs.0.conv.bias)
%374 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%336, %neck.lateral_convs.1.conv.weight, %neck.lateral_convs.1.conv.bias)
%375 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%372, %neck.lateral_convs.2.conv.weight, %neck.lateral_convs.2.conv.bias)
%376 = Shape(%374)
%377 = Constant[value = <Scalar Tensor []>]()
%378 = Gather[axis = 0](%376, %377)
%379 = Shape(%374)
%380 = Constant[value = <Scalar Tensor []>]()
%381 = Gather[axis = 0](%379, %380)
%382 = Unsqueeze[axes = [0]](%378)
%383 = Unsqueeze[axes = [0]](%381)
%384 = Concat[axis = 0](%382, %383)
%385 = Shape(%375)
%386 = Constant[value = <Tensor>]()
%387 = Constant[value = <Tensor>]()
%388 = Constant[value = <Tensor>]()
%389 = Slice(%385, %387, %388, %386)
%390 = Cast[to = 7](%384)
%391 = Concat[axis = 0](%389, %390)
%392 = Constant[value = <Tensor>]()
%393 = Constant[value = <Tensor>]()
%394 = Resize[coordinate_transformation_mode = 'asymmetric', cubic_coeff_a = -0.75, mode = 'nearest', nearest_mode = 'floor'](%375, %392, %393, %391)
%395 = Add(%374, %394)
%396 = Shape(%373)
%397 = Constant[value = <Scalar Tensor []>]()
%398 = Gather[axis = 0](%396, %397)
%399 = Shape(%373)
%400 = Constant[value = <Scalar Tensor []>]()
%401 = Gather[axis = 0](%399, %400)
%402 = Unsqueeze[axes = [0]](%398)
%403 = Unsqueeze[axes = [0]](%401)
%404 = Concat[axis = 0](%402, %403)
%405 = Shape(%395)
%406 = Constant[value = <Tensor>]()
%407 = Constant[value = <Tensor>]()
%408 = Constant[value = <Tensor>]()
%409 = Slice(%405, %407, %408, %406)
%410 = Cast[to = 7](%404)
%411 = Concat[axis = 0](%409, %410)
%412 = Constant[value = <Tensor>]()
%413 = Constant[value = <Tensor>]()
%414 = Resize[coordinate_transformation_mode = 'asymmetric', cubic_coeff_a = -0.75, mode = 'nearest', nearest_mode = 'floor'](%395, %412, %413, %411)
%415 = Add(%373, %414)
%416 = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%415, %neck.fpn_convs.0.conv.weight, %neck.fpn_convs.0.conv.bias)
%417 = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%395, %neck.fpn_convs.1.conv.weight, %neck.fpn_convs.1.conv.bias)
%418 = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%375, %neck.fpn_convs.2.conv.weight, %neck.fpn_convs.2.conv.bias)
%419 = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [2, 2]](%416, %neck.downsample_convs.0.conv.weight, %neck.downsample_convs.0.conv.bias)
%420 = Add(%417, %419)
%421 = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [2, 2]](%420, %neck.downsample_convs.1.conv.weight, %neck.downsample_convs.1.conv.bias)
%422 = Add(%418, %421)
%423 = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%420, %neck.pafpn_convs.0.conv.weight, %neck.pafpn_convs.0.conv.bias)
%424 = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%422, %neck.pafpn_convs.1.conv.weight, %neck.pafpn_convs.1.conv.bias)
%641 = Conv[dilations = [1, 1], group = 16, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%416, %642, %643)
%427 = Relu(%641)
%644 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%427, %645, %646)
%430 = Relu(%644)
%647 = Conv[dilations = [1, 1], group = 64, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%430, %648, %649)
%433 = Relu(%647)
%650 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%433, %651, %652)
%436 = Relu(%650)
%437 = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%436, %bbox_head.stride_cls.(8, 8).weight, %bbox_head.stride_cls.(8, 8).bias)
%438 = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%436, %bbox_head.stride_reg.(8, 8).weight, %bbox_head.stride_reg.(8, 8).bias)
%439 = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%436, %bbox_head.stride_kps.(8, 8).weight, %bbox_head.stride_kps.(8, 8).bias)
%440 = Shape(%437)
%441 = Constant[value = <Scalar Tensor []>]()
%442 = Gather[axis = 0](%440, %441)
%443 = Transpose[perm = [0, 2, 3, 1]](%437)
%446 = Unsqueeze[axes = [0]](%442)
%449 = Concat[axis = 0](%446, %677, %678)
%450 = Reshape(%443, %449)
%out0 = Sigmoid(%450)
%452 = Transpose[perm = [0, 2, 3, 1]](%438)
%455 = Unsqueeze[axes = [0]](%442)
%458 = Concat[axis = 0](%455, %679, %680)
%out3 = Reshape(%452, %458)
%460 = Transpose[perm = [0, 2, 3, 1]](%439)
%463 = Unsqueeze[axes = [0]](%442)
%466 = Concat[axis = 0](%463, %681, %682)
%out6 = Reshape(%460, %466)
%653 = Conv[dilations = [1, 1], group = 16, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%423, %654, %655)
%470 = Relu(%653)
%656 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%470, %657, %658)
%473 = Relu(%656)
%659 = Conv[dilations = [1, 1], group = 64, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%473, %660, %661)
%476 = Relu(%659)
%662 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%476, %663, %664)
%479 = Relu(%662)
%480 = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%479, %bbox_head.stride_cls.(16, 16).weight, %bbox_head.stride_cls.(16, 16).bias)
%481 = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%479, %bbox_head.stride_reg.(16, 16).weight, %bbox_head.stride_reg.(16, 16).bias)
%482 = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%479, %bbox_head.stride_kps.(16, 16).weight, %bbox_head.stride_kps.(16, 16).bias)
%483 = Shape(%480)
%484 = Constant[value = <Scalar Tensor []>]()
%485 = Gather[axis = 0](%483, %484)
%486 = Transpose[perm = [0, 2, 3, 1]](%480)
%489 = Unsqueeze[axes = [0]](%485)
%492 = Concat[axis = 0](%489, %683, %684)
%493 = Reshape(%486, %492)
%out1 = Sigmoid(%493)
%495 = Transpose[perm = [0, 2, 3, 1]](%481)
%498 = Unsqueeze[axes = [0]](%485)
%501 = Concat[axis = 0](%498, %685, %686)
%out4 = Reshape(%495, %501)
%503 = Transpose[perm = [0, 2, 3, 1]](%482)
%506 = Unsqueeze[axes = [0]](%485)
%509 = Concat[axis = 0](%506, %687, %688)
%out7 = Reshape(%503, %509)
%665 = Conv[dilations = [1, 1], group = 16, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%424, %666, %667)
%513 = Relu(%665)
%668 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%513, %669, %670)
%516 = Relu(%668)
%671 = Conv[dilations = [1, 1], group = 64, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%516, %672, %673)
%519 = Relu(%671)
%674 = Conv[dilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]](%519, %675, %676)
%522 = Relu(%674)
%523 = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%522, %bbox_head.stride_cls.(32, 32).weight, %bbox_head.stride_cls.(32, 32).bias)
%524 = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%522, %bbox_head.stride_reg.(32, 32).weight, %bbox_head.stride_reg.(32, 32).bias)
%525 = Conv[dilations = [1, 1], group = 1, kernel_shape = [3, 3], pads = [1, 1, 1, 1], strides = [1, 1]](%522, %bbox_head.stride_kps.(32, 32).weight, %bbox_head.stride_kps.(32, 32).bias)
%526 = Shape(%523)
%527 = Constant[value = <Scalar Tensor []>]()
%528 = Gather[axis = 0](%526, %527)
%529 = Transpose[perm = [0, 2, 3, 1]](%523)
%532 = Unsqueeze[axes = [0]](%528)
%535 = Concat[axis = 0](%532, %689, %690)
%536 = Reshape(%529, %535)
%out2 = Sigmoid(%536)
%538 = Transpose[perm = [0, 2, 3, 1]](%524)
%541 = Unsqueeze[axes = [0]](%528)
%544 = Concat[axis = 0](%541, %691, %692)
%out5 = Reshape(%538, %544)
%546 = Transpose[perm = [0, 2, 3, 1]](%525)
%549 = Unsqueeze[axes = [0]](%528)
%552 = Concat[axis = 0](%549, %693, %694)
%out8 = Reshape(%546, %552)
return %out0, %out1, %out2, %out3, %out4, %out5, %out6, %out7, %out8
}
可以看到其输出有3个dict,一个是 input, 一个是 initializers，以及最后一个是operators把输入和权重 initialization 进行类似于 forward操作，在最后一个dict operators中其返回是 %191，也就是 gemm 最后一个全连接的输出。
利用netron在线工具https://netron.app/ 查看SCRFD模型结构（SCRFD中FPN每一层对应3个head）
3、onnx模型推理
这里使用SCRFD模型进行推理
import cv2
import onnx
from onnx import helper
import onnxruntime
import numpy as np
'''三、onnx模型推理'''
if __name__ == '__main__':
ort_sess = onnxruntime.InferenceSession('./weights/scrfd_500m_kps.onnx') # Create inference session using ort.InferenceSession
# 加载图片
img = cv2.imread("./img/calibrate_glasses.jpg")
img = cv2.resize(img, (640, 640))
img = img.astype(np.float32) / 255.
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img = img.reshape((3,640,640))
if len(img.shape) == 3:
img = np.expand_dims(img, 0)
outputs = ort_sess.run(None, {'images': img}) # 调用实例sess的run方法进行推理
print(f"length of outputs = {len(outputs)}")
---
length of outputs = 9
模型的输出结果和上面的可视化的模型结果是一致的，FPN有3层，每层FPN有3个head，共9个输出。
