0
点赞
收藏
分享

微信扫一扫

不会真有人觉得聊天机器人难吧——使用BERT加载预训练模型得到中文句子向量

思考的鸿毛 2022-07-14 阅读 84
bertpytorch自然语言处理huggingface相似度虚拟化云计算


引言

借用修仙小说体写的标题，主要目的就是吸引你点进来🤭，客官来都来了，看完再走呗~
其实实现一个智能的聊天机器人🤖还是有一定难度的。博主就和大家一起尽可能实现一个智能的聊天机器人。
计划一周🧭更新一次，如果觉得更新慢的话，就在留言里催更吧，很有可能会响应你们的催更哦。

本系列文章📓会基于Pytorch实现一个Seq2Seq 带Attention机制的聊天机器人，在本系列文章中，大家会了解到实现聊天机器人的所有知识，欢迎关注哦。

本文主要介绍了如何通过🤗的Transformer加载哈工大的预训练模型对中文句子进行编码，以得到句向量。这样我们就可以基于这些句向量进行相似度比较，可用于基于检索的聊天机器人实现。

预训练模型

使用哈工大的​​WWM-EXT​​​版，有条件的可以直接从​​谷歌Drive​​​上进行下载，或者下载博主上传到的​​chinese_wwm_ext_pytorch​​。

下载好解压后放到你觉得舒服的位置，注意需要修改里面的​​bert_config.json​​​文件名为​​config.json​​，如下图所示：

不会真有人觉得聊天机器人难吧——使用BERT加载预训练模型得到中文句子向量_相似度

实现步骤

参考bert-as-service，采用倒数第二层的隐藏状态在token上的均值作为句向量表示。

首先引入需要的包：

import torch
from transformers import BertTokenizer,

如果没有​​transformers​​​，通过​​pip install transformers​​即可

定义预训练模型路径：

MODEL_PATH = './pretrained_models/chinese_wwm_ext_pytorch'
VOCAB_PATH = f'{MODEL_PATH}/vocab.txt'

加载预训练分词器和模型:

# 加载预训练的分词器
tokenizer = BertTokenizer.from_pretrained(VOCAB_PATH)
# 加载预训练模型 output_hidden_states=True会返回每个编码器层和嵌入层输出
model = BertModel.from_pretrained(MODEL_PATH, output_hidden_states = True)
model.eval()

输出的模型结构为：

BertModel(
  (embeddings): BertEmbeddings(
    (word_embeddings): Embedding(21128, 768, padding_idx=0)
    (position_embeddings): Embedding(512, 768)
    (token_type_embeddings): Embedding(2, 768)
    (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
    (dropout): Dropout(p=0.1, inplace=False)
  )
  (encoder): BertEncoder(
    (layer): ModuleList(
      (0): BertLayer(
        (attention): BertAttention(
          (self): BertSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (output): BertSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (intermediate): BertIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BertOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
      (1): BertLayer(
        (attention): BertAttention(
          (self): BertSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (output): BertSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (intermediate): BertIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BertOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
      (2): BertLayer(
        (attention): BertAttention(
          (self): BertSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (output): BertSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (intermediate): BertIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BertOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
      (3): BertLayer(
        (attention): BertAttention(
          (self): BertSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (output): BertSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (intermediate): BertIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BertOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
      (4): BertLayer(
        (attention): BertAttention(
          (self): BertSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (output): BertSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (intermediate): BertIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BertOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
      (5): BertLayer(
        (attention): BertAttention(
          (self): BertSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (output): BertSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (intermediate): BertIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BertOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
      (6): BertLayer(
        (attention): BertAttention(
          (self): BertSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (output): BertSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (intermediate): BertIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BertOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
      (7): BertLayer(
        (attention): BertAttention(
          (self): BertSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (output): BertSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (intermediate): BertIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BertOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
      (8): BertLayer(
        (attention): BertAttention(
          (self): BertSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (output): BertSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (intermediate): BertIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BertOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
      (9): BertLayer(
        (attention): BertAttention(
          (self): BertSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (output): BertSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (intermediate): BertIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BertOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
      (10): BertLayer(
        (attention): BertAttention(
          (self): BertSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (output): BertSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (intermediate): BertIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BertOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
      (11): BertLayer(
        (attention): BertAttention(
          (self): BertSelfAttention(
            (query): Linear(in_features=768, out_features=768, bias=True)
            (key): Linear(in_features=768, out_features=768, bias=True)
            (value): Linear(in_features=768, out_features=768, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (output): BertSelfOutput(
            (dense): Linear(in_features=768, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
        (intermediate): BertIntermediate(
          (dense): Linear(in_features=768, out_features=3072, bias=True)
        )
        (output): BertOutput(
          (dense): Linear(in_features=3072, out_features=768, bias=True)
          (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
    )
  )
  (pooler): BertPooler(
    (dense): Linear(in_features=768, out_features=768, bias=True)
    (activation): Tanh()
  )
)

定义测试一些测试句子：

sentences = [
    '今天的月亮又大又圆',
    '十五的月亮十六圆',
    '今天去看电影吧',
]

基于分词器对输入进行预处理：

def preprocess(sentences,tokenizer,max_length=64):
    return tokenizer(sentences, padding=True, truncation=True,max_length=max_length,return_tensors="pt")

这一步主要进行分词、增加特殊标记、填充、截断等操作。

对我们的测试句子进行编码：

input_tensor = preprocess(sentences, tokenizer)

输出：

{'input_ids': tensor([[ 101,  791, 1921, 4638, 3299,  778, 1348, 1920, 1348, 1749,  102],
        [ 101, 1282,  758, 4638, 3299,  778, 1282, 1063, 1749,  102,    0],
        [ 101,  791, 1921, 1343, 4692, 4510, 2512, 1416,  102,    0,    0]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
        [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
        [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]])}

然后直接输入到预训练模型：

# last_hidden_state  包含从最后一个编码器层得到的所有标记的嵌入表示
# pooler_output 指最后一个编码器层[CLS]标记对应的输出的嵌入表示
# hidden_states 包含嵌入层和所有编码器层的输出
# return_dict 返回普通的元组
last_hidden_state, pooler_output, hidden_states = model(**input_tensor, return_dict = False)

根据token的词向量进行avg pooling得到句向量：

# 参考bert-as-service，采用倒数第二层的隐藏状态在token上的均值作为句向量表示。
token_vectors = hidden_states[-2]
# 计算所有输入token向量之和的均值
sentence_embeddings = torch.mean(token_vectors, dim=1)

然后我们计算第一个句子:​​今天的月亮又大又圆​​和另外两个句子的相似度：

for i in range(1,len(sentences)):
    similarity = torch.cosine_similarity(sentence_embeddings[0], sentence_embeddings[i], dim=0)
    print(similarity)

输出：

tensor(0.9129, grad_fn=<DivBackward0>)
tensor(0.7607, grad_fn=<DivBackward0>)

从输出可以看到，虽然成功判断出了前两个句子相似度很高，但是与第三个句子比较时，相似度也不低。

因此直接使用预训练的模型来获取句子相似度是有一定的问题的，我们需要继续进行微调。

在下篇文章中，我们会学到如何进行微调。


举报

相关推荐

不会真有人觉得聊天机器人难吧——微调BERT模型得到句子间的相似度

bert自然语言处理pytorch中文句子相似度数据集Linux系统/运维

小型中文版聊天机器人

数学建模eclipsejavaspringjvmsql

使用websocket开发智能聊天机器人

htmlHTTPjsonJavaScript前端开发

基于GPT搭建私有知识库聊天机器人（三）向量数据训练

gpt机器人milvus数据数据库HarmonyOS后端开发

中文聊天机器人-TensorFlow2.0-Seq2Seq（二）

TensorFlowNLP聊天机器人kerasSeq2SeqLinux系统/运维

大模型增量训练--基于transformer制作一个大模型聊天机器人

pythonjson数据HarmonyOS后端开发

基于Hologres向量计算与大模型免费定制专属聊天机器人

数据jsonpython云服务云计算

Python开发实例（十四）聊天机器人：使用自然语言处理库构建一个简单的聊天机器人

聊天机器人Python自然语言处理后端开发

使用 HelpLook Chatbot，让AI聊天机器人变成销售经理

聊天机器人可扩展性搜索深度学习人工智能

使用 Python 实现一个简单的智能聊天机器人

语音识别python深度学习
0 条评论