0
点赞
收藏
分享

微信扫一扫

解决ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported

做个橙梦 2023-08-10 阅读 52

LLaMA大模型是元宇宙平台公司(Meta)研发的大语言模型。

解决ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported_权重

问题:

load LLaMA 7b的weights的时候报错:

ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported.

出现原因:

新版transformers里面llama的tokenizer命名为LlamaTokenizer

但是旧的模型里面的tokenizer叫LLaMATokenizer

解决方案:

1、改动transformers源码中三个位置:

utils/dummy_sentencepiece_objects.py

models/auto/tokenization_aotu.py

__init__.py

在这三个文件中找到LlamaTokenizer, 改为LLaMATokenizier

2、第1种方法需要改动的地方比较多,很可能改不全,导致依旧出现“ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported.”,另外一种简单的办法就是,找到llama-7b的模型,改动tokenizer_config.json文件,直接把"tokenizer_class": "LLaMATokenizer" 改为 "tokenizer_class": "LlamaTokenizer".

参考:

https://github.com/huggingface/transformers/issues/22222


举报

相关推荐

remote ref does not exist

0 条评论