Having trouble running this model with vLLM not sure why
#1
by
zacksiri
- opened
I'm seeing the same error as mentioned here https://github.com/vllm-project/vllm/issues/15965
Could it be due to missing tekken.json file? I'm using the vllm-openai docker to run the model.
Hi @zacksiri . Although this is a mistral model, it is derived from the HuggingFace definition (in contrast to using the original Mistral definition). Hence, you SHOULD NOT use these arguments: --tokenizer_mode mistral --config_format mistral --load_format mistral --tool-call-parser mistral
Also, I noticed that it works on vllm==0.8.3 but it fails on 0.8.4. I'll notify the vLLM team on this
Thank you!
zacksiri
changed discussion status to
closed