How to use this onnx model
Ask how to use this onnx model, and especially how to configure this tokenizer based on these json files?
I have not used onnx for this model before, but I have one function that works with other models.
The tokeizer are set up outside the function via tokenizer = AutoTokenizer.from_pretrained(model_name)
.
You might need to adjust the below function based on the model and your needs.
# Function to perform inference with ONNX
def onnx_inference(question, answer):
inputs = tokenizer(question, answer, return_tensors="pt")
input_names = ort_session.get_inputs()
inputs_onnx = {
input_name.name: inputs[input_name.name].numpy() for input_name in input_names
}
outputs = ort_session.run(None, inputs_onnx)
return outputs[0][0]
from transformers import AutoTokenizer, pipeline
from optimum.onnxruntime import ORTModelForCausalLM
Explicitly specify the 'transformers' library
model = ORTModelForCausalLM.from_pretrained("openai-community/gpt2", subfolder="onnx") # Add this line
Load tokenizer (must match the original model)
tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2")
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
result = pipe("He never went out without a book under his arm")
print(result[0]["generated_text"])
Device set to use cpu
Setting pad_token_id
to eos_token_id
:50256 for open-end generation.
He never went out without a book under his arm. He never left his books unattended so that I, as well as my siblings, could read them. He could even sit in the armchair listening to those who would listen and write. H