Stop generating token type IDs
#1
by
lysandre
HF Staff
- opened
Offering a quick fix so that the
inputs.pop("token_type_ids")
can be removed from examples. This edit ensures that the tokenizer only generates input_ids and attention_mask when called.
lysandre
changed pull request title from
Update tokenizer_config.json
to Stop generating token type IDs
You can try it out in your example by updating
processor = AutoProcessor.from_pretrained(model_id)
to
processor = AutoProcessor.from_pretrained(model_id, revision="refs/pr/1")
I see the base of the branch seems to have been updated; the changes I propose are only the ones that affect the tokenizer config :)
Thank you so much! Will check it out and merge