Update README.md

README.md

@@ -30,4 +30,54 @@ Benchmarks is as follows for both Qwerky-QwQ-32B and Qwerky-72B models:
 | winogrande | acc | **0.7324** | 0.7048 | **0.7956** | 0.7632 |
 | mmlu | acc | 0.7431 | **0.7985** | 0.7746 | **0.8338** |
 
-> *Note: All benchmarks except MMLU are 0-shot and Version 1. For MMLU, it's Version 2.*
+> *Note: All benchmarks except MMLU are 0-shot and Version 1. For MMLU, it's Version 2.*
+
+
+## Running with `transformers`
+
+Since this model is not on transformers at the moment you will have to enable remote code with the following line.
+
+```py
+# ...
+
+model = AutoModelForCausalLM.from_pretrained("featherless-ai/Qwerky-QwQ-32B", trust_remote_code=True)
+
+# ...
+```
+
+Other than enabling remote code, you may run the model like a regular model with transformers like so.
+
+```py
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_name = "featherless-ai/Qwerky-72B"
+
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    torch_dtype="auto",
+    device_map="auto",
+    trust_remote_code=True,
+)
+
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+
+prompt = """There is a very famous song that I recall by the singer's surname as Astley.
+ I can't remember the name or the youtube URL that people use to link as an example url.
+ What's song name?"""
+messages = [
+    {"role": "system", "content": "You are a helpful assistant."},
+    {"role": "user", "content": prompt},
+]
+text = tokenizer.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True
+)
+model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
+
+generated_ids = model.generate(**model_inputs, max_new_tokens=512)
+generated_ids = [
+    output_ids[len(input_ids) :]
+    for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
+]
+
+response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
+```