Birchlabs commited on
Commit
06effb0
·
1 Parent(s): 58b465a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +22 -7
README.md CHANGED
@@ -8,14 +8,15 @@ This is a fork of [`llama-13b`](https://huggingface.co/huggyllama/llama-13b) + [
8
 
9
  That is: we loaded Llama-13b, we applied Alpaca LoRA, expanded vocabulary, then finetuned from there.
10
 
11
- This expanded tokenizer was used:
12
- https://huggingface.co/Birchlabs/llama-13b-stepwise-tokenizer/blob/main/README.md
13
 
14
- You will also need the finetuned input/output embedding layers:
15
- https://huggingface.co/Birchlabs/llama-13b-stepwise-embeddings/tree/main
 
 
 
16
 
17
- In total, you can load like this (use `evaluate.py` from our [`stepwise`](https://github.com/scottlogic-alex/qlora/tree/stepwise) branch of qlora)):
18
- https://github.com/scottlogic-alex/qlora/blob/stepwise/evaluate.py#L209-L278
19
 
20
  Download `embed_tokens.pt` and `lm_head.pt` from [`Birchlabs/llama-13b-stepwise-embeddings`](https://huggingface.co/Birchlabs/llama-13b-stepwise-embeddings/tree/main), then run evaluator like so:
21
 
@@ -30,4 +31,18 @@ python -m evaluate \
30
  --bf16 \
31
  --use_bos_token_in_prompt \
32
  --overrun_countermeasures False
33
- ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
 
9
  That is: we loaded Llama-13b, we applied Alpaca LoRA, expanded vocabulary, then finetuned from there.
10
 
11
+ Parts:
 
12
 
13
+ - base model [`llama-13b`](https://huggingface.co/huggyllama/llama-13b)
14
+ - LoRA 0 [`chansung/alpaca-lora-13b`](https://huggingface.co/chansung/alpaca-lora-13b)
15
+ - LoRA 1 [`Birchlabs/llama-13b-stepwise-adapter`](https://huggingface.co/Birchlabs/llama-13b-stepwise-adapter) (this)
16
+ - tokenizer [`Birchlabs/llama-13b-stepwise-tokenizer`](https://huggingface.co/Birchlabs/llama-13b-stepwise-tokenizer)
17
+ - finetuned input/output embedding layers: [`Birchlabs/llama-13b-stepwise-embeddings`](https://huggingface.co/Birchlabs/llama-13b-stepwise-embeddings)
18
 
19
+ You can load using [`evaluate.py`](https://github.com/scottlogic-alex/qlora/blob/stepwise/evaluate.py#L209-L278) from our [`stepwise`](https://github.com/scottlogic-alex/qlora/tree/stepwise) branch of [qlora](https://github.com/artidoro/qlora).
 
20
 
21
  Download `embed_tokens.pt` and `lm_head.pt` from [`Birchlabs/llama-13b-stepwise-embeddings`](https://huggingface.co/Birchlabs/llama-13b-stepwise-embeddings/tree/main), then run evaluator like so:
22
 
 
31
  --bf16 \
32
  --use_bos_token_in_prompt \
33
  --overrun_countermeasures False
34
+ ```
35
+
36
+ Example instruction-response conversations:
37
+
38
+ `$ How do I drink a matcha latte?`
39
+ `To drink a matcha latte, I should first remove the matcha from the bag. <|step_end|> <|step_start|> I can do that by opening the bag and shaking out the matcha into a glass or bowl. <|step_end|> <|step_start|> Then, I should add some hot water into the glass or bowl and start stirring the matcha. <|step_end|> <|step_start|> This is to make the matcha easier to mix with the water. <|step_end|> <|step_start|> Next, I should add milk into the glass or bowl and continue stirring. <|step_end|> <|step_start|> This is to make the matcha and the water form a thick mixture. <|step_end|> <|step_start|> Finally, I should add ice cubes into the glass or bowl and stir again. <|step_end|> <|step_start|> This is to make the mixture even thicker and chilled. <|step_end|> <|step_start|> So, a matcha latte is a drink that is a mixture of matcha, water, and milk, with some ice cubes. <|step_end|> <|step_start|> <|answer_start|> To make a matcha latte, I should remove the matcha from the bag, add some hot water into a glass or bowl and stir, add milk into the glass or bowl and stir as well, and finally add ice cubes and stir again. <|step_end|> <|step_start|> <|answer_start|> To drink a matcha latte, I should remove`
40
+
41
+ `$ What is the third Fibonacci number?`
42
+ `I know that the first two Fibonacci numbers are 1 and 1, since they are just the ones that appear in the Fibonacci sequence: 1, 1, 2, 3, 5, 8, 13, and so on, where each term is the sum of the previous two. <|step_end|> <|step_start|> To find the third number, I could try to figure out a pattern or a shortcut, but I'll just use the definition and do some calculations. <|step_end|> <|step_start|> The third Fibonacci number is the sum of the first two, so I have 1 + 1 = 2. <|step_end|> <|step_start|> <|answer_start|> 2 <|answer_end|>`
43
+
44
+ `$ How do I make a cup of tea?`
45
+ `Ok, the first step is to open the cup of tea. <|step_end|> <|step_start|> Then the second step is to jump in the tea. <|step_end|> <|step_start|> And finally the third step is to drink the jump. <|step_end|> <|step_start|> Let's reverse the order of the steps so that they are more intuitive. <|step_end|> <|step_start|> The first step is to jump in the tea. <|step_end|> <|step_start|> Then the second step is to open the cup of tea. <|step_end|> <|step_start|> And finally the third step is to drink the jump. <|step_end|> <|step_start|> So, I think this is a good way to make a cup of tea. <|step_end|> <|step_start|> Yes, it is. <|answer_start|> How do I make a cup of tea? <|answer_end|>`
46
+
47
+ `$ How do I open a jar?`
48
+ `To open a jar, I need to use a key that fits into the lock on the jar. <|step_end|> <|step_start|> I recognize that a common type of key is a skeleton key, which can open many different types of locks, because it has no ridges that would restrict its usage. <|step_end|> <|step_start|> To get a skeleton key, I can cut any old key into pieces and join them back together. <|step_end|> <|step_start|> Alternatively, I can use a special tool that can make cuts exactly perpendicular to each other, forming a key that opens any lock. <|step_end|> <|step_start|> Alternatively, I can use a special trick that can make any old key open the jar. <|step_end|> <|step_start|> This trick involves placing the key on top of the jar and banging it on a hard surface, such as the floor or a table. <|step_end|> <|step_start|> I need to make sure that the key and the jar are both upside-down, and that I am banging them as hard as I can. <|step_end|> <|step_start|> This will create a small hole in the glass on the jar, and then I can use the key to open it. <|step_end|> <|step_start|> I need to be careful not to get cut by the glass, and I need to check if the jar is empty first, otherwise I might poke my finger and`