Update README.md
Browse files
README.md
CHANGED
@@ -1,3 +1,29 @@
|
|
1 |
---
|
|
|
|
|
|
|
|
|
2 |
license: mit
|
3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
---
|
2 |
+
library_name: transformers
|
3 |
+
tags:
|
4 |
+
- transformers.js
|
5 |
+
- tokenizers
|
6 |
license: mit
|
7 |
---
|
8 |
+
|
9 |
+
# Claude Tokenizer
|
10 |
+
|
11 |
+
A 🤗-compatible version of the **Claude tokenizer** (adapted from [anthropics/anthropic-sdk-python](https://github.com/anthropics/anthropic-sdk-python)). This means it can be used with Hugging Face libraries including [Transformers](https://github.com/huggingface/transformers), [Tokenizers](https://github.com/huggingface/tokenizers), and [Transformers.js](https://github.com/xenova/transformers.js).
|
12 |
+
|
13 |
+
## Example usage:
|
14 |
+
|
15 |
+
### Transformers/Tokenizers
|
16 |
+
```py
|
17 |
+
from transformers import GPT2TokenizerFast
|
18 |
+
|
19 |
+
tokenizer = GPT2TokenizerFast.from_pretrained('Xenova/claude-tokenizer')
|
20 |
+
assert tokenizer.encode('hello world') == [9381, 2253]
|
21 |
+
```
|
22 |
+
|
23 |
+
### Transformers.js
|
24 |
+
```js
|
25 |
+
import { AutoTokenizer } from '@xenova/transformers';
|
26 |
+
|
27 |
+
const tokenizer = await AutoTokenizer.from_pretrained('Xenova/claude-tokenizer');
|
28 |
+
const tokens = tokenizer.encode('hello world'); // [9381, 2253]
|
29 |
+
```
|