upload checkpoints

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+checkpoints filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,106 @@
+---
+license: mit
+license_link: https://github.com/microsoft/VidTok/blob/main/LICENSE
+
+tags:
+- tokenization
+- video generation
+- world model
+- vae
+- fsq
+---
+
+# VidTok
+A Family of Versatile and State-Of-The-Art Video Tokenizers
+
+<img src="./assets/radar.png" width="95%" alt="radar" align="center">
+
+VidTok is a cutting-edge family of video tokenizers that delivers state-of-the-art performance in both continuous and discrete tokenizations with various compression rates. VidTok incorporates several key advancements over existing approaches:
+ * ⚡️ **Efficient Architecture**. Separate spatial and temporal sampling reduces computational complexity without sacrificing quality.
+ * 🔥 **Advanced Quantization**. Finite Scalar Quantization (FSQ) addresses training instability and codebook collapse in discrete tokenization.
+ * 💥 **Enhanced Training**. A two-stage strategy—pre-training on low-res videos and fine-tuning on high-res—boosts efficiency. Reduced frame rates improve motion dynamics representation.
+
+VidTok, trained on a large-scale video dataset, outperforms previous models across all metrics, including PSNR, SSIM, LPIPS, and FVD.
+
+<video controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/619b7b1cab4c7b7f16a7d59e/4v2I2YAZJeWSnd7iqntGX.mp4"></video>
+
+Resources and technical documentation:
+
++ [GitHub](https://github.com/microsoft/VidTok)
++ [arXiv](https://arxiv.org/pdf/2412.13061)
+
+
+## Model Performance
+
+The following table shows model performance evaluated on 30 test videos in [MCL_JCL](https://mcl.usc.edu/mcl-jcv-dataset/) dataset, with a sample fps of 30. The input size is `17x256x256` for causal models and `16x256x256` for non-causal models. `VCR` indicates the video compression ratio `TxHxW`.
+
+| Model  | Regularizer | Causal  | VCR  | PSNR  | SSIM  | LPIPS  | FVD |
+|------|------|------|------|------|------|------|------|
+| [vidtok_kl_causal_488_4chn](https://huggingface.co/microsoft/VidTok/blob/main/checkpoints/vidtok_kl_causal_488_4chn.ckpt)  | KL-4chn  | ✔️   | 4x8x8  | 29.64 | 0.852| 0.114| 194.2|
+| [vidtok_kl_causal_488_8chn](https://huggingface.co/microsoft/VidTok/blob/main/checkpoints/vidtok_kl_causal_488_8chn.ckpt)  |  KL-8chn |  ✔️   |4x8x8  | 31.83 | 0.897| 0.083| 109.3|
+| [vidtok_kl_causal_488_16chn](https://huggingface.co/microsoft/VidTok/blob/main/checkpoints/vidtok_kl_causal_488_16chn.ckpt)  | KL-16chn  | ✔️  | 4x8x8   | 35.04 |0.942 |0.047 | 78.9| 
+| [vidtok_kl_causal_41616_4chn](https://huggingface.co/microsoft/VidTok/blob/main/checkpoints/vidtok_kl_causal_41616_4chn.ckpt)  | KL-4chn  | ✔️  | 4x16x16   | 25.05  | 0.711| 0.228| 549.1| |
+| [vidtok_kl_noncausal_488_4chn](https://huggingface.co/microsoft/VidTok/blob/main/checkpoints/vidtok_kl_noncausal_488_4chn.ckpt)  | KL-4chn  |  ✖️ | 4x8x8   | 30.60  | 0.876 | 0.098| 157.9| 
+| [vidtok_kl_noncausal_41616_4chn](https://huggingface.co/microsoft/VidTok/blob/main/checkpoints/vidtok_kl_noncausal_41616_4chn.ckpt)  |  KL-4chn | ✖️  | 4x16x16   | 26.06  | 0.751 | 0.190|423.2 |
+| [vidtok_fsq_causal_488_262144](https://huggingface.co/microsoft/VidTok/blob/main/checkpoints/vidtok_fsq_causal_488_262144.ckpt)  | FSQ-262,144  | ✔️  |  4x8x8  | 29.82  | 0.867 |0.106 | 160.1|
+| [vidtok_fsq_causal_488_32768](https://huggingface.co/microsoft/VidTok/blob/main/checkpoints/vidtok_fsq_causal_488_32768.ckpt)  | FSQ-32,768  | ✔️  | 4x8x8   | 29.16  | 0.854 | 0.117| 196.9|
+| [vidtok_fsq_causal_488_4096](https://huggingface.co/microsoft/VidTok/blob/main/checkpoints/vidtok_fsq_causal_488_4096.ckpt)  | FSQ-4096  | ✔️  | 4x8x8   | 28.36 | 0.832 | 0.133| 218.1|
+| [vidtok_fsq_causal_41616_262144](https://huggingface.co/microsoft/VidTok/blob/main/checkpoints/vidtok_fsq_causal_41616_262144.ckpt)  |  FSQ-262,144 |  ✔️ |  4x16x16 | 25.38 | 0.738 |0.206 | 430.1|
+| [vidtok_fsq_noncausal_488_262144](https://huggingface.co/microsoft/VidTok/blob/main/checkpoints/vidtok_fsq_noncausal_488_262144.ckpt)  |  FSQ-262,144 | ✖️  | 4x8x8 | 30.78 | 0.889| 0.091| 132.1|
+| [vidtok_fsq_noncausal_41616_262144](https://huggingface.co/microsoft/VidTok/blob/main/checkpoints/vidtok_fsq_noncausal_41616_262144.ckpt)  |  FSQ-262,144 | ✖️  |  4x16x16  | 26.37 | 0.772| 0.171| 357.0|
+
+## Training
+### Training Data
+
+The training data of VidTok is divided into two sets based on video quality. 
+1. Training Set 1 consists of approximately 400K of low-resolution videos (e.g., 480p). The videos are natural videos with diverse lightning, motions, and scenarios. 
+2. Training Set 2 includes approximately 10K of high-resolution videos (e.g., 1080p). The videos are natural videos with diverse lightning, motions, and scenarios.
+
+### Training Procedure
+
+Please refer to the [paper](https://arxiv.org/pdf/2412.13061) and [code](https://github.com/microsoft/VidTok) for detailed training instructions.
+
+## Evaluation
+
+Please refer to the [paper](https://arxiv.org/pdf/2412.13061) and [code](https://github.com/microsoft/VidTok) for detailed evaluation instructions.
+
+## Intended Uses
+
+We are sharing our model with the research community to foster further research in this area: 
+* Training your own video tokenizers for research purpose.
+* Video tokenization with various compression rates.
+
+## Downstream Uses
+
+Our model is designed to accelerate research on video-centric research, for use as a building block for the following applications:
+* Video generation on the continuous / discrete latent tokens.
+* World modelling on the continuous / discrete latent tokens.
+* Generative games on the continuous / discrete latent tokens.
+* Video understanding from the latent tokens.
+
+## Out-of-scope Uses
+
+Our models are not specifically designed or evaluated for all downstream purposes. Developers should consider common limitations of video tokenizers (e.g., performance degradation on out-of-domain data) as they select use cases, and evaluate and mitigate for privacy, safety, and fairness before using within a specific downstream use case, particularly for high-risk scenarios. 
+
+Developers should be aware of and adhere to applicable laws or regulations (including privacy, trade compliance laws, etc.) that are relevant to their use case. 
+
+## Risks and Limitations 
+
+Some of the limitations of this model to be aware of include:
+* VidTok may lose detailed information on the reconstructed content.
+* VidTok inherits any biases, errors, or omissions characteristic of its training data.
+* VidTok was developed for research and experimental purposes. Further testing and validation are needed before considering its application in commercial or real-world scenarios.
+
+## Recommendations
+
+Some recommendations for alleviating potential limitations include:
+* Lower compression rate provides higher reconstruction quality.
+* For domain-specific video tokenization, it is suggested to fine-tune the model on the domain-specific videos.
+
+## License
+
+The model is released under the [MIT license](https://github.com/microsoft/VidTok/blob/main/LICENSE).
+
+## Contact
+
+We welcome feedback and collaboration from our audience. If you have suggestions, questions, or observe unexpected/offensive behavior in our technology, please contact us at tianyuhe@microsoft.com.

checkpoints/vidtok_fsq_causal_41616_262144.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:86035579f7037d9ec2ca1ef9e0c310c03882fcbad82b0ce51a40568db786be63
+size 866056490

checkpoints/vidtok_fsq_causal_488_262144.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:56139b893176f11a6bf03f44a384c4a9c838fb7fc05cf97352b1e96a07a8c4bf
+size 699955790

checkpoints/vidtok_fsq_causal_488_32768.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cc7f0039c53ec1de83322698f1a8847feaba95d3060798c28cb0e1313604283d
+size 699844722

checkpoints/vidtok_fsq_causal_488_4096.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:610348b0c8c25df1e92d31e6135089f8daed50fe30af40f4432994d9ce283fb1
+size 699733654

checkpoints/vidtok_fsq_noncausal_41616_262144.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:22127b45eaac642693041be2f5551a488de04ad17bcfb20c7b392d61c99eda99
+size 866052922

checkpoints/vidtok_fsq_noncausal_488_262144.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1dcb479f276e8daef9aacd252912e1efc883669adb335e5a4b82aa17bd5387ce
+size 699952738

checkpoints/vidtok_kl_causal_288_8chn.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:097f9ce6ad8ccf36d83ee6953118d6f426398e89188ea9f2e07afc8872b904b0
+size 665222874

checkpoints/vidtok_kl_causal_41616_4chn.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d92d7b9d639cc0633f23b5447e0a9f7b460403ec1eec4d755ce56575037814c3
+size 866054682

checkpoints/vidtok_kl_causal_444_4chn.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dcc2e0fce3c127effcd17c5ca47f9cd29b8dd2f67a800e054154c56fa5673d72
+size 689923130

checkpoints/vidtok_kl_causal_488_16chn.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5efdf675a98ed6867a454bc4f65130de79b1caddd89d9fcd3a43eb1a981f7eb6
+size 701945558

checkpoints/vidtok_kl_causal_488_4chn.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e10481b370af68b3712d91affd0d5a8a59e83a1d18dcbdcc3fa02376668a682c
+size 699954234

checkpoints/vidtok_kl_causal_488_8chn.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6674a27f4ae661eebf105a336b6ac10d1a09ef7b38edd71470081360a4607331
+size 700617850

checkpoints/vidtok_kl_noncausal_41616_4chn.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:64273b7030a3b3c2d194521e4778cfa8a684cda03d71b05b766e68e4112980c6
+size 866051114

checkpoints/vidtok_kl_noncausal_488_4chn.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a0ebf5e03f4bc1855f98a83c45097e305f2704a3d814e916e90b6b730d4b49e7
+size 699951182