geckon01/anything-llm
1
0
Fork 0
mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-06-12 13:15:38 +03:00
Code Issues Packages Projects Releases Wiki Activity
Files
93d191f162d23d9c54112f73eb18c3ed0f659dde
anything-llm/collector/utils/tokenizer
History
Timothy Carambat d1ca16f7f8 Add tokenizer improvments via Singleton class and estimation (#3072) 
* Add tokenizer improvments via Singleton class
linting

* dev build

* Estimation fallback when string exceeds a fixed byte size

* Add notice to tiktoken on backend
2025-01-30 17:55:03 -08:00
..
index.js
Add tokenizer improvments via Singleton class and estimation (#3072)
2025-01-30 17:55:03 -08:00