v0.5.0

What's Changed

• fix: Updated semantic chunking tutorial by @bhavnicksm in #205
• rewrite backend by @stephantul in #207
• fix bibtex by @stephantul in #208
• feat: Added py.typed file by @Pringled in #214
• fix: pretokenize tokens before checking vocabulary by @stephantul in #215
• feat: add dimensionality during loading by @stephantul in #216
• feat: add quantization by @stephantul in #217
• feat: save load subfolder by @stephantul in #218
• feat: Added quantization for from_sentence_transformers by @Pringled in #219
• feat: faster inference for large vocab by @stephantul in #221
• feat: track token provenance by @stephantul in #222
• fix: typing issues, bug in infernece by @stephantul in #224
• fix: issues with unk and pad by @stephantul in #225
• bug: fix 0 score in evaluate by @stephantul in #226
• fix: precision during training by @stephantul in #228
• fix: issue with unk in unigram by @stephantul in #227
• docs: add info about quantization and dimensionality reduction by @stephantul in #231
• increment version by @stephantul in #232

New Contributors

• @bhavnicksm made their first contribution in #205

Full Changelog: 0.4.1...v0.5.0