SIGN IN SIGN UP
milvus-io / milvus UNCLAIMED

Milvus is a high-performance, cloud-native vector database built for scalable vector ANN search

0 0 146 Go

test: use rjieba for jieba tokenizer tests (#50180)

## What

Switch Python test-side Jieba usage from the Python `jieba` package to
`rjieba`, the Python binding for `jieba-rs`.

## Changes

- Replace Python test requirements from `jieba==0.42.1` to
`rjieba==0.1.11`
- Update Python client and REST test helpers to call `rjieba.cut`,
`rjieba.cut_for_search`, and `rjieba.cut_all`
- Update analyzer expected-token helper to use `rjieba` for supported
mode/hmm cases
- Keep custom-dictionary validation focused on analyzer output because
`rjieba` does not expose dynamic dictionary APIs like Python
`jieba.add_word` or `set_dictionary`

## Test

- `git diff --check`
- `python3 -m py_compile tests/python_client/common/common_func.py
tests/python_client/common/phrase_match_generator.py
tests/restful_client_v2/utils/utils.py
tests/python_client/milvus_client/test_milvus_client_analyzer.py`
- `rjieba` API smoke test
- Against `struct-hybrid-master-d44e5bc` at `10.100.36.224:19530`:
  - `TestMilvusClientAnalyzer::test_jieba_custom_analyzer`: 6 passed
  - `TestMilvusClientAnalyzer::test_analyzer`: 3 passed

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Z
zhuwenxing committed
3517b1ec90501a9eaf13d9cf2b35c76f7bcb8344
Parent: 89659fd
Committed by GitHub <noreply@github.com> on 6/1/2026, 2:36:14 AM