Panic when `add_json` value includes list

[mgao6767]

To reproduce,

import tantivy

schema_builder = tantivy.SchemaBuilder()
schema_builder.add_json_field("data", stored=True)
schema = schema_builder.build()

index = tantivy.Index(schema)

index_writer = index.writer()

json_data = {
    "name": "John Doe",
    "age": 30,
    "email": "john.doe@example.com",
    "interests": ["reading", "hiking", "coding"],
}

doc = tantivy.Document()
doc.add_json("data", json_data)

index_writer.add_document(doc)

index_writer.commit()
index_writer.wait_merging_threads()

searcher = index.searcher()

query = tantivy.Query.all_query()

top_docs = searcher.search(query, limit=10)

for score, hit in top_docs.hits:
    doc = searcher.doc(hit)
    print(doc["data"])  # pyo3_runtime.PanicException: not implemented

If json_data is

json_data = {
    "name": "John Doe",
    "age": 30,
    "email": "john.doe@example.com",
}

then there is no problem.

[cjrh]

EDIT - Ignore this is wrong

On tag 0.22.0, I tried to reproduce but I didn't get the error.

I did have to change a few things in your example to make it work:

def test_json_bug():
    import tantivy

    schema_builder = tantivy.SchemaBuilder()
    schema_builder.add_json_field("data", stored=True)
    schema = schema_builder.build()

    index = tantivy.Index(schema)

    index_writer = index.writer()

    data = {
        "name": "John Doe",
        "age": 30,
        "email": "john.doe@example.com",
        "interests": ["reading", "hiking", "coding"],
    }
    import json
    json_data = json.dumps(data)

    doc = tantivy.Document()
    doc.add_json("data", json_data)
    index_writer.add_document(doc)
    index_writer.commit()
    index_writer.wait_merging_threads()
    index.reload()

    searcher = index.searcher()

    query = "*"
    q = index.parse_query(query)
    top_docs = searcher.search(q, limit=10)

    print(f"Total hits: {top_docs}")
    for score, hit in top_docs.hits:
        doc = searcher.doc(hit)
        print(doc["data"])

Output:

$ pytest -k test_json_bug -vv -s
===================================== test session starts ======================================
platform linux -- Python 3.12.5, pytest-8.3.2, pluggy-1.5.0 -- /home/caleb/Documents/repos/tantivy-py/venv/bin/python3
cachedir: .pytest_cache
rootdir: /home/caleb/Documents/repos/tantivy-py
configfile: pyproject.toml
testpaths: tests, tantivy, src
collected 88 items / 87 deselected / 1 selected                                                

tests/test_deep_query.py::test_json_bug Total hits: SearchResult(hits: [(1, DocAddress { segment_ord: 0, doc: 0 })], count: 1)
[{'email': 'john.doe@example.com', 'interests': ['reading', 'hiking', 'coding'], 'age': 30, 'name': 'John Doe'}]
PASSED

===================================== slowest 10 durations =====================================
0.02s call     tests/test_deep_query.py::test_json_bug
0.00s setup    tests/test_deep_query.py::test_json_bug
0.00s teardown tests/test_deep_query.py::test_json_bug
=============================== 1 passed, 87 deselected in 0.06s ===============================

You can see that the print(doc) line runs fine:

[{'email': 'john.doe@example.com', 'interests': ['reading', 'hiking', 'coding'], 'age': 30, 'name': 'John Doe'}]

Could you try with my version above to see if you still get the error?

[cjrh]

Ok, it seems I manage to reproduce it if I update deps pyo3 and pythonize to versions 0.22 both. Is that what you have done also?

[cjrh]

It has nothing to do with pyo3 and pythonize, the behaviour changed in this commit (I did a bisection): 983364b

This was the upgrade of tantivy to 0.22.

[cjrh]

Working on a fix now.

[cjrh]

Thanks for reporting this - I have a fix in #354. I'd appreciate if you can verify that it fixes this issue, but if you can't that's ok.

[cjrh]

The root cause of the issue that I missed one of the conversion methods during the big upgrade to tantivy 0.22. The PR adds that missing implementation.