TestBed - Cannot use scipy.linalg.eigh for sparse A

[gotsysdba]

Checklist

• I have searched the existing issues for similar issues.
• I added a very descriptive title to this issue.
• I have provided sufficient information below to help reproduce this issue.

Summary

Q&A Generate fails with Scipy error

Steps To Reproduce

LLM: oci/meta.llama-4-scout-17b-16e-instruct
Embedding: ollama/mxbai-embed-large

Generate Q&A

Expected Behavior

No response

Current Behavior

2025-Oct-29 08:42:46 (v1.3.0) - INFO   - (LiteLLM): Wrapper: Completed Call, calling success_handler
2025-Oct-29 08:42:46 (v1.3.0) - INFO   - (uvicorn.access): 127.0.0.1:33460 - "POST /v1/testbed/testset_generate?ll_model=oci%2Fmeta.llama-4-scout-17b-16e-instruct&embed_model=ollama%2Fmxbai-embed-large&questions=8&name=NqfvXNlzOyPHIWan HTTP/1.1" 500
2025-Oct-29 08:42:46 (v1.3.0) - ERROR  - (uvicorn.error): Exception in ASGI application
Traceback (most recent call last):
 File "/app/src/server/api/v1/testbed.py", line 171, in testbed_generate_qa
  test_set = utils_testbed.build_knowledge_base(text_nodes, questions, ll_model, embed_model, oci_config)
...
 File "/app/.venv/lib64/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py", line 1605, in eigsh
  raise TypeError("Cannot use scipy.linalg.eigh for sparse A with "
TypeError: Cannot use scipy.linalg.eigh for sparse A with k >= N. Use scipy.linalg.eigh(A.toarray()) or reduce k.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
...
 File "/app/src/server/api/v1/testbed.py", line 205, in testbed_generate_qa
  shutil.rmtree(temp_directory)
 File "/usr/lib64/python3.11/shutil.py", line 752, in rmtree
  _rmtree_safe_fd(fd, path, onerror)
 File "/usr/lib64/python3.11/shutil.py", line 647, in _rmtree_safe_fd
  onerror(os.scandir, path, sys.exc_info())
 File "/usr/lib64/python3.11/shutil.py", line 643, in _rmtree_safe_fd
  with os.scandir(topfd) as scandir_it:
     ^^^^^^^^^^^^^^^^^
OSError: [Errno 24] Too many open files: PosixPath('/tmp/3531eb49-316e-428f-9ea3-ae012c58a04f/testbed')

Is this a regression?

• Yes, this used to work in a previous version.

Debug info

• Version:
• Python version:
• Operating System:
• Browser:

Additional Information

No response