test_trt_compile test error in pytorch 23.08

[KumoLiu]

[2024-09-02T09:35:38.944Z] ======================================================================
[2024-09-02T09:35:38.944Z] ERROR: test_unet_value_0_fp32 (tests.test_trt_compile.TestTRTCompile)
[2024-09-02T09:35:38.944Z] ----------------------------------------------------------------------
[2024-09-02T09:35:38.944Z] Traceback (most recent call last):
[2024-09-02T09:35:38.944Z]   File "/usr/local/lib/python3.10/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
[2024-09-02T09:35:38.944Z]     return func(*(a + p.args), **p.kwargs, **kw)
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/tests/test_trt_compile.py", line 91, in test_unet_value
[2024-09-02T09:35:38.944Z]     trt_output = model(input_example)
[2024-09-02T09:35:38.944Z]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1505, in _wrapped_call_impl
[2024-09-02T09:35:38.944Z]     return self._call_impl(*args, **kwargs)
[2024-09-02T09:35:38.944Z]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1514, in _call_impl
[2024-09-02T09:35:38.944Z]     return forward_call(*args, **kwargs)
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/monai/networks/trt_compiler.py", line 493, in trt_forward
[2024-09-02T09:35:38.944Z]     return self._trt_compiler.forward(self, argv, kwargs)
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/monai/networks/trt_compiler.py", line 350, in forward
[2024-09-02T09:35:38.944Z]     raise e
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/monai/networks/trt_compiler.py", line 342, in forward
[2024-09-02T09:35:38.944Z]     self._build_and_save(model, build_args)
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/monai/networks/trt_compiler.py", line 483, in _build_and_save
[2024-09-02T09:35:38.944Z]     engine_bytes = self._onnx_to_trt(str(onnx_path))
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/monai/networks/trt_compiler.py", line 411, in _onnx_to_trt
[2024-09-02T09:35:38.944Z]     return engine_bytes_from_network(network, config=CreateConfig(profiles=profiles, **build_args))
[2024-09-02T09:35:38.944Z] TypeError: CreateConfig.__init__() got an unexpected keyword argument 'bf16'
[2024-09-02T09:35:38.944Z] 
[2024-09-02T09:35:38.944Z] ======================================================================
[2024-09-02T09:35:38.944Z] ERROR: test_unet_value_1_fp16 (tests.test_trt_compile.TestTRTCompile)
[2024-09-02T09:35:38.944Z] ----------------------------------------------------------------------
[2024-09-02T09:35:38.944Z] Traceback (most recent call last):
[2024-09-02T09:35:38.944Z]   File "/usr/local/lib/python3.10/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
[2024-09-02T09:35:38.944Z]     return func(*(a + p.args), **p.kwargs, **kw)
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/tests/test_trt_compile.py", line 91, in test_unet_value
[2024-09-02T09:35:38.944Z]     trt_output = model(input_example)
[2024-09-02T09:35:38.944Z]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1505, in _wrapped_call_impl
[2024-09-02T09:35:38.944Z]     return self._call_impl(*args, **kwargs)
[2024-09-02T09:35:38.944Z]   File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1514, in _call_impl
[2024-09-02T09:35:38.944Z]     return forward_call(*args, **kwargs)
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/monai/networks/trt_compiler.py", line 493, in trt_forward
[2024-09-02T09:35:38.944Z]     return self._trt_compiler.forward(self, argv, kwargs)
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/monai/networks/trt_compiler.py", line 350, in forward
[2024-09-02T09:35:38.944Z]     raise e
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/monai/networks/trt_compiler.py", line 342, in forward
[2024-09-02T09:35:38.944Z]     self._build_and_save(model, build_args)
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/monai/networks/trt_compiler.py", line 483, in _build_and_save
[2024-09-02T09:35:38.944Z]     engine_bytes = self._onnx_to_trt(str(onnx_path))
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/monai/networks/trt_compiler.py", line 411, in _onnx_to_trt
[2024-09-02T09:35:38.944Z]     return engine_bytes_from_network(network, config=CreateConfig(profiles=profiles, **build_args))
[2024-09-02T09:35:38.944Z] TypeError: CreateConfig.__init__() got an unexpected keyword argument 'bf16'
[2024-09-02T09:35:38.944Z] 
[2024-09-02T09:35:38.944Z] ======================================================================
[2024-09-02T09:35:38.944Z] FAIL: test_handler (tests.test_trt_compile.TestTRTCompile)
[2024-09-02T09:35:38.944Z] ----------------------------------------------------------------------
[2024-09-02T09:35:38.944Z] Traceback (most recent call last):
[2024-09-02T09:35:38.944Z]   File "/home/jenkins/agent/workspace/Monai-latest-image/tests/test_trt_compile.py", line 67, in test_handler
[2024-09-02T09:35:38.944Z]     self.assertIsNotNone(net1._trt_compiler.engine)
[2024-09-02T09:35:38.944Z] AssertionError: unexpectedly None
[2024-09-02T09:35:38.944Z] 
[2024-09-02T09:35:38.944Z] ----------------------------------------------------------------------
[2024-09-02T09:35:38.944Z] Ran 15905 tests in 4410.586s
[2024-09-02T09:35:38.944Z] 
[2024-09-02T09:35:38.944Z] FAILED (failures=1, errors=4, skipped=342)

[KumoLiu]

Hi @borisfom, could you please take a look at this error? Our daily CI runs tests on the PyTorch base image 23.08. If test_trt_compile isn’t compatible with this version, could you please add a skip or check for the minimum installed version?

MONAI/tests/utils.py

Line 261 in 6a0e1b0

class SkipIfBeforePyTorchVersion:

MONAI/monai/utils/module.py

Line 342 in 6a0e1b0

version_checker: Callable[..., bool] = min_version,

Thanks.

[borisfom]

I have pushed a fix for that issue.

[KumoLiu]

Hi @borisfom, may I ask where did you push the fix?

[borisfom]

Oh, my original PR got merged ...
Here: #8064