Closed
Description
[2024-11-11T23:29:37.758Z] ======================================================================
[2024-11-11T23:29:37.758Z] ERROR: test_trt_export_3_fp16 (tests.test_bundle_trt_export.TestTRTExport)
[2024-11-11T23:29:37.758Z] ----------------------------------------------------------------------
[2024-11-11T23:29:37.758Z] Traceback (most recent call last):
[2024-11-11T23:29:37.758Z] File "/opt/monai/tests/utils.py", line 810, in command_line_tests
[2024-11-11T23:29:37.759Z] normal_out = subprocess.run(cmd, env=test_env, check=True, capture_output=True)
[2024-11-11T23:29:37.759Z] File "/usr/lib/python3.10/subprocess.py", line 526, in run
[2024-11-11T23:29:37.759Z] raise CalledProcessError(retcode, process.args,
[2024-11-11T23:29:37.759Z] subprocess.CalledProcessError: Command '['python', '-m', 'monai.bundle', 'trt_export', 'network_def', '--filepath', '/tmp/tmpiha_eyf8/model_trt_fp16.ts', '--meta_file', '/opt/monai/tests/testing_data/metadata.json', '--config_file', "['/opt/monai/tests/testing_data/inference.json','/tmp/tmpiha_eyf8/def_args.yaml']", '--ckpt_file', '/tmp/tmpiha_eyf8/model.pt', '--args_file', '/tmp/tmpiha_eyf8/def_args.yaml', '--precision', 'fp16', '--input_shape', '[1, 1, 96, 96, 96]', '--dynamic_batch', '[1, 4, 8]']' returned non-zero exit status 1.
[2024-11-11T23:29:37.759Z]
[2024-11-11T23:29:37.759Z] The above exception was the direct cause of the following exception:
[2024-11-11T23:29:37.759Z]
[2024-11-11T23:29:37.759Z] Traceback (most recent call last):
[2024-11-11T23:29:37.759Z] File "/usr/local/lib/python3.10/dist-packages/parameterized/parameterized.py", line 620, in standalone_func
[2024-11-11T23:29:37.759Z] return func(*(a + p.args), **p.kwargs, **kw)
[2024-11-11T23:29:37.759Z] File "/opt/monai/tests/test_bundle_trt_export.py", line 88, in test_trt_export
[2024-11-11T23:29:37.759Z] command_line_tests(cmd)
[2024-11-11T23:29:37.759Z] File "/opt/monai/tests/utils.py", line 816, in command_line_tests
[2024-11-11T23:29:37.759Z] raise RuntimeError(f"subprocess call error {e.returncode}: {errors}, {output}") from e
[2024-11-11T23:29:37.759Z] RuntimeError: subprocess call error 1: b'WARNING:root:Given dtype that does not have direct mapping to torch (dtype.unknown), defaulting to torch.float
[2024-11-11T23:29:37.759Z] WARNING:root:Given dtype that does not have direct mapping to torch (dtype.unknown), defaulting to torch.float
[2024-11-11T23:29:37.759Z] WARNING: [Torch-TensorRT] - Detected and removing exception in TorchScript IR for node: = prim::If(%387) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:562:8 block0(): %388 : str = aten::format(%318, %386) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:563:29 = prim::RaiseException(%388, %317) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:563:12 -> () block1(): -> ()
[2024-11-11T23:29:37.759Z] WARNING: [Torch-TensorRT] - Detected and removing exception in TorchScript IR for node: = prim::If(%401) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:562:8 block0(): %402 : str = aten::format(%318, %400) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:563:29 = prim::RaiseException(%402, %317) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:563:12 -> () block1(): -> ()
[2024-11-11T23:29:37.759Z] WARNING: [Torch-TensorRT] - Detected and removing exception in TorchScript IR for node: = prim::If(%415) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:562:8 block0(): %416 : str = aten::format(%318, %414) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:563:29 = prim::RaiseException(%416, %317) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:563:12 -> () block1(): -> ()
[2024-11-11T23:29:37.759Z] WARNING: [Torch-TensorRT] - Detected and removing exception in TorchScript IR for node: = prim::If(%429) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:562:8 block0(): %430 : str = aten::format(%318, %428) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:563:29 = prim::RaiseException(%430, %317) # /usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py:563:12 -> () block1(): -> ()
[2024-11-11T23:29:37.759Z] WARNING: [Torch-TensorRT] - Conv3d layer with kernel size = 1 configuration incurs a failure with TensorRT tactic optimizer in some cases. Github issue: https://github.com/pytorch/TensorRT/issues/1445. Other conv variants do not have this issue.
[2024-11-11T23:29:37.759Z] ERROR: [Torch-TensorRT TorchScript Conversion Context] - IBuilder::buildSerializedNetwork: Error Code 9: API Usage Error (Target GPU SM 70 is not supported by this TensorRT release.)