Error if Tern calls Scancode-Toolkit

[Jeeppler]

Describe the bug

Tern is unable to scan using Scancode-Toolkit.

With Tern only the image can be scanned and results in an report.

To Reproduce

Scan Alpine based image using Tern + Scancode-Toolkit

tern report -f spdxjson -w "container.tar" -o "result.spdx.json" -x scancode

Error in terminal

2022-12-21 20:48:28,485 - DEBUG - __main__ - Starting...
2022-12-21 20:48:28,485 - DEBUG - prep - Setting up...
2022-12-21 20:48:28,486 - DEBUG - rootfs - Running command: chmod +x /usr/local/lib/python3.9/dist-packages/tern/tools/fs_hash.sh
2022-12-21 20:48:28,488 - DEBUG - run - Starting analysis...
2022-12-21 20:48:28,496 - DEBUG - rootfs - Running command: tar -tf /workspace/workspace/9a44489b-96c1-49f7-a627-cbffc58e366b/upload/extracted/binaries/test.tar
2022-12-21 20:48:28,498 - DEBUG - rootfs - Running command: tar -x -f /workspace/workspace/9a44489b-96c1-49f7-a627-cbffc58e366b/upload/extracted/binaries/test.tar -C /root/.tern/temp
2022-12-21 20:48:28,504 - DEBUG - rootfs - Running command: tar -tf /root/.tern/temp/3e571912155d9bac1a5285bf1c21105bea53585f77a159316eed491882710ab2/layer.tar
2022-12-21 20:48:28,507 - DEBUG - rootfs - Running command: tar -x -f /root/.tern/temp/3e571912155d9bac1a5285bf1c21105bea53585f77a159316eed491882710ab2/layer.tar -C /root/.tern/temp/3e571912155d9bac1a5285bf1c21105bea53585f77a159316eed491882710ab2/contents
2022-12-21 20:48:28,528 - DEBUG - rootfs - Running command: /usr/local/lib/python3.9/dist-packages/tern/tools/fs_hash.sh /root/.tern/temp/3e571912155d9bac1a5285bf1c21105bea53585f77a159316eed491882710ab2/contents
2022-12-21 20:48:28,632 - DEBUG - rootfs - Running command: tar -tf /root/.tern/temp/eb2f2a3b73763d543b7256e39e4a42ea190bd18a1e1c8197aaefa6ae5005ae94/layer.tar
2022-12-21 20:48:28,634 - DEBUG - rootfs - Running command: tar -x -f /root/.tern/temp/eb2f2a3b73763d543b7256e39e4a42ea190bd18a1e1c8197aaefa6ae5005ae94/layer.tar -C /root/.tern/temp/eb2f2a3b73763d543b7256e39e4a42ea190bd18a1e1c8197aaefa6ae5005ae94/contents
2022-12-21 20:48:28,635 - DEBUG - rootfs - Running command: /usr/local/lib/python3.9/dist-packages/tern/tools/fs_hash.sh /root/.tern/temp/eb2f2a3b73763d543b7256e39e4a42ea190bd18a1e1c8197aaefa6ae5005ae94/contents
2022-12-21 20:48:28,639 - DEBUG - rootfs - Running command: tar -tf /root/.tern/temp/cf9a1a27ba287bed93e131f5cb5831b706905811bf91d5e6b5b323187d7cac92/layer.tar
2022-12-21 20:48:28,641 - DEBUG - rootfs - Running command: tar -x -f /root/.tern/temp/cf9a1a27ba287bed93e131f5cb5831b706905811bf91d5e6b5b323187d7cac92/layer.tar -C /root/.tern/temp/cf9a1a27ba287bed93e131f5cb5831b706905811bf91d5e6b5b323187d7cac92/contents
2022-12-21 20:48:28,643 - DEBUG - rootfs - Running command: /usr/local/lib/python3.9/dist-packages/tern/tools/fs_hash.sh /root/.tern/temp/cf9a1a27ba287bed93e131f5cb5831b706905811bf91d5e6b5b323187d7cac92/contents
2022-12-21 20:48:28,646 - DEBUG - rootfs - Running command: tar -tf /root/.tern/temp/ac1cc3694703233d4731e335b6bd4cedc7bd6fe00dead7b76068f1653b147d3a/layer.tar
2022-12-21 20:48:28,647 - DEBUG - rootfs - Running command: tar -x -f /root/.tern/temp/ac1cc3694703233d4731e335b6bd4cedc7bd6fe00dead7b76068f1653b147d3a/layer.tar -C /root/.tern/temp/ac1cc3694703233d4731e335b6bd4cedc7bd6fe00dead7b76068f1653b147d3a/contents
2022-12-21 20:48:28,649 - DEBUG - rootfs - Running command: /usr/local/lib/python3.9/dist-packages/tern/tools/fs_hash.sh /root/.tern/temp/ac1cc3694703233d4731e335b6bd4cedc7bd6fe00dead7b76068f1653b147d3a/contents
2022-12-21 20:48:28,652 - DEBUG - common - Loading packages from cache: layer "d1a00b82ba"
2022-12-21 20:48:28,654 - DEBUG - common - Loading files from cache: layer "d1a00b82ba"
2022-12-21 20:48:28,657 - DEBUG - rootfs - Running command: /usr/local/bin/scancode -ilpcu --quiet --timeout 300 -n 5 --json - /root/.tern/temp/3e571912155d9bac1a5285bf1c21105bea53585f77a159316eed491882710ab2/contents
Traceback (most recent call last):
  File "/usr/local/bin/tern", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/dist-packages/tern/__main__.py", line 311, in main
    do_main(args)
  File "/usr/local/lib/python3.9/dist-packages/tern/__main__.py", line 123, in do_main
    crun.execute_image(args)
  File "/usr/local/lib/python3.9/dist-packages/tern/analyze/default/container/run.py", line 80, in execute_image
    cimage.default_analyze(full_image, args)
  File "/usr/local/lib/python3.9/dist-packages/tern/analyze/default/container/image.py", line 70, in default_analyze
    passthrough.run_extension_layer(image_obj.layers[0], options.extend,
  File "/usr/local/lib/python3.9/dist-packages/tern/analyze/passthrough.py", line 92, in run_extension_layer
    return mgr.driver.execute_layer(image_layer, redo)
  File "/usr/local/lib/python3.9/dist-packages/tern/extensions/scancode/executor.py", line 219, in execute_layer
    file_list, package_list = collect_layer_data(image_layer)
  File "/usr/local/lib/python3.9/dist-packages/tern/extensions/scancode/executor.py", line 162, in collect_layer_data
    files.append(get_scancode_file(f))
  File "/usr/local/lib/python3.9/dist-packages/tern/extensions/scancode/executor.py", line 65, in get_scancode_file
    fd.copyrights = [c['value'] for c in file_dict['copyrights']]
  File "/usr/local/lib/python3.9/dist-packages/tern/extensions/scancode/executor.py", line 65, in <listcomp>
    fd.copyrights = [c['value'] for c in file_dict['copyrights']]
KeyError: 'value'

Expected behavior

Tern can call Scancode-Toolkit without errors.

Environment you are running Tern on

• Tern version 2.11.0
• ScanCode version: 31.2.1
• Debian 11
• Python 3.9.2

[rnjudge]

Interesting! Thanks @Jeeppler I will take a look. Do you have an example container so I can reproduce + debug?

[Jeeppler]

@rnjudge I used an Alpine Image. You can find the build/run script here: https://github.com/mercedes-benz/sechub/tree/develop/sechub-pds-solutions/tern/tests. The Dockerfile can be found here: https://github.com/mercedes-benz/sechub/tree/develop/sechub-pds-solutions/tern/tests/alpine.

I think the main issue is, that the report structure changed from Scancode 30 to 31. I used the new Scancode 31 + Tern.
Previously, I did a similar scan with Scancode 30 + Tern and it worked well.

[rnjudge]

@Jeeppler I have not forgot about this. The issue I am trying to workaround right now is simply installing scancode on my VM on my Mac M1 so I can debug and fix. I opened an issue for it with Scancode here. What I've been told is that they are re-releasing Scancode with fixes for some of the dependent libraries so I'll have to wait for that...

[amallayev]

@rnjudge It seems only some images produce this error, I wonder what do they have in common:

# tern report -x scancode -i docker-daemon:node:18.12.1 -o bitnami-node.18.12.1-prod.tern.$(date +"%d.%m.%Y").txt
2023-01-19 15:17:33,198 - DEBUG - __main__ - Starting...                                                                     [21/535]2023-01-19 15:17:33,198 - DEBUG - prep - Setting up...                                                                               2023-01-19 15:17:37,099 - DEBUG - rootfs - Running command: chmod +x /root/ternenv/lib/python3.10/site-packages/tern/tools/fs_hash.sh2023-01-19 15:17:37,101 - DEBUG - run - Starting analysis...                                                                         2023-01-19 15:17:37,102 - DEBUG - skopeo - Attempting to pull image "docker-daemon:node:18.12.1"                                     2023-01-19 15:17:37,102 - DEBUG - rootfs - Running command: skopeo copy docker-daemon:node:18.12.1 dir:/root/.tern/temp
2023-01-19 15:18:05,324 - DEBUG - rootfs - Running command: tar -tf /root/.tern/temp/58861e9c7a9b85fcb52553413d8725a7ae8d215349b6325cfdb9d0c38b7f5583                                                                                                                     2023-01-19 15:18:05,375 - DEBUG - rootfs - Running command: tar -x -f /root/.tern/temp/58861e9c7a9b85fcb52553413d8725a7ae8d215349b6325cfdb9d0c38b7f5583 -C /root/.tern/temp/1/contents                                                                                    2023-01-19 15:18:06,022 - DEBUG - rootfs - Running command: /root/ternenv/lib/python3.10/site-packages/tern/tools/fs_hash.sh /root/.tern/temp/1/contents                                                                                                                  2023-01-19 15:18:18,438 - DEBUG - rootfs - Running command: tar -tf /root/.tern/temp/8a8dfd8ce7a77963ad1d0d8009a79f22f71c674bcf6198699bfd20ce868ff27b                                                                                                                     2023-01-19 15:18:18,601 - DEBUG - rootfs - Running command: tar -x -f /root/.tern/temp/8a8dfd8ce7a77963ad1d0d8009a79f22f71c674bcf6198699bfd20ce868ff27b -C /root/.tern/temp/2/contents
2023-01-19 15:18:21,603 - DEBUG - rootfs - Running command: /root/ternenv/lib/python3.10/site-packages/tern/tools/fs_hash.sh /root/.tern/temp/2/contents
2023-01-19 15:19:33,438 - DEBUG - common - Loading packages from cache: layer "9d22d5e891"
2023-01-19 15:19:33,932 - DEBUG - common - Loading files from cache: layer "9d22d5e891"
2023-01-19 15:19:35,235 - DEBUG - rootfs - Running command: /root/ternenv/bin/scancode -ilpcu --quiet --timeout 300 -n 1 --json - /root/.tern/temp/1/contents
Traceback (most recent call last):
  File "/root/ternenv/bin/tern", line 8, in <module>
    sys.exit(main())
  File "/root/ternenv/lib/python3.10/site-packages/tern/__main__.py", line 311, in main
    do_main(args)
  File "/root/ternenv/lib/python3.10/site-packages/tern/__main__.py", line 123, in do_main
    crun.execute_image(args)
  File "/root/ternenv/lib/python3.10/site-packages/tern/analyze/default/container/run.py", line 80, in execute_image
    cimage.default_analyze(full_image, args)
  File "/root/ternenv/lib/python3.10/site-packages/tern/analyze/default/container/image.py", line 70, in default_analyze
    passthrough.run_extension_layer(image_obj.layers[0], options.extend,
  File "/root/ternenv/lib/python3.10/site-packages/tern/analyze/passthrough.py", line 92, in run_extension_layer
    return mgr.driver.execute_layer(image_layer, redo)
  File "/root/ternenv/lib/python3.10/site-packages/tern/extensions/scancode/executor.py", line 219, in execute_layer
    file_list, package_list = collect_layer_data(image_layer)
  File "/root/ternenv/lib/python3.10/site-packages/tern/extensions/scancode/executor.py", line 162, in collect_layer_data
    files.append(get_scancode_file(f))
  File "/root/ternenv/lib/python3.10/site-packages/tern/extensions/scancode/executor.py", line 65, in get_scancode_file
    fd.copyrights = [c['value'] for c in file_dict['copyrights']]
  File "/root/ternenv/lib/python3.10/site-packages/tern/extensions/scancode/executor.py", line 65, in <listcomp>
    fd.copyrights = [c['value'] for c in file_dict['copyrights']]
KeyError: 'value'

# scancode --version; tern --version; python --version
ScanCode version: 31.2.4
ScanCode Output Format version: 2.0.0
SPDX License list version: 3.18
Tern version 2.11.0
   python version = 3.10.6 (main, Nov 14 2022, 16:10:14)
Python 3.10.6

[Jeeppler]

@rnjudge and @amallayev is there anything I can do to help to get this fixed?

[rnjudge]

@Jeeppler let me try to install the latest release of scancode. It's hard for me to debug and fix this when I can't install scancode :/

[rnjudge]

@Jeeppler I still cannot install scanacode which is quite concerning for me wrt Tern's integration with the library. I will try to setup a linux environment to debug on my old mac. Stay tuned.

[rnjudge]

@Jeeppler I am able to install scancode-toolkit-mini-31.2.6 and reproduce this error. I'll take a look now.

[rnjudge]

Hi @Jeeppler -- since I am unable to install scancode==30.* to compare the results of my fix, are you able to help? Could you run tern with scancode==30.* (the version before you saw this error) on alpine:latest and share the results with me so I can compare? It should be a 1:1 comparison but I want to make sure.

If you also want to test my changes, which should fix the error you are seeing, they are available here: https://github.com/rnjudge/tern/tree/issue-1202

Also cc amallayev. The issue was with changes to Scancode's JSON attribute names but should be fixed now.

[Jeeppler]

@rnjudge I will have a look, but probably towards the end of the week.