Auto-generated `__MACOSX` folder in tool archive causes `core install` to fail: "`Cannot install platform: searching package root dir: no unique root dir in archive`

[kittaakos]

Describe the problem

I could not install heltec-esp8266:esp8266 with IDE2. It's probably a CLI issue but let's make sure.

Downloading packages
Installing heltec-esp8266:xtensa-lx106-elf-gcc@2.5.0-4-b40a506
heltec-esp8266:xtensa-lx106-elf-gcc@2.5.0-4-b40a506 installed
Installing heltec-esp8266:mkspiffs@2.5.0-4-b40a506
heltec-esp8266:mkspiffs@2.5.0-4-b40a506 installed
Installing heltec-esp8266:mklittlefs@2.5.0-4-69bd9e6
heltec-esp8266:mklittlefs@2.5.0-4-69bd9e6 installed
Installing heltec-esp8266:python3@3.7.2-post1
heltec-esp8266:python3@3.7.2-post1 installed
Installing platform heltec-esp8266:esp8266@0.0.4
Failed to install platform: heltec-esp8266:esp8266.
Error: 13 INTERNAL: Cannot install platform: installing platform heltec-esp8266:esp8266@0.0.4: searching package root dir: no unique root dir in archive, found '/Users/a.kitta/Library/Arduino15/tmp/package-3672416258/__MACOSX' and '/Users/a.kitta/Library/Arduino15/tmp/package-3672416258/esp8266-0.0.4'

To reproduce

I have the following URLs

http://digistump.com/package_digistump_index.json,http://www.leonardomiliani.com/repository/package_leonardomiliani.com_index.json,https://ambasat.com/boards/package_ambasat-1.com_index.json,https://arduino.esp8266.com/stable/package_esp8266com_index.json,https://dl.espressif.com/dl/package_esp32_index.json,https://files.seeedstudio.com/arduino/package_seeeduino_boards_index.json,https://mcudude.github.io/MightyCore/package_MCUdude_MightyCore_index.json,https://rabidprototypes.com/arduino/package_rabidprototypes_index.json,https://raw.githubusercontent.com/Lauszus/Sanguino/master/package_lauszus_sanguino_index.json,https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_index.json,https://resource.heltec.cn/download/package_CubeCell_index.json,https://resource.heltec.cn/download/package_heltec_esp32_index.json,https://resource.heltec.cn/download/package_heltec_esp8266_index.json

Install the platform from the IDE2.

Expected behavior

The archives produced by the macOS built-in archive utility can be used in Arduino boards packages.

Arduino IDE version

HEAD

Operating system

macOS

Operating system version

12.3.1

Additional context

Error log:

daemon INFO {"error":"installing platform heltec-esp8266:esp8266@0.0.4: searching package root dir: no unique root dir in archive, found '/Users/a.kitta/Library/Arduino15/tmp/package-3672416258/__MACOSX' and '/Users/a.kitta/Library/Arduino15/tmp/package-3672416258/esp8266-0.0.4'","level":"error","msg":"Cannot install platform","platform":{"ID":"heltec-esp8266:esp8266","Installed":"0.0.4","Latest":"0.0.4","Name":"Heltec ESP8266 Series Dev-boards"},"time":"2022-07-02T23:04:57+02:00"}

root ERROR Request install failed with error: 13 INTERNAL: Cannot install platform: installing platform heltec-esp8266:esp8266@0.0.4: searching package root dir: no unique root dir in archive, found '/Users/a.kitta/Library/Arduino15/tmp/package-3672416258/__MACOSX' and '/Users/a.kitta/Library/Arduino15/tmp/package-3672416258/esp8266-0.0.4' Error: 13 INTERNAL: Cannot install platform: installing platform heltec-esp8266:esp8266@0.0.4: searching package root dir: no unique root dir in archive, found '/Users/a.kitta/Library/Arduino15/tmp/package-3672416258/__MACOSX' and '/Users/a.kitta/Library/Arduino15/tmp/package-3672416258/esp8266-0.0.4'
    at Object.callErrorFromStatus (/Users/a.kitta/dev/git/arduino-ide/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
    at Object.onReceiveStatus (/Users/a.kitta/dev/git/arduino-ide/node_modules/@grpc/grpc-js/build/src/client.js:349:49)
    at Object.onReceiveStatus (/Users/a.kitta/dev/git/arduino-ide/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:328:181)
    at /Users/a.kitta/dev/git/arduino-ide/node_modules/@grpc/grpc-js/build/src/call-stream.js:187:78
    at processTicksAndRejections (node:internal/process/task_queues:78:11)

I have installed another heltec platform.

Related

• Installation of chipKIT:pic32 platform fails: "no unique root dir in archive" #2131
• core install fails when a core archive doesn't have a single base folder #325
• cannot install arduino:stm32f4 #153

Issue checklist

• I searched for previous reports in the issue tracker
• I verified the problem still occurs when using the latest nightly build
• My report contains all necessary details

[per1234]

This caused by the heltec-esp8266:esp8266@0.0.4 boards platform archive structure violating a limitation imposed by Arduino CLI:

$ arduino-cli version
arduino-cli.exe  Version: git-snapshot Commit: 9b6c9c46 Date: 2022-07-03T15:00:54Z

$ export ARDUINO_BOARD_MANAGER_ADDITIONAL_URLS=https://resource.heltec.cn/download/package_heltec_esp8266_index.json

$ arduino-cli core update-index

[...]

$ arduino-cli.exe core install heltec-esp8266:esp8266@0.0.4
Downloading packages...
heltec-esp8266:xtensa-lx106-elf-gcc@2.5.0-4-b40a506 already downloaded
heltec-esp8266:mkspiffs@2.5.0-4-b40a506 already downloaded
heltec-esp8266:mklittlefs@2.5.0-4-69bd9e6 already downloaded
heltec-esp8266:python3@3.7.2-post1 already downloaded
heltec-esp8266:esp8266@0.0.4 downloaded
Installing heltec-esp8266:xtensa-lx106-elf-gcc@2.5.0-4-b40a506...
heltec-esp8266:xtensa-lx106-elf-gcc@2.5.0-4-b40a506 installed
Installing heltec-esp8266:mkspiffs@2.5.0-4-b40a506...
heltec-esp8266:mkspiffs@2.5.0-4-b40a506 installed
Installing heltec-esp8266:mklittlefs@2.5.0-4-69bd9e6...
heltec-esp8266:mklittlefs@2.5.0-4-69bd9e6 installed
Installing heltec-esp8266:python3@3.7.2-post1...
heltec-esp8266:python3@3.7.2-post1 installed
Installing platform heltec-esp8266:esp8266@0.0.4...
Error during install: Cannot install platform: installing platform heltec-esp8266:esp8266@0.0.4: searching package root dir: no unique root dir in archive, found 'E:\deleteme\arduino-cli\directories\data\tmp\package-2598651179\__MACOSX' and 'E:\deleteme\arduino-cli\directories\data\tmp\package-2598651179\esp8266-0.0.4'

$ wget --output-document="/tmp/esp8266-0.0.4.zip" $(curl $ARDUINO_BOARD_MANAGER_ADDITIONAL_URLS | jq -r .packages[0].platforms[0].url)

[...]

$ unzip -d "/tmp/esp8266-0.0.4" "/tmp/esp8266-0.0.4.zip"

[...]

$ tree -A -F -L 2 "/tmp/esp8266-0.0.4/"
/tmp/esp8266-0.0.4/
├── __MACOSX/
│   └── esp8266-0.0.4/
└── esp8266-0.0.4/
    ├── ISSUE_TEMPLATE.md
    ├── LICENSE
    ├── POLICY.md
    ├── boards.txt
    ├── bootloaders/
    ├── cores/
    ├── keywords.txt
    ├── libraries/
    ├── package.json
    ├── platform.txt
    ├── programmers.txt
    ├── tests/
    ├── tools/
    └── variants/

Note that there are two folders in the root of the platform archive ZIP file. Like this:

esp8266-0.0.4.zip
├── __MACOSX/
└── esp8266-0.0.4/

Arduino CLI allows only a single root folder, which contains the platform:

esp8266-0.0.4.zip
└── esp8266-0.0.4/
    ├── ISSUE_TEMPLATE.md
    ├── LICENSE
    ├── POLICY.md
    ├── boards.txt
    ├── bootloaders/
    ├── cores/
    ├── keywords.txt
    ├── libraries/
    ├── package.json
    ├── platform.txt
    ├── programmers.txt
    ├── tests/
    ├── tools/
    └── variants/

This __MACOSX folder is an annoying quirk of the native macOS archiver. The folder is hidden on macOS, so the users often are not even aware that the structure of the ZIP files they produce has been tampered with.

The same issue previously occurred when installing libraries via arduino-cli lib install --zip-path when the ZIP file had this structure. An exception for the folder in the library archives was added to Arduino CLI (#1228).

Arduino IDE 1.x allows this extra __MACOSX folder in the platform archive.
ⓘ If testing with Arduino IDE 1.x, you may need to use the new package index URL, as specified in the heltec-esp8266:esp8266 platform's installation instructions, as I found that the outdated package index URL mentioned in the report is not compatible with Arduino IDE 1.x ("Error downloading https://resource.heltec.cn/download/package_heltec_esp8266_index.json").

So I think it would be worth adding support to Arduino CLI for this platform archive structure. If not, the package index specification could be made to more clearly describe the required archive structure:

https://arduino.github.io/arduino-cli/dev/package_index_json-specification/#installation-archive-structure

Related

• core install fails when a core archive doesn't have a single base folder #325

[cmaglie]

We may add the exception for __MACOSX and other folder names that seems to appear in the archives artifacts (maybe hidden files with . prefix too?). The exceptions will be added as part of the specification.