upload.py can use system-widely installed esptool instead of expected one

[shimarin]

With change 472faf7, python interpreter uses system-wide esptool(can have been installed by system's package manager) instead of one under tools/ . This may cause unexpected behavior.

[shimarin]

This patch works for me.

--- a/tools/upload.py
+++ b/tools/upload.py
@@ -12,8 +12,8 @@ import os
 sys.argv.pop(0) # Remove executable name
 toolspath = os.path.dirname(os.path.realpath(__file__)).replace('\\', '/') # CWD in UNIX format
 try:
-    sys.path.append(toolspath + "/pyserial") # Add pyserial dir to search path
-    sys.path.append(toolspath + "/esptool") # Add esptool dir to search path
+    sys.path.insert(0, toolspath + "/pyserial") # Add pyserial dir to search path
+    sys.path.insert(0, toolspath + "/esptool") # Add esptool dir to search path
     import esptool # If this fails, we can't continue and will bomb below
 except:
     sys.stderr.write("Error in command line, need pyserial path as 1st arg and esptool path as 2nd.\n")

[earlephilhower]

Seems like a fine solution, @shimarin . I didn't think about pre-existing installed versions of the tools/libs.

Could you create a PR that I could merge and get you credit for it?

[Rob58329]

On my Windows10 PC, I download the "https://github.com/esp8266/Arduino" zip file, and then unzip and manually copy the download into my "arduino-1.8.9\hardware" directory. From command line I then run the "arduino-1.8.9\hardware\esp8266\esp8266\tools\get.py" install file.

HOWEVER, using this method the "get.py" does NOT populate the "esptool" or the "pyserial" subdirectories.

So, if you try to "upload" anything in the Arduino IDE, you get the error message:
"Error in command line, need pyserial path as 1st arg and esptool path as 2nd."

To fix this I copied the contents of an old ESP8266-Ardiono-IDE installation's "esptool" and "pyserial" subdirectories (ESP8266 Board v2.5.2 originally installed using the Arduino-Boards-Manager with "http://arduino.esp8266.com/stable/package_esp8266com_index.json" in the "Preferences/Additional Board Manager URLs" box).

I can now upload fine.

Update (11Sep19):
From @earlephilhower comments below of 10Sep19 ("The alternative is to download ZIPfiles..."), the mentioned github repositories are:

• https://github.com/espressif/esptool
• & https://github.com/pyserial/pyserial

You have to unzip, and then manually copy the contents into the empty arduino directories:

• hardware\esp8266\esp8266\tools\esptool
• & hardware\esp8266\esp8266\tools\pyserial

This also works for me, but note @earlephilhower's warnings below (of 10Sep19).

I hope this info will be useful for someone.

[earlephilhower]

The problem you had is because the .ZIP from github does not populate the git submodules (basically remote references to some other repo). Don't do what you suggested, that's not the way to get a working, sane version.

You need to do a proper "git clone" from the command line of the repo, then use the "git submodule init" and "git submodule update" referenced in the README.md in order to get the consistent versions of these external references. I don't think there's a way to get a working setup from a .ZIP file without way more work and chance of error.

[Rob58329]

@earlephilhower: Very many thanks for your explanation+comments. I understand that you are suggesting installing "Git for Windows" (which I believe then requires supply of a Name & email-address to use), and then instead of using the .zip-download, doing a full "Git Clone" and then a "Git submodule init" and "Git submodule update" to get the missing "esptool" and "pyserial" subdirectories. Ideally I would not want to install yet another piece of software (and supply email-addresses etc) and learn how to use it, just in order to get 2 missing subdirectories.

Might I suggest you therefore perhaps consider including the old Espressive "esptool.exe" file in the standard "zip download" (as was done originally). And then modifing the "platform.txt" file so in addition to your new "esptool.py" based software upload code, you include the old code which used the old "upload.exe" file, in such a way that a user can simply edit the "platform.txt" file to change an "if (true)" to "if (false)" (if possible, or else just add and remove '#' lines), to swap from using your new "esptool.py" file back to the original "esptool.exe" file (which would then not require the install of "Git for Windows" with all the additional actions)?

This might help a number of users, as I note there seem to be several internet threads requesting a solution for this issue.

(PS. I suspect it might also be possible to instead modify your "upload.py" file so if the "import esptool" line fails, instead of just reporting a failure and exiting, it automatically tried instead to use the old "esptool.exe" method - but I'm not sure how much more work this would be...)

[earlephilhower]

Git is open source, free, and doesn't ask for an email or anything. It lets us pull in other open source work (esptool.py is supported by Espressif and actively developed vs. esptool.exe which is ancient, buggy, and unsupported) without having to directly copy files, so when they fix a bug we can get the fix easily.

GIT is a great, free source code control system for your own projects, too, and well worth spending the few minutes to get a basic understanding.

The alternative is to download ZIPfiles for the SWSerial and ESPtool.py archives and place them manually in the tools/ directory. It's not the right way of doing things, and a real pain in the butt to get right.