Intermittent "Permission denied" on Windows

[Manishearth]

Example: http://buildbot.rust-lang.org/builders/auto-win-64-nopt-t/builds/2223

error: linking with `gcc` failed: exit code: 1
note: "gcc" '"-Wl,--enable-long-section-names"' '"-fno-use-linker-plugin"' '"-Wl,--nxcompat"' '"-static-libgcc"' '"-m64"' '"-L"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib"' '"-o"' '"x86_64-pc-windows-gnu\\stage2\\test\\stdtest-x86_64-pc-windows-gnu.exe"' '"x86_64-pc-windows-gnu\\stage2\\test\\stdtest-x86_64-pc-windows-gnu.o"' '"-Wl,--gc-sections"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\libtest-4e7c5e5c.rlib"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\libgetopts-4e7c5e5c.rlib"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\libterm-4e7c5e5c.rlib"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\libserialize-4e7c5e5c.rlib"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\liblog-4e7c5e5c.rlib"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\libstd-4e7c5e5c.rlib"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\libcollections-4e7c5e5c.rlib"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\librand-4e7c5e5c.rlib"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\liballoc-4e7c5e5c.rlib"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\liblibc-4e7c5e5c.rlib"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\libunicode-4e7c5e5c.rlib"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib\\libcore-4e7c5e5c.rlib"' '"-L"' '"x86_64-pc-windows-gnu\\rt"' '"-L"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\llvm\\Release+Asserts\\lib"' '"-L"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\x86_64-pc-windows-gnu\\stage2\\bin\\rustlib\\x86_64-pc-windows-gnu\\lib"' '"-L"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\.rust\\bin\\x86_64-pc-windows-gnu"' '"-L"' '"C:\\bot\\slave\\auto-win-64-nopt-t\\build\\obj\\bin\\x86_64-pc-windows-gnu"' '"-Wl,--whole-archive"' '"-Wl,-Bstatic"' '"-Wl,--no-whole-archive"' '"-Wl,-Bdynamic"' '"-lws2_32"' '"-luserenv"' '"-lkernel32"' '"-lws2_32"' '"-luserenv"' '"-lcompiler-rt"'
note: C:/program files/mingw-w64/x86_64-4.9.1-win32-seh-rt_v3-rev1/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/4.9.1/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot open output file x86_64-pc-windows-gnu\stage2\test\stdtest-x86_64-pc-windows-gnu.exe: Permission denied
collect2.exe: error: ld returned 1 exit status

It doesn't seem to happen in the same place at every time, but basically an exe (usually stdtest) can't be opened for whatever reason.

cc @retep998 @klutzy

[klutzy]

I've thought it's basically this: for some reason stdtest hanged at previous test, then during the second build linker fails to open exe since it's still running.
(The comment only said run-pass tests but while investigating it I also found one some ***test hanged.)

[klutzy]

Ping @alexcrichton: I think you can access to buildbot directly. Could you dump thread/stack status of hanged process (when this occurs again) using Process Explorer? This article explains how to see list of threads and stack trace of each thread.

[Manishearth]

Some various failures this caused (most of these today!):

• http://buildbot.rust-lang.org/builders/auto-win-32-nopt-t/builds/3387
• http://buildbot.rust-lang.org/builders/auto-win-32-nopt-t/builds/3388
• http://buildbot.rust-lang.org/builders/auto-win-64-opt/builds/2230
• http://buildbot.rust-lang.org/builders/auto-win-64-nopt-t/builds/2222
• http://buildbot.rust-lang.org/builders/auto-win-64-nopt-t/builds/2223
• http://buildbot.rust-lang.org/builders/auto-win-64-nopt-t/builds/2224
• http://buildbot.rust-lang.org/builders/auto-win-64-nopt-t/builds/2228
• http://buildbot.rust-lang.org/builders/auto-win-64-nopt-t/builds/2229

This "intermittent" is really becoming a problem, can we prioritize this?

[Manishearth]

Seems like 99b39cc should have fixed it, but d0de2b4 broke it.

We tried b6044c7 but it caused a panic

Edit: I'm talking about the wrong failure here.

[brson]

@Manishearth suspects d0de2b4 may caused the recent spike.

[brson]

I added a patch to buildbot that attempts to kill everything named stdtest before running make check, and changed the max builds per windows slave to 1 rust-lang-deprecated/rust-buildbot@9911005

[brson]

This is how mariadb creates buildbot jobs https://mariadb.com/kb/en/mariadb/buildbot-setup-buildbot-setup-for-windows/

[alexcrichton]

@klutzy is correct in the diagnosis. The fact that stdtest hangs is pretty new to me and I haven't had the chance to investigate it much.

Note that the fix for all the tests in @klutzy's linked comment was to join on all threads. I had a patch to use scoped but it should have instead been spawn + join. @Manishearth I see a few linked commits to this issue which add calls to sleep, but I believe it is better solved with a join.

Could you dump thread/stack status of hanged process (when this occurs again) using Process Explorer? This article explains how to see list of threads and stack trace of each thread.

I took a peek, and unfortunately all I could deduce was that a thread was blocked in accept. I was unable to get any indication of which accept it was blocked in (no symbols were available).

and changed the max builds per windows slave to 1 rust-lang-deprecated/rust-buildbot@9911005

@brson, why change the max builds? I'm under the impression that the error does not come from this but rather the fact that something just hangs for too long. Killing stdtest at the start sounds like a good idea though. Note that this was my impetus for opening #22448.

[Manishearth]

@alexcrichton We can't join() when the child has panicked. That's why I added a sleep()

[Manishearth]

@alexcrichton Max builds has to be one, we're indiscriminately killing all stdtest processes without knowing which build they came from. Peter is working on a job object based solution so that we don't have to do that.