Recent kernel causes -fPIE ASan executables to abort on x86_64

[jcowgill]

This is just a heads-up about this Linux kernel commit recently committed and pending on a number of stable queues:
torvalds/linux@eab0953

It seems to adjust move the default load address for -fPIE executables into the location ASan uses for its shadow memory map (on x86_64). This then causes ASan to abort on startup. Example error:

$ ./a.out
==5661==Shadow memory range interleaves with an existing memory mapping. ASan cannot proceed correctly. ABORTING.
==5661==ASan shadow was supposed to be located in the [0x00007fff7000-0x10007fff7fff] range.
==5661==Process memory map follows:
	0x000cb5280000-0x000cb5281000	/var/tmp/a.out
	0x000cb5480000-0x000cb5481000	/var/tmp/a.out
	0x000cb5481000-0x000cb5482000	/var/tmp/a.out
	0x7f6d4f9ca000-0x7f6d4fd1c000	
	0x7f6d4fd1c000-0x7f6d4fd32000	/lib/x86_64-linux-gnu/libgcc_s.so.1
	0x7f6d4fd32000-0x7f6d4ff31000	/lib/x86_64-linux-gnu/libgcc_s.so.1
	0x7f6d4ff31000-0x7f6d4ff32000	/lib/x86_64-linux-gnu/libgcc_s.so.1
	0x7f6d4ff32000-0x7f6d4ff33000	/lib/x86_64-linux-gnu/libgcc_s.so.1
[...]

With ASLR enabled, you can sometimes get lucky with the load address and the program runs, but most of the time ASan aborts with this error.

Is it possible for ASan to be a bit more flexible about where it places the shadow map on startup to fix this?

[kcc]

It is possible at the code size and execution time cost, which we are not willing to pay.
Any chance to get the kernel to cooperate?

[kcc]

This would not be the first time when the kernel change breaks the sanitizers.
The last significant one was by H.J. Lu when he changed the based from 0x7.... to 0x555....
It caused lots of trouble for us in msan and tsan.

What we really need here is to tell at link time where the shadow is.
AFAICT, there is no such capability currently.

[rnk]

I always wondered if it would be possible to express the shadow mapping as an ELF program header. That would be the ultimate way to communicate shadow memory needs to the kernel.

[jcowgill]

I'm not sure - I'm just a user who happened to stumble across the bug. You might be able to get them to change where the executable gets mapped, but they could argue that PIE executables should be prepared to be loaded at any address.

What we really need here is to tell at link time where the shadow is.

I don't see how that is possible with PIE / ASLR. The entire point is that you don't know where the executable will be loaded, so you can't know what bits of memory will be free until runtime.

[kcc]

@dvyukov @xairy @ramosian-glider FYI

[pcc]

We could have a program header that means "please reserve the first N bytes of the address space for the application". Then the kernel can use that as a minimum for ELF_ET_DYN_BASE.

[kcc]

@dvyukov can you confirm that the fresh kernel breaks the sanitizers?

[bennofs]

I think I am hitting this bug:

$  ./loadaddr 
==16572==Shadow memory range interleaves with an existing memory mapping. ASan cannot proceed correctly. ABORTING.
==16572==ASan shadow was supposed to be located in the [0x00007fff7000-0x10007fff7fff] range.
==16572==Process memory map follows:
	0x04daa6e91000-0x04daa6fc6000	/tmp/loadaddr
	0x04daa71c6000-0x04daa71c7000	/tmp/loadaddr
	0x04daa71c7000-0x04daa71ca000	/tmp/loadaddr
	0x04daa71ca000-0x04daa7e2f000	
	0x7b742c072000-0x7b742c3c4000	
	0x7b742c3c4000-0x7b742c559000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libc-2.25.so
	0x7b742c559000-0x7b742c759000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libc-2.25.so
	0x7b742c759000-0x7b742c75d000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libc-2.25.so
	0x7b742c75d000-0x7b742c75f000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libc-2.25.so
	0x7b742c75f000-0x7b742c763000	
	0x7b742c763000-0x7b742c779000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libgcc_s.so.1
	0x7b742c779000-0x7b742c978000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libgcc_s.so.1
	0x7b742c978000-0x7b742c979000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libgcc_s.so.1
	0x7b742c979000-0x7b742c97c000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libdl-2.25.so
	0x7b742c97c000-0x7b742cb7b000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libdl-2.25.so
	0x7b742cb7b000-0x7b742cb7c000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libdl-2.25.so
	0x7b742cb7c000-0x7b742cb7d000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libdl-2.25.so
	0x7b742cb7d000-0x7b742cc8e000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libm-2.25.so
	0x7b742cc8e000-0x7b742ce8e000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libm-2.25.so
	0x7b742ce8e000-0x7b742ce8f000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libm-2.25.so
	0x7b742ce8f000-0x7b742ce90000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libm-2.25.so
	0x7b742ce90000-0x7b742ce97000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/librt-2.25.so
	0x7b742ce97000-0x7b742d096000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/librt-2.25.so
	0x7b742d096000-0x7b742d097000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/librt-2.25.so
	0x7b742d097000-0x7b742d098000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/librt-2.25.so
	0x7b742d098000-0x7b742d0b1000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libpthread-2.25.so
	0x7b742d0b1000-0x7b742d2b0000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libpthread-2.25.so
	0x7b742d2b0000-0x7b742d2b1000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libpthread-2.25.so
	0x7b742d2b1000-0x7b742d2b2000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/libpthread-2.25.so
	0x7b742d2b2000-0x7b742d2b6000	
	0x7b742d2b6000-0x7b742d2d9000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/ld-2.25.so
	0x7b742d4a8000-0x7b742d4bc000	
	0x7b742d4c0000-0x7b742d4d9000	
	0x7b742d4d9000-0x7b742d4da000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/ld-2.25.so
	0x7b742d4da000-0x7b742d4db000	/nix/store/l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25/lib/ld-2.25.so
	0x7b742d4db000-0x7b742d4dc000	
	0x7fff06e24000-0x7fff06e46000	[stack]
	0x7fff06ef0000-0x7fff06ef2000	[vvar]
	0x7fff06ef2000-0x7fff06ef4000	[vdso]
	0xffffffffff600000-0xffffffffff601000	[vsyscall]
==16572==End of process memory map.
c-cube:/tmp uname -a
Linux c-cube 4.9.39 #1-NixOS SMP Fri Jul 21 05:42:36 UTC 2017 x86_64 GNU/Linux

(Just compiled a trivial Hello world with -fsanitize=address)

[bennofs]

A possible workaround seems to be the following:

$  .../ld-2.25.so ./loadaddr

That way, loadaddr will be loaded by ld.so, which uses mmap so loadaddr ends up in the mmap region which is way higher than the PIE base.

(Yes, my ld.so is in weird path, that's just NixOS things :)

[FSMaxB]

I independently bisected this in the kernel and opened a bug there: https://bugzilla.kernel.org/show_bug.cgi?id=196537 but didn't have a lot of knowledge about the underlying issues.

[richfelker]

Bringing this over from twitter (https://twitter.com/kayseesee/status/894594085608013825), my basic view is that this is a bug in the ASAN library code. Assuming you can use a particular virtual address range is not valid (it could already be in use for some reason, as you're now seeing), and even if it were valid, it's not safe for something that can be used in deployment; it exposes potentially sensitive information at an attacker-known address. ASAN simply needs to pay the cost of using a variable address chosen at runtime.

[kcc]

@richfelker ASAN has been using fixed addresses since 2011.
I know kernel does not guarantee anything like this, but it worked, and it provided performance and code size benefits over using a dynamic shadow base (which we also have now, as an option, off by default on linux)

ASAN simply needs to pay the cost of using a variable address chosen at runtime.

That's one way to look at it. But a much better resolution would be to have a kernel<=>userspace interface that allows to use a fixed address. And in the meantime, revert the change that broke ASAN.

safe for something that can be used in deployment

If you want to discuss this topic, please open a separate issue, let's not mix too many things in a single place.

[richfelker]

Like I said on on the initial Twitter thread, I don't think I have much of value to say beyond "I think what you're doing is badly wrong" and "it happened to work before is not a good argument to do it (or for changing the kernel)". If we disagree then we disagree...