#StackBounty: #gcc #raspberry-pi #linker #arm #cross-compiling segfault before main() is called in cross-compiled armhf

Bounty: 50

I’ve finally cross-compiled and linked a binary on an Ubuntu Xenial x86_64 host to run on the Raspberry Pi 4’s armhf.

My toolchain’s from ARM and placed in $TOOLCHAIN.

My sysroot is a loop-mounted Raspberry OS image placed in $RASPBIAN_ROOT.

This is a sample compilation line:

$TOOLCHAIN/gcc-arm-8.3-2019.03/bin/arm-linux-gnueabihf-g++ -std=c++11 --sysroot=$RASPBIAN_ROOT 
  -D_ARM_ -DOTHER_DEFINES 
  -v -w -fexceptions -fpermissive -pipe 
  -mcpu=cortex-a72 -mfpu=neon-vfpv4 -mfloat-abi=hard 
  -Wabi-tag -D_GLIBCXX_USE_CXX11_ABI=0 
  -fno-use-cxa-atexit 
  -g 
  -I . -I .. -I extlib1/include -I extlib2/include -I $RASPBIAN_ROOT/usr/include/libxml2 
  -I $TOOLCHAIN/gcc-arm-8.3-2019.03/arm-none-linux-gnueabihf/libc/usr/include 
  -I $RASPBIAN_ROOT/usr/include/arm-linux-gnueabihf 
  -I $RASPBIAN_ROOT/usr/include 
  -c File.cpp -o obj/linux/armhf/debug/File.o

There are also some .c files. And this is the linking line:

$TOOLCHAIN/gcc-arm-8.3-2019.03/bin/arm-linux-gnueabihf-ld.gold 
-L ../localdependency 
-L $RASPBIAN_ROOT/opt/vc/lib 
-L $RASPBIAN_ROOT/usr/lib/arm-linux-gnueabihf 
-L $RASPBIAN_ROOT/lib 
-L$RASPBIAN_ROOT/usr/lib 
-L $RASPBIAN_ROOT/usr/lib/gcc/arm-linux-gnueabihf/8 -o bin/linux/armhf/debug/executable obj/linux/armhf/debug/File.o ... 
$RASPBIAN_ROOT/usr/lib/gcc/arm-linux-gnueabihf/8/crtbegin.o 
$RASPBIAN_ROOT/usr/lib/gcc/arm-linux-gnueabihf/8/crtend.o 
$RASPBIAN_ROOT/usr/lib/gcc/arm-linux-gnueabihf/8/libgcc_eh.a 
--verbose --sysroot=$RASPBIAN_ROOT 
-lbcm_host -lvcos -lvchiq_arm -lcurl -lxml2 
-lpthread -lz -lm -ldl -lstdc++ -lc -lgcc -lgcc_s

…which seems to produce an executable. (Both lines edited for readability.)

However, when running on the pi, i get a segfault:

$ ./executable param1 param2
Segmentation fault

If i try to debug, it’s not even reaching main():

$ gdb executable
GNU gdb (Raspbian 8.2.1-2) 8.2.1

...

(gdb) b main
Breakpoint 1 at 0x2c7c74: file main.cpp, line 7029.
(gdb) r param1 param2
Starting program: /home/user/executable param1 param2
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.

Program received signal SIGSEGV, Segmentation fault.
0xb6f97420 in __libc_enable_asynccancel () at ../nptl/cancellation.c:33
33  ../nptl/cancellation.c: No such file or directory.
(gdb) bt
#0  0xb6f97420 in __libc_enable_asynccancel () at ../nptl/cancellation.c:33
#1  0xb6f77968 in __GI___libc_write (nbytes=407, buf=0xb6fdc838 <banner>, fd=1) at ../sysdeps/unix/sysv/linux/write.c:26
#2  __GI___libc_write (fd=fd@entry=1, buf=0xb6fdc838 <banner>, nbytes=nbytes@entry=407) at ../sysdeps/unix/sysv/linux/write.c:24
#3  0xb6ec98d8 in __libc_print_version () at version.c:71
#4  __libc_main () at version.c:71
#5  0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) 

While i see the "warning: Unable to find dynamic linker breakpoint function." (and am trying to figure out where in the packages does ld.so come from) i’m more inclined to believe my toolchain/sysroot/libs is not properly configured.

A simple helloWorld cross-compiles and executes fine:

#include <iostream>

int main(int argc, char* argv[]) {
    int i = 1;
    std::cout << "hello world " << i << std::endl;
    return 0;
}
$ ./rpihello 
hello world 1

$ ldd rpihello 
    linux-vdso.so.1 (0xbef33000)
    /usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so (0xb6f90000)
    libstdc++.so.6 => /usr/lib/arm-linux-gnueabihf/libstdc++.so.6 (0xb6e30000)
    libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0xb6dae000)
    libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0xb6d81000)
    libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xb6c33000)
    /lib/ld-linux-armhf.so.3 (0xb6fa5000)

Edit 1

Since nptl/cancellation.c seems to come from libc, i double-checked i had all the necessary packages, and i believe i do (otherwise i’d be having widespread problems with the Pi):

pi$ cat /etc/ld.so.conf.d/*
/opt/vc/lib
# Multiarch support
/usr/local/lib/arm-linux-gnueabihf
/lib/arm-linux-gnueabihf
/usr/lib/arm-linux-gnueabihf
/usr/lib/arm-linux-gnueabihf/libfakeroot
# libc default configuration
/usr/local/lib

pi$ apt list --installed | grep ^libc
libc-ares2/now 1.14.0-1 armhf [installed,upgradable to: 1.14.0-1+deb10u1]
libc-bin/stable,now 2.28-10+rpi1 armhf [installed]
libc-dev-bin/stable,now 2.28-10+rpi1 armhf [installed,automatic]
libc-l10n/stable,now 2.28-10+rpi1 all [installed,automatic]
libc6-dbg/stable,now 2.28-10+rpi1 armhf [installed]
libc6-dev/stable,now 2.28-10+rpi1 armhf [installed]
libc6-pic/stable,now 2.28-10+rpi1 armhf [installed]
libc6/stable,now 2.28-10+rpi1 armhf [installed]
libcc1-0/stable,now 8.3.0-6+rpi1 armhf [installed,automatic]

pi$ apt list --installed | grep ^libgcc
libgcc-6-dev/stable,now 6.5.0-1+rpi1+b1 armhf [installed,automatic]
libgcc-8-dev/stable,now 8.3.0-6+rpi1 armhf [installed,automatic]
libgcc1-dbg/stable,now 1:8.3.0-6+rpi1 armhf [installed,automatic]
libgcc1/stable,now 1:8.3.0-6+rpi1 armhf [installed]

So i still think i borked the cross-compilation/linking somehow.


Edit 2

I’ve managed to compile my codebase on the raspberry pi and the produced binary exhibits the same results as the cross-compiled one. So either i’ve misscompiled or i have both my sysroot and the pi’s system a bit borked.


Edit 3

rpi $ readelf -dW myexecutable

Dynamic section at offset 0x664e58 contains 48 entries:
  Tag        Type                         Name/Value
 0x00000003 (PLTGOT)                     0x6716d4
 0x00000002 (PLTRELSZ)                   5856 (bytes)
 0x00000017 (JMPREL)                     0x2d940
 0x00000014 (PLTREL)                     REL
 0x00000011 (REL)                        0x2d2e0
 0x00000012 (RELSZ)                      1632 (bytes)
 0x00000013 (RELENT)                     8 (bytes)
 0x00000015 (DEBUG)                      0x0
 0x00000006 (SYMTAB)                     0x8148
 0x0000000b (SYMENT)                     16 (bytes)
 0x00000005 (STRTAB)                     0xfb48
 0x0000000a (STRSZ)                      94185 (bytes)
 0x6ffffef5 (GNU_HASH)                   0x26b34
 0x00000004 (HASH)                       0x2930c
 0x00000001 (NEEDED)                     Shared library: [libbcm_host.so]
 0x00000001 (NEEDED)                     Shared library: [libvcos.so]
 0x00000001 (NEEDED)                     Shared library: [libvchiq_arm.so]
 0x00000001 (NEEDED)                     Shared library: [libjson.so]
 0x00000001 (NEEDED)                     Shared library: [libmylocallib.so]
...
 0x00000001 (NEEDED)                     Shared library: [libpthread.so.0]
 0x00000001 (NEEDED)                     Shared library: [libz.so.1]
 0x00000001 (NEEDED)                     Shared library: [libm.so.6]
 0x00000001 (NEEDED)                     Shared library: [libdl.so.2]
 0x00000001 (NEEDED)                     Shared library: [libstdc++.so.6]
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
 0x00000019 (INIT_ARRAY)                 0x66dd60
 0x0000001b (INIT_ARRAYSZ)               244 (bytes)
 0x0000001a (FINI_ARRAY)                 0x66de54
 0x0000001c (FINI_ARRAYSZ)               4 (bytes)
 0x6ffffff0 (VERSYM)                     0x2c1b0
 0x6ffffffe (VERNEED)                    0x2d0f0
 0x6fffffff (VERNEEDNUM)                 9
 0x00000000 (NULL)                       0x0

I found this interesting:

rpi $ file myexecutable
myexecutable: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /usr/lib/libc.so.1, with debug_info, not stripped
rpi $ file `which ls`
/bin/ls: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=67a394390830ea3ab4e83b5811c66fea9784ee69, stripped

Shouldn’t my executable be using ld as the interpreter?


Edit 4

I added[2] --dynamic-linker=/lib/ld-linux-armhf.so.3 to my linker line, but now i get no stack:

Program received signal SIGSEGV, Segmentation fault.
0x00000000 in ?? ()
(gdb) bt
#0  0x00000000 in ?? ()
(gdb) 

Running strace executable the last lines are these:

rt_sigprocmask(SIG_BLOCK, NULL, ~[ILL TRAP BUS FPE KILL SEGV STOP RTMIN RT_1], 8) = 0
rt_sigaction(SIGILL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0xb5e55120}, NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
clock_gettime(CLOCK_MONOTONIC_COARSE, {tv_sec=6336, tv_nsec=768744979}) = 0
gettimeofday({tv_sec=1631533208, tv_usec=497816}, NULL) = 0
openat(AT_FDCWD, "/dev/urandom", O_RDONLY) = 3
read(3, "3144bJn3616240f203216I257306251r202v232263210374222276TN33,[3A233", 32) = 32
close(3)                                = 0
openat(AT_FDCWD, "/dev/urandom", O_RDONLY) = 3
read(3, "301231203~G6Ey_262Y24134216F2448w2365Z213376223260322zwo334+e", 32) = 32
close(3)                                = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=NULL} ---
+++ killed by SIGSEGV +++
Segmentation fault

/dev/urandom does exist though.


Edit 5

$ readelf -h executable
ELF Header:
  Magic:   7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF32
  Data:                              2's complement, little endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           ARM
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          52 (bytes into file)
  Start of section headers:          40514564 (bytes into file)
  Flags:                             0x5000400, Version5 EABI, hard-float ABI
  Size of this header:               52 (bytes)
  Size of program headers:           32 (bytes)
  Number of program headers:         8
  Size of section headers:           40 (bytes)
  Number of section headers:         37
  Section header string table index: 36

I don’t think entry point address should be 0x0.


Edit 6

I’ve compiled the code on a pi3 and on an arm-qemu chroot – same result.


Get this bounty!!!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.