Hi guys, I compile a native gdb using linaro 2011.10 by “./configure --host=arm-none-linux-gnueabi --target=arm-none-linux-gnueabi”, and the gdb runs on arm target boards directly. # gdb GNU gdb (Linaro GDB) 7.3-2011.10 Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "arm-none-linux-gnueabi". For bug reporting instructions, please see: http://bugs.launchpad.net/gdb-linaro/. (gdb)
I can use it to debug native programs on target boards directly. For example, attach process, set breakpoints, check registers and memory. One issue is I can't see multi-threads, for example:
PID 646 is system_server by ps "646 1000 159m S system_server"
Then I use gdb to attach it:
# gdb attach 646 (gdb) info threads Id Target Id Frame * 1 process 646 "system_server" __ioctl () at bionic/libc/arch-arm/syscalls/__ioctl.S:15
as you see, “info threads” only shows one thread but there are several threads in system_server.
But if I compile a new program based on glibc and gnu libthread, I can see multi-threads by the gdb.
So my questions are:
1. Should I compile the native gdb using android toolchain and android bionic/libthread libraries? 2. Why can’t the current gdb capture multithreads for android processes? This question is actually about the theory for gdb to know multi-threads. In my opinion, both gnu and android use clone() to fork threads and threads in one process have same tgid in kernel and all threads return same getpid() value. Why not gdb just travel process lists to find multi-threads?
Thanks Barry
Barry Song 21cnbao@gmail.com wrote:
So my questions are:
- Should I compile the native gdb using android toolchain and android
bionic/libthread libraries? 2. Why can’t the current gdb capture multithreads for android processes? This question is actually about the theory for gdb to know multi-threads. In my opinion, both gnu and android use clone() to fork threads and threads in one process have same tgid in kernel and all threads return same getpid() value. Why not gdb just travel process lists to find multi-threads?
I'm not sure what exactly is going on on Android with bionic. However, GDB currently does require support from the thread library in order to debug multi-threaded applications. This is needed e.g. to properly handle thread-local storage variables. GDB will also use this support to detect threads of a running process it is attaching to.
(It is true that GDB *could* e.g. look at /proc to find threads, instead of inspecting thread library data structures. However, since the link to those data structures is required anyway, e.g. for TLS, this has not been implemented so far ...)
When using glibc's libpthread, the support routines gdb uses to inspect target thread library datastructures are provided in libthread_db.so.1 (which comes with glibc, and is linked into gdb). I do not know the details on whether/how bionic provides a corresponding service.
However, from looking at the gdbserver sources provided with Android, it seems there are some differences; in particular, there's this patch:
+/* Android doesn't have libthread_db.so.1, just libthread_db.so. */ +#ifdef __ANDROID__ +#define LIBTHREAD_DB_SO "libthread_db.so" +#endif +
If libthread_db is named differently, this would explain why GDB is unable to find and use it.
Mit freundlichen Gruessen / Best Regards
Ulrich Weigand
-- Dr. Ulrich Weigand | Phone: +49-7031/16-3727 STSM, GNU compiler and toolchain for Linux on System z and Cell/B.E. IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz | Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen | Registergericht: Amtsgericht Stuttgart, HRB 243294
Hi Ulrich,
2012/2/3 Ulrich Weigand Ulrich.Weigand@de.ibm.com:
Barry Song 21cnbao@gmail.com wrote:
So my questions are:
- Should I compile the native gdb using android toolchain and android
bionic/libthread libraries? 2. Why can’t the current gdb capture multithreads for android processes? This question is actually about the theory for gdb to know multi-threads. In my opinion, both gnu and android use clone() to fork threads and threads in one process have same tgid in kernel and all threads return same getpid() value. Why not gdb just travel process lists to find multi-threads?
I'm not sure what exactly is going on on Android with bionic. However, GDB currently does require support from the thread library in order to debug multi-threaded applications. This is needed e.g. to properly handle thread-local storage variables. GDB will also use this support to detect threads of a running process it is attaching to.
(It is true that GDB *could* e.g. look at /proc to find threads, instead of inspecting thread library data structures. However, since the link to those data structures is required anyway, e.g. for TLS, this has not been implemented so far ...)
When using glibc's libpthread, the support routines gdb uses to inspect target thread library datastructures are provided in libthread_db.so.1 (which comes with glibc, and is linked into gdb). I do not know the details on whether/how bionic provides a corresponding service.
However, from looking at the gdbserver sources provided with Android, it seems there are some differences; in particular, there's this patch:
+/* Android doesn't have libthread_db.so.1, just libthread_db.so. */ +#ifdef __ANDROID__ +#define LIBTHREAD_DB_SO "libthread_db.so" +#endif
If libthread_db is named differently, this would explain why GDB is unable to find and use it.
there are two ways to handle this issue: 1. ln -s /system/lib/libthread_db.so /system/lib/libthread_db.so.1 2. patching gdb i did have changed linaro-gdb 11.10 release by:
diff --git a/gdb/arm-linux-tdep.c b/gdb/arm-linux-tdep.c index ca0bc30..486faf6 100644 --- a/gdb/arm-linux-tdep.c +++ b/gdb/arm-linux-tdep.c @@ -98,8 +98,8 @@ static const char arm_linux_thumb2_le_breakpoint[] = { 0xf0, 0xf7, 0x00, 0xa0 }; buffer. This is also true for the SoftFPA model. However, for the FPA model the PC is at offset 21 in the buffer. */ #define ARM_LINUX_JB_ELEMENT_SIZE INT_REGISTER_SIZE -#define ARM_LINUX_JB_PC_FPA 21 -#define ARM_LINUX_JB_PC_EABI 9 +#define ARM_LINUX_JB_PC_FPA 24/*21*/ +#define ARM_LINUX_JB_PC_EABI 24/*9*/
/* Dynamic Linking on ARM GNU/Linux diff --git a/gdb/gdb_thread_db.h b/gdb/gdb_thread_db.h index 957ed2c..51ed4fa 100644 --- a/gdb/gdb_thread_db.h +++ b/gdb/gdb_thread_db.h @@ -2,7 +2,7 @@ #include <thread_db.h>
#ifndef LIBTHREAD_DB_SO -#define LIBTHREAD_DB_SO "libthread_db.so.1" +#define LIBTHREAD_DB_SO "libthread_db.so" #endif
#ifndef LIBTHREAD_DB_SEARCH_PATH
But both 1 and 2 can't simply fix the problem. if we compile gdb statically by "make LDFLAGS=-static", it will finally come into a crash: # gdb attach 643 GNU gdb (Linaro GDB) 7.3-2011.10 Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html this is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "arm-none-linux-gnueabi". For bug reporting instructions, please see: http://bugs.launchpad.net/gdb-linaro/... attach: No such file or directory. Attaching to process 643 Reading symbols from /system/bin/app_process...done. BFD: /system/bin/linker: warning: sh_link not set for section `.ARM.exidx'
warning: Could not load shared library symbols for 6 libraries, e.g. gralloc.default.so. Use the "info sharedlibrary" command to see the complete listing. Do you need "set solib-search-path" or "set sysroot"? Reading symbols from /system/bin/linker...(no debugging symbols found)...done. Loaded symbols for /system/bin/linker Reading symbols from /system/lib/libc.so...done. Loaded symbols for /system/lib/libc.so Reading symbols from /system/lib/libstdc++.so...(no debugging symbols found)...done. Loaded symbols for /system/lib/libstdc++.so Reading symbols from /system/lib/libm.so...(no debugging symbols found)...done. Loaded symbols for /system/lib/libm.so ... __ioctl () at bionic/libc/arch-arm/syscalls/__ioctl.S:15 15 bionic/libc/arch-arm/syscalls/__ioctl.S: No such file or directory. in bionic/libc/arch-arm/syscalls/__ioctl.S gdb: ../sysdeps/unix/sysv/linux/getpagesize.c:32: __getpagesize: Assertion `_rtld_global_ro._dl_pagesize != 0' failed. Aborted #
then i simply hardcoded __getpagesize() to return 4096 and avoid the assert:
001d9b70 <__getpagesize>: 1d9b70: f44f 5080 mov.w r0, #4096 ; 0x1000 1d9b74: 4770 bx lr 1d9b76: bf00 nop
this "fixed" Assertion `_rtld_global_ro._dl_pagesize != 0' , but it still can't find multi-threads for android processes: (gdb) info threads Id Target Id Frame * 1 process 645 "system_server" __ioctl () at bionic/libc/arch-arm/syscalls/__ioctl.S:15
(gdb) info threads Id Target Id Frame * 1 process 938 "mediaserver" __ioctl () at bionic/libc/arch-arm/syscalls/__ioctl.S:15
Mit freundlichen Gruessen / Best Regards
Ulrich Weigand
-- Dr. Ulrich Weigand | Phone: +49-7031/16-3727 STSM, GNU compiler and toolchain for Linux on System z and Cell/B.E. IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz | Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen | Registergericht: Amtsgericht Stuttgart, HRB 243294
-barry
Barry Song 21cnbao@gmail.com wrote.
2012/2/3 Ulrich Weigand Ulrich.Weigand@de.ibm.com:
However, from looking at the gdbserver sources provided with Android, it seems there are some differences; in particular, there's this patch:
+/* Android doesn't have libthread_db.so.1, just libthread_db.so. */ +#ifdef __ANDROID__ +#define LIBTHREAD_DB_SO "libthread_db.so" +#endif
If libthread_db is named differently, this would explain why GDB is unable to find and use it.
there are two ways to handle this issue:
- ln -s /system/lib/libthread_db.so /system/lib/libthread_db.so.1
That would seem a good thing in any case. (Is there any reason for Android to use nonstandard shared library naming conventions?)
this "fixed" Assertion `_rtld_global_ro._dl_pagesize != 0' , but it still can't find multi-threads for android processes: (gdb) info threads Id Target Id Frame
- 1 process 645 "system_server" __ioctl () at bionic/libc/arch-arm/syscalls/__ioctl.S:15
(gdb) info threads Id Target Id Frame
- 1 process 938 "mediaserver" __ioctl () at bionic/libc/arch-arm/syscalls/__ioctl.S:15
I had a quick look at the AOSP sources, and found that there actually is an implementation of libthread_db that is supposed to work with bionic. Well, there seem to be multiple versions:
./bionic/libthread_db ./ndk/sources/android/libthread_db/gdb-6.6 ./ndk/sources/android/libthread_db/gdb-7.1.x
I'm not sure why there needs to be a different version for each GDB(server) release ... Maybe there's something in there that also needs changing when it's being used by GDB directly instead of by gdbserver.
I'll probably not be able to help you very much here; I'd suggest you 1) make sure your GDB actually uses the correct version of libthread_db (in particular, not one from glibc, but this one that handles bionic) and 2) if it still doesn't work, you'll have to debug what's going wrong.
Note that from a quick look at the above libthread_db sources, those actually do not look into bionic data structures at all, but rather just traverse /proc to get a thread list. While this is somewhat odd (if you want to do that, you could just do it in gdb/server itself -- the only reason why libthread_db is a separate project is its close integration with the thread library), it ought to make it easier to understand what's going on / wrong.
Mit freundlichen Gruessen / Best Regards
Ulrich Weigand
-- Dr. Ulrich Weigand | Phone: +49-7031/16-3727 STSM, GNU compiler and toolchain for Linux on System z and Cell/B.E. IBM Deutschland Research & Development GmbH Vorsitzende des Aufsichtsrats: Martina Koederitz | Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen | Registergericht: Amtsgericht Stuttgart, HRB 243294
Ulrich, thanks a lot!
2012/2/4 Ulrich Weigand Ulrich.Weigand@de.ibm.com:
Barry Song 21cnbao@gmail.com wrote.
2012/2/3 Ulrich Weigand Ulrich.Weigand@de.ibm.com:
However, from looking at the gdbserver sources provided with Android, it seems there are some differences; in particular, there's this patch:
+/* Android doesn't have libthread_db.so.1, just libthread_db.so. */ +#ifdef __ANDROID__ +#define LIBTHREAD_DB_SO "libthread_db.so" +#endif
If libthread_db is named differently, this would explain why GDB is unable to find and use it.
there are two ways to handle this issue:
- ln -s /system/lib/libthread_db.so /system/lib/libthread_db.so.1
That would seem a good thing in any case. (Is there any reason for Android to use nonstandard shared library naming conventions?)
this "fixed" Assertion `_rtld_global_ro._dl_pagesize != 0' , but it still can't find multi-threads for android processes: (gdb) info threads Id Target Id Frame
- 1 process 645 "system_server" __ioctl ()
at bionic/libc/arch-arm/syscalls/__ioctl.S:15
(gdb) info threads Id Target Id Frame
- 1 process 938 "mediaserver" __ioctl ()
at bionic/libc/arch-arm/syscalls/__ioctl.S:15
I had a quick look at the AOSP sources, and found that there actually is an implementation of libthread_db that is supposed to work with bionic. Well, there seem to be multiple versions:
./bionic/libthread_db ./ndk/sources/android/libthread_db/gdb-6.6 ./ndk/sources/android/libthread_db/gdb-7.1.x
I'm not sure why there needs to be a different version for each GDB(server) release ... Maybe there's something in there that also needs changing when it's being used by GDB directly instead of by gdbserver.
I'll probably not be able to help you very much here; I'd suggest you 1) make sure your GDB actually uses the correct version of libthread_db (in particular, not one from glibc, but this one that handles bionic) and 2) if it still doesn't work, you'll have to debug what's going wrong.
for 1, i think gdb has loaded /system/lib/libthread_db.so of android according to log info after patching gdb: --- a/gdb/gdb_thread_db.h +++ b/gdb/gdb_thread_db.h @@ -2,7 +2,7 @@ #include <thread_db.h>
#ifndef LIBTHREAD_DB_SO -#define LIBTHREAD_DB_SO "libthread_db.so.1" +#define LIBTHREAD_DB_SO "libthread_db.so" #endif
for 2, the answer is it doesn't work yet. i reported this issue at: https://bugs.launchpad.net/gdb-linaro/+bug/926472 as Thiago pointed out. Thanks, Thiago!
Note that from a quick look at the above libthread_db sources, those actually do not look into bionic data structures at all, but rather just traverse /proc to get a thread list. While this is somewhat odd (if you want to do that, you could just do it in gdb/server itself -- the only reason why libthread_db is a separate project is its close integration with the thread library), it ought to make it easier to understand what's going on / wrong.
Mit freundlichen Gruessen / Best Regards
Ulrich Weigand
-barry
Hi Barry,
On Thu, 2012-02-02 at 10:23 +0800, Barry Song wrote:
- Why can’t the current gdb capture multithreads for android
processes? This question is actually about the theory for gdb to know multi-threads. In my opinion, both gnu and android use clone() to fork threads and threads in one process have same tgid in kernel and all threads return same getpid() value. Why not gdb just travel process lists to find multi-threads?
Would you mind opening a bug report at
https://bugs.launchpad.net/gdb-linaro
with this issue? If possible, with a small testcase to reproduce the problem, and the steps to build the testcase.
To be honest I can only look into this issue late next week though...
2012/2/3 Thiago Jung Bauermann thiago.bauermann@linaro.org:
Hi Barry,
On Thu, 2012-02-02 at 10:23 +0800, Barry Song wrote:
- Why can’t the current gdb capture multithreads for android
processes? This question is actually about the theory for gdb to know multi-threads. In my opinion, both gnu and android use clone() to fork threads and threads in one process have same tgid in kernel and all threads return same getpid() value. Why not gdb just travel process lists to find multi-threads?
Would you mind opening a bug report at
https://bugs.launchpad.net/gdb-linaro
with this issue? If possible, with a small testcase to reproduce the problem, and the steps to build the testcase.
To be honest I can only look into this issue late next week though...
Thiago, yes. i will. PS: now there is a platform/external/gdb.git tree in http://android.git.linaro.org/gitweb. i would suggest linaro make the gdb compilable with an Android.mk by arm-eabi-gcc just like other external components something like: LOCAL_PATH := $(call my-dir) include $(CLEAR_VARS)
LOCAL_MODULE_TAGS := eng
LOCAL_MODULE := gdb
LOCAL_SRC_FILES += \ ... \
include $(BUILD_EXECUTABLE)
finally, i hope that will result in a native gdb running on boards and supporting multithreads. that is actually very useful to many users.
-- []'s Thiago Jung Bauermann Linaro Toolchain Working Group
-barry
-
Barry Song -
Thiago Jung Bauermann -
Ulrich Weigand