17. Debugging Remote Programs

If you are trying to debug a program running on a machine that cannot run GDB in the usual way, it is often useful to use remote debugging. For example, you might use remote debugging on an operating system kernel, or on a small system which does not have a general purpose operating system powerful enough to run a full-featured debugger.

Some configurations of GDB have special serial or TCP/IP interfaces to make this work with particular debugging targets. In addition, GDB comes with a generic serial protocol (specific to GDB, but not specific to any particular target system) which you can use if you write the remote stubs--the code that runs on the remote system to communicate with GDB.

Other remote targets may be available in your configuration of GDB; use help target to list them.

17.1 Connecting to a Remote Target		Connecting to a remote target
17.2 Sending files to a remote system
17.3 Using the gdbserver Program		Using the gdbserver program
17.4 Remote Configuration		Remote configuration
17.5 Implementing a Remote Stub		Implementing a remote stub

17.1 Connecting to a Remote Target

On the GDB host machine, you will need an unstripped copy of your program, since GDB needs symbol and debugging information. Start up GDB as usual, using the name of the local copy of your program as the first argument.

GDB can communicate with the target over a serial line, or over an IP network using TCP or UDP. In each case, GDB uses the same protocol for debugging your program; only the medium carrying the debugging packets varies. The target remote command establishes a connection to the target. Its arguments indicate which medium to use:

target remote serial-device

Use serial-device to communicate with the target. For example, to use a serial line connected to the device named `/dev/ttyb':

target remote /dev/ttyb

If you're using a serial line, you may want to give GDB the `--baud' option, or use the set remotebaud command (see section set remotebaud) before the target command.

target remote host:port

target remote tcp:host:port

Debug using a TCP connection to port on host. The host may be either a host name or a numeric IP address; port must be a decimal number. The host could be the target machine itself, if it is directly connected to the net, or it might be a terminal server which in turn has a serial line to the target.

For example, to connect to port 2828 on a terminal server named manyfarms:

target remote manyfarms:2828

If your remote target is actually running on the same machine as your debugger session (e.g. a simulator for your target running on the same host), you can omit the hostname. For example, to connect to port 1234 on your local machine:

target remote :1234

Note that the colon is still required here.

target remote udp:host:port

Debug using UDP packets to port on host. For example, to connect to UDP port 2828 on a terminal server named manyfarms:

target remote udp:manyfarms:2828

When using a UDP connection for remote debugging, you should keep in mind that the `U' stands for "Unreliable". UDP can silently drop packets on busy or unreliable networks, which will cause havoc with your debugging session.

target remote | command

Run command in the background and communicate with it using a pipe. The command is a shell command, to be parsed and expanded by the system's command shell, /bin/sh; it should expect remote protocol packets on its standard input, and send replies on its standard output. You could use this to run a stand-alone simulator that speaks the remote debugging protocol, to make net connections using programs like ssh, or for other similar tricks.

If command closes its standard output (perhaps by exiting), GDB will try to send it a SIGTERM signal. (If the program has already exited, this will have no effect.)

Once the connection has been established, you can use all the usual commands to examine and change data. The remote program is already running; you can use step and continue, and you do not need to use run.

Whenever GDB is waiting for the remote program, if you type the interrupt character (often Ctrl-c), GDB attempts to stop the program. This may or may not succeed, depending in part on the hardware and the serial drivers the remote system uses. If you type the interrupt character once again, GDB displays this prompt:

Interrupted while waiting for the program. Give up (and stop debugging it)? (y or n)

If you type y, GDB abandons the remote debugging session. (If you decide you want to try again later, you can use `target remote' again to connect once more.) If you type n, GDB goes back to waiting.

detach

When you have finished debugging the remote program, you can use the detach command to release it from GDB control. Detaching from the target normally resumes its execution, but the results will depend on your particular remote stub. After the detach command, GDB is free to connect to another target.

disconnect

The disconnect command behaves like detach, except that the target is generally not resumed. It will wait for GDB (this instance or another one) to connect and continue debugging. After the disconnect command, GDB is again free to connect to another target.

monitor cmd

This command allows you to send arbitrary commands directly to the remote monitor. Since GDB doesn't care about the commands it sends like this, this command is the way to extend GDB---you can add new commands that only the external monitor will understand and implement.

17.2 Sending files to a remote system

Some remote targets offer the ability to transfer files over the same connection used to communicate with GDB. This is convenient for targets accessible through other means, e.g. GNU/Linux systems running gdbserver over a network interface. For other targets, e.g. embedded devices with only a single serial port, this may be the only way to upload or download files.

Not all remote targets support these commands.

remote put hostfile targetfile

Copy file hostfile from the host system (the machine running GDB) to targetfile on the target system.

remote get targetfile hostfile

Copy file targetfile from the target system to hostfile on the host system.

remote delete targetfile

Delete targetfile from the target system.

17.3 Using the gdbserver Program

gdbserver is a control program for Unix-like systems, which allows you to connect your program with a remote GDB via target remote---but without linking in the usual debugging stub.

gdbserver is not a complete replacement for the debugging stubs, because it requires essentially the same operating-system facilities that GDB itself does. In fact, a system that can run gdbserver to connect to a remote GDB could also run GDB locally! gdbserver is sometimes useful nevertheless, because it is a much smaller program than GDB itself. It is also easier to port than all of GDB, so you may be able to get started more quickly on a new system by using gdbserver. Finally, if you develop code for real-time systems, you may find that the tradeoffs involved in real-time operation make it more convenient to do as much development work as possible on another system, for example by cross-compiling. You can use gdbserver to make a similar choice for debugging.

GDB and gdbserver communicate via either a serial line or a TCP connection, using the standard GDB remote serial protocol.

Warning: gdbserver does not have any built-in security. Do not run gdbserver connected to any public network; a GDB connection to gdbserver provides access to the target system with the same privileges as the user running gdbserver.

17.3.1 Running gdbserver

Run gdbserver on the target system. You need a copy of the program you want to debug, including any libraries it requires. gdbserver does not need your program's symbol table, so you can strip the program if necessary to save space. GDB on the host system does all the symbol handling.

To use the server, you must tell it how to communicate with GDB; the name of your program; and the arguments for your program. The usual syntax is:

target> gdbserver comm program [ args ... ]

comm is either a device name (to use a serial line) or a TCP hostname and portnumber. For example, to debug Emacs with the argument `foo.txt' and communicate with GDB over the serial port `/dev/com1':

target> gdbserver /dev/com1 emacs foo.txt

gdbserver waits passively for the host GDB to communicate with it.

To use a TCP connection instead of a serial line:

target> gdbserver host:2345 emacs foo.txt

The only difference from the previous example is the first argument, specifying that you are communicating with the host GDB via TCP. The `host:2345' argument means that gdbserver is to expect a TCP connection from machine `host' to local TCP port 2345. (Currently, the `host' part is ignored.) You can choose any number you want for the port number as long as it does not conflict with any TCP ports already in use on the target system (for example, 23 is reserved for telnet).(9) You must use the same port number with the host GDB target remote command.

17.3.1.1 Attaching to a Running Program

On some targets, gdbserver can also attach to running programs. This is accomplished via the --attach argument. The syntax is:

target> gdbserver --attach comm pid

pid is the process ID of a currently running process. It isn't necessary to point gdbserver at a binary for the running process.

You can debug processes by name instead of process ID if your target has the pidof utility:

target> gdbserver --attach comm `pidof program`

In case more than one copy of program is running, or program has multiple threads, most versions of pidof support the -s option to only return the first process ID.

17.3.1.2 Multi-Process Mode for gdbserver

When you connect to gdbserver using target remote, gdbserver debugs the specified program only once. When the program exits, or you detach from it, GDB closes the connection and gdbserver exits.

If you connect using target extended-remote, gdbserver enters multi-process mode. When the debugged program exits, or you detach from it, GDB stays connected to gdbserver even though no program is running. The run and attach commands instruct gdbserver to run or attach to a new program. The run command uses set remote exec-file (see set remote exec-file) to select the program to run. Command line arguments are supported, except for wildcard expansion and I/O redirection (see section 4.3 Your Program's Arguments).

To start gdbserver without supplying an initial command to run or process ID to attach, use the `--multi' command line option. Then you can connect using target extended-remote and start the program you want to debug.

gdbserver does not automatically exit in multi-process mode. You can terminate it by using monitor exit (see Monitor Commands for gdbserver).

17.3.1.3 Other Command-Line Arguments for gdbserver

You can include `--debug' on the gdbserver command line. gdbserver will display extra status information about the debugging process. This option is intended for gdbserver development and for bug reports to the developers.

The `--wrapper' option specifies a wrapper to launch programs for debugging. The option should be followed by the name of the wrapper, then any command-line arguments to pass to the wrapper, then -- indicating the end of the wrapper arguments.

gdbserver runs the specified wrapper program with a combined command line including the wrapper arguments, then the name of the program to debug, then any arguments to the program. The wrapper runs until it executes your program, and then GDB gains control.

You can use any program that eventually calls execve with its arguments as a wrapper. Several standard Unix utilities do this, e.g. env and nohup. Any Unix shell script ending with exec "$@" will also work.

For example, you can use env to pass an environment variable to the debugged program, without setting the variable in gdbserver's environment:

$ gdbserver --wrapper env LD_PRELOAD=libtest.so -- :2222 ./testprog

17.3.2 Connecting to gdbserver

Run GDB on the host system.

First make sure you have the necessary symbol files. Load symbols for your application using the file command before you connect. Use set sysroot to locate target libraries (unless your GDB was compiled with the correct sysroot using --with-sysroot).

The symbol file and target libraries must exactly match the executable and libraries on the target, with one exception: the files on the host system should not be stripped, even if the files on the target system are. Mismatched or missing files will lead to confusing results during debugging. On GNU/Linux targets, mismatched or missing files may also prevent gdbserver from debugging multi-threaded programs.

Connect to your target (see section Connecting to a Remote Target). For TCP connections, you must start up gdbserver prior to using the target remote command. Otherwise you may get an error whose text depends on the host system, but which usually looks something like `Connection refused'. Don't use the load command in GDB when using gdbserver, since the program is already on the target.

17.3.3 Monitor Commands for gdbserver

During a GDB session using gdbserver, you can use the monitor command to send special requests to gdbserver. Here are the available commands.

monitor help

List the available monitor commands.

monitor set debug 0

monitor set debug 1

Disable or enable general debugging messages.

monitor set remote-debug 0

monitor set remote-debug 1

Disable or enable specific debugging messages associated with the remote protocol (see section D. GDB Remote Serial Protocol).

monitor exit

Tell gdbserver to exit immediately. This command should be followed by disconnect to close the debugging session. gdbserver will detach from any attached processes and kill any processes it created. Use monitor exit to terminate gdbserver at the end of a multi-process mode debug session.

17.4 Remote Configuration

This section documents the configuration options available when debugging remote programs. For the options related to the File I/O extensions of the remote protocol, see system-call-allowed.

set remoteaddresssize bits

Set the maximum size of address in a memory packet to the specified number of bits. GDB will mask off the address bits above that number, when it passes addresses to the remote target. The default value is the number of bits in the target's address.

show remoteaddresssize

Show the current value of remote address size in bits.

set remotebaud n

Set the baud rate for the remote serial I/O to n baud. The value is used to set the speed of the serial port used for debugging remote targets.

show remotebaud

Show the current speed of the remote connection.

set remotebreak

If set to on, GDB sends a BREAK signal to the remote when you type Ctrl-c to interrupt the program running on the remote. If set to off, GDB sends the `Ctrl-C' character instead. The default is off, since most remote systems expect to see `Ctrl-C' as the interrupt signal.

show remotebreak

Show whether GDB sends BREAK or `Ctrl-C' to interrupt the remote program.

set remoteflow on

set remoteflow off

Enable or disable hardware flow control (RTS/CTS) on the serial port used to communicate to the remote target.

show remoteflow

Show the current setting of hardware flow control.

set remotelogbase base

Set the base (a.k.a. radix) of logging serial protocol communications to base. Supported values of base are: ascii, octal, and hex. The default is ascii.

show remotelogbase

Show the current setting of the radix for logging remote serial protocol.

set remotelogfile file

Record remote serial communications on the named file. The default is not to record at all.

show remotelogfile.

Show the current setting of the file name on which to record the serial communications.

set remotetimeout num

Set the timeout limit to wait for the remote target to respond to num seconds. The default is 2 seconds.

show remotetimeout

Show the current number of seconds to wait for the remote target responses.

set remote hardware-watchpoint-limit limit

set remote hardware-breakpoint-limit limit

Restrict GDB to using limit remote hardware breakpoint or watchpoints. A limit of -1, the default, is treated as unlimited.

set remote exec-file filename

show remote exec-file

Select the file used for run with target extended-remote. This should be set to a filename valid on the target system. If it is not set, the target will use a default filename (e.g. the last program run).

The GDB remote protocol autodetects the packets supported by your debugging stub. If you need to override the autodetection, you can use these commands to enable or disable individual packets. Each packet can be set to `on' (the remote target supports this packet), `off' (the remote target does not support this packet), or `auto' (detect remote target support for this packet). They all default to `auto'. For more information about each packet, see D. GDB Remote Serial Protocol.

During normal use, you should not have to use any of these commands. If you do, that may be a bug in your remote debugging stub, or a bug in GDB. You may want to report the problem to the GDB developers.

For each packet name, the command to enable or disable the packet is set remote name-packet. The available settings are:

Command Name	Remote Packet	Related Features
fetch-register	p	info registers
set-register	P	set
binary-download	X	load, set
read-aux-vector	qXfer:auxv:read	info auxv
symbol-lookup	qSymbol	Detecting multiple threads
attach	vAttach	attach
verbose-resume	vCont	Stepping or resuming multiple threads
run	vRun	run
software-breakpoint	Z0	break
hardware-breakpoint	Z1	hbreak
write-watchpoint	Z2	watch
read-watchpoint	Z3	rwatch
access-watchpoint	Z4	awatch
target-features	qXfer:features:read	set architecture
library-info	qXfer:libraries:read	info sharedlibrary
memory-map	qXfer:memory-map:read	info mem
read-spu-object	qXfer:spu:read	info spu
write-spu-object	qXfer:spu:write	info spu
get-thread-local- storage-address	qGetTLSAddr	Displaying __thread variables
search-memory	qSearch:memory	find
supported-packets	qSupported	Remote communications parameters
pass-signals	QPassSignals	handle signal
hostio-close-packet	vFile:close	remote get, remote put
hostio-open-packet	vFile:open	remote get, remote put
hostio-pread-packet	vFile:pread	remote get, remote put
hostio-pwrite-packet	vFile:pwrite	remote get, remote put
hostio-unlink-packet	vFile:unlink	remote delete
noack-packet	QStartNoAckMode	Packet acknowledgment

17.5 Implementing a Remote Stub

The stub files provided with GDB implement the target side of the communication protocol, and the GDB side is implemented in the GDB source file `remote.c'. Normally, you can simply allow these subroutines to communicate, and ignore the details. (If you're implementing your own stub file, you can still ignore the details: start with one of the existing stub files. `sparc-stub.c' is the best organized, and therefore the easiest to read.)

To debug a program running on another machine (the debugging target machine), you must first arrange for all the usual prerequisites for the program to run by itself. For example, for a C program, you need:

1. A startup routine to set up the C runtime environment; these usually have a name like `crt0'. The startup routine may be supplied by your hardware supplier, or you may have to write your own.

2. A C subroutine library to support your program's subroutine calls, notably managing input and output.

3. A way of getting your program to the other machine--for example, a download program. These are often supplied by the hardware manufacturer, but you may have to write your own from hardware documentation.

The next step is to arrange for your program to use a serial port to communicate with the machine where GDB is running (the host machine). In general terms, the scheme looks like this:

On the host,

GDB already understands how to use this protocol; when everything else is set up, you can simply use the `target remote' command (see section Specifying a Debugging Target).

On the target,

you must link with your program a few special-purpose subroutines that implement the GDB remote serial protocol. The file containing these subroutines is called a debugging stub.

On certain remote targets, you can use an auxiliary program gdbserver instead of linking a stub into your program. See section Using the gdbserver Program, for details.

The debugging stub is specific to the architecture of the remote machine; for example, use `sparc-stub.c' to debug programs on SPARC boards.

These working remote stubs are distributed with GDB:

i386-stub.c

For Intel 386 and compatible architectures.

m68k-stub.c

For Motorola 680x0 architectures.

sh-stub.c

For Renesas SH architectures.

sparc-stub.c

For SPARC architectures.

sparcl-stub.c

For Fujitsu SPARCLITE architectures.

The `README' file in the GDB distribution may list other recently added stubs.

17.5.1 What the Stub Can Do for You		What the stub can do for you
17.5.2 What You Must Do for the Stub		What you must do for the stub
17.5.3 Putting it All Together		Putting it all together

17.5.1 What the Stub Can Do for You

The debugging stub for your architecture supplies these three subroutines:

set_debug_traps

This routine arranges for handle_exception to run when your program stops. You must call this subroutine explicitly near the beginning of your program.

handle_exception

This is the central workhorse, but your program never calls it explicitly--the setup code arranges for handle_exception to run when a trap is triggered.

handle_exception takes control when your program stops during execution (for example, on a breakpoint), and mediates communications with GDB on the host machine. This is where the communications protocol is implemented; handle_exception acts as the GDB representative on the target machine. It begins by sending summary information on the state of your program, then continues to execute, retrieving and transmitting any information GDB needs, until you execute a GDB command that makes your program resume; at that point, handle_exception returns control to your own code on the target machine.

breakpoint

Use this auxiliary subroutine to make your program contain a breakpoint. Depending on the particular situation, this may be the only way for GDB to get control. For instance, if your target machine has some sort of interrupt button, you won't need to call this; pressing the interrupt button transfers control to handle_exception---in effect, to GDB. On some machines, simply receiving characters on the serial port may also trigger a trap; again, in that situation, you don't need to call breakpoint from your own program--simply running `target remote' from the host GDB session gets control.

Call breakpoint if none of these is true, or if you simply want to make certain your program stops at a predetermined point for the start of your debugging session.

17.5.2 What You Must Do for the Stub

The debugging stubs that come with GDB are set up for a particular chip architecture, but they have no information about the rest of your debugging target machine.

First of all you need to tell the stub how to communicate with the serial port.

int getDebugChar()

Write this subroutine to read a single character from the serial port. It may be identical to getchar for your target system; a different name is used to allow you to distinguish the two if you wish.

void putDebugChar(int)

Write this subroutine to write a single character to the serial port. It may be identical to putchar for your target system; a different name is used to allow you to distinguish the two if you wish.

If you want GDB to be able to stop your program while it is running, you need to use an interrupt-driven serial driver, and arrange for it to stop when it receives a ^C (`\003', the control-C character). That is the character which GDB uses to tell the remote system to stop.

Getting the debugging target to return the proper status to GDB probably requires changes to the standard stub; one quick and dirty way is to just execute a breakpoint instruction (the "dirty" part is that GDB reports a SIGTRAP instead of a SIGINT).

Other routines you need to supply are:

void exceptionHandler (int exception_number, void *exception_address)

Write this function to install exception_address in the exception handling tables. You need to do this because the stub does not have any way of knowing what the exception handling tables on your target system are like (for example, the processor's table might be in ROM, containing entries which point to a table in RAM). exception_number is the exception number which should be changed; its meaning is architecture-dependent (for example, different numbers might represent divide by zero, misaligned access, etc). When this exception occurs, control should be transferred directly to exception_address, and the processor state (stack, registers, and so on) should be just as it is when a processor exception occurs. So if you want to use a jump instruction to reach exception_address, it should be a simple jump, not a jump to subroutine.

For the 386, exception_address should be installed as an interrupt gate so that interrupts are masked while the handler runs. The gate should be at privilege level 0 (the most privileged level). The SPARC and 68k stubs are able to mask interrupts themselves without help from exceptionHandler.

void flush_i_cache()

On SPARC and SPARCLITE only, write this subroutine to flush the instruction cache, if any, on your target machine. If there is no instruction cache, this subroutine may be a no-op.

On target machines that have instruction caches, GDB requires this function to make certain that the state of your program is stable.

You must also make sure this library routine is available:

void *memset(void *, int, int)

This is the standard library function memset that sets an area of memory to a known value. If you have one of the free versions of libc.a, memset can be found there; otherwise, you must either obtain it from your hardware manufacturer, or write your own.

If you do not use the GNU C compiler, you may need other standard library subroutines as well; this varies from one stub to another, but in general the stubs are likely to use any of the common library subroutines which GCC generates as inline code.

17.5.3 Putting it All Together

In summary, when your program is ready to debug, you must follow these steps.

1. Make sure you have defined the supporting low-level routines (see section What You Must Do for the Stub):

   getDebugChar, putDebugChar, flush_i_cache, memset, exceptionHandler.

2. Insert these lines near the top of your program:

   set_debug_traps(); breakpoint();

3. For the 680x0 stub only, you need to provide a variable called exceptionHook. Normally you just use:

   void (*exceptionHook)() = 0;

   but if before calling set_debug_traps, you set it to point to a function in your program, that function is called when GDB continues after stopping on a trap (for example, bus error). The function indicated by exceptionHook is called with one parameter: an int which is the exception number.

4. Compile and link together: your program, the GDB debugging stub for your target architecture, and the supporting subroutines.

5. Make sure you have a serial connection between your target machine and the GDB host, and identify the serial port on the host.

6. Download your program to your target machine (or get it there by whatever means the manufacturer provides), and start it.

7. Start GDB on the host, and connect to the target (see section Connecting to a Remote Target).