This document is a collection of extracts, observations and notes pertaining to the S/390 port of Linux.

On December 18, 1999, IBM published its modifications and additions to the Linux 2.2.13 code base for the support of the S/390 architecture. This port is designed to run under VM/ESA and natively. The code has subsequently been rolled into the 2.2.15 level.

This document contains information specific to the S/390 port of Linux. In it I have reproduced documentation found within the distribution that describes the I/O facilities and DASD handling. In addition, information that I have come across as I have looked at the port such as new source files, system calls, and register conventions have been included.

This chapter describes the common device support routines for Linux/390. Different than other hardware architectures, ESA/390 has defined a unified I/O access method. This gives relief to the device drivers as they don't have to deal with different bus types, polling versus interrupt processing, shared versus non-shared interrupt processing, DMA versus port I/O (PIO), and other hardware features more. i However, this implies that either every single device driver needs to implement the hardware I/O attachment functionality itself, or the operating system provides for a unified method to access the hardware, providing all the functionality that every single device driver would have to provide itself.

The document does not intend to explain the ESA/390 hardware architecture in every detail. This information can be obtained from the ESA/390 Principles of Operation manual (IBM Form. No. SA22-7201).

In order to build common device support for ESA/390 I/O interfaces, a functional layer was introduced that provides generic I/O access methods to the hardware. The following figure shows the usage of the common device support of Linux/390 using a TCP/IP driven device access an example. Similar figures could be drawn for other access methods, e.g. file system access to disk devices.

The common device support layer shown above comprises the I/O support routines defined below. Some of them implement common Linux device driver interfaces, while some of them are ESA/390 platform specific.

• get_dev_info_by_IRQ() / get_dev_info_by_devno()

• get_IRQ_by_devno() / get_devno_by_IRQ()

• read_dev_chars()

• request_IRQ()

• free_IRQ()

• disable_IRQ()

• enable_IRQ()

• do_IO()

• halt_IO()

• do_IRQ()

The following chapters describe the I/O related interface routines the Linux/390 common device support (CDS) provides to allow for device specific driver implementations on the IBM ESA/390 hardware platform. Those interfaces intend to provide the functionality required by every device driver implementation to allow driving a specific hardware device on the ESA/390 platform. Some of the interface routines are specific to Linux/390 and some of them can be found on other Linux platforms' implementations too.

Different to other hardware platforms, the ESA/390 architecture does not define interrupt lines managed by a specific interrupt controller and bus systems that may or may not allow for shared interrupts, DMA processing, etceteras. Instead, the ESA/390 architecture has implemented a so-called channel subsystem, which provides a unified view of the devices physically attached to the systems. Though the ESA/390 hardware platform knows about a huge variety of different peripheral attachments like disk devices (also known as DASD), tapes, communication controllers, they can all by accessed by a well defined access method and they are presenting I/O completion a unified way: I/O interruptions. Every single device is uniquely identified to the system by a so-called subchannel, where the ESA/390 architecture allows for 64k devices to be attached.

Linux, however was first built on the Intel PC architecture, with its two cascaded 8259 programmable interrupt controllers (PICs), that allow for a maximum of 15 different interrupt lines. All devices attached to such a system share those 15 interrupt levels. Devices attached to the ISA bus system must not share interrupt levels (also known as IRQs), as the ISA bus bases on edge triggered interrupts. MCA, EISA, PCI and other bus systems base on level triggered interrupts, and thus allow for shared IRQs. However, if multiple devices present their hardware status by the same (shared) IRQ, the operating system has to call every single device driver registered on this IRQ in order to determine the device driver owning the device that raised the interrupt.

In order not to introduce a new I/O concept to the common Linux code, Linux/390 preserves the IRQ concept and semantically maps the ESA/390 subchannels to Linux as IRQs. This allows Linux/390 to support up to 64k different IRQs, uniquely representig a single device each.

Possible status values are:

DEVSTAT_NOT_OPER - device was found not operational. In this case the caller should disregard the sid_data buffer content.

The ESA/390 I/O architecture defines certain device specific I/O functions. The device returns the device specific command code together with the Sense-ID data in so called Command Information Words (CIW):

The device driver may use these commands as appropriate.

Please see the "ESA/390 Common I/O-Commands and Self Description" manual, with IBM form number SA22-7204 for more details on how to read the Sense-ID output, CIWs and device independent commands.

While some device drivers act on the IRQ (subchannel) only, others take user defined device configurations on device number base, according to the device numbers configured in the IOCDS. The following routines serve the purpose to convert IRQ values into device numbers and vice versa.

 The functions return :

• the requested IRQ/devno values
• -1 if the requested conversion can't be accomplished. This could either be caused by IRQ/devno be outside the valid range (value > 0xffff or value < 0) or not identifying a known device.

This routine returns the characteristics for the device specified.

The function is meant to be called without an IRQ handler being in place. However, the IRQ for the requested device must not be locked or this will cause a deadlock situation. Further, the driver must assure that nobody else has claimed ownership for the requested IRQ yet or the owning device driver's internal accounting may be affected.

The function may be called enabled or disabled.

The read_dev_chars() function returns :

The function can be used in two ways:

In either case the caller must provide the data area length: for the buffer specified or the buffer wanted allocated.

The only value parameter irqflags supports is SA_SAMPLE_RANDOM if appropriate. The Linux/390 kernel does not know about "fast" interrupt handlers, or does it allow for interrupt sharing. Remember, the term interrupt level (IRQ), device, and subchannel are used interchangeably in Linux/390.

Although the interrupt handler is defined to be called with a pointer to a struct pt_regs buffer area, this is not implemented by the Linux/390 generic I/O device driver support layer. The device driver's interrupt handler must therefore not rely on this parameter on function entry.

This function may be called at any time to disable interrupt processing for the specified IRQ. However, as Linux/390 maps IRQs to the device (subchannel) one-to-one, this may require more extensive I/O processing than anticipated, especially if an interrupt status is found pending on the subchannel that requires synchronous error processing.

Unlike the Intel based hardware architecture the ESA/390 architecture does not have a programmable interrupt controller (PIC) where a specific interrupt line can be disabled. Instead the subchannel logically representing the device in the channel subsystem must be disabled for interrupts. However, if there are still interrupt conditions pending they must be processed first in order to allow for proper processing after re-enabling the device at a later time. This may lead to delayed disable processing.

Linux for Intel defines this function

This is suitable for the Intel PC architecture as this only causes to mask the requested IRQ line in the PIC which is not applicable for the ESA/390 architecture. Therefore we allow for returning error codes.

Possible flag values are:

The cpa parameter points to the first format 1 CCW of a channel program:

with the following CCW flags values defined:

The device interrupt handler can use the following definitions to investigate the primary unit check source coded in sense byte 0:

Depending on the device status, multiple of those values may be set together. Please refer to the device specific documentation for details.

The dev_id->dstat field provides the (accumulated) device status:

Please see the ESA/390 Principles of Operation manual for details on the individual flag meanings.

The driver must assure to hold the device specific lock. This can be accomplished by

1. s390irq_spin_lock( IRQ), or
2. s390irq_spin_lock_irqsave(IRQ, flags)

Option (i) should be used if the calling routine is running disabled for I/O interrupts already. Option (ii) obtains the device gate and puts the CPU into I/O disabled state by preserving the current PSW flags.

See the descriptions of s390irq_spin_lock() or s390irq_spin_lock_irqsave() for more details.

However, this should be used with care as it implies the channel will remain busy, not being able to process I/O requests for other devices on the same channel. Therefore, for example, read commands should never use this technique, as the result will be presented by a single interrupt anyway.

The previously mentioned flag allows exploitation of this feature, for example, for communication devices that can handle lost data on the network to allow for enhanced I/O processing.

This section describes various routines to be used in a Linux/390 device driver programming environment.

Linux/390 uses the following major and minor device numbers.

Linux manages S/390_s disk devices (DASD) via the DASD device driver. It is valid for all types of DASDs and represents them to Linux as block devices, namely "DASD". Currently the DASD driver uses a single major number (94) and 4 minor numbers per volume (1 for the physical volume and 3 for partitions). With respect to partitions see the following discussion. Thus you may have up to 64 DASD devices in your system.

The kernel parameter 'dasd=from-to,...' may be issued arbitrary times in the kernel's parameter line or not at all. The 'from' and 'to' parameters are to be given in hexadecimal notation without a leading 0x.

If you supply kernel parameters the different instances are processed in order of appearance and a minor number is reserved for any device covered by the supplied range up to 64 volumes. Additional DASDs are ignored. If you do not supply the 'dasd=' kernel parameter at all, the DASD driver registers all supported DASDs of your system to a minor number in ascending order of the subchannel number.

The driver currently supports ECKD-devices and there are stubs for support of the FBA and CKD architectures. For the FBA architecture only some smart data structures are missing to make the support complete.

We performed our testing on 3380 and 3390 type disks of different sizes, under VM and on the bare hardware (LPAR), using internal disks of the Multiprise as well as a RAMAC virtual array. Disks exported by an Enterprise Storage Server (Seascape) should work fine as well.

We currently implement one partition per volume, which is the whole volume, skipping the first blocks up to the volume label. These are reserved for IPL records and IBM's volume label to assure accessibility of the DASD from other operating systems. In a later stage we will provide support of partitions, maybe VTOC oriented or using a kind of partition table in the label record.

The following is a list of files, and their functions, which were added to the Linux distribution by the Linux/390 developers.

The Linux/390 system calls are implemented via SVC. Each call corresponds to a different SVC.

Linux/390 uses the following control register settings.

Notes:

1. This bit is set for VM configurations only. IUCV is used as a TCP/IP network driver.
2. Floating-point registers 0, 2, 4, and 6, are available on all ESA/390 models. The remaining 12 floating-point registers (1, 3, 5, and 7-15) are referred to as the additional floating-point (AFP) registers. The AFP registers are installed in the CPU when the basic-floating-point-extensions facility is installed and can be used only if bit 13 of control register 0, the AFP-register-control bit, is one. Attempting to use an AFP register when the basic-floating-point-extensions facility is not installed results in a specification exception. Attempting to use an AFP register when the basic-floating-point-extensions facility is installed and the AFP-register-control bit is zero results in an AFP-register data exception (DXC 1).

These registers are used for linkage-stack and address space operations. The registers are saved and restored for each task but never set.

This register is used for Monitor Calls (MC). The register is saved and restored for each task but never set.

These registers are used for Program Event Recording (PER). The registers saved and restored for each task but never set.

This register is used for tracing. The register is saved and restored for each task but never set.

This register is used for linkage-stack operations. The register is saved and restored for each task but never set.

Control Register 7 (secondary space control) and Control Register 13 (Home space control) are set to the user pgdir. The Kernel is running in its own, disjunct address space, and running in primary address space. A "Copy to/from user" is done via access register mode with access registers (AR2 and AR4) set to 0 or 1. For that purpose we need set up CR 7 with the user pgd.

The following section illustrates the IPL process from the VM reader.

When you first download the kernel image you will need to load it, the boot parameters and the RAMDISK from the VM reader. The RAMDISK contains just enough of a normal filesystem to complete the boot process. It will allow you to mount and configure "real" filesystems which can then take over as the root filesystem.

The initial boot parameters are as follows:

These parameters have the following meaning:

• EBCDIC - the parameters are encoded in EBCDIC
• Root=dev/ram0 ro - the root filesystem is located in the ramdisk device (this device is created at boot time when the ramdisk is read into storage) which is to be mounted in r/o mode during the boot process.

When Linux has a "real" root filesystem and booted, it requires only the kernel code and parameters to reside in the VM reader.

The parameters for this boot are more complex. They describe VM minidisks, CTC devices, the location of the root file system and the DASD to be included.

For a "full-functioned" Linux system IBM recommend a 128MB virtual machine:

As part of the boot process the network is brought online. The following netstat display shows the routing table for the system:

The network definitions responsible for the network activation are found in /etc/sysconfig/network:

And in /etc/sysconfig/network-scripts/ifcfg-ctc0:

With the network in place, it is now possible to telnet into the Linux/390 system and discard working from the emulated 3215 session:

Following an initial load of Linux/390 you will need to install either the "small" file system (120MB) or the "large" file system (400MB). If you have no access to TCP/IP you will be unable to FTP or NFS mount the files. You can copy a file system from one Linux system to another using a loopback device.

There are a number of steps to this, but they can be described as follows.

Debugging on Linux for 390

by
Denis Joseph Barrow (djbarrow@de.ibm.com, barrow_dj@yahoo.com)
Copyright (C) 2000 IBM Deutschland Entwicklung GmbH, IBM Corporation

This document is intended to give an good overview of how to debug Linux for S390 it isn't intended as a complete reference and not a tutorial on the fundamentals of C and assembly, it doesn't go into 390 IO in any detail. It is intended to compliment the following books.

• Enterprise Systems Architecture/390 Reference Summary SA22-7209-01 and the
• Enterprise Systems Architecture/390 Principles of Operation SA22-7201-05 and any other worthwhile references you get.

It is intended like the Enterprise Systems Architecture/390 Reference Summary to be printed out and used as a quick cheat sheet self help style reference when problems occur.

S390 Register Set

Address Spaces on Linux

The Linux for S390 Kernel Task Structure

Register Usage and Stackframes on Linux for S390 with glossary

Compiling programs for debugging on Linux for S390

Debugging Tools

Debugging under VM

Stack Tracing under VM

S390 IO Overview

Debugging IO on S390 under VM

GDB on S390

Examining core dumps

LDD

Debugging modules

The proc file system

Starting points for debugging scripting languages etc.

S390 Register Set

The current ESA 390 architecture has the following registers.

• 16 32 bit General propose registers (r0-r15 or gpr0-gpr15) used for arithmetic and addressing
• 16 Control registers (cr0-cr15 kernel usage only ) used for memory management, interrupt control,  debugging control etc.
• 16 Access registers (ar0-ar15) not used by normal programs but potentially could be used as temporary storage. Their main purpose is their 1 to 1 association with general purpose registers and are used in the kernel for copying data between address spaces.
• 16 64 bit floating point registers (fp0-fp15 ) IEEE and HFP floating point format compliant on G5 upwards and a Floating point control register (FPC)
• 4 64 bit registers (fp0, fp2, fp4 and fp6) HFP only on older machines.

Note: Linux (currently) always uses IEEE and emulates G5 IEEE format on older machines, (provided the kernel is configured for this ).

The PSW is the most important register on the machine it is 64 bit and serves the roles of a program counter (PC), condition code register, memory space designator.
In IBM standard notation I am counting bit 0 as the MSB. It has several advantages over a normal program counter in that you can change address translation and program counter  in a single instruction. To change address translation, e.g. switching address translation off requires that you have a logical=physical mapping for the address you are currently running at.

Prefix Page

This per CPU memory area is too intimately tied to the processor not to mention. It exists between the real addresses 0-4096 on the processor and is exchanged with a page in absolute storage by the set prefix instruction in Linux's startup. This page different on each processor. Bytes 0-512 (200 hex) are used by the processor  itself for holding such information as exception indications and entry points for exceptions.

Linux uses bytes after 0xc00 hex for per processor globals. The closest thing to this on traditional architectures is the interrupt vector table. This is a good thing and does simplify some of the kernel coding however it means that we now cannot catch stray NULL pointers in the kernel without hard coded checks.

Address Spaces on Linux

The traditional Intel Linux is approximately mapped as follows:

Now it is easy to see that on Intel it is quite easy to recognize a kernel address as being one greater than user space high memory (in this case 0xC0000000). Addresses of less than this are the ones in the current running program on this processor (if an SMP box). If using the virtual machine (VM) as a debugger it is quite difficult to know which user process is running as the address space you are looking at could be from any process in the run queue. Thankfully you normally get lucky as address spaces don't overlap that and you can recognize the code at by cross-referencing with a dump made by objdump (more about that later).

The limitation of Intels addressing technique is that the Linux kernel uses a very simple real address to virtual addressing technique of Real Address=Virtual Address-User Space Himem. This means that on Intel the kernel Linux can typically only address Himem=0xFFFFFFFF-0xC0000000=1GB and this is all the RAM these machines can typically use. They can lower User Himem to 2GB or lower and thus be able to use 2GB of RAM however this shrinks the maximum size of User Space from 3GB to 2GB they have a no win limit of 4GB unless they go to 64 Bit.

On S/390 our limitations and strengths make us slightly different. For backward compatibility we are only allowed use 31 bits (2GB) of our 32 bit addresses, however, we use entirely separate address spaces for the user and kernel. This means we can support 2GB of non-extended RAM, and more with the extended memory management swap device and 64 Bit when it comes along.

Address Spaces on Linux for S390

Our addressing scheme is as follows:

This also means that we need to look at the PSW problem state bit or the addressing mode to decide whether we are looking at user or kernel space.

The Linux for S390 Kernel Task Structure

Each process/thread under Linux for S390 has its own kernel task_struct defined in linux/include/linux/sched.h. The S390 on initialisation and resuming of a process on a cpu sets the __LC_KERNEL_STACK variable in the spare prefix area for this cpu (which we use for per processor globals). The kernel stack pointer is intimately tied with the task structure for each processor as follows.

What this means is that we don't need to dedicate any register or global variable to point to the current running process and can retrieve it with the following very simple construct: 

static inline struct task_struct * get_current(void)
{
struct task_struct *current;
__asm__("lhi %0,-8192\n\t"
        "nr  %0,15"
        : "=r" (current) );
return current;
}

That is, just and'ing the current kernel stack pointer with the mask -8192. Thankfully because Linux doesn't have support for nested IO interrupts and our devices have large buffers can survive interrupts being shut for short amounts of time we don't need a separate stack for interrupts.

Register Usage and Stackframes on Linux for S390

This is the code that gcc produces at the top and the bottom of each function. It usually is fairly consistent and similar from function to function and if you know its layout you can probably make some headway in finding the ultimate cause of a problem after a crash without a source level debugger.

Note: To follow stackframes requires knowledge of C or Pascal and limited knowledge of one assembly language.

Glossary

int FunctionA(int a) 
{ 
 int b; 
 FunctionC(int c) 
 { 
  b=c+1; 
 } 
 FunctionC(10); 
 return(b); 
}

S390 Register Usage

Notes:

1. The only requirement is that registers which are used by the callee are saved, e.g. the compiler is perfectly capible of using r11 for purposes other than a frame a frame pointer if a frame pointer is not needed.
2. In functions with variable arguments e.g. printf the calling procedure is identical to one without variable arguments and the same number of parameters. However, the prologue of this function is somewhat more hairy owing to it having to move these parameters to the stack to get va_start, va_arg and va_end to work.
3. Access registers are currently unused by gcc but are used in the kernel. Possibilities exist to use them at the moment for temporary storage but it isn't recommended.
4. Only 4 of the floating point registers are used for parameter passing as older machines such as G3 only have only 4 and it keeps the stack frame compatible with other compilers. However with IEEE floating point emulation under Linux on the older machines you are free to use the other 12.
5. A long long or double parameter cannot be have the first 4 bytes in a register and the second four bytes in the outgoing args area. It must be purely in the outgoing args area if crossing this boundary.
6. Floating point parameters are mixed with outgoing args on the outgoing args area in the order the are passed in as parameters.

Stack Frame Layout

A sample program with comments

Comments on the function test

1. It didn't need to set up a pointer to the constant pool gpr13 as it isn't used ( :-( ).
2. This is a frameless function and no stack is bought.
3. The compiler was clever enough to recognise that it could return the value in r2 as well as use it for the passed in parameter ( :-) ).
4. The basr ( branch relative and save ) trick works as follows the instruction has a special case with r0,r0 with some instruction operands is understood as
   the literal value 0, some risc architectures also do this ). So now we are branching to the next address and the address new program counter is in r13,so now we subtract the size of the function prologue we have executed + the size of the literal pool to get to the top of the literal pool

0040037c int test(int b) 
{ # Function prologue below 
40037c: 90 de f0 34 	stm 	%r13,%r14,52(%r15) 	# Save registers r13 & r14 
400380: 0d d0 	basr 	%r13,%r0 		# Set up pointer to constant pool using 
400382: a7 da ff fa 	ahi 	%r13,-6 		# basr trick 
return(5+b); 
# Huge main program 
400386: a7 2a 00 05 	ahi 	%r2,5 			# add 5 to r2 

# Function epilogue below 
40038a: 98 de f0 34 	lm 	%r13,%r14,52(%r15) 	# restore registers r13 & 14 
40038e: 07 fe 	br 	%r14 			# return 
} 

Comments on the function main

The compiler did this function optimally ( 8-) )

Literal pool for main. 
400390: ff ff ff ec 	.long 0xffffffec 
main(int argc,char *argv[]) 
{ # Function prologue below 
400394: 90 bf f0 2c 	stm 	%r11,%r15,44(%r15) 	# Save necessary registers 
400398: 18 0f 		lr 	%r0,%r15 		# copy stack pointer to r0 
40039a: a7 fa ff a0 	ahi 	%r15,-96 		# Make area for callee saving 
40039e: 0d d0 		basr 	%r13,%r0 		# Set up r13 to point to 
4003a0: a7 da ff f0 	ahi 	%r13,-16 		# literal pool 
4003a4: 50 00 f0 00 	st 	%r0,0(%r15) 		# Save backchain 
return(test(5)); 						# Main Program Below 
4003a8: 58 e0 d0 00 	l 	%r14,0(%r13) 		# load relative address of test from 
# literal pool 
4003ac: a7 28 00 05 	lhi 	%r2,5 			# Set first parameter to 5 
4003b0: 4d ee d0 00 	bas 	%r14,0(%r14,%r13) 	# jump to test setting r14 as return 
 								# address using branch & save instruction. 
# Function Epilogue below 
4003b4: 98 bf f0 8c 	lm 	%r11,%r15,140(%r15)	# Restore necessary registers. 
4003b8: 07 fe 		br 	%r14 			# return to do program exit 
} 

Compiling programs for debugging on Linux for S390

Make sure that the gcc is compiling and linking with the -g flag on. This is typically done adding/appending the flags -g to the CFLAGS and LDFLAGS variables Makefile of the program concerned.

If using gdb and you would like accurate displays of registers and stack traces compile without optimization. That is, make sure that there is no -O2 or similar on the CFLAGS line of the Makefile and the emitted gcc commands, obviously this will produce worse code (not advisable for shipment) but it is an aid to the debugging process.

This aids debugging because the compiler will copy parameters passed in in registers onto the stack so backtracing and looking at passed in parameters will work, however some larger programs which use inline functions will not compile without optimisation.

Figuring out gcc compile errors

If you are getting a lot of syntax errors compiling a program and the problem isn't blatantly obvious from the source. It often helps to just preprocess the file, this is done with the -E option in gcc. What this does is that it runs through the very first phase of compilation (compilation in gcc is done in several stages and gcc calls many programs to achieve its end result) with the -E option gcc just calls the gcc preprocessor (cpp). The c preprocessor does the following, it joins all the files #included together recursively ( #include files can #include other files ) and also the c file you wish to compile. It puts a fully qualified path of the #included files in a comment and it does macro expansion. This is useful for debugging because

1. You can double check whether the files you expect to be included are the ones that are being included (e.g. double check that you aren't going to the i386 asm directory ).
2. Check that macro definitions aren't clashing with typedefs,
3. Check that definitions aren't being used before they are being included.
4. Helps put the line emitting the error under the microscope if it contains macros.

For convenience the Linux kernel's makefile will do preprocessing automatically for you by suffixing the file you want built with .i (instead of .o). For example, from the Linux directory type:

make arch/s390/kernel/signal.i 

this will build :

s390-gcc -D__KERNEL__ -I/home1/barrow/linux/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer 
-fno-strict-aliasing -D__SMP__ -pipe -fno-strength-reduce -E arch/s390/kernel/signal.c 
arch/s390/kernel/signal.i

Now look at signal.i you should see something like: 

# 1 "/home1/barrow/linux/include/asm/types.h" 1 

typedef unsigned short umode_t; 
typedef __signed__ char __s8; 
typedef unsigned char __u8; 
typedef __signed__ short __s16; 
typedef unsigned short __u16; 

If instead you are getting errors further down. For example, unknown instruction: 2515 "move.l" or better still unknown instruction:2515 "Fixme not implemented yet, call Martin" you are probably are attempting to compile some code meant for another architecture or code that is simply not implemented, with a fixme statement stuck into the inline assembly code so that the author of the file now knows he has work to do. To look at the assembly emitted by gcc just before it is about to call gas (the gnu assembler) use the -S option. Again for your convenience the Linux kernel's Makefile will hold your hand and do all this donkey work for you also by building the file with the .s suffix. For example, from the Linux directory type:

make arch/s390/kernel/signal.s 
s390-gcc -D__KERNEL__ -I/home1/barrow/linux/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer 
-fno-strict-aliasing -D__SMP__ -pipe -fno-strength-reduce -S arch/s390/kernel/signal.c 
-o arch/s390/kernel/signal.s   

This will output something like, (please note the constant pool and the useful comments in the prologue to give you a hand at interpreting it).

.LC54: 
.string "misaligned (__u16 *) in __xchg\n" 
.LC57: 
.string "misaligned (__u32 *) in __xchg\n" 
.L$PG1: # Pool sys_sigsuspend 
.LC192: 
.long -262401 
.LC193: 
.long -1 
.LC194: 
.long schedule-.L$PG1 
.LC195: 
.long do_signal-.L$PG1 
.align 4
.globl sys_sigsuspend 
.type sys_sigsuspend,@function 
sys_sigsuspend: 
# leaf function 0 
# automatics 16 
# outgoing args 0 
# need frame pointer 0 
# call alloca 0 
# has varargs 0 
# incoming args (stack) 0 
# function length 168 
        STM    8,15,32(15) 
        LR     0,15 
        AHI    15,-112 
        BASR   13,0 
.L$CO1: AHI    13,.L$PG1-.L$CO1 
        ST     0,0(15) 
        LR     8,2 
        N      5,.LC192-.L$PG1(13) 

Debugging Tools

objdump

This is a tool with many options the most useful being ( if compiled with -g).

objdump --source <victim program or object file> > <victims debug listing>

The whole kernel can be compiled like this (doing this will make a 17MB kernel and a 200 MB listing ) however you have to strip it before building the image using the strip command to make it a more reasonable size to boot it.

A source/assembly mixed dump of the kernel can be done with the line:

objdump --source vmlinux > vmlinux.lst 

Also if the file isn't compiled -g this will output as much debugging information as it can (for example, function names). However, this is very slow as it spends lots of time searching for debugging info, the following self explanatory line should be used instead if the code isn't compiled -g.

objdump --disassemble-all --syms vmlinux > vmlinux.lst 

as it is much faster.

As hard drive space is valuable most of us use the following approach.

1. Look at the emitted PSW on the console to find the crash address in the kernel.
2. Look at the file System.map (in the linux directory) produced when building the kernel to find the closest address less than the current PSW to find the
   offending function.
3. Use grep or similar to search the source tree looking for the source file with this function if you don't know where it is.
4. Rebuild this object file with -g on, as an example suppose the file was /arch/s390/kernel/signal.o
5. Assuming the file with the erroneous function is signal.c. Move to the base of the Linux source tree
• Erase the existing object file: rm /arch/s390/kernel/signal.o
• Build the object file: make /arch/s390/kernel/signal.o
• Watch the gcc command line emitted
• Type it in again or alternatively cut and paste it on the console adding the -g option.
• objdump --source arch/s390/kernel/signal.o > signal.lst
  This will output the source and the assembly intermixed, as the snippet below shows. This will unfortunately output addresses which aren't the same as the kernel ones you should be able to get around the mental arithmetic by playing with the --adjust-vma parameter to objdump.

extern inline void spin_lock(spinlock_t *lp) 
{ 
a0: 18 34 		lr 	%r3,%r4 
a2: a7 3a 03 bc 		ahi 	%r3,956 
__asm__ __volatile("	lhi 	1,-1\n" 
a6: a7 18 ff ff 		lhi 	%r1,-1 
aa: 1f 00 		slr 	%r0,%r0 
ac: ba 01 30 00 		cs 	%r0,%r1,0(%r3) 
b0: a7 44 ff fd 		jm 	aa <sys_sigsuspend+0x2e> 
saveset = current-blocked; 
b4: d2 07 f0 68 		mvc 	104(8,%r15),972(%r4) 
b8: 43 cc 
return (set-sig[0] and mask) != 0; 
} 

6. If debugging under VM go down to that section in the document for more info.

strace

Q. What is it ?
A. It is a tool for intercepting calls to the kernel and logging them to a file and on the screen.

Q. What use is it ?
A. You can used it to find out what files a particular program opens.

Example 1

If you wanted to know does ping work but didn't have the source:

strace ping -c 1 127.0.0.1 

and then look at the man pages for each of the syscalls below, (In fact this is sometimes easier than looking at some spaghetti source which conditionally compiles for several architectures). Not everything that it throws out needs to make sense immediately.

Just looking quickly you can see that it is making up a RAW socket for the ICMP protocol. Doing an alarm(10) for a 10 second timeout and doing a gettimeofday() call before and after each read to see how long the replies took, and writing some text to stdout so the user has an idea what is going on.

socket(PF_INET, SOCK_RAW, IPPROTO_ICMP) = 3 
getuid() = 0 
setuid(0) = 0 
stat("/usr/share/locale/C/libc.cat", 0xbffff134) = -1 ENOENT (No such file or directory) 
stat("/usr/share/locale/libc/C", 0xbffff134) = -1 ENOENT (No such file or directory) 
stat("/usr/local/share/locale/C/libc.cat", 0xbffff134) = -1 ENOENT (No such file or directory) 
getpid() = 353 
setsockopt(3, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0 
setsockopt(3, SOL_SOCKET, SO_RCVBUF, [49152], 4) = 0 
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(3, 1), ...}) = 0 
mmap(0, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40008000 
ioctl(1, TCGETS, {B9600 opost isig icanon echo ...}) = 0 
write(1, "PING 127.0.0.1 (127.0.0.1): 56 d"..., 42PING 127.0.0.1 (127.0.0.1): 56 data bytes 
) = 42 
sigaction(SIGINT, {0x8049ba0, [], SA_RESTART}, {SIG_DFL}) = 0 
sigaction(SIGALRM, {0x8049600, [], SA_RESTART}, {SIG_DFL}) = 0 
gettimeofday({948904719, 138951}, NULL) = 0 
sendto(3, "\10\0D\201a\1\0\0\17#\2178\307\36"..., 64, 0, {sin_family=AF_INET, 
sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}, 16) = 64 
sigaction(SIGALRM, {0x8049600, [], SA_RESTART}, {0x8049600, [], SA_RESTART}) = 0
sigaction(SIGALRM, {0x8049ba0, [], SA_RESTART}, {0x8049600, [], SA_RESTART}) = 0
alarm(10) = 0 
recvfrom(3, "E\0\0T\0005\0\0@\1|r\177\0\0\1\177"..., 192, 0, 
{sin_family=AF_INET, sin_port=htons(50882), sin_addr=inet_addr("127.0.0.1")}, [16]) = 84 
gettimeofday({948904719, 160224}, NULL) = 0 
recvfrom(3, "E\0\0T\0006\0\0\377\1\275p\177\0"..., 192, 0, 
{sin_family=AF_INET, sin_port=htons(50882), sin_addr=inet_addr("127.0.0.1")}, [16]) = 84 
gettimeofday({948904719, 166952}, NULL) = 0 
write(1, "64 bytes from 127.0.0.1: icmp_se"..., 
5764 bytes from 127.0.0.1: icmp_seq=0 ttl=255 time=28.0 ms 

Example 2

strace passwd 2>&1 | grep open 
produces the following output 
open("/etc/ld.so.cache", O_RDONLY) = 3 
open("/opt/kde/lib/libc.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/lib/libc.so.5", O_RDONLY) = 3 
open("/dev", O_RDONLY) = 3
open("/var/run/utmp", O_RDONLY) = 3
open("/etc/passwd", O_RDONLY) = 3
open("/etc/shadow", O_RDONLY) = 3
open("/etc/login.defs", O_RDONLY) = 4 
open("/dev/tty", O_RDONLY) = 4

The 2>&1 is done to redirect stderr to stdout and grep is then filtering this input through the pipe for each line containing the string open.

Example 3

Now we are getting sophisticated: telnetd crashes on and I don't know why

1. Replace the following line in /etc/inetd.conf: telnet stream tcp nowait root /usr/sbin/in.telnetd -h with telnet stream tcp nowait root /blah
2. Create the file /blah with the following contents to start tracing telnetd

#!/bin/bash 
/usr/bin/strace -o/t1 -f /usr/sbin/in.telnetd -h

3. chmod 700 /blah to make it executable only to root
4. killall -HUP inetd or ps aux | grep inetd. get inetd's process id and kill -HUP inetd to restart it.

Important options

• -o is used to tell strace to output to a file in our case t1 in the root directory
• -f is to follow children. For example, in our case above telnetd will start the login process & subsequently a shell like bash. You will be able to tell which is which from the process ID's listed on the left hand side of the strace output.
• -p<pid> will tell strace to attach to a running process, yup this can be done provided it isn't being traced or debugged already & you have enough privileges, the reason 2 processes cannot trace or debug the same program is that strace becomes the parent process of the one being debugged & processes (unlike people) can have only one parent.

However the file /t1 will get big quite quickly to test it telnet 127.0.0.1. Now look at what files in.telnetd execve'd

413 execve("/usr/sbin/in.telnetd", ["/usr/sbin/in.telnetd", "-h"], [/* 17 vars */]) = 0 
414 execve("/bin/login", ["/bin/login", "-h", "localhost", "-p"], [/* 2 vars */]) = 0

Whey it worked!

Other hints

If the program is not very interactive (i.e. not much keyboard input) and is crashing in one architecture but not in another you can do an strace of both programs under as identical a scenario as you can on both architectures outputting to a file then. do a diff of the two traces using the diff program. i.e.

and maybe you'll be able to see where the call paths differed, this is possibly near the cause of the crash.

More information

Look at man pages for strace and the various syscalls e.g. man strace, man alarm, man socket.

Debugging under VM

Addresses and values in the VM debugger are always hex never decimal. Address ranges are of the format <HexValue1>-<HexValue2> or <HexValue1>.<HexValue2.> e.g. The address range 0x2000 to 0x3000 can be described as 2000-3000 or 2000.1000.

The VM Debugger is case insensitive.

VM's strengths are usually other debuggers weaknesses you can get at any resource no matter how sensitive e.g. memory management resources, change address translation in the PSW. For kernel hacking you will reap dividends if you get good at it.

The VM Debugger displays operators but not operands, probably because some of it was written when memory was expensive and the programmer was probably proud that it fitted into 2k of memory and the programmers didn't want to shock hardcore VM'ers by changing the interface :-). Also, the debugger displays useful information on the same line and the author of the code probably felt that it was a good idea not to go over the 80 columns on the screen.

As some of you are probably in a panic now this isn't as unintuitive as it may seem as the 390 instructions are easy to decode mentally and you can make a good guess at a lot of them as all the operands are nibble (half byte aligned ). If you have an objdump listing also it is quite easy to follow. If you don't have an objdump listing keep a copy of the ESA Reference Summary and look at between pages 2 and 7 or alternatively the ESA principles of operation. e.g. even I can guess that

0001AFF8' LR 180F CC 0 

is a ( load register ) lr r0,r15

Also it is very easy to tell the length of a 390 instruction from the 2 most significant bits in the instruction (not that this info is really useful except if you are trying to make sense of a hexdump of code):

The debugger also displays other useful info on the same line such as the addresses being operated on destination addresses of branches and condition codes. For example:

00019736' AHI 	A7DAFF0E 			CC 1 
000198BA' BRC 	A7840004 	-> 000198C2'	CC 0 
000198CE' STM 	900EF068 	0FA95E78 	CC 2 

Useful VM debugger commands

I suppose I'd better mention this before I start to list the current active traces do

Q TR 

there can be a maximum of 255 of these per set (more about trace sets later). To stop traces issue:

TR END

To delete a particular breakpoint issue

TR DEL <breakpoint number> 

The PA1 key drops to CP mode so you can issue debugger commands, Doing alt-c (on my 3270 console at least ) clears the screen. Hitting b <enter> comes back to the running operating system from CP mode (in our case Linux ).

It is typically useful to add shortcuts to your PROFILE EXEC file if you have one (this is roughly equivalent to autoexec.bat in DOS or .profile in Linux). Here are a few from mine:

/* this gives me command history on issuing f12 */ 
set pf12 retrieve 
/* this continues */ 
set pf8 imm b 
/* goes to trace set a */ 
set pf1 imm tr goto a 
/* goes to trace set b */ 
set pf2 imm tr goto b 
/* goes to trace set c */ 
set pf3 imm tr goto c 

Instruction Tracing

Setting a simple breakpoint:

TR I PSWA <address> 

To debug a particular function try:

• TR I R <function address range>
• TR I on its own will single step.
• TR I DATA <MNEMONIC> <OPTIONAL RANGE> will trace for particular mnemonics. For example, TR I DATA 4D R 0197BC.4000 will trace for BAS'es (opcode 4D) in the range 0197BC.4000 if you were inclined you could add traces for all branch instructions and suffix them with the run prefix so you would have a backtrace on screen when a program crashes.
• TR BR <INTO OR FROM> will trace branches into or out of an address. For example, TR BR INTO 0 is often quite useful if a program is getting awkward and deciding to branch to 0 and crashing as this will stop at the address before in jumps to 0.  

Displaying and Modifying Registers

• D G will display all the gprs
• D X will display all the control registers
• D AR will display all the access registers
• D AR4-7 will display access registers 4 to 7
• CPU ALL D G will display the GRPS of all CPUS in the configuration
• D PSW will display the current PSW
• ST PSW 2000 will put the value 2000 into the PSW and probably crash your machine.
• D PREFIX will display the prefix register

Displaying Memory

To display memory that was mapped using the current PSW's mapping try:

D <range>

To make VM display a message each time it hits a particular address and continue try:

• D I<range> will disassemble and display a range of instructions.
• ST addr <32 bit word> will store 32 bits of data data at a 32 bit aligned address
• ST Saddr <byte-sequence> will store a byte sequence at a byte aligned address
• D T<range> will display the EBCDIC in an address (if you are that way inclined)
• D R<range> will display real addresses, without DAT, but with prefixing.

There are other complex options to display if you need to get at say home space but are in primary space the easiest thing to do is to temporarily
modify the PSW to the other addressing mode, display the stuff and then restore it.

Hints

If you want to issue a debugger command without halting your virtual machine with the PA1 key, then try prefixing the command with #CP:

#cp tr i pswa 2000 

Also suffixing most debugger commands with RUN will cause them not to stop just display the mnemonic at the current instruction on the console. If you have several breakpoints you want to put into your program and you get fed up of cross referencing with System.map you can do the following trick for several symbols.:

grep do_signal System.map 

which emits the following among other things:

0001f4e0 T do_signal 

Now you can do:

TR I PSWA 0001f4e0 cmd msg * do_signal 

This sends a message to your console each time do_signal is entered. (As an aside I wrote a perl script once which automatically generated a REXX script with breakpoints on every kernel procedure. This isn't a good idea because there are thousands of these routines and VM can only set 255 breakpoints at a time, you nearly had to spend as long pruning the file down as you would enter the messages by hand). However, the trick might be useful for a single object file.

Tracing Program Exceptions

If you get a crash which says something like illegal operation or specification exception followed by a register dump You can restart Linux and trace these using the tr prog <range | value> option.

The most common ones you will normally be tracing for is:

The full list of these is on page 22 of the current ESA Reference Summary. For example:

• tr prog 10 will trace segment translation exceptions.
• tr prog on its own will trace all program interruption codes.

Trace Sets

On starting VM you are initially in the INITIAL trace set. You can do a Q TR to verify this. If you have a complex tracing situation where you wish to wait for instance till a driver is open before you start tracing IO, but know in your heart that you are going to have to make several runs through the code till you
have a clue whats going on.

What you can do is:

TR I PSWA <Driver open address>

Enter b to continue until the breakpoint is reached. Now do your:

TR GOTO B 

To got back to the initial trace set do:

TR GOTO INITIAL 

and the TR I PSWA <Driver open address> will be the only active breakpoint again.

Tracing Linux syscalls under VM

Syscalls are implemented on Linux for S390 by the Supervisor call instruction (SVC) there 256 possibilities of these as the instruction is made up of a 0x0a opcode and the second byte being the syscall number. They are traced using the simple command:

TR SVC <Optional value or range>

The syscalls are defined in linux/include/asm-s390/unistd.h. For example, to trace all file opens just do:

Producing Trace of System Flow

VM's ability to trace branch operations allows the production of system flow data. The output can (and is) quite voluminous but can be made human-friendly by the following process.

1. FTP the System.map to the VM user who will process the flow data or allow the EXEC to FTP it for you each time.
2. Create the trace set that will produce a printout of all branch operations. You can start the trace prior to booting Linux or prior to running your daemon or application:

#CP TR BR PRINT

3. Redirect the output of the command to a user who will process the data:

#CP SP P <user>

4. When you are ready to process the data, end the trace and close the print file:

#CP TR END
#CP CLOSE P

5. Run the following EXEC to process the trace:

/* */
parse upper arg Option .
signal on SYNTAX
MaxSym = 0
Cache. = ''
if Option <> '' then
   Stage  = ''
else
   Stage  = '| nlocate /memset/',
            '| nlocate /memcpy/',
            '| nlocate /memcmp/',
            '| nlocate /update_wall_time/',
            '| nlocate /printk/',
            '| nlocate /strcpy/',
            '| nlocate /strncpy/',
            '| nlocate /strcmp/',
            '| nlocate /strncmp/',
            '| nlocate /strchr/',
            '| nlocate /strlen/',
            '| nlocate /ExternalException/',
            '| nlocate /External+/',
            '| nlocate /do_timer/',
            '| nlocate /vsprintf/',
            '| nlocate /set_bit/',
            '| nlocate /free_pages/',
            '| nlocate /mem_init/'
'PIPE (name READ_MAP)',
   '| ftp ftp://<user>:<password>@<host.domain>/linux/System.map binary',
   '| xlate from 437 to 1047',
   '| deblock c',
   '| drop 1',
   '| strip',
   '| locate 1',
   '| nfind U' ||,
   '| spec w1 x2c 1 w1-* nw',
   '| stem Map.'
'PIPE (name READ_TRACE end ?)',
   '| reader',
   '| mctoasa',
   '| spec 2-* 1',
   '| a: locate /BASR/',
   '| b: faninany',
   '| spec w2-4 1 w6 nw',
   '| stem Trace.',
   '? a:',
   '| c: locate /LPSW/',
   '| b:',
   '? c:',
   '| locate / 07FE /',
   '| b:'
do I_Trace = 1 to Trace.0
   if ((I_Trace // 5000) = 0) then
      say '...'I_Trace
   parse var Trace.I_Trace From Branch BrType To
   if (Cache.From = '') then
   do
      FromSym = GET_ADDR(From,'F')
      Cache.From = FromSym
   end
   else
      FromSym = Cache.From
   if (Cache.To = '') then
   do
      ToSym   = GET_ADDR(To,'T')
      Cache.To = ToSym
   end
   else
      ToSym = Cache.To
   select
      when BrType = '0D1E' then
           Branch = '<---'
      when BrType = '0DEF' then
           Branch = '--->'
      when BrType = '07FE' then
           Branch = '<---'
      otherwise
           Branch = '--->'
   end
   Flow.I_Trace = FromSym LEFT(Branch,8) ToSym
end
Flow.0 = Trace.0
'PIPE (name WRITE_FLOW end ?)',
   '| stem Flow.',
   '| spec w1 1.'MaxSym 'w2-* nw',
   Stage,
   '| spec 1-* 5',
   '| > LINUX FLOW A'
exit

GET_ADDR:
   parse arg VAddr,Type
   parse var VAddr Addr"'"
   XAddr = X2C(Addr)
   Target    = Map.0
   NewTarget = 1
   Disp      = Map.0 % 2
   LastLow   = Target
   do forever
      parse var Map.Target 1 XSym 5 . . Symbol
      if XAddr = XSym then
      do
         LastLow = Target
         leave
      end
      if XAddr < XSym then
      do
         NewTarget = Target - Disp
         if (LastLow = NewTarget) then
            leave
         LastLow   = NewTarget
      end
      else
         NewTarget = Target + Disp
      Disp = Disp % 2
      if (Disp < 1) then
         Disp = 1
      Target = NewTarget
   end
   parse var Map.LastLow 1 XSym 5 . . Symbol
   Disp = X2D(Addr) - C2D(XSym)
   if Disp <> 0 then
      Symbol = Symbol'+'D2X(Disp)
   if ((LENGTH(Symbol) > MaxSym) & (Type = 'F')) then
      MaxSym = LENGTH(Symbol)
return Symbol

SYNTAX:
  say 'Error:' ERRORTEXT(Rc) 'at line' Sigl
  say SOURCELINE(Sigl)
  trace ?r; nop
exit -1

SMP Specific commands

• To find out how many CPUs you have:

• To find the cpu that the current cpu VM debugger commands are being directed at do:

Q CPU

• To change the current CPU VM debugger commands are being directed at do:

CPU <desired cpu no>

• On an SMP guest issue a command to all CPUs try prefixing the command with CPU ALL.

• To issue a command to a particular CPU, try CPU <cpu number>. For example:

CPU 01 TR I R 2000.3000 

• If you are running on a guest with several cpus and you have an I/O related problem and cannot follow the flow of code but you know it isn't SMP related. Then from the bash prompt issue:

DETACH CPU 01-<number of CPUs in configuration> 

• TR SIGP will trace inter processor signal processor instructions.  

Help for displaying ASCII text

Currently, text cannot be displayed in ASCII under the VM debugger (I love EBDIC too), I have written this little program which will convert a command line of hex digits to ASCII text which can be compiled under Linux and you can copy the hex digits from your x3270 terminal to your xterm if you are debugging from a Linux box.

This is quite useful when looking at a parameter passed in as a text string under VM (unless you are good at decoding ASCII in your head). For example, consider tracing an open syscall:

We have stopped at a breakpoint

000151B0' SVC 0A05 - 0001909A' CC 0 

Use D P SVC to check the SVC old PSW in the prefix area and see was it from user-space (for the layout of the prefix area consult page18 of the ESA 390 Reference Summary if you have it available).

SVC 0005  20 OLD 070C2000 800151B2   60  NEW 04080000 8001909A

The problem state bit wasn't set and it's also too early in the boot sequence for it to be a user-space SVC if it was we would have to temporarily switch the PSW to user space addressing so we could get at the first parameter of the open in gpr2. To display the parameter:

Now copy the text till the first 00 hex (which is the end of the string) to an xterm and do hex2ascii on it:

hex2ascii 2F646576 2F636F6E 736F6C65 00 

The resulting output is:

Decoded Hex:=/ d e v / c o n s o l e 0x00 

We were opening the console device.

/* 
* hex2ascii.c 
* a useful little tool for converting a hexadecimal command line to ascii 
* 
* Author(s): Denis Joseph Barrow (djbarrow@de.ibm.com,barrow_dj@yahoo.com) 
* (C) 2000 IBM Deutschland Entwicklung GmbH, IBM Corporation. 
*/ 
#include <stdio.h 
int main(int argc,char *argv[]) 
{ 
 int cnt1,cnt2,len,toggle=0; 
 int startcnt=1; 
 unsigned char c,hex; 

 if(argc1&&(strcmp(argv[1],"-a")==0)) 
    startcnt=2; 
 printf("Decoded Hex:="); 
 for(cnt1=startcnt;cnt1<argc;cnt1++) 
 { 
    len=strlen(argv[cnt1]); 
    for(cnt2=0;cnt2<len;cnt2++) 
    { 
       c=argv[cnt1][cnt2]; 
       if(c='0'&&c<='9') 
          c=c-'0'; 
       if(c='A'&&c<='F') 
          c=c-'A'+10; 
       if(c='a'&&c<='F') 
          c=c-'a'+10; 
       switch(toggle) 
       { 
          case 0: 
             hex=c<<4;
             toggle=1; 
             break; 

          case 1: 
             hex+=c; 
             if(hex<32||hex127) 
             { 
                if(startcnt==1) 
                   printf("0x%02X ",(int)hex); 
                else 
                   printf("."); 
             } 
             else 
             { 
                printf("%c",hex);
                if(startcnt==1)
                   printf(" "); 
             }
             toggle=0;
             break; 
        }
    } 
 } 
 printf("\n");
} 

Alternatively, the following CMS PIPELINE will achieve the same thing:

/* */
parse arg XString
'PIPE (name E2A)',
   '| var Xstring',
   '| change / //',		/* Remove any blanks within string */
   '| spec 1-* x2c 1',    	/* Convert graphic hex to binary   */
   '| xlate from 437 to 1047',    /* Choose the code-page you prefer */
   '| cons' 

Stack tracing under VM

Here are the tricks I use 9 out of 10 times it works pretty well.

When your backchain reaches a dead end

This can happen when an exception happens in the kernel and the kernel is entered twice if you reach the NULL pointer at the end of the back chain you should be able to sniff further back if you follow the following tricks.

1. A kernel address should be easy to recognise since it is in primary space and the problem state bit isn't set and also The Hi bit of the address is set.
2. Another backchain should also be easy to recognise since it is an address pointing to another address approximately 100 bytes or 0x70 hex behind the current stackpointer.

Here is some practice.

• boot the kernel and hit PA1 at some random time

• d g to display the gprs, this should display something like

GPR 0  = 00000001 00156018 0014359C 00000000 
GPR 4  = 00000001 001B8888 000003E0 00000000
GPR 8  = 00100080 00100084 00000000 000FE000
GPR 12 = 00010400 8001B2DC 8001B36A 000FFED8 

• Note that GPR14 is a return address but as we are real men we are going to trace the stack. Display 0x40 bytes after the stack pointer:

D 0.40;BASEF
V000FFED8 000FFF38 8001B838 80014C8E 000FFF38
V000FFEE8 00000000 00000000 000003E0 00000000
V000FFEF8 00100080 00100084 00000000 000FE000
V000FFF08 00010400 8001B2DC 8001B36A 000FFED8   

• Ah now look at what is in sp+56 (sp+0x38) this is 8001B36A our saved r14 if you look above at our stackframe and also agrees with GPR14. Now backchain:

d 000FFF38.40 

• We now are taking the contents of SP to get our first backchain.

V000FFF38 000FFFA0 00000000 00014995 00147094
V000FFF48 00147090 001470A0 000003E0 00000000
V000FFF58 00100080 00100084 00000000 001BF1D0 
V000FFF68 00010400 800149BA 80014CA6 000FFF38 

• Now do d 000FFFA0.40 for our 3rd backchain

V000FFFA0 04B52002 0001107F 00000000 00000000 
V000FFFB0 00000000 00000000 FF000000 0001107F 
V000FFFC0 00000000 00000000 00000000 00000000
V000FFFD0 00010400 80010802 8001085A 000FFFA0 

• Now look at System.map to see if the addresses make any sense:

• Next:

grep -i 00014 System.map 

00014a78 T start_kernel 

• Then:

grep -i 00108 System.map 

00010800 T _stext 

• Congratulations you've performed your first backchain!  

S390 IO Overview

I am not going to give a course in 390 IO architecture as this would take me quite a while and I'm no expert. Instead I'll give a 390 IO architecture summary for Dummies if you have the ESA principles of operation available read this instead. If nothing else you may find a few useful keywords in here and be able to use them on a web search engine like Altavista to find more useful information.

Unlike other bus architectures modern 390 systems do their IO using mostly fibre optics and devices such as tapes and disks can be shared between several mainframes, also S390 can support up to 65536 devices while a high end PC based system might be choking with around 64. Here is some of the common IO terminology.

General Concepts

Input Output Processors (IOP's) are responsible for communicating between the mainframe CPU's and the channel and relieve the mainframe CPU's from the burden of communicating with IO devices directly, this allows the CPU's to concentrate on data processing.

IOP's can use one or more links (known as channel paths) to talk to each IO device. It first checks for path availability and chooses an available one, then starts (and sometimes terminates IO). There are two types of channel path ESCON and the Parallel IO interface.

IO devices are attached to control units. Control units provide the logic to interface the channel paths and channel path IO protocols to the IO devices. They can be integrated with the devices or housed separately and often talk to several similar devices (typical examples would be RAID controllers or a control unit which connects to 1000 3270 terminals).

 
    +---------------------------------------------------------------+ 
    | +-----+ +-----+ +-----+ +-----+ +----------+ +----------+     |
    | | CPU | | CPU | | CPU | | CPU | | Main     | | Expanded |     |
    | |     | |     | |     | |     | | Memory   | | Storage  |     |
    | +-----+ +-----+ +-----+ +-----+ +----------+ +----------+     |
    |---------------------------------------------------------------+
    |        IOP       |      IOP        |      IOP                 |
    |---------------------------------------------------------------| 
    | C | C | C | C | C | C | C | C | C | C | C | C | C | C | C | C | 
    +---------------------------------------------------------------+ 
         ||                                              || 
         ||  Bus & Tag Channel Path                      || ESCON 
         ||  ======================                      || Channel 
         ||  ||                  ||                      || Path 
    +----------+               +----------+         +----------+ 
    |          |               |          |         |          | 
    |    CU    |               |    CU    |         |    CU    | 
    |          |               |          |         |          | 
    +----------+               +----------+         +----------+ 
        |     |                     |                |       | 
+----------+ +----------+      +----------+   +----------+ +----------+ 
|I/O Device| |I/O Device|      |I/O Device|   |I/O Device| |I/O Device| 
+----------+ +----------+      +----------+   +----------+ +----------+ 

CPU = Central Processing Unit 
C = Channel 
IOP = IP Processor 
CU = Control Unit

The 390 IO systems come in 2 flavors the current 390 machines support both the older 360 and 370 interface, sometimes called the parallel I/O interface, sometimes called Bus-and Tag and sometimes Original Equipment Manufacturers Interface (OEMI).

This byte wide parallel channel path/bus has parity and data on the "Bus" cable and control lines on the "Tag" cable. These can operate in byte multiplex mode for sharing between several slow devices or burst mode and monopolize the channel for the whole burst. Upto 256 devices can be addressed on one of these ables. These cables are about one inch in diameter. The maximum unextended length supported by these cables is 125 Meters but this can be extended up to 2km with a fiber optic channel extended such as a 3044. The maximum burst speed supported is 4.5 megabytes per second however some really old processors support only transfer rates of 3.0, 2.0 and 1.0 MB/sec. One of these paths can be daisy chained to up to 8 control units.

IBM introduced ESCON, which is fiber optic based, in 1990. It uses 2 fiber optic cables and uses either LEDs or lasers for communication at a signaling rate of up to 200 megabits/sec. As 10 bits are transferred for every 8 bits of information this drops to 160 megabits/sec and to 18.6 Megabytes/sec once control information and CRC are added. ESCON only operates in burst mode.

ESCONs typical maximum cable length is 3km for the LED version and 20km for the laser version known as XDF (extended distance facility). This can be further extended by using an ESCON director which triples the above mentioned ranges. Unlike Bus and Tag as ESCON is serial. It uses a packet switching architecture. The standard Bus and Tag control protocol is however present within the packets. Up to 256 devices can be attached to each control unit that uses one of these interfaces.

A new fiber architecture has been released by IBM called FICON which improves on the performance of ESCON.

Common 390 Devices

• Network adapters typically OSA2, 3172's, 2116's and OSA-E gigabit Ethernet adapters
• Consoles 3270 and 3215 (a TTY emulated under Linux for a line mode console)
• DASD's direct access storage devices (otherwise known as hard disks)
• Tape Drives.
• CTC (Channel to Channel Adapters), ESCON or Parallel Cables used as a very high speed serial link between 2 machines. We use 2 cables under Linux to do a bi-directional serial link.

Debugging IO on S390 under VM

Now we are ready to go on with IO tracing commands under VM. First, a few self explanatory queries:

Q OSA 
Q CTC 
Q DISK 
Q DASD

Q OSA on my machine returns

OSA 7C08 ON OSA 7C08 SUBCHANNEL = 0000 
OSA 7C09 ON OSA 7C09 SUBCHANNEL = 0001
OSA 7C14 ON OSA 7C14 SUBCHANNEL = 0002 
OSA 7C15 ON OSA 7C15 SUBCHANNEL = 0003 

Now using the device numbers returned by this command we will trace the Io starting up on the first devices 7c08 and 7c09. In our simplest case we can trace the start subchannels:

TR SSCH 7C08-7C09 

Or the halt subchannels

 TR HSCH 7C08-7C09 

You can also trace MSCH's ,STSCH's, but I think you can guess the rest.

Ingo's favourite trick is tracing all the IO's and CCWS and spooling them into the reader of another VM guest so he can ftp the logfile back to his own machine. I'll do a small bit of this and give you a look at the output.

1. Spool stdout to VM guest Linux4's reader

SP PRT TO * 

2. Fill Linux4's reader with the trace

TR IO 7c08-7c09 INST INT CCW PRT RUN

3. Start up Linux
4. Finish the trace

TR END 

5. Close the reader

C PRT 

6. List reader contents

RDRLIST 

7. Copy it to Linux4's minidisk

RECEIVE / LOG TXT A1 ( replace 

8. FILELIST and press F11 to look at it. You should see something like:

00020942' SSCH B2334000 0048813C CC 0 SCH 0000 DEV 7C08 <
CPA 000FFDF0 PARM 00E2C9C4 KEY 0 FPI C0 LPM 80 
CCW 000FFDF0 E4200100 00487FE8 0000 E4240100 ........ 
IDAL 43D8AFE8 
IDAL 0FB76000
00020B0A' I/O DEV 7C08 - 000197BC' SCH 0000 PARM 00E2C9C4 
00021628' TSCH B2354000 00488164 CC 0 SCH 0000 DEV 7C08 
CCWA 000FFDF8 DEV STS 0C SCH STS 00 CNT 00EC 
KEY 0 FPI C0 CC 0 CTLS 4007
00022238' STSCH B2344000 00488108 CC 0 SCH 0000 DEV 7C08 

gdb on S390

Note, compiling for debugging with gdb works better without optimization (see Compiling programs for debugging).

Invocation

gdb <victim program <optional corefile>

Online help

help: gives help on commands. For example:

help 

help display 

Note gdb's online help is very good and we advise you to use it.

Assembly

• info registers: displays registers other than floating point.
• info all-registers: displays floating points as well.
• disassemble: disassembles. For example,

Viewing and modifying variables

• print or p: displays variable or register. For example:

• display: prints variable or register each time program stops. For example:

• undisplay: undo's display's

• info breakpoints: shows all current breakpoints

• info stack: shows stack back trace ( if this doesn't work too well, I'll show you the stacktrace by hand below ).

• info locals: displays local variables.

• info args: display current procedure arguments.

• set args: will set argc and argv each time the victim program is invoked.

set <variable=value>

set argc=100 

set $pc=0 

Modifying execution

• step: steps n lines of sourcecode step with no value steps 1 line.
• next: like step except this will not step into subroutines
• stepi: steps a single machine code instruction.
• nexti: steps a single machine code instruction but will not step into subroutines.
• finish: will run until exit of the current routine
• run: (re)starts a program
• cont: continues a program
• quit: exits gdb.

breakpoints

• break: sets a breakpoint. For example:

break main 
break *$pc 
break *0x400618 

• rbr: Set a breakpoint for all functions matching REGEXP. This is really useful for large programs. For example:

rbr 390 

• info breakpoints: lists all breakpoints

• delete: delete breakpoint by number or delete them all. For example:

• watch: This will set a watchpoint (usually hardware assisted). This will watch a variable till it changes. For example:

• info watchpoints: Display currently active watchpoints

• condition: Specify breakpoint number N to break only if COND is true. Usage is condition N COND, where N is an integer and COND is an expression to be evaluated whenever breakpoint N is reached. This is a particularly another useful command.

User defined functions/macros

• define: Define a macro/function. This is very very useful, simple and powerful. Usage: define <name> <list of commands> end. Examples which you should consider putting into .gdbinit in your home directory:

  define d 
  stepi 
  disassemble $pc $pc+10 
  end 
  define e 
  nexti 
  disassemble $pc $pc+10 
  end   

  Other hard to classify stuff

• signal n: sends the victim program a signal. For example, signal 3 will send a SIGQUIT.

• info signals: what gdb does when the victim receives certain signals.
• list: Examples:

list 		lists current function source 

list 1,10 	list first 10 lines of current file. 

list test.c:1,10   

• directory: Adds directories to be searched for source if gdb cannot find the source. Note it is a bit sensitive about slashes. For example, to add the root of the filesystem to the searchpath do:

directory //   

• call <function>: This calls a function in the victim program, this is pretty powerful. For example:

(gdb) call printf("hello world") 

$1 = 11 

Hints

1. Command completion works just like bash (if you are a bad typist like me this really helps). For example, type br <TAB>
2. If you have a debugging problem that takes a few steps to recreate put the steps into a file called .gdbinit in your current working directory if you have defined a few extra useful user defined commands put these in your home directory and they will be read each time gdb is launched.

break main 
run 
break runtime_exception 
cont   

Stack chaining in gdb by hand

• This is done using a the same trick described for VM:

$5 = 0x528f18 

• Now you can use:

info symbol (*($sp+56))&0x7fffffff 

• Now do:

• Now do:

• Now do:

p/x (*(**$sp+56))&0x7fffffff 

For more information

From your Linux box do:

man gdb or info gdb

Examining Core Dumps

A core dump is a file generated by the kernel ( if allowed ) which contains the registers, and all active pages of the program which has crashed. From this file gdb will allow you to look at the registers and stack trace and memory of the program as if it just crashed on your system. It is usually called core and created in the current working directory. This is very useful in that a customer can mail a core dump to a technical support department and the technical support department can reconstruct what happened. Provided the have an identical copy of this program with debugging symbols compiled in and the source base of this build is available.

In short it is far more useful than something like a crash log could ever hope to be.

In theory all that is missing to restart a core dumped program is a kernel patch which will do the following.

1. Make a new kernel task structure
2. Reload all the dumped pages back into the kernels memory management structures.
3. Do the required clock fixups
4. Get all files and network connections for the process back into an identical state (really difficult).
5. A few more difficult things I haven't thought of.

Why have I never seen one?

Probably because you haven't used the command:

to allow core dumps, now do:

ulimit -a 

to verify that the limit was accepted.

A sample core dump

• To create this I'm going to do:

ulimit -c unlimited 
gdb 

• Now be bad and do the following from another telnet/xterm session to the same machine:

ps -aux | grep gdb 
kill -SIGSEGV <gdb's pid >

killall -SIGSEGV gdb

• Now look at the core dump.

GNU gdb 4.18 
Copyright 1998 Free Software Foundation, Inc. 
GDB is free software, covered by the GNU General Public License, and you are 
welcome to change it and/or distribute copies of it under certain conditions. 
Type "show copying" to see the conditions. 
There is absolutely no warranty for GDB. Type "show warranty" for details. 
This GDB was configured as "s390-ibm-linux"... 
Core was generated by `./gdb'. 
Program terminated with signal 11, Segmentation fault. 
Reading symbols from /usr/lib/libncurses.so.4...done.
Reading symbols from /lib/libm.so.6...done. 
Reading symbols from /lib/libc.so.6...done.
Reading symbols from /lib/ld-linux.so.2...done.
#0 0x40126d1a in read () from /lib/libc.so.6
Setting up the environment for debugging gdb.
Breakpoint 1 at 0x4dc6f8: file utils.c, line 471.
Breakpoint 2 at 0x4d87a4: file top.c, line 2609.
(top-gdb) info stack 
#0 0x40126d1a in read () from /lib/libc.so.6 
#1 0x528f26 in rl_getc (stream=0x7ffffde8) at input.c:402 
#2 0x528ed0 in rl_read_key () at input.c:381 
#3 0x5167e6 in readline_internal_char () at readline.c:454 
#4 0x5168ee in readline_internal_charloop () at readline.c:507 
#5 0x51692c in readline_internal () at readline.c:521 
#6 0x5164fe in readline (prompt=0x7ffff810 "\177¢·¢xx\177¢·¢w¢X\177¢·¢xx¢@") 
 at readline.c:349 
#7 0x4d7a8a in command_line_input (prompt=0x564420 "(gdb) ", repeat=1,
 annotation_suffix=0x4d6b44 "prompt") at top.c:2091 
#8 0x4d6cf0 in command_loop () at top.c:1345
#9 0x4e25bc in main (argc=1, argv=0x7ffffdf4) at main.c:635   

LDD

This is a program which lists the shared libraries which a library needs. For example:

ldd ./gdb 

Outputs:

libncurses.so.4 = /usr/lib/libncurses.so.4 (0x40018000) 
libm.so.6 = /lib/libm.so.6 (0x4005e000) 
libc.so.6 = /lib/libc.so.6 (0x40084000)
/lib/ld-linux.so.2 = /lib/ld-linux.so.2 (0x40000000) 

Debugging modules

As modules are dynamically loaded into the kernel their address can be anywhere to get around this use the -m option with insmod to emit a load map which can be piped into a file if required.

The proc file system

This is a filesystem created by the kernel with files which are created on demand by the kernel if read, or can be used to modify kernel parameters. It is a powerful concept. For example:

cat /proc/sys/net/ipv4/ip_forward 

0 

echo 1 /proc/sys/net/ipv4/ip_forward 

cat /proc/sys/net/ipv4/ip_forward 
1

There is a lot of useful info in here best found by going in and having a look around, so I'll take you through some entries I consider important.

• All the processes running on the machine have there own entry defined by /proc/<pid>.
• Let us have a look at the init process:

cd /proc/1 
cat cmdline 
init [2]

• Now look at the file descriptor entries:

cd /proc/1/fd 

• The storage map for the process may be examined:

cat /proc/29/maps 
00400000-00478000 r-xp 00000000 5f:00 4103 /bin/bash 
00478000-0047e000 rw-p 00077000 5f:00 4103 /bin/bash
0047e000-00492000 rwxp 00000000 00:00 0 
40000000-40015000 r-xp 00000000 5f:00 14382 /lib/ld-2.1.2.so 
40015000-40016000 rw-p 00014000 5f:00 14382 /lib/ld-2.1.2.so
40016000-40017000 rwxp 00000000 00:00 0 
40017000-40018000 rw-p 00000000 00:00 0
40018000-4001b000 r-xp 00000000 5f:00 14435 /lib/libtermcap.so.2.0.8 
4001b000-4001c000 rw-p 00002000 5f:00 14435 /lib/libtermcap.so.2.0.8
4001c000-4010d000 r-xp 00000000 5f:00 14387 /lib/libc-2.1.2.so
4010d000-40111000 rw-p 000f0000 5f:00 14387 /lib/libc-2.1.2.so
40111000-40114000 rw-p 00000000 00:00 0
40114000-4011e000 r-xp 00000000 5f:00 14408 /lib/libnss_files-2.1.2.so 
4011e000-4011f000 rw-p 00009000 5f:00 14408 /lib/libnss_files-2.1.2.so 
7fffd000-80000000 rwxp ffffe000 00:00 0   

• /proc/1/cwd is a softlink to the current working directory.
• /proc/1/root is the root of the filesystem for this process.
• /proc/1/mem is the current running process' memory, which you can read and write to like a file. strace uses this sometimes as it is a bit faster than the rather inefficient ptrace interface for peeking at DATA.

• Use cat /proc/1/status to display the status of the process:

Name: init 
State: S (sleeping) 
Pid: 1 
PPid: 0 
Uid: 0 0 0 0 
Gid: 0 0 0 0 
Groups: 
VmSize: 408 kB 
VmLck: 0 kB
VmRSS: 208 kB 
VmData: 24 kB 
VmStk: 8 kB 
VmExe: 368 kB 
VmLib: 0 kB 
SigPnd: 0000000000000000 
SigBlk: 0000000000000000
SigIgn: 7fffffffd7f0d8fc
SigCgt: 00000000280b2603
CapInh: 00000000fffffeff
CapPrm: 00000000ffffffff
CapEff: 00000000fffffeff 
User PSW: 070de000 80414146
task: 004b6000 tss: 004b62d8 ksp: 004b7ca8 pt_regs: 004b7f68
User GPRS: 
00000400 00000000 0000000b 7ffffa90 
00000000 00000000 00000000 0045d9f4
0045cafc 7ffffa90 7fffff18 0045cb08 
00010400 804039e8 80403af8 7ffff8b0
User ACRS: 
00000000 00000000 00000000 00000000 
00000001 00000000 00000000 00000000
00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000
Kernel BackChain CallChain BackChain CallChain 
004b7ca8 8002bd0c 004b7d18 8002b92c 
004b7db8 8005cd50 004b7e38 8005d12a
004b7f08 80019114 

Starting points for debugging scripting languages etc.

bash/sh

Use the -x option to trace the script: bash -x <scriptname>. For example:

bash -x /usr/bin/bashbug 
+ MACHINE=i586 
+ OS=linux-gnu 
+ CC=gcc 
+ CFLAGS= -DPROGRAM='bash' -DHOSTTYPE='i586' -DOSTYPE='linux-gnu' -DMACHTYPE='i586-pc-linux-gnu' -DSHELL -DHAVE_CONFIG_H -I. -I. -I./lib -O2 -pipe 
+ RELEASE=2.01 
+ PATCHLEVEL=1
+ RELSTATUS=release 
+ MACHTYPE=i586-pc-linux-gnu 

perl

Use the -d option of perl to invoke the interactive debugger: perl -d <scriptname>

Java

Use: jdb <filename> to invoke another fully interactive gdb style debugger. Type "?" for help when the debugger is invoked.

References

• Enterprise Systems Architecture Reference Summary
• Enterprise Systems Architecture Principles of Operation
• Hartmut Penners 390 stack frame sheet.
• IBM Mainframe Channel Attachment a technology brief from a CISCO webpage
• Various bits of man and info pages of Linux.
• Linux and GDB source.
• Various info and man pages.
• CMS Help on tracing commands.
   

Without a telnet connection into your Linux system you cannot do much. Unfortunately getting an IP address and routing in place is not always easy to arrange on short notice.

The following describes a very minimal network when running under VM. We created a private subnet for the Linux virtual machine(s) and use CMS Pipelines TCP/IP support to connect that subnet to your primary TCP/IP stack.

This approach does not require any changes to your production TCP/IP.

In the examples setup I'll use the following userids:

• LINUXTCP

• TCPIP

• WARPHOLE

• LINUXVM

The updates to configuration files are described within this section.

When I defined this I didn't have the IUCV options in the directory so I did a CTC link. You should be able to do an IUCV link as well.

The 192.168.16 subnet is one of the private subnets that you can use internally but that can not go out to the Internet.

The following will let you tn3270 from your Linux virtual machine to your VM system, if you need that.

When you add move Linux guests you need to think about routing.

This one works for me.

Note that the CTC's are cross-connected here. Your LINUXVM userid should do something similar so you get yout connection also when LINUXVM starts up after LINUXTCP.

Now put a TCPIP DATA on your A-disk with tcpipuserid linuxtcp

First try to ping 192.168.16.254 (the stack itself). When that works you should be able to ping 192.168.16.2 (to verify the connection to your Linux virtual machine).

The next test could be to telnet or ftp to 192.168.16.2 (still using the modified TCPIP DATA file.

The WARPHOLE userid runs the WARPHOLE EXEC. The program is in fact a stripped down version of TCPSNIFF from John Hartmann.

The core pipeline has a number of segments like this:

You need one of those for each port on the production network that you want to connect to your Linux virtual machine. If you reverse them you can make a port on the LINUXTCP stack that goes out.

Rename your modified TCPIP DATA file and check for listening ports:

netstat ( select user warphole

This should give you something like this

From VM you can now telnet 127.0.0.1 10023 to logon to Linux, or from your workstation to your production stack port 10023.

This section describes the packages that we've ported to Linux/390 and any special steps that needed taking.