[Contents] [Chapter 2] [Chapter 4] [Index]

3 Running Compaq MPI Programs

This chapter describes how to run a Compaq MPI program. The chapter consists of the following sections:

3.1 Executing Compaq MPI Programs

To execute a Compaq MPI program, enter the command dmpirun and the name of the program you wish to execute:

: dmpirun myprog

3.1.1 Executing with Two Processes

You can execute a Compaq MPI program with a small set of options. You can refer to /usr/examples/mpi for an example program.

For example, to run myprog using two processes on one host, enter the following:

: dmpirun -np 2 myprog

To run the same program across two hosts, for example, black and white, follow these steps:

Create a file, for example, myhosts, with the host names separated by white space (including newline), for example:
black white
Add the hostfile option, -hf, to the command line:
dmpirun -np 2 -hf myhosts myprog

3.1.2 Executing with Options

Options can be specified via command line options and arguments, or via environment variables. Command line options override the setting of environment variables. This section describes the available options and arguments. Compaq MPI Environment Variables describes the Compaq MPI environment variables.

Note that the dmpirun command and its helper programs use the UNIX® commands wdir, hostname, and rsh. It is assumed the commands are located in your UNIX path.

Option	Argument	Description

-wdir	directory	Specifies the working directory for the application.
-np	number-of-processes (or tasks)	Default of one per host.
-hf	host-file-name	This option contains the names of the hosts on which to run the Compaq MPI program. It has the format of a list of hostnames separated by whitespace (including a newline). The host on which dmpirun is started must be in the host file. If a `Permission denied' message appears when you execute dmpirun, ensure you have permission to perform an `rsh' on each host in the host file. Check that the ~/.rhosts file is set up correctly. The last line of the hostfile must be terminated with a newline. Hostnames may now take the form hostname:directory. If :directory is supplied, this will be used in place of the DMPI_DIRECTORY value or the -wdir value. This allows processes on each host to run with a different working directory.
-dbx		Allows you to run the application processes inside the dbx debugger.
-ladebug		Allows you to run the application processes inside the ladebug debugger.
-xdbx		Allows you to run the application processes inside the xdbx debugger.
-gdb		Allows you to run the application processes inside the gdb debugger.
-xterm		Allows you to run each of the application processes inside its own xterm.
-cmddbx	dbx-command	The supplied program will be used in place of the default dbx command when the -dbx option is used. The full pathname must be supplied.
-cmdladebug	ladebug-command	The supplied program will be used in place of the default ladebug command where the -ladebug option is used. The full pathname must be supplied.
-cmdxdbx	xdbx-command	The supplied program will be used in place of the default xdbx command when the -xdbx option is used. The full pathname must be supplied.
-cmdgdb	gdb-command	The supplied program will be used in place of the default gdb command when the -gdb option is used. The full pathname must be supplied.
-cmdxterm	xterm-command	The supplied program will be used in place of the default xterm command when the -xterm option is used. The full pathname must be supplied.
-cmdrsh	rsh-command	The dmpirun program invokes the /usr/bin/rsh to create remote processes. Use this option to select a different command to use. The full pathname must be supplied.
-cmddmpirun	dmpi-command	dmpirun invokes a copy of itself on each host. The default is /usr/bin/dmpirun. Use this option to select a different location for the command.
-display	x-display	The X display on which to use xterm or debugger.
-ump_key	ump-key	The ump_key option is used to generate unique identifiers for Memory Channel regions used by applications. If you want to run more than one application, each application must have a unique key. When you execute dmpirun, a pseudo-random key is chosen by default unless the environment variable UMP_KEY is set. If you specify keys using the ump_key option, they must be at least 4000 keys apart.
-ump_bufs	ump_buffer_size	Determines the size of a channel buffer. Default of 32768 or the environment variable UMP_BUFS.
-ump_multirail	string	This option controls the assignment of UMP "channels" to the underlying Memory Channel physical rail structure. The default rail assignment method evenly distributes UMP "channels" over the available hardware rails. The argument string takes the form: [report\|avoid:N] where report enables the printing of a channel distribution/usage report and avoid:N will cause no channel to be assigned to rail N. Examples: > setenv UMP_MULTIRAIL "report" > dmpirun ... will use the default "pressure" assignment and generate a final usage report. > setenv UMP_MULTIRAIL "avoid:1,report" > dmpirun ... will NOT assign rail 1 and will also generate a final report.
-ump_thread_mode	mode	This option controls the use of additional data-moving threads within UMP. By default, an extra thread is not used. This can be modified by setting the option to one of the following: "merging" selects an optimized multithread implementation. This may be used to some advantage when there are more CPUs available than MPI processes. "single" (*) is the default value.
-ump_yield	method	This option is effective when using multiple threads to control the method by which executing threads yield to a competing thread. "kernel" informs the kernel level scheduler when a thread is willing to yield. "user" informs the thread level scheduler when a thread is willing to yield. "swap" will alternate between each of the above. "none" will effectively disable a thread from ever "yielding".
-ump_timeout	timeout	The value of the integer argument is used to determine the timeout value (in seconds) when draining UMP channels during an ump_close(). The default value is 300.
-ump_fragsize	fragsize	The integer value of this option determines the fragmentation of UMP data transfers. By default it has a value of 4096 (bytes). Its value also cannot exceed that of DMPI_UMPBUFS. Data transfers that are of UMP_FRAGSIZE or less are not fragmented. All other data transfers will be fragmented into chunks of UMP_FRAGSIZE or less. Example: A message transfer of 16k bytes will be fragmented into 4 chunks of 4k bytes.
-ump_error_mode	mode	This option specifies the level of error checking to be performed for Memory Channel transactions. Note that the higher the level of error checking set, the greater the performance degradation. "none" performs no error checking. "one" checks for errors once at start up and shutdown. "inter" checks for errors intermittently. This is the default setting. "always" checks for errors on every transaction. "recover" will resend data in the event of an error.
-mpid_shrt	mpid_shrt limit	Compaq MPI messages are normally sent internally as a header packet followed by a data packet. However, for messages smaller than the mpid_shrt size, the data is sent as part of the header packet in two UMP¹ operations; this improves performance. The default size is the value specified by the MPID_SHRT environment variable, or 256 bytes if the environment variable is not specified. The short message limit must be a multiple of four, and must not be larger than the UMP buffer size. Some applications perform a large number of short non-blocking sends; the short protocol may cause such sends to block. If this happens, set the value of mpid_shrt to zero.
-mpid_tiny	mpid_tiny limit	Compaq MPI messages are normally sent internally as a header packet followed by a data packet. However, for messages smaller than the mpid_tiny size, the data is sent as part of the header packet using a single UMP operation; this improves performance. The default size is the value specified by the MPID_TINY environment variable, or 64 bytes if the environment variable is not specified. The value specified must be a multiple of 4, less than or equal to 64, and greater than zero. In some circumstances the tiny protocol can lead to the UMP channel becoming clogged up, which causes non-blocking sends to block. If this happens, set the value of mpid_tiny to zero.
-block		Along with -cyclic and -random, -block is used to select the process layout when a -hf option is used. -block is the default. The processes are spread across the hosts with a block distribution. For example, with two hosts and four processes, Ranks 0 and 1 will be on the first host, with Ranks 2 and 3 on the second. The processes are spread evenly across the host, but no account is taken of the number of processors on each host.
-cyclic		Along with -block and -random, -cyclic is used to select the process layout when a -hf option is used. In a -cyclic distribution, the processes are located in round-robin fashion. For example, with two hosts and four processes, Ranks 0 and 2 will be on the first host. Ranks 1 and 3 on the second. The processes are spread evenly across the host, but no account is taken of the number of processors on each host.
-random		Along with -block and -cyclic, -random is used to select the process layout when a -hf option is used. In a -random distribution, processes are placed at random on hosts. The processes are spread evenly across the host, but no account is taken of the number of processors on each host.
-pf	file	In place of a command line application specification, a file containing the required application processes can be provided. Process File Format describes the format of this file. The -pf option should not be used with the -np or -hf options, or a command line executable.

^1. UMP is the communication layer used by Compaq MPI. This release includes support for multiple rails.

3.2 Compaq MPI Environment Variables

All the command line variables for dmpirun have environment variable counterparts. These may be set to avoid the need to use the command line option if something other than the default is required, but will not change frequently. For example, when using ssh instead of rsh, rather than use the -cmdrsh option every time, the DMPI_CMDRSH environment variable can be set.

The table below lists all the environment variables used by Compaq MPI. The first column is the variable name, the second is the default value if the variable is unset, and the third column is the command line flag which takes priority.

Name	Default	Override Flag
DMPI_DIRECTORY	current working directory	-wdir
DMPI_HOSTFILE	none	-hf or -machinefile
DMPI_PROCFILE	none	-pf
DMPI_NP	1	-np
DMPI_UMPKEY	random	-ump_key
DMPI_UMPBUFS	32768	-ump_bufs
DMPI_MPIDSHRT	256	-mpid_shrt
DMPI_MPIDTINY	64	-mpidtiny
DMPI_DEBUG	.	-dbx, -ladebug, -xdbx, -xterm, -gdb
DMPI_LAYOUT	block	-block, -cyclic, -random
DMPI_LAYOUT_SEED	dmpirun process id	none
DMPI_SERVERPORT	0	none
DMPI_CMDXTERM	/usr/bin/X11/xterm	-cmdxterm
DMPI_CMDRSH	/usr/bin/rsh	-cmdrsh
DMPI_CMDDMPIRUN	/usr/bin/dmpirun	-cmddmpirun
DMPI_CMDDBX	/usr/bin/dbx	-cmddbx
DMPI_CMDXDBX	/usr/local/bin/X11/xdbx	-cmdxdbx
DMPI_CMDLADEBUG	/usr/bin/ladebug	-cmdladebug
DMPI_CMDGDB	/usr/local/bin/gdb	-cmdgdb
UMP_MULTIRAIL	pressure	-ump_multirail
UMP_THREAD_MODE	single	-ump_thread_mode
UMP_YIELD	swap	-ump_yield
UMP_TIMEOUT	300	-ump_timeout
UMP_FRAGSIZE	4096	-ump_fragsize
UMP_ERROR_MODE	inter	-ump_error_mode

Notes:

Variable	Comment
DMPI_DEBUG	The special value . means no debugger. Valid values for this option are thus ., dbx, xdbx, ladebug, xterm, and gdb. xterm is not strictly a debugger, but shares the same mechanism.
DMPI_LAYOUT	May take the values block, cyclic, or random.
DMPI_LAYOUT_SEED	Is used to seed the random number generator used by the random layout option. By default, the dmpirun process id is used. To allow repeatability, a fixed seed may be supplied. There is no command line override for this option.
DMPI_SERVERPORT	dmpirun creates a socket which is then used by the host managers to communicate information about the application processes created. This environment variable allows the port number to be set. By default, the value 0 is used, This has special meaning: it requests a currently unused port from the operating system. This avoids collisions with any other well-written software which uses sockets. It has no command line override.

3.3 Process File Format

The new -pf option allows more complex application layouts to be specified. In particular it allows multiple executables to be used within one application. This section describes the file which must be used with the -pf option.

The file consists of one or more lines of the following format:

: number hostname[:path] executable [arguments]

[...] indicates optional features.

Each component has the following meaning:.

Name	Explanation
number	Takes one of two forms. Either a single number which indicates the number of processes required, or a triple of the form lowest;highest;step. In the first form, the next available ranks will be used after all ranks used by the second form have been allocated. In the second, the ranks are specified explicitly as lowest, lowest+step, lowest+2*step,...<= highest.
hostname	The host on which to create these processes.
path	Run these processes with path set as the working directory in preference to the -wdir or DMPI_DIRECTORY options.
executable	The executable to be used for these processes.
arguments	The arguments to be used for these processes.

All explicit ranks will be assigned first, then any gaps will be filled with the remaining processes in the order they are declared in the process file. Overlapping ranks, or missing ranks will result in an error being reported.

The following examples illustrate common cases:

Example 1: An uneven block distribution

: 3 black a.out -x 1
: 4 white a.out -x 1
: 2 brown a.out -x 1
: 3 green a.out -x 1

The result will be ranks 0-2 on black, 3-6 on white, 7-8 on brown and 9-11 on green.

Example 2: A non-SPMD application

: 1 black master
: 7 black slave

Rank 0 will be an instance of master, ranks 1-7 will be instances of slave.

Example 3: A cyclic distribution

: 0;15;4 black a.out -y 0
: 1;15;4 white a.out -y 0
: 2;15;4 brown a.out -y 0
: 3;15;4 green a.out -y 0

Ranks 0,4,8,14 will be on black; Ranks 1,5,8,13 on white; 2,6,10,14 on brown and 3,7,11,15 on green.

Example 4: Setting some ranks explicitly

: 0;0;1 black source
: 7;7;1 black sink
: 6 black worker

Rank 0 will be an instance of source, rank 7 will be an instance of sink. The remainder, 1-6, will be instances of worker.

3.4 TotalView Support

This version of Compaq MPI is compatible with the Dolphin Interconnect Solutions' TotalView™ debugger, Version 3.7.7 and later. It is used with TotalView in the following way:

: totalview dmpirun -a dmpirun_arguments

where dmpirun_arguments are the command line arguments you wish to give to dmpirun. For example,

: totalview dmpirun -a -np 2 a.out 0 1

The use of TotalView in combination with any other debug option, including xterm, is not permitted. See the TotalView documentation for usage details.

[Contents] [Chapter 2] [Chapter 4] [Index]

mpi@ilo.dec.com