[Contents] [Preface] [Chapter 2] [Index]


1 Getting Started with Compaq MPI

This chapter provides a description of Compaq MPI, and the information required to install it. The chapter consists of the following sections:

1.1 Introducing Compaq MPI

Compaq MPI is a proprietary implementation of MPI software for Alpha systems running the Compaq Tru64™ UNIX (formerly DIGITAL® UNIX) operating system, and is supported on both standalone SMP systems and Memory Channel clusters. Compaq MPI is derived from MPICH Version 1.1.1 from Argonne National Laboratories. Compaq MPI is optimized to give you low latency, high bandwidth message passing in SMP and SMP cluster environments.

The goal of MPI software is to provide a practical, portable, efficient, and flexible standard for message passing programs.

1.2 Installing Compaq MPI

This section describes the requirements to check before installing Compaq MPI; how to install the Compaq MPI kit; and, after installation, how to verify the installation, initialize machines in a cluster, and set shared memory and Memory Channel parameters.

1.2.1 Checking Hardware Requirements

You need the following hardware to run Compaq MPI:

1.2.2 Checking Software Requirements

You need the following software to run Compaq MPI:

Note: If you are using multiple Memory Channel rails, you must use Tru64 UNIX Version 4.0D and TruCluster Version 1.5.

1.2.3 Installing the Kit

To install Compaq MPI from a CD-ROM, follow these steps.

  1. Read any release notes that come with the kit. (See the PSE Installation Guide for more information on installing Compaq MPI.)
  2. Log in to the machine running the Tru64 UNIX operating system, as root.
  3. To install the kit, enter the following commands, substituting your CD-ROM drive name for rzxxc:
cd /
mount -rd /dev/rzxxc /mnt
setld -l /mnt/pse1xx/kit
  1. The Compaq MPI installation comprises one subset, MPIBIN1XX, which consists of the libraries, include files, man pages, scripts, and examples.
  2. Install the kit on all the systems in the cluster if you are using Memory Channel.
  3. To verify Compaq MPI is installed correctly, enter the following command:
setld -v MPIBIN1XX
  1. If you intend using Upshot, as root, modify the first line of the Upshot file to refer to the correct path name for the wish interpreter on your system.

1.2.4 Initializing Machines for Memory Channel

If you are using Memory Channel, you must check that each machine in the cluster is initialized for user-level access.

To check, and initialize if necessary, follow these steps:

  1. To search for the process imc_mapper on each machine, enter this command:

    ps ax | grep imc_mapper | grep -v grep

    for example, the return message below might appear:
    657 ttyp2 U 0:00.01 /usr/sbin/imc_mapper

  2. If the process described in step 1 does not exist on a machine that you intend using as part of the cluster, enter the following command as root on the machine:

    /usr/sbin/imc_init

    You enter this once and it is in effect until the machine is shut down. (There is no problem in executing this if the command was entered previously. See the imc_init man pages for details.)

  3. Ensure that the /etc/sysconfigtab file contains the following entry:

    rm:
    rm-no-inheritance = 1

    This prevents Memory Channel resources from being inherited by a forked process.

  4. If you are using multiple Memory Channel rails, the default setting is to use them as a failover pair. While this provides maximum availability, you can achieve increased performance by configuring your cluster to use the single-rail style. This allows applications to access the aggregate address space of all logical rails and utilize aggregate bandwidth.

    To configure the use of the single-rail style, change the /etc/sysconfigtab file by adding or modifying the following entry:

    rm:
    rm_rail_style = 0

    1.2.5 Setting Shared Memory and Memory Channel Parameters

    Compaq MPI uses shared memory for communication within a host, and Memory Channel for communication between hosts. This is achieved by setting up "communication channels", each of which is either a shared memory segment or a group of Memory Channel pages.

    The size of each communication channel is set by the user using the -ump_bufs option. (See Executing with Options.) The default value is 32K.

    If very large applications are being run, it may be appropriate to increase the data and stack size using the limit command; for example,

    limit datasize unlimited

    1.2.5.1 Changing Shared Memory Parameters

    1. To increase the maximum shared memory size, edit the /etc/sysconfigtab file and include or modify the following entry:

      ipc:
      shm-max=<size in bytes>

    2. There are kernel limits on the number of shared memory segments a process can attach to and the total number of system-wide shared memory segments available.

      To modify the value of the shm-seg variable, edit the /etc/sysconfigtab file and include or modify the following entry:

      ipc:
      shm-seg=<number of segments attached per process>

    3. The total number of system-wide shared memory segments allowed is controlled by the shm-mni variable.

      To change the system parameter, edit the /etc/sysconfigtab file and include or modify the following entry:

      ipc:
      shm-mni=<total number of system-wide segments>

    4. If you have changed any of the parameters as described in 1, 2, or 3 above, reboot the system.

    1.2.5.2 Changing Memory Channel Parameters

    Communication across Memory Channel uses a similar scheme to shared memory. The maximum Memory Channel allocation allowed is set by default to 10MB. There is a limit of 128MB on Memory Channel allocation using Memory Channel Version 1.5, and a limit of 512MB on Memory Channel Version 2.0.

    It is desirable before running an MPI program to increase the default values. Increasing the default values does not use up memory resources but allows MPI to access more Memory Channel space.

    To increase from the default 10MB allocation on each host, for example to 100MB, follow one of these steps:

    1. Log in as root.

      Enter the following command:

      imc_init -a 100 -r 100

      or

    2. Edit the /etc/rc.config file and modify the IMC_MAXALLOC and IMC_MAXRECV entries to read:

      IMC_MAXALLOC="100"
      IMC_MAXRECV="100"

      It is recommended that you increase these values to the maximum allowed for your configuration.

      Reboot the system.

    1.2.5.3 Changing Virtual Memory Resources

    Each shared memory or Memory Channel communication channel uses up a virtual memory object. The number of virtual memory objects allowed is a kernel-defined parameter. This parameter needs to be at least twice as large as the number of communication channels between tasks.

    To change this value, follow these steps:

    1. Edit the /etc/sysconfigtab file and include or modify this entry:

      vm:
      vm_mapentries=<number of VM objects allowed>

    2. Reboot the system.

    [Contents] [Preface] [Chapter 2] [Index]

    mpi@ilo.dec.com

    Copyright © 1999, Digital Equipment Corporation. All rights reserved. Unpublished rights reserved under the copyright laws of the United States.