Compaq KAP transforms Fortran 90 source programs so that, when compiled and linked, they execute as multithreaded processes. These threads can run simultaneously --- that is, in parallel --- on symmetric multiprocessor (SMP) systems. The result is a program whose start-to-finish time is less than a Fortran 90 program that does not execute as a multithreaded process. More specifically, at run time the instructions from DO loops in a transformed Fortran 90 program execute in parallel mode. Parallelization is the process that transforms DO loops into instructions in an .EXE file that execute as multithreaded processes.

DO loops occur both explicitly and implicitly. The familiar sequence

DO 10 I = . . . 10 CONTINUE

forms an explicit DO loop that Compaq KAP views as a candidate for parallelization. A statement such as A = B * C, where A and B and C are arrays according to Fortran 90 syntax, is also viewed by Compaq KAP as a candidate for parallelization. Compaq KAP transforms this matrix multiplication statement into explicit DO loops. In general, any statement that Compaq KAP can transform into DO loops is an implicit DO loop.

Compaq KAP considers all DO loops in a program as candidates for parallelization. Each loop is or is not parallelized according to:

• Parallel processing directives that you have inserted
• KAP determination of data dependencies among the loop's iterations
• KAP determination of the amount of runtime work (the overhead of parallelization might require too much time); the -minconcurrent switch affects this determination

This chapter describes the three basic methods of controlling parallel processing (automatic, directed, and combination). It explains, for each method, how to:

• Change a source program for parallel processing
• Give commands, in the form of command line switches and values, to Compaq KAP that transform the source programs
• Direct the compilation, linking, and execution of the parallel-mode program

Compaq KAP provides three methods for programmers to control parallel processing. Their summaries follow:

• Automatic Detection --- Use this method for programs that do not contain either OpenMP (C$OMP) or Parallel Computing Forum (C*KAP*) directives.
  Compaq KAP automatically looks at these programs' DO loops and statements with Fortran 90 arrays. If these loops are good candidates for parallelization, then Compaq KAP transforms them so that they will be executed by multiple threads. This is the recommended method for initial experiences with parallelization, because the other methods require detailed knowledge of parallel programming concepts and implementation statements. Also, Compaq KAP sets the compiler and linker switches correctly.
  Section 3.2 shows how to direct Compaq KAP to perform automatic parallelization of your program. An example of using KAP automatic detection, selection, and transformation of loops is giving the following command line for Fortran source program my_prog.f:
  kf90 -fkapargs='-concurrent' my_prog.f
  The results include a transformed source program and its processing by the compiler and linker to create executable file a.out. The transformed source file will contain OpenMP directives for the loops that Compaq KAP has automatically decided to parallelize when the -psyntax=openmp switch is set. The OpenMP directives are passed onto the Compaq Fortran compiler for processing. If the -psyntax=kap switch is set, then Compaq KAP will automatically insert into the source file calls to Compaq KAP parallel run-time library, libkmp_osfp10.a.

  Note A program must be parallelized by using either OpenMP directives or calls to Compaq KAP parallel run-time library. Mixing of these methodologies within a program is not supported.

• Directed --- Use this method for programs that contain parallel directives and for which you want only the loops surrounded by parallel directives to be parallelized. These directives explicitly control where and when Compaq KAP performs parallelization inside your program. These directives can be standard OpenMP ones; they begin with C$OMP. The other source of directives is the Compaq KAP implementation of the X3H5 standard that was produced by the Parallel Computing Forum (PCF). These directives begin with C*KAP*. While you can have both kinds of directives in a program, Compaq KAP will only recognize one kind of directive, according to the -psyntax switch, as it transforms the program.
  Section 3.3 shows how to use KAP to perform directed parallelization of your program. An example of using KAP directed detection and transformation of loops is giving the following command line for Fortran source program my_prog.f containing OpenMP directives:

  kf90 -fkapargs='-noconc' my_prog.f \ -omp -pthread -call_shared

  
  The results include a transformed source program and its processing by the compiler and linker to create executable file a.out.

• Combination --- Use this method for programs where you control, with directives, the parallelization of selected individual loops in the source program file and want KAP to perform automatic detection, transformation, and parallelization of the remaining loops. Section 3.4 shows how to direct KAP to perform combined parallelization of your program. A possible command line for Fortran source program my_prog.f is:
  kf90 -fkapargs='-concurrent' my_prog.f
  The results include a transformed source program and its processing by the compiler and linker to create executable file a.out. Because -psyntax=openmp is the default switch, KAP automatically parallelizes loops by inserting OpenMP directives. OpenMP directives inserted automatically by KAP and manually by the programmer are then processed by the compiler. The compiler switch -omp tells the compiler to recognize the OpenMP directives.
  To view the compiler and linker switches that a kf90 command sets, include the -v switch. In the case of the automatic method, a command could be the following:
  kf90 -fkapargs='-concurrent' my_prog.f -v
  When using any of these three methods you must be aware of the values of environment variables, because they affect the run-time behavior of your program. The setting of the -psyntax switch determines which environment variables you should set. If you select the -psyntax=kap switch, then the environment variables to set are the ones read by the KAP Parallel run-time library, libkmp_osfp10.a. On the other hand, if you select the -psyntax=openmp switch, then the environment variables to set are the ones read by the Compaq Fortran compiler. The -psyntax=openmp switch is the default and is recommended in almost all cases.

Environment Variables to Set When -psyntax=openmp Is Selected

setenv OMP_SCHEDULE (static,dynamic,guided,runtime) setenv OMP_DYNAMIC (true,false) default is false. setenv OMP_NESTED (true,false) default is false. setenv OMP_NUM_THREADS (number) default value is the number of processors on the current system.

For further information on environment variables read by the Compaq Fortran compiler, see your Fortran user's guide.

Environment Variables to Set When -psyntax=kap Is Selected

setenv KMP_SPINLOCKS on setenv KMP_STACKSIZE 1000000 setenv PARALLEL 2

(The value of PARALLEL must always be less than or equal to the number of processors.)

To execute the program, whose default file name is a.out, give this command:

a.out

Details of each method of parallelization, including corresponding switches for the kf90 command line, are in the method's description.

3.1.2 Parallel Processing Controls --- Summary

KAP provides the following parallel command switches, directives, and assertions for use with automatic parallel processing. Refer to the appropriate sections for explanations and code examples as follows:

• Parallel Processing Switches
  • -chunk
  • -concurrent
  • -directives
  • -minconcurrent
  • -parallelio
  • -pdefault
  • -psyntax
  • -scheduling
• Parallel Processing Directives and Parallel Processing Assertions, which are used to assist automatic parallelization:

  !*$* [no]concurrentize !*$* minconcurrent !*$* assert concurrent call !*$* assert do (concurrent) !*$* assert do (concurrent call) !*$* assert do (serial) !*$* assert do prefer (concurrent) !*$* assert do prefer (serial)

• OpenMP Directives, which describe parallel processing directives used for directed parallelization:
  • Control directives
  • Storage directives
  • Synchronization directives
  • Scheduling directives
• PCF Directives, which describe parallel processing directives used for directed parallelization:
  • Processor section directives
  • Parallel region directives
  • Parallel DO loop directives
  • Critical section directives

Two types of command lines, kf90 and kapf90, invoke Compaq KAP software:

• Invoking the kf90 command results in these events:
  1. Transformation of the source program using software known as the kapf90 preprocessor.
  2. Compilation of the transformed source program using the Fortran 90 compiler.
  3. Linking of the object program from the previous event with runtime libraries.
  
  Compaq recommends using the kf90 command to process your files.
• Invoking the kapf90 command only transforms the source program. No compilation and no linking occur.

As a programmer, you should always remember that you implement a parallel processing method (automatic, directed, or combination) by making choices from the previous command line options, directives, and assertions. Your choices affect the following actions:

• DO loop detections made during Compaq KAP preprocessing
• DO loop transformations made during Compaq KAP preprocessing
• Compiler and linker output after Compaq KAP preprocessing
• Runtime behavior of the executable file after preprocessing, compiling, and linking

For example, suppose you choose combination detection and parallelization for source program test_1.f90. This program contains some or none of the parallel processing directives, parallel processing assertions, and OpenMP directives. Consider the following command:

kf90 -fkapargs='-concurrent -minconcurrent=1000 \ -psyntax=openmp -directives=akpv' test_1.f90

This command tells Compaq KAP to:

• Automatically detect (-concurrent) and transform loops with at least 1000 units of work (-minconcurrent=1000)
• Parallelize loops by inserting parallel directives of type OpenMP (-psyntax=openmp)
• Respond to parallel processing directives and parallel processing assertions and directives such as !*$* assert concurrent call (the effect of the -directives=akp switch)
• Give the transformed source program file to the compiler and linker for the creation of executable program file a.out (the default behavior of the kf90 command).

Compaq KAP parallel processing options, such as -concurrent, are enclosed in single quotation marks and are values of the -fkapargs option. The kf90 driver responds to the options enclosed in these single quotation marks by passing them as arguments to the kapf90 preprocessor (which actually transforms the source program file).

The default values of the parallel processing options also control Compaq KAP loop detections, loop transformations, calling of the compiler and linker, and runtime scheduling. They are:

-directives=akpv -minconcurrent=1000 -parallelio -default=safe -psyntax=openmp (parallel directives of type OpenMP are generated) -scheduling=e -chunk=1

Read the explanations of each of the three methods of parallelization in light of how your choices of options, directives, and assertions affect Compaq KAP detection of loops, changes to loops, compiler and linker behavior, and runtime behavior of executable file a.out.

3.2 Automatic Parallelization Using the kf90 Driver

Parallelization --- that is, creating an executable file that executes as a multithreaded application on symmetric multiprocessor systems --- via the automatic method, is most useful for large programs under the following circumstances:

• The explicit and implicit DO loops are obviously parallelizable.
• You are unfamiliar with the code.

For programs with loops that have not already been explicitly parallelized using OpenMP directives, Compaq recommends that you use automatic parallelization.

An example of using KAP automatic detection, selection, and transformation of loops is giving the following command line for Fortran source program my_prog.f:

kf90 -fkapargs='-concurrent' my_prog.f

The results include a transformed source program (my_prog.cmp.f) containing OpenMP directives around loops that KAP has decided to parallelize. and processing of the transformed source by the compiler and linker to create the executable file a.out.

3.2.1 Changing Source Programs

Compaq KAP cannot automatically parallelize loops with data dependencies between loop iterations and loops with calls to external routines. You can help Compaq KAP automatic detection of these loops by placing parallel processing assertions and parallel processing directives (each beginning with !*$*) in the source program. These assertions and directives are:

!*$* assert concurrent call !*$* assert do (concurrent) !*$* assert do (concurrent call) !*$* assert do (serial) !*$* assert do prefer (concurrent) !*$* assert do prefer (serial) !*$* [no]concurrent !*$* minconcurrent

3.2.2 Giving Command Line Switches

Command line switches you can give to Compaq KAP that affect its transformation of DO loops under the automatic method are:

• -concurrent
  The -concurrent switch, which is often abbreviated to -conc, is required for automatic parallelization.
• -minconcurrent
  So that the overhead of parallelization does not occur on loops with relatively small execution time, give the switch and value -minconcurrent=1000. A common abbreviation is -minc.
• -parallelio
  If you know that I/O operations in DO loops will not execute, give this switch.
• -psyntax
  This switch's setting determines whether KAP transforms DO loops by inserting OpenMP directives (psyntax=openmp, the default) or by inserting calls to routines in its parallel processing runtime library, libkmp_osfp10.a (-psyntax=kap)

To construct a program for parallel execution under the automatic method, you normally need to give only the -concurrent switch to the command, kf90, that invokes the kf90 driver software as follows:

kf90 -fkapargs='-concurrent' my_prog.f

The -concurrent (often abbreviated to -conc) switch tells KAP to automatically restructure the DO loops for parallel processing. The -conc switch also sets the compiler and linker switches correctly. DO loops are parallelized by insertion of OpenMP directives because -psyntax=openmp is the default. The Fortran 90 compiler will in turn process these directives and implement the parallelization.

Finally, you may want to create a completely non-parallelized program so you can compare its execution time with the times of programs that are parallelized in various ways (such as the automatic method and the directed method). The following command does this:

kf90 -fkapargs='-noconc -directives=ak' -noomp myprog.f90

The -noconc switch prevents automatic parallelization of DO loops and the absence of "p" from the string following -directives= prevents Compaq KAP from responding to any parallel directive statements. The -noomp switch prevents the Fortran compiler from responding to any OpenMP parallel directive statements in the transformed source file it receives.

3.3 Directed Parallelization Using the kf90 Driver and OpenMP Directives

Under the directed method, Compaq KAP does not do any automatic parallel detection. As always, any OpenMP directives in the original source program are passed to the Fortran 90 compiler for processing.

Parallelization --- that is, creating an executable file that executes as a multithreaded application on symmetric multiprocessor systems --- via inserting OpenMP directives is most useful for programs under the following circumstances:

• You are experienced with parallel programming.
• You want to explicitly control which DO loops are parallelized
• Some of the loops have OpenMP directives and you do not want parallelization of the other loops (that is, the ones without these directives).

The directed method applies only to DO loops with parallel directives. Consider a program with the following loops:

C$OMP PARALLEL DO WHILE (I ... DO WHILE (J ... ... ... END DO END DO C$OMP END PARALLEL

Compaq KAP passes the OpenMP directives of the DO WHILE I loop onto the compiler for processing. Compaq KAP does not parallelize the DO WHILE J loop. So, "directed" means any loops not surrounded with parallel directive statements are not parallelized. If instead Compaq KAP were to attempt to transform both DO WHILE loops, then it would be running under the combination method.

An example of how to use KAP to process a program for which no automatic parallelization is desired is given below:

kf90 -fkapargs='-noconc' my_prog.f \ -omp -pthread -call_shared

The results include a transformed source program and its processing by the compiler and linker to create executable file a.out. Because of the -noconc switch, Compaq KAP does not automatically set compiler and linker switches related to parallel processing. Therefore the user must explicitly set the -omp and -pthread compiler and -call_shared linker switches.

3.3.1 Changing Source Programs

Insert OpenMP directives (beginning with C$OMP) only with loops that are safe to parallelize. When Compaq KAP sees a loop prefaced with OpenMP directives, it does not perform data dependence analysis on that loop and does not prevent you from using a parallel directive incorrectly. The OpenMP directives are described in the DIGITAL Fortran 90 User's Guide. For reference, the following table correlates OpenMP directives to Cray's CMIC parallel directives:

OpenMP	Cray
Specifying Regions of Parallel Execution
C$OMP PARALLEL	CMIC$ PARALLEL
C$OMP END PARALLEL	CMIC$ END PARALLEL
Specifying Parallel Loops
C$OMP DO	CMIC$ END DO
Specifying Synchronized Sections of Code
C$OMP CRITICAL	CMIC$ GUARD
C$OMP END CRITICAL	CMIC$ END GUARD
C$OMP SINGLE	(equivalent coded with guard flag)
C$OMP END SINGLE
Specifying Sections of Code for Parallel Execution
C$OMP SECTIONS	CMIC$ CASE
C$OMP SECTION (before each section)
C$OMP END SECTIONS	CMIC$ END CASE
Controlling Subroutines Called within Parallel Regions
C*$* ASSERT CONCURRENT CALL	CMIC$ CONTINUE
Specifying Commons Local to each Thread
C$OMP THREAD PRIVATE	CMIC$ TASKCOMMON

3.3.2 Giving Command Line Switches

No Compaq KAP switches affect the processing by the compiler of OpenMP directives inserted by the user.

3.3.3 Directing the Compilation and Linking Process

To parallelize a program containing OpenMP directives, you normally need to give only the kf90 command with the -noconc KAP switch , the -omp and -pthread Fortran 90 compiler switches, and -call_shared linker switch.

An example follows:

kf90 -fkapargs='-noconc' myprog.f90 \ -omp -pthread -call_shared

Because of the -noconc switch, Compaq KAP does not automatically set the compiler and linker switches needed for parallelization. Correct ones appear here.

To parallelize a program containing PCF directives, you need to give the kf90 command with the -noconc KAP switch and a specification of the PCF library.

An example follows:

kf90 -fkapargs='-noconc -psyntax=kap' myprog.f90 \ -lkmp_osfp10 -pthread -call_shared

Finally, you may want to create a completely non-parallelized program so you can compare its execution time with the times of programs that are parallelized in various ways (such as the automatic method and the directed method). The following command does this:

kf90 -fkapargs='-noconc -directives=ak' -noomp myprog.f90

The -noconc switch prevents automatic parallelization of DO loops and the absence of "p" from the string following -directives= prevents Compaq KAP from responding to any PCF parallel directive statements (OpenMP directives are not processed by Compaq KAP). The -noomp switch prevents the Fortran 90 compiler from responding to any OpenMP directive statements in the transformed source file.