Exit print view

Using the NEC SX-6

The purpose of this document is to assist new users of the DKRZ compute server during their first projects on the NEC SX-6. This introduction is linked with other more in-depth documents on this webserver (some of them in German) and also with the original NEC Manuals (in English). Access to these manuals is restricted to users of DKRZ. You can only access these manuals if you are logged in to the DKRZ web-server

Contents

Login
Interactive usage
Cross compiler
Filesystems
Batch jobs

Login

Usually you  access the system with

 ssh cross.dkrz.de

On cross you have access to all filesystems und can submit  your jobs on HLRE and track the execution.

Access to the NEC SX-6 itself is possible via   ssh hurrikan.dkrz.de.

You can verify the hostkey's fingerprint here (more information on => ssh ). Telnet and rlogin are not supported for security reasons.

Interactive usage

Interactive sessions on hurrikan are restricted to avoid interference with jobs in the queueing system. For program development and compiling we provide a powerful cross compiler environment which allows compiling and performance analysis of SX-6 executables on other platforms like Linux PCs or SUN workstations. Compiling a program is much faster on our cross compile server than on hurrikan. This is why you should avoid compiling interactively or, even worse, in a batch job.

Cross compiler

DKRZ provides on  cross.dkrz.de  a complete cross compiler environment to all users of DKRZ. The server cross  is also the reference installation.

The following commands are available in the cross compiler environment:

  

sxc++    

C++ und C compiler

sxf90  

 Fortran90 compiler

sxld 

Linker

sxar

ar archiver

sxmake

replaces make if sxar is used (update of libraries, headerformat!)

sxgmake

replaces gmake

sxmpic++   

MPI C und MPI C++     (MPI: Message Passing Interface)

sxmpif90

MPI Fortran90

psuite

Integrated development environment "psuite"

Some environment variables need to be set for correct operation of the cross compiler. DKRZ provides the command

module load cross-compiler

which initializes the complete cross compile environment.


Please do no mix the above listed "sx-commands" with the respective native commands. For examle you should not use the standard unix make together with sxar (see also ->).

Compiler options

The NEC manuals describe all options in detail. A short overview for the fortran compiler will be given here:

sxf90   [external opts] \
        [-Wp "<preprocessor options>"] \
        [-Wf "<internal options>"] \
        [-Wl "<linker options>"]
        <files>

The most important external options are:

      -C [ hopt | vopt | sopt | vsafe | ssafe | debug ]

They set the optimization level. Default is vopt (= vector optimization). The letters v and s mean vector and scalar, resp. hopt is the highest level of optimization and should be used with care. Be sure to check your results! (The -O option is a NEC internal option and not related to the parameter of the CRAY compiler.)

        -EP oder -Ep

activates the C preprocessor for suffixes f and f90. (-EP creates a file with preprocessor output, -Ep not)


               -ftrace

is used for detailed performance analysis.

      -eC

tells the compiler to instrument the code with the ABC test (=Array Bounds Check). Optimization is turned off then.

              -P[ auto | multi | openmp | stack | static ]

creates parallel code. Default is static (no parallel code, all variables are static). Parameter auto specifies that the automatic parallelizing function -similar to the CRAY autotasking- will be used. (see Parallelverarbeitung auf SX-6 )


        -Rn (n=0,1,2,3,4,5)
creates compile-time output lists. The default value is -R0, which means no list output.

The iteration count is important for vectorization. The compiler makes an assumption depending on array dimensions in a do loop. If this is not possible in complex loops the default value (5000) is taken. This may be too small in some cases and should be increased to the maximum possible loopcount (further information in the manual, Programmer's Guide ...).

       -Wf "-pvctl noassume loopcnt=n"
The value of n should be bigger than the loop count of the largest loop in your program. 

Libraries

You can also link libraries in SX-6 format in the cross environment. Some important libraries are installed on cross. Specially optimized versions of BLAS, LAPACK and other libs which are distributed as the MathKeisan package can be found in /pool/SX-6/MathKeisan. The MathKeisan manual describes these libs in detail.

Further you find versions of the mathematical libraries NAG and IMSL as well as of the netCDF I/O libraries in /pool/SX-6. An overview of the installed Libraries on cross and hurrikan can be found in SX-6 software list

Some of the libraries are installed in the default search path of the linker. Example for compile runs:

sxf90 -o foo foo.f90  -lblas

sxf90 -o foo foo.f -L/pool/SX-6/netcdf/netcdf-<version>/lib -lnetcdf

If a library you want to use is missing on cross,  you can still compile on cross and link your program with the missing library on hurrikan later.


 

Data transfer and filesystems

Detailed information on filesytems see here.

Batch jobs

Batch jobs for the SX-6 (hurrikan) can be submitted from cross  (and also from hurrikan)  with

qsub <parameterlist> <jobscript>

NQSII is configured in such a way, that you just have to specify the CPU time, memory consumption and number of CPUs. You do not give a queue name. Low requirements for time and/or memory will increase the internal priority of the job. Your job will be automatically queued in the right queue. (See batch jobs for more information)

Fortran and C runtime options are set through environment variables (export or setenv F_... ). A typical job script would look like:

#PBS -S /bin/csh
#PBS -l cpunum_prc=1 # 1 cpus
#PBS -l cputim_job=2:00:00 # 2 h cputime
#PBS -l memsz_job=1gb # 1 Gbyte memory
#PBS -j o # join err and out to out

#PBS -N job_II # job name
#PBS -M myname@mymail.de # you should always specify your email
# address for error messages etc

setenv F_FILEINF YES    # information about I/O to stderr {NO|YES|DETAIL},
# may increase I/O time if not {NO}

setenv F_PROGINF DETAIL # program information about speed, vectorization
# {NO|YES|DETAIL}

setenv F_FTRACE  YES    # analysis list from compile option -ftrace {NO|YES}
# compile option -ftrace should be used only during
# optimization runs (causes overhead in execution time)

/bin/echo " job started at: " \\c
date
/bin/echo " Account code: " \\c
showacct # print account code used in this hurrikan job
/bin/echo " ExecutionHost: " \\c
hostname # print name of current host

echo $PBS_O_WORKDIR      # helpful variables for the specific request
echo $PBS_JOBNAME
echo $PBS_JOBID

cd $PBS_O_WORKDIR

./prog [argument_1] [argument_2]

/bin/echo " job completed at: " \\c
date

###########################

Useful commands

qstat  lists your own active batch jobs,   qs  gives an overview of all jobs.

qdel <RequestID> deletes your batch job.

For more information please refer to the NQSII man pages.

Hint: If you are subscribed to more than one project,  you can specify which account to use for each job you submit:
 qsub -A <accountcode> ...
See Working with several accounts for more information.
Deutsches Klimarechenzentrum GmbH | Impressum