The purpose of this document is to assist new users of the DKRZ compute server during their first projects on the NEC SX-6. This introduction is linked with other more in-depth documents on this webserver (some of them in German) and also with the original NEC Manuals (in English). Access to these manuals is restricted to users of DKRZ. You can only access these manuals if you are logged in to the DKRZ web-server
Login
Interactive usage
Cross compiler
Filesystems
Batch jobs
Usually you access the system with
ssh cross.dkrz.de
On cross you have access to all filesystems und can submit your jobs on HLRE and track the execution.The following commands are available in the cross compiler environment:
|
sxc++ |
C++ und C compiler |
|
sxf90 |
Fortran90 compiler |
|
sxld |
Linker |
|
sxar |
ar archiver |
|
sxmake |
replaces make if sxar is used (update of libraries, headerformat!) |
|
sxgmake |
replaces gmake |
|
sxmpic++ |
MPI C und MPI C++ (MPI: Message Passing Interface) |
|
sxmpif90 |
MPI Fortran90 |
|
psuite |
Integrated development environment "psuite" |
Some environment variables need to be set for correct operation of the cross compiler. DKRZ provides the command
module load cross-compiler
which initializes the complete cross compile environment.The NEC manuals describe all options in detail. A short overview for the fortran compiler will be given here:
sxf90 [external opts] \
[-Wp "<preprocessor
options>"] \
[-Wf "<internal
options>"] \
[-Wl "<linker
options>"]
<files>
The most important external options are:
-C [ hopt | vopt | sopt | vsafe | ssafe | debug ]
They set the optimization level. Default is vopt (=
vector optimization). The letters v and s mean vector and scalar, resp.
hopt is the highest level of optimization and should be used
with care. Be sure to check your results! (The -O option is a NEC
internal option and not related to the parameter of the CRAY compiler.)
-EP oder -Ep
activates the C preprocessor for suffixes f and f90. (-EP creates a file with preprocessor output, -Ep not)
-ftrace
is used for detailed performance analysis.
-eC
tells the compiler to instrument the code with the ABC test (=Array Bounds Check). Optimization is turned off then.
-P[ auto | multi | openmp | stack | static ]
creates parallel code. Default is static (no parallel code, all variables are static). Parameter auto specifies that the automatic parallelizing function -similar to the CRAY autotasking- will be used. (see Parallelverarbeitung auf SX-6 )
creates compile-time output lists. The default value is -R0, which means no list output.
-Rn (n=0,1,2,3,4,5)
The iteration count is important for vectorization. The compiler makes an assumption depending on array dimensions in a do loop. If this is not possible in complex loops the default value (5000) is taken. This may be too small in some cases and should be increased to the maximum possible loopcount (further information in the manual, Programmer's Guide ...).
-Wf "-pvctl noassume loopcnt=n"The value of n should be bigger than the loop count of the largest loop in your program.
You can also link libraries in SX-6 format in the cross environment. Some important libraries are installed on cross. Specially optimized versions of BLAS, LAPACK and other libs which are distributed as the MathKeisan package can be found in /pool/SX-6/MathKeisan. The MathKeisan manual describes these libs in detail.
Further you find versions of the mathematical libraries NAG and IMSL as well as of the netCDF I/O libraries in /pool/SX-6. An overview of the installed Libraries on cross and hurrikan can be found in SX-6 software list
Some of the libraries are installed in the default search path of the linker. Example for compile runs:
sxf90 -o foo foo.f90 -lblas
sxf90 -o foo foo.f -L/pool/SX-6/netcdf/netcdf-<version>/lib -lnetcdf
If a library you want to use is missing on cross, you can still compile on cross and link your program with the missing library on hurrikan later.
Batch jobs for the SX-6 (hurrikan) can be submitted from cross (and also from hurrikan) with
qsub <parameterlist> <jobscript>
NQSII is configured in such a way, that you just have to specify the CPU time, memory consumption and number of CPUs. You do not give a queue name. Low requirements for time and/or memory will increase the internal priority of the job. Your job will be automatically queued in the right queue. (See batch jobs for more information)
Fortran and C runtime options are set through environment variables (export or setenv F_... ). A typical job script would look like:
#PBS -S /bin/csh
#PBS -l cpunum_prc=1 # 1 cpus
#PBS -l cputim_job=2:00:00 # 2 h cputime
#PBS -l memsz_job=1gb # 1 Gbyte memory
#PBS -j o # join err and out to out
#PBS -N job_II # job name
#PBS -M myname@mymail.de # you should always specify your email
# address for error messages etc
setenv F_FILEINF YES # information about I/O to stderr {NO|YES|DETAIL},
# may increase I/O time if not {NO}
setenv F_PROGINF DETAIL # program information about speed, vectorization
# {NO|YES|DETAIL}
setenv F_FTRACE YES # analysis list from compile option -ftrace {NO|YES}
# compile option -ftrace should be used only during
# optimization runs (causes overhead in execution time)
/bin/echo " job started at: " \\c
date
/bin/echo " Account code: " \\c
showacct # print account code used in this hurrikan job
/bin/echo " ExecutionHost: " \\c
hostname # print name of current host
echo $PBS_O_WORKDIR # helpful variables for the specific request
echo $PBS_JOBNAME
echo $PBS_JOBID
cd $PBS_O_WORKDIR
./prog [argument_1] [argument_2]
/bin/echo " job completed at: " \\c
date
###########################
qstat lists your own active batch jobs, qs gives an overview of all jobs.
qdel <RequestID> deletes
your batch job.
For more information please refer to the NQSII man pages.
qsub -A <accountcode> ...See Working with several accounts for more information.