Info
Alle Inhalte des Nutzerportal sind nur auf Englisch verfügbar.
Sie sind hier: Startseite / Services / Code Tuning / Collect Performance Data / Lightweight MPI analysis

Lightweight MPI analysis

MPI implementations usually provide a feature to analyse the MPI usage at runtime without any modification to the src code.

Intel MPI

The Intel MPI library has a built in statistics gathering facility that collects essential performance data without disturbing the application execution. This data can afterwards be analysed using the Intel MPI Performance Snapshot utility to get an overview of the application performance.

To record the data, you need to built (or at least link) your application with the IntelMPI version 5.1.2.150 or later. While the jobs batch script can be used as usual, the additional environment variable I_MPI_STATS needs to be set to one of the values 1,2,3,4,10 or 20. The higher the value, the more information that will be provided, ie.

export I_MPI_STATS=20

will give you most detailed information on MPI runtime.

One could further restrict the collection of data to some MPI subsystems only via

export I_MPI_STATS_SCOPE="<subsystem>[:<ops>][;<subsystem>[:<ops>][...]]"

Where subsystem is one of coll (for collective MPI operations), p2p (for point-to-point MPI operations) or all (which is the default) - ops can further be used to specify MPI operations explicitely. For details refer to the Intel MPI developer reference.

Once the application has finished, the results are stored in a plain text file called stats.txt. If you wanna store the data in a different file, set the filename to be used via the environment variable I_MPI_STATS_FILE.

You can further analyse the data using the Intel mps tool (installed at /sw/rhel6-x64/intel/traceanalyzer/9.1_update2/itac/9.1.2.024/intel64/bin/mps) and produce a more readable summary in txt or html format

$ /sw/rhel6-x64/intel/traceanalyzer/9.1_update2/itac/9.1.2.024/intel64/bin/mps -g -t -D -o stats.txt

bullxMPI

For the current installation of bullxMPI with mellanox tools (ie bullxmpi_mlx/bullxmpi_mlx-1.2.8.3) we provide a corresponding installation of two different tools:

  1. mpiP, which is a lightweight profiling library for MPI applications. In addition to the MPI summary profiling, mpiP can provide "call site" statistics showing which calls in the code are dominating MPI execution time.
  2. IPM, which provides a low-overhead performance profile of the performance aspects and resource utilization in a parallel program.

mpiP

Using mpiP does not require code recompilation. Instead, LD_PRELOAD is used to dynamically load the mpiP library (libmpiP.so) as a wrapper to the MPI runtime. Therefore, you just need to modify the batch script as show here

export LD_PRELOAD=/sw/rhel6-x64/mpi/mpiP-3.4.1-bullxmpi-intel14/lib/libmpiP.so
srun ... # as usual
unset LD_PRELOAD

The mpiP profiler generates a text based output file named like

<appname>.<ntasks>.20290.1.mpiP

For details refer to the official mpiP website.

IPM

Using IPM does not require code recompilation. Instead, LD_PRELOAD is used to dynamically load the IPM library (libipm.so) as a wrapper to the MPI runtime. Therefore, you just need to modify the batch script as show here

export LD_PRELOAD=/sw/rhel6-x64/mpi/ipm-0.983-bullxmpi-intel14/lib/libipm.so
srun ... # as usual
unset LD_PRELOAD

The IPM profiler generates by default a summary of the performance information for the application on stdout. IPM also generates an XML file that can be used to generate a graphical webpage. This file is named something like

<username>.1459775774.278838.0

The postprocessing then needs to be done on your local machine:

    1. make sure that ploticus is installed on your local machine (see here)
    2. copy the XML output file, the ipm_parse script and ipm_key file to your machine (located at /sw/rhel6-x64/mpi/ipm-0.983-bullxmpi-intel14)
    3. create the webpage using
IPM_KEYFILE=./ipm_key ./ipm_parse -html <ipm_xml_file>
  1. you will be find a new directory with an index.html file that can be opened with any browser

For details refer to the official IPM website.

Artikelaktionen