Dewedseed's Blog

Fortran MPI 并行编程

记录 Fortran MPI 编程相关问题及实现

Question

  1. MPI 基本思想 (多进程)
  2. 为什么要使用 MPI
  3. MPI 数据通信方式
  4. 多进程调试 Debug
  5. 非阻塞通信
  6. 负载均衡: 内存分布比算力分布更重要

Introduction

  • Message Passing Interface
  • MPI standard
  • MPI implementations
    • OpenMPI
      • Layer: OMPI, ORTE, OPAL
      • Framework
        • OSC: MPI one-side communications
        • PML: MPI point-to-point management Layer
    • MPICH
  • why MPI
    • DSM-NUMA: memory access conflict
    • Cluster/COW => MPI

Concept

  • Hardware
    • compute node
    • physical core and logical core
    • thread
  • Software
    • MPI
      • Process
      • Communicator 通信域
        • Group
      • Envelope 信封
        • Buffer

Programming

  1. Paradigm – SPMD Single Program Multiple Data
  2. Data transfer – Point to Point, Collectives

Point to Point

one – one

  1. Send & Receive
  2. Message Envelope
  3. Status

Collective

one – many

  1. Single data: Bcase & Reduce
  2. Multiple data: Scatter & Gather
  3. Implement by point-point
    1. one to many (binary tree)

Group

  • World
    • Group

Implementation

flowchart LR
  mpi(MPI)
  bg(Begin)
    init[MPI_Init]
  comm(Communicator)
    commp(MPI_Comm_rank </br> MPI_Comm_size)
  gp(Group)
    commgp(MPI_Comm_Split </br> MPI_Comm_rank)
  parad(Paradigm)
  pp(Point-Point)
    ppb(MPI_Send </br> MPI_Recv</br> MPI_Sendrecv)
    ppnb(MPI_ISend </br> MPI_IRecv</br> MPI_Wait </br> MPI_Test)
  cl(Collective)
    clb(MPI_Bcast </br> MPI_Reduce)
  sy(Synchronization)
    syb(MPI_Barrier)
  time(Wall time)
    mtime(MPI_Wtime)
  type(Type)
    mtype(MPI_Type_Vector </br> MPI_Type_Commit </br> MPI_Type_Indexed)
  ed(End)
    final[MPI_Finalize]

  mpi --> bg --> init
  mpi --> comm --> commp
          comm --> gp --> commgp
  mpi --> parad
          parad --> pp --> |Blocking| ppb
                    pp --> |Non-Blocking| ppnb
          parad --> cl --> clb
  mpi --> sy --> syb
  mpi --> time --> mtime
  mpi --> type --> mtype
  mpi --> ed
          ed --> final

Compile

Environment

  • 安装 OpenMPI
1
2
3
4
5
6
7
8
# download release package
wget https://download.open-mpi.org/release/open-mpi/v5.0/openmpi-5.0.6.tar.gz
tar -xvf openmpi-5.0.6.tar.gz
# build
cd openmpi-5.0.6 && mkdir build && cd build
../configure --prefix=$HOME/Program/OpenMPI/build 2>&1 | tee config.out
make -j 8 all 2>&1 | tee make.out
make install 2>&1 | tee install.out
  • 配置环境
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
#%Module

proc ModulesHelp {} {
  puts stderr "Program:     OpenMPI $::version"
  puts stderr "Description: Open Message Passing Interface."
  puts stderr "Home Page:   https://www.open-mpi.org/"
  puts stderr "Github:      https://github.com/open-mpi/ompi"
  puts stderr {}
}

set             version     5.0.6
set             prefix      $HOME/Program/OpenMPI/build

module-whatis "OpenMPI -- Open Message Passing Interface"

setenv          OpenMPI_HOME       $prefix
setenv          OpenMPI_INCLUDE    $prefix/include
setenv          OpenMPI_LIB        $prefix/lib
prepend-path    PATH               $prefix/bin
prepend-path    LD_LIBRARY_PATH    $prefix/lib
prepend-path    MANPATH            $prefix/share/man

Make

  • Makefile
1
2
3
4
# link lib
gfortran xxx.f90 -I/path/to/include -L/path/to/lib -lmpi_mpifh
# auto: mpifort, mpif90
mpifort xxx.f90 -I/path/to/include -L/path/to/lib -lmpi_mpifh
  • Cmake
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# find mpi
find_package(MPI REQUIRED)
if(MPI_FOUND)
    include_directories(${MPI_Fortran_INCLUDE_PATH})
    message(STATUS "OpenMPI library: ${MPI_Fortran_LIBRARIES}")
    message(STATUS "OpenMPI include: ${MPI_Fortran_INCLUDE_PATH}")
else()
    message(FATAL_ERROR "OpenMPI library not found")
endif()

# set paths
set(OPENMPI_ROOT ${PROJECT_SOURCE_DIR}/ext/openmpi-5.0.6)

find_path(
  MPI_Fortran_INCLUDE_PATH
  NAMES mpif.h
  PATHS ${OPENMPI_ROOT}/include
)

find_library(
  MPI_Fortran_LIBRARIES
  NAMES mpi_mpifh
  PATHS ${OPENMPI_ROOT}/lib
  NO_DEFAULT_PATH
)

if(MPI_Fortran_INCLUDE_PATH AND MPI_Fortran_LIBRARIES)
  set(MPI_FOUND TRUE)
  set(MPI_Fortran_LIBRARIES ${MPI_Fortran_LIBRARIES})
  set(MPI_Fortran_INCLUDE_PATH ${MPI_Fortran_INCLUDE_PATH})
else()
  set(MPI_FOUND FALSE)
endif()

Debug

Non-Blocking

  • Deadlock 死锁
    • Process 1 中先 send a 再 send b
    • Process 2 需先 recv b 再 recv a
    • 解决方法： 非阻塞通信
  • Non-Blocking
    • 允许计算和通信重叠进行，提高并行效率
    • 注意不能修改发送或接收缓冲区中的数据

Optimization

  • 负载均衡 Load Balance
    • split array => blance computation and memory
    • cycle => blance communication
  • 通信优化 Minimize Communication
    • 并行 IO

Reference

0%