Known issues in parallel

From Gerris

(Difference between revisions)
Jump to: navigation, search
Revision as of 11:49, 10 March 2012
Zaleski (Talk | contribs)
(A two-box simulation gives different results with one or two pids - added residual.bias mystery)
← Previous diff
Revision as of 11:55, 10 March 2012
Zaleski (Talk | contribs)
(A two-box simulation gives different results with one or two pids - added mpi version info)
Next diff →
Line 39: Line 39:
m4: yes m4: yes
</pre> </pre>
 +First we run without mpi on MacOS 10.7.3 on a MacBook Pro with a four-core intel i7 system,
 +then with mpi. The mpi and compiler versions are
 +<pre>
 +% mpicc --version
 +Apple clang version 3.1 (tags/Apple/clang-318.0.54) (based on LLVM 3.1svn)
 +Target: x86_64-apple-darwin11.3.0
 +Thread model: posix
 +% mpirun --version
 +mpirun (Open MPI) 1.5.4
-Here is the result of running without mpi on MacOS 10.7.3 on a MacBook Pro with a four-core intel i7 system.+Report bugs to http://www.open-mpi.org/community/help/
 +</pre>
 +Here is the result without mpi:
<pre> <pre>
% gerris2D twobox-twopid.gfs % gerris2D twobox-twopid.gfs
Line 51: Line 62:
residual.infty: 6.400e+00 6.251e-04 2 residual.infty: 6.400e+00 6.251e-04 2
</pre> </pre>
-On the other hand, if we run the same simulation with mpi (openmpi installed with macports) +On the other hand, if we run the same simulation with mpi
and two pids, this is the result: and two pids, this is the result:
<pre> <pre>
Line 72: Line 83:
We have also run the same test case on an Ubuntu system. Here are the results. We have also run the same test case on an Ubuntu system. Here are the results.
 +
1. non-mpi 1. non-mpi

Revision as of 11:55, 10 March 2012

A two-box simulation gives different results with one or two pids

We run a simplified version of the Benard Von-Karman test case for zero time steps. Only the approximate projection is performed. The modified simulation file is

2 1  GfsSimulation GfsBox GfsGEdge {} {
  Time { iend = 0 }
  Refine 6
  Solid (x*x + y*y - 0.0625*0.0625)
  AdvectionParams{
  scheme = none
}

  OutputTime { istep = 1 } stderr
  OutputProjectionStats { istep = 1 } stderr
  OutputSimulation { start = 0.1 step = 0.1} simulation.gfs {
      variables = U,V,P
  }
}

GfsBox { id=1 pid=0
  left = Boundary {
       BcDirichlet U 1
  }
}
GfsBox { id=2 pid=1 right = BoundaryOutflow }
1 2 right

We use the following version of Gerris

% gerris2D -V
gerris: using 2D libgfs version 1.3.2 (120310-112425)
  compiled with flags:  -DBSD_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_C_SOURCE
  MPI:          yes
  pkg-config:   yes
  m4:           yes

First we run without mpi on MacOS 10.7.3 on a MacBook Pro with a four-core intel i7 system, then with mpi. The mpi and compiler versions are

% mpicc --version
Apple clang version 3.1 (tags/Apple/clang-318.0.54) (based on LLVM 3.1svn)
Target: x86_64-apple-darwin11.3.0
Thread model: posix
% mpirun --version
mpirun (Open MPI) 1.5.4

Report bugs to http://www.open-mpi.org/community/help/

Here is the result without mpi:

% gerris2D twobox-twopid.gfs
step:       0 t:      0.00000000 dt:  1.000000e-01 cpu:      0.12000000 real:      0.12236900
Approximate projection
    niter:   13
    residual.bias:   -1.000e-01 -1.984e-04
    residual.first:   5.020e-02  9.960e-05    1.6
    residual.second:  5.668e-01  1.330e-04    1.9
    residual.infty:   6.400e+00  6.251e-04      2

On the other hand, if we run the same simulation with mpi and two pids, this is the result:

% mpirun -np 2 gerris2D twobox-twopid.gfs
step:       0 t:      0.00000000 dt:  1.000000e-01 cpu:      0.04000000 real:      0.03555900
Approximate projection
    niter:    4
    residual.bias:   -1.000e-01 -7.446e-05
    residual.first:   5.020e-02  3.839e-05      6
    residual.second:  5.668e-01  4.914e-05     10
    residual.infty:   6.400e+00  2.713e-04     12

The two results are different: the pre-iteration Projection statistics (first column) are the same but the post-iteration Projection statistics are different. However, since there are the same number of boxes, the mpi communication should send the same information that is exchanged between boxes in the non-mpi run. Thus something is amiss in the way information is exchanged between boxes.

Using larger number of boxes and pids (typically 24), we found cases where the non-mpi runs converge but the mpi run do not converge, i.e. the residual is not reduced below the required minimum of 0.001 .

We have also run the same test case on an Ubuntu system. Here are the results.

1. non-mpi

% gerris2D cylinder.gfs
step:       0 t:      0.00000000 dt:  5.263158e-03 cpu:      0.19000000 real:      0.20179100
Approximate projection
    niter:   13
    residual.bias:   -5.020e-02 -9.960e-05
    residual.first:   5.020e-02  9.960e-05    1.6
    residual.second:  5.668e-01  1.330e-04    1.9
    residual.infty:   6.400e+00  6.251e-04      2

2. mpi

% mpirun -np 2 gerris2D cylinder.gfs
step:       0 t:      0.00000000 dt:  5.263158e-03 cpu:      0.05000000 real:      0.05292300
Approximate projection
    niter:    4
    residual.bias:   -5.020e-02 -3.737e-05
    residual.first:   5.020e-02  3.839e-05      6
    residual.second:  5.668e-01  4.914e-05     10
    residual.infty:   6.400e+00  2.713e-04     12

Notice the results are the same, except for the cpu times and residual.bias. the pre and post iteration residual.bias values are divided by two when going from MacOS to Ubuntu.

Personal tools
communication