# Known issues in parallel

### From Gerris

Revision as of 09:07, 10 March 2012Zaleski (Talk | contribs) (link fixed) ← Previous diff |
Revision as of 11:31, 10 March 2012Zaleski (Talk | contribs) (→A two-box simulation gives different results with one or two pids - updated Gerris version) Next diff → |
||

Line 28: |
Line 28: | ||

<pre> | <pre> | ||

% gerris2D -V | % gerris2D -V | ||

- | gerris: using 2D libgfs version 1.3.2 (120229-075733 + local changes) | + | gerris: using 2D libgfs version 1.3.2 (120310-112425) |

- | compiled with flags: -DBSD_SOURCE -D_DARWIN_C_SOURCE | + | compiled with flags: -DBSD_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_C_SOURCE |

MPI: yes | MPI: yes | ||

pkg-config: yes | pkg-config: yes | ||

Line 38: |
Line 38: | ||

<pre> | <pre> | ||

% gerris2D twobox-twopid.gfs | % gerris2D twobox-twopid.gfs | ||

- | step: 0 t: 0.00000000 dt: 5.263158e-03 cpu: 0.12000000 real: 0.12951800 | + | step: 0 t: 0.00000000 dt: 1.000000e-01 cpu: 0.12000000 real: 0.12236900 |

Approximate projection | Approximate projection | ||

niter: 13 | niter: 13 | ||

Line 50: |
Line 50: | ||

<pre> | <pre> | ||

% mpirun -np 2 gerris2D twobox-twopid.gfs | % mpirun -np 2 gerris2D twobox-twopid.gfs | ||

- | step: 0 t: 0.00000000 dt: 5.263158e-03 cpu: 0.03000000 real: 0.03374700 | + | step: 0 t: 0.00000000 dt: 1.000000e-01 cpu: 0.04000000 real: 0.03555900 |

Approximate projection | Approximate projection | ||

niter: 4 | niter: 4 |

## Revision as of 11:31, 10 March 2012

## A two-box simulation gives different results with one or two pids

We run a simplified version of the Benard Von-Karman test case. The modified simulation file is

2 1 GfsSimulation GfsBox GfsGEdge {} { Time { iend = 0 } Refine 6 Solid (x*x + y*y - 0.0625*0.0625) OutputTime { istep = 1 } stderr OutputProjectionStats { istep = 1 } stderr OutputSimulation { start = 0.1 step = 0.1} simulation.gfs { variables = U,V,P } } GfsBox { id=1 pid=0 left = Boundary { BcDirichlet U 1 } } GfsBox { id=2 pid=1 right = BoundaryOutflow } 1 2 right

We use the following version of Gerris

% gerris2D -V gerris: using 2D libgfs version 1.3.2 (120310-112425) compiled with flags: -DBSD_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_C_SOURCE MPI: yes pkg-config: yes m4: yes

Here is the result of running without mpi on MacOS 10.7.3 on a MacBook Pro with a four-core intel i7 system.

% gerris2D twobox-twopid.gfs step: 0 t: 0.00000000 dt: 1.000000e-01 cpu: 0.12000000 real: 0.12236900 Approximate projection niter: 13 residual.bias: -1.000e-01 -1.984e-04 residual.first: 5.020e-02 9.960e-05 1.6 residual.second: 5.668e-01 1.330e-04 1.9 residual.infty: 6.400e+00 6.251e-04 2

On the other hand, if we run the same simulation with mpi (openmpi installed with macports) and two pids, this is the result:

% mpirun -np 2 gerris2D twobox-twopid.gfs step: 0 t: 0.00000000 dt: 1.000000e-01 cpu: 0.04000000 real: 0.03555900 Approximate projection niter: 4 residual.bias: -1.000e-01 -7.446e-05 residual.first: 5.020e-02 3.839e-05 6 residual.second: 5.668e-01 4.914e-05 10 residual.infty: 6.400e+00 2.713e-04 12

The two results are different: the pre-iteration Projection statistics (first column) are the same but the post-iteration Projection statistics are different. However, since there are the same number of boxes, the mpi communication should send the same information that is exchanged between boxes in the non-mpi run. Thus something is amiss in the way information is exchanged between boxes.

Using larger number of boxes and pids (typically 24), we found cases where the non-mpi runs converge but the mpi run do not converge, i.e. the residual is not reduced below the required minimum of 0.001 .

We have also run the same test case on an Ubuntu system. Here are the results. 1. non-mpi

% gerris2D cylinder.gfs step: 0 t: 0.00000000 dt: 5.263158e-03 cpu: 0.19000000 real: 0.20179100 Approximate projection niter: 13 residual.bias: -5.020e-02 -9.960e-05 residual.first: 5.020e-02 9.960e-05 1.6 residual.second: 5.668e-01 1.330e-04 1.9 residual.infty: 6.400e+00 6.251e-04 2

2. mpi

% mpirun -np 2 gerris2D cylinder.gfs step: 0 t: 0.00000000 dt: 5.263158e-03 cpu: 0.05000000 real: 0.05292300 Approximate projection niter: 4 residual.bias: -5.020e-02 -3.737e-05 residual.first: 5.020e-02 3.839e-05 6 residual.second: 5.668e-01 4.914e-05 10 residual.infty: 6.400e+00 2.713e-04 12

Notice the results are the same, except for the cpu times and residual bias.