Skip to content
Snippets Groups Projects
Commit 6b137f48 authored by Christopher Subich's avatar Christopher Subich
Browse files

Added MPI_Barrier to split file reader

This update adds an explicit MPI Barrier to the end of the split 2D file
reader.  This should address a race condition discovered by Nancy when
resuming a 2D case as 3D on Scinet, where the initially-2D files were
(correctly) overwritten by their 3D versions.  It appears that the
overwrite happened on some processes before others had opened the file,
causing the later processes to read incomplete files.  The exact
sequence appears to have been:

* Process 1 opens 'u.0' as a 2D-split file to read
* Process 1 finishes its reading and assigns the contents to its
  internally-3D array
* Process 1 begins writing out its segment of the 3D array, as 'u.0'.
  (It blocks on the write-synchronization, but has already written to
  disk.)
* Process 2 wakes up (on a remote node) and opens 'u.0' as a
  presumably-2D file.  It dies.

This race condition could, under different circumstances, also cause
data corruption (if process N wasn't reading past the end of the file in
a linear sense; possible for large Ny and small Np).  The corrective
measure was to apply an MPI Barrier at the end of read_2d_slice in
Split_reader_impl.cc (the implementation fucntion used by the more
user-friendly functions in Science.cpp).
parent b19ed904
No related branches found
No related tags found
No related merge requests found
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment