GitLab

Diagnostic values (like total KE/ total enstrophy/ mass/ etc)
are often desired outputs at each time-step. These values
are easily printed to a txt file using two functions:
add_diagnostic and write_diagnostics.

add_diagnostic adds the name and value of a diagnostic into a
pair of strings which will be written into diagnostics.txt
which the function write_diagnostics.

Usage of add_diagnostic is as follows:
add_diagnostic(name, value, header, line);
This will add the string, 'name', onto the string, 'header', which
will be the header of the diagnostics.txt file. The double (or
int), 'value', will be added onto the string, 'line', which will
be a row (or line) in the diagnostics.txt file.

write_diagnostics will write these lines (header and line) into
diagnostics.txt.

A new case file, derivatives.cpp, gives a means to compute
derivatives of any field or to compute vorticity components.
The vorticity calculation in Science is completely re-written
so-as to also work for mapped grids.

The derivatives file is built to compute secondary variables
after a run has already been completed. This uses spins' built-in
derivative toolkit allowing multiple variables to be calculated
in parallel. A derivative field (one that is the derivative of
another) is denoted by an underscore followed by the direction
the derivative was taken in (ie. u_x is the x derivative of u).

The method get_dt_max inside BaseCase was added in a
previous commit as part of the effort to include the
'default' timestep-checking code inside BaseCase.

The intent is for user code to override this method with a
problem-specific maximum timestep, but if the user code did
not do this the previous implementation would return '0'.

This would cause SPINS to error, as a timestep of <= 0 is
obviously invalid.  However, the error message would be
unintuivie, referring to the <=0 timestep rather than the
root cause of the user not implementing get_dt_max().

This change replaces the previous default method with one
that immediately aborts, with an assertion failure if
assert() is enabled at compile time.

A new case file, derivatives.cpp, gives a means to compute
derivatives of any field or to compute vorticity components.
The vorticity calculation in Science is completely re-written
so-as to also work for mapped grids.

The derivatives file is built to compute secondary variables
after a run has already been completed. This uses spins' built-in
derivative toolkit allowing multiple variables to be calculated
in parallel. A derivative field (one that is the derivative of
another) is denoted by an underscore followed by the direction
the derivative was taken in (ie. u_x is the x derivative of u).

The empty shell of check_timestep in BaseCase has been filled
with what was essentially the same thing in each case file.
Again, this is just more case file house cleaning.

A few ancillary functions have also been written to help.
The most important might be get_dt_max() which returns dt_max.
dt_max can either be specified in spins.conf or defined in the
case file based on the buoyancy frequency. Currently, if
dt_max > 0 and in spins.conf, then that value is used. Otherwise,
0.5/sqrt(N2_max) is used.

The re-definition of check_timestep in each case file can now be
removed so long as the associated ancillary functions and
parameters are included.

The generic code block to check the restart sequence is not needed
in the case file and has been moved into Options.cpp. This
streamlines the case file and makes a single copy for all other
case files to use.

The re-definition of get_restart_sequence() was missing in
gravity_current.cpp. This resulted in all restarts to read output
zero (*.0). Restarting will now properly read the output number
specified by restart_sequence.

This commit moves the helper function that process
expansion/boundary conditions ("FOURIER" -> PERIODIC, etc)
to BaseCase.cpp, from Options.

This change should be invisible to most case files, which
habitually include BaseCase for helper functions as-is, but
it allows Options to once again be independent of
NSIntegrator.  This improves modularity for other uses of
the code, such as a hypothetical SPINSoff (get it?  it's a
pun) that would like to do some spectral voodoo with an
option file but doesn't need the full Navier-Stokes
machinery.

gravity_current.cpp accepts filter parameters (f_cutoff, f_order,
f_strength) as optional input arguments. Easy adjustment of the
filter is useful when the flow become suddenly becomes too chaotic.

Parsing of expansion types in gravity_current.cpp moved to
Options.cpp. This is a general procedure that other case files
will and should make use of. It also simplifies the case file
to the specific set-up of that case.

See http://stackoverflow.com/questions/16839658 -- although
in most cases 16 decimal digits suffices for a double
precision value, in a handful of instances 17 are necessary.
At the command line, see the difference between:

$ printf %.17g 1.0000000000000001

and

$ printf %.16g 1.0000000000000001

Since SPINS does not provide bit-for-bit compatibility when
restarting (because of the startup timestepping approach),
this change is unlikely to make any significant difference
to existing or future cases.

The accuracy of the time used in dump_time.txt (the dump time)
just before writing dump fields was increased from 12 significant
figures to 16. 12 was probably fine, but since a double is used
we might as well save all digits of accuracy.

Moved the section of gravity_current.cpp regarding the adjustment
of temporal values (restarting, initial_time, etc.) when restarting
from dump into Options.cpp as it is a standard piece of code and
should be in a centralized location.

The gravity_current case was located in a subdirectory, but
its #include directives had only one ".." in the path.  This
updates those directive to properly refer to the grandparent
subdirectory (src/), allowing the case to be built as:

make -j# cases/gravity_current/gravity_current.x

This case file is restartable and will dump all variables at the end of the requested computation time (stated in spins.conf) when run on SCINET or SHARCNET. The spins.conf enables the quick set-up of a parameter sweep as parameters are set within it and not within the case file which would need to be re-compiled.
This file will be used as a basis for the upcoming additions and changes to BaseCase.cpp and Science.cpp.

This commit amends the 'hood' system-sepcific makefile to use the MPI
libraries loaded from the user's shell profile.  This is handy because
hood.math supports multiple versions of OpenMPI, which need to be used
consistently.

Now, the makefile generation process on hood locks in the MPI version
**when the makefile is generated**.  This should carry through in using
the makefile-specified MPI version at compile time, but if the user's
environment is inconsistent they'll encounter a runtime error.

This commit adds support for hood.math at the University of Waterloo
using the ICC compiler.  Before compiling and running, the user's
environment should be set up with:

. /opt/intel/bin/iccvars.sh intel64

and

. /opt/intel/composer_xe_2013_sp1.1.106/mkl/bin/mklvars.sh intel64

This change modifeis Options.hpp to add the possibility of
vector-valued options to the config file and command line.
From the 'case' code, this option is used by (for example)

std::vector<int> intvec;
add_option("Name",&intvec,"Description);

When run, options of the form --Name 1 2 3 4 (at the command
line) or Name = 1 2 3 4 (in the config file) will be added
to the vector 'intvec' as separate entries.

Because add_option is a template function, this works
identically for double (floating point)-valued options.  It
has not been tested with string options.

Since an empty vector is a perfectly cromulent and
unambiguous possibility, vector-valued options do *not*
permit the specification of defaults.

The multigrid solver regularly "rebalances" arrays, by which
it takes an input array and moves complete rows (i fixed, j
variable) such that the same global arrays divides
differently.  This is used on initial calling (to adjust the
input problem specification), on coarsening, and finally to
collapse "down" processors when near the coarsest level.

However, this code was written naïvely: it built (and
divided) the global array split at each invocation, using a
number of MPI_Allgather calls on each processor.  Profiling
(courtesy Kris Rowe) has found that this can be agonizingly
slow; in at least one test case on one particular system it
consumed over 50% of wallclock-time.

This patch fixes the problem by allowing rebalance_array
(now moved into the multigrid solver class) to cache these
array divisions.  It requires the caller to specify one of a
few categories, with the division being computed (with
Allgathers) only on the first invocation per multigrid
level and type.  The divisions currently are:

FFine, UFine, FCoarse, and UCoarse

where 'F' and 'U' refer to the error and solution terms
respectively.  Confusingly, coefficients of the Laplacian
itself (such as the uxx term) belong to the 'error' space in
terms of call structure.

This caching eliminates all but setup calls to Allgather.

After some experimentation, the previous commit of MPI-IO file
output/input had (and has) lingering bugs.  To prevent breaking existing
code, these changes have been moved into a 'testing' branch and the
'master' branch has been reverted to its prior state with this merge.

Once the MPI-IO output is fully working, it will be restored into the
master branch.

- Added 1D and 2D outputs
- Added parallel writing and reading

The automatic_grid() helper function in BaseCase was not properly
functional if called without supplied arguments for the 1D grids
xx/yy/zz.  Now, if these arguments are null or not supplied, the
function will do its grid generation using temporarily-allocated
internal 1D arrays, deleted at the end of its execution.

Previously, the automatic_grid function was creating Chebyshev-type
grids with points ordered as:

{x,z} = L/2 + L/2*cos(pi*ii/(N-1))

This is incorrect.  The expected ordering is actually the reverse, or

{x,z} = L/2 - L/2*cos(...)

and write vertical chain was moved to BaseCase (can write out vertical profile at a high temporal frequency)

Safety dump estimates output write time and writes fields with enough time before the computation limit is reached