A second calling of analysis was previously added to run on the
initial configuration, but this was repeated for each subsequent
time-step. This repetition has been fixed.

Also reorganize how some diagnostics are added to the file.
This is now a little clearer to read since similar diagnostics
are placed near each other.

The dimensional rho option should not be after the mapping options
since the mapping options will need to be specified for an
unmapped dimensional density case. This now makes the optional
argument ordering more logical.

In the previous merge, compilation of Sorter.o went missing from the
Makefile, and a typo was introduced in the gravity_current case file;
each would prevent a build from finishing.  These are fixed.

In certain cases, diagnostic computation required Nx/y/z > 1, but this
check was not repeated for adding the diagnostic output to the
respective file; this could result in unexpectedly writing 0 or
meaningless data to the file.

Conflict in Makefile, fixed with formatting corrections

This changeset adds a compile-time option on whether to include the
timing code.  If SPINS is compiled without TIMINGS=true, the timing code
is #defined to a no-op and should not otherwise affect the performance
of the software; timing_stack_report will instead print a short error
message.

Because this affects #defines in header files, changing from
TIMINGS=true to false (or vice versa) should only be done with a 'make
clean'.  Not doing so could leave the timing code in an inconsistent
state and result in either link-time errors (if older code calls timing
code that is #defined away) or a corrupted/growing timing stack (if for
example a timing_push() is compiled in but the corresponding
timing_pop() is removed.)

This change allows the gravity_current case to compute the v^2 component
of the kinetic energy term even if NY=1 when rot_f != 0; in that case
the solution can still develop a 2D geostrophic balance where v-velocity
balances a pressure gradient.

Additionally, the respective component sums are explicitly initialized to
0.

Christopher Subich authored Nov 01, 2017 and Christopher Subich committed Nov 01, 2017

The template form of add_diagnostic<x> is attractive, but to
be properly used the template implementation must be visible
at the same time as the header file itself.  The
instantiations formerly in BaseCase.cpp were not likely to
be visible to a program, increasing compile time (if they
were re-instantiated) or causing an error (if the
implementation was not visible).  This change:

*) Moves the implementation of add_diagnostic to a new
BaseCase_impl.cc
*) Includes this impementation file from BaseCase.hpp, with
a double-inclusion guard
*) Preserves instantiations of <double> and <int>, with new
'extern' declarations in BaseCase.hpp to prevent
double-definition.

The spinscase.cpp file is the case file showing what was run in
a given simulation. If derivatives are calculated afterwards
(using derivatives.x) then this file gets over-written and case
specific parameters/instructions will be lost. Rather than be
over-written, a different file (derivatives.cpp) will be created
when derivatives.x is run.

First and most trivially, this change adds a few more timing push/pop
invocations relating to multigrid and especially to some of the MPI
synchronization steps that might cause processors to wait on each other.

More substantively, this change also adds a new tridiangonal solver
based on the bog-standard forwards/backwards elimination Thompson
algorithm (see Wikipedia).  This should be acceptable because the 1D
problems being solved themselves come from a 2D problem, so we don't
expect ill-conditioning; calling out to the GMRES banded solver was
surprisingly a computational bottleneck perhaps because of pivoting.

This change seems to decrease the line-solve time by about 80%, which in
turn decreases the overall runtime (tank_rho test case) by 40%.

This is the last energy conversion term to get a complete
energy budget.

Some diagnostics are only computed in a specific configuration
(ie. a 2D field does not need to compute all three vorticity
components).

The MPI_BroadCast call in the check_and_dump appears to be time
consuming. When the compute_time is not specified (or is <=0)
then checking for the dump is unnecessary. Re-wrote the code to
not broadcast in this instance.

Creating a rank 3 matrix requires all three indices in the declaration
statement. This is only necessary when debugging, but should be explicit
wherever possible. Otherwise, too much time will be spent trying to
remember this fact when it matters.

The BPE calculation had assumed that the density field was
non-dimensionalized as the density anomaly. An optional argument
allows the user to set the density field as dimensional.

The main while loop in the triple loop has been better optimized to
remember the location from the previous loop step. This results in
only one Nx loop over the entire triple loop. Reducing the loop
from O(Nx^2*Ny*Nz) to O(Nx*Ny*Nz).

Now tracking the partial sums of dx, and the volume of the hill.
This makes comparisons for finding depths easier later on.

And other minor coding changes.

Some fprintf statements were written as stderr and should be stdout

The mapping affects how to spead voxels out at a given depth.
A sorted hill makes loops easier, and thus the calculation of the
surface area for a given voxel that intersect the hill.

This commit adds stack-structured timing code, such that bits of SPINS
can be instrumented with

>  timing_push("short name")
>  [... work goes here]
>  timing_pop()

This is recursive, such that already-timed portions of the code can call
the timing code to further instrument subroutines or discrete code
segments, and the resulting structure follows the -dynamic- call stack.
This does have the odd result that portions of the code that are re-used
(namely spectral derivatives) can show up as several 'leaf' nodes in the
graph.

To print out the report to standard error, call

>  timing_stack_report()

The exact format of the report should be considered tentative.

See also commit: 46ca2c72

The background Potential Energy (BPE) is calculated in
compute_Background_PE in Science.cpp. The APE can
be calculated afterwards from the BPE and PE from the
diagnostic file.

Diagnostic values (like total KE/ total enstrophy/ mass/ etc)
are often desired outputs at each time-step. These values
are easily printed to a txt file using two functions:
add_diagnostic and write_diagnostics.

add_diagnostic adds the name and value of a diagnostic into a
pair of strings which will be written into diagnostics.txt
which the function write_diagnostics.

Usage of add_diagnostic is as follows:
add_diagnostic(name, value, header, line);
This will add the string, 'name', onto the string, 'header', which
will be the header of the diagnostics.txt file. The double (or
int), 'value', will be added onto the string, 'line', which will
be a row (or line) in the diagnostics.txt file.

write_diagnostics will write these lines (header and line) into
diagnostics.txt.

A new case file, derivatives.cpp, gives a means to compute
derivatives of any field or to compute vorticity components.
The vorticity calculation in Science is completely re-written
so-as to also work for mapped grids.

The derivatives file is built to compute secondary variables
after a run has already been completed. This uses spins' built-in
derivative toolkit allowing multiple variables to be calculated
in parallel. A derivative field (one that is the derivative of
another) is denoted by an underscore followed by the direction
the derivative was taken in (ie. u_x is the x derivative of u).

The method get_dt_max inside BaseCase was added in a
previous commit as part of the effort to include the
'default' timestep-checking code inside BaseCase.

The intent is for user code to override this method with a
problem-specific maximum timestep, but if the user code did
not do this the previous implementation would return '0'.

This would cause SPINS to error, as a timestep of <= 0 is
obviously invalid.  However, the error message would be
unintuivie, referring to the <=0 timestep rather than the
root cause of the user not implementing get_dt_max().

This change replaces the previous default method with one
that immediately aborts, with an assertion failure if
assert() is enabled at compile time.