Combined the mapped and unmapped cases for this into a single
case file.

With the retirement of belize, which formerly hosted a fixed
copy of the dependency libraries, make_deps.sh would no
longer download and build said libraries.

These libraries are now instead hosted in a dedicated
repository at UWGit, so make_deps.sh has been changed to
clone this repository.

The order of the input arguments for compute_vort_x,y,z was changed
some time over the last year, but this was not accounted for in
the calculation for enstrophy density.

This case has uncovered that the BPE calculation returns
incorrect values when a Chebyshev grid is used on a field scale.
The BPE calculation uses the quadrature weights as the volumes
of each cell. This is true when the grid is regular, but incorrect
for Chebyshev grids.

To first approximation the Cheb quadrature weights are close to
the volumes of the cells. This leads to the BPE calculation
being accurate to 8 significant figures. 8 sig figs is not enough
on fields scales since the magnitude of the PE can be very, very
large. The difference of the BPE from the PE can thus be smaller
than these 8 sig figs.

Some simulations have a long clock time per step, but have a short
write to disk time. If only the short write to disk time is
accounted for in the check_and_dump command then insufficient
time may be allocated for the dump to disk. Both are now accounted
for.

The energy transfer rate from internal energy to BPE is generalized
to handle both mapped, and unmapped cases. What had been there
was in fact correct for both, though it wasn't realized at first.

Archived old wave_reader cases.

Minor fixes in other case files.

The full xgrid and ygrid are unnecessary since x and y only
depend on a single variable (or index - ii for x, and jj for y).
The variables xx and yy can be used instead of xgrid and ygrid,
respectively. Creating the full grids for these fields is a
waste of memory.

This commit amends Cheb_2dmg::resid_dot (dot-product of
residual vectors in the context of the 2D GMRES/multigrid
iteration) to perform the dot-product in the simplest way,
as a sum over grid elements.

Before this, the code (which is left in the file but
disabled by #if 1 / #else) tried to be clever and
evenly-weight contributions from the interior, the boundary
conditions, and the mean-pressure error (if the problem was
indeterminate).  This evidently caused the long-standing
problem of the GMRES inner iteration being far more
optimistic about error (as reported by LAPACK) than would be
computed by directly measuring the residual error.

This problem was ultimately discovered when the
doc_map_iwave case broke.  Even though this broken residual
code has been in SPINS since before it was
version-controlled via GIT, the map_iwave case would still
converge (and relatively quickly) when the multigrid
iteration had a relatively large coarse-solve size.

Git bisect pinned the broken documentation case on the
change that reduced the coase-grid size, which reduced the
GMRES convergence rates enough that the errors caused by
*this* problem broke convergence.

Since this change replaces the error norm calculations used
by GMRES, it should be treated with caution and mapped-grid
cases should have a mark-1-eyeball inspection of results for
a while.

--

Additionally, for debugging utility this change also
includes crude residual/basis output for the 2D GMRES
solver.  All current uses are commented out.

The addition of an analysis call on the initial data resulted
in the Nth time step to correspond with iteration N+1. This isn't
a problem, just dissatisfying.

The stress along the bottom and top surfaces are calculated
and written to a file (stresses_(top/bottom).txt) using the function
stresses_top and stresses_bottom in BaseCase. These functions are
especially useful with mapped cases.

If dt_max is not given in the spins.conf, it is now explicitly
set to 0 so as to be adjusted later to use the buoyancy frequency.
The dt_max is also printed in the initialization for clarity.

A second calling of analysis was previously added to run on the
initial configuration, but this was repeated for each subsequent
time-step. This repetition has been fixed.

Also reorganize how some diagnostics are added to the file.
This is now a little clearer to read since similar diagnostics
are placed near each other.

The dimensional rho option should not be after the mapping options
since the mapping options will need to be specified for an
unmapped dimensional density case. This now makes the optional
argument ordering more logical.

In the previous merge, compilation of Sorter.o went missing from the
Makefile, and a typo was introduced in the gravity_current case file;
each would prevent a build from finishing.  These are fixed.

In certain cases, diagnostic computation required Nx/y/z > 1, but this
check was not repeated for adding the diagnostic output to the
respective file; this could result in unexpectedly writing 0 or
meaningless data to the file.

Conflict in Makefile, fixed with formatting corrections

This changeset adds a compile-time option on whether to include the
timing code.  If SPINS is compiled without TIMINGS=true, the timing code
is #defined to a no-op and should not otherwise affect the performance
of the software; timing_stack_report will instead print a short error
message.

Because this affects #defines in header files, changing from
TIMINGS=true to false (or vice versa) should only be done with a 'make
clean'.  Not doing so could leave the timing code in an inconsistent
state and result in either link-time errors (if older code calls timing
code that is #defined away) or a corrupted/growing timing stack (if for
example a timing_push() is compiled in but the corresponding
timing_pop() is removed.)

This change allows the gravity_current case to compute the v^2 component
of the kinetic energy term even if NY=1 when rot_f != 0; in that case
the solution can still develop a 2D geostrophic balance where v-velocity
balances a pressure gradient.

Additionally, the respective component sums are explicitly initialized to
0.

Christopher Subich authored Nov 01, 2017 and Christopher Subich committed Nov 01, 2017

The template form of add_diagnostic<x> is attractive, but to
be properly used the template implementation must be visible
at the same time as the header file itself.  The
instantiations formerly in BaseCase.cpp were not likely to
be visible to a program, increasing compile time (if they
were re-instantiated) or causing an error (if the
implementation was not visible).  This change:

*) Moves the implementation of add_diagnostic to a new
BaseCase_impl.cc
*) Includes this impementation file from BaseCase.hpp, with
a double-inclusion guard
*) Preserves instantiations of <double> and <int>, with new
'extern' declarations in BaseCase.hpp to prevent
double-definition.

The spinscase.cpp file is the case file showing what was run in
a given simulation. If derivatives are calculated afterwards
(using derivatives.x) then this file gets over-written and case
specific parameters/instructions will be lost. Rather than be
over-written, a different file (derivatives.cpp) will be created
when derivatives.x is run.

First and most trivially, this change adds a few more timing push/pop
invocations relating to multigrid and especially to some of the MPI
synchronization steps that might cause processors to wait on each other.

More substantively, this change also adds a new tridiangonal solver
based on the bog-standard forwards/backwards elimination Thompson
algorithm (see Wikipedia).  This should be acceptable because the 1D
problems being solved themselves come from a 2D problem, so we don't
expect ill-conditioning; calling out to the GMRES banded solver was
surprisingly a computational bottleneck perhaps because of pivoting.

This change seems to decrease the line-solve time by about 80%, which in
turn decreases the overall runtime (tank_rho test case) by 40%.

This is the last energy conversion term to get a complete
energy budget.