NAME¶
mgd77sniffer - Scan for errors in MGD77 data
SYNOPSIS¶
mgd77sniffer NGDC-ids [
-Afieldabbrev,
scale,
offset ] [
-Cmaxspd ] [
-Dd|
e|
E|
f|
l|
m|
s|
v[
r] ] [
-gfieldabbrev,
imggrid,
scale,
mode ] [
-G fieldabbrev,
grid ] [
-H[
i][
nrec]
] [
-Ifieldabbrev,
rec1,
recN ] [
-K ] [
-L custom limits file ] [
-N ] [
-Q[
b|
c|
l|
n][[
/]
threshold] ]
[
-Rwest/
east/
south/
north[
r] ] [
-S d|
s|
t ] [
-Tgap ] [
-V ] [
-Wc|
g|
o|
s|
t|
v|
x ] [
-bo[
s|
S|
d|
D[
ncol]|
c[
var1/...]]
]
DESCRIPTION¶
mgd77sniffer scans old (pre-Y2K) and new format ASCII MGD77 files for
errors using point-by-point sanity checking, along-track detection of
excessive slopes, and optional comparison of cruise data with global gravity
and predicted bathymetry grids. Detected data problems are output by default
as verbose descriptions of each detected error, often resulting in multiple
messages per scanned record. Data problems are optionally output (
-De
option) using a computer-parseable format (see E77 ERROR FORMAT description
below). Default error thresholds are derived from histograms of all MGD77
geophysical data collected between 1952 and January, 2006. Thresholds are
adjustable with the
-L option.
- NGDC-ids
- Can be one or more of five kinds of specifiers:
1) 8-character NGDC IDs, e.g., 01010083, JA010010etc., etc.
2) 2-character <agency> codes which will return all cruises from each
agency.
3) 4-character <agency><vessel> codes, which will return all
cruises from those vessels.
4) =<list>, where <list> is a table with NGDC IDs, one per line.
5) If nothing is specified we return all cruises in the data base.
(See mgd77info -L for agency and vessel codes). The
".mgd77" or ".nc" extensions will automatically be
appended, if needed (use -I to ignore certain file types). Cruise
files will be looked for first in the current directory and second in all
directories listed in $MGD77_HOME/mgd77_paths.txt [If
$MGD77_HOME is not set it will default to
$GMT_SHAREDIR/mgd77].
REQUIREMENTS¶
The mgd77sniffer links with Generic Mapping Tools 4.0 or later along with the
supplemental GMT packages x2sys and mgd77. See
http://gmt.soest.hawaii.edu for
GMT details. Grids for comparison with cruise data may be downloaded via the
web.
OPTIONS¶
- -A
- Apply scale factor and DC adjustment to specified data field. Allows
adjustment of cruise data prior to along-track analysis. CAUTION: data
must be thoroughly examined before applying these global data adjustments.
May not be used for multiple cruises.
- -C
- Set maximum ship speed in m/s, or knots with -N option. Ship speeds
exceeding 10 m/s (~20 knots) are flagged as excessive by default.
- -D
- Suppress default warning output and only dump cruise data row-by-row such
as values, gradients, grid-cruise differences, E77 error summaries for
each record, re-created MGD77 records or sniffer limits. Append r to
include all records (default omits records where navigation errors were
detected).
-Dd output differences between cruise and grid data. Requires
-G option. Output columns include:
lat lon dist cruiseZ gridZ diff [cruiseZ2 gridZ2 diff2 ...]
Note: grid values are subtracted from cruise data so a positive difference
implies cruise > grid. For multiple grid comparison, cruiseZ gridZ
diff are repeated for each grid comparison in command line order.
-De output E77 error classification format. Error output is divided
into (1) a header containing information globally applicable to the cruise
and (2) individual error records summarizing all errors encountered in
each cruise record. mgd77sniffer writes E77 directly to
<ngdc_id.e77> file handle. See E77 ERROR FORMAT below for
additional details.
-DE Same as -De but no regression tests will be carried out.
-Df output delta Z (change in geophysical field) column and delta S
(change in distance) for each geophysical field. Distance between
observations often differ for different fields depending on instrument
sampling rate, so ds is included for each geophysical observation. Output
columns include:
d[twt] ds d[depth] ds d[mtf1] ds d[mtf2] ds d[mag] ds d[diur] ds d[msd]
ds d[gobs] ds d[eot] ds d[faa] ds
-Dl display mgd77sniffer limits. Customize this output to create a
custom limits file for the -L option. No additional arguments are
required. Output columns include:
fieldabbrev min max maxSlope maxArea
-Dm output MGD77 format records in Y2K-compliant MGD77 format
-Dn output distance to coast for each record. Requires -Gnav
or -gnav option. Output columns include:
lat lon dist distToCoast
-Ds output calculated gradients for speed and geophysical fields.
Gradients correspond to the gradient type selected in the -S option
(spatial derivatives by default). Output columns include:
speed d[twt] d[depth] d[mtf1] d[mtf2] d[mag] d[diur] d[msd] d[gobs]
d[eot] d[faa]
See MGD77 FIELD INFO below for field and abbreviations descriptions.
-Dv display values for the twelve position and geophysical fields
for each MGD77 data record (in this order):
lat lon twt depth mtf1 mtf2 mag diur msens gobs eot faa
See below for MGD77 FIELD INFO.
- -g
- Compare cruise data to the specified grid in Sandwell/Smith Mercator
format. Requires a valid MGD77 field abbreviation (see MGD77 FIELD
INFO below) followed by a comma, the path (if not in current
directory) and grid filename, a scale to multiply the data (1 or 0.1), and
mode which stand for the following: (0) Img files with no constraint code,
returns data at all points, (1) Img file with constraints coded, return
data at all points, (2) Img file with constraints coded, return data only
at constrained points and NaN elsewhere, and (3) Img file with constraints
coded, return 1 at constraints and 0 elsewhere.
- -G
- Compare cruise data to the specified grid. Requires a valid MGD77 field
abbreviation (see MGD77 FIELD INFO below) followed by a comma, then
the path (if not in current directory) and grid filename. Multiple grid
comparison is supported by using separate -g or -G calls for
each grid. See GRID FILE INFO below.
Grid comparison activates several additional error checks. (1) Re-weighted
Least Squares Regression of ship versus grid data determines slope and DC
shift, which when differing from expected 1 and 0, respectively, may
indicate incorrectly scaled ship data, including incorrect units or
instrument drift as well as erroneous gravity tie-in. (2) Accumulated ship
grid offsets are computed along-track and excessive offsets are flagged
according to maxArea threshold (use -L option to adjust
maxArea). Warning: predicted bathymetry grids are constrained by
cruise data so grids and cruise data are not always independent.
Comparison of cruise bathymetry with predicted bathymetry grids also
activates a "navigation crossing over land" check.
- -H
- (with -G|g only) disable (or force) decimation during RLS
analysis of ship and gridded data. By default mgd77sniffer analyses both
the full and decimated data sets then reports RLS statistics for the
higher correlation regression.
-Hb analyze both (default), report better of two.
-Hd to disable data decimation (equivalent to -H with no argument).
-Hf to force data decimation.
- -I
- Append a field abbreviation and the first and last record in a range of
records that should be flagged as bad (and set to NaN prior to the
analysis). Repeat as many times as needed. May not be used for multiple
cruises.
- -K
- Reverse navigation quality flags (good to bad and vice versa). May be
necessary when a majority of navigation fixes are erroneously flagged bad,
which can happen when a cruise's first navigation fix is extremely
erroneous. Caution! This will affect sniffer output and should only be
attempted after careful manual navigation review.
- -L
- Override mgd77sniffer default error detection limits. Supply path and
filename to the custom limits file. Rows not beginning with a valid MGD77
field abbreviation are ignored. Field abbreviations are listed below in
exact form under MGD77 FIELD INFO. Multiple field limits may be modified
using one default file, one field per line. Field min, max, max slope and
max area may be changed for each field. Max slope pertains to the gradient
type selected using the -S option. Max area is used by the
-G option as the threshold for flagging excessive offsets from the
specified grid. Dump defaults -Dl to view syntax or to quickly
create an editable custom limits file.
Example custom default file contents (see below for units):
# abbrev min max maxSlope maxArea
twt 0 15 1 0
depth 0 11000 500 5000
mag -800 800 - -
faa -300 300 100 2500
Use a dash '-' to retain a default limit. Hint: to test your custom limits,
try: mgd77sniffer -Dl -L<yourlimitsfile>
- -N
- Use nautical units.
- -P
- Flag regression statistics that are outside the specified confidence
level. (i.e., -P5 flags coefficients m, b, rms, and r that fall outside
95%.)
- -Q
- Quick mode, use bilinear rather than bicubic interpolation [Default].
Alternatively, select the interpolation mode by adding b for
B-spline smoothing, c for bicubic interpolation, l for
bilinear interpolation or n for nearest-neighbor value. Optionally,
append threshold in the range [0,1]. This parameter controls how
close to nodes with NaN values the interpolation will go. E.g., a
threshold of 0.5 will interpolate about half way from a non-NaN to
a NaN node, whereas 0.1 will go about 90% of the way, etc. [Default is 1,
which means none of the (4 or 16) nearby nodes may be NaN]. -Q0
will just return the value of the nearest node instead of interpolating.
This is the same as using -Qn.
- -R
- west, east, south, and north specify the Region of interest,
and you may specify them in decimal degrees or in
[+-]dd:mm[:ss.xxx][W|E|S|N] format. Append r if lower left and
upper right map coordinates are given instead of w/e/s/n. The two
shorthands -Rg and -Rd stand for global domain (0/360 and
-180/+180 in longitude respectively, with -90/+90 in latitude).
Alternatively, specify the name of an existing grid file and the -R
settings (and grid spacing, if applicable) are copied from the grid.
- -S
- Specify gradient type for along-track excessive slope checking.
-Sd Calculate change in z values along track (dz). Output is given
in geophysical units, e.g., mGal.
-Ss Calculate spatial gradients (dz/ds). Output is given in
geophysical units per km along the survey track, e.g., mGal/km.
-St Calculate time gradients (dz/dt) [default]. Output is given in
geophysical units per second along the survey track, e.g., mGal/sec.
- -T
- Adjusts mgd77sniffer gap handling. By default, data gaps greater than 5 km
are skipped. Set to zero to de-activate gap skipping.
- -W
- Print out only certain warning types for verbose error messages. Comma
delimit any combination of c|g|o|s|t|v|x: where (c) type
code warnings, ( g)radient out of range, (o)ffsets from grid
(requires -G|g), (s)peed out of range, (t)ime
warnings, ( v)alue out of range, ( x) warning summaries. By
default ALL warning messages are printed.Not compatible with any -D
options.
- -V
- Selects verbose mode, which will send progress reports to stderr [Default
runs "silently"].
- -b
- output binary data for -D d|f|s|v option. Append s for
single and d for double precision [Default is double].
MGD77 FIELD INFO¶
Field Abbreviation Units
Two-way Travel Time twt sec
Corrected Depth depth m
Mag Total Field1 mtf1 nT
Mag Total Field2 mtf2 nT
Residual Magnetic mag nT
Diurnal Correction diur nT
Mag Sensor Depth/Alt msens m
Observed Gravity gobs mGal
Eotvos Correction eot mGal
Free Air Anomaly faa mGal
GRID FILE INFO¶
For
-g the grids must be in the format used by Sandwell & Smith,
which is a spherical Mercator 2-byte grid with no header. For
-G the
grid files can be of any grid type supported by GMT and therefore must contain
a GMT header. A correctly formatted *.i2 grid file can be generated using
grdraster as shown below.
gmtset GRIDFILE_SHORTHAND TRUE
Create/edit .gmt_io file to include the following rows:
# GMT I/O shorthand file
# suffix format_id scale offset NaN
grd 0 - - -
i2 2 - - 32767
grdraster 1 -R0/359:55/-90/90 -Getopo5_hdr.i2
The new grid, etopo5_hdr.i2 in this example, contains a GMT header and can be
used in the
-G option to compare cruise depth with grid values.
- Header
- Information pertaining to an entire cruise, such as NGDC and survey
institution identification codes, cruise examination time, two-way travel
time corrector information, data precision warnings, as well as systematic
scales, DC shifts and correlation coefficients from global grid
comparisons are reported as E77 header information.
Sample
# Cruise 08010039 ID 74010908 MGD77 FILE VERSION: 19801230 N_RECS: 3066
# Examined: Wed Oct 3 16:30:13 2007 by mtchandl
# Arguments: -De -Gdepth,/data/GRIDS/etopo5_hdr.i2
N Errata table verification status
# mgd77manage applies corrections if the errata table is verified (toggle
'N' above to 'Y' after review)
# For instructions on E77 format and usage, see
http://gmt.soest.hawaii.edu/mgd77/errata.php
# Verified by:
# Comments:
# Errata: Header
Y-E-08010039-H13-02: Invalid Magnetics Sampling Rate: (99) [ ]
Y-W-08010039-H13-10: Survey year (1975) outside magnetic reference field
IGRF 1965 time range (1965-1970)
Y-I-08010039-depth-00: RLS m: 1.00053 b: 0 rms: 127.851 r: 0.973422
significant: 1 decimation: 0
Y-W-08010039-twt-09: More recent bathymetry correction table available
Y-W-08010039-mtf1-10: Integer precision
Y-W-08010039-mag-10: Integer precision
- Error Record
- Individual error records have strict format. Included is a time or
distance column followed by record number, a formatted error code string,
and finally a verbose description of errors detected in the record. Three
error classes are encoded into the error code string with different
alphabetic characters representing unique error types. See below for error
code format description.
Format
<time/distance> <record number> <error code string>
<description>
Sample
# Errata: Data
Y 08010039 1975-05-10T22:16:05.88 74 C-0-0 NAV: excessive speed
- Error Code Description
- Each of the three error classes is separated by a dash '-' and described
by a combination of alphabetic characters or 0 signifying no detected
problems.
Error classes: NAV-VAL-GRAD
Error Class Descriptions
NAV (navigation):
0 - fine
A - time out of range
B - time decreasing
C - excessive speed
D - above sea level
E - lat undefined
F - lon undefined
VAL (value):
0 - fine
K - twt invalid
L - depth invalid
O - mtf1 invalid
etc.
GRAD (gradient):
0 - fine
K - d[twt] excessive
L - d[depth] excessive
O - d[mtf1] excessive
etc.
The NAV error class has unique cases while VAL and GRAD classes are
described by alphabetic characters for each of the 24 numeric fields in
MGD77 format order.
MGD77 bit-pattern w/ E77 alpha characters
|-------------------------------------------------|----------|
| X W V U T S R Q P O N M L K J I H G F E D C B A | E77 Code |
| - - - - - - - - - - - - - - - - - - - - - - - - | - - - - -|
| n f e g m d m m m m b b d t p l l m h d m y t d | F I |
| q a o o s i s a t t t c e w t o a i o a o e z r | i D |
| c a t b d u e g f f c c p t c n t n u y n a t | e |
| s r n 2 1 t r t r | l |
| s h h | d |
| - - - - - - - - - - - - - - - - - - - - - - - - | - - - - -|
| 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 | Bit place|
| - G C G C C - G G G - - G G - - - T T T T T - - | Bit type |
|-------------------------------------------------|----------|
Bit types: (G)eophysical, (C)orrection, (T)ime
EXAMPLES¶
To scan for excessive values or gradients, try
mgd77sniffer 08010001
To dump cruise gradients, try
mgd77sniffer 08010001 -Ds
To compare cruise depth with ETOPO5 bathymetry and gravity with Sandwell/Smith 2
min gravity version 11, try
mgd77sniffer 08010001
-G depth,/data/GRIDS/etopo5_hdr.i2
-g
faa,/data/GRIDS/grav.11.2.img,0.1,1
SEE ALSO¶
mgd77list(1),
mgd77track(1) x2sys_init(1)
REFERENCES¶
Wessel, P., and W. H. F. Smith, 2014, The Generic Mapping Tools (GMT) version
4.5.12 Technical Reference & Cookbook, SOEST/NOAA.
Wessel, P., and W. H. F. Smith, 1998, New, Improved Version of Generic Mapping
Tools Released, EOS Trans., AGU, 79 (47), p. 579.
Wessel, P., and W. H. F. Smith, 1995, New Version of the Generic Mapping Tools
Released, EOS Trans., AGU, 76 (33), p. 329.
Wessel, P., and W. H. F. Smith, 1995, New Version of the Generic Mapping Tools
Released,
http://www.agu.org/eos_elec/95154e.html, Copyright 1995 by the
American Geophysical Union.
Wessel, P., and W. H. F. Smith, 1991, Free Software Helps Map and Display Data,
EOS Trans., AGU, 72 (41), p. 441.
The Marine Geophysical Data Exchange Format - "MGD77", see
http://www.ngdc.noaa.gov/mgg/dat/geodas/docs/mgd77.txt