CTD corrections applied to delayed-mode data¶
This notebook performs analysis and correction of GPCTD data from the following C-PROOF glider deployment:
** Calvert Island Line: glider dfo-bb046** ************ * Deployment: dfo-bb046-20210324 * Sensor: CTD_0278 * Processing steps will be saved in: CTD_dfo-bb046-20210324.html * Processed by AH, IOS Data Products team, Ocean Sciences Division, Fisheries and Oceans Canada * Processing date: 20241211
1.0 Preamble¶
This document describes conductivity, temperature, and pressure data processing steps applied to delayed mode data collected using Sea-Bird Scientific Glider Payload Conductivity Temperature Depth (GPCTD) sensors mounted on C-PROOF Slocum and SeaExplorer autonomous ocean gliders. This sensor has a nominal sampling rate of 1 Hz and was designed specifically for Slocum gliders. This document covers the application of the sensor alignment correction and the thermal lag correction, as well as removal of questionable conductivity values and salinity profiles.
1.1 Set up the processing¶
The processing steps below are applied to delayed mode data stored in a single netcdf timeseries file created using the Pyglider data processing package (https://github.com/c-proof/pyglider).
The metadata and sensor calibration sheets are available via the deployment page on the C-PROOF website at: https://cproof.uvic.ca/gliderdata/deployments/dfo-bb046/dfo-bb046-20210324
1.2 Profile Check¶
Check that upcasts and downcasts are being properly identified. Negative values should be associated with upcasts.
Loading: e:/Glider/data//dfo-bb046-20210324_delayed.nc ************ * There are 301312 data points in total, with 808.0 profiles * Time period: 2021-03-25 to 2021-04-08 * Depth range: 0 - 704 metres ************ Identifying upcasts and downcasts. The left panel shows pressure vs. time and the right panel shows profile direction vs. time for a small subset of the time series:
1.3 Delayed-mode data prior to corrections¶
Checking fields (temperature, salinity, conductivity and density) in the delayed-mode data, before any CTD corrections:
2.0 Corrections applied to delayed mode data for this mission¶
Processing steps:
- Identification of anomalous conductivity values
- Identification of questionable salinity profiles
- Sensor alignment correction
- Thermal lag correction
2.1 Identification and removal of anomalous conductivity values¶
We identify and remove any conductivity values that are obviously unphysical, which is typically caused by air bubbles in the conductivity cell. We use a simple criterion applied to the raw conductivity data. The criterion temporarily flags any data points that are more than 5 standard deviations away from the overall time series mean for a given depth bin and profile bin, then recomputes the mean and standard deviation, excluding the temporarily flagged values. Conductivity values that still differ from the mean by more than 3 standard deviations are flagged as 'bad' and set to NaN in the time series. If the difference between the 'bad' values and the mean is less than the accuracy of the sensor, which is 0.0003 S/m for the GPCTD, then those points are not excluded.
This criterion is applied to data binned first by profile index, in increments of 50 profiles, then binned by depth, in increments of 5 m. The use of profile index bins rather than time or temperature bins is designed to allow for the removal of unphysical values in both Hakai Pass and crossing the continental shelf in Queen Charlotte Sound, two mission segments where the distributions of conductivity differ significantly.
Adjustments to this correction are based on examining the data and making a judgment call about which conductivity values are undeniably 'bad'. In this case, we want to exclude the extremely low values occurring at the surface consistent with air bubbles in the cell. Some unphysical values are missed by this correction, and may be caught during the removal of unphysical salinity profiles in further stepsbelow.
Note that for this mission:
************ The mode of the sampling rate for the GPCTD is one sample every 4 seconds. ************
Temperature vs. conductivity (left), depth vs. conductivity (middle), and conductivity vs. profile index (right), with the red dots showing the unphysical values flagged as bad and removed:
Salinity, temperature, conductivity and density shown, with conductivity outliers removed from the salinity, conductivity and density fields:
When plotting the fields shown above with the conductivity filter, we can see that the salinity and conductivity fields now have "speckles" showing data removed. This filtering is applied to remove values from the following fields:
- conductivity
- salinity (re-calculated)
- density (re-calculated)
The new field is called conductivityClean. The other fields mentioned were replaced.
2.2 Identifying questionable salinity profiles¶
Here, potentially suspicious salinity profiles are identified in order to prevent them from being used in the thermal lag correction. While these questionable salinity profiles are not included in the following steps, these profiles are not removed from the final corrected salinity product.
Profiles flagged as bad due to questionable salinity values: [ -0. 1. 2. 4. 9. 10. 11. 12. 13. 14. 15. 16. 17. 19. 20. 333. 335. 336. 397. 398. 399. 531. 793. 794. 795. 796. 797. 798. 801. 802. 803.]
Binned salinity plotted as a function of temperature (left) and vs. profile index (right), with the salinity profiles identified as bad due to questionable values and set to NaN shown in red and indicated by the red arrows at the top of the panel on the right:
Temperature vs. salinity diagram. Black contours give density in kg/m^3:
2.3 Sensor alignment correction¶
We now test application of a sensor alignment correction. In the literature this correction is often used to align the temperature and conductivity in time, relative to the pressure. This correction reduces the occurrence of salinity spikes near sharp gradients in T and S, and ensures calculations are made using the same parcel of water for all variables. The misalignment between the sensors is caused by:
- The physical separation between sensors causing a transit time delay for water being pumped through the CTD, and,
- Different sensor response times
Janzen and Creed (2011) provided constants used for the GPCTD sensor alignment correction "... based on the transit time between the T-C sensors, the temperature response time, and the estimated response time of the conductivity sensor in a 10 ml/s flow." For their Sea-Bird GPCTD alignment corrections, temperature was advanced by +0.5 s relative to pressure, and conductivity was advanced by +0.4 s relative to pressure, with a sampling rate of 1 Hz. In prior processing, offset values for conductivity and temperature were preferentially used from Janzen and Creed, 2011. However, their glider had differing geometry from the SeaExplorers and Slocums used by C-PROOF.
Instead, we directly estimate the sensor alignment correction constant for conductivity relative to temperature, $\tau_C$. We apply a preliminary correction using a linear time interpolation method, based on the one in the Sea-Bird CTD data processing manual and the Sea-Bird CTD-processing toolbox on Github (https://github.com/rejectedbanana/Sea-Bird-Toolbox/blob/master/CTD_PROCESSING/SBE_alignCTDW.m). We determine the correction using only water below the highly variable near-surface layer, which is approximately identified using the T-S diagram below. We choose as an isopycnal cutoff a value above the main thermocline, but which excludes that highly variable surface layer.
To directly estimate the sensor alignment correction constant following the methodology of Ferrari and Rudnick (2000):
- Assume conductivity is approximately linearly related to temperature
- Calculate the cross-spectrum between the standardized temperature and conductivity timeseries
- Calculate the coherence, to determine where the cross-spectrum is valid
- Fit the function $2 \pi f \tau_C$ to the phase of the cross-spectrum, where $f$ is frequency and $\tau_C$ is the correction constant we are looking for
These steps are applied to each individual profile, with fresh surface water less dense than a threshold excluded based on the T-S diagram. The cross-spectrum is estimated using Welch's method, with a Hann window of length 128 and 50% overlap. The spectrogram (below) shows only values for which the coherence is at least 75% and exceeds the 95% confidence level. The choice of 75% is arbitrary, and results are not sensitive to this choice within the range 50% to 90%.
Excluding data less dense than: 1023.0 Examining filename: ./dfo-bb046-20210324_conductivityClean.nc Applying density cutoff Total number of profiles: 808 Ready to loop over profiles ****** Spectrogram for T and C cross-spectrum magnitude (top), phase (middle), and squared coherence (bottom) as a function of frequency and profile index. Only points exceeding the 95% confidence level and with coherence above 75% are shown:
****** The value of tau_C is 0.04453 with standard error 0.02608. ****** Cross-spectrum phase of temperature and conductivity plotted as small gray points, with the mean plotted as large dots coloured by mean squared-coherence. The gray envelope shows the standard error for the mean over all profiles. The red line shows the function fit to points for which the frequency is less than the chosen cutoff:
Salinity before (top) and after (middle) shifting the conductivity by 0.044525736663971825. The difference is shown on the bottom:
Temperature vs. salinity diagram (left), where black contours give density in kg/m^3. On the right, the difference between the aligned and uncorrected temperature vs. the difference between the aligned and uncorrected salinity is shown:
We fit the function only to frequencies below 0.3 Hz (above). Above this frequency, we lose coherence and the slope of the cross-spectrum phase changes sign. We see that the resulting value for the time constant $\tau_C$ is well within the required $\pm 0.5$ s of the value specified for Sea-Bird GPCTDs, $\tau_C = -0.1$ s.
Previously, Janzen and Creed (2011) constants would be used instead of our directly estimated value for $\tau_C$, since we have found that the direct estimate is sensitive to the choices of frequency and density cutoff used, with reasonable choices producing constants that can vary by up to $\pm 0.5$ s. However, as we see here, the conductivity and thus salinity changes are very small, leading us to decide not to apply sensor alignment adjustment to this glider mission.
2.4 Thermal lag correction¶
The thermal lag effect is caused by the thermal inertia of the conductivity cell affecting the temperature of the water as it passes through the cell. To determine the thermal lag correction, the temperature inside the conductivity cell is estimated, then salinity is recalculated using the estimated temperature and the measured conductivity. To estimate the temperature, a recursive filter is applied to the temperature field with parameters 𝛼 (the amplitude of the error), and 𝜏 (the time constant for the thermal lag). Two methods for this are mentioned below.
Sea-Bird GPCTDs are pumped with a constant flow rate. As such, we expect the thermal lag to be approximately constant over the full mission, and it is sufficient to find a single value of 𝛼 and 𝜏 for the entire mission. It is ideal to use profile pairs from regions with large temperature gradients, but small conductivity gradients, when comparing up- and down-casts.
Janzen and Creed (2011) determined a cell thermal mass correction for the GPCTD using data from a prototype CTD that sampled twice as rapidly as the GPCTD nominally samples, with a pumped flow rate of 10 ml/s. They found 𝛼 = 0.06 and 𝜏 = 10s. These values are considered when retrieving $\alpha$ and $\tau$ to see how much the results differ.
This mission on the Calvert Line occurred in a highly energetic environment, so near-surface differences between a downcast and the subsequent upcast are likely to be caused by spatiotemporal variability. As such, we exclude segments of each profile in the upper water column for which the density is $<$1023 kg/m$^3$ from the minimization routine.
Considerations for using Janzen and Creed values:¶
Prior processing used Janzen and Creed (2011) values $\alpha$ and $\tau$ for the thermal lag. For each mission, the thermal lag parameters were directly estimated. The steps are outlined below, but can be found in much greater detail here: https://cproof.uvic.ca/gliderdata/deployments/reports/C-PROOF_SBE_CTDProcessingReport_v0.2.pdf
- It was confirmed that the directly estimated value of $\tau$ was within $\pm$10s of the Janzen and Creed (2011) value of 10s
- The improvement with the directly estimated, as well as Janzen and Creed, parameters was quantified.
- The thermal lag correction was applied with the parameters that resulted in the greater improvement, and did not result in an over-correction.
- If a given sensor had a directly estimated value of $\tau$ that is significantly higher or lower than 10s, investigate further
Note that the thermal lag correction parameters are more likely to vary slightly between individual GPCTD sensors than the alignment correction constants.
With this method, the recursive filter seeks to minimize the root-mean squared difference (RMSD), which is calculated as the square root of the sum of the squared areas between pairs of salinity profiles (binned by temperature), normalized by the number of pairs of profiles. The values of 𝛼 and 𝜏 that minimize the area between pairs of profiles (each dive and subsequent climb along the glider path) were determined using a brute force minimization scheme. This method also uses a subset of the remaining data, consisting of 100 pairs of profiles equally spaced in time, to determine the correction.
While this was employed during previous processing, here we are preferring to directly estimate the parameters using a mission-specific subset of profiles. However, the results are still compared to Janzen and Creed values.
Updated thermal lag correction procedure:¶
Same to the above, this is based on Morison et al's (2011) second method which derives a modified temperature that is the "best guess" for what the temperature is in the conductivity cell based on the temperature observed by the thermistor. The temperature is corrected using lfilter
which is just a recursive filter:
$T_T(n) = -b T_T(n-1) + aT(n) - a T(n-1)$
where
$$ a = \frac{4f_n\alpha \tau}{1+4f_n\tau} $$and $$ b = 1 - \frac{2a}{\alpha}$$
$\tau$ can be thought of as the time constant of the thermal lag (in seconds) and $\alpha$ as its strength. Following Gaurau et al, $f_n$ is the sampling frequency. The cell temperature is then"
$$T_c(n) = T(n) - T_T(n)$$and can be used to calculate salinity with the measured conductivity and pressure.
2.4.1 Pre-processing steps:¶
We exlude profiles from the correction for which the area between subsequent downcasts is more than one standard deviation from the mission mean. This ensures that no data crossing fronts or intrusions is included in the correction, in line with the key assumption that a downcast and the subsequent upcast be identical.
Furthermore, we impose a cutoff for the area between pairs of profiles that will be included in the subset used to estimate the parameters. Any pair of profiles whose area is more than 3 standard deviations away from the mean will be excluded from the determination of the RMSD. This ensures that a small number of anomalous profiles do not bias the results.
During this step, the suspicious salinity profiles identified earlier are excluded as well.
Loading filename ./dfo-bb046-20210324_conductivityClean.nc Total number of profiles: 808
RMSD for uncorrected data = 0.02752312912643408
Restricting profiles Max and min area between downcasts = 2.4002970773785597 1.7109212405424509e-06 List of profiles to exclude: [357. 358. 361. 362. 372. 373. 376. 377. 382. 383. 384. 385. 386. 387. 390. 391. 406. 407. 424. 425. 426. 427.]
Red indicates profile pairs that were identified in this process, where the area between profile pairs was considered to be too large, and so are not included in the thermal lag correction. White bands indicate salinity profiles removed during step 2.2.
2.4.2 Defining the range to calculate $\alpha$ and $\tau$:¶
From examining the asymmetry in up-down profiles, we manually choose a range to apply the thermal lag correction to. It is ideal to pick areas with high temperature gradients in the water column, but generally low salinity gradients.
Density cutoff: 1023 Uncorrected salinity (top), shown with suspicious profiles and low-density data points removed. Salinity differences between sequential profiles, showing asymmetry between upcasts and downcasts:
Zooming in along the glider deployment to visualize up-down asymmetry:
Ranges that appear to fulfill this requirement include:
- 100-160
- 475,530
- 600,660
Using profile limits [475, 520] for tau and alpha calculation
Comparing the error measurements between estimated alpha & tau and Janzen and Creed values:¶
Below shows the subset of profiles, limited to 200 m depth, without any thermal lag correction, and using literature values (Janzen and Creed 2011). The Janzen and Creed $\alpha$ and $\tau$ values visibly reduce the error in the water column, but better results can be retrieved by calculating our own.
Finding alpha and tau values with lowerest error estimates:¶
By scanning over a range of alpha and tau values, error minima can be retrieved. We can see that, below, from scanning for errors, there is a consistent band with lower error. We'll select the cell with lowest error and other minima nearby.
The range of alpha and tau values tested, with log10-transformed errors coloured. The dark blue indicates the alpha and tau with lowest errors:
Applying determined alpha and tau for futher validation¶
The following plots show the application of these tau and alpha values to the beginning, end, and entire glider mission. The Janzen and Creed error statistics, as well as errors from no correction applied, are shown as well.
Applied to the end of the mission:¶
****** Applying alpha = 0.075 and tau = 18 *****
Applied to the beginning of the mission:¶
Middle of the mission¶
Finally, applied to the overall mission:¶
***** Saving with alpha = 0.075 and tau = 18 applied *****
<xarray.Dataset> Dimensions: (time: 301312) Coordinates: * time (time) datetime64[ns] 2021-03-25T16:56:31.95000012... latitude (time) float64 ... longitude (time) float64 ... depth (time) float64 ... Data variables: (12/19) heading (time) float64 ... pitch (time) float64 ... roll (time) float64 ... conductivity (time) float64 ... temperature (time) float64 ... pressure (time) float64 ... ... ... profile_direction (time) float64 ... salinity (time) float64 ... potential_density (time) float64 ... density (time) float64 ... potential_temperature (time) float64 ... conductivityClean (time) float64 ... Attributes: (12/67) Conventions: CF-1.6 Metadata_Conventions: CF-1.6, Unidata Dataset Discovery v1.0 acknowledgement: Funding from Fisheries and Oceans Canada, Cana... cdm_data_type: Trajectory comment: Calvert Island deployment contributor_name: James Pegg, Jody Klymak, Tetjana Ross, Jennife... ... ... summary: Glider deployed near Calvert Island BC as part... time_coverage_end: 2021-04-08T15:43:55.704000000 time_coverage_start: 2021-03-25T16:56:31.950000000 title: dfo-bb046-20210325T1656 transmission_system: IRIDIUM wmo_id: 4803918
<xarray.Dataset> Dimensions: (time: 301312) Coordinates: * time (time) datetime64[ns] 2021-03-25T16:56:31.95000012... latitude (time) float64 ... longitude (time) float64 ... depth (time) float64 ... Data variables: (12/22) heading (time) float64 ... pitch (time) float64 ... roll (time) float64 ... conductivity (time) float64 nan nan nan nan ... nan nan nan 3.092 temperature (time) float64 6.912 7.033 6.898 ... 7.235 7.266 7.25 pressure (time) float64 -0.13 -0.03 0.19 ... 3.49 2.77 2.04 ... ... density (time) float64 ... potential_temperature (time) float64 ... conductivityClean (time) float64 ... salinity_corrected (time) float64 nan nan nan nan ... nan nan nan 30.01 temperature_adjusted (time) float64 6.423 6.589 6.514 ... 7.266 7.251 density_adjusted (time) float64 nan nan nan nan ... nan nan 1.023e+03 Attributes: (12/67) Conventions: CF-1.6 Metadata_Conventions: CF-1.6, Unidata Dataset Discovery v1.0 acknowledgement: Funding from Fisheries and Oceans Canada, Cana... cdm_data_type: Trajectory comment: Calvert Island deployment contributor_name: James Pegg, Jody Klymak, Tetjana Ross, Jennife... ... ... summary: Glider deployed near Calvert Island BC as part... time_coverage_end: 2021-04-08T15:43:55.704000000 time_coverage_start: 2021-03-25T16:56:31.950000000 title: dfo-bb046-20210325T1656 transmission_system: IRIDIUM wmo_id: 4803918
Before and after plots of the thermal lag correction¶
The following plots show the adjusted salinity, temperature, and density, respectively. The insets, right, correspond to the red box, delineating the profile range used to calculate the alpha and tau values. Further, unadjusted salinity, temperature, and density outliers, identified during the pre-processing steps, which were removed for the tau and alpha calculation to not skew results, are shaded in a lighter colour.
These fields, adjusted and re-calculated using the new alpha and tau values, are saved in the output file as fields salinity, temperature and density. The original delayed-mode, uncorrected fields are saved as salinity0, temperature0 and density0.
Density cutoff: 1023
Salinity before (top) and after adjustment (middle), with salinity difference (bottom). Profiles and data points that were ignored for thermal lag correction are shaded in the top plot. The profile range used to determine the thermal lag correction is shown on the right:
Temperature before (top) and after adjustment (middle), with temperature difference (bottom). Profiles and data points that were ignored for thermal lag correction are shaded in the top plot. The profile range used to determine the thermal lag correction is shown on the right:
Density before (top) and after adjustment (middle), with density difference (bottom). Profiles and data points that were ignored for thermal lag correction are shaded in the top plot. The profile range used to determine the thermal lag correction is shown on the right:
2.4.3 Quantifying improvement¶
To quantify the improvement of the thermal lag correction further, the area of profile pairs was re-calculated and compared to the area between profiles prior to correction. This is calculated from the same quality-controlled data, with suspicious profiles and profile-pairs removed, as during the thermal lag correction, but using evenly spaced profile pairs across the deployment.
When examining the corrected data (shown in orange) relative to the uncorrected data (shown in black), we can see that up-down asymmetry was reduced, as well as a greater reduction compared to using Janzen and Creed constants (shown in red). Overall, we can see a large reduction in the area between profiles when using the calculated alpha and tau values. When looking at evenly spaced profiles across the deployment, some pairs have much larger area (e.g. in the middle of the deployment), whereas others remained quite low. When comparing upcasts and downcasts before and after correction in a T-S plot, we also see that they are much better aligned.
Total number of profiles: 808
Total area mean before correction = 0.027365326882586 Total area mean after correction = 0.024069943586583535 Total area anomaly mean = -0.003295383296002462 Total area anomaly median = -0.004758825451055063 ***** Top: The area between 100 profile pairs when uncorrected (black), using Janzen and Creed alpha and tau values (red) and when corrected with alpha = 75.0 and tau = 18, shown by profile index and as a histogram. Bottom: the area change between profile pairs when corrected with calculated alpha and tau. Shown by profile index (left) and histogram (right):
Temperature-salinity diagrams for all profiles, showing the difference between downcasts (blue) and uncorrected data (black) on the left, and upcasts (red) and uncorrected data (black) on the right panel: ***** alpha = 0.075, tau = 18
Temperature-salinity diagrams for all profiles, showing the difference between upcasts (red) and downcasts (blue), for the data without the thermal lag correction applied (left panel) and the data with the thermal lag correction applied (right panel):
3.0 Summary of corrections applied to delayed mode data for this mission¶
Identification of anomalous conductivity values:
- Anomalous conductivity values at the surface caused by air bubbles in the cell were set to NaN.
Sensor alignment correction:
- No sensor alignment correction was applied.
Identification of questionable salinity profiles:
- Numerous salinity profiles were flagged as 'bad' and their values set to NaN for the thermal lag correction. The range of profiles examined was not limited, and these profiles were not removed from the final corrected salinity dataset.
Thermal lag correction:
- The directly determined values for the thermal lag correction produced an improvement that was larger than the recommended values from Janzen and Creed (2011).
- The correction overall significantly reduced the root-mean squared difference for the area between between pairs of profiles.
- The final thermal lag correction was applied using the calculated values of:
alpha = 0.075 and tau = 18
The corrected temperature and salinity fields shown in a T-S diagram with density contours:
'e:/Glider/data//dfo-bb046/dfo-bb046-20210324/dfo-bb046-20210324_grid_adjusted.nc'
The corrected salinity and temperature, shown with filtered conductivity and adjusted density:
References¶
Ferrari, R., and Rudnick, D. L. Thermohaline variability in the upper ocean, J. Geophys. Res., 105(C7), 16857-16883, 2000.
Garau, B., Ruiz, S., Zhang, W. G., Pascual, A., Heslop, E., Kerfoot, J., & Tintoré, J. Thermal Lag Correction on Slocum CTD Glider Data, J. Atmos. Oceanic Technol., 28(9), 1065-1071, 2011.
Janzen, C. D., and Creed, E. L. Physical oceanographic data from Seaglider trials in stratified coastal waters using a new pumped payload CTD, OCEANS'11 MTS/IEEE KONA, Waikoloa, HI, USA, 1-7, 2011.
Morison, J., Andersen, R., Larson, N., D’Asaro, E., & Boyd, T. The correction for thermal-lag effects in Sea-Bird CTD data, J. Atmos. Oceanic Technol., 11, 1151-1164, 1994.
Sea-Bird Seasoft V2:SBE Data Processing - CTD Data Processing & Plotting Software for Windows, Sea-Bird Scientific, software manual revision 7.26.8, 2017.
Sea-Bird User Manual - GPCTD Glider Payload CTD (optional DO) - Conductivity, Temperature, and Pressure (optional DO) Sensor with RS-232 Interface, Sea-Bird Scientific, manual version 008, 2021.