Welcome to the Data Distribution Working Group (DDWG)


NOTE: This website is being deprecated in favor of the DDWG github repo(external link).

This group keeps tabs on issues arising from the distribution of SuperDARN data per the PI agreement. Of note, this group deals with collecting data from all of the radars in the SuperDARN network and making them available to the SuperDARN PI institutions. Historically, the distribution to the institutions was done by shipping hard drives containing several months or more of data. However, as of the start of 2014 this distribution is being performed through a data mirror located at the University of Saskatchewan. Several ways to download data and to keep sync'ed with this data mirror have been developed using sftp as well as rsync. Notes on these methods can be found below.

In addition to the distribution of files, this working group takes on the task of quality control for SuperDARN data files ensuring that accurate files are available for distribution. This group deals with erroneous files and works to correct the errors in the files either within this working group or with other working groups. Lastly, files that cannot be corrected are recommended for removal after discussion within this group. A record of data issues, known data problems, and corrected data are provided below.

What's New


Usask globus and BAS mirrors operational The new globus mirror at the University of Saskatchewan is up and running, as discussed at the 2017 workshop in Italy. As well, the BAS mirror is operational and
has an API (see documentation here: https://docs.api.bas.ac.uk/superdarn/mirror/v3/)(external link)

We encourage users to contact Kevin Krieger or Paul Breen to have access to these mirrors.

At the University of Saskatchewan the temporary mirror with only the latest SuperDARN data is still operational and supported for the short term.


<<


Loosely tied to this working group out of discussions in Issue #2, some updates/enhancements to the hdw.dat repository have come out of recent discussions about how new files could be sent around. The github repository is now setup with hdw.dat releases, which are all of the files in once package. As well, whenever a file is updated, a new release will be made via an automatic script running at Virginia Tech. The release page can be found at:

https://github.com/vtsuperdarn/hdw.dat/releases(external link)


Current topics of discussion by Issue number (for more detail see Tasks & Issues):
Policy for dealing with problem files
(#2) 'Fixrawacf' string found in many rawacf files.
(#12) Blacklisted lyr files prior to public release
(#13) Blacklisted Hokkaido East files with no data in January 2017

Basic Information


The Data Distribution Working Group (DDWG) was formed in the Fall of 2013 after the creation of a data mirror at the University of Saskatchewan and the hand over from JHU/APL to Virginia Tech of the responsibility of collecting data from all of the radars. The charter can be found at:

DDWG Charter

The current membership of the DDWG as of May 1, 2014 is:

DDWG Membership

The data distribution mailing list along with other SuperDARN mailing lists are operated by the University of Leicester. An archive of the ddwg mailing list can be found at:

http://lists.le.ac.uk/pipermail/darn-ddwg/(external link)

Data Mirrors


As a result of PI discussions during the SuperDARN 2014 Workshop at Svalbard, the distribution of data files is now being done electronically via data mirrors located at the University of Saskatchewan (USask) and the British Antarctic Survey (BAS). The routine mailing of data on hard drives to PI institutions was discontinued as of December 31, 2013. The new method facilitates rapid delivery of files and the dissemination of corrected files. The USask globus mirror is accessible via the globus.org software and interface. The USask temporary mirror accepts rsync, scp and sftp
connections as well as the BAS mirror. In addition to allowing rsync connections, a configuration file has been added to allow for more clever coding/scripting with the data mirror. Consult with Kevin Krieger (USask) or Paul Breen (BAS) for more information.

Additionally, following the SuperDARN 2015 workshop, the responsibility of collecting data files would be divided between three hubs: USask, British Antarctic Survey (BAS), and a hub yet to be determined in Japan.

At the start of 2016, the BAS data mirror came on line. This mirror has been undergoing testing during much of 2015 and is now transitioning to assume the responsibility for collecting files for a subset of the radars. Here the BAS mirror will collect data for radars allocated to BAS in addition to radars allocated for the Japanese data mirror. The radars are allocated to each mirror as noted in the table below.
Click here for a summary of data distribution and data mirrors as of June 29, 2020.

USask ADE ADW BKS CVE
CVW CLY FHE FHW
GBR INV KAP KOD
PGR RKN SAS WAL
MCM SPS
BAS HAN PYK STO DCE
FIR HAL KER SAN
Japan HOK HKW KSR BPK
SYE SYS TIG UNW
ZHO



Supplemental Material


These documents should get you up to speed on data distribution working areas.

Tasks & Issues


Listing of major tasks & issues. These can branch off into blogs or their own page so that we can keep a running tally of what is going on. Of course, back build this so that the latest issue is on top.

Open Issues


Issue #38: Clyde River borealis files removed
Issue #37: Corrupt rkn file
Issue #36: BPK File time error
Issue #35
Issue #34
Issue #33
Issue #32 - Empty/skipped?
Issue #31
Issue #30
Issue #29
Issue #28
Issue #27: Fwd: LYR data, new versions...
Issue #26: 526 Corrupt rawacf files
Issue #25
Issue #24
Issue #23: Corrupted Stokkseyri rawacf file
Issue #22
Issue #21
Issue #20
Issue #19
Issue #18
Issue #17
Issue #16
Issue #15
Issue #14
Issue #13: blacklisted hok files with no data in January 2017
Issue #12: lyr files prior to public release
Issue #11: Incorrectly labelled bpk/hal data
Issue #10: Updated PGR data with corrected frang and lagfr
Issue #9: Empty/blacklisted 2013 han, kod files
Issue #2 Hard Disks and differences in rawacf files REOPENED

Closed Issues


Issue #8 Missing files added to mirror CLOSED
Issue #7 Duplicate named unw files CLOSED
Issue #6 Duplicate dat files CLOSED
Issue #5 High Velocity ksr data - 201110 CLOSED
Issue #4 Break-in Period Christmas Valley radars CLOSED
Issue #3 Corrected Canadian Data RECLOSED
Issue #1 Incorrectly named multi-channel files: CLOSED


Known Data Problems


Sorted by reverse time:


19960101 - present: hal These files have xcfd, or elevation angle data even there is currently no interferometer array at the Halley radar. From BAS: The previous engineer/PI (Mike Pinnock) did some experiments with 4 whip antenna and reflectors c. 2000 for ~ 12 months but I am not sure what the outcome of these findings were. Last activity: The identification that there isn't an interferometer array at Halley except for the period above was the last of it. Here steps should be taken to identify the period around 2000 that the interferometer experiments were conducted.

20181219.04: hok Nozomu Nishitanit noted that the data for this file is contained within the previous 2 hour file. It's possible this file can be split into 2 files. Last Activity: File was identified during Fall 2019 data gap checks.

20181217.10: hok Nozomu Nishitanit noted that the data for this file is contained within the previous 2 hour file. It's possible this file can be split into 2 files. Last Activity: File was identified during Fall 2019 data gap checks.

20181214.14: hok Nozomu Nishitanit noted that the data for this file is contained within the previous 2 hour file. It's possible this file can be split into 2 files. Last Activity: File was identified during Fall 2019 data gap checks.

20181213.06, *.18: hok Nozomu Nishitanit noted that the data for this file is contained within the previous 2 hour file. It's possible this file can be split into 2 files. Last Activity: File was identified during Fall 2019 data gap checks.

20181212.06, *.18: hok Nozomu Nishitanit noted that the data for this file is contained within the previous 2 hour file. It's possible this file can be split into 2 files. Last Activity: File was identified during Fall 2019 data gap checks.

20181209.04, *.08, *.20: hok Nozomu Nishitanit noted that the data for this file is contained within the previous 2 hour file. It's possible this file can be split into 2 files. Last Activity: File was identified during Fall 2019 data gap checks.

20181208.22: hok Nozomu Nishitanit noted that the data for this file is contained within the previous 2 hour file. It's possible this file can be split into 2 files. Last Activity: File was identified during Fall 2019 data gap checks.

20181207.1800: hok Nozomu Nishitanit noted that the data for this file is contained within the previous 2 hour file: 20181207.1602.00.hok.rawacf*. It's possible this file can be split into 2 files. Last Activity: File was identified during Fall 2019 data gap checks.

20181205.1800: hok Nozomu Nishitanit noted that the data for this file is contained within the previous 2 hour file: 20181205.1801.06.hok.rawacf*. It's possible this file can be split into 2 files. Last Activity: File was identified during Fall 2019 data gap checks.

20181203.00 and 20181203.02: hok Nozomu Nishitanit noted that the data for these files are contained within the previous 2 hour file: 20181202.2201.05.hok.rawacf*. It's possible this file can be split into 2 files. Last Activity: File was identified during Fall 2019 data gap checks.

20181130.1800: hok Nozomu Nishitanit noted that the data for this file is contained within the previous 2 hour file: 20181130.1601.19.hok.rawacf*. It's possible this file can be split into 2 files. Last Activity: File was identified during Fall 2019 data gap checks.

20181023.1400, 20181023.2200: hok Nozomu Nishitanit noted that the data for this file is contained within the previous 2 hour files: 20181023.1201.00.hok.rawacf*, 20181023.2009.00.hok.rawacf*. It's possible this file can be split into 2 files. Last Activity: File was identified during Fall 2019 data gap checks.

20181013.0400: hok Nozomu Nishitanit noted that the data for this file is contained within the previous 2 hour file: 20181013.0201.00.hok.rawacf*. It's possible this file can be split into 2 files. Last Activity: File was identified during Fall 2019 data gap checks.

20160202.02: sto This file appears to have had a byte corruption or some kind of error that is not allowing the rawacf file to be processed correctly. Last Activity: For now this file has been blacklisted per Issue #23.

20160131.20 - 20160201.18: tig It appears that files during this time in most of the beams contain very high velocity returns from some of the farther range gates. Last activity: Put these to the DDWG list to see if they need to be removed from distribution.

20151020.12 - 20151022.1206: bks During this time the clock of the main QNX computer was off by an estimated 30 minutes in the past. So at 1200, the time was showing about 1130. The clock on the computer was reset at 1206 on the 22nd and the radar started up at this time. Last activity: Identified the possible times this could be off, need to reset or adjust the times of the data?

20150416.12 - 20150417.08: hal During this time, there was a control issue at the radar. These files have been found to process with the RST 2.xx, but not the VTRST 3.5. Last activity: These files were removed from the VT archives and backed up. The DDWG was notified of this issue and proposed testing on other places to see if these files need to be moved off of the USask data mirror. Several attempts were trying to see where in the make_fit code that things were breaking down, but no exact point could be identified. The last memory is that a memory allocation was being corrupted.

20150216.0554 - 20150328.0908: hkw From Nozomu: During this time the transmitting signal input to the phasing matrix was interchanged between channel A and B AND the radar operated with different beams between channel A and B. For these files data should be handled with care when there are echoes along the camping beam (bm5) direction (mixing of two beams will happen). When there are no echoes along the camping beam the data are much less problematic.

20141008.1201: tig This file was damaged in the transfer from Bruny Island to La Trobe University and produces a segmentation (seg) fault whenever trying to process with several different versions of fitacf. From Keith K.: the process seg faults at line 1018 of dmap.c. Something about this particular file is causing the buffer offset to increment far outside its memory size and then the program seg faults when it tries to dereference the address at that offset. Last activity: The file was first identified by Tomo Hori and confirmed no replacement can be found at the radar by Andrew McDonald. This file has been moved out of circulation in the VT archives and backed up.

20140527.0600: tig This file contains a data record at 06:52.28 that appears to have some errors in the dmapfile. With a dmapdump command, the record header's time.yr was found to be noted at time.xr. There are also other errors with this record that do not appear to be correct. Last activity: The file was identified by Evan Thomas at VT and confirmed by Kevin Sterne. The file in the VT archives was checked against the file on the La Trobe server and it was found that the two files are the same.

20130921.06 - 20130921.12: kap During this time it appears as though two different control programs tried to run at the same time. It is unsure how this happened, but the files during these times reflect two control programs recording data. Last activity: The issue was originally pointed out by Evan Thomas and then investigated by Kevin Sterne. Unsure of how these files could be fixed up and so they may need to be removed if they cause analysis problems.

20130912.0600: tig This file contains data from many more hours than just the two hour block. This file needs to be separated into two hour files. The file currently exists in the VT archives.

20130814.0735.46 - 20130909.2200: san Data files during this period were set with a computer that was set to the wrong time zone. As a consequence, the files here are one hour ahead of when they actually should be. Several of the 2200 files may also be cut off due to the way download scripts were running. Last activity: these files were identified, but no adjustment in time taken. Judy Stephenson also noted that the radar was running on only 4 transmitters, so the data may be too weak to be worth anything.

20130712.00 - 20130712.20: kod.d Data from this timeframe appears to be missing, but is actually contained in a larger than usual file that has a time/radcode stamp of: 20130711.2342.24.kod.d. Last activity: This file was idenitified by Jon Klein during the last data gaps request in March 2013.

20120321 - 20121227 : tig Rawacf files during this time randomly have erroneous values in the ltab array. These erroneous values are typically -23600. Not every file during this time contains these erroneous values, but for example, 20120326.2201.00.tig.rawacf has a bad lag table in the very first record of the file. Last activity: Need to search for exact files that have this error to identify for correction or removal from the data archives.

20111219(?) - 20130719(?) : cve, cvw Data during this time includes elevation angle data that does not make sense due to hardware and/or software issues. Last activity: The issue was identified however definitely dates of when bad elevation angle data started and ended being collected have not been identified. The dates here are a quick guess from quickly looking at RTI plots. Have asked Simon Shepherd for beginning and end dates.

20111014.0000 - 20111025.2110.13: ksr Data recorded here have incorrect velocities, all between -1000 m/s and -3500 m/s. Tsutomu-san explains that the problem was a bad PTS clock frequency. This was fixed temporarily by modifying the software on Oct. 25 and totally fixed with the next site visit (summer 2012?). Last activity: These files have been removed from the VT data archive; the grid and map files for these dates have been reprocessed here without the erroneous data. As well, this information was sent around to the DDWG for review and removal. This error is discussed in issue #5.

20101220.1135 - 20101221.0801: han Data recorded in the rawacf files has a smsep of zero as well as having other parts of data arrays that are bad. Last activity: Need to identify which files contain smsep = 0 and recommend that these files should be removed from the archive.

20061202.0000 - 20071119.0000: hok During this time, all of the phasing cards were operating, however the interferometer data is useless. These files can be distributed with the condition that the interferometer data should not be used.

Before 20061202.0000: hok Data taken during this time was purely at a testing phase. At this time, 3 of the 8 phasing matrix cards were not operating. This data should not be distributed.




Corrected Files


Sorted by reverse time:

20150313.1800 - 20150723.0300: pgr Data originally distributed during this period for Prince George had an incorrect receiver delay setting. The data has been since corrected in issue #10.

20120911.0000 - 20121007.1801: ade, adw Data originally distributed during this period for Adak East and West had the channel information lost when being processed by JHU/APL. This has been corrected in issue #1

20120110.0000 - 20121015.2201: mcm Data originally distributed during this period for McMurdo had the channel information lost when being processed by JHU/APL. This has been corrected in issue #1

20120201.0000 - 20120331.22: sas Data originally distributed during this perdiod for the Saskatoon radar had an incorrect receiver delay. This has been corrected in issue #3

20110718.2221 - 20120831.2201: kod Data originally distributed during this period for Kodiak had the channel information lost when being processed by JHU/APL. This has been corrected in issue #1

20100930.00 - 20110831.22: inv, pgr, rkn, sas Data originally distributed during this period for the Inuvik, Prince George, Ranken Inlet and Saskatoon radars had an incorrect receiver delay. This had been corrected in issue #3

Halley files data issue


Hello DDWG,

I know I have some loose ends to look into this, but it's becoming too outstanding without taking some kind of action. I've also talked this through with some of the folks at BAS, so it shouldn't be too much of a shock.

It seems as though from the period of 20150416.12 to 20150417.08, the Halley radar (hal) rawacf files have a problem with them. The more troubling piece here is that older versions of the RST (2.x I think) appear to not have a problem with these files. However, I've had a problem with trying to process these files here at Virginia Tech with the VTRST 3.5 code.

The long story is that I looked into it a while ago and it appeared as though some kind of memory allocation was being corrupted and causing things to crash. If anyone is interested more, e-mail me directly and I can get back to my notes on when I was working on this a few months ago.

For now, I have moved these from the usual VT archive and backed them up so that I can continue to investigate why RST 2.x processes these files, but VTRST 3.5 does not.

Does anyone else have these issues with these files? If so, maybe we should look to putting them in a separate place on the USask mirror as well.

Cheers,

-Kevin S.

Print