On the morning of Jan. 25th, it was observed on VT’s tenplot, a sample of which is shown here, that the velocity sign of the Blackstone radar was incorrect. The ionospheric returns from the pre-noon data that appeared on the “tenplot” on SuperDARN @ Virginia Tech’s website showed that the returns had the same velocity sign between Fort Hays East, Blackstone, and Wallops Island. This is incorrect as Blackstone faces westward and Fort Hays East and Wallops Island face eastward and thus all three radars should not have the same velocity sign from returns occurring at the same time.
The issue was brought up immediately and e-mailed around to Mike Ruohoniemi, Ray Greenwald and Rob Barnes. Rob responded to the e-mail noting that the hdw.dat.bks file was the proper place to fix this error. However, Mike and Ray both responded that they had not found an issue when watching the same returns on the real-time display. Any rash action to changing the hdw.dat.bks file anywhere was then stopped until clearer thinking and understanding of how data is processed and plotted could be written.
The issue with the real-time display displaying expected data and the tenplot displaying unexpected data was tracked down to an inconsistency in the hdw.dat.bks file for the real-time display. However, the inconsistency was still not changed immediately in order to let issues be thought out better. One such issue was how the tenplot was being created. Questions began arising on how the velocity sign from the hdw.dat file was being used in data processing and in data visualization. The following figure shows how the data processing is done between the Radar Operating Software and the data storage directories on sd-data:
Of note the figure shows that the velocity sign is applied when creating fitacf or fitex files from rawacf files. From these files, the DaViT software and real-time display let someone visualize the data in the following ways:
However, before the data processing and plotting was sorted out, Ray believed that the I & Q data was being opposite of what it should have been. This is to say that the I channel was being written with Q data and the Q channel was being written with I data. This reversal would effectively reverse the velocity sign apparent in the data. The order in which the two data sets are written is controlled in the hdw.c file located in:
~/usr/code/src.lib/radarqnx4/site/site.2.00/src/
In this file two define statements are located towards the top of the file:
- define REAL_BUF_OFFSET 0
- define IMAG_BUF_OFFSET 1
So, going on the assumption that the I and Q data sets were being written backwards, on Jan. 28, 2012, 14:31:00 utc the two values were changed such that the real value was defined as 1 and the imaginary value was defined as 0.
However, by this point it was realized that the velocity sign of hdw.dat.bks file needed to be changed to +1 for data after the Jan. 17th software reinstall. If the values in the hdw.c file that were set on Jan. 28 were to continue being used, then the hdw.dat.bks file would need to use a -1 for the velocity sign.
Several e-mails from Ray indicated that we should strive to use a velocity sign of +1 and so on Jan. 31, 2012, 17:10:55.0 utc the real and imaginary offset values in the hdw.c file were returned to the original values (real = 0, imag = 1). Also, as this time, the hdw.dat.bks file was changed to use a +1 for the velocity sign.
Where are we now?
The hdw.dat.bks file has been updated with 3 new entries to account for the flip in the velocity sign after the installation of the new software on Jan. 17th. The other 2 entries account for the flip and subsequent reverted in the real and imaginary offset values in the hdw.c file. After the hdw.dat.bks file was updated in DaViT, all of the fitacf and fitex files for Jan. 2012 were remade to ensure the correct velocity sign was used. Then with these new files, all of the tenplots for Jan. 2012 were remade. The new entries for the hdw.dat.bks file are as follows:
UNTIL 12/Sep/2011 00:00:00.0
33 2011 21945600 37.100 -77.950 50.0 -32.0 3.86 1 0 -0.324 1 0.0 -58.9 -2.7 0.0 0 110 16
UNTIL 17/Jan/2012 15:00:00.0
33 2012 1436400 37.100 -77.950 50.0 -40.0 3.24 -1 0 -0.324 1 0.0 -58.9 -2.7 0.0 0 110 24
UNTIL 28/Jan/2012 14:31:00.0
33 2012 2385060 37.100 -77.950 50.0 -40.0 3.24 1 0 -0.324 1 0.0 -58.9 -2.7 0.0 0 110 24
UNTIL 31/Jan/2012 17:10:55.0
33 2012 2653855 37.100 -77.950 50.0 -40.0 3.24 -1 0 -0.324 1 0.0 -58.9 -2.7 0.0 0 110 24
UNTIL 29/Jan/2999 00:00:00.0
33 2999 2419200 37.100 -77.950 50.0 -40.0 3.24 1 0 -0.324 1 0.0 -58.9 -2.7 0.0 0 110 24
EOF
The data after the reprocessing and replotting with the new hdw.dat.bks entries is seen to the right.
Ongoing Discussions
One ongoing discussion is the merit of controlling the velocity sign with the hdw.dat file or with the offset values in the hdw.c file.
Arguments for hdw.dat control:
1. The hdw.dat files are easily distributed amongst SuperDARN groups and keep a good record of start and end times which is easy to put with data timestamps
2. The start and end times allow for easy reprocessing of data using software that is already written, being used and works. Reprocessing data of I and Q values relies on software that is not regularly used but has been tested.
3. We've been keeping the velocity sign in the hdw.dat files for a number of years, so for consistency, it should stay there.
4. Since we aren't the source of the hdw.c file and the ROS.1.25 software, this file can easily be overwritten if another software reinstall is performed. Remembering a change to this bit of software will rely on memory.
Counter arguments to the hdw.dat and ones being for the hdw.c control:
1) It is much better to have all of MSI radars have the same vdir sign and that this sign be +1. This assures that rawacf and fitacf files at Blackstone are compatible in terms of their Doppler sense. Currently, you have compatible fitacf velocities with the other MSI radars, but acfs that are not. Having vdir=1 assures that both the acfs and the fitted velocities are compatible with the other radars and that when we develop new software we do not have to worry about exceptions for Blackstone. Once you reverse the IQ order it is going to stay that way until you reinstall the software in the QNX machine. This could be years from now. All you have to do is remember to reverse the IQ after the reinstall. If you do not remember, the data will remind you.
2) The velocity sign in the hdw.dat files was originally intended to tell you that there had been been a cabling error from the digitizers and that the velocity sign needed to be flipped. There was no correction for velocity sign in the original fitacf software. I believe this change only came into effect when the Australians rewrote the fitacf software about 5-6 years ago. When they did this they effectively created two meanings for Vdir. When I wrote up the text for the hdw.dat files, I only used the newer definition, but it is by no means the only one. Rob wrote the buffer offset definitions so that they could be easily changed. That was his way of allowing good data to be assigned a vdir of +1. There is actually a third way provided by the manufacturer of the digital receiver card. A register can be set using software to reverse the order of the IQ data. This the card to compensate for receivers that flip the Doppler sign. Rob's approach is the easiest to implement.
3) I already commented on this in point 1)
