hdw.dat Updater

Background: Jef Spaleta of University of Alaska Fairbanks and Kevin Sterne of Virginia Tech were recently talking about how a hdw.dat file change seems to have trouble propagating throughout the SuperDARN community. However, changes to these files occur so infrequently that it is difficult to keep things fresh on how to update the hdw.dat file.

Synopsis: The hdw.dat updater is a utility to help PI groups easily check on and download the most up-to-date hdw.dat files. Detailed instructions on setting up the utility follow. (It involves using a github account to set up a git repository in the the directory where you store your hdw.dat files - this is easy to do.)

Simple way of updating a hdw.dat file, web interface version

1. If you have a github account, you can edit the file from the web interface (no installation needed!). Kevin Sterne will need to add you to the hdw.dat team, so if you do have a github account, pass along your username to Kevin.

2. Once I've added you, you'll be able to go to:

https://github.com/vtsuperdarn/hdw.dat/

and click on the hdw.dat file that you would like to edit. You have to be logged in to make an edit, so go ahead and login at this point.

3. Once you are to the file you want to edit, click on the pencil icon on the top right of the text file near "Blame" and "History". This will allow you to directly edit the file. I encourage you to add in a short description in the line under "Commit changes" as well as a bit of a longer description in the bigger text box underneath of that.

4. Then it's simply a click of the "Commit changes" button to save the changes you've made.

Setting up hdw.dat github, command line version

Last revised April 9, 2015

1. If you do not already have one, create a github account at: github.com. If necessary, install the software associated with github on your computer. You can follow the instructions in creating a repository in a test directory to get the hang of git.

2. Once you have a feel for git, change into the directory where you keep your hdw.dat files. Make a backup of the files you have on hand currently because your current version of the hdw.dat files may be different than what is posted in the hdw.dat repository.

3. Then create a git repository on your local computer by typing in a terminal:

> git init

This command will add a .git directory to your current directory. The .git directory contains information used by the git program. In the git language, a repository is a collection of files and/or directories. You can think of it as being an untarred .tar file. For a more proper definition from github as well as other git terminiology, please visit: https://help.github.com/articles/github-glossary/

4. Once you’ve set up git to run on your local computer, you’ll want to copy, or clone, the current hdw.dat repository with:

> git clone https://github.com/vtsuperdarn/hdw.dat.git

This will create a hdw.dat subdirectory from the one that you're currently in. Once you've create this directory, move the prompt into this subdirectory with something such as:

> cd hdw.dat

5. Then to pull down the files from the vtsuperdarn/hdw.dat.git repository type in:

> git pull

This will download the files on the hdw.dat repository located at github.com which includes several hdw.dat files. You may not get satisfaction right away as you have not been added to the list of people allowed to pull from hdw.dat repository. The git command will download a hdw.dat file if the version posted online has been changed more recently than the file that you have stored on your local computer. If there is an error with pulling some of the files, you might try entering:

> git status

This command will tell you which hdw.dat files need to be checked out or which files you have in your local repository that are not a part of the hdw.dat repository on github.com

For more info: origin will act as a shortcut to the github.com server name. To access what remotes you have setup on your local computer you can enter: > git remote –v . Likewise, the git clone will by default create a master branch on your local computer. To see what branches you have created on your local computer, you can enter > git branch. (For more info on managing remotes and branches, I suggest searching for managing github remotes or managing github branches. You should find several websites either from github.com or another forum that should list commands to copy, creates, rename, etc. remotes and branches.)

7. From here, examine the contents of the hdw.dat file for the radar(s) that your institution is/are responsible for. If the contents of the hdw.dat file that you downloaded from the github repository is different than the copy that you made previously, then you need to update the github repository!

8. To update the github repository, you will want to contact Kevin Sterne at or Jef Spaleta so that we can add you to the hdw.dat team and allow you to push your file to the repository.

From here:

a. Copy the hdw.dat file that has the latest/correct information back into your local directory in which you created the git repository earlier.

b. Then, to add a file to the list of files to be uploaded or ‘committed’ to the hdw.dat github repository, type in:

> git add hdw.dat.bks

Where the ‘hdw.dat.bks’ is the file that you’d like to upload to the github repository.

c. Once you have the list of files that you would like to upload type in:

> git commit –m 'Univ. of Alaska Fairbanks version of hdw.dat.ade'

Where the message between the single quotes should briefly describe the changes between your file and the github repository file. This line allows the changes to the file to be tracked and explained; it can be as simple as ‘changed range gates to 105 for hdw.dat.han’.

d. Then, to upload the files to the github repository, simply type:

> git push

If you look at the repository at: https://github.com/vtsuperdarn/hdw.dat

you will see that your file has been uploaded with the 'commit' note beside the file. If you have not yet been added to the hdw.dat team, the people on this team will see your push request and will be able to add you to the team.

NOTE: If you get an error here that says:

git push origin master
fatal: remote error:
You can't push to git://github.com/vtsuperdarn/hdw.dat.git

Use https://github.com/vtsuperdarn/hdw.dat.git

Then enter:
> git remote set-url origin https:/github.com/vtsuperdarn/hdw.dat.git

9. If you would like to setup an automated pull of the files, to be performed (say) every night, you can setup an ssh login without password entry much as you would other ssh connections. Here, you will want to add your id_rsa.pub key to github.com through the following page:

https://help.github.com/articles/generating-ssh-keys

NOTE: Skip down to step 4 on that page if you already have an id_rsa.pub generated, otherwise you will mess up all of your other automatic logins.

a. Once you are able to login to github without a password, you will want to change the config file that the repository on your local computer uses to access github. So, go to the directory of your repository and enter

> cd .git

b. From there, pick your favorite text editor to edit the config file. In this file you will want to change the url line to:

url = :vtsuperdarn/hdw.dat.git

save the file and test out things out by changing one unimportant character of one of your hdw.dat files and type:

> git add hdw.dat.yourradar
> git commit –m ‘File mostly unchanged, testing auto login feature’
> git push

If you did not need to enter a username and password then you have it right!

10. If you get lost, feel free to contact Kevin Sterne at or search around the github website as there are a lot of useful help pages there.

Github interfacing script

automates the day-to-day commands mentioned before can be found at:

git_pull_hdwdat.sh

This script should be downloaded and run on a computer that can run shell scripts. Be sure to edit this code for your particular uses as your groups radars will be different than what is listed in the code.

Execute the script with the command (or similar command):

> ./git_pull_hdwdat.sh

What to expect upon running the script

The script will run once, checking for any changes on the files posted in the github repository. If a change in a file is detected, the script will replace your local version with the updated file. You will be notified by the output upon executing the script.

The current hdw.dat github repository has the following files updated on an automated basis:

Virginia Tech	BKS	FHE
	FHW	GBR
	KAP
University of Saskatchewan	SAS	PGR
	INV	RKN
	CLY
La Trobe University	TIG	UNW
La Trobe University	BKP
University of Alaska Fairbanks	ADE	ADE
	KOD	MCM
	SPS
IRAP/CNRS	KER

NOTE:This script will only check for updated hdw.dat files once. In order to automate this process it is expected, for the time being, that this script is executed as a cronjob or similar scheduled executing program and the output is piped into a log file. It is advisable to set up a log file so that changes to your hdw.dat files due to the updater can be tracked.