Software Distribution Working Group
Welcome
This is an area for basic information and updates on the system we are developing for software development and distribution. Although not officially constituted as a Working Group, a number of people are in fact teaming on this project and the SuperDARN PIs and wider community should be kept aware of developments. Communications amongst team members are maintained via the DaViTPy listserv.
What's New
During the CEDAR 2014 workshop Nathaniel held a meeting of software developers on June 23, 2014 to discuss the protocols for revising software via GitHub. Ashton's notes are given in the Supplemental Material section.
On July 13, 2014, Nathaniel created a new master branch on github, based off of the setup branch. (See following sections for an explanation of these terms.) In his words:
>> This means we should be good to continue code development from this new master branch.
1. The new master branch (officially titled “master”) is a merge of the old master branch with the setup branch. This will hopefully keep the recent developments we have made in the old master branch while giving us the more standard python setup procedure. I tested the new installation procedure on Ubuntu 14.04 and re-wrote the README.
2. I’ve created 3 new releases: the old master branch, the setup branch before anything was merged into it, and the setup branch with master merged into it.
3. I tried to create a init.py for the top-level of davitpy, but I ran into some complications so I will save that job for another night.
Please go ahead and test this if possible. I am sure things will break! I already know there are issues with magnetic coordinate plotting… <<
Ashton's notes from the Developers' meeting held during the CEDAR Workshop, June 23, 2014:
In attendance: Nathaniel Frissell (Virginia Tech), Ashton Reimer (Univ. of Saskatoon), Jef Spaleta (Univ. of Alaska, Fairbanks), Chister van der Meeren (Univ. of Bergen, Norway), Sebastien de Larquier (VT Alumnus, via Google Hangouts), AJ Ribeiro (VT Alumnus, via Google Hangouts)
Status of RST and FITACF
Summary of discussion of RST and FITACF motivated by Mark’s email of July 27, 2015 (derived from email thread between Mark, Pasha, Jef, and Mike):
The semi-official version of RST software is accessible via the VT GitHub from the VTRST3.5 repository:
https://github.com/vtsuperdarn/VTRST3.5
This was derived from the last official release from APL with corrections to buggy routines.
Other groups (U. Saskatchewan, UAF) may have their own similarly derived versions of RST.
Note that the GitHub site has active code development going on that follows categorization into master and development branches.
A user can download a zipped version of RST3.5 from the GitHub site. The total size of the download is 127 MB. This includes the binaries. This should run easily on a 64-bit machine with Linux Ubuntu as the operating system once the pathways are set.
Action >> It is not clear that the source code will compile. One of the groups (besides VT) should try to compile the source code as an exercise.
Note: The download will include the version of FITACF that is running at VT which is not much changed from that which accompanied the last official APL release.
Note: Another, smaller package of software also bundled at the VT GitHub is RSTLite. This was generated by AJ as a stripped down version that contains only basic code to open and read files with no proprietary code or plotting code. This has proved useful for people who simply want to open and read the files and then do analysis with their own software.
New about FITACF (summarizing from Pasha):
The new software engineer at U Sas (Keith Kotyk) has refactored FITACF and stamped it as FITACF 2.6. This is a significant advance for making changes to FITACF. The 2.6 version reproduces the results of the original version (stamped as 2.5) while the 2.7 version has corrections for straightforward velocity error estimates and CRI levels.
Both updates are available from the VT GitHub repository under the "refactoring" branch of the VTRST3.5 repository. Here is the direct link:
https://github.com/vtsuperdarn/VTRST3.5/tree/feature/refactorfitacf
Action >> other groups should try to download and run the updates and compare against running their in-house version of FITACF.
Repositories:
We agree that it is desirable to have a special (non-GitHub) repository where only compilable source code is available. Active development of software will take place using the GitHub resources while the repository will contain static code that has been vetted for this release.
Action >> The source code associated with the current RST could be checked out (to see that it will compile) and then entered into said repository.
Copy of Pasha’s email of July 27, re: RST and fitacf availability and versions:
Mark,
RST:
I was looking at this issue since coming back from Europe late in June. It looks like at the moment we only have a semi-official RST 3.5 downloadable from VT through github. The problem with that package is that you need to download it with all binaries (actually, this is the only way to get it from github anyway!) because it has problems with compiling. This is several hundred Mb zipped and might take along time to download.
After making inquiries here, I discovered that some time ago Dieter produced a compilable version of RST 3.5 from the last Rob's release, but I need to check if it is really working.
FITACF:
Since mid-May, our new software engineer Keith Kotyk and me have been working on a gradual transition form the latest available version, FITACF 2.5, towards a fully-reshuffled FITACF 3.0 which would (a) be modularized and well-documented, (b) contain optimal implementations of least-square fits for both power and phase and (c) produce realistic error estimates for the major parameters.
At the moment we have produced two updates to FITACF 2: 2.6 — re-factored FIATCF 2.5 which produces the same fit results as the original version; 2.7 — it includes modifications recommended by the WG to get more realistic velocity error estimates and CRI levels.
Both updates are available from the VT github repository under "refactoring" branch.
However, while 2.6 has been tested during re-factoring by byte-to-byte comparison of the FITACF files with their original versions, 2.7 still has to be tested against both simulated and real data sets.
Finally, Keith has already done some basic work on FITACF 3.0, but it still far from being finished.
Repositories:
We have discussed this with Mike and come to the conclusion that there should be a depository which is separate from github. This depository should contain only official compilable code, in contrast to git stuff with all development branches, binaries etc. At this moment such a depository does not exist because there is nothing yet to put in it.
In summary, the latest official version of FITACF would still be 2.5. It produces reliable estimates of the major parameters (velocity, "power", width, elevation) but the errors are incorrect. Testing v2.7 will take some time, so if you have issues with FITACF, we can try to fix them without waiting for the tests to be completed.
Copy of Mark’s email of July 27, re: Latest versions of RST and FITACF
Dear Pasha, cc Jef, Mike,
Hope you are well. It was good to see you at the workshop in June. We are starting to look at the versions of the software that we have here as we seem to be hitting some issues with data analysis. There seem to be a couple of problems that we have but it would seem sensible to start at the beginning and check the latest versions of the code Hence the following questions:
Is there a standard implementation for RST that includes the most recent accepted version of FITACF?
What are the versions?
Where can we find these?
Is there a software repository from where we can download them?
Thanks for your help.
Cheers, Mark.
Basic Information
Intro to new SuperDARN system for software development and distribution
A vast array of software products including the core SuperDARN analysis software is available for download. We have recently moved to an on-line model for software development and distribution that is based on GitHub, a commercial Git repository web-based hosting service. Github provides a graphical interface and desktop with directory-like structure that simplifies management tasks.
Getting Started (Overview of basic access and installation)
The relevant GitHub area is hosted by the VT SuperDARN group and is accessed at
https://github.com/vtsuperdarn/
The key directory is labeled
davitpy
for the Data Visualization Toolkit (DaVit) Python project. Note that our aim is to have as much software as possible utilize Python, which is an open-source programming language that is rapidly replacing IDL for scientific analysis. Be aware that the Toolkit includes older code written in Fortran and C that utilizes python wrappers and the appropriate compilers are needed (provided with installation).
Click through davitpy and find instructions on how to install and extensive documentation. Installation works best on Ubuntu 14.04. The install steps have recently been simplified. Although the code can be downloaded in zip format, it is recommended that the user Clone in desktop to a URL, as this will create a complete copy of the software with proper directory structure with linkage to the tracking features of GitHub. So, as changes are made, you will be notified. Developers are encouraged to join the development Google group, davitpy-dev.
See the README file at
https://github.com/vtsuperdarn/davitpy/blob/master/README.md
Concept of software development and distribution
Very briefly, GitHub utilizes a branching model whereby code is developed on a branch and then, after testing, merged onto a more authoritative branch, a process that culminates in merging onto the master branch. If a person is interested only in accessing the latest official release of the software, he need only draw on the master branch. This user should still be aware that the master branch might be modified because of a 'hotfix' that is implemented to fix an obvious bug, leading to an increase in the tag (or revision) number of the master branch from 0.0 to 0.1, 0.2, etc. A major step in revision of the master branch is indicated by an increase in whole-integer tag number, e.g., to 2.0, 3.0, etc. Developers work on a ‘develop’ branch which was drawn originally from the master at some tag level. Here, they may work on pieces of the code, introduce changes, ask for other developers to test the change, and then issue a ‘merge’ request to have the revised software implemented in the develop branch. Eventually, once enough changes have been realized and there is general agreement, a 'release' branch is derived from the develop branch. The release branch is a holding area for code to be double-checked and only minor fixes are allowed. Once a satisfactory state of stability is reached and all are agreed, the release branch is merged onto the master branch as a major upgrade.
Here is a schematic and deeper explanation of this development model:
http://nvie.com/posts/a-successful-git-branching-model/
With an installation that uses Cloning to desktop, the user will have access to all branches and also to notifications of changes, merge requests, etc. There is a defined procedure for checking out a chunk of code to a local branch for revision; if the revision is a hotfix for an obvious error at the master level then the checking out is done to the master; if it involves code development then the checking out is done to the develop branch. The local branch thus created is also called a ‘feature’ branch and if your revision ultimately checks out then you will issue a merge request (via the DavitPY listserv), for the master branch in the case of a hotfix and for the develop branch in the case of code development. In the fullness of time a revision to the develop branch will process through a release branch and finally appear at the master branch with a new integer tag number.
The account of official releases can be tracked via the releases tab at the davitpy GitHub site. This also shows the number of ‘commits’, that is, requests to check out (download) the code since the release was made.
See the record of releases at
https://github.com/vtsuperdarn/davitpy/releases
Summary of DaViT-py development group resources
Main discussion area (listserv): https://groups.google.com/forum/#!forum/davitpy
Projects main page (public face): http://vtsuperdarn.github.com/davitpy/
Tutorials and other information on our github wiki: http://github.com/vtsuperdarn/davitpy/wiki
ToDoS and other progress details, see our Trello page: https://trello.com/b/djb82luC
If you have any questions on the code development then use this listserv. For questions on accessing/using the code, please read the files on the github repository.
Supplemental Material
Ashton's notes from the Developers' meeting held during the CEDAR Workshop, June 23, 2014:
In attendance: Nathaniel Frissell (Virginia Tech), Ashton Reimer (Univ. of Saskatoon), Jef Spaleta (Univ. of Alaska, Fairbanks), Chister van der Meeren (Univ. of Bergen, Norway), Sebastien de Larquier (VT Alumnus, via Google Hangouts), AJ Ribeiro (VT Alumnus, via Google Hangouts)
Restructuring
-> Setup branch (Ben from VT), tag current master as 0.2, make current setup branch the new master branch, create devel, create releases, create hotfixes.
-> Make a davitpy module that loads all modules (need an init.py file)
-> Ensure that we have separate functions for fetching files, reading files, plotting files. Can have wrappers to make plots all in one go.
-> Ashton will work on the fetching files routines (can incorporate environment variables to set server preference etc.) Ideally the data servers should have config information available that pydarn can parse.
-> Plotting without X (for .matplotlibrc you can change the backend) can change the backend using matplotlib.use('Agg')
-> Remove raydarn for now (Sebastien is working on it)
Testing
-> use the python "unittests" library (AJ and Jef will work on this)
-> keep test for each piece of code in the same file as the code to be tested
-> "We shalt depend on unittest." -Jef
Documentation
-> if matplotlib style documentation works with sphinx, then we can switch over to it (Christer and Angeline will be responsible for this).
-> We need to keep each other honest on code submissions (make sure we always adequately document the code).
-> "If it is clever, document it." -Jef
Summary of on-line meeting of 08/15/2014 hosted by VT
All,
Thanks to all who participated in today's experimental telecon, especially if you weren't able to be heard. Here is a summary:
(i) Nathaniel briefed us on the status of distributed software development model and explained about master and development branches, pull requests and merging. The indexing of pull requests has now reached #76 with two pull requests (both from Matt) open. The link to the github repository is
https://github.com/vtsuperdarn/davitpy
(ii) Mike explained that he has set up a web page at the VT's web site for high-level explanation of the workings of the software development team. This is accessible under 'SD Working Groups' via the link labelled for 'Software Distribution Working Group', an outfit not yet officially sanctioned, or click through to http://vt.superdarn.org/tiki-index.php?page=Software+Distribution+Working+Group
(iii) Matt gave us a review of his coordinate transformation code. This is a signficant development involving coordinate families that offers a general way to effect coordinate transformations and also a good demonstration of development testing procedures. To review, see Pull request #74 (now closed) or click through to https://github.com/vtsuperdarn/davitpy/pull/74
(iv) As Jeff has pointed out, WebEx is not ideal for a development team that is Linux-based. Also, some of the equipment at VT is clunky for hosting on-line meetings. After the meeting, Kevin and Nathaniel started setting up a more high-powered computer and Mike offered to find a better mike/speaker system. Also, the VT group will try out Google Hangouts with the 'record to youtube' feature described by Jeff next week. We might then try it out with a subset of developers before we call another on-line meeting.
Echoing Jeff, I am reluctant to call meetings unless they are needed and will be productive.
Mike
