StationPlaylist Add-on Internals: final thoughts and add-on development process overview

Hi,
Now that we've visited internals of StationPlaylist Studio add-on, I'd like to give you a tour of my lab where I develop this add-on. Along the way you'll learn how an add-on is born, coded, tested, released and maintained.
Lab setup, development equipment and software
For all my software development, I use two computers: a touchscreen laptop and a desktop, both running Windows 10 and latest NVDA next branch snapshots. Both also run Cygwin to run various command-line tools (Git, SCons, etc.), and in case I need to compile NVDA from source code, installed Visual Studio 2012 with latest update and other dependencies.
In case of SPL add-on, I have different Studio versions installed: 5.01 on my laptop and 5.10 on the desktop. This allows me to work on both versions at once (both computers have the full source code of the add-on, though I tend to write bug fixes on my laptop and experiment with new things on my desktop).
Git: a "smart" source code manager
Like other NVDA developers and many add-on writers, I use Git for source code management (contrary to its slogan, Git is very smart). This is a distributed system, meaning a local repository contains the complete record of how the source code is managed (no need to connect to a server to commit and fetch updates). For example, using just my local copy of the SPL add-on source code, I can view commit history and generate older add-on releases.
Another advantage of Git is extensive support for branches. A branch is a development workflow separate from other branches. For example, NVDA screen reader uses at least three branches for its workflow: next (alpha), master (live beta) and rc (release candidate, used to build official releases). SPL add-on uses this approach as well: there are at least two branches in use, called master and stable used for ongoing development or release and maintenance, respectively (we'll come back to branches in a second).
How an add-on feature is born
Let's go through a typical development process for an add-on feature by looking at how broadcast profiles was developed (for more information on broadcast profiles, refer to configuration management article).
I started working on broadcast profiles in March 2015 while developing add-on 5.0. This was a natural extension of add-on settings dialog: whereas this dialog (and the configuration database it uses) only dealt with a single profile, I thought it might be a good idea to allow multiple profiles to be defined and to let the settings dialog respond to profile changes.
There was an important reason for writing this feature: Since NVDA supports multiple configuration profiles and since some broadcasters were hosting multiple shows, I thought it would be a good idea to implement a similar feature in the SPL add-on. Thus, I envisioned broadcast profiles to be used primarily by people hosting multiple shows, with each show defined as a profile.
In March and April 2015, I started rewriting certain parts of add-on configuration manager (splstudio.splconfig) in preparation for developing broadcast profiles (now included as part of add-on 6.0-dev). I started by writing todo comments (where appropriate) describing what the future feature should be like. I then modified initConfig and saveConfig (discussed in app module articles), initially telling them to work with the default profile (the one and only configuration map then), then I left it alone until add-on 5.0 was released in June 2015.
In June 2015, I opened a new branch (initially using the codename "starfish") to house code related to broadcast profiles. Before any "real" code was written, I studied NvDA source code dealing with configuration profiles to learn more about how Jamie (James Teh from NV Access) implemented this feature. Once I understood how it worked, I copied, pasted and changed the code to match the overall add-on code base (giving nV Access the credit they deserve).
One of the first things I had to decide was how to store profiles. I experimented with using ConfigObj sections, one per profile, but this proved to be problematic (a profile could be given the name of an existing configuration map key). I then went back to NVDA source code to find out how NV Access solved this problem (using separate ini files), implemented it, and was met with another problem: transfering values between profiles. This was resolved by specifying whether a setting was "global" (applies to all profiles) or specific to a profile. Next came profile controls in the add-on settings dialog and using choice events to set alarm values using values from the selected profile.
The last thing I did before merging the broadcast profiles branch to master branch in July was revising configuration error dialog and writing documentation for broadcast profiles. Once the documentation was ready and small issues were fixed after going through many rounds of testing (on my own computer and from the profiles branch itself), broadcast profiles branch was merged into master. But the development didn't stop there: thanks to provisions I made, it was quite simple to implement instant switch profiles (again it had issues which are now largely resolved).
Dealing with threaded code: headaches during development of background encoder monitoring feature
You may recall our discussion of Cart Explorer and how it went through extensive testing to arrive at the current state (this was a difficult code segment). When it comes to difficulty, nothing beats multithreaded code, especially if it involves multiple threads working in parallel (rather, almost parallel), and I tasted this when writing background encoder monitor (add-on 5.0). This involved tracking how many threads were running to make sure no unnecessary threads were running, catching suttle errors and race conditions (a connection attempt could run a thread without checking if the encoder is being monitored) and so on. Thankfully, I went through a similar set of problems a few months earlier when I struggled with library scan (add-on 4.0), and that experience taught me to be careful with threads (and to experience fewer headaches).
Add-on development process
Follow me as I show you how a typical SPL add-on version is developed, released and maintained:
1. Before starting work on the new add-on version, I write down some goals the add-on should achieve, including feature ideas, user (your) suggestions and so on.
2. I then hold a conference call with add-on users to see what they think about some ideas and gather feedback (these are also written down).
3. I then create separate branches for each feature in order to isolate  code and not to break existing code.
4. Next, I write todo comments reminding myself as to what the feature should be like, then I start working on it. As each feature is being developed, I do mental simulations as to how you might use the feature under development, such as possible errors, messages spoken and so on.
5. Once the feature is quite stable, I test the feature to uncover bugs and to fill in the missing pieces. When it comes to testing, I test the new feature branch on both of my computers running different versions of Studio to make sure it works across versions (if not, I go back and modify the code to recognize differences between Studio versions).
6. After testing the feature for a while and if the feature is stable, I merge the feature branch into master.
7. Every few weeks, I publish master branch snapshots to gather feedback from users willing to test drive snapshots.
8. At some point, I set release target window for the next add-on version (for 6.0, it is January 2016). This determines when feature freeze should be and beta release window (for 6.0, beta is scheduled for sometime in December 2015). Between feature freeze and the first beta release, I concentrate on code refinements and bug fixes.
9. After going through several beta cycles (typically two), I ask NVDA community add-on reviewers to review my add-on code and request add-on release during the release window (this is done by merging master branch into stable branch).
10. Once the add-on version is released, subsequent maintenance versions (localization updates, bug fixes, minor tweaks) will be released from the stable branch, with the master branch containing the code for the next major version.
11. Once the next version enters beta cycle, further maintenance releases may or may not happen (an exception is long term support release, described below).
Long term support release
A typical add-on version is supported until the next add-on version is released (currently several months). However, there are times when an add-on version receives extended support (termed long term support (LTS) release). This happens if the next major version of Studio is released or a version of Studio with user interface changes is released.
A LTS version is a major version of the SPL add-on with some notable differences:
* Support duration: A LTS version is supported for at least two add-on development cycles (at least a year).
* Features: A LTS version may contain some features from the next major add-on release.
* Studio version supported: A LTS version is the last version to support the oldest supported Studio version. This is designed to give people plenty of time to upgrade to newer Studio releases.
As of August 2015, the most recent (and so far the only) LTS version was add-on 3.x (September 2014 to June 2015). Add-on 3.x was maintained thus:
1. Add-on 3.0 was released in September 2014.
2. Add-on 3.5 could have been the last maintenance version for add-on 3.x if it was not a LTS version.
3. When add-on 4.0 was released (January 2015), add-on 3.6 was released, backporting some features from 4.0. Users were told that add-on 3.x will be the last version to support Studio versions earlier than 5.00. From that time on, add-on 3.x was taken off the stable branch and was moved to an internal branch.
4. When add-on 5.0 beta was released (May 2015), add-on 3.x (3.9 was available then) entered end of support countdown (no more maintenance releases).
5. A few weeks later, when add-on 5.0 came out (June 2015), add-on 3.x became unsupported.
Final thoughts
As I close this series of articles on StationPlaylist Studio Add-on Internals, I feel it is time I reveal why my add-ons are free: it is because I love you users and as a service for NVDA user and developer community (and in extension, to all blind broadcasters using SPL Studio). What brings me joy as an add-on writer is the fact that this add-on (and accompanying documentation) has made impact in your lives and lives of listeners to your shows, as well as to other NVDA users and developers around the world. Thank you users for your continued support and feedback, and I promise once again that all my add-on code (including SPL Studio add-on) will be free and anyone is welcome to study and improve upon it.
For add-on writers looking for quality add-on documentation, I hope this series gave you an inspiration as to how to write amazing documentation in your future projects. For people new to add-on writing or for those interested in writing an add-on, I hope this Add-ons Internals series served as a handy resource for your projects, and in extension, gave you an idea as to how certain NVDA functions work. If you'd like to reference this documentation or use it as a blueprint, you are more than welcome to do so. Thank you community add-on reviewers for your continued support and reviews.
Important notices and credits
I'd like to thank StationPlaylist staff for continued collaboration with screen reader users in regards to accessibility of Studio. A special thanks goes to Jamie Teh from NV Access and Geoff Shang (original add-on author) for giving me and others a foundation for future goodies. As always, the biggest thanks goes to you, the users of SPL add-on for your continued feedback and teaching me new things about studio.
Source code notice: to protect copyrights, parts of Studio API has not been documented. Also, source code discussed throughout this series may change as future add-on versions are developed.
Copyrights: StationPlaylist Studio, Track Tool and StationPlaylist Encoders are copyright StationPlaylist.com. NonVisual Desktop Access is copyright 2006-2015 NV access Limited (released under GPL). SAM Encoders is copyright Spatial Audio. Microsoft Windows and Windows API are copyright Microsoft Corporation. Python is copyright Python Software Foundation. StationPlaylist Studio add-on for NvDA is copyright 2011, 2013-2015 Geoff Shang, Joseph Lee and others (released under GPL). Other products mentioned are copyrighted by owners of these products (licenses vary).