StationPlaylist Add-on Internals: The magic behind Cart Explorer

IMPORTANT: The following article superseeds a previous blog entry (summer 2014) regarding this subject.
Hi,
A live radio broadcast would not be complete without jingles. This can range from station promotions (often called "promos"), advertisements, jingles to convey the mood of a show, segment jingles and more. Many station automation programs, including StationPlaylist Studio includes facilities to manage jingles (sometimes called carts), including defining a cart to be played when cart keys are pressed, announcing the name of the playing cart and saving uer specific carts to a safe location.
For blind broadcasters, one of the things they worry is pressing a wrong jingle key by accident, thus script writers were asked to implement a way for broadcasters to learn which carts are assigned to cart keys. As of time of this post, all three screen readers (JAWS for Windows (script author: Brian Hartgen), Window-Eyes (script author: Jeff Bishop), NVDA (script author: Joseph Lee (I, the author of this article)) includes a feature to learn jingle assignments. As this is a series of articles on internals of an NVDA add-on, I'll give you an overview of what happens when you activate and explore cart assignments (in fact, this section was the most interesting and feedback driven portion of the add-on). Along the way you'll learn where Cart Explorer (NVDA's version of cart learn mode) draws its power and why it is very important.
Carts in StationPlaylist Studio
Studio comes in three editions: Demo (same as Pro but for limited time trial), Standard and Pro. The first user visible difference between Standard and Pro is number of cart assignments: Standard can store 48 jingles, while Pro can work with 96 of them.
To play jingles, a broadcaster would use Cart Edit Mode Control+T), then assign a hotkey to a file. For Studio Standard, you can assign F1 through F12 by themselves or in combination with Control, Shift or Alt. In Demo and Pro, number row can be assigned (1 through 9, 0, hyphen (-) and equals (=) either by themselves or in combination with Control, Shift or Alt, for a grand total of 96 jingles). Once jingles are assigned, they will appear under cart menus (there are four cart menus, one for standalone keys (called main) and one each for Control, Shift and Alt).
Where does Studio store carts?
Studio's "carts" are housed in Studio installation folder. There are four cart files (called banks) in use: a .cart file for each of the cart banks (main, Shift, Control, Alt). During normal business hours, Studio will work with these four banks unless told by a broadcaster to load carts from a different cart bank file.
Cart Explorer: my own Summer of Code
It was a hot day in June 2014 when I sat down to design a way to let broadcasters learn cart assignments. Since I was developing add-on 3.0 back then, I decided that this feature should be a top priority feature to be included in the upcoming release.
When I started writing this feature, the first thing I thought about was its name. I felt "cart learn mode" didn't really convey the picture - after all, I reasoned that broadcasters will use this feature to explore cart assignments. Thus the name "Cart Explorer" was chosen - in effect, when you use this feature, you are browsing jingle assignments in preparation for a show.
Next, I read JAWS scripts documentation to get a glimpse of how Brian has managed to implement cart learn mode. In JAWS scripts, script settings are stored in the user configuration directory (typically this is %systemdrive%\Users\%username%\AppDate\Roaming\Freedom Scientific\JAWS\%JAWSVersion%\Settings\Enu; Brian, please correct me if I'm wrong). A section of this script configuration file is dedicated to carts, and JAWS scripts use a map of key names and cart values to announce cart information while cart learn mode is active.
Based on this information, I started writing an ini file parser, seeing that broadcasters would store cart assignments in a configuration database. This was prone to a number of errors, including wrong cart name format, nonexistent cart key assignment, invalid configuration format and others. I once wrote a blog post (on this blog) explaining how this worked (times have changed, as you'll see very soon).
Then I became curious as to how Studio stores its own cart banks, and naturally, I opened the folder where carts were stored and opened each .cart file in Notepad++ (a very handy text editor). From reading the cart bank format (explained below), I thought it might be useful to write a cart bank file parser. Thus I resumed writing Cart Explorer routines, this time incorporating the cart bank format, not forgetting to handle suttle errors, and this is the routine used in add-on releases up to 5.x (6.0 uses a completely different yet related routine, as you'll see).
While writing the first version of Cart Explorer, I realized that this feature needed some real life testing, so I asked a seasoned blind broadcaster to test this feature. We spent a better part of Independence Day writing, debugging and rewriting this routine until we felt satisfied. In the end, our hard work paid off, as you can see in subsequent paragraphs.
Introducing Cart Explorer version 1
Cart Explorer version 1, shipped as part of add-on 3.0, worked as follows:
1. You press Control+NvDA+3 to activate Cart Explorer. When this happens, NVDA will make sure you are in main playlist viewer, then it will set a flag indicating that Cart Explorer is active.
2. NVDA will then open and parse cart bank files, storing cart assignments in a dictionary of cart keys to cart names. This parser also takes care of some corner cases, including skipping unassigned carts and determining Studio edition in use. Once carts were parsed, NVDA says, "Entering Cart Explorer", and if errors occur, NVDA will inform you that it cannot enter Cart Explorer (this happens if the cart bank file doesn't exist).
3. While using Cart Explorer, if you press a cart key, NVDA will look up the name of the key in the carts dictionary, and announce the cart name associated with it (if found, otherwise, NVDA says, "cart unassigned"). If you are using Standard edition and press number row keys, NVDA will warn you that cart commands are unavailable.
4. It so happens that some people will activate Cart Edit Mode to modify cart assignments while in the middle of exploring carts. If this happens, NVDA will remind you (via doExtraAction function used by name change event) that Cart Explorer is active, and when Cart Edit Mode is turned off, NVDA will ask you to reenter Cart Explorer (this was done to parse newly updated cart bank files).
5. You press Control+NVDA+3, and NVDA will clear carts dictionary, thereby leaving Cart Explorer.
But there was a major concern with this approach: what if a future version of Studio uses a different cart bank format? Thus, I revisited cart bank files again in July 2015, and this time, I noticed a familiar structure: comma-separated values. To test my hypothesis, I opened .cart files in Excel, and voila, it presented itself just like any CSV file. Thus I worked on modifying cart parsing routine, this time using Python's CSV module to parse "cart" files (cart bank files are really CSV files in disguise). This new routine (described below) will make its appearance as part of add-on 6.0.
The magic behind Cart Explorer: handling CSV files
Since Python comes with a library to handle CSV files and since cart banks are CSV files, I rewrote Cart Explorer routine (a function in splmisc module which returns the carts dictionary) as follows:
1. When entering Cart Explorer, Car Explorer preparation routine (splmisc.cartExplorerInit) will take a snapshot of your user name and Studio edition (Studio's title bar indicates which version is in use). Then it initializes the carts dictionary and stores the Studio edition in use.
2. Next, the preparation function will write down names and paths to cart banks. In case a user other than default user is using Studio, it'll modify the cart file names to match the name likely to be used by Studio to present user-specific cart banks. These cart names form one part of the cart bank path (the other part is the path to the folder where the carts live, obtained by using an environment variable).
3. For each cart bank, NVDA will ask Python to parse the cart bank as a CSV file (csv.reader; when finished, it returns a list of lists, with each list representing one row of a CSV table).
4. Once the csv version of the selected cart bank is ready, the row containing cart keys and cart names, together with the cart bank modifier and the carts dictionary are sent to a helper function (part of splmisc module) that will do the following:
A. Depending on Studio edition, this helper will work with only the first half (Standard will only work with function keys, which are the first twelve columns of this row) or the entire row (the rest are number row keys) will be processed.
B. Depending on column position (items in the row list), it will see if function keys or number row keys should be assigned to the selected cart entry. This routine also checks the type of the cart bank (modifiers or none (main)) and modifies the cart key name accordingly.
C. Next, the helper routine will try to locate the name of the jingle assigned to the cart key in question, and if there is one, it'll add the cart key and jingle name pair into the carts dictionary.
5. Back at the cartExplorerInit function, if no erorrs were found while parsing a cart bank, it'll move onto the next one, otherwise it will inform the Studio app module by modifying a flag value in the carts dictionary (stored as an integer, representing number of cart bankks with errors).
6. By now cartExplorerInit is desperate to pass the carts dictionary to someone, and this someone turns out to be the Studio app module - once picked up by the app module, carts dictionary is hired by you to look up cart names for cart keys while you use Cart Explorer (to fire the carts dictionary, simply deactivate Cart Explorer by pressing Control+NvDA+3).
In effect, the routine above (the "magic" behind Cart Explorer) replaced a hand-written cart bank parser and simplified the add-on code (I regret not investigating CSV last year). As far as user experience is concerned, this is same as Cart Explorer 1, with the difference being the parsing routine. With the addition of splmisc.cartExplorerInit, the current version of the splmisc module (miscellaneous services, containing the Track Finder/Column Search combo dialog, column retriever and Cart Explorer preparation tool) was completed.
Conclusion
Cart Explorer has come a long way; from a simple suggestion to the CSV parsing routine above, Cart Explorer has changed to meet the needs of broadcasters using Studio and NVDA. I would like to improve this further in future releases (another suggestion I received was ability to specify cart file names for individual banks, and I'm thinking about implementing this in the near future).
One of the things you may have noticed as you read this article is how I and other developers continue to research better ways of accomplishing something. You also saw a glimpse of how developers and users shape a feature and how much work is involved to bring a feature suggestion to life. These activities (research and feature development collaboration) are just two of the pillars supporting Studio add-on for NVDA, and highlights how design philosophy and product development approach affects future course of product development.
This ends our detailed tour of internals of major features in Studio app module. When we come back, we'll visit our friend from the past: SPL Assistant layer and inner workings of various layer commands.
References:
1. Comma-separated values (Wikipedia): https://en.wikipedia.org/wiki/Comma-separated_values
2. RFC 4180 (Common Format and MIME Type for Comma-Separated Values (CSV) Files), Internet Engineering Task Force: https://tools.ietf.org/html/rfc4180
3. Import or export text (.txt or .csv) files, Microsoft Office Support for Microsoft Excel: https://support.office.com/en-za/article/Import-or-export-text-txt-or-csv-files-5250ac4c-663c-47ce-937b-339e391393ba
4. CSV (Python documentation, Python Software Foundation): https://docs.python.org/2/library/csv.html