Tuesday, November 3, 2015

Windows 10 TH2: What's new, screen reader development strategies, do not trust all information regarding release dates and such

Hi,
Welcome to another episode of Windows 10 articles by yours truly. Those who are regular visitors to my blog might be familiar with a series of articles describing state of accessibility of Windows 10, advisories on screen readers and so on.
As Windows 10 Fall Update (AKA Threshold (th) 2, Fall Refresh and so on) is just around the corner, I'd like to take you on a tour of what's new and changed in Windows 10 Fall Update from screen reader point of view. I would also like to take a moment to talk about predictions for Windows 10 adoption, implications of Windows as a service for screen readers, key things distinguishing public release builds of Windows 10 and my attitude on forwarding news sources to email lists.
What to expect in Windows 10 Fall Update
Windows 10 TH2 (Fall update) brings exciting features. In addition to multimedia casting in Edge, things Cortana can do now and other changes, it includes various changes to state of accessibility. Compared to RTM (10240), it is more refined, slightly faster and looks more polished.
The features introduced or changed in Fall Update can be categorized into three broad areas: more integration and socialization, deep and subtle tweaks, and responses to feedback. In terms of integration and socialization, Skype is now integrated into the operating system in the form of three new apps: Skype phone (voice calls), Messaging (IM's) and Skype Video (video calls). Cortana is better at understanding input from "digital pens" (stylus), as well as providing event reminders and offering Uber rides to attend an event. It is also possible to stream games and other multimedia content through Xbox and Microsoft Edge, respectively.
In terms of deep and subtle tweaks, TH2 includes a much better way to tweak environment variables (a series of global settings used by operating systems for certain operations such as executable path search). Previously, if you had to edit things such as path used to look for programs, one had to use an edit field which was prone to mistakes, whereas in TH2, it is organized as a list and an edit field. Also, you can now tell Windows to use the last printer used regardless of network location, and Hyper-V allows running virtual machine guests inside guests (termed "nested virtualization"). Other tweaks include making the operating system respond faster, particularly in memory management and in areas where UI Automation (UIA) is used (slightly faster than RTM build).
In regards to responding to feedback, TH2 includes ability to specify accent colors in title bars, refined context menus and other tweaks in response to feedback received from users, Insiders, developers and enterprises. As far as accessibility and screen readers are concerned, it is great to see Microsoft willing to work with screen reader vendors to make the oS better respond to accessibility queries from screen readers.
So when will we get a chance to play with Windows 10 Fall Update and what will be the build number?
Throughout 2015, some news sources claimed that there will be a minor update to Windows 10 after it is released to the general public, and we're about to see this claim come to pass in a near future. In October 2015, reputable sources such as Windows central, Win Beta and others reported that TH2 will be released sometime in November, and it appears Microsoft is on track this time (based on frequency of Insider build releases and according to these sites). In addition, some sites claim that it'll be out in early November (some articles claimed November 2nd as the release date, but we've passed that point; some of the sites now claim November 10th or 12th as the big day when TH2 will hit the air).
However, I'd like to question some of these claims:
* Build number: Although some sources such as Windows central say Microsoft has signed off Windows 10.0.10586 as the official TH2 build, there's no official confirmation from Microsoft that November 12th will indeed be the big day.
* Build revision: RTM or public releases of Windows usually have a four-part version identifier: major (Windows NT kernel) version, minor release, build number, and build revision. For example, the RTM build of Windows 10 has the build string of 10.0.10240.16384. Similarly, Windows 8.1 general distribution release (GDR) sent out in April 2014 has the build string of 6.3.9600.17031. Technically, this four-part version string denotes the revision of Windows NT hardware abstraction layer (HAL), a part of the operating system that allows Windows to talk to motherboard and other fundamental hardware of a computer system. Because the latest publicly disclosed build of Windows 10 reported to sites such as Build Feed have "0" as the build revision, we cannot say build 10586 is indeed the TH2 release version for now (in order for this to happen, the fourth part (build revision) must be something other than 0).
* Watermark still present in public Windows Insider builds: As many other Windows Insiders can testify, desktop watermark (displaying the version of Windows one is testing) is still present in the latest publicly available Insider build (10576). One sure way to test if the claims of November 12th release date is true is if Microsoft releases a build of TH2 with the watermark removed (last time it happened, it was released to both fast and slow ring fliers, with two weeks to go before general availability of the RTM build).
There are two key evidences that may support the above claims made by Windows Central and other sites:
* Insider build release frequency: Just before RTM build was released in July 2015, fast ring Windows Insiders had a chance to play with up to three builds in a single week, with one of them eventually landing at the doors of slow ring fliers via Windows Update. We saw a similar activity within the past few weeks: two fast ring Insider builds released just a few days apart, with one of them eventually going out to slow ring Insiders.
* The identity of the new Windows 10 update: The version of Windows reported by windows Version (winver.exe) has changed. Instead of saying "Windows version 10.0", it now says "Windows version 1511" (build 10576), confirming earlier rumors that TH2 will be released in November 2015 (yymm, similar in format to Ubuntu's version identifier of yy.mm).
Until Microsoft (or at least Terry Myerson or Gabriel Aul) gives official confirmation, I'd not trust some of these sources, nor should be something that should be forwarded to mailing lists, especially ones dealing with screen readers.
As a side note, one of the questions you might have is if TH2 is a service pack. No, it is not. Some journalists (such as Mary Jo Foley and Paul Thurrott, two influencial and reputable reporters) wrote earlier in 2015 that there will be a series of huge updates codenamed Redstone, with the first one scheduled for June 2016, bringing Microsoft Edge extensions. I'm hoping that Microsoft would step up to the game, as it'll give people second thoughts about taking the free upgrade offer (one concern critics raised, which I agree somewhat, is privacy and collection of some data to improve the user experience).
Windows as a service: what does this mean to screen reader users, developers and observers
A few months ago, Microsoft proclaimed that Windows 10 will become an example of "software as a service". Software as a service means a given software product will receive continuous refinements, with some products using subscriptions (Source: Wikipedia). In other words, instead of a big release every two to three (and sometimes, five) years, Windows 10 will receive ongoing updates (serviced). As for a claim that Microsoft may debut a subscription-based Windows version, I don't know (Microsoft did try this approach a few years back with Windows XP in certain countries).
As a user, developer and an observer of screen readers and their landscape, I'm concerned that screen reader vendors might not be prepared for this change of strategy. Continuous refinement opens a possibility that future Windows 10 builds may break existing functionality, especially if it involves a workaround that has worked in previous builds (a good example is toast/Action Center notifications not announced for some JAWS 17 users). Thus, it is more important now for screen reader vendors, users, observers and Microsoft to work together to come up with a long-lasting strategy of partnership, feedback loop and understanding in order to make Windows 10 the truly universal operating system for many, including for those depending on screen reading solutions.
Windows 10 adaption: A hyperbola of thoughts, setbacks, recommendations and second thoughts
According to Microsoft and other sources, Windows 10 powers over 120 million devices today. Although the first weeks saw sharp rise in Windows 10 adaption, privacy concerns and a potentially major mistake by Microsoft (Windows 10 update mistakenly selected in Windows Update for some) signaled decline in Windows 10 adaption rates. However, I think the adaption rate for Windows 10 will rise sharply again next summer when end of gree upgrade offer approaches, and I hope Microsoft will do something fantastic with Redstone 1 to make that happen.
In conclusion, I think Windows 10 TH2 (Fall Refresh/Update) is quite an improvement compared to TH1 (Build 10240). New capabilities offered by Cortana, Edge and other features opens up new possibilities,and I'm glad to see Microsoft taking feedback seriously, including ones dealing with accessibility and performance. Although it has room for improvement, it appears to be a fine operating system, and I hope screen reader users, developers and observers would use this time to think about future development strategy when it comes to Windows 10 support.
If I'm to give a grade to TH2 as currently stands, I would give this a solid B: exciting, has lots of potential yet needs some more polishing such as improved accessibility of some Skype features and apps. Time will tell if TH2 will receive a better grade and embraced by many, including users of screen readers. Looking forward to using TH2 and the Redstone update with various screen readers next year.
//JL

Wednesday, October 28, 2015

An open letter to Microsoft and Windows 10 universal app developers and users concerning accessibility of universal apps

To leaders at Microsoft specializing in accessibility, customer and developer relations, Windows 10, universal app architecture and ecosystem and others, fellow developers of screen readers, Win32, WinRT and universal apps, users of Windows 10, screen readers, Windows 10 observers including Paul Thurrott, Mary Jo Foley and others:
My name is Joseph S. Lee, a blind college student from Los Angeles, a translation and code contributor to an open-source screen reader for Microsoft Windows and one of millions of blind and visually impaired users of Microsoft Windows operating system. First, a sincere thanks goes to Microsoft for your continued commitment to accessibility and starting a new era of Windows through release of Windows 10 in July 2015. Thanks also goes to millions of app developers for Microsoft Windows operating system for making your apps accessible for blind people, providing opportunities for blind users of your software to be productive in their workplace and use your apps in unexpected places.
One of the promises of Windows 10 is universality. although Windows 10 has opened up many possibilities for unification between devices, software ecosystem and so on, there is one major issue that many in the Windows 10 culture may have not grasped or just beginning to grasp: accessibility of universal apps, or as I see it, lack of such accessibility for people with disabilities, specifically blind and visually impaired users using screen readers. By writing this open letter to the parties addressed above, I hope to start a dialogue or perhaps let the general public know the impact that lack of accessibility of universal app platform could have on lives of millions of current and potential users of Windows 10, and in extension, universal apps from Windows store.
According to a recent fact sheet from World Health Organization (WHO), as of August 2014, there are about 285 million blind people around the world, with nine out of ten blind people living in low income regions (footnote 1). Even in developed nations such as United States, South Korea (where I'm originally from), parts of Europe and others, many blind and visually impaired users are classified as poor or low income, or in some cases, unemployed. Because of this, coupled with trends in computing and other factors, blind and visually impaired people are some of the most affected by information blackout, considering that access to information has become a must in 21st century.
One route taken by blind people to gain access to information is through a computer and a piece of software called a screen reader. A screen reader is a program that identifies what is on screen and presents this information to users via speech or braille. Technically, recent trends in screen reader ecosystem has focused on understanding and tracking events fired by various controls using a specific accessibility layer such as Microsoft Active Accessibility (MSAA) or UI Automation (UIA), and it has become a requirement for screen readers to include support for accessibility API's from the beginning.
A good example of mechanics of a screen reader is when toasts appear in Windows 10 (sometimes called Action Center notifications). When such notifications appear, the screen reader will gather necessary information about the toast (knowing that it was a UIA object firing an "alert" event) which has just fired an event. Once the screen reader gathers this information (which includes text of the notification), it will speak the notification so the user would be informed as to which app has fired this event and press Windows+V to go to and interact with the toast notification.
Another common use case of screen reading is web browsing. Because web page presentation is hierarchical in nature, screen reader vendors had to implement a way to give screen reader users an experience similar to reading a document from top to bottom. One such screen reader uses code injection to obtain just the needed information from web browsers such as Internet Explorer, Firefox, Chrome, Edge and others, and will construct a "linear" representation of this web document (footnote 2). This gets interesting when the web application provides extra information or coded in such a way that allows users to use a web application as though one has installed the app on his or her computer (a good example is Google Docs). Thanks to recent efforts such as WAI ARIA (Accessible Internet Rich Applications), websites have become accessible portal of information for screen readers.
However, when we examine accessibility of apps, particularly those powered by an emerging ecosystem called universal apps (Windows 10, we're seeing a bag of mixed accessibility. The level of accessibility of such apps (and for Windows 8.x, Windows Runtime apps) ranges from fully accessible (Windows Feedback, Facebook and others), partial accessibility (unlabeled or misuse of markup or other coding in Twitter, unlabeled radio buttons in Insider Hub and others) to outright inaccessibility (Maps, games and others).
One note of concern is partially accessible apps where screen reader developers such as I had to write workarounds for inaccessible parts. A good example is universal app version of Twitter, where tweets (or timeline) are shown as a list and thus exposed by UIA as list items. However, as far as screen readers are concerned, the items (or the labels for them) are images, which indicates poor UIA (Ui Automation) implementation. In contrast, the same UIA items used to show feedback in Windows Feedback app are coded correctly - feedback text are spoken by screen readers. This comparison shows that either something is going on with UIA, or Twitter developers might want to optimize certain parts of the app to make it shown correctly to screen readers when they request UIA to show them list item labels.
However, I'm afraid that this is just a tip of an iceberg. The Twitter example is just one of many apps that exhibit behavior that makes me and others wonder if developers thought about accessibility of universal apps in mind when they wrote the code and XAML (eXtensible Application Markup Language), the major components that powers such apps. Unless something is done to ask citizens of universal app ecosystem (users, developers, leaders at Microsoft, observers and others) to think about universal accessibility of universal app architecture and the products that will power or be powered by them (Windows 10 ecosystem and universal apps, respectivley), this app ecosystem will be ignored by current and potential users of Windows 10 who are blind, a minor yet important customer base for Windows 10 and various universal apps. If accessibility is not addressed early, it will come back to haunt us later, and I believe it is important for people to start talking about universal app ecosystem accessibility given Microsoft's proclamation that Windows will be a service and hopes of seeing one billion devices running Windows 10, some of them belonging to users with disabilities.
Thus, I would like to ask Microsoft and its partners, users and developers of universal app ecosystem, other users, screen reader users and developers and others to start thinking about accessibility of universal apps, as this app platform opens new opportunities and will be used by many, including people with disabilities for years to come. To developers of universal apps, as a fellow programmer, Windows Insider and as a user of some universal apps, I would like to ask you to think about accessibility of your apps, as it will mean great reviews by those with disabilities using your apps, which will translate to increase in user base, prominence in the ecosystem and more. To users of universal apps, please remember that, for a population of users who have experienced information blackout, accessibility of an app is one of their top priorities when they wish to join the band wagon. To Microsoft, I would like to seriously ask you to let universal app developers know about options available to make their apps more accessible, such as good use of UIA and learning about accessibility standards and why it is important to promote universal accessibility in their universal apps.
I hope to see the universal app architecture and the ecosystem flourish and increase and prominence, not only due to its potential, but also due to accessibility and efforts put into making universal apps universally accessible. Thank you.
Sincerely,
Joseph S. Lee

Footnotes:
1. Visual Impairment and Blindness, World Health Organization. http://www.who.int/mediacentre/factsheets/fs282/en/
2. NVDA architecture and design overview: http://community.nvda-project.org/wiki/DesignOverview

Thursday, September 3, 2015

NVDA Add-on Internals: SysTray List: Implanting a feature from another screen reader

Hi,
When transitioning between screen readers, one of the first things people might ask is inclusion of familiar features from old screen reader in the new product. As a user who have worked with multiple screen readers, I often ask this question when moving from one screen reader to another.
As more and more people are transitioning from JAWS for Windows to NVDA, a frequently asked question is whether NVDA has a dialog to show system tray (notification area) icons. By default, NVDA does not ship with such a feature, but it can be brought back with an add-on appropriately called "SysTray List". This add-on, developed by Rui Fontes and Rui Batista, allows you to view a list of system tray or taskbar icons, and this is the add-on we'll find out how it works in this add-on internals article.
To download the add-on, go to http://addons.nvda-project.org/addons/systrayList.en.html, and for the source code, go to http://bitbucket.org/nvdaaddonteam/systraylist. Just like the previous series on StationPlaylist Studio, you don't have to use the add-on to learn about its internals, but having the source code and/or using the add-on could help you as you understand how this add-on works behind the scenes.
Important disclaimer: This add-on was not developed by the author of this article, and views expressed in this article are strictly the author's. SysTrayList add-on is copyright Rui Fontes and Rui Batista, Windows API is copyright Microsoft Corporation, NVDA is copyright NV Access, Python is copyright Python Software Foundation.
Introduction to SysTray List add-on features
After installing this add-on, you can press NVDA+F11 to view a list of system tray icons. This dialog lists the icons in the notification area, as well as buttons to cilck, double-click and do right mouse clicks.
A hidden gem of this add-on is that there is no command to view taskbar icons (that command is taken by NVDA's review cursor selection copy command (NvDA+F10)). Instead, to view taskbar icons, press NVDA+F11 twice quickly (more on how this is possible in a second). The list view changes to show taskbar icons (including pinned items). Pressing ENTER will cause NVDA to perform a left mouse click.
Source code layout
This global plugin lives in its own directory named sysTrayList (globalPlugins/sysTrayList/__init__.py). Some add-ons, especially those that use helper modules uses package-style directory structure.
The global plugin file is composed of the following sections:
* Like any Python module, various modules are imported.
* An event executor (more on this below) that sends mouse events specified in the parameter.
* The global plugin (class GlobalPlugin(globalPluginHandler.GlobalPlugin)) which contains the systray script and helper functions.
* The dialog class (globalPlugins.systraylist.SystrayListDialog) for displaying the dialog itself.
NVDA+F11: A behind the scenes tour
So what exactly happens when you press NVDA+F11? When you press this command after installing the add-on, NVDA will do the following:
1. Determines if you pressed this command once or twice by calling scriptHandler.getLastScriptRepeatCount. If you press NvDA+F11 and then you press NVDA+F11 within the next half a second, NvDA will treat this as multiple press of this command.
2. If NVDA sees that you have pressed NVDA+F11 once, NvDA will locate system tray icons, otherwise it'll fetch taskbar icons.
3. The icon locators (both are private functions) will do the following to obtain a list of icons needed:
A. Each locator has a list of window paths (window class names) to be searched to locate the needed icons and their locations, and this list is used by another private function to obtain a list of icon names.
B. The private function will use user32.dll's FindWindowEx to locate the handle to the icons list where the icons live (more on this in the next section).
C. Once the handle is found, NVDA will locate the first icon via NVDAObjects.IAccessible.getNVDAObjectFromEvent function. Then NvDA will use object navigation emulation (object, does something, object = object.next) to locate the icons and store their names and location in a list, which will then be returned to the locator routines.
4. Once the locators obtain the list of icons and their locations, it'll call another private function to open a dialog and show the requested icons. Depending on script count, it'll change the title and the label for the icons list view.
5. After you select an icon and what to do (left click (default), right click, double-click), NVDA will perform the following:
A. NVDA will look up the location of the selected icon.
B. NVDA will perform a series of mouse clicks (mouseEvents function, more on this routine below).
FindWindow versus FindWindowEx and its relation to icon locators
Windows API has changed a lot in more recent versions of Windows. This came as a response to security concerns, to extend API functionality and so on. Because of this, you may see many Windows API functions that ends with "Ex" (short for "extended").
In the original version, FindWindow would return a handle to a window given the class name and child window class name. For example, if we need to obtain a handle to a menu bar in an app, we would use:
hwnd = FindWindow("WindowClassName", "MenuBar")
Specifying NULL (None) for the second parameter would search for the top-level window.
As opposed to FindWindow, FindWindowEx takes two additional parameters, namely the handle to a window where the search should begin and which child window to search (or a group of windows to search). This results in the following signature:
hwnd = FindWindowEx(handle, childGroup, className, childClassName)
For example, the above call to FindWindow could become:
parentHwnd = FindWindow("Desktop", None)
hwnd = FindWindowEx(parentHwnd, None, "WindowClassName", "MenuBar")
The "magic" behind icon name locator function
When asked by NVDA, the icon name locator function (noted above) will try its best to locate the first system tray or taskbar icon. This is how it is done:
1. For element in the path to be searched, it calls FindWindowEx to locate the needed handle. At first, this handle is 0, and search begins from the shell (desktop) object (the root of all windows).
2. Depending on the icons you are looking for, NvDA will search the following windows:
A. For system tray list: "shell_TrayWnd" (desktop object), "TrayNotifyWnd" (notification area), "SysPager" (system tray), "ToolbarWindow32" (first system tray icon).
B. Taskbar icons in Windows XP/Server 2003: "Shell_TrayWnd" (Desktop), "RebarWindow32" (Taskbar), "MSTaskSwWClass" (Taskbar), "ToolbarWindow32"), (first taskbar icon).
C. Taskbar icons in Windows Vista/Server 2008 and later: Shell_TrayWnd" (Desktop), "RebarWindow32" (Taskbar), "MSTaskSwWClass" (Taskbar), "MSTaskListWClass") , (taskbar icon).
3. For each path (window handle if found), this function will ask Windows to search for the next window in the path and store the resulting handle.
Conclusion
Although the idea of listing system tray icons seems easy, there is a lot going on under the hood, involving locating the correct window via Windows API and carefully designing the user interface. This shows that designing a simple add-on like this involves careful thinking, especially if it'll be used by many users around the world. I hope this article was helpful in letting you understand how simple add-on suggestions are developed and designed.
A big hint: you don't have to use this add-on to access list of system tray icons. To access system tray, press Windows+B.
Lastly, I would like to stress the fact that JAWS for Windows and NonVisual Desktop Access are two completely different screen readers. Although they have borrowed features from each other, the underlying philosophy, license type (proprietary and commercial for JAWS versus open-source and GPL for NvDA), design and programming language in use are different. Thus, please do not expect all JAWS functionality suggestions to be investigated by NV Access, or Freedom Scientific to borrow every feature from NVDA.
References:
1. FindWindow (user32.dll) reference (Windows API): https://msdn.microsoft.com/en-us/library/windows/desktop/ms633499(v=vs.85).aspx
2. FindWindowEx (user32.dll) reference (Windows API): https://msdn.microsoft.com/en-us/library/windows/desktop/ms633500(v=vs.85).aspx

Wednesday, August 26, 2015

StationPlaylist Add-on Internals: final thoughts and add-on development process overview

Hi,
Now that we've visited internals of StationPlaylist Studio add-on, I'd like to give you a tour of my lab where I develop this add-on. Along the way you'll learn how an add-on is born, coded, tested, released and maintained.
Lab setup, development equipment and software
For all my software development, I use two computers: a touchscreen laptop and a desktop, both running Windows 10 and latest NVDA next branch snapshots. Both also run Cygwin to run various command-line tools (Git, SCons, etc.), and in case I need to compile NVDA from source code, installed Visual Studio 2012 with latest update and other dependencies.
In case of SPL add-on, I have different Studio versions installed: 5.01 on my laptop and 5.10 on the desktop. This allows me to work on both versions at once (both computers have the full source code of the add-on, though I tend to write bug fixes on my laptop and experiment with new things on my desktop).
Git: a "smart" source code manager
Like other NVDA developers and many add-on writers, I use Git for source code management (contrary to its slogan, Git is very smart). This is a distributed system, meaning a local repository contains the complete record of how the source code is managed (no need to connect to a server to commit and fetch updates). For example, using just my local copy of the SPL add-on source code, I can view commit history and generate older add-on releases.
Another advantage of Git is extensive support for branches. A branch is a development workflow separate from other branches. For example, NVDA screen reader uses at least three branches for its workflow: next (alpha), master (live beta) and rc (release candidate, used to build official releases). SPL add-on uses this approach as well: there are at least two branches in use, called master and stable used for ongoing development or release and maintenance, respectively (we'll come back to branches in a second).
How an add-on feature is born
Let's go through a typical development process for an add-on feature by looking at how broadcast profiles was developed (for more information on broadcast profiles, refer to configuration management article).
I started working on broadcast profiles in March 2015 while developing add-on 5.0. This was a natural extension of add-on settings dialog: whereas this dialog (and the configuration database it uses) only dealt with a single profile, I thought it might be a good idea to allow multiple profiles to be defined and to let the settings dialog respond to profile changes.
There was an important reason for writing this feature: Since NVDA supports multiple configuration profiles and since some broadcasters were hosting multiple shows, I thought it would be a good idea to implement a similar feature in the SPL add-on. Thus, I envisioned broadcast profiles to be used primarily by people hosting multiple shows, with each show defined as a profile.
In March and April 2015, I started rewriting certain parts of add-on configuration manager (splstudio.splconfig) in preparation for developing broadcast profiles (now included as part of add-on 6.0-dev). I started by writing todo comments (where appropriate) describing what the future feature should be like. I then modified initConfig and saveConfig (discussed in app module articles), initially telling them to work with the default profile (the one and only configuration map then), then I left it alone until add-on 5.0 was released in June 2015.
In June 2015, I opened a new branch (initially using the codename "starfish") to house code related to broadcast profiles. Before any "real" code was written, I studied NvDA source code dealing with configuration profiles to learn more about how Jamie (James Teh from NV Access) implemented this feature. Once I understood how it worked, I copied, pasted and changed the code to match the overall add-on code base (giving nV Access the credit they deserve).
One of the first things I had to decide was how to store profiles. I experimented with using ConfigObj sections, one per profile, but this proved to be problematic (a profile could be given the name of an existing configuration map key). I then went back to NVDA source code to find out how NV Access solved this problem (using separate ini files), implemented it, and was met with another problem: transfering values between profiles. This was resolved by specifying whether a setting was "global" (applies to all profiles) or specific to a profile. Next came profile controls in the add-on settings dialog and using choice events to set alarm values using values from the selected profile.
The last thing I did before merging the broadcast profiles branch to master branch in July was revising configuration error dialog and writing documentation for broadcast profiles. Once the documentation was ready and small issues were fixed after going through many rounds of testing (on my own computer and from the profiles branch itself), broadcast profiles branch was merged into master. But the development didn't stop there: thanks to provisions I made, it was quite simple to implement instant switch profiles (again it had issues which are now largely resolved).
Dealing with threaded code: headaches during development of background encoder monitoring feature
You may recall our discussion of Cart Explorer and how it went through extensive testing to arrive at the current state (this was a difficult code segment). When it comes to difficulty, nothing beats multithreaded code, especially if it involves multiple threads working in parallel (rather, almost parallel), and I tasted this when writing background encoder monitor (add-on 5.0). This involved tracking how many threads were running to make sure no unnecessary threads were running, catching suttle errors and race conditions (a connection attempt could run a thread without checking if the encoder is being monitored) and so on. Thankfully, I went through a similar set of problems a few months earlier when I struggled with library scan (add-on 4.0), and that experience taught me to be careful with threads (and to experience fewer headaches).
Add-on development process
Follow me as I show you how a typical SPL add-on version is developed, released and maintained:
1. Before starting work on the new add-on version, I write down some goals the add-on should achieve, including feature ideas, user (your) suggestions and so on.
2. I then hold a conference call with add-on users to see what they think about some ideas and gather feedback (these are also written down).
3. I then create separate branches for each feature in order to isolate  code and not to break existing code.
4. Next, I write todo comments reminding myself as to what the feature should be like, then I start working on it. As each feature is being developed, I do mental simulations as to how you might use the feature under development, such as possible errors, messages spoken and so on.
5. Once the feature is quite stable, I test the feature to uncover bugs and to fill in the missing pieces. When it comes to testing, I test the new feature branch on both of my computers running different versions of Studio to make sure it works across versions (if not, I go back and modify the code to recognize differences between Studio versions).
6. After testing the feature for a while and if the feature is stable, I merge the feature branch into master.
7. Every few weeks, I publish master branch snapshots to gather feedback from users willing to test drive snapshots.
8. At some point, I set release target window for the next add-on version (for 6.0, it is January 2016). This determines when feature freeze should be and beta release window (for 6.0, beta is scheduled for sometime in December 2015). Between feature freeze and the first beta release, I concentrate on code refinements and bug fixes.
9. After going through several beta cycles (typically two), I ask NVDA community add-on reviewers to review my add-on code and request add-on release during the release window (this is done by merging master branch into stable branch).
10. Once the add-on version is released, subsequent maintenance versions (localization updates, bug fixes, minor tweaks) will be released from the stable branch, with the master branch containing the code for the next major version.
11. Once the next version enters beta cycle, further maintenance releases may or may not happen (an exception is long term support release, described below).
Long term support release
A typical add-on version is supported until the next add-on version is released (currently several months). However, there are times when an add-on version receives extended support (termed long term support (LTS) release). This happens if the next major version of Studio is released or a version of Studio with user interface changes is released.
A LTS version is a major version of the SPL add-on with some notable differences:
* Support duration: A LTS version is supported for at least two add-on development cycles (at least a year).
* Features: A LTS version may contain some features from the next major add-on release.
* Studio version supported: A LTS version is the last version to support the oldest supported Studio version. This is designed to give people plenty of time to upgrade to newer Studio releases.
As of August 2015, the most recent (and so far the only) LTS version was add-on 3.x (September 2014 to June 2015). Add-on 3.x was maintained thus:
1. Add-on 3.0 was released in September 2014.
2. Add-on 3.5 could have been the last maintenance version for add-on 3.x if it was not a LTS version.
3. When add-on 4.0 was released (January 2015), add-on 3.6 was released, backporting some features from 4.0. Users were told that add-on 3.x will be the last version to support Studio versions earlier than 5.00. From that time on, add-on 3.x was taken off the stable branch and was moved to an internal branch.
4. When add-on 5.0 beta was released (May 2015), add-on 3.x (3.9 was available then) entered end of support countdown (no more maintenance releases).
5. A few weeks later, when add-on 5.0 came out (June 2015), add-on 3.x became unsupported.
Final thoughts
As I close this series of articles on StationPlaylist Studio Add-on Internals, I feel it is time I reveal why my add-ons are free: it is because I love you users and as a service for NVDA user and developer community (and in extension, to all blind broadcasters using SPL Studio). What brings me joy as an add-on writer is the fact that this add-on (and accompanying documentation) has made impact in your lives and lives of listeners to your shows, as well as to other NVDA users and developers around the world. Thank you users for your continued support and feedback, and I promise once again that all my add-on code (including SPL Studio add-on) will be free and anyone is welcome to study and improve upon it.
For add-on writers looking for quality add-on documentation, I hope this series gave you an inspiration as to how to write amazing documentation in your future projects. For people new to add-on writing or for those interested in writing an add-on, I hope this Add-ons Internals series served as a handy resource for your projects, and in extension, gave you an idea as to how certain NVDA functions work. If you'd like to reference this documentation or use it as a blueprint, you are more than welcome to do so. Thank you community add-on reviewers for your continued support and reviews.
Important notices and credits
I'd like to thank StationPlaylist staff for continued collaboration with screen reader users in regards to accessibility of Studio. A special thanks goes to Jamie Teh from NV Access and Geoff Shang (original add-on author) for giving me and others a foundation for future goodies. As always, the biggest thanks goes to you, the users of SPL add-on for your continued feedback and teaching me new things about studio.
Source code notice: to protect copyrights, parts of Studio API has not been documented. Also, source code discussed throughout this series may change as future add-on versions are developed.
Copyrights: StationPlaylist Studio, Track Tool and StationPlaylist Encoders are copyright StationPlaylist.com. NonVisual Desktop Access is copyright 2006-2015 NV access Limited (released under GPL). SAM Encoders is copyright Spatial Audio. Microsoft Windows and Windows API are copyright Microsoft Corporation. Python is copyright Python Software Foundation. StationPlaylist Studio add-on for NvDA is copyright 2011, 2013-2015 Geoff Shang, Joseph Lee and others (released under GPL). Other products mentioned are copyrighted by owners of these products (licenses vary).

StationPlaylist Add-on Internals: encoder support

Hi,
We have now arrived at the penultimate article in this Add-on Internals series for StationPlaylist add-on: encoder support, the second pillar for the SPL Utilities global plugin. We'll talk about how encoder support is implemented, how NVDA can detect stream labels and a behind the scenes overview of what happens when you connect to a streaming server.
Encoder support: From suggestion to implementation
Originally, I wasn't planning on including encoder support into the SPL add-on. However, after talking to some Studio users who were using SAM encoders and seeing how other screen readers supported it, I decided to investigate SAM encoder support in summer 2014.
The first issue I had to solve was making NVDA recognize the encoder entries themselves. Once that was solved, the next task was announcing connection error messages, which led to figuring out how SAM encoders react when connected to a streaming server.
Originally, I manipulated text written to the screen to obtain needed status messages (via text infos). This routine caused some to experience screen flickering issues when connecting to a streaming server. This was resolved by using encoder description (obj.description), which opened up a possibility to monitor changes to this text via a background thread (more on this routine below), which also eliminated a need to stay on the encoders window until connected.
While I was resolving problems with SAM encoders, I also worked on refactoring encoder support code to support StationPlaylist encoders (add-on 4.0). Initially, encoder support code was optimized for SAM encoders, but the current code structure (explained below) was written to extend basid encoder support easily, and as a result, both SAM and SPL encoder entries present similar interfaces and commands.
Encoder entries: Yet another overlay class family
Just like Studio track items (see the article on track items), encoder entries are overlay classes. Each encoder type (SAM or SPL) inherits from a single encoder object (SPLStudioUtils.encoders.EncoderWindow) that provides basic services such as settings commands, announcing stream labels and so on. Then each encoder type adds encoder-specific routines such as different connection detection routines, ways of obtaining stream labels and so on. Speaking of stream labels and settings, the base encoder class is helped by some friends from the encoder module itself, including a configuration map to store stream labels and basic settings, a routine to obtain encoder ID (encoder string and the IAccessible child ID) and so on.
On top of the base encoder class are two encoder classes, representing SAM encoder entries and SPL encoder entries. SAM encoder entries (SPLStudioUtils.encoders.SAMEncoderWindow) is laid out just like Studio's track items, whereas SPL encoder entries (SPLStudioUtils.encoders.SPLEncoderWindow) is a typical SysListView32 control (see an article on column routines for more information). Both classes provide similar routines, with the only difference being how connection messages are handled.
Common services: basic settings, stream labels and related methods
All encoder classes provide the following common services:
* Configuring settings: three settings can be configured:
A. Pressing F11 will tell NVDA if NVDA snuld switch to Studio when the encoder is connected.
B. Pressing Shift+F11 will ask Studio will play the next track when connected.
C. Pressing Control+F11 will enable background encoder monitoring (more on this in a second).
D. Once these settings are changed, the new values will be stored in appropriate flag in the encoder entry, which in turn are saved in the configuration map.
* Applies settings. This is done by initOverlayClass method - once called, this method will look up various settings for the encoder from the configuration map (key is the setting flag, value is the encoder ID). Another job of this routine is to load stream labels when an encoder first gains focus (if this was loaded earlier, it could be a waste of space, especially if encoders are never used).
* Announces stream labels (if defined) via reportFocus method. In contrast with the Studio track item version, an encoder's reportFocus routine:
A. Locates stream labels for the current encoder (the configuration map stores stream labels as dictionaries (sections), with each dictionary representing the encoder type, key is the encoder position and the value is the label; each encoder, when told to look up stream labels, will consult its own labels dictionary).
B. If a label is found, NVDA will announce the label (in braille, surrounded by parentheses).
* Define and remove stream labels. This is done via stream labels dialog (F12) that'll make sure you entered a label (if not, the encoder position is removed from the encoder-specific stream labels dictionary).
* Updates stream label position when told to do so (via a dialog, activated by pressing Control+F12). This is needed if encoders were removed, as you may hear stream label for an encoder that no longer exists. This is implemented as a variation of find predecessor algorithm.
* Announces encoder columns. The base class can announce encoder position (Control+NVDA+1) and stream label (Control+NVDA+2), while SAM can announce encoder format, status and description and SPL allows one to hear encoder format and transfer rate/connection status.
Encoder ID's
An encoder ID is a string which uniquely identifies an encoder. This consists of a string denoting the encoder type (SAM for SAM encoder, for instance), followed by the encoder position (separated by a space). For instance, the first SAM encoder is given the ID "SAM 1". The ID's are used to look up stream labels, configure settings and to identify encoders being monitored (SPL Controller, E).
More and more threads: connection messages and background encoder monitoring
As we saw in a previous article, threads allow developers to let programs perform certain tasks in the background. Even in encoder support, threads are employed for various tasks, including connection message announcement and background encoder monitoring.
Each encoder overlay class (not the base encoder) includes dedicated connection handling routines (reportConnectionStatus). Depending on how you invoke this, it starts up as follows:
* If background encoder monitoring is off and you press F9 to connect, NVDA will run this routine in a separate thread. For SAM, this is checked right after sending F9 to the application, and for SPL, this is done after clicking "connect" button (manipulates focus in the process).
* If background encoder monitoring is on before pressing F9, the routine will run from another thread when this setting is switched on. Then when you press F9, NvDA knows that the background monitoring thread is active, thus skipping the above step.
The connection handling routine performs the following:
1. Locates status message for the encoder entry. For SAM, it is the description text, and for SPL, it is one of the entry's child objects (columns). This will be done as long as Studio and/or NVDA is live (that is, if the thread is running).
2. Announces error messages if any and will try again after waiting a little while (fraction of a second).
3. If connected, NvDA will play a tone, then:
A. Do nothing if not told to focus to studio nor play the next track.
B. Focuses to studio and/or plays the next track if no tracks are playing.
4. For other messages, NVDA will periodically play a progress tone and announce connection status so far as reported by the encoder.
5. This loop repeats as long as this encoder is being monitored in the background.
Encoder-specific routines
In addition to basic services, each encoder routine has its own goodies, including:
For SAM encoders:
* To disconnect, press F10.
* You can press Control+F9 or Control+F10 to connect or disconnect all encoders (does not work well in recent SAM releases, according to my tests).
For SPL encoders:
* When you press F9 to connect, NVDA does the following:
1. Locates "connect" button, and if it says "Connect", clicks it (obj.doAction).
2. Moves focus back to the entry (self.SetFocus).
* To disconnect, press TAB until you arrive at "Disconnect" button and press SPACE.
Conclusion
We've come a long way: from add-on design and app module contents to encoder support, we discussed internals of a project that makes a difference in lives of many blind broadcasters around the world. I learned many things developing this add-on, and I hope to improve this add-on in the future (which also means publishing future editions of this internals series as new add-on versions are released). To put it all together, let me give you a tour around my lab where I think about, code, test and talk about StationPlaylist Studio add-on for NVDA (and in extension, NVDA screen reader itself).

StationPlaylist Add-on Internals: introduction to SPL Controller layer and focus to studio window routine

Hi,
Now that we've covered the "kernel" (innermost parts) of the Studio add-on, it is time to talk about the icing: SPL Utilities global plugins and its contents. The next few articles will talk about what the global plugin does, introduce you to inner workings of SPL Controller layer and tour how encoder support is implemented.
Studio app module versus SPL Utilities global plugin
As described in the add-on design article, SPL Studio add-on comes with two app modules and a global plugin. This was needed not only to differentiate between module types and expertese, but also allow Studio functions to be invoked from other programs. With the introduction of encoder support in add-on 3.0 (fall 2014), the global plugin portion of the add-on (SPL Utilities) took on an important role: working as an encoders manager to report connection status and to perform other routines.
SPL Utilities package contents
The SPL Utilities global plugin consists of the following modules:
* Main plugin code (__init__.py), containing essential global routines such as SPL Controller (described below) and a procedure to focus to Studio window upon request. This module defines constants used by Studio to receive messages, a function to focus to Studio window and the global plugin class containing definitions for SPL Controller layer commands.
* Encoder support (encoders.py), outlining NVDA's support for various encoders (see the next article; the main global plugin module plays an important part in helping the encoder module as you'll see in the next article).
SPL Controller layer
The SPL Controller layer (entry command unassigned, same reason as the Assistant layer entry command) is used to invoke Studio functions from anywhere. The entry routine is similar to the app module counterpart (SPL Assistant) except for the following:
* NVDA will make sure Studio is running (if so, it'll cache the window handle value just as in the Studio app module), otherwise it cannot enter SPL Controller layer.
* All commands (except two) use Studio API (Studio API and use of user32.dll's SendMessage was described in a previous article).
For mechanics of layer commands, see a previous article on add-on design where layer commands were discussed.
The following commands utilize Studio API:
* A/Shift+A: Automation on/off.
* L/Shift+L: Line in on/off.
* M/Shift+M/N: Microphone on/off/instant on/off toggle.
* P: Play.
* R: Remaining time for the currently playing track (if any).
* Shift+R: Library scan progress and umber of items scanned.
* S/T: Stop with fade/instant stop.
* U: Play/pause.
For readers familiar with Studio keyboard commands, you'll find yourself at home (they are indeed Studio commands except pressing Shift will turn a feature off and Shift+R will remind you of Control+Shift+R for library scan from Insert Tracks dialog).
Here are the two exceptions
* E: If you tell NVDA to monitor one or more encoders in the background, this command will announce number of encoders being monitored (see the next aritlce on the format of this message).
* F1: Opens a dialog displaying Controller layer commands (does this sound familiar?).
Focusing to Studio window from anywhere
As you are broadcasting a show with Studio, you may find yourself in a situation where you need to switch to Studio quickly to take care of automation, insert new tracks and so on. An ideal situation is to switch to Studio when you press Alt+TAB (this isn't the case if you have more than two programs opened). For this reason, screen reader scripts for Studio includes a command to switch to Studio upon request (unassigned in NVDA).
In NVDA world, this is accomplished with a function in the SPL Utilities module (SPLStudioUtils.fetchSPLForegroundWindow). This is employed not only by the main global plugin module (called from a script to focus to Studio window), but also used in encoders for various functions. The routine is as follows:
1. The focus to Studio script will check if Studio is running, and if so, it'll call the fetch window function, which in turn locates the desktop (shell) window to serve as the starting point for locating Studio window.
2. NVDA will scan top-level windows (children of desktop object) until a Studio window (where the window's app module is the Studio app module) is found, and if found, NVDA will increment a Studio window candidate counter.
3. Once top-level window scanning is complete, NVDA will take action based on what the Studio window candidate counter says before passing the foreground object back to the main script. It can do one of the following:
A. If counter is 0 (fg is None), NVDA will know that you have minimized Studio, so it'll tell you that Studio is minimized.
B. If counter is 1, NVDA will locate the Studio window by looking for the main Studio window (user32.dll is involved).
C. For all other values, NVDA will assume the last window found is the Studio window (held in fg variable) and return it.
4. Back at the focus to Studio script, NvDA will either announce if Studio is minimized or switch to the foreground window returned by the fetch window function (fg.SetFocus).
Conclusion
The routines discussed above (SPL Controller and the command to switch to Studio window) is one of the two pillars of the SPL Studio Utilities global plugin (the other is encoder support). With these routines, it became possible to perform playback operations without focusing to studio, and you can switch to Studio window from anywhere, anytime. We'll visit the other side of this global plugin story in the next StationPlaylist Add-on Internals article, and after that, we'll conclude with an interview with the maintainer of the add-on to learn about how he (I) develop new add-on features.

Monday, August 24, 2015

StationPlaylist Add-on Internals: configuration management, settings dialog and broadcast profiles

Hi,
We have arrived at our last station stop for Studio app module internals: configuration management. This facility allows a broadcaster to configure various options such as alarms, column announcement order and so on, as well as package settings for a show as a broadcast profile to be invoked during the show. Along the way you'll learn how NVDA screen reader stores various settings, what happens if something goes wrong and get into internals of how broadcast profiles work.
ConfigObj: NVDA's configuration manager assistant
NVDA uses ConfigObj library to manage settings. This Python module, inspired by Python's own Config Parser, allows developers to store settings in a text file, read and interpret settings and validate options against default configuration options.
NVDA comes with a collection of default options. They live in source/config/__init__ and are used for various things, including presenting preferences, validating user configuration and so on. The config management module also includes facilities to handle profiles (a package of user settings to be used in an app, during say all or reserved for manual activation).
NVDA configuration profiles: multiple configuration files, one online database
A number of users asked NV Access if it would be possible to have profiles where certain settings can take effect while one is using apps or during say all. NV Access listened and introduced configuration profiles in late 2013. As of August 2015, one can create a manual or an automated (triggered) profile, with the latter further divided into say all profile and app-specific one.
Configuration profiles involve a few support routines and a careful coordination between configuration files. In essence, each configuration profile (stored in profiles folder in user configuration folder) is a snapshot of differences between the profile and the main user configuration file (named nvda.ini). When a profile becomes active, NVDA will load the profile file associated with the given profile and modify user settings according to values stored in the newly activated profile, and wwill record the name of the profile file to remind itself as to which profile is active (the default user configuration profile is named "normal configuration" with the file name of nvda.ini).
What if settings had errors? As part of the startup routine (portions of main function (source/core.py) prior to entering the main loop), NVDA will display a configuration error dialog if it detects serious issues with configuration values (in reality, ConfigObj notifies NVDA of this problem). You'll see this is also implemented in the Studio app module to deal with add-on configuration issues.
All about Studio add-on Configuration Manager
Until a few months ago, Studio app module handled all add-on configuration routines. With the advent of add-on 5.0 which introduced add-on settings dialog, configuration management routines were split into a dedicated Configuration Manager (splstudio.splconfig). The new module takes care of configuration routines, including validating the user configuration, presenting add-on settings dialog and other dialogs inside it, handling broadcast profiles and more. We'll cover each routine in this article.
How settings are loaded, used and saved
As mentioned in an article on life of the Studio app module, one of the things the app module does is load the add-on configuration database by calling splconfig.initConfig function. The job of this function is to load the add-on configuration map from various places (for add-on 5.x, it will be the main configuration map only, while 6.0 also searches appModules/profiles folder to load broadcast profiles). The format of the configuration file is that of a typical ini file, and as far as NVDA is concerned, it is a dictionary.
When the configuration database is ready, Studio app module will then use values stored in this settings dictionary to perform various tasks, including microphone alarm, announcing listener count and so on. If multiple profiles are defined, NVDA will start with the first configuration map (normal profile), and the active profile is denoted by splconfig.SPLConfig map (more on profiles in a moment).
After you are done using Studio, close Studio so settings can be saved to disk. This involves saving individual profiles, copying global settings to the normal profile and saving the normal profile to disk.
The Studio Add-on Settings Dialog
Studio app module allows you to configure various settings in two ways: via a shortcut key (discussed in an article on configuring basic settings) or via the settings dialog. When you use a shortcut key to change settings, NVDA will look up the value for the setting, change it, announce the new setting and store the newly changed value in the settings map.
Alternatively, you can configure settings via the add-on settings dialog (Control+NVDA+0). As it is a settings dialog (powered by gui.SettingsDialog), it will look just like any NVDA preferences dialog. For some advanced options, this dialog is the only gateway to access them (covered below).
The add-on settings dialog (splconfig.SPLConfigDialog) contains following options:
* Broadcast profile controls (add-on 6.0 and later): inspired by NVDA screen reader's configuration profiles dialog, this group of controls shows a list of profiles loaded and buttons to create a brand new profile or a copy of an existing profile, rename and delete profiles. It also contains a button (really a toggle button) that tells NVDA to switch to the selected profile upon request (more on this in a second).
* Global settings: these are settings not affected by profiles. These include status announcements, column announcement order, announcing listener count, toggling Track Dial, library scan options and so on.
* Profile-specific settings: Currently alarm settings are profile-specific. These are end of track alarm and the option to use this alarm, song ramp (intro) time and the setting to use this alarm, and microphone alarm. In case of microphone alarm, Studio stores this value as an integer in the map while it is shown as a string in the add-on settings dialog. For other alarm settings, it is a spin control (wx.SpinCtrl; use up or down arrow keys to change them).
* Reset settings: NVDA will ask if you wish to reset settings in the currently active profile back to factory defaults. This is done by using a function in splconfig module (splconfig.resetConfig) that will set current profile values to defaults (a default configuration map is included for this purpose; this map uses a configuration specification (confspec, part of defining validation routine via validator module (a close friend of ConfigObj), and this confspec is defined in the splconfig module).
When you press Control+NVDA+0 from Studio to open this dialog, the following will happen:
1. Just like alarm dialogs (see previous articles), NVDA will make sure no other dialogs are opened.
2. It'll then call a function in splconfig module, which in turn will prepare the dialog to be shown.
3. The preparation routine (SettingsDialog.makeSettings) will populate the dialog with controls and options associated with each control, with current options coming from configuration values stored in the active profile.
4. Once the dialog is ready, it'll pop up and you'll land on the status message checkbox or list of active profiles depending on the add-on version (former is 5.x, latter is 6.0). You can then use typical dialog navigation commands to navigate through various options.
After configuring some settings, click OK. NVDA will then locate the selected profile and tell SPLConfig to use this profile, then store options from the settings dialog into the configuration map. There is one showstopper, however: if you enter a wrong value into the microphone alarm, NVDA will tell you to enter something else and will return you to microphone alarm field.
In case you discard new settings (clicking Cancel), NVDA will check to see if an instant switch profile is defined, and if so, it'll perform various operations depending on whether the instant profile has been renamed or deleted.
All about broadcast profiles
In Studio app module world, a broadcast profile (usually shortened to profile) is a group of settings to be used in a show. This is achieved by using a configuration profile list (splconfig.SPLConfigPool) for storing these profiles, and one of them is used at any given time (by default, the first profile).
There are two ways of creating a profile: brand new or as a copy. Both uses the same dialog (splconfig.NewProfileDialog), with the difference being the base profile in use. For a brand new profile, settings from the normal profile will be used (minus profile-specific settings, which are set to default values), and for a copy, the new profile will contain all settings from the base profile. In both cases, a memory resident profile will be created and initialized just like other profiles (splconfig.unlockConfig, taking the name of the new profile as a parameter); this was done to reduce unnecessary disk writes. Also, new/copy profile dialog (and other dialogs invoked from the main configuration dialog) will disable the main settings dialog.
In case the selected profile is deleted, the profile will be removed from the profiles list, the configuration file associated with the profile will be deleted (if it exists) and normal profile will be set as the active profile. In case of a rename operation, it'll look for a profile with the old name and change some properties to reflect name change. There is an important step the app module will performed if an instant switch profile is renamed or deleted (if renamed, the instant profile variable will hold the new name, and if deleted, instant profile value will be None).
How does instant profile switching work
An instant switch profile is a profile to be switched to if told by the user. This is used before you connect to a streaming server to load settings appropriate for a show (as of time of this writing, only one can be selected as an instant switch profile; to define this profile, select a profile to be used as a show, then go to profile switching button and select it).
Once a profile is defined as an instant switch profile (via add-on settings dialog), invoke SPL Assistant layer and press F12. NVDA will call splconfig.instantProfileSwitch, which will perform the following:
1. Performs some checks, including:
* Checks if an instant switch profile is defined.
* If a profile is defined, it'll make sure you are not using the instant switch profile yet.
2. Saves the index of the active profile.
3. Locates the name of the instant switch profile and the profile associated with it and switches to the switch profile (reassigns SPLConfig to use the switch profile).
4. If told to return to the previously active profile, it'll tell SPLConfig to use the previously active profile (the index for the previously active profile is located and is used to pull the profile with the given index from the config pool).
Conclusion
Configuration management, if used right, could open up new possibilities and allow you to tailor the program to your liking. Studio add-on for NVDA is no exception to this idea: the configuration management routines discussed above allows you to configure how NVDA acts while using Studio, create profiles to be used in a show and so forth. In essence, all the features we talked about so far (library scan, Track Dial and so on) cannot become personal unless ability to configure settings is in place, and with broadcast profiles, the fun is just beginning.
This article concludes a detailed tour of Studio app module internals. The rest of this series will focus on SPL Studio Utilities global plugin, encoder support and a few thoughts on how the add-on is developed, starting with a tour of SPL Controller layer commands.
References:
1. Configparser (Python documentation, Python Software Foundation): https://docs.python.org/2/library/configparser.html
2. ConfigObj documentation: http://www.voidspace.org.uk/python/configobj.html
3. Validate module documentation: http://www.voidspace.org.uk/python/validate.html
4. Spin control (wx.SpinCtrl) documentation (WXPython): http://wxpython.org/Phoenix/docs/html/SpinCtrl.html

StationPlaylist Add-on Internals: All about SPL Assistant layer commands

Hi,
You may recall visiting two layer command sets in a previous article: SPL Controller and SPL Assistant, the former used to perform Studio functions from any program and the latter for status announcements. I mentioned throughout this series that we'll tour these layer sets, and we'll start with SPL Assistant layer.
Talk about layer commands
One of the common features of Studio scripts for JAWS, Window-Eyes and NVDA is extensive use of layer commands. This was popularized by JAWS and its Studio layer (grave key). Some of the benefits of this approach include saving keyboard commands, reminding users as to commands available in Studio and so on.
Birth of SPL Assistant layer
As mentioned previously, since version 1.0 in January 2014, Studio add-on comes with two layer commands to represent the global plugin and the studio app module. In case of Studio app module and its layer set (SPL Assistant), I borrowed some commands from both JAWS and Window-Eyes scripts with some notable differences, namely some commands and how things were announced.
When I sat down to design this layer set, I felt it would be helpful for broadcasters if most of the Assistant layer commands borrowed from Studio command assignments. For example, a broadcaster will press M to toggle microphone on and off, and in SPL Assistant layer, pressing M announces microphone status. Another example was Cart Edit Mode - pressing Control+T in Studio will toggle this, and pressing SPL Assistant, T will announce whether this mode is on or off (the reason for assigning T for Cart Edit Mode status will be discussed later).
Originally, one could invoke SPL Assistant layer by pressing Control+NVDA+grave key from within Studio. However, some NVDA translators told me that this key combination is used for NVDA screen reader commands in their language. Thus, in add-on 2.0 (late spring 2014), I decided to remove this command, which means in order for you (broadcasters) to invoke SPL Assistant layer, you need to go to Input Gestures dialog while focused in Studio, expand StationPlaylist category and look for the Assistant entry (I personally use Control+NvDA+grave, and in recent add-on development builds, I told Studio add-on to let SPL Controller layer command (discussed in a future article) to invoke Assistant layer).
Categorizing SPL Assistant commands
Once you invoke SPL Assistant layer (a beep will be heard), you can perform one of the following operations:
* Status announcements (automation, microphone, etc.).
* Tools (library scan, track time analysis and so on).
* Configuration (switching broadcast profiles).
* Ask for help (opening SPL Assistant help dialog or the online user guide).
For the first two categories, they can be divided further into commands which uses studio API (via statusAPI function discussed in a previous article) and those relying on object navigation (multiple components are involved and is sensitive to user interface changes). We'll go through each of these categories in turn.
SPL Assistant 1: status announcements
These commands allow you to obtain various status information such as title and duration of the next track, cart edit mode status and so on. These can be divided into those which uses object navigation (old style) and Studio API (new style) commands.
The following commands (sorted alphabetically) utilize Studio API to perform needed functions:
* D: Remaining time for the opened playlist.
* H: Duration of tracks in the selected hour.
* P: Playback status.
Revisiting the past: object navigation
Before the new style routines were written, all commands used object navigation. Typically, the command will use a helper function and an object map to locate the needed object and will announce what you are looking for (typically obj.name or an error message). The process was as follows:
1. The Studio app module contains a map of indecies where the object lives in relation to the foreground window. For example, for a given object, if index was 0, NVDA nows that the object is the first child of the foreground object. Technically, it is a dictionary of lists, with each list item (indecies) corresponding to the version of Studio the add-on supports.
2. To fetch needed objects and to record the command type, a number of constants are defined in the app module (all integers, denoting what you wish to hear). These constants serve as keys to the object index map.
3. After entering SPL Assistant layer and once you press one of the commands below, NVDA will do the following:
A. Each command will obtain the object in question by calling object fetcher (status function) with the announcement type as the parameter (status(SPLConstant; for example, for cart edit mode, the signature is self.status(self.SPLPlayStatus), with the constant denoting a status bar).
B. The object fetcher (status function) will first consult an object cache (part of the Studio app module) hoping that the needed object is ready for use (for performance reasons).
C. If the object was not cached, the fetcher will first write down the foreground window, then use the directions specified in the object index map (the constant passed into the status function is the key to this map and different values are returned based on Studio version) to locate, cache and return the object in question (in that order).
D. Back at the routine for the command, it is up to the routine as to what to do with it (sometimes, the actual object is a child of the just returned object).
The commands which utilizes object navigation steps above include:
* A. Automation.
* C: Title of the currently playing track.
* Shift+H: Duration of selected tracks in the current hour slot.
* I: Listener count (I have tried using Studio API to obtain this information, but after experimenting with it, object navigation routine was more stable).
* L: Line in.
* M: Microphone.
* N: Title and duration for the next track.
* Shift+P: Track pitch.
* R: Record to file.
* S: Track scheduled for.
* T: Cart Edit Mode (I assigned T to this command for efficiency reasons).
* U: Studio up time.
* W: Weather and temperature (if configured).
* Y: Playlist modification.
For example, if you press A to obtain automation status from Studio 5.10:
1. Invoke SPL Assistant, then press A.
2. The status function (object fetcher) is called, taking the status bar constant (SPLPlayStatus, which is 0) as the key to the index map.
3. Object fetcher will see if the status bar object (cache dictionary with the key of 0) has been cached. For this example, it isn't.
4. Seeing that the status bar isn't cached, object fetcher will now look at the index map and will decide which column to read (for this example, it is column 2 (index 1)). The column records the child position of the status bar relative to the foreground window (in our case, index is 6 or the seventh child).
5. Once the child object position index is obtained, object fetcher will locate the actual object and cache it (self._cachedStatusObjs[infoIndex] = fg.children[statusObj]), then returns the object to the automation announcement routine.
6. Back at the script routine, NVDA will be reminded that it needs to look at one of the object's children (status bars can contain child objects if exposed by accessibility API's), then will announce one of it's contents (second child object, which records automation status).
SPL Assistant 2: tools
These are miscellaneous commands in SPL Assistant, and all of them (except one) uses Studio API:
* Shift+R: Library scan. This is a convenience function to start library scan in the background, useful if you have added new tracks from a number of folders via Studio Options dialog. Consult a previous article on library scan for details on library scan internals.
* F9: Marks the current position of the playlist as start of track time analysis (more on this feature below).
* F10: Performs track time analysis (add-on 6.0).
Track time analysis: Duration of "selected" tracks
A few months ago, during a Skype chat with a number of add-on users, someone suggested a feature where NvDA will tell you how long it'll take to play selected tracks. Since I was familiar with this concept from JAWS scripts, I decided to work on it as part of add-on 6.0.
The resulting routine (which is available if you are focused on the main playlist viewer with the playlist loaded) is as follows:
1. Move to the position in a playlist to mark as start of track time analysis.
2. Enter SPL assistant, then press F9.
3. Move to another track in the playlist, open SPL Assistant then press F10. NVDA will then:
A. Determine analysis range. For most cases, it'll be top to bottom analysis, but in some cases, it could be reverse (bottom to top). Also, a variable to hold total duration will be prepared.
B. For each track in the analysis range, NvDA will obtain file name and track duration via Studio API. Once the track duration is received, it is then added to the total duration variable.
C. Once time analysis (calculating total duration) is done, NVDA will announce number of tracks selected and the total duration using mm:ss format.
If you are a seasoned NVDA user, you may have noticed a familiar pattern: the command to set a marker to copy review cursor text is NVDA+F9, and you would move to a different location and press NvDA+F10 to copy the selected text to the clipboard. Replacing the NvDA modifier key with SPL Assistant produces the commands above: F9 to mark current position for time analysis, and F10 to perform the actual calculation. I intentionally chose these two function keys to provide consistent experience and to reenforce concepts used in NvDA screen reader: review cursor.
SPL Assistant 3: configuration
There is another function key assigned to SPL Assistant: pressing F12 will switch to an instant switch profile (if defined). We'll come back to what is meant by "instant switch profile" and the mechanics of it (and internals of SPL Assistant, F12) in the next article.
SPL Assistant 4: getting help
I believe that a product isn't complete without a good quality documentation. For this reason, SPL Assistant provides two commands to help you use the layer commands or the add-on itself. They are:
* F1: Displays a dialog presenting a list of SPL Assistant layer commands.
* Shift+F1: Opens the online user guide (os.startfile).
A surprise: some Assistant layer commands can be invoked without entering the layer first
There are times when a broadcaster will need to obtain certain information quickly. So the question becomes, "is there a way to announce something without first invoking Assistant layer?" Yes, you can assigg a custom command for the following Assistant commands:
* Name of the next track.
* Name of the current track.
* Weather and temperature.
* Track time analysis marker.
* Track time analysis.
For these routines, an extra step is performed to make sure that SPL Assistant flag is turned off automatically after the shortcut for these routines are pressed. Without this step, you might end up with a situation like the following:
1. You invoke Assistant layer.
2. You then press the shortcut key (not the layer counterpart) for the layer command you wish to use.
3. You press another key which may announce something else, or you hear the same thing twice if you do press the layer command counterpart to the command you have pressed. In effect, you have invoked two layer commands in one sitting (the purpose of the layer set is to let you hear one announcement at a time).
Conclusion
By now you should have a better understanding of how SPL Assistant layer commands work. You also learned that some commands do not require you to invoke the Assistant layer, and toured the necessary infrastructure to support certain layer commands. We'll visit the Controller layer commands in a future article. Our next station stop is an overview of add-on configuration management internals, including how NVDA can remember various settings, present options in add-on settings dialog and an introduction to broadcast profiles.
References:
1. Cache (Wikipedia): https://en.wikipedia.org/wiki/Cache_(computing)
2. Os (Python documentation, Python Software Foundation): https://docs.python.org/2/library/os.html

Saturday, August 22, 2015

StationPlaylist Add-on Internals: The magic behind Cart Explorer

IMPORTANT: The following article superseeds a previous blog entry (summer 2014) regarding this subject.
Hi,
A live radio broadcast would not be complete without jingles. This can range from station promotions (often called "promos"), advertisements, jingles to convey the mood of a show, segment jingles and more. Many station automation programs, including StationPlaylist Studio includes facilities to manage jingles (sometimes called carts), including defining a cart to be played when cart keys are pressed, announcing the name of the playing cart and saving uer specific carts to a safe location.
For blind broadcasters, one of the things they worry is pressing a wrong jingle key by accident, thus script writers were asked to implement a way for broadcasters to learn which carts are assigned to cart keys. As of time of this post, all three screen readers (JAWS for Windows (script author: Brian Hartgen), Window-Eyes (script author: Jeff Bishop), NVDA (script author: Joseph Lee (I, the author of this article)) includes a feature to learn jingle assignments. As this is a series of articles on internals of an NVDA add-on, I'll give you an overview of what happens when you activate and explore cart assignments (in fact, this section was the most interesting and feedback driven portion of the add-on). Along the way you'll learn where Cart Explorer (NVDA's version of cart learn mode) draws its power and why it is very important.
Carts in StationPlaylist Studio
Studio comes in three editions: Demo (same as Pro but for limited time trial), Standard and Pro. The first user visible difference between Standard and Pro is number of cart assignments: Standard can store 48 jingles, while Pro can work with 96 of them.
To play jingles, a broadcaster would use Cart Edit Mode Control+T), then assign a hotkey to a file. For Studio Standard, you can assign F1 through F12 by themselves or in combination with Control, Shift or Alt. In Demo and Pro, number row can be assigned (1 through 9, 0, hyphen (-) and equals (=) either by themselves or in combination with Control, Shift or Alt, for a grand total of 96 jingles). Once jingles are assigned, they will appear under cart menus (there are four cart menus, one for standalone keys (called main) and one each for Control, Shift and Alt).
Where does Studio store carts?
Studio's "carts" are housed in Studio installation folder. There are four cart files (called banks) in use: a .cart file for each of the cart banks (main, Shift, Control, Alt). During normal business hours, Studio will work with these four banks unless told by a broadcaster to load carts from a different cart bank file.
Cart Explorer: my own Summer of Code
It was a hot day in June 2014 when I sat down to design a way to let broadcasters learn cart assignments. Since I was developing add-on 3.0 back then, I decided that this feature should be a top priority feature to be included in the upcoming release.
When I started writing this feature, the first thing I thought about was its name. I felt "cart learn mode" didn't really convey the picture - after all, I reasoned that broadcasters will use this feature to explore cart assignments. Thus the name "Cart Explorer" was chosen - in effect, when you use this feature, you are browsing jingle assignments in preparation for a show.
Next, I read JAWS scripts documentation to get a glimpse of how Brian has managed to implement cart learn mode. In JAWS scripts, script settings are stored in the user configuration directory (typically this is %systemdrive%\Users\%username%\AppDate\Roaming\Freedom Scientific\JAWS\%JAWSVersion%\Settings\Enu; Brian, please correct me if I'm wrong). A section of this script configuration file is dedicated to carts, and JAWS scripts use a map of key names and cart values to announce cart information while cart learn mode is active.
Based on this information, I started writing an ini file parser, seeing that broadcasters would store cart assignments in a configuration database. This was prone to a number of errors, including wrong cart name format, nonexistent cart key assignment, invalid configuration format and others. I once wrote a blog post (on this blog) explaining how this worked (times have changed, as you'll see very soon).
Then I became curious as to how Studio stores its own cart banks, and naturally, I opened the folder where carts were stored and opened each .cart file in Notepad++ (a very handy text editor). From reading the cart bank format (explained below), I thought it might be useful to write a cart bank file parser. Thus I resumed writing Cart Explorer routines, this time incorporating the cart bank format, not forgetting to handle suttle errors, and this is the routine used in add-on releases up to 5.x (6.0 uses a completely different yet related routine, as you'll see).
While writing the first version of Cart Explorer, I realized that this feature needed some real life testing, so I asked a seasoned blind broadcaster to test this feature. We spent a better part of Independence Day writing, debugging and rewriting this routine until we felt satisfied. In the end, our hard work paid off, as you can see in subsequent paragraphs.
Introducing Cart Explorer version 1
Cart Explorer version 1, shipped as part of add-on 3.0, worked as follows:
1. You press Control+NvDA+3 to activate Cart Explorer. When this happens, NVDA will make sure you are in main playlist viewer, then it will set a flag indicating that Cart Explorer is active.
2. NVDA will then open and parse cart bank files, storing cart assignments in a dictionary of cart keys to cart names. This parser also takes care of some corner cases, including skipping unassigned carts and determining Studio edition in use. Once carts were parsed, NVDA says, "Entering Cart Explorer", and if errors occur, NVDA will inform you that it cannot enter Cart Explorer (this happens if the cart bank file doesn't exist).
3. While using Cart Explorer, if you press a cart key, NVDA will look up the name of the key in the carts dictionary, and announce the cart name associated with it (if found, otherwise, NVDA says, "cart unassigned"). If you are using Standard edition and press number row keys, NVDA will warn you that cart commands are unavailable.
4. It so happens that some people will activate Cart Edit Mode to modify cart assignments while in the middle of exploring carts. If this happens, NVDA will remind you (via doExtraAction function used by name change event) that Cart Explorer is active, and when Cart Edit Mode is turned off, NVDA will ask you to reenter Cart Explorer (this was done to parse newly updated cart bank files).
5. You press Control+NVDA+3, and NVDA will clear carts dictionary, thereby leaving Cart Explorer.
But there was a major concern with this approach: what if a future version of Studio uses a different cart bank format? Thus, I revisited cart bank files again in July 2015, and this time, I noticed a familiar structure: comma-separated values. To test my hypothesis, I opened .cart files in Excel, and voila, it presented itself just like any CSV file. Thus I worked on modifying cart parsing routine, this time using Python's CSV module to parse "cart" files (cart bank files are really CSV files in disguise). This new routine (described below) will make its appearance as part of add-on 6.0.
The magic behind Cart Explorer: handling CSV files
Since Python comes with a library to handle CSV files and since cart banks are CSV files, I rewrote Cart Explorer routine (a function in splmisc module which returns the carts dictionary) as follows:
1. When entering Cart Explorer, Car Explorer preparation routine (splmisc.cartExplorerInit) will take a snapshot of your user name and Studio edition (Studio's title bar indicates which version is in use). Then it initializes the carts dictionary and stores the Studio edition in use.
2. Next, the preparation function will write down names and paths to cart banks. In case a user other than default user is using Studio, it'll modify the cart file names to match the name likely to be used by Studio to present user-specific cart banks. These cart names form one part of the cart bank path (the other part is the path to the folder where the carts live, obtained by using an environment variable).
3. For each cart bank, NVDA will ask Python to parse the cart bank as a CSV file (csv.reader; when finished, it returns a list of lists, with each list representing one row of a CSV table).
4. Once the csv version of the selected cart bank is ready, the row containing cart keys and cart names, together with the cart bank modifier and the carts dictionary are sent to a helper function (part of splmisc module) that will do the following:
A. Depending on Studio edition, this helper will work with only the first half (Standard will only work with function keys, which are the first twelve columns of this row) or the entire row (the rest are number row keys) will be processed.
B. Depending on column position (items in the row list), it will see if function keys or number row keys should be assigned to the selected cart entry. This routine also checks the type of the cart bank (modifiers or none (main)) and modifies the cart key name accordingly.
C. Next, the helper routine will try to locate the name of the jingle assigned to the cart key in question, and if there is one, it'll add the cart key and jingle name pair into the carts dictionary.
5. Back at the cartExplorerInit function, if no erorrs were found while parsing a cart bank, it'll move onto the next one, otherwise it will inform the Studio app module by modifying a flag value in the carts dictionary (stored as an integer, representing number of cart bankks with errors).
6. By now cartExplorerInit is desperate to pass the carts dictionary to someone, and this someone turns out to be the Studio app module - once picked up by the app module, carts dictionary is hired by you to look up cart names for cart keys while you use Cart Explorer (to fire the carts dictionary, simply deactivate Cart Explorer by pressing Control+NvDA+3).
In effect, the routine above (the "magic" behind Cart Explorer) replaced a hand-written cart bank parser and simplified the add-on code (I regret not investigating CSV last year). As far as user experience is concerned, this is same as Cart Explorer 1, with the difference being the parsing routine. With the addition of splmisc.cartExplorerInit, the current version of the splmisc module (miscellaneous services, containing the Track Finder/Column Search combo dialog, column retriever and Cart Explorer preparation tool) was completed.
Conclusion
Cart Explorer has come a long way; from a simple suggestion to the CSV parsing routine above, Cart Explorer has changed to meet the needs of broadcasters using Studio and NVDA. I would like to improve this further in future releases (another suggestion I received was ability to specify cart file names for individual banks, and I'm thinking about implementing this in the near future).
One of the things you may have noticed as you read this article is how I and other developers continue to research better ways of accomplishing something. You also saw a glimpse of how developers and users shape a feature and how much work is involved to bring a feature suggestion to life. These activities (research and feature development collaboration) are just two of the pillars supporting Studio add-on for NVDA, and highlights how design philosophy and product development approach affects future course of product development.
This ends our detailed tour of internals of major features in Studio app module. When we come back, we'll visit our friend from the past: SPL Assistant layer and inner workings of various layer commands.
References:
1. Comma-separated values (Wikipedia): https://en.wikipedia.org/wiki/Comma-separated_values
2. RFC 4180 (Common Format and MIME Type for Comma-Separated Values (CSV) Files), Internet Engineering Task Force: https://tools.ietf.org/html/rfc4180
3. Import or export text (.txt or .csv) files, Microsoft Office Support for Microsoft Excel: https://support.office.com/en-za/article/Import-or-export-text-txt-or-csv-files-5250ac4c-663c-47ce-937b-339e391393ba
4. CSV (Python documentation, Python Software Foundation): https://docs.python.org/2/library/csv.html

StationPlaylist Add-on Internals: Library scan and microphone alarm: threads, threads and more threads

IMPORTANT: The following article on threads, especially a discussion on library scan may change in a future add-on release (I'm experimenting with a new routine for library scan that will use timers instead of a threaded while loop; I'll provide an update via an addendum if timer routine experiment is successful).
Hi,
Of all the work done on Studio add-on, one of them stands out the most: background tasks. I spent many hours and months perfecting this concept, read documentation on this feature and learned a lot through this experience. Today, this work is used in various parts of the Studio app module and beyond, and we'll take a look at two most important results of this work: library scan and microphone alarm.
Brief feature overview
When you are producing a live show, you may forget that your microphone is active. The microphone alarm feature lets NVDA notify you if microphone has been active for a while. This happens even if you are using another program.
Library scan comes in handy when you want to see the progress of a background library scan. Typically, you would initiate library scans from Insert Tracks dialog (press Control+Shift+R). NvDA will then tell you how the scan is going, and if you close Insert Tracks dialog, NVDA will continue to monitor library scans in the background.
But there's more to it than a simple description when it comes to looking at internals of these features. As you'll see, these features use a concept that is gaining traction: running multiple tasks at once, or at least try to emulate it. We'll visit this concept first before returning to our regularly scheduled program of describing the internals of the two features above.
Recent trends in computing: more and more processors in a single computer
A decade ago, people thought a single core CPU was enough to run multiple programs. This involved the processor spending small fraction of a second devoted to each program. Nowadays, it has become common to see desktops, laptops, smartphones and other small devices using at least two cores (termed multi-core; two cores is dubbed "dual core"). As of August 2015, many computers use processors with four cores (dubbed "quad core"), while enthusiasts prefer more cores (the current record holder for desktop computers (as of August 2015) is 8 (octa core), held by Intel Core I7-5960X, a 3 GHz unlocked processor which costs about a thousand dollars; for servers where more cores are used, the current record holder is a package of eight processors (octa processor) called Intel Xeon E7-8895V3, with each processor boasting eighteen cores with base speed of 2.6 GHz).
Despite the fact that many computers come equipped with multi-core processors, not all programs take advantage of this. Python interpreter is one of those programs, and since NVDA is a Python-based screen reader and due to its operational architecture, many of its operations cannot take advantage of multiple processors. Fortunately, Python provides a way to simulate this - run certain tasks in the background, and this is utilized by NVDA and some of its add-ons as you'll see in this article on library scan and microphone alarm.
A gentle introduction to threads: multiple tasks at once
During normal business hours, a program will run from beginning to end with some interuptions (keyboard input, switching to a different part of the program and so on). However, there are times when the program will need to work with many things simultaneously, such as calculating distance between many points, adding multiple numbers at once, comparing many pairs of strings and so on. Fortunately, a mechanism called threads allow a program to do multiple things simultaneously.
A thread is a procedure independent of other tasks. If one thread is busy with something, other threads can work on other tasks. The best analogy is multiple bank tellers in a bank: customers can talk to different tellers, with one teller working on updating customer records for a customer while another customer discusses fraudulent credit card charges with a different teller.
A thread can be involved with parts of a task, devoted to a single task or multiple tasks. For example, an antivirus program could have multiple threads (workers) working independently of each other. One worker can display the overall progress of a scan, while other threads can scan multiple drives at once, with each thread devoted to scanning files and folders on separate drives. In NVDA world, multiple workers are involved to perform various tasks, including making sure NVDA is responsive, handling browse mode in different web browsers and so on.
Threads: a more geeky introduction
A thread (sometimes termed "thread of execution) is an independent path of execution. A single process (app) can have as many threads as it desires (minimum is one for the main thread). Each thread can be asked to perform certain operations with other threads in parallel, which can range from a single, repetative task (part of a function) to being responsible for an entire module or a significant part of the program. In case of antivirus example above, each scanner thread is responsible for scanning an entire drive, with each of them reporting its progress to a manager thread which displays overall progress of a virus scan.
Using threads means each thread can execute on a processor core on a multi-core system. Because of this, many people would want many programs to take advantage of this and finish their jobs faster. However, threads introduce disadvantages, namely many days spent designing careful coordination routines between threads, preventing attempts by multiple threads to change a critical value that a manager thread depends on (called race condition) and so forth.
Python's way of managing threads and the threading module
Python interpreter (and programs which uses them, including NVDA) is not exactly multithreaded. Because of internal issues, Python uses so-called global interpreter lock to prevent multiple threads from messing with each other. One way to bring true parallelism in Python is use of multiprocessing module (multiple Python interpreters, each one devoted to a single task), which has its own advantages and drawbacks (NVDA does not ship with multiprocessing module in the first place).
To manage threads, Python programs (including NVDA) use Python's threading module. This library includes various ways of managing threads, including defining which function can execute in a separate thread, coordinating sharing of information between threads (locks, semaphores (resource access counter) and so on), and letting a thread run its task after waiting for a while (called timers). Even with multiple threads defined, NVDA is mostly single-threaded (serial execution).
To use threads, a programmer will define the thread type (regular thread, timer and so on), define some properties and tell the thread which routine to execute. Once the thread is defined, the start (thread.start) method is called to let the thread do its work.
Threads in Studio app module
For the most part, Studio app module uses only one thread (NVDA's main thread) to do its job. However, there are times when multiple threads are used - up to three can be active at a time: NVDA's main thread (announcing status changes, alarms, Cart Explorer and others), microphone alarm (a timer) and library scan (a background thread). Another situation threads are used is when background encoder monitoring is enabled (see the encoder routines article for details and use of threads there).
The main reason for using threads is to prevent background tasks from blocking user input (commands will not work when a long running task is run from the main NVDA thread). This is more noticeable when library scan is active as you'll find out soon. For now, let's take a look at microphone alarm.
Microphone alarm: A timer waiting to do its work
Simply put, microphone alarm is a timer (akin to a countdown timer). When the microphone becomes active, Studio app module will tell a timer thread to come alive. This timer's only job is to play the alarm sound and display a warning message, and it will wait a while (microphone alarm value in seconds; for example, five seconds).
The master switch which flips the microphone alarm timer is:
alarm = threading.Timer(micAlarm, messageSound, args=[micAlarmWav, micAlarmMessage])
Where "micAlarm" denotes how long this timer will wait and the second argument is the task to be performed (messageSound function). If microphone alarm is off (value is 0), this switch will be left alone forever (turned off until you enable the alarm by specifying a value above 0).
However, microphone alarm is more than a timer: a unique feature of timers is responding to events (a cancel event, that is). When the microphone becomes active, microphone alarm timer will become active. If you happen to turn off your microphone before microphone alarm kicks in, NVDA instructs microphone alarm to quit (timer is canceled). In other words, the "real" master switch is status change, and one of the activities performed by name change event handler (event_nameChange function described earlier) is to manage microphone alarm timer via doExtraAction method (in fact, microphone alarm and Cart Explorer are managed from this function).
Library scan: a unique combination of Studio API and a background thread
When NVDA is told to keep an eye on background library scanning, it calls up another thread to perform this duty. This thread will ask Studio for number of items scanned so far and take appropriate action after scanning is complete (in fact, multiple helper functions are used).
The library scan routine is performed as follows:
1. NVDA will make sure you are not in Insert Tracks dialog (if you are, background library scan routine will not be invoked, as event_nameChange will perform this duty instead).
2. If you do close Insert Tracks while a scan is in progress, or invoke library scan from SPL Assistant (Shift+R), NVDA will instruct a thread to keep an eye on scan progress in the background(see below for signature) to allow you to use Studio commands and to let you hear scan progress from other programs.
3. Library scan thread will ask Studio to return number of items scanned (this is done every second) and will store the result for record keeping.
4. After the scan result is obtained, the thread will check where you are in studio, and if you are back in Insert Tracks dialog, the thread will terminate (see step 1).
5. Every five seconds, library scan thread will call a private function (which wants to see how many items were scanned and current library scan announcement setting) to announce library scan results as follows:
A. If you tell NVDA to announce scan progress, NVDA will say, "scanning" and/or play a beep (if told to do so).
B. If NVDA is told to announce scan count, number of items scanned so far will be announced (again with or without a beep).
C. This reporter function will not be invoked if you tell NVDA to ignore library scan completely or ask it to interupt you only when the overall scan is complete (you can press Alt+NVDA+R to cycle through different library scan announcement settings).
6. Once library scanning is complete (after checking scan result value every second and seeing that the previous scan result and the current one have same values), NVDA will announce scan results (in some cases, number of items scanned will be announced).
You can imagine what would have happened if the above operation was not a background task: cannot perform other NvDA commands until library scan is complete, cannot cancel this operation and what not. And this is the signature of the thread that performs the above operation:
libraryScanner = threading.Thread(target=self.libraryScanReporter, args=(_SPLWin, countA, countB, parem))
There are important arguments in use: the function (task) to be performed and arguments for this function. The most important argument is the last one: Studio 5.0x and 5.10 expects different arguments when told to report number of items scanned so far.
Conclusion
Despite limitations of Python's threading routines, if used properly, it can open new possibilities, and you saw some of them above: microphone alarm and background library scan. Use of threads in the Studio app module also allows NvDA to be responsive while using Studio and allows background tasks to be faithful to the tasks at hand. We'll come back to threads when we talk about encoder connection routines. There is a more "magical" feature we'll visit, and this is our next stop on the SPL Studio Add-on Internals: Cart Explorer.
References:
1. Thread (Wikipedia): https://en.wikipedia.org/wiki/Thread_(computing)
2. Multi-core processor (wikipedia): https://en.wikipedia.org/wiki/Multi-core_processor
3. Multi-core introduction, Intel Developer Zone, March 5, 2012: https://software.intel.com/en-us/articles/multi-core-introduction
4. Intel Core I7-5960X specifications (Intel ARK): http://ark.intel.com/products/82930/Intel-Core-i7-5960X-Processor-Extreme-Edition-20M-Cache-up-to-3_50-GHz
5. Intel Xeon E7-8895V3 specifications (Intel ARK): http://ark.intel.com/products/84689/Intel-Xeon-Processor-E7-8895-v3-45M-Cache-2_60-GHz
6. Global Interpreter Lock (Python Wiki): https://wiki.python.org/moin/GlobalInterpreterLock
7. Threading (Python documentation, Python Software Foundation): https://docs.python.org/2/library/threading.html
8. Multiprocessing (Python documentation, Python Software Foundation): https://docs.python.org/2/library/multiprocessing.html#module-multiprocessing