Building Blocks of a Beta-Testing Campaign


Beta testing is a strategic phase in the software development cycle. Since it involves participants who are outside the development team and is of limited duration, it requires some preparation. Here we introduce the building blocks of a beta testing campaign.

https://www.reachout-project.eu/download/Main/Guidelines/BuildingBlocks.jpg

Goals
Think of the goals pursued in the beta testing campaign. In a research project it usually is to verify basic characteristics such as: deployability, integration of the different components, clarity of the purpose of the software, etc.

Scope
While the goal is generic and can be shared, the scope is more concerned with specific functionalities of the software that are made available to bets testers. In the case of a platform it may be only basic modules instead of the whole platform. The scope must be well introduced in the beta testing documentation.

Timeline
Being a phase in the software development lifecycle, beta testing should have a limited duration. In a research project, it must be strategically positioned in the project timeline. If it is too early, partners will not have the time to deliver software mature enough to be tested, and if it is too late, there would be no time to incorporate feedbacks before the end of the project.

Beta testers
Beta testers are those unknown users who are asked to try the software and report their feedbacks. A good definition of goal, scope and timeline helps define the profile of beta testers. Research projects usually require highly trained IT professionals. Beta testers are likely to be developers rather than end users. However, depending on what is to be tested, business users could also be a good target.

Participants
Beta testing participants include development partners, use case partners and third party beta testers. It is a good idea to leverage use case partners as internal beta testers. Their role would be to provide early feedback that will help refine the beta version of the software that will be tested so as to make testing more valuable. 

Support
A contact point must be appointed to handle incoming questions from beta testers. This is a pivotal role in beta testing. The role of the contact point is not only to answer beta testers questions but to identify critical issues and to file them and report them in a way that is beneficial for the whole project. It is a good idea to set-up a support mailing list or forum to which all beta testers are subscribed: they can access archives and benefit from previous questions and answers. ReachOut automatically creates such a mailing list for each new campaign.

Beta Documentation
The existing software documentation could be too broad, or too complex and might not not adapted to the specific goals and scope of beta testing. It is a good practice to develop specific beta testing documentation and beta testers can always refer to the full software documentation if necessary. Proper beta test documentation will decrease the proportion of beta testers who abandon the test before completing it (abandon rate).

Internal Communication
While it may be developed as a dissemination activity, all project partners must understand that beta testing is a required phase in the development of the project outcome. Beta testing goal, scope and timeline must be clearly shared by all project partners and particularly those who will deliver the beta version to be tested. 

Feedback collection
Beta testing aims at producing feedback from developers who have not participated in the development of the software, or real life users. Collecting this feedback is of paramount importance. This can be done in various ways, via the support contact point, at a workshop, and through a questionnaire. In research projects, the questionnaire is the preferred method. It should be carefully drafted.

Beta Campaign Manager
Since beta testing being must deliver maximum return (feedback) in a limited time frame it must be executed effectively and this is why we use the term campaign. Like a military operation, a beta-testing campaign combines resources of different nature. Resources must be committed and in place before the campaign goes public. One person should lead the campaign and ensure all resources are aligned: the Beta-Testing Campaign Manager. 

Campaign web page
The campaign web page is the public face of the beta-testing campaign. It is the central place where beta testers can find all resources to participate in beta testing. The campaign page should include key information and links to the presentation of the software, to documentation, download sites, etc. It should include a simple form where beta testers can sign up and indicate the campaign start and end dates. ReachOut provides a template of the Campaign page.

Packaging
Software to be beta tested should be easy to access and to deploy. While deployability might be right in the scope of beta testing, complex build processes are a huge deterrent and must be avoided at all cost. Depending on the scope a SaaS instance of the software will facilitate beta testers participation, otherwise Docker images are also a good practice. As a packaging option, ReachOut offers project the possibility to use the ReachOut Factory to package their beta version.

Beta versions
A beta-testing campaign can be comprised of several cycles. At each cycle a new build can be released after identified issues have been addressed and fixed. It may be necessary to update the questionnaire if a new feature or a major fix has been added to the software but such changes should be kept to a minimum as they make exploitation of questionnaire results more complicated. 

Incentives
Signing up for a beta-testing campaign is an opportunity for beta testers to preview software  that advance the state-of-the-art of technology. “Most beta testers get involved because they have a passion for software and technology. In return, most only request that their feedback be acknowledged.  For the most part, this recognition doesn’t require any incentive (though a free T-shirt to say thank you never hurts).” From an article in FastCompany.

Fine print
The nature and limitations of a beta-testing campaign should be well understood between all participants. When signing up, beta testers should be clearly advised of such nature and limitations.  ReachOut provides a disclaimer template adapted to beta-testing campaigns. This disclaimer includes a GDPR notice and is to be automatically acknowledged by beta testers when signing up. Of course research project must ensure that conducting a beta test will not have unwanted, adverse effect of beta testers. 

Promotion
A beta-testing campaign has to be promoted, beta testers will not find it by themselves. Promotion should be conducted in the media most likely to reach out to beta testers with the desired profile and competences. While the ReachOut platform is a means of promotion itself, research projects must develop their own promotion activities in the framework of their dissemination plans. 

Beta test report
While consumer good and services testing which can happen in person in a controlled environment, in research projects beta testers are distant and usually unknown. Collecting feedback via questionnaire is probably the most efficient method. ReachOut provides a beta test report template which is a questionnaire consisting of 2/3 generic questions and 1/3 specific questions which closely reflect the scope of the beta testing.

Photo credit: CC BY Holger Zscheyge