Table of Contents

Alpha Review Process

Overview

What is the Alpha Process?

Alpha test is the name of the first limited-public review stage in product development, particularly in the tech industry. At eCornell, we adopted this Alpha testing process initially for our programming and technical courses. It offers us a chance to get course content in front of real users who work through the content and check the integrity of the coursework, noting any errors, bugs, glitches, typos, or challenges they might run into.

The Alpha process is different from QA or our review phase in development because it is the first opportunity to get the course in front of users, rather than our internal reviewers. Therefore the Alpha process typically happens prior to the course going to QA, and after the course has been reviewed and approved by faculty and IDDs from a content perspective; although it can also happen after QA has been completed, depending on the project schedule.

When or why do we run an Alpha Test?

Stakeholders will typically decide if the content is targeted for Alpha Review in the early chartering sessions for the certificate. The decision is often based on technical or STEM related content, in particular any coursework that might require ITG-maintained canvas add-ons, like Codio, Jupyter Notebooks, etc. There may be a reason for non-STEM or technical coursework to be alpha tested, and it will be discussed and decided as early in the development process as possible.

Who is involved in the Alpha Process?

Alpha is typically coordinated by the ITG team in collaboration and communication with the Instructional Designer assigned to each project. The Facilitation team, led by the Program Facilitation Manager, will identify appropriate test participants, which we split between two qualifiers: Student Experience testers and Expert testers (see below for more info between these two roles).

The ID assigned to the certificate is responsible for filling out the Alpha Template with details for the Alpha Review plans. This will initiate the Alpha Process: proper facilitators will be enlisted to participate as Expert testers, and related appropriate student experience testers will be brought in.

IDs should:

  1. Make a copy of the Alpha Template and name it appropriately for the program this Alpha is for (i.e. CIS570s - Alpha Review Plan)
  2. Put the copy in the google drive folder structure for that program, and share it with the relevant Program Facilitation Manager, Lori Hoyt, and Jason Carroll.
  3. Fill the copy out with the relevant dates and details, as they are defined

As each course moves through development to the Alpha testing phase, the ID and the Technical Lead (ITG resource) coordinate with Course Operations to ensure that the Alpha sections of the course are cloned from the latest course version (Dev, QA or -- in rare cases -- Master) into a separate instance on Canvas; that the linked technical components (e.g., Codio exercises) are copied/cloned properly; that the Alpha section is correctly linked to the course’s BugHerd instance; and that the students and facilitator are enrolled in the Alpha section. 

Expectations for Alpha Facilitator

The following information should be sent to new alpha facilitators:

  • Reviewing the course’s instructional content, including all Codio activities, from a facilitator's perspective, and logging questions, edits & observations in BugHerd.
  • Specifically recording in BugHerd any information that should be added to the Facilitator Notes section of the course, for future facilitators to refer to after the course launches.
  • Grading student assignments and ensuring the grading functionality is working as expected.
  • Answering student questions, posting notifications, and providing feedback to the Alpha student testers.
Difference between Student Experience testers and Expert testers:
  • Student Experience testers are individuals enlisted to participate in an alpha review with a minimum related experience for the content being tested. These are typically testers who would be reviewing the content with less direct experience or background relating to the coursework, but fit the target audience and meet any prerequisite standards. These testers typically review in a more generalized form and will provide feedback on "course flow" or difficulty levels of exercises, as well as any general bugs and instructional issues.
  • Expert testers are selected Alpha participants who have subject expertise in the content area being reviewed. These are typically testers who are targeted to be future facilitators for the course being reviewed. They will have extensive knowledge and experience in the related content area, and will focus on testing the technical accuracy and practicality of the content, so that all exercises are correct and working as intended.
What do testers do as they review?

The main method testers use to log notes, feedback, bugs, and general course issues is through the browser plugin BugHerd. This plugin allows testers to screenshot or video capture elements of a page (useful for documenting errors or bugs), annotate or describe their issue with it, and submit it to an ITG-managed ticket organization workflow.

After the Alpha is finished and all reviewers have logged any specific comments through BugHerd tickets, a post-alpha BugHerd review is scheduled by the ITG team to organize and assign the related tickets for addressing by related parties. Content issues are addressed by the designers, and technical issues are addressed by ITG.

For more information on BugHerd and the processes, see the related BugHerd Knowledge Base articles here.
What is the Student Experience Survey?

Similar to the end of our live eCornell courses, we include a custom Student Experience Qualtrics survey specifically for the Alpha testers. This survey is a modified and shorter version of the standard end-of-course survey, more focused on capturing the general observations and feedback of the course from the student perspective.

This custom survey is mainly for Product to review and track suggestions and issues at the overall “course experience” level, as opposed to the BugHerd tickets testers submit when logging an issue concerning a content error, a functional bug, or a specific question. General feedback logged in BugHerd tickets tends to be less actionable and get lost in the shuffle of iteration after an Alpha, so this custom survey will serve as documentation of the testers’ perspective immediately post-Alpha.

Process

Timeline

Details

2 months prior to Alpha Review Date

Task: Alpha Review Plans (Document)

  • ID fills out the Alpha Review Plan Template
    • Copy the template and rename it "[course code] - Alpha Review Plan"; e.g., "CEEM580s - Alpha Review Plan"
  • Note any potential timing complications
  • Think about your audience and what type of Alpha reviewer would most help improve the learner experience (i.e. beginner students for an intro course)
  • Alpha course length: generally, either two or three weeks, matching the proposed length of the courses; but This can vary depending on your situation
  • Share the document with Ted, IDD, Program Facilitation Manager, Product Manager, and Lori Hoyt
  • ID & Program Facilitation Manager identify Alpha Team

Goal: 

  • The purpose of this planning document is to make the Facilitation, Course Operations, and ITG teams aware of the alpha needs, and signal them to begin the process of scouting out appropriate and qualified testers.

1-2 weeks prior to Alpha (ideally post-faculty review of course targeted for Alpha)

Task: Tech Team Review Plans

  • Varies by course content; coordinate with Ted/ITG team
  • ID ensures Tech Review/QA is finished before Alpha section created (each course)
  • Course Ops & ITG create & confirm Alpha section (each course)
    • Make sure the Custom EOC Survey is implemented in the Alpha section for your courses. Add this HTML snippet on the Course Exit Survey page to embed the Alpha survey:
<div class="qualtrics">SV_25I5AHq2A8LPndY</div>
  • ITG adds Alpha section to course BugHerd project

Email week before

Meeting ideally the Monday before the first Alpha begins

Task: Review Reminder

  • Make sure your reviewers remember to review the course
  • Remind reviewers about your goals for the review, and about what kind of feedback you need from them
  • If they weren’t hitting the course milestone dates, remind them of those

Task: Meet with Alpha Review Team

  • Write email with meeting agenda and alpha guidelines/timing
  • Alert Alpha reviewers to what you want guidance on
  • Learn about reviewers’ backgrounds
  • Introduce reviewers to BugHerd
  • Invite ITG, IDD, Program Facilitation Manager, Product

Goal: 

  • Set and align expectations
  • Understand the backgrounds/experiences of the Alpha testers 

After an Alpha Review finishes

Task: Thank-you email

  • ID sends Review Reminder and Thank You emails to Alpha team (each course)
  • Thank the reviewers and If you want, note any feedback that was particularly helpful.
  • Remind them of the dates of the next alpha 

Task: Alpha Review of BugHerd Bugs

  • ITG will schedule a session to cover the feedback and organize the BugHerd tickets logged in the alpha.
    • Include: ID; ITG; IDAs; Alpha Facilitator; certificate Product Owner
  • In this session, tickets will be assigned to most relevant roles (technical fixes to ITG members, content fixes to designers, etc.)

Task: Implement Identified Changes/Bug fixes

  • Assigned roles address and complete the identified BugHerd tickets to fix and change the course post-alpha.
    • ID/IDAs/ITG/Faculty fix & confirm Alpha bugs tracked in BugHerd (each course)

Task: Update Facilitator Notes as necessary

  • ID works with Alpha facilitator to edit/update Facilitator Notes doc as necessary

Resources

Name and Link

Role

Purpose

Alpha Review Plan Template

ID

ID Clones this template plan in the early certificate chartering.

Alpha Kickoff Email (Start)

Facilitator/Alpha coordinator

Template email used by the alpha coordinator to send to all alpha participants.

Alpha Reminder (Day before course opens)

Facilitator/Alpha coordinator

Reminder email to let all alpha participants know the day before the alpha course goes live. This also serves as a brief reminder how to log bugs/issues vs. the general feedback comments for the EOC student survey.

Alpha Thank you Email (Finish)

Facilitator/Alpha coordinator

Template email to close out an alpha review; Note any feedback that was particularly helpful; Remind them of the next Alpha dates;

Custom End of Course Student Survey

ID

Custom created EOD Student survey with unique feedback/questions for the Alpha testers. Make sure this is implemented at the end of each of your Alpha course instances.

Alpha Reviewing Tips

What is a “good bug” to log, from an Alpha Reviewer perspective? 
  • It’s a real bug. Reset the Codio unit to ensure you are reviewing the latest version
  • It’s a well-described bug. Make sure to detail the issue fully and completely
  • It’s a necessary and non-redundant bug. Make sure to check the page for other bugs calling out the same issue
    • When responding to a bug, make sure to check its Comments thread!
  • It’s a properly linked bug. If you’re in an Alpha, cross-link the issue from the Alpha to the Master or Dev version (or vice versa) in the comments
  • It’s a well-supported bug. If you are adding to or replacing a file, attach it directly to the bug
    • Make sure to be clear for the Tech Developer as to whether the file is being replaced..

How did we do?

6. Alpha - Wrike Task Definitions

Prepare a course for Alpha review

Contact