Table of Contents

7. Conduct Technical Review of Course (STEM-only)

Pavel Dimens Updated by Pavel Dimens

Overview

A technical review is performed by eCornell staff members to assess the completeness and accuracy of a course. There are various kinds of technical or STEM-focused courses offered through eCornell (e.g. focused on coding; more conceptual; and inclusive of physical kits); and these different courses will occasionally require different considerations in their technical reviews.

The General Process

A technical review requires the reviewer to experience the course through two simultaneous lenses: a student meeting the minimal requirements to take the course, and a highly critical assessor going through things with a fine-toothed comb. Holding both perspectives requires the reviewer to maintain empathy towards a student trying to learn this material. Ultimately, everything needs to be fluid and maintain a level of excellence consistent with the eCornell brand. In more specific project terms, the purpose of the Technical Review is to ensure that course is ready for the Alpha Test (see the Alpha Review Process article for more details).

The focus for the technical review is essentially threefold:

  1. to make sure all technical elements in the course function correctly: i.e., there are no bugs or crashes that stop forward progress;
  2. to make sure that the instructions for completing the technical exercises are clear and complete: i.e., there is no missing information that students will need; and
  3. to make sure all the grading functionality is working as expected.

For steps 1 and 2 above, the Technical Reviewer will use a Student Canvas user (either logging in directly, or using the Canvas "Act As" function to masquerade as a test student). For step 3, if there are manually graded projects or exercises, the Technical Reviewer will need to first complete and submit the exercises as a Student., and then manually grade the exercises as an Admin/Teacher, in order to test the grading functionality. The correct grading function, percentages and arrangement should be detailed in the certificate's Grading Scheme Document, in this folder.

In addition, the Technical Reviewer should reference the Charter-Starter spreadsheet for the Certificate (the certificate's Instructional Designer will be able to provide a link to this). In particular, the PM Audience tab of this document provides information about the level of knowledge and experience the course's audience will be expected to have, and the Tech Specs tab covers the initial technical specifications.

Logging bugs into BugHerd

BugHerd is the bug logging system eCornell uses, and all issues will be logged using this system. See this article for detailed information on BugHerd. Issues should be logged on the page they appear, as near as possible to the issue itself (text/image). If there is an issue with a provided PDF document, attach the issue to the text with the link to the PDF; then either take a screenshot of the PDF and upload it to the bug, or download the PDF itself and upload that to the bug. Alternatively, you can use BugHerd's video capture functionality (still in Beta, but fairly stable), since this captures the entire browser window instead of just the course page. Open the PDF in another tab; click the video icon in BugHerd to open a video-capture bug; and then navigate to the PDF and highlight the issue.

There are various kinds of issues that can crop up, and to improve organization and triaging, a series of standard tags should be used to classify issue types. Please use these tags when logging issues on Canvas pages:

  • clarification - text or image needs clarification
    • examples: ambiguous wording, undefined technical concepts
  • content order - text/image/quiz/assignment appears out of order with the presentation of other topics/materials
    • examples: quizzes require knowledge appearing later in the course
  • missing content - text/video suggests presence of content that is not on the page
    • examples: missing cheat sheet, missing assignment
  • issue - generic tag for issues that don't fit the other categories
    • examples: typos, broken links

Areas of Focus

Concentrate on the following when reviewing the course.

Course Pages
  • Correct page layouts
  • Correct hyperlinks
  • Typos
  • Page content (video/text) consistent with page title
  • Video consistent with page text
  • In-text inaccuracies
  • Assignment typos
  • Correct hyperlinks
Tools and Other PDFs
  • Consistency (within the document and with relation to the part of the course it's being presented in)
  • Typos
  • Hyperlinks not broken
Assignments
  • Assignment/quizzes can be completed with knowledge provided up to that point
  • Assignments/quizzes do not introduce new undefined terms
  • Assignment/quiz autograding correct (if applicable)

Note the time it takes to complete each assignment/quiz

  • gives a sense of QA effort per section
Projects
  • Projects must be verified to be submittable by the students
    • during QA, complete and submit any course projects through Canvas
    • check (as a Canvas Admin/Teacher) that Project submission was successful

Style Guide

Always update the course style guide sheet with technical terms, clarifications, programming language and programming environments unique to a particular course. Note any technical terms or phrases that a copy editor may need to refer to during the QA phase.

Access the eCornell Style and Settings Guide here

Overall Process for Courses with Physical Kits

On courses that contain physical kits which the students use for completing exercises, the Technical QA should be divided into two "passes": one just to review and examine the online elements, and one using the physical kit to complete assignments. Keeping mind that individual courses will have individual requirements, the general process for courses with physical kits is:

  1. Do a single pass through the course, ignoring assignments and quizzes
  2. Do a second pass through the course, this time taking all the quizzes and doing all the assignments

Ideally, you will have noted all content-related issues in the first pass and can instead focus on assignment/quiz correctness and consistency in the second pass.

The first pass (no assignments/quizzes)

The point of the first pass is to just focus on video and written content. Check for:

  • Correct page layouts
  • Correct hyperlinks
  • Typos
  • Page content (video/text) consistent with page title
  • Video consistent with page text
  • In-text inaccuracies

Pay attention to Tools and other included PDFs for:

  • Consistency (within the document and with relation to the part of the course it's being presented in)
  • Typos
  • Hyperlinks not broken
The second pass (all content)

The second pass has you review the text and videos again, but now you will also take any quizzes and perform any assignments. With the inclusion of this content, you will need to pay attention to a new set of things. Check for:

  • Assignment typos
  • Correct hyperlinks
  • Assignment/quizzes can be completed with knowledge provided up to that point
  • Assignments/quizzes do not introduce new undefined terms
  • Assignment/quiz autograding correct (if applicable)
  • Assignment submission works as intended
    • Codio: Education >> Mark as complete is present in the Codio unit
    • Canvas: Assignment/Project submission works, the assignment is received by Teacher/Admin
  • Note the time it takes to complete each assignment/quiz
    • gives a sense of QA effort per section

Considerations for take-home kits

Courses with kits are unique because we have limited ability to correct errors in a kit and do it quickly. Therefore, it's very important that the kits themselves get quality assessed, along with how the kits are referenced in the course.

  • Kit parts are labelled (if applicable)
  • Kit parts are labelled correctly
  • Kit contains all expected parts
  • Kit parts work as expected
  • Kit parts are referenced accurately in the course: i.e. the part is mentioned in the course with the same name that appears on its label
    • Example: "100K Trim Potentiometer" in kit is referenced as "100K Trim Potentiometer" in course, not "100k pot" or "100k Trim pot"
  • Images of parts in use reflect appearance of parts in kit, or have a note/disclaimer explaining the part may look different in the kit
  • Note any part troubleshooting you had to do
  • Note whether it is practical to expect students to troubleshoot it
  • Note the time it took to troubleshoot
  • Note the amount of physical space required to interact with the kit while using a laptop or desktop computer to take the course
  • Note redundancies for easy-to-lose or easy-to-break parts

How did we do?

6. Implement Faculty Edits

2. Conduct IDD or Sr ID Review

Contact