From New Currents 1.1 November 1994

Testing Software: a review

Donna J. Fremont and Brad Jones
Department of Computer Science

Financial support for this project was partly provided by a Government of Alberta STEP grant, a Royal Bank Teaching Development grant, and research funds from Dr. Nader Bshouty.

In the summer of 1994, members of the Computer Science department embarked on a project to implement a computerized test question database which would allow instructors to construct paper examinations based on a number of question selection criteria. One objective was to specify requirements and find software to maintain the test question database. This article summarizes our findings and conclusions in the hopes that these will assist other people on campus who may have similar course development work in mind.

Requirements Specification

The general requirements were to provide a way to easily create and maintain a database of questions, and to be able to create a test from these questions using criteria for question selection. An additional consideration was to anticipate integration into a Computer Aided Learning (CAL) environment, consistent with work being done by the Faculty of Science CAL Committee.

A summary of the specific requirements follows:

  1. Ability to store a variety of question types in the database, including multiple choice, fill in the blanks, short answer, true/false, and long answer.
  2. Ability to include text, graphic images for diagrams, and properly display mathematical symbols for all questions.
  3. Ability to link each question to additional fields, which can be used to store: question type; source (textbook author or instructor who made the question); chapter; content topic; learning objective; cognitive level; level of difficulty; the weight of a question on an exam; when the question was last used; and the actual usage statistics, difficulty and discrimination, as reported from optical scoring of multiple-choice exams.
  4. Exam preparation allows for both instructor selection and random selection using criteria based on all of the fields mentioned above.
  5. Allows flexibility when formatting paper exams, to meet the instructor and departmental requirements.
  6. Supports optical scoring of multiple choice and true/false question types, so that feedback about student and class performance can be provided to the instructor, and item analysis for each question can be imported directly to the question database to update appropriate fields.
  7. Communication with other software products through text files and conversion utilities.
  8. System should be easy to learn and use; a graphical user interface (GUI) was preferred. A GUI allows better integration with other applications such as graphics programs, through cut-and-paste and clipboard operations.
  9. System availability on more than one computer platform.
  10. Availability of testing software. We would like the software to be used by many departments on campus so that development, maintenance, and expertise can be shared. This last requirement implies that the software must not be proprietary to a particular publisher and licensed only for use with particular adopted textbooks.

Software evaluation

We decided to search for commercial software, because this is far more economical than custom development. After a broad search, we finally considered eight software products. The software products reviewed can be divided into two main classes: publisher (i.e., provided free with textbook adoption) and market (i.e., available for purchase.) Publisher products were not preferred because they are proprietary; this limits the desired sharing among departments.

Publisher products

The publisher products reviewed were:

Market products

The market products reviewed were:

Table 1 summarizes some characteristics of the products tested. The products are generally arranged from best to worst in terms of meeting our criteria. In general, the publisher products are deficient in many desired characteristics.

LXR*TEST 5.0

Of the market products, LXR*TEST 5.0 stands out. We evaluated both the Macintosh and MS Windows versions, discovering few functional differences. Questions are initially classified using an objective and sequence number, but further classification can be done with eight user-defined fields. Fields for last date used and full item analysis are available for each question. Any type of graphic, including Quicktime movies, can be included anywhere in a question, that is, in a stem, in a foil, in an instruction. Standard cut and paste techniques are supported on both Windows and Macintosh versions. The program can import and export question banks using two different text formats (one is free form; the other is strict but well documented.) Test construction allows question selection both by the user and random selection using criteria based on any of the fields mentioned above. The program supports many optical-mark readers, providing automatic updating of the database. This updates question statistics such as difficulty and discrimination. The program also produces reports on student performance and updates the class record book. An add-on interactive module is available which allows the user to create interactive tests using a proprietary system on Windows, and HyperCard on Macintosh. Overall, LXR*TEST 5.0 is a full-featured, moderately easy-to-learn test generation and question bank creation software.

Conclusion

After reviewing the eight software products, we concluded that the best product and the only one that satisfied our specifications was LXR*TEST 5.0 by Logic eXtension Resources. We have presented demonstrations of LXR*TEST to the Faculty of Science CAL Committee, interested Science Department Heads and faculty members, and New Media Centre staff Janice Bakal and Charmaine Connop-Scollard.

The Faculty of Science has purchased a faculty-wide site license for the Scoring Edition of LXR.TEST 5.0, with Spelling and Interactive Testing options. The license includes both Macintosh and Windows versions. Members of the faculty can obtain a copy by contacting Dr. Hans Laue in the Department of Physics. Dr. Laue can also provide further information on the license and the software.

If you would like a copy of the full evaluation of the above products, or addresses for suppliers, please contact Donna Fremont: 220-5109, e-mail: fremont@cpsc.ucalgary.ca