WebCT in Computer Science

- Michele Jacobsen,
Rob Kremer
Roberto Flores
WebCT is a Web server designed to support on-line, Web-based instruction. Faculty with little or no prior experience with HTML will find the structure and design of WebCT helpful starting points for creating on-line documents. The tools offered by WebCT provide excellent entry-level supports and templates as materials and on-line examinations are created. Disciplines which use timed, selected-response exam questions for assessment and evaluation may find the WebCT on-line examination environment useful for future course offerings.

This winter, WebCT was used to construct and administer a midterm exam in a fourth year computer science course in artificial intelligence.

Considerations for On-line Midterm Design and Administration

WebCT exams represent a considerable amount of instructor preparation time, which renders it less viable for class sizes of fewer than forty students. Grading is done automatically, reducing the marking time essentially to zero. WebCT also provides statistical and graphical summaries of results, as well as a complete item analysis valuable in analyzing student results and the performance of the on-line examination. The ease of marking and availability of detailed statistical results makes it a worthwhile tool for large classes. This is particularly true if the statistical results are used in conjunction with a question bank (also supported by WebCT) and re-use of selected questions in subsequent courses.

A feature of WebCT is that exam items are randomized only once per student. In the event that a student experiences a technical problem, they would be able to log in again and continue at the point they left off. A drawback is that the timer keeps ticking while the student is disconnected.

Screen print of typical test question For our exam, we constructed a test bank of 51 items from which 33 questions were randomly selected when the student logged in. Some questions were chosen from groups of unique items, while others were variations of the same question. WebCT offers short and long question styles; selected-response type items were chosen in order to take advantage of the WebCT automatic grading feature. Exam questions were either multiple choice (see screen print to left) for typical question) or matching type items.

It is critical to give students the opportunity to practice with WebCT prior to the actual exam, so students were invited to complete a Web pre-exam survey that used a similar interface to the actual exam. The survey gathered information about students¹ prior on-line exam experience, expectations, and concerns. The pre-exam survey also served as practice with logging in and changing passwords prior to the exam date. While the pre-test survey appeared to help familiarize students with the WebCT interface, some students reported feeling unduly pressured by time constraints during the midterm.

A few students mentioned that they were very aware of the timer during the exam (there where no time constraints on the survey). Thus, before a high- stake on-line midterm or final exam, it may be beneficial to expose students to some short, timed quizzes in WebCT to familiarize them with WebCT¹s examination timer.

Student Results

This was not a controlled experiment, but instructors believe that students did as well on the on-line midterm as they would have on a paper-based exam, though a number of students were surprised by their performance, which was lower than expected.

There are a number of possible explanations for this. The midterm exam was designed to be challenging to complete in the time given if students were not adequately prepared. Those students who had attended lectures, participated in labs, and had read all assigned materials had a greater chance of success. One reason for designing a more challenging exam was that it was open book, so students who studied for memorization would not perform as well as those who studied and attempted to understand higher order concepts. If students tried to look up answers in the textbook, then they would not have had time to complete the entire exam.

One item, worth 10%, required students to complete a lengthy calculation. Many students spent an inordinate amount of time on this question, and may have had to guess at other questions as a result. It may be better to divide the exam into two separately timed parts to mitigate the possible random effects of long calculation vs. standard multiple choice.

Environmental conditions may have affected some students' exam performance, as they were able to choose the time and location to write the exam. Students had the option of writing the one-hour examination from anywhere, and had some flexibility with the time of writing - from 8:00 am to 9:00 pm on the designated day. Some students were concerned about slow access times from home, and wrote the exam at a public computer on campus. However, a number of students were frustrated by the noise and distractions in the campus computer labs, so, students who did not have access to a private study area equipped with an Internet-connected computer on campus may have been disadvantaged.

Student Reaction to the On-line Exam

Completing a midterm exam on-line was a novel experience for most students. The majority of students liked being able to write the exam at a time most convenient to them - the 'any time, any where' access was an important strength and benefit. Students could complete the exam from home, from a work place setting, or on campus. One student completed the exam from another province! Students appreciated being able to define the atmosphere and set up their own work space for the exam; for example, listening to music at home, spreading out books, resources and tools. Finally, some students reported increased confidence because they could access their textbook and notes during the course of the open book exam.

The two main concerns with on-line exams were cheating and access to the instructor during the exam. Students were concerned that they would not have immediate access to the instructor during the exam for questions, such as explaining wording, providing missing information, clarifying ambiguous items, and so on. A good number of students were very concerned about the potential for cheating, or group writing of the exam. They described the many ingenious ways in which students could cheat, which they believed would penalize the more ethical students.

Some concern was raised regarding the differences between a Web interface and paper- based exams. There was no ability to write or do calculations in the margins. There was little sense of how long the exam was or how much they had completed, and students were not sure how to set up the workspace for the exam. Others commented on the relative benefits of submitting answers using point-and-click radio buttons versus colour-in-the-bubbles scantron sheets.

Instructors who wish to experiment with WebCT's on-line testing are well advised to post the exam on a dedicated server in order to reduce potential access and load problems. Although a number of students indicated that they worried about the browser or server crashing, very few students reported any difficulties accessing and completing the on-line exam. One student prepared to answer the test at home early in the evening only to discover that his Internet connection was down. The student had to drive to the University and complete the exam in a public area.

One feature of WebCT testing that concerned students was the timer that only updated the posted 'time remaining' when they submitted an answer and that the timer provided somewhat misleading information - though ultimately, it is the student's responsibility to keep track of time. Finally, because the questions were selected-response, with one best answer, some students were disappointed there was no opportunity for partial marks.

Next Steps

We believe this experiment was successful, and will continue to refine our on-line testing and grading procedures. We believe it is crucial to use exemplary test construction procedures and principles for an on-line exam. For example, to alleviate student concerns about content and the exam structure, the instructor should provide a clear table of specifications, and provide practice opportunities.

In order to address student concerns about cheating, a supervised, on-line exam situation could be arranged whereby students complete the midterm during a scheduled time in a supervised lab. This would require booking sufficient computer workstations for the class, assigning students to a particular lab, and employing several laboratory assistants who could check student identity, monitor the exam, and answer student questions during the exam. However, a scheduled supervised exam would sacrifice the benefit of "any time, anywhere" access.

For those who want to experiment with the exam environment in WebCT, we have created a Canadian Trivia quiz that can be accessed by pointing your browser to this URL: http://sern.ucalgary.ca:8900/SCRIPT/QUIZ101/scripts/serve_home.pl Use this login: guest Use this password: guest
For more information, please contact:

Michele Jacobsen dmjacobs@ucalgary.ca
Rob Kremer rkremer@ucalgary.ca
Roberto Flores robertof@cpsc.ucalgary.ca

Back to Volume 6, No.3 Table of Contents.