WebCT in Computer Science
- Michele Jacobsen,
Rob Kremer
Roberto Flores
WebCT is a Web server designed to support on-line, Web-based instruction.
Faculty with little or no prior experience with HTML will find the structure
and design of WebCT helpful starting points for creating on-line documents.
The tools offered by WebCT provide excellent entry-level supports and
templates as materials and on-line examinations are created. Disciplines
which use timed, selected-response exam questions for assessment and
evaluation may find the WebCT on-line examination environment useful for
future course offerings.
This winter, WebCT was used to construct and administer a midterm exam in
a fourth year computer science course in artificial intelligence.
Considerations for On-line Midterm Design and Administration
WebCT exams represent a considerable amount of instructor preparation time,
which renders it less viable for class sizes of fewer than forty students.
Grading is done automatically, reducing the marking time essentially to zero.
WebCT also provides statistical and graphical summaries of results, as well
as a complete item analysis valuable in analyzing student results and the
performance of the on-line examination. The ease of marking and availability
of detailed statistical results makes it a worthwhile tool for large classes.
This is particularly true if the statistical results are used in conjunction
with a question bank (also supported by WebCT) and re-use of selected
questions in subsequent courses.
A feature of WebCT is that exam items are randomized only once per student.
In the event that a student experiences a technical problem, they would be
able to log in again and continue at the point they left off. A drawback is
that the timer keeps ticking while the student is disconnected.
For our exam, we constructed a test bank of 51 items from which 33 questions
were randomly selected when the student logged in. Some questions were chosen
from groups of unique items, while others were variations of the same
question. WebCT offers short and long question styles; selected-response
type items were chosen in order to take advantage of the WebCT automatic
grading feature. Exam questions were either multiple choice (see screen print to left)
for typical question) or matching type items.
It is critical to give students the opportunity to practice with WebCT prior
to the actual exam, so students were invited to complete a Web pre-exam
survey that used a similar interface to the actual exam. The survey gathered
information about students¹ prior on-line exam experience, expectations,
and concerns. The pre-exam survey also served as practice with logging in and
changing passwords prior to the exam date. While the pre-test survey appeared
to help familiarize students with the WebCT interface, some students
reported feeling unduly pressured by time constraints during the midterm.
A few students mentioned that they were very aware of the timer during the
exam (there where no time constraints on the survey). Thus, before a high-
stake on-line midterm or final exam, it may be beneficial to expose students
to some short, timed quizzes in WebCT to familiarize them with WebCT¹s
examination timer.
Student Results
This was not a controlled experiment, but instructors believe that students
did as well on the on-line midterm as they would have on a paper-based exam,
though a number of students were surprised by their performance, which was
lower than expected.
There are a number of possible explanations for this. The midterm exam was
designed to be challenging to complete in the time given if students were not
adequately prepared. Those students who had attended lectures, participated
in labs, and had read all assigned materials had a greater chance of success.
One reason for designing a more challenging exam was that it was open book,
so students who studied for memorization would not perform as well as those
who studied and attempted to understand higher order concepts. If students
tried to look up answers in the textbook, then they would not have had time
to complete the entire exam.
One item, worth 10%, required students to complete a lengthy calculation.
Many students spent an inordinate amount of time on this question, and may
have had to guess at other questions as a result. It may be better to divide
the exam into two separately timed parts to mitigate the possible random
effects of long calculation vs. standard multiple choice.
Environmental conditions may have affected some students' exam performance,
as they were able to choose the time and location to write the exam.
Students had the option of writing the one-hour examination from anywhere,
and had some flexibility with the time of writing - from 8:00 am to 9:00 pm
on the designated day. Some students were concerned about slow access times
from home, and wrote the exam at a public computer on campus. However,
a number of students were frustrated by the noise and distractions in the
campus computer labs, so, students who did not have access to a private study
area equipped with an Internet-connected computer on campus may have been
disadvantaged.
Student Reaction to the On-line Exam
Completing a midterm exam on-line was a novel experience for most students.
The majority of students liked being able to write the exam at a time most
convenient to them - the 'any time, any where' access was an
important strength and benefit. Students could complete the exam from home,
from a work place setting, or on campus. One student completed the exam from
another province! Students appreciated being able to define the atmosphere
and set up their own work space for the exam; for example, listening to music
at home, spreading out books, resources and tools. Finally, some students
reported increased confidence because they could access their textbook and
notes during the course of the open book exam.
The two main concerns with on-line exams were cheating and access to the
instructor during the exam. Students were concerned that they would not have
immediate access to the instructor during the exam for questions, such as
explaining wording, providing missing information, clarifying ambiguous
items, and so on. A good number of students were very concerned about the
potential for cheating, or group writing of the exam. They described the many
ingenious ways in which students could cheat, which they believed would
penalize the more ethical students.
Some concern was raised regarding the differences between a Web interface and
paper- based exams. There was no ability to write or do calculations in the
margins. There was little sense of how long the exam was or how much they
had completed, and students were not sure how to set up the workspace for
the exam. Others commented on the relative benefits of submitting answers
using point-and-click radio buttons versus colour-in-the-bubbles
scantron sheets.
Instructors who wish to experiment with WebCT's on-line testing are well
advised to post the exam on a dedicated server in order to reduce potential
access and load problems. Although a number of students indicated that they
worried about the browser or server crashing, very few students reported any
difficulties accessing and completing the on-line exam. One student prepared
to answer the test at home early in the evening only to discover that his
Internet connection was down. The student had to drive to the University
and complete the exam in a public area.
One feature of WebCT testing that concerned students was the timer that
only updated the posted 'time remaining' when they submitted an answer
and that the timer provided somewhat misleading information - though
ultimately, it is the student's responsibility to keep track of time.
Finally, because the questions were selected-response, with one best answer,
some students were disappointed there was no opportunity for partial marks.
Next Steps
We believe this experiment was successful, and will continue to refine our
on-line testing and grading procedures. We believe it is crucial to use
exemplary test construction procedures and principles for an on-line exam.
For example, to alleviate student concerns about content and the exam
structure, the instructor should provide a clear table of specifications,
and provide practice opportunities.
In order to address student concerns about cheating, a supervised, on-line
exam situation could be arranged whereby students complete the midterm during
a scheduled time in a supervised lab. This would require booking sufficient
computer workstations for the class, assigning students to a particular lab,
and employing several laboratory assistants who could check student identity,
monitor the exam, and answer student questions during the exam. However, a
scheduled supervised exam would sacrifice the benefit of "any time, anywhere"
access.
For those who want to experiment with the exam environment in WebCT, we have
created a Canadian Trivia quiz that can be accessed by pointing your browser
to this URL:
http://sern.ucalgary.ca:8900/SCRIPT/QUIZ101/scripts/serve_home.pl
Use this login: guest
Use this password: guest
For more information, please contact:
Michele Jacobsen dmjacobs@ucalgary.ca
Rob Kremer rkremer@ucalgary.ca
Roberto Flores robertof@cpsc.ucalgary.ca
Back to Volume 6, No.3 Table of Contents.