|
|
|
|
Assessment in Online CourseA Case StudyIntroduction
This paper is a
case study discussing the forms of assessment used in the online courses at
Dakota State University. The case
study is comprised of the a background section describing the online program at
DSU, a literature section, and a discussion on the survey sent to the online
faculty about online course assessment and its results. Background
Dakota
State University began offering online courses in 1991.
The University has very innovative faculty and as the Web developed
faculty began to develop courses online. This concept is in line with the DSU
mission to “provide instruction in computer management; computer information
systems; electronic data processing; other related undergraduate and graduate
programs; two-year, one-year, and short courses for application and systems
training in areas authorized; and the preparation of elementary and secondary
teachers emphasizing the use of computers and information processing”.
Dakota State’s goal is to provide quality education through the use of
technology and integrates technology is all of its curricula.
In 1998, DSU was rank 12th in the Yahoo listing of the 100
most wired colleges, moving to #10 in 1999. By
the academic year 1995-96 the University made a strategic decision to move into
distance education in a planned way, not just courses by individual faculty. The Office of Distance Education was formed and set up with a
staff of three, the Director of Distance Education, an Instructional
Technologist, and a secretary. Faculty
interested in developing courses for online delivery or alternative delivery in
the classroom can apply for the summer grant program with awards of $1500 to
$3500 for course development (Attachment A).
The stipulation of the grant is that the course(s) be delivered in the
next academic year. Grants are
awarded for projects other than course development, such as the establishment of
a proxy server for distance students to access the Library and other University
services and a web site designed specifically to support research of distance
students. Faculty
teach the online courses as part of their load.
Understanding the additional time required for online teaching and
understanding the issues related to the COHE agreement in South Dakota, the
faculty are provided with the opportunity to request an online student assistant
(Attachment B). These assistants
are (for the most part) upper-class undergraduate students hired by the faculty
to do any number of things from assisting in web page development to organizing
course email and assisting in grading. The pay is set higher than that of work-study as an incentive
to attract good students. The
students can work eight hours a week for 15 weeks. This is the first year that
Dakota State has had masters programs and for the first time now DSU has
graduate assistants working with faculty teaching online. The
Office of Distance Education (ODE) was established with an operating budget and,
for the first year, reinvestment funds from the state.
As the program grows it will become self-sufficient.
Last year the program took over a salary, the secretary’s.
The University supports the program and currently carries the other
salaries. The courses were offered
under state tuition with off-campus delivery fees, and now the courses are
delivered in a self-support mode. This
will change the funding base for the program, but the advantage is that all of
the self-support rates stays on campus. Over
the three years that I have been director the program enrollments have grown
168.6% and continue to grow. We have implemented the Fast Track Program,
designed to provide high school students the opportunity to take general
education classes online and earn dual credit for the courses.
A high school junior taking one course a semester can complete their
first semester of college before graduating from high school if so desired.
Dakota State also has its first accredited totally distance delivered
degree program. It is the
bachelor’s in Health Information Administration.
This is a degree completion program for students with the associate
degree and have passed the Accredited Record Technician (ART) examination. Review of Assessment Literature
It is important to clearly define the performance or skills you will be
assessing in the course. According
to Worthen et. al. (1999), you start by defining the performance you plan to
assess:
1.
The set of skills, proficiencies, and behaviors that skillful performers
are likely to use.
2.
The subject matter domains that successful performers are likely to draw
from in making decisions about what to do, how to do it, and why it should be
done.
3.
The relevant criteria that should be used to judge the quality of a
student’s performance.
4.
Whether the assessment will focus on observing and judging the process
students use, the products they create, or both process and products (page 297).
Clear
course and lesson objectives provide the means for students to understand what
skills are to be learned and what performance will be assessed.
Guthrie (1992), Sendov and Eskenasi (1990), and Worthen et. al. (1999)
suggest the use of Bloom’s taxonomy of educational objectives for cognitive
domain as an excellent tool for writing objectives. The following table from Worthen et. al. summarizes the
taxonomy:
Page 195
Guthrie
(1992) conducted a study of the types of assessment used by faculty at three
institutions. The following table describes the medium and the percentage of
faculty using the medium out of a group of 239.
These media can be used both in the classroom and at a distance and I
will make a comparison between this data and my survey in the results section.
Page 74. Brown and Knight (1994) provide some characteristics of sound assessment: ·
There
will be clarity of purpose ·
Assessment
will enable the learner to review progress and plan further learning ·
Assessment
will allow the provider to review progress and adjudge teaching effectiveness ·
It will
be clear what is being assessed and how judgments are reached ·
Assessment
will essentially assess what it claims to assess (validity) ·
It will
appear credible to tutors, learners and institutions ·
It will
be cost-efficient ·
An
outcome will be clear records of attainment which are useful to third parties ·
The
system itself will be subject to quality assurance procedures (page 22). Brown
and Knight also comment that multiple methods of assessment are best. Nixon (1990) comments that assessment should focus on the ways in which students’ judgments and understanding are enhanced by conceptual representation and exploratory models; ways in which students review their practice in the light of theory; and the extent to which students explore the interdependence and interpenetration of theory and practice.
Prus and Johnson (1994) did a critical review of thirteen assessment
options used in higher education, providing advantages, disadvantages and
discussing the bottom line on the assessment.
They summarized with the following facts on higher education assessment: ·
There
will always be more than one way to measure any objective; there are always
options. ·
No single
method is good for measuring a wide variety of different student abilities;
there is no one true way. ·
We cannot
rely on popularity, tradition, sales propaganda, or education practices; it
isn’t simply a matter of choosing the most attractive available option. ·
The best
methods usually take longer and cost more faculty time, student effort, and
money. ·
The only
way to be certain that a particular methodological option is good for your
program is to pilot-test it on your students, in your curriculum, with your
faculty – an educated trial-and-error approach (pages 69-70). Banta et. al. (1996) discuss assessment as a collaborative process. It requires participation and direction. Designing assessment to examine complex issues such as students’ critical thinking skills is not an easy task and may require team efforts. Banta et. al. also view assessment as a form of faculty development. When faculty devote time to become fully involved in assessment, they create and become active participants in their own program of professional growth and development (page 37). Sendov and Eskenasi (1990) dealt in particular with
assessment in the age of technology. They
comment that analysis shows that there are three main tendencies: 1.
the
computer as an object of study 2.
the
computer as a training device 3.
the
computer as an extension of the human mind. In their study they view the computer as an extension of the human mind.
They note that it follows then that the aim of education in an
information age is the training of children and young people who are equipped to
use computers (page 57). Sendov and Eskenasi contend that this will lead to the
most profound changes in both methods and content of education.
Their study goes to discuss a computer based testing system called TEST.
From their results, they discovered that the system was both an interest
and a frustration to students. The
students liked the immediate feedback to their answers but were frustrated that
they could not change an answer after it was keyed. The system was designed to let the student skip an item and
go back but not to change an answer. Their
study also showed that training the teacher to use the system was not
significant, a total of 18 hours.
How does the literature relate to assessment for online courses?
Basically our online faculty are using or adapting classroom assessment
to their online courses. The
literature I mention in this paper point to three key principles: 1.
An
instructor needs to clearly define the skills/knowledge to be learned or
acquired in the course through the course objectives. The instructor must also make sure the forms of assessment
used in the course clearly assess that skill/knowledge and that the forms of
assessment are well defined and understood by the learner. 2.
There is
no one best way to assess objectives. Instructors
should use multiple methods. It is not a process to be taken lightly but determining the
best methods to use for a particular subject may come form a trial and error
method. Common classroom methods
were presented. 3.
Assessing
skills/knowledge in the technology age is changing education.
The change is going on around us now and we as educators must be aware
that they are in the middle of educational reform.
The online faculty in this case study are demonstrating that change in
action. These principles are
important to assessment both in the classroom and online.
As the survey results are discussed in the next section, differences in
techniques will become apparent. The Survey and its Results
I developed a five question open-ended survey that I distributed to the
faculty who have or are now teaching online at Dakota State (Attachment C). Although analyzing open-ended surveys is often more difficult
than quantifiable surveys I needed to do my survey this way so the faculty could
tell me what they do. I sent the
survey to 23 faculty and had a return of 15 or 65%, which I felt was a good
return. The 15 faculty that
completed the survey teach (or have taught) a total of 38 courses which are
listed in Attachment D.
The questions and all responses are listed in Attachment E.
Here I will summarize comments from each question.
In question 1 – What types of assessment are you using to determine
student performance in your online course? – I found from the 15 faculty that
they use on average 3.53 different types of assessment in the online courses.
The range is from two to eight different types of assessment in a course
and the median is 3. In the
following table I have listed the different types of assessment described and
the total percentage used. Assessment medium Total Percentage
Comparing this data to that of the Guthrie study, the most commonly used
form of assessment in the traditional classroom and for the DSU online faculty
is examinations, followed by papers. It
is evident that faculty use the same forms of assessment all forms of course
delivery, which is what I addressed in question 2 – Are the types of
assessment of student performance used in your online course different from the
forms of assessment used in your traditional courses?
Seven of the fifteen faculty survey stated that they use the same
assessment in their traditional classroom and online courses.
The remaining eight faculty had no major changes, but rather had made
adaptations to their forms of assessment such as, changing quizzes and exams to
multiple choice and other objective formats rather than short answer or essay to
make grading easier. Using
discussion board in place of classroom discussion was a change.
How assessment was weighted also changed.
Question 3 – Please discuss the forms of assessment that have been
successful in your online course provided a variety of responses.
It becomes clear that success of a form of assessment relies on the
comfort level of the faculty member using it.
Proctored exams were considered successful and I believe that comes for
the security provided by the proctor. As
in the field of distance education in general, faculty are concerned that the
learner taking the exam and writing the papers is actually the learner signed up
for the course. Online quizzes were
considered successful because they provide immediate feedback to the students
and lightens the faculty members grading load.
Overall written assignments were considered successful because writing
demonstrates whether the learner was organizing and analyzing the content. Discussion boards were successful and for many faculty
provided more in depth discussion than in the classroom.
Your course does not always go the way you plan it. In question 4 –
Please discuss the forms of assessment that have not worked well in your online
course – problems with assessment was discussed.
There were technical problems with online quizzes and problems with
opening email attachments. Another
major area was communication. Faculty
assumed that instructions were clear and found that was not case.
For some faculty discussion was a success but for others getting the
discussion boards started was hard. Some
faculty found it hard to reorganize their subject matter to online delivery.
Worries about cheating appeared here.
The final question was for additional comments.
It was interesting to find that some faculty felt online assessment was
simpler. Many faculty were
considering using both discussion boards and online quizzes and exams in their
classroom courses. Generally
faculty didn’t consider assessment for the online courses different from their
classroom courses. In My Humble Opinion
What have I learned from this case study?
For the most part the faculty teaching the online courses have tried to
produce the same course online as they teach in the classroom.
That is not a bad thing and makes our program academically sound.
However, presenting the same content does not mean that the assessment
used needs to be the same. A couple
of the faculty have tried using alternative assessment, such as portfolios.
I noted frustration from the faculty in trying to conduct their classroom
assessment online. There was
concern about security of exams and quizzes.
There were comments about the time that was spent in making quizzes and
exams secure. Faculty were making
as many as five different versions of quizzes and exams to insure a random
selection when students accessed exams and quizzes.
Faculty felt restricted by the forms of assessment they were using.
The picture was not all bad. There
were comments from faculty about successes in what they tried and some faculty
were considering using their online assessment in the classroom setting.
As the administrator of the program I am not in a position to “tell”
faculty what type of assessment to use in their courses but I see that I might
be able to help some faculty to break out of the “traditional” assessment
mode by providing a workshop in alternative assessment that could be used for
distance delivery. The online faculty meet on a regular basis through the
semester and my plans now are to work with faculty who have used alternative
forms of assessment to provide a workshop for faculty.
The Office of Distance Education has faculty development as one of its
strategic goals and such a workshop will fit well into our strategic plan. References
Banta, T. W., Lund, J. P., Black, K. E. and Oblander, F. W. 1996. Assessment
in Practice: Putting Principles to Work on College Campuses. San Francisco:
Jossey-Bass.
Brown, S. and Knight, P. 1994. Assessing Learners in Higher Education.
London: Kogan Page.
Guthrie, D. S. 1992. “Faculty goals and methods of Instruction:
Approaches to Classroom Assessment”. In Assessment and Curriculum Reform. James
Ratcliff, editor. New Directions for Higher Education. No. 80, Winter 1992.
Pages 69-80.
Nixon, N. 1990. “Assessment Issues in Relation to Experience-based
Learning on Placements within Courses”. In World Yearbook of Education
1990: Assessments and Evaluation. Bell, C. and Harris, D. editors. London:
Kogan Page/New York: Nichols Publishing.
Prus, J. and Johnson, R. 1994. “A Critical Review of Student Assessment
Options”. In Assessment and Testing: Myths and Realities. Bers, T. H.
and Mittler, M. L. editors. New Directions for Community Colleges, No. 88,
Winter 1994. Pages 69-83.
Sendov, B. and Eskenasi, A. 1990. “Educational change and Assessment in
the Age of Information Technology”. In World Yearbook of Education 1990:
Assessments and Evaluation. Bell, C. and Harris, D. editors. London: Kogan
Page/New York: Nichols Publishing.
Worthen, B. R., White K. R., Fan, X. and Sudweeks, R. R. 1999. Measurement
and Assessment in Schools. New York: Addison Wesley Longman, Inc. Attachment ARequest
for Proposal Summer
Grant 1999 Request for ProposalsDistance Education Delivery Course Development Grants Summer 1999 Introduction: Again for summer President Tunheim announces the availability of funding for faculty development of alternative delivery courses. The focus for the grants is the improvement of instruction using technology. Of special interest this year are proposals that will use the capabilities of RDTN, the Governor’s Electronic Classroom, and WWW for distance delivery. We need to continue developing new approaches to teaching and learning, using these resources. In addition, new technology that faculty can work with this summer includes real video, CD-ROM and database support for online courses. The Distance Education Coordinating Committee is charged with effecting the distribution of the funds to faculty on a merit basis. The Committee solicits proposals from DSU faculty members for grants to support their development of innovative, creative alternative delivery materials and methods for DSU courses, especially those that utilize the targeted technology. The Committee encourages a broad, but forward-looking approach to technological innovation in submitted proposals. In addition to traditional proposals, faculty are invited to collaborate on projects that use the targeted technology to link different courses and disciplines. Eligibility: All courses in all disciplines, regardless of enrollment or year of offering and all approaches to alternative delivery, are eligible for development support under this program. Faculty applying for the Governor’s summer grants are not precluded from applying for the DSU summer grants. Funding Level: These grants in support of faculty time are anticipated to be in the range of $1500 to $3500. The level of funding will be determined by proposal merit, project complexity, actual cost of the project, and the number of proposals received. Program Timeline: Proposal submission deadline: April 23, 1999 Notification of Awards: May 5, 1999 Proposal Submission: Deliver proposals to the Office of Distance Education by 5:00 p.m., Friday, April 23, 1999. If you have questions regarding proposal preparation or submission, e-mail or call Deb Gearhart at 5049. Expectations of Faculty who receive grants: 1. Completion and submission of a proposal by April 23, 1999. 2. Development of alternative delivery materials during Summer 1999. 3. Completion of a final report at the end of the development effort (Deadlines are January 14, 2000 for Fall course delivery and May 26, 2000 for Spring course delivery). 4. Presentation of highlights of the developed material at a campus-wide development workshop during orientation week of Fall 1999. 5. Incorporation of the developments into the target course for Fall 1999 or Spring 2000. 6. Submission of a follow-up evaluation report (at the end of the semester the course is offered. If the course does is not taught, for whatever reason, the evaluation criteria planned for the course should still be submitted). Preparation of
Proposals: Proposal Narrative - A successful proposal must include a narrative section of no more than 5 pages (double-spaced) which addresses all of the following: A. Cover Sheet (attached) B. Objectives of the project. C. Description of course.
D. Identify and discuss delivery methods.
E. Significance of project to the college and the University. F. Identify any special facilities or material needed for successful completion of project. G. Indicate competence and preparation of primary investigator. H. Project as a model for other courses.
I. Evaluation of the deliverables of project.
Other Proposal Sections:· Vita (3 page maximum) of faculty member, emphasizing experience and skills relevant to the proposed development activity. ·
Budget-Faculty Time: The emphasis
of this grant is to provide faculty time to engage in development.
The dollar amount requested should relate to the intended amount of
development planned. The suggested range is $1500 to $3500 per course.
Other Related Costs: There is a small possibility that additional funds of a significantly lesser amount may be available to support acquisition of essential software or other support materials. If such additional support is important to the proposed development, then the faculty member should include it as a separate budget category, with the realization that such funding is not assured. The faculty member should discuss with her/his college dean the possibility of acquiring those support materials with college funds. Support materials that provide a truly innovative development opportunity or that have a wider application in actual or potential development effort by others in the college or on campus are most likely to be considered for such supplemental funding. Review Criteria:In evaluating proposals received, the Committee will be guided by the following criteria. The proposal narrative should address these and indicate how the proposed development meets the criteria. Intrinsic Merit
· Does the project represent a significant, innovative advance over current teaching practice or a significant expansion and modification of alternative delivery methods already practiced? · Does the proposed development effectively address the needs of the University and the discipline for introduction/expansion of alternative delivery courses and use of technology available at DSU? · Is the proposed use of alternative delivery methods appropriate for the target course(s)? · Is the proposed development easily extensible (likely to lead to and facilitate similar changes in delivery in other courses)? Performance Competence
· Does the faculty member have the expertise needed to accomplish the proposed course development in the given time period? Relevant information to support this competence should be included in the proposal and/or the vita. · Does the proposal show an awareness of current pedagogical issues, alternative delivery methods? Prioritization: In the evaluation of proposals, priorities will be established based on: 1. Creative, innovative proposals which meet the needs of the institution and the discipline(s). 2. Proposals which show promise of establishing pilot efforts that can proliferate into other courses and disciplines on campus. 3. Address the special focus of this year’s request. 4. Likelihood of success, based on experience and competence of the faculty member and the feasibility of the course development program outlined in the proposal. 5. The existence of "deliverables" (developed course materials) which will be available at the end of the development effort. 6. Equity of distribution of support across campus. Ownership of Materials:Materials developed as part of this grant process will be the exclusive property of the faculty member with the exception that the University retains the right to use the materials for the purpose for which they were developed without being assessed licensing fees. Disbursement of Grant Funds:Half of the awarded funds will be disbursed in July 1999. The remaining funds will be disbursed in September 1999. Distance Education/Alternative Delivery Course Development Proposal Cover Sheet Dakota State University Project Title: ______________________________________________________ _____________________________________________________ Faculty Name: _______________________________________________ Course(s) for which development is proposed (include course prefix, course number and course title): ___________________________________________ Collaborative projects – Colleges or disciplines participating: ____________________________________________________ Planned term for implementation: ___________________________ Funding Requested: $______________ Planned Interval of Development activity: __/__/99 to __/__/99 Attachment
B Online
Assistant Policy OFFICE OF RECORD: Academic Vice President/Provost ISSUED BY: Office of Distance Education APPROVED BY: 02-66-00 EFFECTIVE DATE: 8/10/98 Page 1 of 2 ON-LINE INSTRUCTIONAL ASSISTANTS POLICY The purpose and function of on-line instructional assistants is to assist faculty in the preparation/delivery of on-line or other distance education courses involving the significant use of technology. PROCEDURES
Allocation Each college offering an on-line or other qualifying course will be allocated a pool of 8 hours of student labor and/or work-study per 3 hr. course per week. The pool will equal 8 hours of labor x # of 3 hr. courses x 15 weeks x $8 per hour. For example, a college offering four 3 credit hour courses would receive a labor pool equal to 8 x 4 x 15 x $8.00 = $3,840. The allocation will be prorated for courses with greater or fewer credits. This is used to determine the amount of salary paid to the student. The actual amount of compensation per hour paid to any student may be more or less than the $8.00 rate depending on student qualifications and responsibilities. (See also Student Labor Salary Schedule policy #01-04-00.) This labor pool will be utilized by all qualifying courses in the college in a manner determined by each college each semester to best serve the needs of the faculty teaching the qualifying courses. The need and utilization may vary throughout the semester and all qualifying faculty desiring to utilize assistance from the pool will receive assistance from the pool. The college pool may be used to assist faculty in the preparation of courses to be offered in the succeeding semester. Qualifications/Compensation Because the needs of each qualifying faculty for support in offering on-line or other qualifying courses will vary, the colleges will determine for each semester the number and qualifications of on-line instructional assistants to be hired with their allocated labor pool. The colleges are encouraged to identify and recruit qualified students for these positions. Final PARs will be submitted to and administered by the Office of Distance Education (ODE). On-line Instructional Assistants must be in good academic standing and must meet all qualifications determined by the college to effectively perform the functions needed to support the qualifying courses taught by the college during the period of hire. Depending on the types of support needed, it is anticipated that individual student abilities and compensation rates may vary within and among semesters. 02-66-00 Page 2 of 2 General Support Because of the varying manner in which on-line instructional assistants are used within each college, workspace and furnishing arrangements are determined and provided by each college. A pool of traveling computers is also available to assist the colleges in effectively utilizing on-line assistants supporting courses in session. Evaluation As the result of working interactions between college personnel, on-line instructional assistants and the on-line instructional support team, an evaluation of the utilization of the On-line Instructional Assistant pool will be reported each semester. The report will include for each course: number of on-line instructional assistant labor hours worked by type of work; for example, e-mail management, assignment grading, responding to routine student questions, Web page design, Web page mark-up and other supporting functions. The Office of Distance Education and the Instructional Technology Support Team will compile and distribute the report. The information in the report will be used to gauge the type of student assistance needs for future semesters and provide a sampling of different ways of utilizing On-line Instructional Assistants for reference by new as well as continuing on-line faculty. The composition of the Instructional Technology Support Team is the Instructional Technologist (ODE), the Communication Network Analyst (CS), Director of Distance Education (ODE), Computer Support Specialist (CS) and on-line instructional assistants. General On-line
Support Effort In addition to working closely with college faculty, like the on-line faculty, the On-line Instructional Assistants will also work as appropriate with the On-line Instructional Technology Support Team from Computing Services and the Office of Distance Education to provide quality on-line educational experiences for enrolled students. See also DSU Policy #04-07-00, Distance Education Coordinating Committee.
Attachment
C Online Faculty SurveyOnline Faculty Survey Student Assessment in Online Courses
Attachment
D Courses
taught by online faculty surveyed Courses taught by the Faculty
responding to the survey Undergraduate Courses BUS 313 Cooperative Programs BUS 425 Production and Operations Management CSC 120 Logo CSC 150 Principles of Programming CSC 270 Special Topics: Develop and Manage Websites ENGL 101 Composition ENGL 201 Advanced Composition ENGL 210 Introduction to Literature ENGL 305 Grant and Report Writing ENGL 341 American Literature I ENGL 342 American Literature II ENGL 350 Computer Text Analysis HIM 350 Research Methods in Health Care HIM 360 Management of Health Information Centers I HIM 361 Management of Health Information Centers II HIM 443 Current Trends in Health Care Delivery HIM 444 Advanced Health Data Systems HIM 498 Seminar in Health Information Administration HIST 379 Native American Studies INFS 130 BASIC Programming INFS 221 COBOL I INFS 222 COBOL II INFS 267 Systems Analysis and Design MATH 341 Math Concepts I MATH342 Math Concepts II MUS 100 Introduction to Music MUS 312 Form & Analysis MUS 333 Music History Survey NFP 315 Fund Raising PSYC 101 General Psychology SOC 110 Career Planning SOC 240 Rural Sociology SOC 285 The Information Society SOC 360 Social Systems & Technology SOC 380 Social Institutions Graduate CoursesCED 596 K-12 Network Planning & Administration CED 685 Computer-based Educational Research and Statistics CHUM650 Computing for the Humanities Attachment
E Survey
Results ED814 faculty survey results Question 1 What types of assessment are you using to determine student performance in your online course? (examples – quizzes, exams, papers, participation, etc.)
ED814 faculty survey results Question 2 Are the types of assessment of student performance used in your online course different from the forms of assessment used in your traditional courses? If so, how?
ED814 faculty survey results Question 3 Please discuss the forms of assessment that have been successful in your online course. Explain why.
|