Chapter 23:

 Assessment of Undergraduate Education Beyond the Major


.

INTRODUCTION

.

This chapter addresses SFSU's assessment of several components of the undergraduate curriculum, specific programs, and issues that affect all undergraduate students, regardless of their degree objectives (1). The topics include:

.

• Basic skills development, including written English literacy at the lower division and upper division levels; and

.

• The General Education (GE) program, which consists of basic subjects, arts and sciences, and relationships of knowledge plus the overlay component of cultural, ethnic, and social diversity.

.

SFSU makes a distinction between the GE program itself and other graduation requirements; for example, although the information competence and U.S. history and government requirements must be satisfied by all undergraduates for graduation, they are, by campus policy, outside the GE program (2). In a similar manner, SFSU's written English literacy requirement (freshman and sophomore written English plus the upper division writing requirement) is partially inside of and partially separate from GE. Obviously the overlap in goals of these requirements necessitates consideration in terms of the role they play in the larger picture of the undergraduate's degree.

.

DEVELOPMENTAL INSTRUCTION AND
BASIC SKILLS DEVELOPMENT

.

SFSU's implementation of new admission requirements and measures to ensure early compliance with placement test requirements and time limits on completion of remedial courses have been described in Chapter 3. Actions to improve access and outcomes were also covered earlier. The focus of this section is the assessment of the basic subjects curriculum in written communication, oral communication, and critical thinking (3).

.

1. The information in this chapter serves to complement the chapter on undergraduate education beyond the major (Chapter 3).

2. These areas have not been the focus of current assessment endeavors and are therefore not covered in this chapter.

3. Assessment efforts in quantitative reasoning have focused on the university's remedial mathematics courses. These efforts, and their results, were discussed in Chapter 3..

.

Written English Proficiency

.

In the area of written English literacy requirements at the lower division and upper division levels, some brief background is necessary. CSU English Placement Test (EPT) results determine which English courses students take at SFSU. Remediation, if needed, must begin during the first semester of enrollment at SFSU, and all required remedial courses must be completed within one year from the time of admission to SFSU. Native speakers of English who score between 150 and 142 on the English Placement Test (EPT) are required to take ENG 50: Writing Skills Workshop and ENG 115: Effective College Reading (paired developmental reading and writing courses) as prerequisites to First Year Composition. Students who score 141 or below on the English Placement Test are required to take both ENG 49 and 50, a special two-semester sequence of composition instruction, along with ENG 115.

.

In addition to the EPT, all non-native speakers of English must take the SFSU English as a Second Language Placement Test (ESLPT) before they can enroll in ESL courses. Placement in ESL courses is determined by the ESL coordinator solely on the basis of ESLPT scores and not on any course work or tests taken at other institutions. ESL English courses are designed in a sequence, with the following courses equating to First and Second Year Written Composition: ENG 209 for ENG 114 and ENG 310 for ENG 214 and its variants. Undergraduate students must complete the CSU Graduation Writing Assessment Requirement by taking the Junior English Proficiency Essay Test (JEPET) after completing 48-semester units and before completing 80-semester units. Students must pass ENG 114 and 214 or their equivalents before they are eligible to take JEPET. Those who pass the JEPET examination will have satisfactorily met the upper division written English requirement. Those who do not pass JEPET must enroll in ENG 414 before completing 90 semester units. Passing ENG 414 satisfies the upper division written English requirement for those who did not pass JEPET. Non-native English speakers complete ENG 410 or 411 instead of JEPET.

.

Since this chapter is concerned primarily with GE and basic skills graduation requirements, the focus here will be on ENG 214: Second Year Written Composition, BLS (Black Studies) 214, RAZA (La Raza) 214, and ENG 310: English as a Second Language (ESL) Composition 2—all of which are GE Segment I courses at SFSU that conclude the lower division written literacy sequence. The following are summaries of the studies and what has been learned.

.

Second Year Composition

.

During Spring 2000, 26 English Department faculty teaching 33 sections of ENG 214 (632 students) participated in a study based on a scoring rubric derived from GE Segment I learning objectives for written communication. (Learning objectives for the four areas of Segment I may be seen at www.sfsu.edu/~senate/S99-64.htm.) In the assessment study of the ENG 214 course, a common final essay exam was given to all sections and evaluated by the faculty. In addition to assessing student learning in terms of the GE outcomes, the study endeavored to see whether the students' scores on the outcomes correlated with their final grade in the course and how well they understood literary text (since these were courses taught by the English Department). Based on the data, the coordinator of the English Composition Program concluded that the English Department has developed a valid instrument for assessment and has already achieved a high level of success and consistency in its teaching of ENG 214. The report also indicated several intended uses of the findings for program improvement, as well as a proposal to repeat the assessment yearly with a sample of the sections (20%) as a way to check to see that they continue to meet their goals in the course.

.

A similar assessment project was conducted by faculty teaching ENG 310, the course that fulfills the GE Segment I written English communication requirement for speakers of English as a second language (ESL) who have been in the U.S. for six years or less. ESL faculty developed and applied a scoring guide derived from the GE Segment I written English communication learning objectives. The guide developed for ESL courses was written to closely parallel that being used in the ENG 214 assessment project.

.

The essays of 97 students in ENG 310 were evaluated; this represents 25% of the students enrolled in ENG 310 during that semester. Based on the findings, ENG 310 students, on the average, performed adequately or better in all the skills covered in the GE Segment I learning objectives. Variations in the results of performance on the individual learning objectives will be useful in focusing the on-going work of refining course design and methods and choosing materials for the course. The results demonstrated that the ESL faculty members are teaching the strategies that enable students to become effective academic writers.

.

Second Year Written Composition courses taught in the College of Ethnic Studies include Black Studies 214, Raza 214, and Asian American Studies 214. Only the Black Studies and Raza Studies courses were taught in Spring 2000, the term of the Ethnic Studies assessment project. Black Studies described its assessment tools as papers written throughout the course and individual conferences with students. Students’ diagnostic essays revealed severe problems with thesis development, organization, and mechanics. Students corroborated lack of sufficient instruction in these areas during conferences. So, in addition to referring students to tutorial services for mechanics and usage remediation, faculty in Black Studies resolved to emphasize instruction in thesis development and organization of the whole essay. Communication with other instructors was also identified as essential, in order to build consistent efforts toward improving students’ writing.

.

The RAZA 214 assessment report compared the teaching approach and experience in RAZA 214 to the learning objectives for Segment I written communication. The conclusion was quite similar to that identified above—problems with thesis development were particularly prevalent, but significant improvement occurred following concentrated attention through instruction, practice, and instructor feedback.

.

JEPET

.

Assessments of student performance on SFSU's Junior English Proficiency Essay Test (JEPET) have been conducted for many years. JEPET is the upper division written English proficiency exam at SFSU. The upper division course, ENG 414, that is required for those who fail the test, has also been assessed for many years. Non-native English speakers take ENG 410/411 in lieu of JEPET and these courses have also undergone periodic evaluation.

.

The most recent JEPET report summarizes JEPET data from 1993-2000 and compares those data to outcomes from ten years earlier. What is highly significant in the data, according to the report, is the generally stable fail rate of approximately 40%. This statistic, more than any other in the report:

.

underscores the significant and on-going problem of how literacy is addressed in higher education. Clearly a mere 32 weeks of lower division course work—comprised of one freshman and one sophomore writing course (the latter of which focuses primarily on literary analysis)—continue to be insufficient to prepare our linguistically diverse and often academically under prepared student population to meet the kinds of literacy standards we hold for our upper division students.

.

The report went on to cite the inability of faculty teaching large classes to require and carefully evaluate extended writing assignments. Additionally, the report cited the failure of many students to take JEPET in a timely way as a factor that contributes to the difficulty in preparing students for their upper division course work.

.

English 414

.

The second report summarized the assessment project conducted on the upper division course (ENG 414) required of those who fail JEPET. Briefly, in order to receive credit for 414, students must demonstrate the consistent ability to write upper-division expository/argumentative essays. In this study, 20% of the ENG 414 sections (196 students) during Spring 2000 were identified as participants. By the end of the semester, the actual number of students with complete files—background information sheets, a copy of the JEPET exam failed (the pre-test), and the final exam in the course (the post-test)—was lower (150) because of missing information, attrition, etc. Nevertheless, some important findings emerged. Despite the challenges of teaching students to be proficient writers of upper-division expository prose in only 16 weeks, 55% of the students were able to move from failing JEPET to passing the writing test at the end of the course. The remaining 45% had not reached the level of competence required to pass the sample JEPET-like post-test, but many of them improved sufficiently on the individual measures in the ENG 414 grading criteria to receive credit for the course and thus complete the university written communication requirement. (The average pass rate in ENG 414 is 92%, with the remainder of the students having to take the course again.) Throughout the ENG 414 courses, students are provided with individual conferences with faculty and referred to campus tutorial programs for additional support. Areas in which the greatest gains were reported include syntax, providing adequate evidence to support a thesis, and responding to the rhetorical task. Implications of some of these data were discussed in the report, as were conditions for best use of the findings in program improvement.

.

Basic Skills in Oral Communication

.

The oral communication requirement in GE Segment I is fulfilled by taking either of two courses—Speech 150: Fundamentals of Oral Communication or ENG 210: ESL Oral Communication. Two assessment projects involving assessment of learning objectives in oral communication will be described here. A Spring 1999 study involved five assessment measures for SPCH 150: pre and post measures of oral communication competence and apprehension, plus students' and faculty's evaluation of achievement of the GE learning objectives for oral communication. In addition, faculty members evaluated students' public speaking skills after completing the SPCH 150 course. The assessment tools were a combination of published and department generated forms. Each of the assessment measures provided a positive evaluation of the learning taking place in SPCH 150. Students increased their perceived communication competence and reduced their communication apprehension in multiple contexts. Students and faculty reported that the GE learning objectives were achieved, and faculty rated the public speaking skills of students as being "adequate" to "good." The report pointed out similarities among the results of this study and two other studies: "credible evidence" and "critical analyses" were often absent in the speaking and writing of students assessed. (Details will be presented later in this chapter on one of those studies.) The Speech 150 Spring 1999 study offered this conclusion:

.

In combination, these data from both beginning students and students about to graduate indicate that students may benefit from additional instruction on constructing and assessing claims, including the use of relevant supporting materials and citations. This instruction would be most effective if it occurred not only in Segment I (GE Basic Subjects) oral communication classes, but if it was also reinforced across a student's college experience in multiple classes.

.

Further, the report prompts, the GE Council:

.

may wish to consider whether the skills taught in Segment I oral communication classes ought to be reinforced in other GE classes, and if so how.

.

In June 2000, a second study was conducted to again assess the degree to which SPCH 150 meets its stated objectives. Three measures were selected for assessing achievement of the learning objectives. One was a published instrument for detection and comparison of self-reported levels of communication apprehension (i.e., personal reports about communication with other people) that could be used as a pre- and post-test. Faculty evaluations of communication competence were performed on the first and last presentations of students in SPCH 150. The assessment instrument used for this purpose was developed by the department as a standardized form for evaluating the content, structure, style, and impact of the oral presentations. Twenty faculty teaching thirty-one sections measured twenty-four performance criteria on a scale from 1–5 (869 students). At the end of the semester, faculty completed post-test assessments in 24 sections (624 students). At the end of the semester, students also completed self-report measures of the degree to which they individually perceived the course as having achieved its objectives.

.

The findings of the study indicated that, in the area of communication apprehension, improvement was reported for all of the items measured. Outcomes of the self-reported portion were very positive, as evidenced by ratings indicating that students agreed that all learning objectives were achieved. On the instructor critiques, students were seen as improving in all but two of the performance criteria—the use of "credible evidence and examples" and "effective use of visual aids."

.

The outcomes here were remarkably similar to those described for written communication in the previous section of this chapter. The course had the greatest impact on the students’ abilities to provide a thesis or purpose statement, preview the organization of the presentation, provide an effective summary, reinforce the central idea, etc. In terms of uses of the findings, again there were similarities. The assessment projects helped the department identify components where additional or sustained emphasis was needed. The appropriateness of various assessment tools or combinations of tools will continue to generate both experimentation and debate. Also, the use of consistent faculty ratings will involve ongoing departmental as well as campus-wide communication.

.

Instruction in Critical Thinking

.

The Basic Subjects requirement in critical thinking at SFSU may be met by completion of any one of nine different approved critical thinking courses. Five of the courses are taught by the College of Ethnic Studies and carry the same title (Critical Thinking: The Third World Experience) but are taught in each of five departments/programs (American Indian Studies, Asian American Studies, Black Studies, Ethnic Studies, and Raza Studies). Three other courses are taught by the College of Humanities in three different departments (Writing Logically is offered as English 200, Introduction to Critical Thinking I is offered as Philosophy 110, and Argumentation and Advocacy is offered as Speech Communication 250). Finally, one course is taught by the College of Behavioral and Social Sciences (The Logic and Psychology of Critical Thinking is offered as Psychology 111). Assessment reports were submitted by the College of Ethnic Studies and by the Philosophy Department.

.

The Philosophy Department regularly offers the largest number of sections of the critical thinking course (at least twenty sections each semester). The department assessment report described the results of the different assessment reports prepared by each of the sections of Philosophy 110. The department report gave equal attention to both the convergences and divergences among the different reports. Questions important in the process of any assessment endeavor were raised:

.

To the extent that the outcomes described in different classes are different, then it forces more careful attention to the source of these divergences and hence a deeper understanding of the definition of our goals or learning objectives in teaching critical thinking. Are the divergences in outcome an artifact of different assessment procedures or do they reflect genuine differences among the students in different classes? Alternatively, are they the result of different teaching techniques that are especially effective in the attainment of one or more of the learning objectives? If so, then typical members of our critical thinking staff have much to learn from one another and a period of healthy experimentation will have been ushered into being in the teaching and evaluation of Philosophy 110. Moreover, the goals of the course as described in the learning objectives will be able to be formulated in a way that is more clear and precise while, equally important, individual instructors will learn from one another about different ways to improve their teaching of Philosophy 110.

.

The report described the process by which the faculty teaching critical thinking learned from one another in significant ways, as they agreed to submit their plans for assessment to the critical scrutiny of their colleagues to take full advantage of the cross fertilization that inevitably occurs in any serious, practical discussion among teachers about teaching methods. Ten faculty members submitted reports on their sections of PHIL 110 (some who taught multiple sections submitted individual reports on each section and some submitted a single report covering multiple sections).

.

The report described the outcomes as follows:

.

Without exception, these many different measures of success in critical thinking resulted in the conclusion that our classes had attained a high level of success at meeting the learning objectives of the course. Though this success is described in a number of different ways, the convergence of each of these different reports to the same result is both telling and reassuring.

.

The reported divergence regarding the attainment of learning objective #13 (writing reasoned discourse that provides a detailed critique of a complex argument) seems to indicate, according to the report, that: "careful attention to this crucially important learning objective is now in order." It is important to point out that, as a GE basic subjects course, this and other critical thinking courses may be taken by first year students even though their other courses may be remedial English courses designed to help them attain college level written literacy. There are no prerequisites for critical thinking courses. The difficulty in meeting this learning objective may well be related to the entry-level skills of the students in written communication.

.

The above appears to reinforce several of the outcomes of the Basic Subjects (both written and oral communication) assessment reports described earlier; namely that: (1) while the courses are achieving considerable success in accomplishing the learning objectives of the Segment I areas, there may be a need for more intensive and perhaps more extended remedial/developmental writing instruction; (2) there is a need to reinforce/promote the skills taught in basic subjects courses throughout the curriculum; and (3) the outcomes point to the importance of departmental communication regarding both instructional strategies and evaluation criteria. This is obviously not a statement intended to criticize the work that has been accomplished; it is a restatement of the findings in several of the reports and reinforced in the detailed report from Philosophy.

.

The College of Ethnic Studies produced two reports on GE Basic Subjects critical thinking courses offered in the college. A Fall 1999 survey of faculty teaching critical thinking in the college resulted in a preliminary report on the various ways that the 13 learning objectives are assessed. The assessment activities in Spring 2000 included data collected from the faculty on more specific teaching and learning evaluation activities as well as feedback from students participating in critical thinking sections. The first report presented the course descriptions (from course syllabi) of each of the critical thinking courses taught by the college, followed by a listing of each of the 13 learning objectives for GE critical thinking. Under each learning objective was a brief description of the instructional method and evaluation tool employed by each course instructor.

.

The report also indicated that, during the discussion and data collection process, several Ethnic Studies faculty identified additional objectives relating to development of an "ethnic studies" perspective on critical thought and issues. The report summarized the outcomes of the collection of instructor methods and evaluation tools as follows:

.

This compilation of faculty reports revealed consistent evidence that the 13 student learning objectives are being covered in the critical thinking sections and that students are being evaluated on their understanding and mastery of the objectives. There also was evidence that diverse methods, textbooks, and approaches are being used to impart the knowledge and skills of reasoning in the different sections including, for example, philosophical inquiry, social scientific methodology, legal reasoning, and ethnic studies critical analysis.

.

The goal of the college’s second assessment project was to learn how the learning outcome objectives of critical thinking courses could be assessed and what the common threads were among sections of this course in the college. The faculty also wanted to learn how a critical thinking course would be different in the College of Ethnic Studies from critical thinking courses in other colleges. In terms of outcomes, the following was reported: "The sections of critical thinking offered in the College of Ethnic Studies do reflect the mission of the college and its emphasis on ethnic studies 'added value' curricular requirements expected of all students." Emphasis on ethnic studies "added value" facilitated focusing on practical applications of reasoning skills. In some instances, this approach gave students an opportunity to develop problem-solving skills in addition to other expected learning outcomes.

.

GENERAL EDUCATION

.

Assessment endeavors were part of the original design of the General Education (GE) program at SFSU. Academic Senate Policy F80-64, from which the current program originates, presented the goals of the program as guidelines, along with procedures for developing detailed objectives and evaluating new and existing courses/programs as to their appropriateness for General Education. The task of developing the specific objectives and learning outcomes was assigned to the several General Education committees: Basic Subjects, Arts and Sciences Core, and Relationships of Knowledge.

.

Although the 1980 policy recognized the need for a system by which courses in the program could be evaluated for the purpose of making decisions about GE, the focus of committee activity became the evaluation of proposals for new courses. Detailed proposal guidelines and requirements that objectives be clearly identified have been in use since 1980. With the exception of some departmental endeavors, ongoing courses have not been systematically evaluated in terms of their consistency over time with the original intent of the GE program or their deviation from these objectives. However, at the time of this writing, SFSU can show several notable examples of assessment projects incorporating both assessment of student learning/subject matter competency and assessment of GE learning outcomes.

.

Development of GE Learning Objectives

.

The development or refinement of a complete set of specific objectives and learning outcomes for all components of the program (not just new courses) has been an evolutionary process:

.

• As of February 24, 1989, the GE Segment III Relationships of Knowledge policy includes learning objectives for this component of GE and a stipulation that evaluation will be conducted in terms of the objectives and requirements [see www.sfsu.edu/~ugs/three.htm].

.

• As of January 1993, GE Segment II Arts and Sciences policy includes learning objectives for Segment II and a stipulation that evaluation will be conducted on a regular basis and that this monitoring and evaluation will provide the major basis for considering continuation, modification, or termination of each course [see www.sfsu.edu/~ugs/two.htm].

.

• As of May 11, 1999, the GE Segment I Basic Subjects policy includes learning objectives for each area plus an evaluation component that addresses assessment of the extent to which new and existing courses meet the learning outcomes, a regular reporting schedule, and a remedy for non-compliance with the assessment plan [see www.sfsu.edu/~senate/S99-64.htm].

.

Assessment Activities and What's Been Learned

.

Assessing General Education in a broad sense is a monumental task due to the diversity of content and courses that are encompassed within the GE program. This task has been the responsibility of the General Education Council (GEC), the Office of Undergraduate Studies, and the many departments that offer General Education courses. Over the last few years, several different assessment activities have been undertaken to determine student and faculty perceptions of the GE program at SFSU and, more recently, student learning outcomes.

.

Assessment activities that will be discussed in greatest detail are:

.

• The 1995-96 GE Program Review including the Public Research Institute (PRI) survey of General Education.

.

• The 1999-2000 feasibility study of assessment of student learning outcomes in basic subjects and arts and sciences breadth using the ETS Academic Profile.

.

• The Segment III Committee assessment of student opinions and learning in the Segment III: Relationships of Knowledge component of the GE program.

.

• The University Interdisciplinary Council assessment of interdisciplinary thinking among SFSU students.

.

• The study of the basic critical thinking and communication skills of graduating seniors in capstone courses in academic majors.

.

Other, more limited, assessment endeavors will be described in chronological order as they occurred among the major endeavors.

.

In Spring 1995, the Office of Undergraduate Studies conducted a survey of transfer student evaluations (Advanced Standing Evaluations) of 750 Fall 1994 admits. The purpose of this study was to determine the percentage of transfer students who had completed the following requirements prior to admission: Segment I: Basic Subjects; Segment II: Arts and Sciences Core; U.S. History and Government; and a course equivalent to First Year Composition (our English 114). Sixty percent of the Fall 1994 admits had already completed all four of their Segment I: Basic Subjects courses. This information, along with the current system-wide mandate that upper-division transfer admits must have completed their Basic Subjects courses to qualify for admission, allowed SFSU to better predict the need for sections of Basic Subjects courses.

.

Previous GE program reviews and accreditation studies called for attention to issues of numbers of courses in the GE program, student access, and resources. In response, the General Education Council continued to monitor the functioning of the Segment III component of GE. In Fall of 1995, upon the recommendation of the Segment III Committee, several Segment III clusters were dropped after consultation with faculty coordinators, and other clusters were substantially revised to improve availability to students. Annual reports of cluster coordinators, in which course offering histories were reported, along with consultation with GE advising coordinators regarding GE petition traffic, provided helpful data for these Segment III Committee actions.

.

In Fall 1995, the Segment III Committee again analyzed the cluster coordinator annual reports from the previous academic year and the results of the Segment III survey of graduating students from the previous spring semester. Results were summarized for the GEC and incorporated into the program review process. Also in Fall 1995, the Segment III Committee formalized the qualities of the writing component of Segment III by issuing writing guidelines for new Segment III courses.

.

In Spring 1996, Undergraduate Studies conducted a detailed survey of the General Education course and cluster selection patterns in the May 1995 graduation applications. Detailed information was provided for both native students and transfer students. The report included data on courses not selected and on those courses most frequently selected by students. More than 75% of courses in the GE program serve multiple purposes in the curriculum. The Undergraduate Studies Office used the data in conjunction with the General Education Council in discussions with deans and department chairs to determine where curricular adjustments needed to be made.

.

1995-96 GE Program Review

.

In Spring 1995, the scheduling of the 1995-96 GE Program Review was announced. The General Education Council divided into subcommittees to plan the program review for 1995-96. In planning for the review, the General Education Council members had to consider many of the same limitations noted by their predecessors in 1987. Limited time and limited resources continued to be principle factors in the slow evolution of assessment of our General Education program. Although some routine ongoing assessment tools were in place, the program still lacked regular assessment of some portions. As noted by WASC, assessment efforts to date had concentrated on design, accessibility, and enrollment patterns much more than measuring learning outcomes and whether courses were serving the objectives of the program. Among the issues surrounding program assessments, one of the most delicate and difficult concerns the distinction that must be drawn between reviewing the program and reviewing the work of an individual. Collective bargaining agreements as well as academic freedom and tradition provide that individual faculty performance may only be reviewed by appropriate peer committees and administrative officers. This consideration had been raised in the past when the GEC was planning assessment for portions of the program and was an important factor in the 1995-96 design process.

.

By mid-fall 1995, the General Education Council had commissioned the SFSU Public Research Institute (PRI) for a comprehensive survey of students and faculty that would include some questions from the 1986 survey for comparison purposes. This survey included faculty, General Education instructors, students, and recent graduates. The goals of the 1995 survey were to re-use all but the obsolete questions from the 1986 survey and to collect additional information as determined by the GEC from previous recommendations, long-standing issues, and philosophical goals. The 1986 survey included only recent graduates; PRI recommended that current students be included in 1995 in order to gather information from students who may not have experienced as much success with General Education as actual graduates. Students of color were over-sampled in 1995 in order to make statistically reliable comparisons of attitudes and experience among ethnic groups.

.

In the 1986 study, students were asked to rate the overall quality of their experiences and to indicate the importance and usefulness of the basic objectives for each segment of the program. In 1995, both faculty and students were asked to rate the importance and usefulness of a revised set of objectives reflecting the current program. The two surveys collected other similar attitudinal data and information on how students select courses. The 1995 surveys added new sub-elements to continuing questions and added questions seeking information on how well informed students and faculty felt they were.

.

The 1995 student survey added several questions aimed at determining accessibility and eliciting reasons that students found it difficult to complete requirements. The faculty survey added questions about alternative models for the upper-division Segment III requirement, class size, and preferred proportions among GE, the major, and other requirements in completing a B.A. Important questions were also asked of the faculty concerning the mixing of majors and GE students in the same courses and concerning proposed additions to General Education such as foreign languages, service learning, reduction of prejudice and discrimination, and information skills. For the first time, the faculty survey also asked GE instructors in Segments II and III to what extent they required students to use basic skills and to what extent the typical student demonstrated proficiency in those basic skills.

.

The General Education Council also sought participation in other ways by a broad spectrum of interested individuals and groups. To supplement the PRI survey, the GEC gathered opinions and information through a variety of meetings and through a supplemental survey for department chairs. Focus groups were organized to solicit the views of faculty and administration. The questions were tailored to each event, but typically included questions of clarity, resources, problems, and solutions. Those meetings and the results of the department chair survey were summarized.

.

1995 General Education Surveys: As was mentioned above, in 1995, a comprehensive survey was conducted of students, graduates, and faculty to determine their ratings of the GE program at SFSU. This study was conducted by PRI under the auspices of the GEC, the dean of undergraduate studies, and the vice president for academic affairs. The purpose of the survey was to determine student, recent graduate, and faculty opinions, beliefs, and experiences regarding the GE program. Questionnaires were sent to all faculty, to samples of students, and to May 1995 graduates. The response consisted of 350 completed surveys from students, 361 from 1995 graduates, and 423 from faculty.

.

Results from the students and graduates indicated that they felt their GE experiences were excellent or good. They felt the goals and requirements of the GE courses were important and useful and that the GE program was informative, comprehensive, challenging, practical, enjoyable, and theoretical. The students indicated only one area of concern, which was access to courses. This problem was most evident with Segment III GE courses, and the students who were most likely to have this problem were those with heavy work schedules.

.

According to the survey, the GE Program increased student knowledge of diverse cultural groups. Seventy-nine percent of the students responded that they agreed with the statement. (The percentage had risen from the 64% who agreed with the same statement in a 1986 study.) In addition, student agreement with the statement that cultural, ethnic, and social diversity was a focus in their Segment III course work ranged from 87-89% among the respondents in the biennial survey conducted by the Segment III Committee of students applying to graduate between 1992 and 1996.

.

Faculty were somewhat less likely than students to rate the overall quality of the GE program as excellent or good. However, faculty were more likely than students to rate most GE goals and requirements as important. The areas rated highest by faculty were intellectual content and quality of instruction and those rated lowest were the structure of the program and GE committees.

.

Suggestions for change of the program ranged from complete restructuring to modifications. Faculty who taught GE courses rated the program higher and were less likely to recommend that it be changed. In comparing the 1995 findings to those of 1986, faculty in 1995 were substantially more likely to rate the GE program, its program structures, and the structure of GE committees excellent or good than in 1986.

.

Perhaps the most compelling information in the PRI survey was also the least amenable to categorization and interpretation—the transcribed comments from open-ended questions. Both faculty and student comments represented a wide range of opinion and ran the gamut from coolly cerebral to highly emotional. Many took the time to provide thoughtful and constructive feedback and many students expressed their thanks for asking for their opinions.

.

Comprehensive Assessment of GE

.

To check the health of the GE program in terms of its success in achieving learning outcomes, several additional measures have been employed. As described earlier in this chapter, numerous departmental and interdepartmental assessment projects have contributed data and analysis to facilitate this process. On the agenda of the GE Council, however, was a plan to administer a comprehensive measure of the academic skills acquired through GE. The research subcommittee of the GE Council investigated several means of assessing the effectiveness of the GE program. Among them was a proposal to develop our own comprehensive instrument or at least to conduct a pilot test of the feasibility of developing such an instrument. The GE Council planned to consult the GE Segment Committees for their perspectives and cooperation.

.

With the exception of the Segment III Committee, the attempt was a failure. The Segment III Committee examined Segment III learning objectives, developed its own assessment instrument, designed its own study, and ultimately analyzed its own results. But efforts to entice/cajole/conscript/incite the committees for the other two GE Segments were futile. There are lessons learned. Closer contact between the GEC and the sub-committees is crucial. The GEC has since begun to designate liaison relationships between the GEC and the subcommittees and explore other ways to increase communication and interaction. Especially in relation to Segment II, the Arts and Sciences breadth area, the GEC developed a proposal to reconfigure the Segment committee which is currently not one committee but five—one for each of the breadth areas and one for each overlay requirement. A Segment II "super committee" was also proposed. The GEC agenda for 2000-2001 includes follow-up on this and other lessons learned in this process of GE assessment.

.

The decision, in 1999-2000, after unsuccessful attempts to reactivate each of the GE committees, was: (1) to support the Segment III Committee in developing its own instrument; and (2) to select a commercial, standardized instrument with which to conduct a pilot study to explore methods of assessment for Segment I (Basic Subjects) and Segment II (Arts and Sciences Core).

.

The GEC had endorsed the position that assessment of student learning is best achieved with measures developed by campus faculty. However, due to limited time and other constraints indicated earlier, the GEC decided to pilot test a standardized instrument, the Educational Testing Service’s Academic Profile, to determine if it would meet our assessment needs in these basic areas. This test would provide norm-referenced total scores and criterion-referenced scores in the basic subject areas of writing, math, and reading/critical thinking. Norm-referenced scores would also be provided in the areas of humanities, social sciences, and natural sciences. In these areas, students weren't tested for knowledge content specifically, but were tested regarding broad issues, themes, and ideas.

.

The short version of this instrument was tested on a small group of junior and senior students in Spring 2000. Initial plans to administer the instrument to a random sample of students applying to graduate were unsuccessful. After generating a random sample, postcards were sent to the sample of students asking them to come to the assessment sessions for which they would be compensated with a small gift or token payment. After only a very small number of students responded, faculty involved in the project recruited students so that the project could be conducted despite the loss of randomness in the sample. Initial results indicate that our junior and senior students scored comparably to other students in similar institutions.

.

Spring 2000 Academic Profile Norm-Referenced Results (4)

 

Total

Reading

Writing

Critical Thinking

Math

Humanities

Social Sciences

Natural Sciences

SFSU

450.3

120.8

118.0

113.2

114.0

118.3

116.9

117.2

Comparable Universities


448.9


120.1


116.8


112.6


114.8


116.2


115.3


117.7

.

(4) SFSU upperclass student scores (n = 48) compared to student scores at comparable universities from The Academic Profile Comparable Data Guide, Educational Testing Service, Princeton, New Jersey, 1998.

.

Criterion-referenced scores provided more detailed information regarding the number of students who scored at various levels in writing, math, and reading/critical thinking. These scores can be of value if this test is utilized over time to evaluate student improvements. Interventions that are implemented to strengthen student achievement in these areas could be monitored to determine their success.

.

In Fall 2000, the long form of the Academic Profile was administered to a selected group of incoming freshmen with the intention of providing information regarding where students score prior to taking college-level General Education courses. The value of these data will be in administering the examination again as the students progress through and finish their education at SFSU.

.

The administration of the instrument in the pilot study has been for evaluation purposes only; and if the GE Council decides to continue to use the Academic Profile, funding and support from administration and the Office of Academic Planning and Assessment will be essential. Implementing this examination was costly and required considerable administrative staff support.

.

Assessment of GE Segment III: Relationships of Knowledge

.

Each CSU campus interprets the system-wide upper division General Education requirement in different ways. Here at SFSU, this requirement is a cluster of three courses around a central theme. An objective of the clusters is that students "gain an appreciation of the complexity of problems and issues that confront individuals and societies and gain a greater appreciation of human achievements." Assessing the knowledge acquired by students in the 59 clusters and more than 400 courses with one assessment instrument is impossible. Faculty in these courses evaluate individual learning through course assignments, so the Segment III committee decided to assess student learning through a campus-derived instrument that focused on the broad goals of Segment III.

.

The development of this assessment of student learning was through two phases. In Phase One, the members of the Segment III committee developed the assessment questions. The instrument was then pilot tested on a small group of students, and a focus group was conducted to provide the students an opportunity to give feedback regarding the instrument and their Segment III experience. Results from the focus group discussion were fairly positive regarding their Segment III experience. One area of concern to the faculty researchers, however, was little evidence of the existence of a significant writing requirement in several of the Segment III classes the students had taken. This is a requirement of all Segment III courses and will be assessed further in subsequent assessment activities to see if this is a real problem or an aberrancy noted only in this small group. Based on the feedback of the students in the focus group, the instrument was revised. The revised instrument was then administered as Phase Two in Spring 2000 to 30 senior students. The number of participants in the study was intentionally small; however, attaining the participation of even the intended 30 students was difficult. Student participants were compensated with small gifts or a token payment.

.

In Phase Two, the participants included 43% men and 57% women. Caucasian and Asian students each represented 37%, with the next largest group being Hispanic at 13%. Sixty percent were transfer students from another college. Eighteen different clusters were represented, with 27% of the responses representing the Human Sexuality cluster.

.

Analysis of the quantitative data regarding the students' perceptions indicated that they rated all indicators towards the positive (1=strongly agree, 3=neutral, and 5=strongly disagree). Most could identify the cluster theme, its multidisciplinary aspects, and that the instructors demonstrated excellent knowledge of the subject matter. The students did perceive that cultural, social, and/or ethnic diversity (CESD) content was presented in the cluster. Although the mean responses on all questions were positive (indicating "agree" to "strongly agree"), some areas that may be improved were primarily academic areas, including the responses to the following: the assignments were challenging (x = 2.76); writing assignments encouraged perspectives from different disciplines (x = 2.66); the assignments encouraged discussion, analysis, and evaluation (x = 2.44); the courses helped students to develop their abilities to relate theory and practice (2.37).

.

Most students were able to indicate the name of their cluster (97%) and the courses they took (86%). However, only 52% were able to explain the theme of the cluster, and only 28% were able to provide an example of how each course contributed to the cluster. Of concern was that only 14% were able to give a comprehensive response regarding how the cluster courses expanded their awareness of the cultural, social, and/or ethnic diversity (CESD) presented in the courses.

.

Assessment of Interdisciplinary Thinking

.

During academic year 1998-1999, the University Interdisciplinary Council (UIC) embarked on a process to assess interdisciplinary thinking among SFSU students. The UIC wanted to find out if students could demonstrate:

.

1. Awareness of connections across fields of knowledge;

.

2. An understanding of the meaning of discipline as a paradigm for obtaining and interpreting knowledge, and the necessity, therefore, for interdisciplinary efforts to transcend the limits of disciplines when the problem at hand requires a less specialized and more comprehensive approach;

.

3. The capability of drawing information, insights, and theoretical perspectives from multiple disciplinary and interdisciplinary sources for the purpose of seeking solutions to actual social, expressive, and technical problems.

.

Given the obvious convergence of UIC goals with GE learning objectives (especially Segment III: Relationships of Knowledge), the project was conducted in conjunction with Segment III professors, who offered students in their courses as some of the participants in the UIC project.

.

In addition to assessing the use and understanding of interdisciplinarity among SFSU students, several companion objectives were built into the design of the project. Fostering interdisciplinary approaches to examining issues is, of course, an intended learning outcome of GE Segment III: Relationships of Knowledge. So while the UIC was looking at interdisciplinary teaching/learning across the curriculum, this assessment project had implications for GE assessment in particular. Furthermore, the development of critical thinking skills, written English proficiency, and information competence are GE and undergraduate degree objectives that could be observed in the outcomes of this UIC assessment project. Therefore, results of the demographic and objective/short answer items were to be analyzed from multiple perspectives, and a scoring rubric was developed for the written responses.

.

Getting at these different aspects and levels required both quantitative and qualitative measures. The approach of providing a case study that could be understood and/or solved only by using interdisciplinary knowledge and seeing how the students performed was selected by the UIC. After extended consideration and consultation concerning approaches, and discovering no comparable instrument at another institution, the UIC developed its own instrument consisting of 32 objective and short answer items plus 4 case studies/problems designed to elicit application (or lack of application) of an interdisciplinary approach to responding to the case study/problem in question. Students were asked to select one passage and produce an explanation of the problem; an identification of the types of information and expert consultation that would be helpful in designing a solution to the problem; a critical examination of several strategies for dealing with the problem; and a recommendation, based on the students’ experiences, of any courses offered at SFSU that members of a community group interested in solving the problem might take.

.

Over 400 participants were involved in the Spring 2000 project, and a subsequent study to increase the number of participants was conducted in Fall 2000. Outcomes from the Spring 2000 study showed that GE Segment III does have "a significant impact on students' awareness of and understanding of interdisciplinary thinking." The results suggested that, by graduation, most SFSU students have some proficiency in interdisciplinary thinking. Some idea of areas that could benefit from further attention was also provided by the surveys. The essay portion of the survey presented an opportunity to observe performance in the areas of critical thinking, writing proficiency, information competence, and interdisciplinary thinking. The essay completion rate, however, was disappointing. Only 58% of the students who completed the survey also completed the essay. Time restraints were a factor, as were lack of incentives for students to engage in a complex activity as part of a voluntary project. Analytical reports produced by SFSU's Public Research Institute enumerated the strengths and weaknesses of this initial endeavor.

.

The potential for success in the further development and implementation of this kind of study is substantial. Also substantial was the quality and quantity of inter-departmental consultation and input. An instrument that began as an outgrowth of a university committee of interdisciplinary faculty ultimately encompassed the goals of a much broader set of faculty and administrative units. It can stand as a model of consultation among GE committees, the English Department, the Library, the UIC, and the Liberal Studies Council.

.

Assessing Basic Skills of Graduating Seniors

.

An innovative project designed to assess students' abilities to use basic communication skills in the context of their subject matter disciplines—that is, their ability to speak, write, and think about concepts important to their chosen fields—was developed jointly by faculty from Speech Communication and English and conducted along with several faculty from other disciplines: business analysis and computing systems, consumer and family studies, international relations, management, geosciences, history, and art history. The researchers designed an assessment plan that asked students to: (1) compile a portfolio that consisted of writings completed in upper-division courses as well as written evaluation of those writings; and (2) give a speech on one of the portfolio texts to a group of faculty evaluators and peers. The project researchers worked with the subject matter faculty to develop guidelines and criteria for evaluating critical thinking and oral and written communication skills, developed instruments to evaluate the writing and speaking assignments, and pilot tested them with senior-level students. Finally, the researchers carried out the assessment process with the subject matter faculty members and collected feedback from and provided feedback to participating students.

.

According to the report, the input of subject-matter faculty was crucial as they considered criteria needed to assess content issues in speaking and writing; for example, the types of evidence or analysis that would be expected in the varied disciplines. A great deal of discussion was needed to decide how critical thinking could be best assessed within writing and speaking assignments; in the end, they agreed that critical thinking overlapped with speaking and writing in that all three of the skill areas involved: (1) using clear and unambiguous language; (2) organizing in a coherent form that provides a logical argument; and (3) providing substantive content with evidence used to back up claims. The team members therefore decided that the criteria for critical thinking would be best merged with the criteria for writing and speaking in their assessment forms.

.

The subject matter faculty had the task of selecting senior student participants who represented the range of abilities found within the major. Student selection turned out to be one of the most difficult tasks; even with a $50 stipend and a promise of certification of competence for students' resumés, students were reluctant to participate because the assessment had no true bearing on their degree work. As a result of this difficulty, the strategy was changed during the next semester, enlisting faculty participants who were teaching capstone courses. All students enrolled in these capstones were automatically enrolled in the project, with faculty members writing the project requirements into their syllabi. All student participants were given a certificate of completion, and those who passed both activities were certified as competent in the basic skills of writing, speaking, and critical thinking.

.

The researchers concluded that their instruments were basically sound since they were able to distinguish between adequate and inadequate performances in oral and written communication. A statement from the assessment report summarized an important outcome of the assessment project in terms of faculty perspectives and highlights a dilemma faced across campus:

.

In most cases basic subjects and subject matter faculty members agreed on the pass/fail decision; when they disagreed, discussion always resolved the matter. Most often disagreements occurred when the subject matter faculty focused exclusively on content and the basic subjects faculty member was heavily influenced by organization or language/delivery issues.

.

This is the message that we will find reiterated over and over; basic skills must be reinforced throughout the curriculum, including, as indicated here, in content area upper division capstone courses.

.

Another statement from the project report concerned the impact of the project on subject matter faculty participants as well as their basic subjects counterparts:

.

Not surprisingly, as subject matter faculty members got involved in the assessments, they began to take more responsibility for students' on-going development by reinforcing basic skills in the context of the subject matter.

.

Further, the assessment report repeatedly cited skepticism that a standardized test (or a test about the ability to practice effective basics skills) could assess skills in writing, speaking, and critical thinking. The researchers' primary rationale for designing and testing a performance-based assessment plan was that certain types of student learning, they believed, "could be best evaluated directly in the context of the students' performance of the target skills, rather than in a traditional test venue, where competence is measured indirectly by examining discrete skills." They wanted to evaluate students' abilities to use basic communication skills in the context of their subject matter disciplines. The researchers concluded their study with a restatement of their commitment to evaluating speaking and writing in the context of actual speaking and writing, rather than by means of multiple-choice tests about speaking and writing. While this assumption may seem self-evident to teachers of oral and written communication, it is by no means an institutionally accepted truth by those who must be concerned about the costs of such assessments, according to the report. Though likely to be more costly than multiple choice tests, performative assessments of students' communicative competencies yield something such tests cannot: direct and meaningful evidence of students' learning—evidence that can readily inform the curricular practices and institutional policies which shape the basic skills of our university graduates.

.

GE Assessment Summary

.

In large part, securing faculty participation in discussion of the GE program at SFSU is not a problem. The WASC team acknowledged this in 1992:

.

In contrast with other universities in which general education seems ever a stepchild, at San Francisco State the program has been the object of continuous debate, review, and creative development…. Hundreds of faculty members came to be involved in this most recent review.

.

Intense interest across campus in the quality of our General Education curriculum continues to this date, and intense debates on the appropriate ways to measure its quality continue to occur. The assessment process in which this campus has been engaged since 1992 has prompted even more debate and interest, and it has also spawned enormous creativity and innovation.

.

Assessment is always a challenge, and assessing student learning outcomes in a program as broad as General Education is especially difficult. The measures tested in the above assessment activities can focus only on limited elements of General Education. In these activities we are not assessing the full breadth and depth of the experience. We do not know the extent that students develop an appreciation for diversity, life long learning, and sensitivity to the multiple issues and individuals that they may encounter in their lives. We have not measured if they develop a sense of citizenship and civic responsibility as well as the ability to work collaboratively with others. We cannot measure the extent that they are inspired to care for others or are encouraged to create beautiful works of art, music, or science. The breadth of the General Education experience can stimulate all of the above, much of which is intense and inspirational yet also illusive and immeasurable.

.

CONCLUSION

.

San Francisco State University has accomplished a great deal since the last campus-wide accreditation cycle. The faculty and administration have wrestled with critically important questions concerning performance and assessment and documentation and uses of the data. There have been large town meetings in the campus theatre, and there have been electronic listserves and small subgroup task force teams. We have attended presentations by visiting assessment experts, and we have drawn on our own deeply felt commitments to finding ways to develop and select the measurement tools that can be of value to the institution, the programs, and the students and faculty, without sacrificing too much of what is held to be the purview of individual faculty to determine their own evaluation measures. As we have seen, in many instances it was the process that was as valuable as the product. We have identified many strengths and some weaknesses in the outcomes, and the faculty will continue to compare syllabi and discuss learning objectives after this report is set on the shelves. We will continue to track students through their remediation steps and through their acquisition of basic information competence skills. We will continue to see lower division basic skills faculty consulting with faculty of capstone courses about the latter’s expectations for final course projects. We will continue to debate the adequacy, completeness, and currency of the GE program. Assessment requires on-going dialogue and self-examination. This is completely in line with our campus mission and tradition.

Return to Part Two: University-Wide Assessment Endeavors | Return to Accreditation | SFSU Home | Top of page