John Jay College of Criminal Justice The City University of New York. Report from the Committee on General Education Assessment. September 11, PDF

Please download to get full document.

View again

of 22
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Information Report
Category:

Self Improvement

Published:

Views: 7 | Pages: 22

Extension: PDF | Download: 0

Share
Related documents
Description
John Jay College of Criminal Justice The City University of New York Report from the Committee on General Education Assessment September 11, 2012 Table of Contents Introduction: General Education Competence
Transcript
John Jay College of Criminal Justice The City University of New York Report from the Committee on General Education Assessment September 11, 2012 Table of Contents Introduction: General Education Competence among John Jay Competency 1: Written Communication Competency 2: Oral Communication Competency 3: Scientific Reasoning Competency 4: Critical Thinking Competency 5: Information Literacy and Technological Competence Conclusions and Recommendations Appendix A: AAC&U VALUE Rubrics Appendix B: Sources of Data for Quasi-Direct Assessments Appendix C: Chart: Alignment of John Jay Mission Goals, Learning Outcomes for Undergraduate Education, Pathways Learning Outcomes, Middle States Competencies, AAC&U Value Rubrics, and the John Jay College Gen Ed Option Prepared and submitted to Dean of Undergraduate Studies Anne Lopes by the General Education Assessment Committee: Professor Amy S. Green, chair Professor Andrea Balis Professor Ellen Sexton Professor Andrew Sidman Virginia Moreno, Director of Assessment 1 John Jay College of Criminal Justice The City University of New York Report from the Committee on General Education Assessment This report assesses outcomes for John Jay s existing (i.e. outgoing) general education curriculum. A new, outcomes-based general education curriculum was developed and approved for implementation in May 2011, but that process was interrupted when CUNY announced a new, University-wide general education program. Pathways will to be implemented in fall The Pathways learning outcomes will form the basis of general education assessment at John Jay moving forward. In the meantime, assessment of general education at John Jay has mostly been conducted at the course level for such required classes as English composition, history, speech, math, science, and philosophy. This is the first comprehensive study of student learning in general education. Members of the General Education Assessment Committee that prepared this report included Professors Amy Green (Communications and Theater Arts and Interdisciplinary Studies Program, Committee Chair), Andrea Balis (History and ISP), Professor Andrew Sidman (Political Science), and Ellen Sexton (Library) and John Jay s Director of Assessment, Virginia Moreno. Because there had been no systematic study of gen ed outcomes at the college before, the Committee decided to pilot a comprehensive study that would accomplish two goals. First, it would tell us about student achievement in general education under our existing program, which could be used a baseline for future assessments. Second, the pilot would enable us to make concrete recommendations for both general education curriculum and pedagogy and for ways to design a long-term comprehensive plan for gen ed assessment under the incoming general education program. We believe we have achieved both goals. The report takes a broad approach to assess general education abilities across the curriculum by focusing on students outcomes at the capstone level. We recognize the limitations of this approach. Assessing the impact of general education on seniors and recent graduates means that our sample includes only the most successful students, those who stick with it long enough to take their capstone courses. This approach also fails to distinguish between achievements of native John Jay seniors, who have been here throughout their college careers, with transfer students, who have may have taken some or all of their general education requirements elsewhere. The pilot study assessed student learning at the capstone level in six of the seven Middle Statesrecommended areas of general education competence: 1) written communication, 2) oral communication, 3) scientific reasoning, 4) critical analysis and reasoning, 5) technological competence, and 6) information literacy. Quantitative reasoning was assessed through the learning outcomes in mathematic courses (College Algebra to Calculus). Wherever possible, competency areas have been assessed using relevant AAC&U VALUE rubrics because they provide a reliable, national standard against which to measure our students learning. Some of the competencies were assessed together using modified rubrics that combine criteria from more 2 than one of the original VALUE rubrics. The following competencies were assessed using these original or modified VALUE rubrics: Middle States Competency Written communication Oral communication Scientific and quantitative reasoning Critical analysis and reasoning Technological competence and information literacy AAC&U VALUE Rubric Written Communication Oral Communication Inquiry and Analysis Problem Solving Critical Thinking Information Literacy (there is no VALUE rubric for technological competence) Descriptions of the criteria assessed within the instruments are included in separate sections on each competency. To get a well-rounded snapshot of general education outcomes at the capstone level, we conducted three types of assessment: Indirect, based on extant data from institutional surveys, including the National Survey of Student Engagement (2010), the Faculty Survey of Student Engagement ( ), the 2011 CUNY Student Experience Survey, and the 2011 John Jay survey of graduates; quasi-direct, which drew data from capstone assessment reports from eleven majors; and direct, in which the members of the Committee applied the VALUE rubrics to a random sample of capstone papers (#30) from 13 majors. Because the data sets were collected independently, the amount of data available for each item and the language, scales of achievement, common definitions and values vary widely, especially among the capstone assessment reports that were used for quasi-direct assessment. (Appendix B includes tables that indicate the sources of data for each rubric criterion.) While all the data was scored by ordinal scales, Exceeding Expectations for a given undergraduate program does not necessarily mean students are performing at the Capstone level according to the criteria established by the AAC&U. We therefore counsel caution in comparing the three sets of results. In the most general terms, John Jay students are functioning at well below national norms across all five areas of competence studied, although they consistently and generously overestimate their proficiency when surveyed. Capstone papers scored at Milestone 1 on five out of five criteria in written communication; for scientific reasoning, they scored highest at Milestone 2 in one area out of seven, Milestone 1 in four areas, and at Benchmark in the remaining two. Out of five criteria for information literacy, the largest number of capstone papers scored at Milestone 1 for three, Benchmark for 1, and tied between Milestone 2 and Capstone for one (access and use information ethically and legally). Not surprisingly, the majority of capstone students are at least moderately proficient in the basic academic skills: they can select a topic, identify and access sources, and manage the mechanics of writing. Their performance is weaker on more challenging tasks such as analyzing data, drawing conclusions, and putting their subject, ideas, and writing in context. Three of our five competencies - oral communication, critical thinking, and information literacy and technological 3 competence - emerged through our study as curricular and/or assessment stepchildren, with lots of lip service but little actual attention paid in the majority of capstone learning outcomes and assessments. Happily, selected majors seem to be producing highly competent students. Papers from capstone courses in the Political Science, Humanities and Justice Studies, English, Global History, and International Criminal Justice majors scored higher than others. One thing we learned by studying multiple sources is that there appears at this point to be little consensus among the John Jay faculty on which general education outcomes are most important and what expectations for them should be. We recommend developing universal rubrics for general education assessment (perhaps based on the VALUE models) with common criteria, standards, scales and language to be used in every undergraduate department and program alongside the discipline-specific instruments appropriate to their majors and programs. What follows are the results of indirect, quasi-direct, and direct assessment of five broad areas of general education competence among John Jay College seniors and recent graduates. 4 Competency 1: Written Communication Writing was assessed at the capstone level using indirect, quasi-direct, and direct measures of competence according to the five criteria on the VALUE Rubric for Written Communication. Under Context of and Purpose for Writing, papers were assessed for how well they demonstrated awareness of the surrounding circumstances, target audience, and intended effect on the audience. The Content Development criteria evaluated the papers on the extent to which they used compelling and relevant content to support significant exploration of the topic and ideas. Genre and Disciplinary Conventions looked at how the papers did or did not observe formal and informal rules inherent in the expectations for writing in particular forms and/or academic fields, while Sources and Evidence evaluated the appropriateness, reliability, and adequacy of source materials to support the ideas and intent of the paper. Grammar, language, and clarity of ideas were examined under Control of Syntax and Mechanics. Other criteria typical of writing assessment, such as thesis and argumentation, are assessed in the VALUE Rubrics for Critical Thinking and Scientific Reasoning. 1.a Indirect assessment of written communication Indirect assessment of writing focused on student and faculty perceptions of the improvement in writing ability as a result of studying at John Jay. Overall, both students and faculty report significant improvement in writing, with scores ranging between 47 and 86%, with gaps between student and faculty reports. 86% of 2011 graduates report significant improvement in their ability to write clearly and effectively as a result of their studies, with 34% saying they have improved somewhat and 52% saying they have improved a lot. 66.5% of the seniors credit English 101as a contributor to that improvement. However, consistent with the pattern Arum and Roska document among underprepared college students in Academically Adrift: Limited Learning on College Campuses 1, John Jay students evaluate their skills more highly faculty do. Faculty who teach seniors perceive that 59% have improved quite a bit or very much, a full 27 points lower than the seniors self evaluation. A possible indicator of the difference between student and faculty perceptions can be found when we look more closely at Sources and Evidence and Content Development. An overwhelming majority of seniors, (88%) report having integrated information from multiple sources in a paper, but only 43% claim to have prepared multiple drafts before submitting their work. The faculty assessment of revision is even lower, at 33%. 1 Arum, Richard and Joseph Roska. Academically Adrift. University of Chicago Press (2011). 5 Indirect Assessment of Writing Table 1: Comparison of Student and Faculty Perceptions Content Development Prepared 2 or more drafts of a paper or assignment before turning it in (NSSE Table 2; FSSE Table A4) 1 st Year Sources and Evidence Worked on a paper or project that required integrating ideas or information from various sources (NSSE Table 2; FSSE Table A4) 1 st Year Often Never 24% 36% 33% 7% 18% 25% 40% 7% Often Often Never 44% 45% 9% 2% 48% 40% 10% 2% Often Never 10% 20% 34% 36% 14% 19% 48% 18% Often Often Students (NSSE) Often Sometimes Faculty (FSSE) Often Sometimes Sometimes Sometimes Never 24% 34% 22% 20% 39% 38% 14% 9% Control of Syntax and Mechanics Writing clearly and effectively (NSSE Table 9; FSSE Table A12) 1 st Year much 44% 42% Quite a bit 37% 39% Some 16% 16% little 3% 3% much 15% 19% Quite a bit 32% 40% Some 47% 36% Little 6% 5% Indirect Assessment of Writing Table 2: Items from 2011 CUNY Student Experience Survey and 2011 John Jay Survey of Graduates Survey item How much has taking the following courses improved your ability to speak and /or write clearly and effectively? (CUNY Student Experience Survey Table 5) English 101 or SEEK English English 201 (used to be English 102) How much did your ability to write clearly and effectively improve as a result of your studies at John Jay College? (John Jay Alumni Survey, Table 6) Student Response Improved Improved a lot somewhat Improved a little Did not improve % 34% 9% 4% 1.b. Quasi-direct assessment of written communication Quasi-direct data on writing in capstone and 400-level courses gives a more nuanced perspective, although the overall assessment of writing is strong. This data set is least consistent because of the wide variety in the ways that different majors do and do not include or assess writing in their learning outcomes. Table 3 shows that the largest number of capstone papers were assessed for Content Development (238 papers), with Control of Syntax and Mechanics seeing the next largest number (122), while far fewer attended to the other component skills. According to capstone reports, in three of the five writing criteria, the largest percentage of students meet 6 expectations (Content Development 45%, Sources and Evidence 50.5%; and Control of Syntax and Mechanics 56%). Only in Context of and Purpose for Writing and Genre and Disciplinary Conventions, perhaps the most sophisticated criteria and the two least frequently assessed in capstone reports, do the largest percentage of students fall under Approaches Expectations (75% and 62% respectively). Control of Syntax and Mechanics is the writing skill most highly rated in our quasi-direct assessment, with 85% meeting (56%) or exceeding expectations (29%). Next in line are Sources and Evidence, with 69.5% meeting (50.5%) or exceeding (19%) expectations, and Content Development, in which 78.5% score at the two highest levels, with 45.5% meeting and 33% exceeding expectations. VALUE Item Table 3: Quasi-Direct Assessment of Writing from Major Capstone Assessment Reports Number of Papers Exceeds Expectations Meets Expectations Approaches Expectations Fails to meet Expectations Context of and Purpose for Writing % 75.00% 8.00% Content Development % 45.50% 15.00% 5.50% Genre and Disciplinary Conventions % 31.00% 62.00% 6.75% Sources and Evidence % 50.50% 19.50% 8.00% Control of Syntax and Mechanics % 56.00% 12.50% 2.50% Note: Cells contain the percentage of all capstone papers assessed by the majors falling under each performance category. The first substantive column lists the number of capstone papers included in the meta-analysis for that particular VALUE item. As a cautionary note, papers were independently assessed by each major whose papers were included. There is, therefore, no guarantee that the criteria used to place each paper in a particular performance category are consistent across majors. 1.c. Direct assessment of written communication The results of the Committee s direct assessment of written communication, which is based on the national standards articulated for the five component skills on the VALUE Rubric for Written Communication, indicates a less robust level of achievement than in the indirect and quasi-direct studies. In all but one skill, Sources and Evidence, the largest percentage of capstone papers scored at Milestone 1, roughly equivalent to Approaches Expectations in the capstone reports, although, unlike the quasi-direct data, in which two categories had 0% in the top rank, our assessment found at least 13.33% or more placed in the Capstone level for every skill. Control of Syntax and Mechanics had its largest group, 43.33% in Milestone 2. 7 Overall, the direct assessment of written communication at the capstone level showed a wide range of student performance, with between 36.66% (in Genre and Disciplinary Conventions) and 59.99% (in Control of Syntax and Mechanics) scoring relatively highly in Milestone 2 and Capstone, despite the fact that most skills found the single largest percentage of students performing at only Milestone 1. The data suggest that achievement in writing is inconsistent among John Jay seniors, and scores for capstone papers in such liberal arts majors as Gender Studies, English, and Humanities and Justice Studies tended to be higher than for papers written in other majors. Although it is possible that the liberal arts majors attract students with better writing skills, it would be worth looking at whether and how these majors emphasize writing skills in their curricula and whether other majors can emulate the ways that they advance the writing skills introduced in general education. Table 4: Direct Assessment of Capstone Papers using VALUE Rubric for Written Communication VALUE Item Capstone Milestone 2 Milestone 1 Benchmark Below Benchmark Context of and Purpose 23.33% 16.66% 40.00% 20.00% 0.00% for Writing Content Development 23.33% 33.33% 36.66% 6.66% 000% Genre and Disciplinary Conventions 13.33% 23.33% 36.66% 10.00% 16.66% Sources and Evidence 20.00% 23.33% 43.33% 10.00% 3.33% Control of Syntax and Mechanics 16.66% 43.33% 33.33% 6.66% 0.00% 1.d. Conclusions and Recommendations Despite a strong, shared belief among students and faculty that writing experiences at the College, especially English 101 and 201, are having a positive impact on skills, John Jay seniors are writing below national standards in every component criteria on the AAC&U VALUE Rubric for Written Communication. In four out of five categories, our students score at Milestone 1, two levels below capstone. Their greatest strength on this scale is in the mechanics of writing, where 60% of papers scored ranked at Milestone 2 or Capstone. When faculty rated the mechanics of writing using departmental scales and criteria in capstone assessment reports, a whopping 85% were judged to meet or exceed expectations. If the most basic elements of writing competence, grammar and mechanics, are areas of relative strength for John Jay capstone students, their areas of greatest weakness are in establishing the Context and Purpose for Writing and using Genre and Disciplinary Conventions (although large numbers of student products were ranked low in these areas because they were not required by the capstone assignments). Student accomplishments in Content Development and using Sources and Evidence cluster at Milestone 1 and above on the national scale, and meets expectations and 8 above on local measures. Clearly, we face the challenge of whether and how to align local expectations with national norms. We have two recommendations for improving and monitoring the development of student writing: 1. We recommend that a multidisciplinary team, including faculty from the English Department writing program, be charged with reconciling the bases on which we teach and assess writing competence across the curriculum. It will be interesting to see what results look like when we evaluate comparable sets of skills using the same criteria. 2. We recommend that a more deliberate hand-off of responsibility for writing development be articulated between the two required English composition courses and other areas of the curriculum, especially in upper-division courses that prepare students for capstone work in the major. 9 Competency 2: Oral Communication One of the most striking discoveries of our pilot inquiry is the disjuncture between the emphasis placed on oral communication in the earliest stage of the general education curriculum, the limited data on student learning in the course, the absence or vagueness of references to oral communication skills in the majors and their capstone courses, and the absence of data at any level in relation to the criteria on the AAC&U VALUE Rubric for Oral
Recommended
View more...
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks