1. Questions Raised at the April 13 Forum and March 23 FC Meeting
  2. Background on the Achievement Index
  3. Proposed Policy
  4. AI and Grading practices.
  5. Honors
  6. Grading in Different Disciplines
  7. Additional Questions

Is the proposed use of AI transparent?

Transparency exists on many levels. The proposal charges the implementation committee with providing basic explanations of AI (e.g., it measures student academic performance independently of course grading practices, it is based on a student's performance relative to other students in the class taking into account how those students' overall record of academic achievement) and of justification for its use (e.g., it is a better indicator of relative student performance than is GPA). In addition, the implementation committee will be charged with developing materials that show students how their AI was influenced by specific courses. While care will be needed in crafting these explanations, there is little doubt that it can be explained successfully.

It is not expected that most people have the background in statistics to understand exactly how AI is calculated. Here, AI has a different type of transparency. The algorithm has been published in a respected peer-reviewed journal and is available for scrutiny by disinterested experts. This type of transparency is important in a complex society where individuals and institutions frequently rely on methods established by experts.

While students cannot calculate their own AI scores to check for errors, we believe this to be a minor concern. First, the source of errors is almost certain to be in the data provided to the algorithm, not in the algorithm itself, which is available for public scrutiny. Second, it is always impossible for an individual to calculate her own position in a ranked system. A student cannot now calculate her class rank, for example.

What about competitiveness between students?

Collegiality and cooperation among students is very important. The committee does not believe that AI is likely to undermine that spirit on campus because the proposed system does not greatly change the mix of incentives for cooperation and competition that exist currently. There are courses on campus now that grade on relatively strict curves but there have not been complaints of competitiveness about those classes. In a hard curve, when one student moves from A- to A another student must move down. With the AI when one student moves from A- to A there is a very minor dilution of the achievement associated with an A in that class. If current classes with relatively hard curves are not generating excessive competition between students there is little reason to believe that AI would cause it. At present students form study groups (which faculty heartily endorse) because it helps all students in the group learn the material and because it makes studying more enjoyable. Those considerations will persist with AI. Some students also spend time tutoring other students in situations where the tutor does not need to study the material further. There is no reason why AI should affect such altruistic behavior. Furthermore, if synergies exist that make successful cooperative learning in one class a benefit to students in their other classes, cooperation among students would be enhanced by the AI.

At a practical level, an item assessing competitiveness could be added to course evaluations before use of AI begins. This would provide a way of tracking any changes in competitiveness due to use of AI. If increased competitiveness proves problematic, then changes could be made in administration of the policy, or in the policy itself.

What is the problem being addressed?

The University uses Grade Point Average (GPA) as a summary measure of student academic accomplishment but GPA is flawed because substantial disparities in grading practices across courses means that GPA reflects course selection as well as student accomplishment. The inaccuracies of GPA lead to inequities when GPA is used for purposes such as awarding University Distinction, screening job candidates, and maintaining scholarship eligibility. The existence of such inequities creates grade-based incentives for course selection. It also means faculty face undue upward pressure on grades based on student demands for high GPAs.

Can the problem be solved by better communication?

Past experience indicates that it is unlikely that problems with grading can be changed by better communication among faculty. There was substantial discussion about grading at Carolina following the EPC report on grade inflation in February 2000. Four years later, EPC found found no change in disparities in grading following the 2000 report. It is essential that there be communication about grading, but communication will always be limited in an academic setting where faculty have very different academic training and where they typically have contact with only a small number of faculty outside their own department.

What are the basic policy options for addressing grading disparities?

There are two basic policy options for addressing grading disparities: mandatory target grade distributions (curves) for departments or classes and statistical adjustment.

Why not curves?

The 2000 EPC report on grading advocated consideration of common target grade distributions for all departments, but Faculty Council did not act on that recommendation. Curves are problematic because the meaning of grades varies across academic disciplines and because administrative mandates on grade distributions undermine both the faculty's autonomy in and responsibility for grading.

Why statistical adjustment and why AI?

Statistical adjustment of grading disparities has the philosophical and practical advantages of leaving responsibility for grading in the hands of the faculty. Many advantages of the AI over other statistical adjustment methods are summarized in other documents. The principal advantage is that the AI does not penalize a strong student who does well in a course where grades are typically high. Thus, the AI does not replace the current incentive to avoid classes where grades are typically low with an incentive to avoid classes where grades are high.

When would AI be implemented if passed?

The proposal contains no specific timetable. However, given the work that would go into design and implementation of the system, it is very unlikely that it could be implemented before the entering class of Fall, 2009.

What is the Achievement Index?

The Achievement Index (or AI), like grade-point average (or GPA), is a measure of students' performance that is calculated from the grades that students receive in their classes. Calculation of AI differs from calculation of GPA in that it takes into account differing grading practices in different classes.

What is the goal of the Achievement Index?

The goal of the AI is to measure the academic performance of students independently of the grading practices employed in the particular classes taken by different students.

Where did the AI come from?

The AI was created by a statistician, Valen Johnson, who at the time was on the faculty at Duke University. It builds on statistical methods that were developed for use in standardized tests but it is appropriate for college grading because it does not require standardization of course content or of methods for evaluating student achievement.

Do other universities use the AI?

No. Carolina would be the first university to use the AI. However, the goal of providing information about grading practices across courses has been addressed by other universities (notably Dartmouth College and Indiana University) by listing the median grade in a class on transcripts next to the student's grade.

What advantages are there in using the AI as compared to listing median class grades?

The AI allows a more direct comparison with GPA than does median class grade. For purposes of class rank and awarding distinction, where we need to compare performance across different disciplines, we need to be able to compare students' performance in numerous classes, not just in any single class.

Where is information about the AI available?

Brief descriptions of the AI and how it is calculated can be found in the report on Adopting the Achievement Index and in the Primer on the AI. Those summaries provide references to the primary sources.

How will AI be used?

AI will be listed on students' transcripts. In addition, the University will use it for those purposes that involve comparing the academic performance of different students, such as providing information about students' class rank and awarding of Distinction and Highest Distinction upon graduation.

Why focus on University Distinction?

The University awards undergraduate degrees with Distinction or Highest Distinction to those students who have excelled in their coursework. At present, these awards are based solely on GPA; students with a GPA of at least 3.5 but under 3.8 receive Distinction and those with a GPA of 3.8 or higher receive Highest Distinction. As a University-wide form of recognition, Distinction requires comparing the grades of students who have pursued very different courses of study. AI provides a way of comparing the performance of students independently of the grading practices in the courses that they have taken. Therefore, it is a more accurate method than GPA for identifying those students who have excelled academically.

Will AI be used for purposes other than awarding University Distinction?

The proposed legislation establishes an implementation committee whose charge would include determining whether other uses of AI are warranted. Possible other uses include: admission to undergraduate professional programs, admission to first/second year Honors for students who are already enrolled and admissions to Major Honors programs. Decisions on whether or how it might be appropriate to use AI for such purposes require discussion with the faculty, students and administrators involved in those programs.

Will AI replace GPA?

No, students' GPAs and the grades earned in their classes will be listed on their transcripts as in the past.

Will AI be used in setting standards for continuing academic eligibility and graduation?

No, the proposal recommends no change to standards for continuing academic eligibility or graduation. Over the last two years EPC and Faculty Council have made a number of important changes to standards for continuing eligibility with the goal of increasing graduation rates. No change in those new standards is proposed.

Why does it make sense to use AI for University Distinction but not for eligibility and graduation?

AI is useful in comparing the performance of different students, not in setting absolute standards. Criteria for eligibility and graduation should be based on absolute standards not comparative ones. Carolina admits only those students who it believes have the potential to master the standards for graduation; achieving a high graduation rate is a legitimate and important goal for the University. In contrast, awarding Distinction involves comparing the performance of different students and recognizing those who have done the best in their academic work at Carolina. As a comparative judgment, the award of Distinction should be based on the most accurate comparative measure available.

Will AI cause changes in grading that would indirectly affect eligibility or graduation?

The question of whether use of AI would have any effect on faculty grading practices is addressed in Section 3 on AI and Grading Practices. Even if the proposed use of AI prompted faculty to assign somewhat lower grades, such a change would be very unlikely to affect eligibility or graduation rates. There are 7 grades (C through A) which fully satisfy eligibility and graduation standards. At present, instructors in many classes distinguish student performance using only a small portion of this range.

How will people learn about AI?

The implementation committee will be charged with developing ways of describing AI to students and faculty at the University as well as to those outside of the University. In addition, the committee will develop methods for providing information to students about their academic progress in terms of AI and about how particular classes might influence their AI.

Does adoption of AI dictate how faculty should grade?

AI provides a statistical context for interpreting faculty grades when comparing the academic performance of different students, it does not provide any guidelines about how faculty should grade.

Does implementing the AI privilege courses that grade "on a curve"?

No. The impact of a grade in a class on a student's AI does depend in part on the extend to which the grades in a class differentiate levels of student performance. Strictly speaking, grading on a curve means that the proportions of students receiving certain grades is specified in advance for the course, a method that ensures differentiation of levels of student performance. However, differentiation of levels of student performance can be achieved by specifying standards that must be achieved for certain grades, as long as those standards lead to differentiation of student performance.

How will AI affect professors' grading practices?

There is no way to determine with certainty whether professors' grading practices would change or how they might change. We assume that, currently, professors grade students based on their performance in class and their mastery of the course material. To the extent that professors are pressured by students to award high grades for other reasons, the introduction of the AI may tend to discourage that behavior.

How can a student strategically plan to raise his/her AI ?

Do well in classes in which other strong students are enrolled, and in which instructors issue a wide range of grades.

What is the difference between Distinction and Honors at Carolina?

University Distinction is awarded at graduation based solely on GPA; it does not involve any particular type of academic work or program of study. At Carolina, students can pursue two types of honors; the Honors Program centers on coursework while Honors in a major field of study requires successful completion of a Senior Honors Thesis. A student who successfully completes a Senior Honors Thesis graduates with Honors or Highest Honors in a specific discipline. The nature of the work in a Senior Honors Thesis is shaped by the academic discipline in which the student is working and the thesis is evaluated using the standards of that discipline by a faculty committee. In this way, Honors in a major field of study is a much more focused form of recognition than University Distinction.

Will AI be used in selecting students for Honors programs?

Some students are accepted into the Honors Program upon admission to Carolina; obviously AI could not be used in selecting those students. Other students apply to the Program in the second semester of their first year or the first semester of their second year. As discussed above, the AI implementation committee would evaluate whether AI should should play a role in decisions about admissions.

Currently, a student must meet a minimum GPA standard to be eligible to undertake a Senior Honors Thesis. Once again, consideration of whether an AI standard should also be used would be left to the implementation committee.

How will use of the AI affect Honors students?

The academic records of Honors students over the last twelve years are slightly stronger when evaluated by AI rather than raw GPA. For students in the first- and second-year honors program, the average raw GPA is 3.59, while the average achievement-adjusted GPA is 3.65; for 75% of these students, achievement-adjusted GPA is higher than raw GPA. For students who earn Departmental Honors, the average raw GPA is 3.70 while the average achievement-adjusted GPA is 3.73; for 68.4% of these students, achievement-adjusted GPA is higher than raw GPA. See AI and Honors.

How will use of the AI affect Honors classes?

Carolina offers Honors sections of departmental course offerings. Students in the Honors Program (and some other students) enroll in these sections to fulfill general education and major field requirements. Because these sections enroll students who have chosen a rigorous course of study and have met high selection standards, the grades in honors sections tend to be higher than in non-honors sections. For example, the average percentage of As awarded in Honors sections was 42.3 while it was 24.6 in the comparable non-honors sections. The impact of grades in a section on AI depends on two factors: the extent to which the instructor's grades differentiate levels of student performance and how well the students in the class have done in their other classes. For Honors sections on average, the AI calculation for the grade of A places students above the 82.7th percentile of the student body as a whole while for non-honors section an A places students above the 75.8th percentile. Thus, the AI calculation awards slightly more credit for an A in an Honors section than for an A in a non-honors section, even though a higher percentage of students receive As in Honors sections. See AI and Honors.

Do grades vary across academic disciplines?

Yes. Analyses of grading patterns at Carolina and elsewhere show systematic differences in grading across disciplines (see the EPC reports in 2001 and 2004. Additional analysis by the grading subcommittee suggests that over 15% of the variation in students' grades can be explained by the departments they studied in and the instructors who taught their classes.

How will AI affect different academic majors?

As discussed in the reports referenced above, there are substantial differences in grading practices across disciplines. Those differences are incorporated into GPA as measure of student performance but are not incorporated into AI. The impact on different majors of using AI to award University Distinction is shown in the tables that are linked here.

Do disciplinary differences account for all variation in grading practices?

No. There is often substantial variation in the grades assigned by different instructors within a department even when they teach the same course.

Does implementing the AI imply that some departments are better, or more rigorous, than others?

No. Professors are the experts in their fields, including in the standards and expectations for grades and the pedagogical goals they pursue. The AI does not interfere with that expertise. But the university, in comparing students' performance across department and instructor, should take into account these variations in grading practices. The AI is a way of allowing the university to make those comparisons in a fair way without interfering with instructors' autonomy.

What is the intended consequence of adopting AI?

The intended consequence is to make the comparison of student performance across the university fair by reducing or eliminating the effect of departments' and instructors' grading practices.

Will there be unintended consequences of adopting AI?

Of course it is possible that adopting AI will have unintended consequences. The proposal charges the implementation committee with developing procedures for tracking the consequences of adopting AI. In that regard, it is important to note that grades in classes and GPAs will still be presented as in the past. The continued presence of these familiar measures is likely to moderate any unintended consequences of adopting AI. Some possible, though unintended, consequences of AI are discussed below.

How will AI affect the competitive climate of the university?

There is no way to determine with certainty how the competitive climate would change following the introduction of the AI. In general, students who tend to be competitive academically will likely continue to do so. After implementation of the AI, students in any given section would have incentives both to do well themselves and to have high-performing classmates.

How will AI affect student selection of courses and majors?

There is no way to determine with certainty how students' selection of courses and majors would change. We assume that, currently, students select courses and majors either to fulfill requirements or to pursue intellectual and career goals; the introduction of the AI would support those reasons for course and major selection. If students are currently selecting courses and majors with reputations for "easy" grading in the hope of improving their GPAs, the introduction of the AI would tend to discourage that behavior.

How will AI affect students in graduate/professional school admissions and the job market?

There is no way to determine with certainty how students' success in graduate and professional school admissions or in the job market would change. We believe that introducing the AI will help reinforce UNC's reputation for academic excellence, which would in turn help UNC graduates in their academic and professional careers. Since prospective schools and employers will continue to have access to raw GPA as well AI, they may choose to use either or both measures to evaluate prospective candidates.