Freshman Academy Evaluation- 1999-2000
David D. Williams
Updated June 2001
Table of Contents
Executive Summary
Why an evaluation?
How was the evaluation conducted?
What was learned?
Who Cares?
About What?
What Are Their Expectations?
How Well Are Expectations Being Met?
Interim Report
Findings
Case Studies
Student Focus Groups
Faculty Focus Groups
Student Journals
Community Time Reports
Staff Interviews
Peer Advisor Focus Group
Historical Study
Previous Evaluations
Conclusions and Recommendations
Conclusions
Recommendations
Full Report
Why an evaluation?
How was the evaluation conducted?
What was learned?
Who cares?
What Do Stakeholders Care About?
What Do Stakeholders Expect of Freshman
Academy?
Common expectations
Most Important
Least Important
Expectation differences
Students' unique
expectations:
Parents' unique
expectations:
Academic administrators'
unique expectations:
Support administrators'
unique expectations:
How well is Freshman Academy meeting expectations?
Interim Report
Retention
Academics
Attitudes
Conclusions in
Interim Report
Case Study Results
Quantitative Results
Qualitative Results
Summary
Student Focus Group Results
Academics
Social Aspects
Community Time
Teachers
Small Class Size
Spiritual Aspects
Summary
Faculty Focus Group Results
Social Aspects
Academic Aspects
Teaching Aspects
Suggestions for
Improving Freshman Academy
Summary
Student Journal Results
Summary
Analysis of Community Time
Reports
Staff Interviews
Peer Advisor Focus Group Results
Historical Study
Studies Conducted Since 1993
Conclusions and Recommendations
Conclusions
Recommendations
References
APPENDICES
Appendix A. Data
Collection Methods
Appendix B. Expectations
Survey
Appendix C. Expectations
Survey Results
Appendix D. Interim
Report
Support
Document 1-Original proposal
Support
Document 2-Data sources for entire study
Support
Document 3-GPA data
Support
Document 4-Copy of the end-of-semester survey
Support
Document 5-Summary of the end-of-semester survey data for all three groups,
presented in Appendix E
Appendix E. Fall
1999 Comparison Data
Appendix F. Case
Study Summaries
Appendix G. Case
Study Statistics
Appendix H. Student
Focus Groups
Appendix I. Faculty
Focus Groups
Appendix J. Student
Journal Summary
Appendix K. Student
Journal Entries
Appendix L. Community
Time Analyses
Appendix M. Staff
Interviews
Appendix N. Peer
Advisor Focus Group Transcript
Appendix O. Gary
Daynes' Historical Study
Appendix P. Pre-1999
Evaluation Efforts
At the request of the Dean of General Education and Honors and the administrators of Freshman Academy at Brigham Young University (BYU), an evaluation of the Academy was conducted during the 1999-2000 school year. This summary of the study clarifies why and how the evaluation was conducted, what was learned, and recommendations the faculty, administrators, and others might consider for improving the program or developing similar programs elsewhere.
Freshman Academy has undergone continual evaluation since it began as the Student Housing Initiative in Education (SHINE) program in 1993. Once again in 1999-2000, key administrators, both within and without Freshman Academy felt it was time to formally evaluate the program to assess how well it was meeting its purposes and to seek ways to improve it.
How was the evaluation conducted?
In accord with several evaluation principles, the study began by identifying key stakeholders, ascertaining their definitions of and criteria for judging Freshman Academy, and then gathering descriptions of the program from several perspectives to compare to the criteria, make evaluative conclusions and generate recommendations.
Key findings are presented briefly below. Details and supporting evidence are presented in the full report, beginning on page 11.
The stakeholders, or those who care about the evaluand (the object of the evaluation), include students, faculty, parents, academic administrators, support administrators, the Board of Trustees, similar audiences at other universities, and scholars interested in higher education.
The main object of interest is Freshman Academy-- a complicated entity, created and maintained by many parts of the university. Freshman Academy includes these elements:
The stakeholders do not define the evaluand or the criteria for judging it in identical ways. They make different assumptions that complicate efforts to guide and improve the Academy as well as to hold it accountable.
Although the stakeholders do agree on some goals, particularly that the Academy should help ease students' transition from high school to the university setting, this and other goals are interpreted in a variety of ways.
How Well Are Expectations Being Met?
Results from several data sources address a variety of expectations held by many of the stakeholders, although much work remains to be done to clarify and prioritize the expectations Freshman Academy should address.
In March 2000, the Associate Academic Vice-president's office asked for answers to three questions. The request assumed comparisons between Freshman Academy students and non-Academy students would be a reasonable means of measuring the program's success on three objectives which received mixed prioritization by participants in the surveys completed in August 1999. Answers to these three questions were prepared in an "interim report" which was presented orally and in writing and are summarized below.
Case studies of students from 17 of 19 learning communities were conducted by peer observers. They made 63 observations by attending Academy classes and community time sessions with one randomly selected student from each community and interviewing the students. Based on these observations, they estimated how well Freshman Academy helped ease the students' transition from high school to university, enhanced their academic and spiritual growth, and provided social support for each student.
In 13 focus group sessions, 73 students made many comments addressing the role of Freshman Academy in their academic, spiritual, and social growth, the value of small class sizes associated with the Academy, the quality of the teachers, and community time.
Focus groups were held with 33 faculty in October and December 1999 to invite them to comment on their experiences with Freshman Academy, clarify their criteria for judging it and express their evaluations of it.
Names of 190 Freshman Academy students (10 from each of the 19 learning communities) were drawn randomly. The students were invited to answer a weekly questionnaire (journal) posted on the internet throughout the Fall 1999 semester. The questionnaire asked students to report on their own participation in Freshman Academy classes, community time activities, and other events. They were also asked to rate the impact of these experiences on their transition from high school to university life, their academic and spiritual growth, and the social support and encouragement they felt from participation. Finally, they were asked for comments on the nature and impact of their experiences.
A key element of the Freshman Academy program is participation in learning communities-- three different courses for the participating students, upper classmen Peer Advisors who help organize out-of-class activities, housing arrangements that put students near one another, and a weekly community time during which many of these extra-curricular activities are held. The Peer Advisors were required to fill out planning and evaluation sheets for every community time and to keep these on file with the Academy office. Analyses of these community time reports indicate a wide variety of experiences across the different communities and lessons learned about how to use community time effectively.
Interviews were conducted with Academy administrators throughout the study. One interview is presented as indicative of administrator and staff thinking and vision.
A focus group session with 14 Peer Advisors (upperclassmen employed to work with Freshman Academy communities) clarified the roles they played, benefits of the Academy they perceived, and concerns they felt need to be addressed.
Professor Gary Daynes and some of his History 389 students developed 25 oral histories about the origins of Freshman Academy and the perceptions of various participants regarding the program.
Since the Academy began as SHINE (Student Housing Initiative in Education) in 1993, regular internal evaluations (surveys, focus groups, interviews, and so on) have been a part of the program's development. The reports from these studies (available in the Freshman Academy office) were reviewed.
Conclusions and Recommendations
Based on results from all the different sources reported here, the following conclusions and recommendations are offered.
Freshman Academy Evaluation- 1999-2000
At the request of the Dean of General Education and Honors and the administrators of Freshman Academy at Brigham Young University (BYU), an evaluation of the Academy was conducted during the 1999-2000 school year. This report clarifies why and how the evaluation was conducted, what was learned, and recommendations the faculty, administrators, and others might consider for improving the program or developing similar programs elsewhere.
Efforts to provide freshmen with experiences to help them grow academically, personally, socially, and spiritually have been prominent at BYU since its inception. Over the last decade, one effort, originally known as SHINE (Student Housing Initiative in Education), later as Freshman Academy, has emerged which now enrolls nearly 40% of the entering freshmen at BYU.
Entering freshmen who are in the Academy register for three classes during their first semester (in addition to other non-Academy courses) taught by departmentally-selected faculty who collaborate with other facudty to provide a more personal and coordinated effort across courses. Students share their three Academy classes with about 20 other students (in the smallest sections, which combine with other sections to create the other two, larger classes) and often share the same housing units.
Students participate in study groups, service projects, meals, and social events with these same students and with faculty throughout the semester in an effort to combine academics with other aspects of college life as the literature recommends (e.g., Terenzini, Pascarella, & Blimling, 1996; Boyer, 1987 [College]).
Academy administrators have from the beginning (i.e., SHINE in 1993) collected data systematically on all aspects of the program. Recently, university administrators requested that a more formal evaluation be done, given the significance of Freshman Academy to the first year of studies at BYU. A faculty member specializing in program evaluation (David Williams, Instructional Psychology and Technology), who had not been involved with the Academy previously was asked to conduct a study, with the assistance of a number of others at the university (Kris Kristensen, Marilyn Nelson, Natalie Williams, Rebecca Cardon, Monte Shelley, Eric Rogers, Delys Cowles, Julienne Vance, Jennifer Ellet, Heidi Widmer, Mike Bohn, Brooke Robinson, Megan Boyle, Wendy Woodfield, Ryan Davis, BJ Lynes, Emily Snyder, Lynette Lake, The Faculty Center, Julene Bassett, Darcy Haycock, Sally Allen, Richard Ang, Sungup Moon, and everyone in the Freshman Academy office).
Pat Esplin and Clark Webb, administrators of Freshman Academy, took particular interest in building evaluation into their daily operations. They noted in a memo (see Support Document 1 in Appendix D) to the Dean of General Education that to properly evaluate a program such as Freshman Academy, two criteria were to be met:
First, there are three levels of questions about program adequacy or function, to be asked in a particular order:
Second, unless stakeholders' (or their representatives') values and expectations are taken into account, they will not credit the results of an assessment of the quality of the program.
How was the evaluation conducted?
Over the last several years, a number of new approaches to the theory and practice of educational evaluation have been produced and implemented. In virtually every case the changes have encouraged greater attention to the interests and values of the participants. Some of these participant-oriented approaches are responsive evaluation (Stake, 1984), democratic evaluation (House & Howe, 1999; Ryan & DeStefano, 2000), fourth generation evaluation (Guba & Lincoln, 1989), empowerment evaluation (Fetterman, 1996), utilization-focused evaluation (Patton, 1997), participatory evaluation (Cousins & Whitmore, 1998), collaborative evaluation (Cousins, Donohue, & Bloom, 1996) and an integrated framework (Mark, Henry, & Julnes, 2000).
Although these approaches to evaluation differ in many ways, they all emphasize the fact that evaluations are done for particular participants whose values vary. Thus on a criterion of fairness, the evaluations have to take discrepant value positions seriously and systematically. Additionally, attention should be paid to values if the participants are to have sufficient interest in using the evaluation results. Indeed, over time, evaluation has become increasingly attentive to the needs and interests of wider and more diverse groups of people associated with the things being evaluated.
Some elements of one participant-oriented approach fundamental to the Freshman Academy evaluation are summarized below. This approach takes a broad perspective on the nature of most kinds of evaluands (things being evaluated), ranging from organizations to instructional products and from their conception to their completion, as first proposed by Stufflebeam (1971) in his CIPP (context, input, process, product) approach as shown in Figure 1. Stufflebeam's CIPP Approach.
Figure 1
Stufflebeam's CIPP (Context, Input, Process, Product) Model The CIPP approach assumes that anything that might be evaluated could be usefully evaluated at various stages in its development. As indicated in the figures below, the proposed evaluation framework organizes the interests, questions, values, and participation of potential evaluation users and stakeholders around four types of evaluation which parallel four stages of development:
Ideally, evaluations of all four types will occur simultaneously and repeatedly throughout the life of an organization such as a university(at the macro level) that has multiple projects, programs, initiatives, and courses, and throughout the life of a particular program, such as Freshman Academy (at the micro level).
The participant oriented approach presented here combines Stufflebeam's approach with Patton's utilization-focused approach (Patton,1997), illustrated in Figure 2, into a comprehensive model, presented in Figure 3.
Figure 2. Patton's Utilization-focused Approach
Figure 3. CIPP and Utilization-Focused Evaluation combined
As represented in Figure 2, Patton argues that the key to evaluation utility is to identify people who are disposed to learning from evaluations. He outlines several procedures for identifying these users and then working with them to clarify what they want to know and what they are likely to do with information gathered by an evaluation. He gives many examples to persuade evaluators to not only organize their studies around users' questions and criteria but to also involve the "clients," stakeholders, or participants in gathering and interpreting the evaluation data as much as possible.
As shown in Figure 3. CIPP and Utilization-Focused Evaluation Combined, combining Stufflebeam's and Patton's approaches suggests that different users with different questions, criteria, and information needs may be more or less crucial at different stages in the life of an evaluand. To be most helpful, evaluations should be organized to meet the greatest needs of the most people at each of these stages.
In response to these principles, this evaluation began by identifying key stakeholders, ascertaining their definitions of and criteria for judging Freshman Academy, and then gathering descriptions of the program from several perspectives to compare to the criteria. The results of this effort and associated recommendations are presented in this report. Data collection methods are summarized in Appendix A.
Data collected for this study are summarized in appendices. The main conclusions are presented in response to the following questions:
Several parties with interests in the results were identified in the early stages of the evaluation while others emerged during the study (see Figure 4. Stakeholders, for a summary).
Administrators. Administrators of the Academy have been evaluating their efforts internally from its beginnings (as SHINE) in 1993. In 1999, administrators in the Academic Vice-president's office called for a formal external evaluation as well. Academy administrators and the Dean of General Education and Honors invited David Williams from the Department of Instructional Psychology and Technology to conduct the study.
The Academy directors began the effort by interviewing or exchanging memos with twenty-five administrators representing the Academic Vice-president's office, the office of Planning and Assessment, Student Life, Housing services, General Education and Honors, Admissions and Records, the Marriott School of Management, the David O. McKay School of Education, and the colleges of Fine Arts, Family, Home and Social Sciences, Engineering, Biology and Agriculture, Physical and Mathematical Sciences, Health and Human Performance, Religious Education, and Humanities.
Figure 4. Stakeholders
Faculty. Faculty who have taught Freshman Academy courses were considered to be principal stakeholders. Early in the evaluation, interviews and focus groups were held with them to clarify their questions and concerns. Surveys were sent to all faculty who were scheduled to teach classes associated with Freshman Academy.
Support staff. Non-faculty staff who serve freshmen were also considered to be important stakeholders. Interviews were held with representatives from the support units listed above to clarify their questions and concerns. Surveys were sent to 55 support staff members who were associated with Freshman Academy.
Students. Students are clearly important stakeholders who have expectations and can report on their experiences with Freshman Academy. Current students from Fall 1999 and others from previous years were involved in the evaluation through interviews, focus groups, and questionnaire responses.
Parents. Parents of students associated with Freshman Academy are key audiences with expectations of Freshman Academy. A few parents of students involved during Fall 1999 and previous semesters were involved in the evaluation through telephone interviews and completion of surveys.
Others. Wider audiences with related interests are anticipated as recipients of our findings because the value of the first year experience in university is being addressed by many educators and little is known about how to evaluate freshman programs effectively. We hope to share what we learn through general presentations and publications based on this report of the evaluation
What Do Stakeholders Care About?
What is the evaluand (the thing being evaluated) these people care about? Obviously, the main evaluand was Freshman Academy. Details will be presented throughout this report; but in brief, Freshman Academy consists of the following elements, based on the comments of the users listed above:
See Freshman Academy Home Page (http://academy.byu.edu/transition.html) for more details on the nature of Freshman Academy.
What Do Stakeholders Expect of Freshman Academy?
Based on extensive interviews and focus group discussions, a questionnaire was developed to ascertain stakeholders' expectations or criteria for performance for Freshman Academy. The questionnaire was administered in Spring 1999 to representatives of all the audiences/stakeholders/ information users (33 academic administrators, 67% responding; 61 support administrators, 48% responding; 243 faculty, 21% responding; 610 current and former Academy students, 10% responding; and 703 parents of Academy students, 22% responding). The instrument used is presented in Appendix B, " Expectations Survey." The results are summarized in Appendix C.
The analysis of these data leads to the main conclusion that although stakeholder groups do have some common expectations of Freshman Academy, they disagree extensively on many. As a result these should be serious efforts undertaken to come to a more widely-shared understanding of the aims and policies of Freshman Academy.
Thirteen putative expectations for Academy focus were ranked in this order among the top 20 (out of 55 originally identified expectations) both by mean ratings of 2.0 or lower and by rankings when responses from all groups were combined:
It is also interesting to note which expectations identified through interviews and focus groups were rated and ranked as lowest priorities for the Academy. For example, some of these were considered by key administrators to be of highest priority, but the majority of stakeholders had a different view. The lowest ranked objectives (considering all respondents) were:
Although, as noted, there was substantial agreement on a number of expectations, the groups of respondents ranked and rated their expectations differently for most items on the fifty-five item questionnaire. For example, the following expectations were ranked most highly by all respondents but the students:
Other differences are summarized in Appendix C and discussed below. As shown in Appendix C, each stakeholder group had some expectations they shared in common with at least one other group and some expectations that were unique to them. The unique expectations by group are listed below.
Students' unique expectations:
Academic administrators' unique expectations:
Support administrators' unique expectations:
The fact that the stakeholders do not agree on what Freshman Academy should be doing is no surprise. Participants have a variety of values and interests, as do similar stakeholders throughout the world, according to a review of related literature and experiences by Kuh and Banta (2000)
Nevertheless, in order to shape the Academy's program toward a more coherent set of expectations, these differences in perspective not only should be recognized but provision should be made for reducing discrepancies among stakeholder groups. One way to do this would be for the university administration to mandate a general direction for Freshman Academy, and at the same time encourage the Academy to maintain on-going dialogue among relevant stakeholder groups so that changing needs of freshmen (or of the university) could be ascertained and addressed over time.
How well is Freshman Academy meeting expectations?
Ideally, the 1999-2000 evaluation would have paused to allow the dialogue called for in the previous section. However, pressures to produce answers to the question, "How is Freshman Academy doing" and the fact that data collection activities were already underway when we discovered how disparate stakeholders' expectations were, dictated that we continue the study based on "some" of the criteria, if not the ideal set. In this section results associated with several criteria are presented briefly with details presented in appendices.
First, results associated with three questions asked by the Associate Academic Vice-president's office to guide budgetary decisions are summarized because the criteria included in this request departed slightly from the criteria discussed earlier but were considered essential by the senior administrators of the university and were included in an Interim Report (March, 2000) at their request. Next, results from all other data sources (Case Studies, Student Focus Groups, Faculty Focus Groups, Student Journals, Community Time Reports, Staff Interviews, Peer Advisor Focus Group, A Historical Study, and Previous Evaluations) are summarized to estimate how well many of the highest priority objectives for Freshman Academy are being met, based on what is known about these priorities currently.
In March 2000, the Associate Academic Vice-president's office asked for answers to three questions that assumed comparisons between Freshman Academy students and non-Academy students would be a reasonable means of measuring the program's success on three objectives which received mixed prioritization by participants in the surveys completed in August 1999:
To address the first question, data from BYU's Institutional Research office were gathered. These figures suggest that retention of students at the university was a major problem prior to 1993 (coincidentally the date when SHINE-- the first version of Freshman Academy-- began operation). However, during the years Freshman Academy has existed, retention has not been a serious concern at BYU. Likewise, interviews and surveys conducted for this study revealed the fact that according to most stakeholders who care about its success, retention of students is not a major purpose for Freshman Academy. Nevertheless, comparisons were made between Freshman Academy and non-Freshman Academy students during the seven years it has existed to ascertain how well Freshman Academy students are retained at the university compared to non-Academy students.
As summarized in Table 1 in Appendix D. Interim Report, students in Freshman Academy appear to be retained (percentage completing three or more semesters at BYU after starting) at a higher rate than non-Academy students
In conclusion, Freshman Academy is not increasing retention substantially (though there is a statistically significant difference favoring Academy students), according to the data provided by Institutional Research. Likewise, overall, the retention rate appears to be as high as it has ever been at BYU, at around 92%.
To address the second question regarding academic performance, grade distribution data from Institutional Research were employed. Although grades are defined in a wide variety of ways by different faculty and departments (including the forcing of grades to follow a normal distribution in some departments), they continue to be the only common metric for comparing achievement and student performance across courses and programs at the university. Therefore, grade point average information was obtained from Institutional Research for all students in Freshman Academy from fall 1993 through fall 1999 to compare against grade point averages for all non-Academy Freshmen from the same years in all courses they took.
To give some additional perspective, GPAs for the students' first semester as Freshmen were compared and then the GPAs from their most recent semester were compared to estimate the possible impact of Freshman Academy during the semester or term the students were involved in it, as well as later in their schooling experience.
As indicated in Table 2 in Appendix D. Interim Report, there were very few statistically significant differences (in terms of F tests) in performance between the Freshman Academy students and their counterparts in terms of overall GPA. As exceptions, there were significant differences favoring Freshman Academy students who began in Fall 1996 (and again during their latest semester) and in Summer 1997 (that term only). In contrast, the non-Freshman Academy students who began in Fall 1997 had higher GPAs than the Freshman Academy students that semester and during their latest semester. Otherwise, there were no statistically significant differences between groups in terms of GPA.
Expected GPA Differences. After the Interim Report was presented, additional analyses were conducted to compare Freshman Academy and non-Freshman Academy GPAs to their expected GPAs and to each other. This was done to test the hypothesis that the Academy students had improved more than the non-Academy students between their first and last semesters. The expected GPAs were calculated based on the students' high school and admission test information. The results are presented in Attachment A, "Comparing Estimated to Actual GPA."
It appears that when comparing the two groups on estimated GPA, the Freshman Academy students had higher expected GPAs than the non-Academy students in Fall 1996 (3.17 versus 3.08) and overall across all years (3.11 versus 3.08) but the non-Academy students had higher expected GPAs in Fall 1997 (3.14 versus 3.08). Also, it appears that in Summer 1997 (2.90 versus 3.28 or a difference of .385), the Freshman Academy students' estimated GPA was significantly lower than their actual first semester GPA and in Summer 1999 (2.78 versus 3.19 or a difference of .413) the non-Freshman Academy students' estimated GPA was significantly lower than their actual first semester GPA at statistically significant levels.
In sum, there do not appear to be any significant differences between Freshman Academy and non-Freshman Academy students on first semester and last semester GPAs when taking Estimated GPA into account. And, although some of the differences described earlier are statistically significant, they aren't really practically meaningful when taking into account the large numbers of students included in the analysis (large sample sizes usually result in statistically significant but practically unimportant test statistics). Therefore, although the Freshman Academy students were expected to do a little better due to their higher expected GPA numbers across all seven years of the analysis and although they did a little better some years and not so well other years on GPA, these differences should not be taken very seriously. The bottom line is, the Freshman Academy students were expected to perform and did perform about as well as their non-Academy students in terms of estimated and obtained GPA. The Freshman Academy experience is neither raising GPA nor hurting GPA of participating students beyond what would be expected.
Failure Rates. Similar analyses of students' "failure rates" (defined both as an obtained GPA of less than 2.00 and as a course grade of E or UW) for Freshmen from these two groups across several courses they shared in large numbers (ranging from three courses in Fall 1993 to 23 in Fall 1999), indicated some statistically significant but practically insignificant differences as shown in Table 3 in Appendix D. Interim Report.
The difference between 2.59% and 4.37% failure rates as shown there may be statistically significant but probably isn't practically meaningful. The failure rate for all Freshmen appears to be quite low at Brigham Young University and it may be even lower for Academy students.
The final question regarding students' attitudes about their experiences in the Academy was addressed briefly in the interim report based on data from a survey (presented in supporting documents associated with Appendix D) administered to Academy students (with parallel items administered to a sample of non-Academy students) during the final weeks of fall semester, 1999. However, more extensive data on students' feelings and perceptions about their experiences in Freshman Academy were collected via the other sources to be reported in later sections of this report.
The following statements summarize the students' perceptions about their experiences during their first semester at BYU in terms of 25 of the most highly rated Academy expectations as shown in the survey results in Appendix E.
In conclusion, looking across all the items and comparing their responses to responses from non-Freshman Academy students, Freshman Academy students appear to feel that the Academy experience was a positive one. In general, they were more involved or engaged and more positive than non-Academy students about their initial semester at BYU.
Finally, based on the data reported in the interim report, the following conclusions are reasonable:
Case studies of students from 17 of 19 learning communities were conducted by student observers who worked for the Faculty Center. They attended Academy classes and community times with one randomly selected student from each community, interviewed the student, and estimated how well four goals of Freshman Academy (composites of the highest priority objectives from the August 1999 survey: easing the transition from high school to university, enhancing academic and spiritual growth, and providing social support) were being accomplished in that student's experience.
Brief summaries of each case study are presented in Appendix F. Case Study Summaries to illustrate individual students' experiences. The observers' quantitative estimates of how well the students' experiences matched Freshman Academy's four highest priority objectives are summarized in Appendix G. Case Study Statistics in frequencies tables.
As indicated in Appendix G, in 87% of the 63 observation sessions with the 17 students, the observers felt the Freshman Academy experience they were observing positively eased the student's transition from high school to university life. In 76% of the instances they felt there was a positive impact on the student's academic growth by their participation in the Freshman Academy that day. In 84% of the instances, they felt there was a positive impact on the student's efforts to find social support through their participation in the Academy. And in 65% of the instances, the observers felt the Academy experience had a positive impact on helping the student grow spiritually.
The classes observed had enrollments ranging from 15 to 831 with attendance ranging from 7 to an estimate of 715. Attendance ratios ranged from 6% to over 100% in some small classes with visitors. The observations were made in a variety of classes (19% in American Heritage, 27% in religion, 10% in English, 6% in humanities, 5% in biology, and 2-3% in psychology, math, child development, sociology, PS100, music, and philosophy classes) and various Community Time sessions (16%). Visits were also made throughout the semester between October 4 and December 10.
Based on the descriptions made by the case study writers, it was estimated that 54% of the sessions observed were either lectures or media presentations (such as the showing of a video, while 18% consisted mostly of discussion sessions, 18% were community time activities and 10-11% were labs or lab-like activities.
Analysis of the case descriptions also allowed estimates of the comportment of the students and a rating of how close to the Academy ideal students and instructors were (in terms of criteria similar to those encouraged by Entwistle (2000) and others in a growing literature on university student learning that encourages students to be deep learners). Based on one interpretation of the cases, in 54% of the situations observed the students were actively participating in the instructors' or community leaders' planned activities. In 5% of the situations the students appeared to be "out of control" and not participating as invited. And in 41% of the situations there were students in both categories, spread across the entire range of participation fidelity.
On the other hand, in classroom situations, 36% of the instructors were clearly following the ideals set forth by Freshman Academy in their faculty training sessions while 35% were oriented toward the ideal and 29% were not teaching in the mode suggested by Freshman Academy.
An undergraduate research assistant studied the cases and prepared the following qualitative analysis summary.
Freshman Academy is strong in these areas:
Freshman Academy has relatively less influence in these areas:
Freshman Academy needs improvement in these areas:
In summary, the case study observers positively rated the elements of Freshman Academy on a quantitative scale in terms of how well it helps freshmen transition from high school to college, encourages them academically, helps them socially, and supports them spiritually (although they were more positive about the transition and social advantages than they were about the academic and spiritual). However, in their qualitative descriptions and comments, these observers found room for improvement in all these areas as well, particularly in terms of improving academic skills and attitudes.
Student focus group results substantiate results from other data sources from 73 students' perspectives. Thirteen Student Focus Groups were conducted (seven between 6 and 29 October, and six between 1 and 17 November, 1999), tape recorded, and summarized in field notes. The field notes were created by observers and then corrected and expanded by listening to the tape recordings.
The groups were composed of Freshman Academy students responding to an email inviting them to share their Freshman Academy experiences with a research team, independent of the Freshman Academy offices. Two hundred and eighty students representing all 19 learning communities were randomly selected and sent an email message inviting them to come eat pizza and talk about their Academy experiences with researchers who were independent of Freshman Academy. All 280 students were invited to come twice, once in October and once in November. Seventy-three or about 26% came to at least one session; but only four students came twice.
The Focus Groups were loosely structured so students could guide the direction of the conversation. Some of the groups focused on specific elements of Freshman Academy, like Community Time, while others discussed the experience more generally. Most student comments fit into six main topic categories: Academics, Social Aspects/Transition from HS to College, Community Time, Teachers, Small Class Size, Spiritual Aspects.
The analysis of the transcripts consisted of counting the number of positive and negative comments relating to each category, and then writing a short summary of each comment, as well as how many times it was stated. The results are discussed in these sections: Academics, Social Aspects, Community Time, Teachers, Small Class Size, and Spiritual Aspects.
Although there were more positive than negative comments, academics was an area where many students in the focus groups felt that Freshman Academy needed improvement. Negative comments focused on what many students felt was Academy classes' tendency to be too much like high school, not academically rigorous enough. Positive comments were mostly related to the benefits of Study Groups. Most students agreed that the close contact with their classmates facilitated the formation of study groups.
A majority (58%) of the students' comments about academics were positive as this statement illustrates: "Living by people in our community makes it easier to talk to them and get questions answered when we're confused about class assignments. Also we can learn from each others' learning styles and pick up other ways of succeeding."
The remaining 42% of the comments about academics were less positive, such as this comment: "I would rather have more responsibility. It feels too much like high school that we are in our own little group. I don't want to be coddled."
The social aspect of Freshman Academy received the most comments of any category (and the majority,140 or 69% of all the comments about the social aspects, were positive). Nearly everyone agreed that the Academy was socially beneficial. Negative comments, however, dealt with questions of whether or not the social aspect was emphasized too much, at the expense of study and learning.
The following quote illustrates the positive sentiments of the majority of participants in the focus groups: "You can build stronger friendships with other students because you live around them and can do things with them. That's what I like most about FA as a whole."
A less positive example: "The FA classes are loud. Students think they are still in high school" illustrates a concern expressed in nearly half the comments.
This element of Freshman Academy received more negative than positive comments (74% or 96 out of 129 comments). Few students felt community time was being used effectively. Most students wanted either to have community time less frequently, or to have better organized, more study-oriented community time sessions. Students did generally like the service projects and review sessions held during community time, but even then, attendance was poor because of lack of interest in Community Time concept in general. Many students felt their time could be better used elsewhere.
Sample comments include this positive one: "In our community time a TA was brought in right before a test and it really helped."
The more common negative comments are illustrated by this one: "I don't like the community time except for the social aspect. Otherwise it's a waste of time."
Nearly all the students agreed that they liked their Freshman Academy teachers, and felt closer to them than they did to other teachers. A high 90% of the 132 comments about the quality of the Academy teachers were positive. They liked having out-of-class activities and getting to know their teachers on a more personal level. The students were more comfortable in their classes because they felt they were important to their teachers. Many students commented on how they liked having their professors call them by name. Negative comments were few, and varied; there was no consensus.
An example of the typical positive comment: "I am able to get to know the teachers on a personal level by having lunch with them, etc. I feel like they are on the same side, and are really pulling for us."
Students enjoyed the small class size of at least one of their Freshman Academy classes because of the chance it gave them to get to know other students, and the teacher, better. They liked the feeling of being an individual rather than just a number. Some students, however, worried that they wouldn't be ready for university classes when the time came to leave the Academy.
About 95% of the 56 comments about small class sizes were positive as this quote illustrates: "College in general is more difficult academically, and so it is good to have the smaller classes and get into study groups. In other classes you don't get the opportunity to know other students."
Nearly everyone agreed they had grown spiritually from coming to BYU, but that Freshman Academy had little or no effect on that growth. The negative comments were not so much that the Academy had been a negative spiritual experience, but simply that it had no real spiritual effect except in the fact that roommates and others in the housing area shared both Academy and other university experiences with them. Instead, BYU Wards, religion classes, and associations with other LDS young people (which are available to all students, not just those in Freshman Academy), were the things that fostered spiritual growth.
Yet, 61% of the 41 comments about spiritual aspects of the Academy experience were positive as this quote illustrates, "I like being able to talk about scriptures and spiritual things with people I live with, and we do!"
The following negative comment illustrates the fact that there isn't a concern because this dimension of BYU is being met without the need for Academy attention: "Religion class in FA helps, but I'd be in a religion class anyway. Spirituality doesn't seem to be a major focus of FA. FA has not really impacted me spiritually."
In summary, the students in focus groups held in October and then late in the semester confirmed the findings from the case studies, surveys and other data sources indicating that Freshman Academy was particularly helpful to them in terms of social adjustment to the university. This help came both through smaller classes and personal attention from excellent teachers. Efforts to strengthen the students spiritually and academically didn't have as great an impact from the students' perspectives. However, it is possible that social adjustment may impact both academic and spiritual development. It seems quite clear from these group discussions that Community Time as it was conceived during this year needs adjusting to be more academically oriented. See Appendix H. Student Focus Groups for a detailed summary.
Focus groups were held with 33 faculty in October and December 1999 to invite them to comment on their experiences with Freshman Academy, clarify their criteria for judging it and express their evaluations of it. Transcripts of the sessions were reviewed to identify positive and negative comments faculty made in terms of the two main purposes for the Academy that they seemed to be noting-- to help students find social support and to grow academically. They also noted pros and cons for teachers and made suggestions for improving Freshman Academy.
The majority of positive comments (36) from faculty were focused on the social support and adjustment afforded by Freshman Academy. For example, four professors noted that students seemed to make friends and find a social support system through Freshman Academy. Five noted that they enjoyed inviting the students to their homes and felt that in this non-threatening environment the students opened up to them better.
In contrast, there were 12 negative comments regarding social aspects of Freshman Academy. Several of these centered on the experience some faculty had with students who didn't seem to take them seriously because they were caught up in group dynamics. For example, two of the comments were about rowdiness or lack of discipline and one noted that the students were too comfortable and their chattering made them appear more like high school than university students.
There were 9 positive comments regarding academics. The positives included three professors who felt Freshman Academy students were more willing to form study groups and study together, and others who noted that their students were awakened quickly to university academics, emphasized the value of academic study in all disciplines, and talked about research and sharing/learning with others. These are interesting, not to say puzzling, comments, given the students' (reported above) perceived absence of academic enhancement by the Academy.
Some of the positive comments focused on advantages to the faculty as well as the students, such as one professor who noted that it was fun to help students with their total development and not just with one subject. Another felt he was part of a community of scholars working together.
The 12 negative comments included four focused on the need to identify students who are struggling and help them sooner and the comments of a couple of teachers who felt that study groups really weren't helping that much because they were disorganized or didn't involve much academic interaction.
According to the professors in the focus groups, there were certain advantages to teaching in Freshman Academy, some they were surprised to discover. Of the 16 positive comments, the following are illustrative. Three teachers felt that teaching in the Academy was almost a religious experience. Three said it helped them get to know the students better than they could in regular classes. Three commented on how good it was to have out-of-class experiences with these students. One said it was enjoyable working with the other Freshman Academy professors and another felt these students were more focused than most freshmen.
But, there were 27 negative comments made as well. Three voiced the concern that the purposes for the Academy were undefined. Five complained that there was not enough coordination among the faculty in preparing integrated curricula for the students or for helping students who might be struggling. Four said they felt out of place or intrusive when eating with the students in the cafeteria or at Community Time events. Two were unsure what Community Time was supposed to accomplish. A couple of teachers felt inadequate in the role of being an advisor/counselor to the students as well as a teacher. Two others said the Academy doesn't mesh well either with some majors or with the honors program.
Suggestions for Improving Freshman Academy
Finally, the faculty offered several suggestions for improving Freshman Academy. These included (parentheses indicate numbers making this comment if greater than one):
Overall, the faculty who took the time to participate in the focus groups were quite positive about Freshman Academy, in particular the social element it provides and the assistance many freshmen seem to get from that. There weren't as many positive comments about the academic advantages of the Academy but there were few complaints as well. These faculty seem to enjoy teaching Academy students but they have several suggestions for improving the experience that ought to be considered. See Appendix I. Faculty Focus Groups for a detailed summary.
One hundred and ninety Freshman Academy students (10 from each of the 19 learning communities) were drawn randomly. The students were invited to answer a weekly questionnaire on the internet throughout the fall 1999 semester. The questionnaire asked students to report on their own participation in Freshman Academy classes, community time activities, and other events. They were also asked to rate the impact of these experiences on their transition from high school to university life, their academic and spiritual growth, and the social support and encouragement they felt from participation. Their ratings are summarized in Appendix J. Student Journal Summary and additional written comments are illustrated in Appendix K. Student Journal Entries.
A brief review of the data from 200 responses (we do not have a way of identifying how many of the 190 students responded over the course of the semester or how many times a given student responded) in Appendix J. Student Journal Summary indicates that the students attended most of their small Academy classes (89%), their Book of Mormon Freshman Academy class (94%), and their larger Freshman Academy class (90%). However, as would be expected from results reported earlier, only 48% of the weekly reports indicated that the students were attending their regular community time activity while only 35% attended a special community time, with 20% saying this question didn't apply to them.
When asked if they participated in a study group that week, only 54% of the time did they say yes. This is a lower percentage than might be expected given the focus of Freshman Academy on study groups. Similarly, 41% said they had eaten a meal with a Freshman Academy faculty member, while nearly half (49%) of the responses indicated they had used a resource learned from Freshman Academy faculty or staff (such as the library, online resources, labs, etc.; see comments in Question 9 in Appendix K. Student Journal Entries.)
About 71% of the ratings indicated a positive impact while 21% said Freshman Academy had no impact on the transition they were making from high school to the university setting (see their explanations for these responses in Question 15 in Appendix K. Student Journal Entries.).
Likewise, 60% of the responses reported a positive impact of their participation in Freshman Academy on their academic growth (see their explanations in Question 17 in Appendix K. Student Journal Entries) and 53% reported they had been positively influenced in their spiritual growth by participating in Freshman Academy (see their explanations in Question 19 in Appendix K. Student Journal Entries.
Finally, 74% of the responses indicated they felt Freshman Academy was successfully trying to help them find social support and encouragement (see their explanations in Question 21 in Appendix K. Student Journal Entries.).
For a summary of the quantitative results see Appendix J. Student Journal Summary and for the students' actual comments regarding the ups and downs of their experiences with Freshman Academy throughout the semester, see Appendix K. Student Journal Entries.
Relatively few journal entries were made-- assuming that a total of 190 (sampled) students over 11 weeks should have yielded well over 1500 entries and we only received 200. However, those that were submitted provide an insight into the experiences that these students had during October, November and December 1999, that should prove useful to anyone who would like to review them.
Taken as a whole, their comments confirm what was learned through questionnaires, focus groups, and observations. The majority of journal entries express the students' sentiment that Freshman Academy helped them most with social support and encouragement, next with transition from high school to university expectations, next with academic growth, and least with spiritual growth. They also confirmed that although some of the Community Time activities were valued, most were not and some actually interfered with academic pursuits.
Analysis of Community Time Reports
As mentioned earlier, a key element of the Freshman Academy program is participation in learning communities which combine three different courses for the participating students, peer advisors who are upperclassmen who help organize extra-curricular (non-class) activities, and a weekly community time during which many of these extra-curricular activities are held. The Peer Advisors were required to fill out planning and evaluation sheets for every community time and to keep these on file with the Academy office. These sheets asked the following questions:
Planning Sheet
Evaluation Sheet
Since peer advisors for each community were responsible to complete and save these planning and evaluation sheets, we decided to analyze a sample of the sheets to document the nature and quality of the activities from their perspectives. We made copies of the sheets, read through each of them, and entered the details listed there into a spreadsheet. A narrative summary of these details as well as a copy of the spreadsheet are presented in Appendix L. Community Time Analyses.
Analysis of the community time reports indicates a wide variety of experiences across the different communities and important lessons learned about how to use community time effectively. The analysis confirms the findings from other data sources that community time was not the most successful element of the Academy during this semester. Various approaches had been used in previous semesters and others will be tried in future years. Lessons were learned from this experience that should shape future efforts.
A student reviewer of the community leaders' reports identified the following ways to improve community time: Plan more effectively - understand how to handle food orders and payment, avoid low attendance due to conflicting activities (tests - com. H, dance-sport, etc. - com. C, classes during community time. - com. F, tests times- com. I), and lack of sufficient advertising.
The most common suggestions for improvements were: ensure quality test reviews; get a good TA or the professor to conduct study or review sessions; choose appropriate tests (not ones that people did well on); plan structure ahead of time to avoid wasting time and losing students' interest; advertise well enough in advance and thoroughly enough so that students are excited to attend; take time at the beginning of the semester to find out what kind of activities the students would like to do; some communities preferred all academic-based activities and other communities would only attend the fun-based activities.
Throughout the evaluation process, regular meetings and conversations were held with Freshman Academy administrators and staff members. These meetings and associated interviews provided vital information about the day-to-day operations of the Academy office as well as perspective on the purposes, processes, and outcomes associated with the Academy itself.
A major purpose of these interviews as well as the collaborative approach taken throughout the evaluation was to involve Academy administrators and staff as much as possible in the evaluation process. They helped shape the evaluation questions, gathered data from academic and support administrators through interviews, helped design and refine questionnaires, helped analyze and interpret results, and will review this report before it is distributed.
In addition, the administrators of Freshman Academy are actively involved in designing ongoing evaluation activities as they plan future iterations of the Academy itself. They are integrating evaluation into the program as an integral part of its development and actualization. They are joining with others throughout the university in creating an "assessment culture" that will use feedback loops to automatically generate guiding information about how their program is being implemented, what students and faculty are experiencing, and what results are being achieved.
As one example of these efforts, Clark Webb and Pat Esplin not only invited David Williams to conduct this formal evaluation but also invited two faculty members, Cody Carter (philosophy) and Bruce Smith (BioAg) to conduct their own investigations into Freshman Academy. They interviewed several participants, reflected on their own experiences as faculty members in the Academy and submitted brief reports. In addition, several studies have been conducted throughout the life of Freshman Academy by Pat Esplin and the various faculty associates she has worked with in administering the program. See Appendix P. Pre-1999 Evaluation Efforts for more information.
An extensive interview with Pat Esplin, one of the key administrators of the Academy for several years, is presented in Appendix M. Staff Interviews to provide insight into the philosophy and involvement of the administrators and staff. From this interview, the following conclusions were reached:
Peer Advisor Focus Group Results
To support the Freshmen in the Academy, a leadership structure has been organized. Students who have been in Freshman Academy previously or who are upperclassmen are selected to be Peer Advisors by the Academy administrators. Each Peer Advisor is responsible for at least one community, although some have two communities if they are smaller.
In addition, once they are accepted into Freshman Academy, students are invited to be considered for leadership council membership. If accepted, they take a class from Pat Esplin and others throughout the semester they are involved in the Academy in which they learn and practice various leadership principles. They also participate in council meetings with representatives from their own and other communities. They collaborate with their community's Peer Advisor to ascertain the needs and interests of their community members and to plan and organize activities for their community. They also work with the Peer Advisor to resolve problems the community faces as a whole and issues individuals have as well.
To better understand the Academy from the Peer Advisors' points of view, a focus group was held near the end of the Fall 1999 semester by Dr. Williams. The Peer Advisors were invited to talk about what they thought the purposes of the Academy were, and in particular what role they and the leadership council were to be playing in helping the members of the Academy fulfill their purposes. A summary of the focus group session is in Appendix N. Peer Advisor Focus Group Transcript. The key points learned during this session were:
Professor Gary Daynes and some students from his History 389 class conducted 25 oral histories into the origins of Freshman Academy (starting as the SHINE program) and the perceptions of various participants regarding the program. In summary, they found that:
See Appendix O. Gary Daynes' Historical Study for more details.
Since 1993, SHINE/Freshman Academy has evolved through various forms as participants have defined and redefined freshmen needs and the program to meet them. Throughout this evolution, administrators of the program have conducted internal evaluations to describe and assess the students and their needs, program efforts to address those needs, and plans for next generations of the program. A brief summary of some of these efforts is found in Appendix P. Pre-1997 Evaluation Efforts.
As part of this external evaluation, reports and files from many of these previous studies were reviewed. They are available for perusal in the Freshman Academy office. However, they do not follow a standard reporting format and did not address the same questions year by year. Therefore, it was decided that doing a quantitative meta-analysis across the studies would not be worth the effort. A more qualitative synthesis is reported here.
Across all the studies, it is clear that the objectives and procedures associated with Freshman Academy have been in flux throughout its life. It began as an effort to retain students at the university through a Housing initiative, based on the assumption that if students who live together take some of the same classes together, perhaps they will study together, help each other, and have greater success. And if they succeed in class, perhaps they will continue at the university for a longer term.
After a year or so, the academic administrators joined with Student Life and support administrators to explore ways to enhance this program even more. Faculty training in how to work with freshmen was planned and administered with varying degrees of success. Many other elements of the current program were added one by one, including the use of Peer Advisors, student leaders, focus on the "Aims of a BYU Education," use of feedback instruments to let students know how they compared to national norms on study habits and other characteristics, various approaches to new student orientation, alternative community envelopes (including "major-friendly" alignments of courses), varying efforts to invite faculty to collaborate in unifying courses within given envelopes, and much more.
As these and other changes were instigated, a variety of surveys, focus groups, interviews, records analyses, and other data gathering activities were conducted to estimate how well the Academy was performing on a wide variety of outcomes. Gender differences were assessed; experiences were documented; and performance by major, community, and many other predictor variables were plotted on colorful charts to tease out emerging patterns. Many of these reports are still available in the Freshman Academy offices and may be reviewed there.
Looking across all these inquiries, it seems reasonable to conclude that students in Freshman Academy tend to like their experience. The growing number of participants (from about 200 in the first year to 1800 or so currently) confirms that finding. Faculty and students had plenty of suggestions for improving the experience and they shared these openly during the focus groups and interviews. But no participants argued for the demise of this program. It was considered valuable according to all the studies internally conducted.
The results of previous evaluations are not presented uniformly, nor are they similarly categorized from year to year. This fact probably has two antecedents. One is due to the reality that there has never been a unified and universally accepted set of objectives for the Academy. This reality confirms the need for a "meeting of the minds" among various constituencies, particularly the University administration, regarding the educational purposes for which Freshman Academy should be held accountable. The other is that Freshman Academy has never been the same program for two consecutive years, consequently, as the practical realities have changed (larger numbers, more departments and faculty, etc.) the organization and operation have changed correspondingly.
Conclusions and Recommendations
Based on results from all the different sources reported here, the following conclusions and recommendations are offered.
Boyer, ?? (1987). College.
Cousins, J. B., Donohue, J. J., & Bloom, G. A. (1996). Collaborative evaluation in North America: evaluators' self-reported opinions, practices, and consequences. Evaluation Practice 17 (3), 207-226.
Cousins, J. B. & Whitmore, E. (1998). Framing participatory evaluation. In E. Whitmore (ed.), Participatory evaluation approaches. San Francisco: Jossey-Bass.
Entwistle, N. (2000). Promoting deep learning through teaching and assessment. Paper presented at the AAHE Assessment Conference, Charlotte, NC, June 14-18, 2000.
Fetterman, D. M. (1996). Empowerment evaluation: knowledge and tools for self-assessment and accountability. Thousand Oaks, CA: Sage.
Guba, E. G. & Lincoln, Y. S. (1989). Fourth generation evaluation. Thousand Oaks, CA: Sage.
House, E. R. & Howe, K. R. (1999). Values in education and social research. Thousand Oaks, CA: Sage.
Kuh, G. D. & Banta, T. W. (2000). Faculty-Student Affairs collaboration on assessment: lessons from the field. About campus, January-February, 4-11.
Patton, M. Q. (1997). Utilization-focused evaluation. (3rd ed.) Thousand Oaks, CA: Sage.
Mark, M. M., Henry, G. T., & Julnes, G. (2000). Evaluation: an integrated framework for understanding, guiding, and improving policies and programs. San Francisco, CA: Jossey-Bass.
Ryan, K. E. and DeStefano, L. (Eds) (2000). Evaluation as a democratic process: promoting inclusion, dialogue, and deliberation. New Directions for Program Evaluation, no. 30. San Francisco: Jossey-Bass.
Sanders, J. R. (Chair) (1994). The program evaluation standards. (2nd ed.) Thousand Oaks, CA: Sage.
Scriven, M. (1980). The logic of evaluation. Inverness, CA: Edgepress.
Scriven, M. (1991). Evaluation thesaurus. (4th ed.) Thousand Oaks, CA: Sage.
Stake, R. E. (1984). Program evaluation, particularly responsive evaluation. In G. F. Madaus, M. Scriven, and D. L.
Stufflebeam (eds.), Evaluation models. Boston: Kluwer-Nijhoff.
Stufflebeam, D. L. (1971). The relevance of the CIPP evaluation model for educational accountability. Journal of Research and Development in Education. 5(1), 19-25.
Stufflebeam, D L. (1975). Metaevaluation. Occasional Paper Series, no. 3. Kalamazoo, Mich.: Evaluation Center, Western Michigan University.
Terenzini, P. T., Pascarella, E. T., & Blimling, G. S. (1996). Students' out-of-class experiences and their influence on learning and cognitive development: a literature review. Journal of college student development, 37(2), 149-162.
Worthen, B. R., Sanders, J. R., & Fitzpatrick, J. L. (1997). Program evaluation: alternative approaches and practical guidelines. White Plains, NY: Longman.