by George Lorenzo

When you look at the history behind Chalk and Wire (C&W), it becomes clear why it is a successful company with extremely satisfied customers. For one, C&W was founded by a passionate educator, Geoff Irvine, with a well-seasoned background. Irvine, who is from Ottawa, Canada, taught high school history for 22 years; was an education professor at the university level; authored several textbooks; worked as a private consultant, specializing in education technology and school leadership; and has received a good number of professional awards and citations, including international recognition for innovation in the integration of digital technologies in education from the Nortel Institute for Excellence in Education.

Start-Up of a Web-Based Portfolio and Assessment System
Secondly, during the early development phase of C&W, going back to the late 1990s, Irvine started working with the Canadian government's main research arm for communications technology R&D, the Communications Research Centre (CRC). At CRC, Irvine hooked up with a talented senior researcher and technology expert, Thomas E. Whalen. Together, and with help from other talented programmers, they developed a web-based portfolio and assessment system. Within a few years, Irvine moved out of the research phase at CRC and into the CEO position of C&W, a small and intimate company that entered the marketplace on the cusp of a trend happening mostly in schools and colleges of education.

That trend was about the development of electronic portfolio (eportfolio) assessment systems, being driven, in large part, by a National Council for Accreditation of Teacher Education (NCATE) standard in 2000, establishing that teacher education units must implement an assessment system that collects and analyzes data on graduate performance. The technology and pedagogical leanings built inside C&W’s early-stage eportfolio assessment system were a natural fit to help meet this NCATE standard, giving Irvine’s company a good running start in the field.

Putting Usability First
Irvine and Whalen developed C&W with a software-development mindset that focused on having live usability factors take the highest precedence for fixes and upgrades. In other words, they launched C&W on their own servers in an application service provider (ASP) model and partnered with a small group of dedicated institutions, listened to their users’ complaints and suggestions and then implemented fixes and upgrades as quickly as possible, oftentimes within hours.

This usability mindset and ASP structure still holds true today at C&W. "The basic argument is really simple," Irvine says. "Stop paying attention to marketing (C&W’s marketing strategy is minimalist today as well) and go right to the users. What do they need? Ignore the usual processes that get in the way of developing software and build it. Release it early. Get it in front of people. If it breaks, somebody will report it, and we will fix it before most people even notice that it’s broken. We will just keep moving on. We will just keep building software. Building, building, building . . ."

Early Builders
The first "core partner" during the company’s early launch and development phase was Southern Utah University (SUU). It started when SUU Professor of Education Deborah Hill met Irvine at an educational conference eight years ago. At the time Hill was experimenting with having her teacher candidate students build eportfolio websites, mostly for displaying artifacts (assignments tied to teacher standards), using Macromedia Dreamweaver (now Adobe Dreamweaver). She and Irvine decided to jointly work on further developing the C&W eportfolio and assessment tools for SUU’s College of Education. The process started out with five tech-savvy education students using C&W the first semester and then moved up to 30 students, all of whom provided important usability feedback.

Today all of the College of Education teacher candidates at SUU, plus graduate-level educational leadership and administrative/supervisory licensing students, create electronic portfolios, Hill says. "We score between 250 and 400 eportfolios a year." In addition to being fully operational in SUU’s College of Education, "we are moving towards an institutional eportfolio for all students as a way of documenting their competencies," she adds.

ePortfolio2 and CWReporter^TM
Hill also notes that, overall, C&W’s software and the services it provides "have grown leaps and bounds. They are doing what they should be doing." The end result thus far is ePortfolio2 and CWReporter. As noted on the C&W website, the company’s strength is "listening to what our users need and then providing no cost solutions to them that are not only quickly implemented, but are also elegantly simple to use, stable and scaleable. We then follow that up with long-term, personalized support."

ePortfolio2 is an eportfolio authoring tool that has unlimited storage, secure sharing capabilities and is completely portable. Users can import and align performance data from any database source and aggregate and disaggregate that data using both performance and demographic variables. Its data analysis functions allow users to conduct high-end statistical operations that flag results. Users can also create printable custom reports with tables and graphs.

CWReporter is a desktop reporting tool that allows users to download assessment data to their desktops and perform complex statistical analyses. Users can view data from a variety of perspectives, filter reports by any demographic data they choose and see the output in an easy-to-read report.

Building New Solutions
When describing the relationship that SUU’s College of Education had with C&W over the years, Hill notes that "they modify it (the C&W software) to meet your needs. It is wonderful. You call them and say ‘could we do this, or I wonder if, or would you think about that,’ and they are willing to try. I have never heard them say they would not consider an idea. Every time I have come up with something, it has been ‘hmmm, that’s interesting.’"

This kind of testimonial is a common refrain from C&W customers. Kelly McClure, assistant professor of Instructional Technology in Cameron University’s Department of Education, was instrumental in the selection of C&W and the design and re-design of the department’s assessment system. McClure explains that "what I like about Chalk and Wire is that the company itself allowed me to say, ‘hey, I need this done or I need this kind of programming,’ and they would make those changes within the software for me."

Lauren Andreson, is an instructional support specialist at Southern New Hampshire University (SNHU) and the person responsible for C&W eportfolio training of faculty and students. She also helps manage and implement the assessment system reporting mechanism sides of the software. "What I like about Chalk and Wire is even if we are the only school asking to make some changes, they will consider it - and they have implemented them for us," Andreson explains. "I don’t know how they are able to do that, but I get almost instantaneous responses from them. I feel like they are this hidden jewel."

Three filters help to determine whether or not a new feature is added to C&W products and services: 1) Is it pedagogically sound or, put another way, Irvine asks, "is it good for kids?" 2) Is there any reasonable chance that somebody other than the people who asked for it would want to use it? 3) Is it cost effective to run in the long run?

"We are building 130 to 140 new features a year," Irvine says. "Our clients have become our marketing department."

The ORU Story
Oral Roberts University (ORU), perhaps unwittingly, has become a very big word-of-mouth ambassador for C&W, going back to 2003 when ORU, like SUU, met Irvine at an education conference. ORU also became a core partner in the development of the C&W product and services that exist today.

The story of how ORU successfully implemented C&W in its School of Education, and later across campus, is an exceptional model of how an institution can actually develop an eportfolio assessment system by creating an institutional culture of evidence with sophisticated eportfolio and reporting features and functions.

As proof of their success, ORU was recognized with a 2007 Council for Higher Education (CHEA) Award for Institutional Progress in Student Learning Outcomes.

Beginning with the School of Ed
ORU’s partnership with C&W began when David Hand, dean of ORU’s School of Education, and Kim Boyd, associate dean, first visited with Irvine, at a small booth he was manning at the 2003 American Association of Colleges for Teacher Education Conference, held in New Orleans, LA. Hand and Boyd were dealing with their first foray into obtaining NCATE accreditation, which they had been working on for about one year. "NCATE was looking for hard data," Hand says. "We needed a means to aggregate and disaggregate data. The state of Oklahoma required portfolios for teacher education, and we said ‘why not use our portfolios electronically.’ In our naivete we thought that there must be some company out there that has an electronic portfolio where once students put something into it and we grade it, then we can just push a button to synthesize all that."

"We started working with Chalk and Wire," adds Boyd. "They were able to take the work that we had completed over a year’s time and pretty much take our assessment system and put it into their system. It allowed us to start entering data. Plus, Chalk and Wire was able to aggregate and disaggreate the data in a way that we needed."

Irvine says that the performance and demographic variables that NCATE wanted to see were actually "middle zone statistics" that were not difficult to produce. Basically, in concert with the School of Education, C&W expanded the capabilities of the rubrics used to rate the student artifacts that were aligned with ORU and NCATE teacher education standards and housed in what was now their eportfolios. The eportfolios also housed each student’s demographic profile, such as gender, age, education, race, etc. The C&W software aggregated and disaggregated the data collected from the rubrics and eportfolios, essentially marrying the two and using both performance and demographic variables to compile printable status reports, with tables and graphs, for the NCATE accreditation team to review during their visits.

"In the end what happened was that Oral Roberts University was widely successful in their NCATE visits," Irvine adds.

Boyd offers a telling anecdote from the final NCATE visit: "They asked for one additional report. They wanted to know how our students were doing by their ethnic status. So I went in and printed out a report for them in 30 seconds. They thought it was going to take the rest of the day. I just handed it to them, and they were literally blown away."

Time-consuming Effort
Since gaining NCATE accreditation in 2006, 29 higher education institutions have visited with Hand and Boyd at the ORU campus to see first hand how they have utilized C&W. Plus, Hand and Boyd have frequently presented and held workshops and seminars on their success with C&W. They both agree that any school or college of education must realize that to incorporate a similar eportfolio assessment system requires a concerted three-year effort and process, at minimum.

"I think one of the problems and one of the challenges that a lot of institutions face when they are trying to implement this kind of system is that they don’t understand the preliminary work that has to take place," Boyd explains. "Even when you know what you want the system to do for you, before you can implement the system it takes at least one year’s worth of work to develop it. What happens a lot of times is that institutions have their site visits coming up, and they think they can implement something like this in six months. It takes much longer than that."

Going Across Campus
The much larger level of C&W’s partnership with ORU came from Ralph Fagin, executive vice president for academic affairs and currently interim president. In 2003, Fagin saw the potential of eportfolios to go across campus in every department, and he had the foresight to see how this could be utilized for an upcoming 2007-08 comprehensive reaccreditation visits by the Higher Learning Commission (HLC).

Enter Cal Easterling, ORU’s director of institutional research. . . Easterling notes that Fagin’s proposal "did not meet with universal horror. Our academic vice president was really into student learning, and we were turning a corner from being primarily involved in teaching to emphasizing learning. It does not matter if you get a standing ovation from your students or if you get teaching awards. At the end of the day what counts is did they learn something, and can they take home skills." (The kind of thinking that fits well with an eportfolio assessment system implementation . . .)

An Involved, Expansive Process
The first major task at hand entailed determining student learning outcomes for an entire university. Over the 2003-04 academic year, a general education committee, as well as committees from every department, consisting of students, faculty and administrators, came together and ultimately devised five student learning outcomes and related proficiencies. The five outcomes are spiritually alive, intellectually alert, physically disciplined, socially adept and professionally competent. An eportfolio assessment system was constructed around these outcomes and each of the related proficiencies. Today, every ORU student has an eportfolio aligned to the outcomes. As noted in a paper about how the system was built:

These outcomes were then correlated with specific required student assignments (artifacts), and customized faculty-created assessment rubrics were designed for every academic program at Oral Roberts University, including General Education and co-curricular programs. Each academic program finalized its own mission statement and designed its student learning outcomes, required artifacts, and evaluation rubrics to correlate with and reflect the university’s overall mission and student learning outcomes.

All of this information was then uploaded to the ePortfolio, which is an institution-wide Internet-based system for displaying and evaluating student work via the online rubrics, with the capability of collecting aggregated and disaggregated assessment data electronically and giving valuable feedback to the student, to each department and school, and to the university. ¹

Essentially the entire process of building out this university-wide eportfolio assessment system took about four years, which, considering what they built, is a relative short period of time. "When the accreditors came - we had our site (HLC) visit in January 2008 - they loved it," Easterling says. "The HLC called Doctor Fagin and I and talked with us for about an hour and a half (about ORU’s eportfolio assessment system), and they are holding it as an example. They want other schools to adopt eportfolios, whether Chalk and Wire or not. However, we found Chalk and Wire has been able to do what we need. They still make changes for us. They really try to work with us."

Personal Growth Assessment
Easterling and Fagin recently presented on ORU’s eportfolio assessment system at the 113^th Annual Meeting of the Higher Learning Commission, held in Chicago in April 2008. In an accompanying paper to the PowerPoint presentation they gave at the meeting, Easterling and Fagin describe the logistics of developing a personal growth assessment (PGA) that directly measures a student’s progress on learning outcomes. ^{2, 3}  A PGA is an average of multiple assessment results that were graded within rubrics on key learning outcomes. Easterling says that currently there are at least a total of 900 rubrics across departments that he has recorded. "It is pretty impressive," he adds. ORU also has a complete curriculum map that shows every course at the university and how each course relates to five major student learning outcomes, what assignments are related to the outcomes, and what eportfolio artifacts accompany those assignments and outcomes.

At the end of a grading period, in addition to receiving grade point averages on their courses, ORU students receive a PGA score, based on a four-point scale, but much different than a GPA, with 4 being exemplary, 3 being competent, 2 being average, 1 being unacceptable and 0 meaning not attempted.

In addition, in relation to university-side reporting statistics, students are able to see how their overall scores within the five student learning outcomes areas compare to other ORU students. For example, the average overall mean score by outcomes and proficiencies, covering 17,001 ORU students in the intellectually alert outcome for the 2006-07 academic year in general education was 3.1. Intellectually alert has proficiencies on critical thinking, information literacy, global and historical perspectives, aesthetic appreciation and intellectual creativity. Each proficiency is also broken out, with mean scores of 2.9, 3.0, 2.8, 3.5 and 3.2 respectively. Students can see, by major, as well as by all ORU students, how their scores compare. ⁴

Linking to Broader Outcomes
This kind of reporting generated by the C&W software is only the tip of the proverbial iceberg when it comes to aggregating and disaggregating data. A wide variety of reports can be generated at the individual, course, program and university-wide levels, all custom segmented under numerous demographics. As noted in Easterling’s and Fagin’s paper,

The ability to portray average scores on learning outcomes from assessments employed across the curriculum offers major benefits. Our goal in using this method of reporting levels of achievement is to change the thinking and viewpoints of students, faculty members and university constituents. Such a paradigm shift enables students to become aware that learning transcends individual courses and majors. With assessments being accumulated across a range of courses, a link is established between outcomes learned at lesson and course levels with broader outcomes representing the acquisition or enhancement of complex integrated skills and abilities during the course of an entire academic degree program. ⁵

Institutional Reporting
When talking about C&W’s reporting features and functions, Irvine says, "our reporting was designed from the very beginning for institutional research purposes, so it has many levels to it and answers very complex questions. It does linear aggression. It does multiple variable analysis. It does pattern analysis. It is like an Olympic swimming pool with a very deep end. But you don’t have to go there. If you want to do a fairly straight forward analysis, such as the mean, the medium and the standard deviation, okay, it is down there in the shallow end."

Irvine adds that C&W is "not earmarked for educational faculties only. We ask what standards you want to use. You tell us where the authoritative standard is and how you want it to be portrayed, or if you want a skeleton that you can edit however you like. We basically customize the site to the way you want it."

Additionally, the assessment side of the C&W product can stand on its own without a portfolio, Irvine notes. "You can import whatever data you want, including data that you already have, and get some meaningful reports out of it and align it at the same time.

"The bottom line is I want to put you in a position where you can leave the door open and say bring it on. We want our clients to be able to say, ‘we don’t care who shows up (accreditors, institutional administrators and/or stakeholders). If you want to see what we do, we’ll show you.’"

Endnotes:

3. Cal Easterling and Ralph Fagin, "Personal Growth Assessment (PGA) Developing Composite Scores That Directly Measure Learning," A Collection of Papers on Self Study and Institutional Improvement, The Higher Learning Commission of the North Central Association of Colleges and Schools, 2008, Volume 2, Chapter 4.

4.See http://eportfolio.oru.edu and click on the "portfolio data" link to view a variety of reports.

5. Cal Easterling and Ralph Fagin, "Personal Growth Assessment (PGA): Developing Composite Scores That Directly Measure Learning," A Collection of Papers on Self Study and Institutional Improvement, The Higher Learning Commission of the North Central Association of Colleges and Schools, 2008, Volume 2, Chapter 4.