8th Northumbria International Conference on
Performance Measurement in Libraries and Information Services
17-20th August 2009, Florence, An IFLA satellite conference
Dr. Lynne Porat
LibQUAL+® Exchange meeting
Monday 17th August 2009 – 9:00-13:00
Powerpoints from the LibQUAL+(R) Exchange are available through the LibQUAL+(R) website under Publications.
The first paper was presented by Stephen Town from the University of York, UK (formerly from Cranfield University), who discussed the use of LibQUAL +® around the world and outlined specific outcomes that had been implemented in UK libraries as a direct result of LibQUAL+® – such as receiving additional funding for web site development, for increased opening hours and for library refurbishment. Next Bruce Thompson from Texas A & M University, USA explained the rigorous scientific basis of LibQUAL +®.
After a “Round Robin” during which participants shared their ESP (Effective, Sustainable and Practical Assessment) and LibQUAL +® experiences, Jim Self from the University of Virginia, USA gave a fascinating paper on how the one LibQUAL+® question on faculty satisfaction with the electronic journal collection is a reliable indication of overall satisfaction with the library. Over 40 ARL libraries have consistently received negative scores to this question (i.e., the perceived score was below the minimum). Jim recommended conducting 5-minute follow-up telephone interviews by a neutral person to clarify the specific problems raised in LibQUAL+® by asking such questions as “How can we improve the electronic journal collection for you?”
Next, I gave a paper on LibQUAL +®Lite in Hebrew which received a lot of attention due to the fact that the University of Haifa was one of the first libraries in the world to run the LibQUAL+® Lite protocol consisting of 8 questions instead of 22. As predicted, response rates were higher and completion times were faster. I also mentioned our “roving laptops” concept whereby we sent four student employees around the campus with laptops to encourage people to complete the survey.
Then, Marja Hjelt and Saija Nieminen from the Helsinki University of Technology, Finland, and Monica Hammes from the University of Pretoria, South Africa reported on their experiences with LibQUAL+®. These presentations were very interesting not only because of their similar results and response rates to the University of Haifa’s, but also because of their marketing activities – such as sending SMS messages, handing out bookmarks and putting stickers on the floors -, and because of the way they involved all the staff in the survey process by providing daily updates of response rates and interesting comments.
After the break, Brinley Franklin, from the University of Connecticut, USA and the president-elect of ARL spoke about the importance of getting all staff members involved with the assessment process and strongly recommended using external experts to explain the results of LibQUAL+® and other surveys, and to bring about change.
Then, Colleen Cook from Texas A&M University, USA showed us the upward trends in her university’s LibQUAL+® results based on ten years of performing the survey on an annual basis. Next Stephen Town gave a particularly interesting presentation in which he compared Cranfield University’s excellent LibQUAL+® results with the University of York’s poor results. He described his attempts to improve the results in York by influencing staff attitudes and encouraging them to value the library. Martha Kyrillidou, Senior Director of ARL Statistics and Service Quality Programs adjourned the session by expressing her thanks for the informative and interesting presentations and by stating her pleasure at meeting all the participants.
Northumbria International Conference on Performance Measurement in Libraries and Information Services (PM8)
Monday 17th August 2009 – 14:00-18:00
On the afternoon of the 17th August, the PM8 conference officially began. In the keynote paper Robert Hayes from UCLA discussed how a costing model (ACM) can be used to explain the role of libraries in national economies. Next, Michiel Moll from the Cape Peninsular University of Technology (CPUT) in South Africa discussed his formula for calculating the number of staff necessary in small branch libraries. The next lecture by Colin Darch and Karin de Jager from the University of Cape Town, South Africa was of particular interest to me. It demonstrated the success of their “Research Commons” which provides a comfortable and inviting space for graduate students to write their papers away from the regular reference and study areas. The idea is based on the need for graduate students to produce whereas the “Learning Commons” or “Information Commons” are designed for undergraduate students to seek information. Last was John Bertot from the University of Maryland, USA (and the editor of Library Quarterly) who discussed the growing use of web-based surveys in public libraries.
Tuesday 18th August 2009 – 9:15-18:15
Brinley Franklin opened the day with a very interesting review of the ongoing discussions in the literature on whether users are deserting the library. He tested the debate against the evidence from ARL figures and came to the conclusion that although some counts – such as circulation and reference enquiries – are decreasing, students are still using libraries, but as study space and not for the resources they hold. He also discussed some results of the MINES for Libraries™ (Measuring the Impact of Networked Electronic Services) survey which is “an online, transaction-based survey that collects data on the purpose of use of electronic resources and the demographics of users”. Brinley reported that the majority of undergraduates use electronic resources outside the library – but on the university campus.
The next lecture by Niels Pors from the Royal School of Library and Information Science, Denmark focused on how public libraries in Denmark are dealing with their changing roles in the face of declining resources and use. Next, Elizabeth Curach from the University of Western Sydney, Australia, talked about how management and leadership skills turned her library from being one of the poorest ranking in client satisfaction in 1999 (using the Rodski/Insync survey) to being within the top quartile in national satisfaction in 2008 (based on CAUL – Council of Australian University Libraries – benchmarking). Interestingly, one of the ways of bringing about improvement was reducing the number of staff and encouraging early retirement – thereby bringing the salary portion of the library budget to a level comparable to other Australian academic libraries.
Next, Stephen Town reported on the survey of Performance Measurement and Assessment Practice in 108 UK research libraries. The results showed that although most library managers were committed to assessment, only one had a full-time assessment position, whereas 25% had an assessment committee and 25% had a part time assessment coordinator. None of the people involved in assessment had formal assessment training. Nearly all the libraries regularly assessed the following library activities: reference/enquiry services, electronic resources, circulation, acquisitions, ILL, web site, information literacy and the online catalog. Most just used statistics, suggestion boxes, data mining, outcomes evaluation, benchmarking and KPIs (Key Performance Indicators). Few libraries used ROI (Return on Investment), impact assessment, Balanced Scorecard, physical orientation, and mystery shoppers. The most frequently-cited improvements resulting from assessment activities were: longer opening hours, improved web sites, more IT facilities, improved re-shelving processes, more e-resources, better arrangement of library space and changed staffing structures.
Next Stephen gave a presentation on ARL’s “Effective, Sustainable, and Practical Assessment” program – ESP (given by Steve Hiller, Jim Self and Martha Kyrillidou) which has now expanded from North America to Israel, South Africa and UK. According to Stephen, the benefits of the program include: getting a broad, external view from experts which encourages a culture of assessment in the organization, receiving specific recommendations outside decision-makers’ comfort zones, and receiving a succinct and compelling report which outlines the current and optimal assessment situations in the institution.
Last was Roswitha Poll from the University of Munster, Germany who talked about measuring the quality and quantity of staff performance. She discussed soft methods such as asking users, colleagues, supervisors, and self-assessment and hard methods such as accuracy, cost and time-measurement.
Wednesday 19th August 2009
The day began with a very interesting presentation by Isabella Trahn from Nanyang Technological Universities (NTU), Singapore (formerly University of New South Wales (NSW), Australia). She discussed the difficulty of finding benchmarking partners in her region and outlined the importance of finding partners with the same cultural background. She cited literature that showed the cultural tendencies of Asians when filling in questionnaires: such as avoiding extremes and scoring conservatively, avoiding negative criticism of people, and not complaining directly. She suggested that if non-American libraries used the D-M score they might be able to offset the cultural biases that are inherent in LibQUAL+®. The D-M score is based on combining all the (minimum, perceived and desired) scores to get one score (from 1 to 100) for each question. She suggested that this should be an automated part of LibQUAL+® “Analytics” and that a standard definition of “good academic library practice” is needed by which libraries should measure themselves. The other advantage of D-M scores (as opposed to radar charts) according to Trahn is, the ease with which assessment librarians can explain LibQUAL+® results to stakeholders and staff.
Next Stephen Town gave a theoretical and practical presentation on the importance of defining values and disseminating them to library staff. Specifically, he spoke about the role of leadership in imparting the library’s values to its staff and patrons. He began with two key questions to ask staff:”Do you know our mission statement?” and “Does our mission statement inform or affect your day-to-day work?” He then outlined the “Competing Values Framework” (CVF) as a basis for defining library culture: 1. the ‘Clan’ culture which focuses on customer services and is like “an irresponsible country club”, 2. The ‘Adhocracy’ culture which focuses on digital library development and is like a “tumultuous anarchy”, 3. the ‘Hierarchy’ culture which focuses on content control and is like a “stifling bureaucracy”, and 4. the ‘Market’ culture which focuses on academic liaison and is like an “oppressive sweatshop”. He asked his staff to comment on the dominant culture in their institution. He also created a “values debate” blog for staff and began by asking staff to comment on the following two statements: “We are focused on users and responsive to their needs” and “We add value and contribute to educational outcomes”. In addition, he took the staff on a “values away day” to hear specifically what they value and disvalue in the library.
Steve Hiller from the University of Washington, Seattle and I followed with a presentation on the “In-Library Use” (ILU) survey that was carried out three times in Washington and once in Haifa (in Hebrew). The survey is designed to find out who is using the library and what they are doing there. In addition, it highlights the services users think are important and how they rate them. The main finding was that undergraduates tend to use the library as study space and not for the collections or staff, whereas graduates and faculty hardly use the physical library at all. Students in the sciences in particular, use the library for group work, whereas humanities students use it for individual study. An interesting difference between Washington and Haifa was that in Haifa students use the collections and request staff assistance much more than in Washington, possibly due to cultural differences such as lack of familiarity with libraries and the necessity of reading English texts (as opposed to works in their native Hebrew or Arabic). Steve and I both emphasized the importance of using the comments to identify actionable items.
Next was a panel session on “impact measurement” with Stephen Town, Steve Hiller, Roswitha Poll and John Bertot. Stephen discussed the importance of SCONUL’s (Society of College, National and University Libraries) VAMP (Value and Impact Measurement Program) which provides a toolkit for directors to show the value of library services to their stakeholders by emphasizing: value for money (VFM) and economic impact, impact on learning and teaching, and impact on research. In addition, Stephen mentioned SCONUL’s WGPI (Working Group on Performance Improvement) and SCONUL’s Performance Portal and Wiki as sources for information on current impact measurement activities.
The afternoon ended with an excellent workshop by Raynna Bowlby and Martha Kyrillidou, both from ARL, on “Guiding library staff to act on user survey results”. Raynna began by asking “how many of you are the person at library meetings who says ‘but where’s the data?’ and then your colleagues reply ‘we don’t need data we know what our patrons want’” – which is a very familiar situation for me. Raynna and Martha suggested taking four or five LibQUAL+® questions and presenting them to staff from the relevant departments to remind them of the questions and to show them that they are responsible for these issues. Then they suggested, focusing on one or two issues that received high LibQUAL+® scores and can therefore be considered to be working well, one or two issues that need improvement and one or two that are already being improved. They suggested that asking staff to define SMART (Specific, Measureable, Attainable, Relevant and Time-bound) goals for each problematic issue e.g. if re-shelving is a problem a SMART goal could be “shelf-read the five most used areas based on circulation statistics once a year” or if lighting is a problem “add lights to 45% of tables by the end of the year” or “improve LibQUAL+® grade on electronic journals from 7.02 to 7.70 by 2010”. It was very interesting for me to hear at this workshop how little support assessment librarians got for their assessment activities from their managers and colleagues – a situation that is very different from the situation at the University of Haifa, where there is a part-time assessment librarian (me) and eight part-time members of the Assessment Team. Raynna showed a free web site (wordle.net) for analyzing the comments which is especially useful if “Atlas.ti” is not available. This site allows the user to copy and paste all the LibQUAL+® comments and get a graphical display of the words that appear most frequently.
Thursday 20th August 2009
The day began with Bruce Thompson from Texas A&M University presenting the LibQUAL+® Lite protocol based on the University of Haifa’s results. The findings show that there were no major differences between the Lite and Long results, only a higher response rate and shorter completion times for the Lite. However, the results for the IS (Information Control) and LP (Library as Place) were slightly higher in the Long version.
Next, Monica Hammes from the University of Pretoria, South Africa talked about the third generation Balanced Scorecard as a strategic management tool, and Charles Lowry from ARL presented ClimateQUAL™ which is designed to assess staff attitudes and their perceptions on the organizational climate and culture. The idea is that by changing staff attitudes you can bring about change.
The conference closed with a keynote paper by Alfredo Serrai from the University or Rome who talked about the ethical, scientific and cultural duty of preserving libraries in times of change.
I had the privilege at the end of the conference of flying from Florence to Rome with Martha Kyrillidou. During the time we spent together she told me about Aaron Cohen Associates who had conducted a Visual Scan Space Audit at Duquesne University’s Gumberg Library following poor LibQUAL+® LP (library as place) results and that following changes the next survey results showed major improvements. She also mentioned TICER (Tilburg Innovation Centre for Electronic Resources) a Dutch company that offers consultancy on digital libraries and IT infrastructure, and organizes courses and seminars.
The conference was enjoyable both personally and professionally. I met all the leading people in the field of assessment and had the pleasure of spending time with them in the evenings as well as during the conference.
For more news and pictures also see Libraries Plus Blog