The forum focused on the recent IMLS grant on Return on Investment (ROI) awarded to ARL, University of Illinois at Urbana-Champaign, and the University of Tennessee with a presentation by Carol Tenopir, co-PI, who led a discussion on the research that has taken place to this date and engaged the community to shape the ROI methodologies developed. The Forum also included a presentation by Linda Plunket (Boston University) regarding their efforts to create a culture of assessment. An update and community discussion about the 2010 Library Assessment Conference to be held on October 25–27 in Baltimore, Maryland, concluded the meeting.

In October, I was lucky enough to attend the very first “Undergraduate Research Practices Workshop” hosted by CLIR and run by Nancy Fried Foster from the University of Rochester. One of our topics was “Asking Good Questions,” and during the discussion I offered up the process we use here at Columbia. I hope some of you find our approach helpful, too.

This process is used at Columbia University Libraries to facilitate discussions about information needs in assessment projects. All of the questions that staff may want to ask student or faculty, via a survey, focus group, interview, or ethnographic study can be developed using this process. Many assessment projects start with “let’s do a survey.” A survey is a useful tool for gathering information, and we use them often. However, it’s not always the best tool for the job. No matter the methodology you decide on, it’s important to have good questions to collect relevant data. We can ask any number of “interesting” questions, but we want to be sure to prioritize our information needs, which this process facilitates. Assessment supports decision making at CUL, and I try to make that as direct a relationship as possible.

Another reason to write good questions is to make our assessment tools more “usable.” There’s often anxiety about the length of a survey. In my opinion, it’s not the length, but the usability of the survey that matters. If you have 20 questions that are relevant to the survey taker, and easy to understand and answer, you’ll get a higher response rate than having 5 poorly-worded, leading or confusing questions. We consistently get pretty high completion rates to our surveys, which hints at the general usability of the questions we write.

Step One: Using the “Project Team Brainstorming Activity” chart below identify what you know and what you don’t know about the project at hand. I call these items, simply, “Knowns” and “Unknowns.” Before we gather information from our users, it’s important to be sure that we don’t already have data on-hand that may answer our questions. Examples of Knowns can be anything from budget information, website usage statistics or gate count statistics to general limitations of the project or things that you know you will do during the project. Unknowns can be anything that team members want to know about the project or from the user population you’re working with. Team members should be encouraged to be exhaustive in their brainstorming – this is our opportunity to ask all of our questions! Team members shouldn’t worry at all about the phrasing of the question at this point, this is a brainstorming activity.

I usually give team members a paper worksheet (like the one below) before the meeting, and ask them each to brainstorm on their own, in preparation for a group brainstorming session.

At the group brainstorming session, start with the Knowns, and then move onto the Unknowns. Team members share the items they came up with on their own, and then move on with further brainstorming of more items.

Usually, as the Assessment Librarian, I am facilitating the brainstorming session, and recording all of the ideas on flip chart paper.

Step Two: Transfer all of the Unknowns to the “Writing Good Questions” document.

Step Three: Write an Information Need statement for each Unknown. A good way to do this is to write “I want to know” statements for each Unknown, as recommended by Nancy Fried Foster. For example, an Unknown might be “Should the new science library be open 24/7?” The corresponding Information Need would be “I want to know if undergraduate science students need library services overnight.” There may be multiple Information Needs for each Unknown. (This isn’t an exact science! The goal is to clarify the questions you brainstormed, to identify the exact information you need, and help you write a more clear question.)

Step Four: This is a good time for the project team to prioritize the information needs – what do you need to know now for the project at hand, and what would simply be nice to know? (You could add a “Priority” column to the worksheet.)

Step Five: Assign the appropriate audience and methodology (survey, focus group, interview, observation, ethnographic study, etc.) to each Information Need. Some questions will be appropriate for undergraduates, some just for faculty. You may want to use different methodologies for different populations.

Step Six: Based on the Information Need, audience, and methodology, write the text for each question you will ask. Try to be as clear as possible. Play around with the wording, maybe write two or three options for each Information Need, before choosing the final question text. I like to do this with a partner, so we can challenge each others word choices, and find the “best” way to ask a question. One good thing to try is “If we ask this question, people will give us this information. Will that help us?”

Step Seven: Test the questions, rewrite the questions as necessary, test, rewrite, test, rewrite, test, rewrite… I usually send the questions to a couple of students workers who will not be participating in the study. I ask them to tell me if any of the questions are unclear, confusing, use terms they don’t understand, or don’t allow them to give the answer they want to express. I also send the questions to colleagues who have experience with research, and a critical eye. At some point, you’ll need to get the study under way, so don’t spend too much time over-thinking the questions.  A couple of rounds of testing will pick up the red flag issues, and then you’ll notice that the feedback you’re getting from colleagues sounds nitpicky. Your questions are probably ready to go!

I encourage everyone to join the Anthrolib list serve run by Nancy Fried Foster a the University of Rochester.

For more information on our approach to assessment at Columbia, take a look at the materials available from the 2009 ACRL conference workshop Assessment Project Management in the Real World. I’d love to hear how others approach question writing – please post a comment and share your experience!

Agnes Tatarka and David Larsen, University of Chicago

Particularly with the recent publication of Studying Students (The Undergraduate Research Project at the University of Rochester), I think this approach may be gaining ground as a research method in libraries. I’d like to find out more about who is doing what and to share strategies, etc.

If you are not familiar with the the Rochester work, I recommend it highly. It’s available to download from the Association of College and Research Libraries at [http://www.ala.org/ala/acrl/acrlpubs/downloadables/downloads.cfm]

Thanks,

With the beginning of the school year just around the corner, it might provide an opportunity to find out something useful about users and especially about non-users. The idea was sparked by a recent discussion about focus groups held last year as part of a building renovation project. There was low turnout but some of the ideas captured there were very useful and provocative.

The question raised was how to move forward getting input from users. Rather than try to lure students away from or interupt their other activities, the beginning of the year afforded a rare opportunity to find them standing around with nothing to do; waiting in lines, at the bookstore, at the registrar, at the photo ID booth, at the parking office.

Since this would be too early to get anything about user experience with information seeking or library use, this survey would focus on the material circumstances of the students. We’re particularly interested in the technological world they live in, the hardware and the systems they use, the services they depend upon and expect to have available, and what the library needs to consider when we design our services.

I’d also like to capture more about students’ study habits. During the focus groups we had a strong indication that students want the library to be open 24×7, but I’d like to test that view against some concrete information about where and when they actually do their school work. However, I think it is too early to gather this sort of information now.

Any thoughts? Anyone have experience with a similar project and is willing to share what they did? Thanks for any suggestions.

– Lee Jaffe, UC Santa Cruz

This international study will:

• investigate factors influencing the selection and use of chat-based VRS
• study user and staff perceptions of satisfaction
• investigate why non-users of these services do not choose VRS
• seek to develop research-based recommendations for VRS staff to increase satisfaction.

A two day workshop in Denver, CO (September 28-30, 2006). Day One is themed “Understanding users and usage” and includes sessions on:

• Measuring Your Performance to Communicate Your Story
• Using the COUNTER Code of Practice: A Tutorial
• SUSHI at Work: A Tutorial
• Closing the Loop with Usability Testing: A Solutions Forum

2) Focussing on Users: evaluating the impact of libraries and information services on users and potential-users

A pair of action workshops: 21 September 2006 and 23 January 2007
King’s College, London, UK

The introductory one-day workshop will help you to explore how to get to grips with the impact of your services on users to choose how to evaluate service impact on targeted users/potential users to decide an action plan.

I am intrigued by all aspects of assessment and most recently have been looking at ways to measure the nature of the use of the academic library physical space. I’d be interested to hear from anyone else on this if you have thoughts, advice or experience.

Right now I am running a faculty survey on digital repositories that is getting a good response and most importantly, it is also generating interest and discussion about open access on campus. The survey response results will help us decide the direction that we should take with an institutional repository. This study is a follow-up to the University of Regina Task Force on University Publishing for which the hearings have just concluded on our campus.

We completed our first run of LibQUAL+ in spring 2006 and we were very pleased with the process and the results. Early in the process, through the text box contributions we received from our respondents in which they described their problems and frustration, we were able to identify a large systemic problem relating to access to full text resources from off campus. We would not have had the insight into this problem were it not for LibQUAL+ and the participation of our community. We have reported back to the community the good news about the resolution of the problem and our thanks for their participation. I am working now on our summary of LibQUAL+ findings.

Thank you Pam for setting up this forum and for bringing us all together!