Archive
Assessment News & Resources
If you have not seen the latest Library Connect newsletter (March 2012), please take a look. Donna Carroll, Academic Service Development Manager, University of Warwick Library, Conetry, UK has written a feature article entitled “Get to know your users better: Seven secrets for successful surveys”. Additionally, there are several articles on “Power the Librarian: Exploring the impact of the 21st Century Librarian”.
Assessment Ideas & Topics Welcomed
If anyone has an idea or question regarding an assessment topic or panel/ presentation for an upcoming conference please feel free to share it with us by using the ‘Ideas’ tab at the top of the blog page.
General assessment topics of interest or questions may be posted to the blog’s main page (here).
Jennifer, Erin, and I look forward to working with you this year.
Calculating per capita expenditures
What is the standard for calculating per capita academic library expenditures? What typically is used for the per capita part– total full-time students? FTE? Total full- and part-time students? Something else? Thanks…
Institutional Effectiveness
How many of you work in a library that produces an institutional effectiveness assessment plan (IEAP)?
How many of you have some responsibility for your library’s IEAP?
Anyone willing to share their IEAPs?
Here at Georgia State, all academic and administrative units are required to complete an IEAP at the end of every fiscal year. The structure is goal, outcomes, measures, and targets. The goal is a new piece this fiscal year. Since we were working with a new goal at GSU, we came up with an entirely new IEAP with new outcomes, measures, etc. We decided to make the IEAP work for us this time, instead of just an exercise we were completing for the Provost, and we created concrete, practical outcomes, measures, and targets.
I asked the campus assessment people to meet with me to go over the new IEAP. Their major concern was that some of our targets are really action items, since the targets are yes, we did it/no, we did not. Since this is a new year with no comparison data, we sort of had to set it up this way to establish benchmarks.
Here are:
The Georgia State University Library IEAP and
The Odum Library, Valdosta State University Archives and Special Collections IEAP from Deborah Davis.
Please share yours!
Evaluating Customer Service
We all know what constitutes good customer service…right? I think we do, but we don’t always provide it. Good customer service requires more energy! Smiling! Making eye contact! Active listening! It can be exhausting.
At Georgia State University Library, we have a customer service policy that describes the service our patrons can expect from our employees. We provide regular customer service training that is unique to our customer service policy, but we don’t have a method in place for evaluating customer service.
We currently are planning a mystery shopper evaluation program. Undercover “patrons” will visit our in-person, virtual, and phone service points and evaluate the service provided. We plan to use a rubric as the evaluation tool, and most of the rubric items are based on tenets of the customer service policy.
What methods have you used to evaluate the customer service provided by your employees? Did you have trouble separating reference skills from customer service? Please add a comment (the comments link is just under the post title) and join in the discussion!
Jennifer@GSU
Promoting Surveys
(From the AIG wiki; originally posted March 2009)
Hi everyone. I have created some online library surveys and I’m wondering about the best way to promote them (besides placing a link on the home page). I’ve thought about posters/flyers, but I’m unsure if the students will notice them. Does anyone have any suggestions?
Thanks!
Stacy
———–
jewell.anderson@… said at 11:06 am on Mar 25, 2009
Hi Stacy,
If you can, try sending a blanket e-mail request with a link to the survey(s) to student e-mail accounts. In the e-mail be sure to tell them it will only take a few minutes of their time and would really help the library perform better, for them.
Think about working through the campus student organizations, as well for promotional and participation ideas.
Also, edible incentive works wonders. Consider the possibility of a printable (pdf?) certificate at the end of the survey that students can bring to the lib and exchange for mini candy bar/ cookie/ whatever. That’ll encourage them to both fill out the survey and come to the library…
Good luck!
Jewell Anderson
————
Stacy said at 8:18 am on Mar 26, 2009
Hi Jewell,
Thank you for your ideas. They are great! I will definitely talk to Student Life about a blanket email. I like your edible incentive plan, too. Maybe I could also find some cheap little items, like highlighters or key chains. If I was getting candy or cookies after the survey, I’d complete it for sure!
Have you ever used posters and/or flyers to capture the attention of students? If so, do you think they work?
Reference Service Assessment
(From the AIG wiki; originally posted June 2008)
June 2008 Discussion Topic: Assessing Reference Service
This month please post your ideas for assessing service provided from reference points—in-person or virtual. Do you go beyond hash marks on a paper form? If so, how do you measure the quality of reference service: Peer observations, secret shoppers, surveying patrons, patron evaluation forms…?
Posted by Bella Karr Gerlich, Re Reference Assessment – I can discuss my personal experiences with experimenting with alternate data gathering for reference. I felt very frustrated having to explain the effort / skills etc behind the reference desk, so to go beyond the tick mark I developed the READ Scale (Reference Effort Assessment Data) tool. The READ Scale is a 6 point scale tool for recording the effort / knowledge / skills / teaching that occur during a reference transaction. My research partner (Lynn Berard, Carnegie Mellon) and I did a national study in the spring of 2007 testing the Scale at 14 diverse academic institutions, including our own. ( We are presenting the results of the national study at ALA this summer, but I would be happy to share more information with anyone interested.) At GCSU, we used the data gathered from our participation in the study to identify those times where professional reference staff were needed on the desk (based on number of transactions and READ Scale rankings) and when they were not – we finally had data to say that it was more cost effective and benefitted our users to have the professional staff on duty during earlier in the day as opposed to being on the desk after 9 pm during the week and on Saturdays, when the only questions received were in the ‘1’ category (ie, where is the bathroom?). The professional staff are not only happier to be here earlier and for more hours during the regular work day, but it benefits the institution because staff can participate on additional committees, attend more meetings / assist with projects and cover additional reference shifts allowing for greater flexibility in the schedule overall. The Scale also gives reference staff a chance to qualify their work, and they feel good about the service they give to the students and the faculty.
Assessment of Library Promotional Activities
(From the AIG wiki; originally posted March 2008)
March 2008 Discussion Topic: Assessment of Library Promotional Activities
Ciara Healy suggested this month’s discussion topic, which is based on a message she received from the ALA ACRL Community and Junior College Libraries Section (CJCLS) discussion list. Thanks, Ciara.
Think about libraries hosting non-traditional (in the library sense) events to promote library awareness. These events don’t promote library services and resources directly, but draw users’ attention to the library. For example, the Learning Commons and Special Collections at GA State University Library soon will sponsor a film screening to promote Women’s History Month. The example that was included in the CJCLS discussion list item is GA Tech’s CeLIBration, which gets freshmen in the library to particpate in very un-library activities, like speed dating, retro (board) games, and DDR.
Does anyone have any experience assessing this type of activity? How can we determine the impact of these events on users’ perceptions and use of the library?
The use of the Library as a hospitality space for the powers that be at the school can have some sidestream visibility benefits to the Library.
To my experience, folks coming into the Library for a non-Library purpose see only their purpose – be it meeting, viewing a presentation or attending a gathering. The folks who arrive early, and alone, might browse if they don’t fall back on checking their cellphone or leaving to wait for more people to arrive. If the librarian knows the person that arrives early, then an opportunity is available to “talk up” the Library or show whatever may interest that particular person.
Assessment material would be any followup from the visitors that isn’t related to their original purpose in coming into the Library – ie a request for information, a book, “where you got that poster from”, etc.
Tim Wojcik
I can only imagine a few ways to assess the use of the library as a hospitality space (I like Tim’s description):
1. anecdotally- hope that those who come for the non-library event mention this on a later visit and that the library employee who hears it will record it, etc. Inefficient and not good for gathering enough data to measure impact.
2. survey the non-library event attendees about their perceptions of the library at the end of the event. “Now that you’ve visited the library, would you return for research purposes?” or whatever. Value?
3. focus group of non-library event attendees. Invite them at the end of the event or hold the focus group at the end of the event. I don’t think this would really work.
This made me think about my freshman college year. Having come from a town of 2,500 with the public library in an old house, I was very intimidated about going to the 7-story campus library—so intimidated that I didn’t go for months until I had to do some scavenger hunt assignment. If I’d walked into the library to attend a non-library event and seen what it was like, I would’ve been inclined to go back much sooner than I did. There’s one anecdote for you.
Jennifer
This is an interesting topic. Ingram Library recently formed a friends of the library group and we are starting to host events either during or after library hours. It would be nice to have some type of assessment in terms of how many people who attend an event come back later to use our resources or if it has changed their perception of the library.
Diane
So, how can we determine who comes back after attending an event at the library and whether their library perceptions changed? Any ideas? –Jennifer
ALA in DC hosted illuminating speakers as part of their “Marketing Library to GenX” workshop, The two in particular that caught my interes, Ciara may have seen mention of them on the CJCLS List, are from St. Charles Public Library in IL and started an activities group targetd at twenties and thrities called T N T which very cleverly entices library awareness through non-trad events. If I recall correctly, they gauge/assesse at the end of each event via a brief survey insturment- which seems sensible to me. Check out their site for some great ideas.
As far as what I’ve any experience with (here in a academic setting), our local PR team conceptualized an ad campaign for the school paper that was based on an ad campaign originally run by a local establishment (bar) in the community’s free weekly paper that featured a picture of a patron with the tag line “Guess Who I Saw?” The bar used it to garner interest by showing that interesting/popular/good looking people were there, we did the same to show interesting/popular/good looking people were at the library. And, while I can’t actually address the efficacy of the campaign, the ad space did bring the library to people’s attention. That same season we ran ads (featuring pics of student workers) to promote our feature film/DVD holdings; again, non-trad services to entice the populus…
Finally, to address the value of the survey insturment I would suggest that the brief survey may be the easiest and most concrete way to assess non-trad event attendees change of perception (or lack thereof) regarding the library. The keyword is “brief” though, as anything too involved may sour participant’s perception at just the wrong time.
Jewell
I am a librarian at the Cartersville campus library of Georgia Highlands College and we are “teaming up” with the coordinator of student life and the president of the Book Club (started by a student) to host non library-related events in the library.
For instance, in celebration of National Poetry Month and Jazz Appreciation Month the Book Club has organized a poetry reading for April 23rd and the event will be held in the library. We have contacted the advisor in charge of production of The Old Red Kimono (http://www.highlands.edu/ork/) ,GHC’s literary magazine, and asked her to contact the students who have poetry in the magazine to see if they want to read their stuff. We are hoping to advertise the event in the upcoming issue of ORK and we also plan to advertise the event through student life. There will be food and jazz music playing (hopefully by a student band) and we have set up a display of poetry and jazz books at the entrance of the library just in case the students want to browse afterwards (or beforehand). The library’s involvement is minimal—our role is to provide a venue for the reading—but these events get students to come into our library… and hopefully come back.
Jennifer I like your #2 assessment idea.
Stacy
Hi Diane!
It would also be cool to have a coffee shop in the library. That would certainly draw students–Stacy
GA State opened its coffee shop a few months ago. It would be easy enough to survey students in the coffee shop about how/whether it’s changed their perception of the library
Jennifer
Focus Groups
(From the AIG wiki; originally posted April 2008)
Let’s discuss focus groups this month! Some questions to get you started:
- How did you successfully recruit focus group participants?
- What incentives did you use, if any, and what seemed to work best?
- How did you decide what questions to ask? (Complaints from patrons? Based on survey results? Other?)
- Academic libraries, did you need approval from your IRB to conduct the focus group?
- Please provide any details that you think others would benefit from knowing: Number of subjects, number of moderators, method(s) of recording discussions, time of day you held the group, reporting results, acting on findings . . .
If you haven’t conducted any focus groups and have questions for those in the group who have, feel free to post your questions.
I conducted a student focus group at the end of March that worked out well. I had 9 subjects and asked them 5 questions. The questions were based on our fall 2007 LibQUAL+ data–mainly repeated comments that we didn’t quite understand or that we wanted validated. I served pizza and snacks, and each participant received a $5 campus debit card at the conclusion of the focus group. The subjects were great; they provided thoughtful answers and seemed engaged the whole time. I moderated the session and used a digital audio recorder and a video camera that also captured audio (which was good, because the audio recorder cut out quite a bit, and I would’ve missed much of the audio without the camera’s audio.)
At my insitution (Georgia State), we do have to get IRB approval to conduct a focus group. I requested a waiver of consent, since all reports of the focus group’s discussions would not have subjects’ names associated with them. Subjects still had to give consent according to the informed consent document, but they were allowed to give verbal consent. I told the IRB that I planned to do more focus groups in the future and did I have to submit an application for each one. I’m allowed to edit my current application and revise the date, time, and questions to be asked.
In the past, successful recruitment has been a problem. This time I targeted organized student groups and emailed members. I asked those interested to tell me when they were available and scheduled the focus group at the most available time. This meant editing my IRB protocol for date and time, but it wasn’t a big deal. I will definitely use this approach again.
I look forward to reading others’ comments–I’m especially interested in recruiting techniques!
Jennifer
Some focus group resources:
Becher, M. L., & Flug, J. L. (2005). Using student focus groups to inform library planning and marketing. College & Undergraduate Libraries, 12(1/2), 1-18.
Bloor, M. (2001). Focus groups in social research. Thousand Oaks, CA: SAGE Publications.
Greenbaum, T. L. (1998). The handbook for focus group research. Thousand Oaks, CA: SAGE Publications.
Greenbaum, T. L. (2000). Moderating focus groups: a practical guide for group facilitation. Thousand Oaks, CA: SAGE Publications.
Hughes-Hassel, S., & Bishop, K. (2004). Using focus group interviews to improve library services for youth. Teacher Librarian, 32(1), 8-12. Available through ProQuest Research Library.
Lee, D. (2005). Can you hear me now?: Using focus groups to enhance marketing research. Library Administration & Management, 19(2), 100-101. Available through ProQuest Research Library and ABI Inform Complete.
Walden, G. (2006). Focus group interviewing in the library literature: A selective annotated bibliography 1996-2005. Reference Services Review, 34(2), 222-241. Available through Emerald and ProQuest Research Library.
Using focus groups for library renovations
Some members of this committee have recently renovated their library. Before renovating your library did you create a focus group or groups to obtain input from students or other library users about the changes they wanted incorporated into the renovation? If so can you share your success stories and also how did you attract and select people to join the focus group? What were some of the drawbacks of using focus groups? The reason I ask is because we are to going to renovate our library and it seem appropriate we use focus groups to help us determine some of the changes we should incorporate into the renovation.
Thanks,
Diane
Hi Diane,
We did a small renovation last year and did not use focus groups for that project, however for the new building (scheduled to break ground this summer) we formed the Library Planning Group. It is made up of “key” players such as the Director of Student Success, representatives from Educational Technology, library faculty members etc. We also included students and faculty members (volunteers) that had an interest in the project. We conducted “brain storming” sessions as the full group. In these sessions we asked questions of all members and also just allowed time for everyone to say what is important to them in a library building. We also divided up the members and did small groups focusing on interior, exterior and technology for the building, with library faculty interviewing the participants. Group members also went on “field trips” to several schools and brought that information back to the table. The Library Planning Group is very much like a focus group, we are engaging our users to find out what they want and need. The group produced a Planning Document that was given to the project manager and architects, both of which have found it to be very helpful. From our experience I would think that focus groups would be extremely helpful in planning a renovation. These folks do know what works in the real world of academics (something an architect might not always understand) and can definitely help make the library better. The only drawback I can think of is that a lot of folks have a lot of great ideas but you can’t give everyone everything, so some participants may be a bit disappointed in the end if their “great idea” wasn’t put into place (usually because of space, budget or practicality.) Good luck!
Joy
Hi Joy,
Thanks for sharing your experiences. There has been talk of a new library in our future and I like your ideas about how to get the students, the faculty and the staff involved in the planning process.
Stacy
Bibliographic/Library Instruction Assessment
(From the AIG blog; originally posted February 2008)
February 2008 Discussion Topic: Bibliographic/Library Instruction Assessment
1. What are your ideas for assessment of bibliographic or library instruction? How can we get away from quantitative assessment—”I taught 27 classes during fall semester”—and incorporate meaningful, qualitative assessment methods that measure instructional effectiveness against factors such as how students are using what they’ve learned?
– I’ve successfully used a web-based evaluation form. Since the students had acces to computers during the session (to follow along) it was easy to direct them to the survey at the end. It was run by survey monkey and it gives instant stats, as soon as the survey is complete. I’ve also thought of using this same survey (it linked through the college blog) as a kind of quiz at the end – not graded – to see what the students know or have learned. If you wanted to go assessment crazy, you could have a pre and post test of just a few questions.
Here is a link that came through on assessment that showed a comprehensive report on an extended assessment project. Might be a nice template for someone looking to evaluate their BI sessions: assessment report
-Posted by Jennifer: In a previous position, I worked with a faculty member who pre- and post-tested her students before brining them for library instruction. Interestingly, she used a general library tutorial that I created (that was in quiz format) as the pre- and post-test. Her students took the quiz prior to library instruction and then again after library instruction. Their test scores did improve, but (to me) the most worthwhile outcome was not assessment-related: They seemed to pay more attention during library instruction, perhaps because they’d already read about many of the concepts I was presenting.
– Our plan (not there yet) is to use a Blackboard “Library” course page by which students would transmit to the BI librarian citations gathered (after a BI session) for their assignment. The librarian would then offer feedback to the student on how relevant the cites were, where else they might want to look, etc. In fact, we might ask the student to send us their bibliography gathered via RefWorks – another tool we teach in BIs. Faculty here love RefWorks, so it likely would be easy to pull from some test classes.
-Posted by Jennifer: This sounds like a great idea. Will the citations be annotated? And what a great way to get students to use RefWorks!
Diane Fulkerson passed along this link to Assessments of Information Literacy, a directory of online assessment tools. You might want to take a look before you create your own pre- or post-test
-Posted by Diane: One issue that came up in a session I attended about assessment at ALA mid-winter was making sure we are doing assessment instead of evaluation. The focus of our assessments should be on what the students come away with from the session and not an evaluation of our teaching style. Several people in the discussion group mentioned they often e-mail the students after the BI session and ask them to write a “one minute paper.” It serves two purposes one it allows more time for the BI session and second it provides feedback to the instructor on what the students learned during the session. I have gone to the one minute paper for my BI session and ask students to include a least one thing I can do to make the class better.
2. What are your ideas for assessing how well library employees are conducting library instruction?
-Posted by Jennifer: Does anyone use peer evaluation or peer review? I attended a GaCOMO session given by some UGA librarians in which they talked about their peer evaluation program. Just wondering, because some people seem very turned off/threatened by this practice.
-Posted by Diane: I have heard of other schools going to peer evaluation and if you think about it is really that different from submitting an article to a peer reviewed journal? Okay, so you are standing in front of your peers while teaching, but it is also a way for us to improve our teaching skills since we can probably learn from each other.
GLA Assessment Interest Group 