Collecting data
Data collection
You need to collect data on your indicator (or indicators) in order to answer your question. Data collection can be resource-intensive, so choose your data sources and collection strategy wisely. Ask the following questions (adapted from page 53 of the CAISE Principal Investigator’s Guide) to guide you through the process:
- What kind of information do you need? Do you need to know about opinions or attitudes? Knowledge levels? Details about implementation?
- What is the best method to obtain that information? You have many choices for data collection—surveys, focus groups, interviews, and more.
- What is the best strategy for collecting that information via that method for this project? From whom do you need to collect information? Where is the best place to reach them via the method(s) you’ve chosen? When should this happen? If there is a facilitator, who should that be?
Quantitative or qualitative?
In the simplest terms, quantitative data means numbers. Traditional library assessments tend to be quantitative: the number of items borrowed or the number of program attendees.
Qualitative data is anything other than numbers — usually written or spoken words, but images, sounds, or behaviors can also be collected. Qualitative data, collected by listening to or observing people, can be more useful for understanding individual experiences, thoughts, and attitudes.
An assessment can collect both qualitative and quantitative data. For instance, a survey can have quantitative and qualitative questions, and an and an observation may involve both counting the number of times something happens (quantitative) and listening to someone’s comments (qualitative). You can also quantitatively analyze qualitative data (see the next section on analyzing data).
Charities [often throw] out a bunch a numbers, wrongly assuming that hard data is all funders need to see as proof of impact. However, without a story – without that compelling dialogue that charities can share – the numbers lose value as impact indicators… Nonprofits that are able to present their impact with a strong story will often be more successful at attaining grants than those who only present their numbers.
— Measuring Impact: How Small and Medium-Sized Nonprofits Can Benefit From Effectively Measuring Their Impact
Data collection tools
The data collection process usually involves tools (also called instruments) — for example, surveys, interview guides, or observation records — that directly capture the indicators that will tell you if you’ve achieved your desired outcomes. If possible, it can be helpful to reuse a tool that someone else has already created (you can find examples in the Module Resources). However, connected learning programs are often unique, with one-of-a-kind elements that may need one-of-a-kind assessments. You can modify someone else’s tool to fit your own assessment needs, or even create one that is entirely original.
Pilot-testing the process
Before diving headlong into data collection, test the entire process. For a small and simple assessment, this could simply be asking a co-worker to review your plan. A larger effort should be tested more thoroughly, ideally with a sample that resembles who or what you will be assessing. A large survey, for example, would benefit from being tried out with a handful of teens first. As you conduct your pilot tests, consider the following questions:
- Does the instrument you are using give you the kind of data you need to answer your question? Do you need to make changes to wording, format, time or place?
- Are there ways you can automate or simplify data collection? For instance, you could provide computers at the library for people to fill out a survey online, rather than handing them a sheet of paper to fill out .
- Do you have the capacity to handle the data collection you’re planning? If an interview takes much longer than expected, for example, you may need to either shorten your interview instrument, add more staff to the effort, or increase the amount of time your assessment will run.
- Do you need to collect any baseline data so you can make before and after comparisons?
- Is there a way that data collection can happen seamlessly (for instance, data is collected from teens as an integrated part of the program), and not intrusive (such as taking time out from the program to have the teens fill out a feedback form)?
Privacy and consent
Be sure to respect your participants’ privacy and autonomy as you are collecting data (and any time outside stakeholders are involved). This is particularly important when you are interacting with minors. Project Outcome is a good starting place for learning about issues of privacy and consent in library assessment (requires free registration).
Collection methods
Analyzing artifacts
You can gain a great deal of insight from the items learners create through your initiatives, whether they are musical performances, poetry, or 3-D models. Knowledge, skill, development, attitudes, behaviors—these can all be observed in learners’ creations. You can also ask students to write about the learning experience or their creative process, and analyze their responses.
Good for:
- Demonstrating achievement first-hand
Potential drawbacks:
- There can be a high level of subjectivity; be careful not to read too much into a creation (particularly a non-textual one) without input from the creator
- People may find the process intrusive
Passive feedback
Collecting data through active feedback means that you, the researcher, take action to get data from particpants — such as making observations, holding focus groups, or asking for feedback after a program. Passive feedback means that the data collection happens in the background, without direct interaction from the researcher — you can think of it as “always-on” data collection.1 Along with on-demand surveys and questionnaires, you can collect passive feedback through methods such as comment cards or “talkback boards”.
Good for:
- Collecting data unobtrusively
Potential drawbacks:
- Responses may come only from people with extreme or passionate opinions—not from people “in the middle”
Observations
Using an observation guide or protocol, you can collect data by simply observing what is happening during a program or in a space and taking detailed “field notes.” This can be done unobtrusively in the background, or in a more engaged manner. If observing an individual directly (perhaps as they use a resource or do an activity) you can ask them to use a “think-aloud” strategy to help you understand why they do what they do.
Good for:
- Uncovering unexpected outcomes
- Directly observing what is happening, not just relying on participants’ self-reports
Potential drawbacks:
- People often behave differently when they know they are being observed
- Effective observations require a great deal of attention and focus
- People may not like being watched
Testing knowledge or skills
Formal tests of knowledge or skills feel too much like school for a connected learning library setting, but you can find small, creative ways to “test” a teen’s progress without making it feel like an exam, such as integrating a competitive quiz towards the end of a program or presenting challenges for participants to work on.
Good for:
- Directly assessing what a teen knows or can do
Potential drawbacks:
- They can feel too formal and school-like
- They can feel like too much pressure, especially for youth who are hanging out or messing around
Worksheet #3: Design a Talkback Board
Talkback boards can be a quick and easy way to gather data about the people who visit your library. Look through this Talkback Board Repository from the Connected Learning Lab, and the Talkback Board Instructions from Impact Libraries. Now, let’s design a talkback board that you can put up in your library or teen space in the next few days.
- What question do you have about your community, your patrons, or programming? Don’t start with the question you’ll put on the talkback board — start with a bigger question about something you want to know. Example: Are youth building relationships through this program?
- Will it be easier to answer your question with an open-ended prompt (teens write their own responses), a close-ended prompt (teens can place stickers underneath the answer they choose), or a mix of both? Your answer will not only guide the way you write the prompt, but what supplies you have on hand for participants (e.g., markers for writing or just stickers for selecting an answer).
- Now, word the prompt (either as an open-ended or close-ended question) in a way that is engaging and will elicit the responses you need.
- Create the talkback board, and present it to your visitors. You’ll analyze the data in the next worksheet!