26
Matthew DeCarlo
Chapter Outline
- Ethical responsibility and cultural respectfulness (8 minute read)
- Critical considerations (5 minute read)
- Informing your dissemination plan (11 minute read)
- Final product taking shape (10 minute read)
Content warning: Examples in this chapter contain references to research as a potential tool to stigmatize or oppress vulnerable groups, mistreatment and inequalities experienced by Native American tribes, sibling relationships, caregiving, child welfare, criminal justice and recidivism, first generation college students, Covid-19, school culture and race, health (in)equity, physical and sensory abilities, and transgender youth.
Your sweat and hard work has paid off! You’ve planned your study, collected your data, and completed your analysis. But alas, no rest for the weary student researcher. Now you need to share your findings. As researchers, we generally have some ideas where and with whom we desire to share our findings, but these plans may evolve and change during our research process. Communicating our findings with a broader audience is a critical step in the research process, so make sure not to treat this like an afterthought. Remember, research is about making a contribution to collective knowledge-building in the area of study that you are interested in. Indeed, research is of no value if there is no audience to receive it. You worked hard…get those findings out there!
In planning for this phase of research, we can consider a variety of methods for sharing our study findings. Among other options, we may choose to write our findings up as an article in a professional journal, provide a report to an organization, give testimony to a legislative group, or create a presentation for a community event. We will explore these options in a bit more detail below in section 21.4 where we talk more about different types of qualitative research products. We also want to think about our intended audience.
Exercises
For your research, answer these two key questions as you are planning for dissemination:
- Who are you targeting to communicate your findings to? In other words, who needs to hear the results of your study?
- What do you hope your audience will take away after learning about your study?
Of course, we can’t know how our findings will be received, but we will want to work to anticipate what expectations our target audience might have and use this information to tailor our message in a way that is clear, thorough and consistent with our data (keep it honest!). We will further discuss different audiences you might want to reach with your research and a few considerations for each type of audience that may shape your dissemination plan. Before we tackle these sections, we are going to take some time to think through ethical and critical considerations that should be at the forefront of your mind as you plan for the dissemination of your research. Remember, your research is NOT a static, inanimate object. It is a reflection of your participants. Furthermore, others will review, interpret and potentially form opinions based on your research Your research has real implications for individuals, groups, communities, agencies, programs, etc.
21.1 Ethical responsibility and cultural respectfulness
Learning Objectives
Learners will be able to…
- Identify key ethical considerations in developing their qualitative research dissemination plan
- Conceptualize how research dissemination may impact diverse groups, both presently and into the future
Have you ever been misrepresented or portrayed in a negative light? It doesn’t feel good. It especially doesn’t feel good when the person portraying us has power, control and influence. While you might not feel powerful, research can be a powerful tool, and can be used and abused for many ends. Once research is out in the world, it is largely out of our control, so we need to approach dissemination with care. Be thoughtful about how you represent your work and take time to think through the potential implications it may have, both intended and unintended, for the people it represents.
As alluded to in the paragraph above, research comes with hefty responsibilities. You aren’t off the hook if you are conducting quantitative research. While quantitative research deals with numbers, these numbers still represent people and their relationships to social problems. However, with qualitative research, we are often dealing with a smaller sample and trying to learn more from them. As such, our job often carries additional weight as we think about how we will represent our findings and the people they reflect. Furthermore, we probably hope that our research has an impact; that in some way, it leads to change around some issue. This is especially true as social work researchers. Our research often deals with oppressed groups, social problems, and inequality. However, it’s hard to predict the implications that our research may have. This suggests that we need to be especially thoughtful about how we present our research to others.
Two of the core values of social work involve respecting the inherent dignity and worth of each person, practicing with integrity, and behaving in a trustworthy manner.[1] As social work researchers, to uphold these values, we need to consider how we are representing the people we are researching. Our work needs to honestly and accurately reflect our findings, but it also needs to be sensitive and respectful to the people it represents. In Chapter 6 we discussed research ethics and introduced the concept of beneficence or the idea that research needs to support the welfare of participants. Beneficence is particularly important as we think about our findings becoming public and how the public will receive, interpret and use this information. Thus, both as social workers and researchers, we need to be conscientious of how dissemination of our findings takes place.
Exercises
As you think about the people in your sample and the communities or groups to which they belong, consider some of these questions:
- How are participants being portrayed in my research?
- What characteristics or findings are being shared or highlighted in my research that may directly or indirectly be associated with participants?
- Have the groups that I am researching been stigmatized, stereotyped, and/or misrepresented in the past? If so, how does my research potentially reinforce or challenge these representations?
- How might my research be perceived or interpreted by members of the community or group it represents?
- In what ways does my research honor the dignity and worth of participants?
Qualitative research often has a voyeuristic quality to it, as we are seeking a window into participants’ lives by exploring their experiences, beliefs, and values. As qualitative researchers, we have a role as stewards or caretakers of data. We need to be mindful of how data are gathered, maintained, and most germane to our conversation here, how data are used. We need to craft research products that honor and respect individual participants (micro), our collective sample as a whole (meso), and the communities that our research may represent (macro).
As we prepare to disseminate our findings, our ethical responsibilities as researchers also involve honoring the commitments we have made during the research process. We need to think back to our early phases of the research process, including our initial conversations with research partners and other stakeholders who helped us to coordinate our research activities. If we made any promises along the way about how the findings would be presented or used, we need to uphold them here. Additionally, we need to abide by what we committed to in our informed consent. Part of our informed consent involves letting participants know how findings may be used. We need to present our findings according to these commitments. We of course also have a commitment to represent our research honestly.
As an extension of our ethical responsibilities as researchers, we need to consider the impact that our findings may have, as well as our need to be socially conscientious researchers. As scouts, we were taught to leave our campsite in a better state than when we arrived. I think it is helpful to think of research in these terms. Think about the group(s) that may be represented by your research; what impact might your findings have for the lives of members of this group? Will it leave their lives in a better state than before you conducted your research? As a responsible researcher, you need to be thoughtful, aware and realistic about how your research findings might be interpreted and used by others. As social workers, while we hope that findings will be used to improve the lives of our clients, we can’t ignore that findings can also be used to further oppress or stigmatize vulnerable groups; research is not apolitical and we should not be naive about this. It is worth mentioning the concept of sustainable research here. Sustainable research involves conducting research projects that have a long-term, sustainable impact for the social groups we work with. As researchers, this means that we need to actively plan for how our research will continue to benefit the communities we work with into the future. This can be supported by staying involved with these communities, routinely checking-in and seeking input from community members, and making sure to share our findings in ways that community members can access, understand, and utilize them. Nate Olson provides a very inspiring Ted Talk about the importance of building resilient communities. As you consider your research project, think about it in these terms.
Key Takeaways
- As you think about how best to share your qualitative findings, remember that these findings represent people. As such, we have a responsibility as social work researchers to ensure that our findings are presented in honest, respectful, and culturally sensitive ways.
- Since this phase of research deals with how we are going to share our findings with the public, we need to actively consider the potential implications of our research and how it may be interpreted and used.
Exercises
- Is your work, in some way, helping to contribute to a resilient and sustainable community? It may not be a big tangible project as described in Olson’s Ted Talk, but is it providing a resource for change and growth to a group of people, either directly or indirectly? Does it promote sustainability amongst the social networks that might be impacted by the research you are conducting?
21.2 Critical considerations
Learning Objectives
Learners will be able to…
- Identify how issues of power and control are present in the dissemination of qualitative research findings
- Begin to examine and account for their own role in the qualitative research process, and address this in their findings
This is the part of our research that is shared with the public and because of this, issues like reciprocity, ownership, and transparency are relevant. We need to think about who will have access to the tangible products of our research and how that research will get used. As researchers, we likely benefit directly from research products; perhaps it helps us to advance our career, obtain a good grade, or secure funding. Our research participants often benefit indirectly by advancing knowledge about a topic that may be relevant or important to them, but often don’t experience the same direct tangible benefits that we do. However, a participatory perspective challenges us to involve community members from the outset in discussions about what changes would be most meaningful to their communities and what research products would be most helpful in accomplishing those changes. This is especially important as it relates to the role of research as a tool to support empowerment.
Ownership of research products is also important as an issue of power and control. We will discuss a range of venues for presenting your qualitative research, some of which are more amenable to shared ownership than others. For instance, if you are publishing your findings in an academic journal, you will need to sign an agreement with that publisher about how the information in that article can be used and who has access to it. Similarly, if you are presenting findings at a national conference, travel and other conference-related expenses and requirements may make access to these research products prohibitive. In these instances, the researcher and the organization(s) they negotiate with (e.g. the publishing company, the conference organizing body) share control. However, disseminating qualitative findings in a public space, public record, or community-owned resource means that more equitable ownership might be negotiated. An equitable or reciprocal arrangement might not always be able to be reached, however. Transparency about who owns the products of research is important if you are working with community partners. To support this, establishing a Memorandum Of Understanding (MOU) or Memorandum of Agreement (MOA) early in the research process is important. This document should clearly articulate roles, responsibilities, and a number of other details, such as ownership of research products between the researcher and the partnering group(s).
Resources for learning more about MOUs and MOAs
Center for Community Health and Development, University of Kansas. (n.d.). Section 9. Understanding and writing contracts and memoranda of agreement.
Collaborative Center for Health Equity, University of Wisconson Madison. (n.d.). Standard agreement for research with community organizations.
Office of Research, UC Davis. (n.d.). Research MOUs.
Office of Research, The University of Texas at Dallas. (n.d.). Types of agreements.
In our discussion about qualitative research, we have also frequently identified the need for the qualitative researcher to account for their role throughout the research process. Part of this accounting can specifically apply to qualitative research products. This is our opportunity to demonstrate to our audience that we have been reflective throughout the course of the study and how this has influenced the work we did. Some qualitative research studies include a positionality statement within the final product. This is often toward the beginning of the report or the presentation and includes information about the researcher(s)’s identity and worldview, particularly details relevant to the topic being studied. This can include why you are invested in the study, what experiences have shaped how you have come to think about the topic, and any positions or assumptions you make with respect to the topic. This is another way to encourage transparency. It can also be a means of relegating or at least acknowledging some of our power in the research process, as it can provide one modest way for us, as the researcher, to be a bit more exposed or vulnerable, although this is a far cry from making the risks of research equitable between the researcher and the researched. However, the positionality statement can be a place to integrate our identities, who we are as an individual, a researcher, and a social work practitioner. Granted, for some of us that might be volumes, but we need to condense this down to a brief but informative statement—don’t let it eclipse the research! It should just be enough to inform the audience and allow them to draw their own conclusions about who is telling the story of this research and how well they can be trusted. This student provides a helpful discussion of the positionality statement that she developed for her study. Reviewing your reflexive journal (discussed in Chapter 20 as a tool to enhance qualitative rigor) can help in identifying underlying assumptions and positions you might have grounded in your reactions throughout the research process. These insights can be integrated into your positionality statement. Please take a few minutes to watch this informative video of a student further explaining what a positionality statement is and providing a good example of one.
Key Takeaways
- The products of qualitative research often benefit the researcher disproportionately when compared to research participants or the communities they represent. Whenever possible, we can seek out ways to disseminate research in ways that addresses this imbalance and supports more tangible and direct benefits to community members.
- Openly positioning ourselves in our dissemination plans can be an important way for qualitative researchers to be transparent and account for our role.
21.3 Informing your dissemination plan
Learning Objectives
Learners will be able to…
- Appraise important dimensions of planning that will inform their research dissemination plan, including: audience, purpose, context and content
- Apply this appraisal to key decisions they will need to make when designing their qualitative research product(s)
This section will offer you a general overview of points to consider as you form the dissemination plan for your research. We will start with considerations regarding your audience, then turn our attention to the purpose of your research, and finally consider the importance of attending to both content and context as you plan for your final research product(s).
Audience
Perhaps the most important consideration you have as you plan how to present your work is your audience. Research is a product that is meant to be consumed, and because of this, we need to be conscious of our consumers. We will speak more extensively about knowing your audience in Chapter 24, devoted to both sharing and consuming research. Regardless of who your audience is (e.g. community members, classmates, research colleagues, practicing social workers, state legislator), there will be common elements that will be important to convey. While the way you present them will vary greatly according to who is listening, Table 21.1 offers a brief review of the elements that you will want your audience to leave with.
| Element | Purpose |
| Aim | What did my research aim to accomplish and why is it important |
| Process | How did I go about conducting my research and what did I do to ensure quality |
| Concepts | What are the main ideas that someone needs to know to make sense of my topic and how are they integrated into my research plan |
| Findings | What were my results and under what circumstances are they valid |
| Connection | How are my findings connected to what else we know about this topic and why are they important |
Once we determine who our audience is, we can further tailor our dissemination plan to that specific group. Of course, we may be presenting our findings in more than one venue, and in that case, we will have multiple plans that will meet the needs of each specific audience.
It’s a good idea to pitch your plan first. However you plan to present your findings, you will want to have someone preview before you share with a wider audience. Ideally, whoever previews will be a person from your target audience or at least someone who knows them well. Getting feedback can go a long way in helping us with the clarity with which we convey our ideas and the impact they have on our audience. This might involve giving a practice speech, having someone review your article or report, or practice discussing your research one-on-one, as you would with a poster presentation. Let’s talk about some specific audiences that you may be targeting and their unique needs or expectations.
Below I will go through some brief considerations for each of these different audiences. I have tried to focus this discussion on elements that are relevant specific to qualitative studies since we do revisit this topic in Chapter 24.
Research community
When presenting your findings to an academic audience or other research-related community, it is probably safe to a make a few assumptions. This audience is likely to have a general understanding of the research process and what it entails. For this reason, you will have to do less explaining of research-related terms and concepts. However, compared to other audiences, you will probably have to provide a bit more detail about what steps you took in your research process, especially as they relate to qualitative rigor, because this group will want to know about how your research was carried out and how you arrived at your decisions throughout the research process. Additionally, you will want to make a clear connection between which qualitative design you chose and your research question; a methodological justification. Researchers will also want to have a good idea about how your study fits within the wider body of scientific knowledge that it is related to and what future studies you feel are needed based on your findings. You are likely to encounter this audience if you are disseminating through a peer-reviewed journal article, presenting at a research conference, or giving an invited talk in an academic setting.
Professional community
We often find ourselves presenting our research to other professionals, such as social workers in the field. While this group may have some working knowledge of research, they are likely to be much more focused on how your research is related to the work they do and the clients they serve. While you will need to convey your design accurately, this audience is most likely to be invested in what you learned and what it means (especially for practice). You will want to set the stage for the discussion by doing a good job expressing your connection to and passion for the topic (a positionality statement might be particularly helpful here), what we know about the issue, and why it is important to their professional lives. You will want to give good contextual information for your qualitative findings so that practitioners can know if these findings might apply to people they work with. Also, as since social work practitioners generally place emphasis on person-centered practice, hearing the direct words of participants (quotes) whenever possible, is likely to be impactful as we present qualitative results. Where academics and researchers will want to know about implications for future research, professionals will want to know about implications for how this information could help transform services in the future or understand the clients they serve.
Lay community
The lay community are people who don’t necessarily have specialized training or knowledge of the subject, but may be interested or invested for some other reason; perhaps the issue you are studying affects them or a loved one. Since this is the general public, you should expect to spend the most time explaining scientific knowledge and research processes and terminology in accessible terms. Furthermore, you will want to invest some time establishing a personal connection to the topic (like I talked about for the professional community). They will likely want to know why you are interested and why you are a credible source for this information. While this group may not be experts on research, as potential members of the group(s) that you may be researching, you do want to remember that they are experts in their own community. As such, you will want to be especially mindful of approaching how you present findings with a sense of cultural humility (although hopefully you have this in mind across all audiences). It will be good to discuss what steps you took to ensure that your findings accurately reflect what participants shared with you (rigor). You will want to be most clear with this group about what they should take away, without overstating your findings.
Regardless of who your audience is, remember that you are an ambassador. You may represent a topic, a population, an organization, or the whole institution of research, or any combination of these. Make sure to present your findings honestly, ethically, and clearly. Furthermore, I’m assuming that the research you are conducting is important because you have spent a lot of time and energy to arrive at your findings. Make sure that this importance comes through in your dissemination. Tell a compelling story with your research!
Exercises
Who needs to hear the message of your qualitative research?
What will they know in advance?
- Example. If you are presenting your research about caregiver fatigue to a caregiver support group, you won’t need to spend time describing the role of caregivers because your audience will have lived experience.
What terms will they likely already be familiar with and which ones will require additional explanation?
- Example. If you are presenting your research findings to a group of academics, you wouldn’t have to explain what a sampling frame is, but if you are sharing it with a group of community members from a local housing coalition, you will need to help them understand what this is (or maybe use a phrase that is more meaningful to them).
What will your audience be looking for?
- Example. If you are speaking to a group of child welfare workers about your study examining trauma-informed communication strategies, they are probably going to want to know how these strategies might impact the work that they do.
What is most impactful or valued by your audience?
- Example. If you are sharing your findings at a meeting with a council member, it may be especially meaningful to share direct quotes from constituents.
Purpose
Being clear about the purpose of your research from the outset is immeasurably helpful. What are you hoping to accomplish with your study? We can certainly look to the overarching purpose of qualitative research, that being to develop/expand/challenge/explore understanding of some topic. But, what are you specifically attempting to accomplish with your study? Two of the main reasons we conduct research are to raise awareness about a topic and to create change around some issue. Let’s say you are conducting a study to better understand the experience of recidivism in the criminal justice system. This is an example of a study whose main purpose is to better understand and raise awareness around a particular social phenomenon (recidivism). On the other hand, you could also conduct a study that examines the use of strengths-based strategies by probation officers to reduce recidivism. This would fall into the category of research promoting a specific change (the use of strengths-based strategies among probation officers). I would wager that your research topic falls into one of these two very broad categories. If this is the case, how would you answer the corresponding questions below?
Exercises
Are you seeking to raise awareness of a particular issue with your research? If so,
- Whose awareness needs raising?
- What will “speak” most effectively to this group?
- How can you frame your research so that it has the most impact?
Are you seeking to create a specific change with your research? If so,
- What will that change look like?
- How can your research best support that change occurring?
- Who has the power to create that change and what will be most compelling in reaching them?
How you answer these questions will help to inform your dissemination plan. For instance, your dissemination plan will likely look very different if you are trying to persuade a group of legislators to pass a bill versus trying to share a new model or theory with academic colleagues. Considering your purposes will help you to convey the message of your research most effectively and efficiently. We invest a lot of ourselves in our research, so make sure to keep your sights focused on what you hope to accomplish with it!
Content and context
As a reminder, qualitative research often has a dual responsibility for conveying both content and context. You can think of content as the actual data that is shared with us or that we obtain, while context is the circumstances under which that data sharing occurs. Content conveys the message and context provides us the clues with which we can decode and make sense of that message.
While quantitative research may provide some contextual information, especially in regards to describing its sample, it rarely receives as much attention or detail as it does in qualitative studies. Because of this, you will want to plan for how you will attend to both the content and context of your study in planning for your dissemination.
Key Takeaways
- Research is an intentional act; you are trying to accomplish something with it. To be successful, you need to approach dissemination planfully.
- Planning the most effective way of sharing our qualitative findings requires looking beyond what is convenient or even conventional, and requires us to consider a number of factors, including our audience, the purpose or intent of our research and the nature of both the content and the context that we are trying to convey.
21.4 Final product taking shape
Learning Objectives
Learners will be able to…
- Evaluate the various means of disseminating research and consider their applicability for your research project
- Determine appropriate building blocks for designing your qualitative research product
As we have discussed, qualitative research takes many forms. It should then come as no surprise that qualitative research products also come in many different packages. To help guide you as the final products of your research take shape, we will discuss some of the building blocks or elements that you are likely to include as tools in sharing your qualitative findings. These are the elements that will allow you to flesh out the details of your dissemination plan.
Building blocks
There are many building blocks that are at our disposal as we formulate our qualitative research product(s). Quantitative researchers have charts, graphs, tables, and narrative descriptions of numerical output. These tools allow the quantitative researcher to tell the story of their research with numbers. As qualitative researchers, we are tasked with telling the story of our research findings as well, but our tools look different. While this isn’t an exhaustive list of tools that are at our disposal as qualitative researchers, a number of commonly used elements in sharing qualitative findings are discussed here. Depending on your study design and the type of data you are working with, you may use one or some combination of the building blocks discussed below.
Themes
Themes are a very common element when presenting qualitative research findings. They may be called themes, but they may also go by other names: categories, dimensions, main ideas, etc. Themes offer the qualitative researcher a way to share ideas that emerged from your analysis that were shared by multiple participants or across multiple sources of data. They help us to distill the large amounts of qualitative data that we might be working with into more concise and manageable pieces of information that are more consumable for our audience. When integrating themes into your qualitative research product, you will want to offer your audience: the title of the theme (try to make this as specific/meaningful as possible), a brief description or definition of the theme, any accompanying dimensions or sub-themes that may be relevant, and examples (when appropriate).
Quotes
Quotes offer you the opportunity to share participants’ exact words with your audience. Of course, we can’t only rely on quotes, because we need to knit the information that is shared into one cohesive description of our findings and an endless list of quotes is unlikely to support this. Because of this, you will want to be judicious in selecting your quotes. Choose quotes that can stand on their own, best reflect the sentiment that is being captured by the theme or category of findings that you are discussing, and are likely to speak to and be understood by your audience. Quotes are a great way to help your findings come alive or to give them greater depth and significance. If you are using quotes, be sure to do so in a balanced manner—don’t only use them in some sections but not others, or use a large number to support one theme and only one or two for another. Finally, we often provide some brief demographic information in a parenthetical reference following a quote so our reader knows a little bit about the person who shared the information. This helps to provide some context for the quote.
Kohli and Pizarro (2016)[2] provide a good example of a qualitative study using quotes to exemplify their themes. In their study, they gathered data through short-answer questionnaires and in-depth interviews from racial-justice oriented teachers of Color. Their study explored the experiences and motivations of these teachers and the environments in which they worked. As you might guess, the words of the teacher-participants were especially powerful and the quotes provided in the results section were very informative and important in helping to fulfill the aim of the research study. Take a few minutes to review this article. Note how the authors provide a good amount of detail as to what each of the themes meant and how they used the quotes to demonstrate and support each theme. The quotes help bring the themes to life and anchor the results in the actual words of the participants (suggesting greater trustworthiness in the findings).
Figure 21.1 below offers a more extensive example of a theme being reported along with supporting quotes from a study conducted by Karabanow, Gurman, and Naylor (2012).[3] This study focused on the role of work activities in the lives of “Guatemalan street youth”. One of the important themes had to do with intersection of work and identity for this group. In this example, brief quotes are used within the body of the description of the theme, and also longer quotes (full sentence(s)) to demonstrate important aspects of the description.
Work-based Identity: Creativity and Self-WorthWork, be it formal or informal, is beneficial for street youth not only for its financial benefits; but it helps youth to develop and rebuild their sense of self, to break away from destructive patterns, and ultimately contributes to any goals of exiting street life. Although many of the participants were aware that society viewed them as “lazy” or “useless,” they tended to see themselves as contributing members of society earning a valid and honest living. One participant said, “Well, a lot of people say, right? ‘The kid doesn’t want to do anything. Lazy kid’ right? And I wouldn’t like for people to say that about me, I’d rather do something so that they don’t say that I’m lazy. I want to be someone important in life.” This youth makes an interesting and important connection in this statement: he intrinsically associates “being someone” with “doing something” – he accepts the work-based identity that characterizes much of contemporary capitalist society. Many of the interviews subtly enforced this idea that in the informal economy, as in the formal economy, “who one is” is largely dependent on “what one does.” This demonstrates two important ideas: that street youth working in the informal sector are surprisingly ‘mainstream’ in their underlying beliefs and ambitions, and that work – be it formal or informal – plays a crucial role in allowing street youth, who have often dealt with trauma, isolation and low self-esteem, to rebuild a sense of self-worth. Many of the youth involved in this study dream of futures that echo traditional ideals: “to have my family all together…to have a home, or rather to have a nice house…to have a good job.” Several explained that this future is unattainable without hard work; many viewed those who “do nothing” as people who “waste their time” and think that “your life isn’t important to you.” On the other hand, those who value their lives and wish to attain a future of “peace and tranquility” must “look for work, that’s what needs to be done to have that future because if God allows it, in the future maybe you can find a partner, form a family and live peacefully.” For these youth, working – be it in the formal or informal sector – is essential to a feeling of “moving forward (seguir adelante).” This movement forward begins with self-esteem. Although the focus of this study was not the troubled pasts of the participants, many alluded to the difficulties they have faced and the various traumas that forced them onto the streets. Several of the youth noted that working was a catalyst in rebuilding positive feelings about oneself: one explained, “[When I’m working,] I feel happy, powerful…Sometimes when I go out to sell, I feel happy.” Another said: For me, when I’m working I feel free because I know that I’m earning my money in an honest way, not stealing right. Because when you’re stealing, you don’t feel free, right? Now when you’re working, you’re free, they can’t arrest you or anything because you’re selling. Now if you’re stealing and everything, you don’t feel free. But when you’re selling you feel free, out of danger. This feeling of being “free” or “powerful” rests on the idea that money is “earned” and not stolen; being able to earn money is associated with being “someone,” with being a valid and contributing member of society. In addition, work helps street youth to break away from destructive patterns. One participant spoke of her experience working full time at a café: For me, working means to be busy, to not just be there….It helps us meet other people, like new people and not to be always in the same scene. Because if you’re not busy, you feel really bored and you might want to, I don’t know, go back to the same thing you were in before…you even forget your problems because you’re keeping busy, you’re talking to other people, people who don’t know you. For this participant, a formal job was beneficial in that it supplied her with a daily routine and allowed her to interact with non-street people – these factors helped to separate her from the destructive lifestyle of the street, and helped her to “move forward.” Although these benefits are indeed most obvious with formal employment, many participants spoke of the positive effects of informal work as well, although to varying degrees. In Guatemala, since the informal economy accounts for over half of the country’s GNP, there is a wide range of under-the-table informal work available. These jobs frequently bring youth out of the street context and, therefore, provide similar benefits to a formal job, as described by the above participant. As to informal work that takes place on the street, such as hawking or car watching, the benefits of work are present, although to a different degree. Even hawking, for example, gives young workers a routine and a chance to interact with non-street people. As one young man continuously emphasized throughout his interview, “work helps you to keep your mind busy, to be in another mind-set, right? To not be thinking the same thing all the time: ‘Oh, drugs, drugs, drugs…’” As explained earlier, the code of the hawking world dictates that vendors cannot sell while high – just like a formal job, hawking helps to distance youth workers from some of their destructive street habits. However, as one participant thoughtfully noted, it is difficult to break these habits when one is still highly embroiled in street culture; “it depended on who was around me because if they were in the same problems as I was, I stopped working and I started doing the same as they did. And if I was surrounded by serious people, then I got my act together.” While certain types of informal work, like cleaning or waitressing, can help youth to distance themselves from destructive patterns, others, such as car watching and selling, may not do enough to separate youth from their peers. While the routine and activity do have positive effects, they often are not sufficient. Among some of the participants, there was the sentiment that informal work could function as a transition stage towards exiting the street; it could “change your life.” One participant said “there are lots of vendors who’ve gotten off the streets, if you make an effort, you go out to sell, you can get off the street. Like myself, when I was selling, I mean working, I got off the street, I went home and I managed to stay there quite a long time.” One might credit this success to several factors: first, the money the seller may have been able to save and accumulate; second, the routine of selling may have helped the seller to break from destructive patterns, such as drug use, and also prepared the seller for the demands of formal sector employment; and, thirdly, selling may have enabled the seller to develop the necessary confidence and sense of self to attempt exiting the street. |
Pictures or videos
If our data collection involves the use of photographs, drawings, videos or other artistic expression of participants or collection of artifacts, we may very well include selections of these in our dissemination of qualitative findings. In fact, if we failed to include these, it would seem a bit inauthentic. For the same reason we include quotes as direct representations of participants’ contributions, it is a good idea to provide direct reference to other visual forms of data that support or demonstrate our findings. We might incorporate narrative descriptions of these elements or quotes from participants that help to interpret their meaning. Integrating pictures and quotes is especially common if we are conducting a study using a Photovoice approach, as we discussed in Chapter 17, where a main goal of the research technique is to bring together participant generated visuals with collaborative interpretation.
Take some time to explore the website linked here. It is the webpage for The Philidelphia Collaborative for Health Equity’s PhotoVoice Exhibit Gallery and offers a good demonstration of research that brings together pictures and text.
Graphic or figure
Qualitative researchers will often create a graphic or figure to visually reflect how the various pieces of your findings come together or relate to each other. Using a visual representation can be especially compelling for people who are visual learners. When you are using a visual representation, you will want to: label all elements clearly; include all the components or themes that are part of your findings; pay close attention to where you place and how you orient each element (as their spatial arrangement carries meaning); and finally, offer a brief but informative explanation that helps your reader to interpret your representation. A special subcategory of visual representation is process. These are especially helpful to lay out a sequential relationship within your findings or a model that has emerged out of your analysis. A process or model will show the ‘flow’ of ideas or knowledge in our findings, the logic of how one concept proceeds to the next and what each step of the model entails.
Noonan and colleagues (2004)[4] conducted a qualitative study that examined the career development of high achieving women with physical and sensory disabilities. Through the analysis of their interviews, they built a model of career development based on these women’s experiences with a figure that helps to conceptually illustrate the model. They place the ‘dyanmic self’ in the center, surrounded by a dotted (permeable) line, with a number of influences outside the line (i.e. family influences, disability impact, career attitudes and behaviors, sociopoltical context, developmental opportunities and social support) and arrows directed inward and outward between each influence and the dynamic self to demonstrate mutual influence/exchange between them. The image is included in the results section of their study and brings together “core categories” and demonstrates how they work together in the emergent theory or how they relate to each other. Because so many of our findings are dynamic, like Noonan and colleagues, showing interaction and exchange between ideas, figures can be especially helpful in conveying this as we share our results.
Composites
Going one step further than the graphic or figure discussed above, qualitative researchers may decide to combine and synthesize findings into one integrated representation. In the case of the graphic or figure, the individual elements still maintain their distinctiveness, but are brought together to reflect how they are related. In a composite however, rather than just showing that they are related (static), the audience actually gets to ‘see’ the elements interacting (dynamic). The integrated and interactive findings of a composite can take many forms. It might be a written narrative, such as a fictionalized case study that reflects of highlights the many aspects that emerged during analysis. It could be a poem, dance, painting or any other performance or medium. Ultimately, a composite offers an audience a meaningful and comprehensive expression of our findings. If you are choosing to utilize a composite, there is an underlying assumption that is conveyed: you are suggesting that the findings of your study are best understood holistically. By discussing each finding individually, they lose some of their potency or significance, so a composite is required to bring them together. As an example of a composite, consider that you are conducting research with a number of First Nations Peoples in Canada. After consulting with a number of Elders and learning about the importance of oral traditions and the significance of storytelling, you collaboratively determine that the best way to disseminate your findings will be to create and share a story as a means of presenting your research findings. The use of composites also assumes that the ‘truths’ revealed in our data can take many forms. The Transgender Youth Project hosted by the Mandala Center for Change, is an example of legislative theatre combining research, artistic expression, and political advocacy and a good example of action-oriented research.
Counts
While you haven’t heard much about numbers in our qualitative chapters, I’m going to break with tradition and speak briefly about them here. For many qualitative projects we do include some numeric information in our final product(s), mostly in the way of counts. Counts usually show up in the way of frequency of demographic characteristics of our sample or characteristics regarding our artifacts, if they aren’t people. These may be included as a table or they may be integrated into the narrative we provide, but in either case, our goal in including this information is to offer the reader information so they can better understand who or what our sample is representing. The other time we sometimes include count information is in respect to the frequency and coverage of the themes or categories that are represented in our data. Frequency information about a theme can help the reader to know how often an idea came up in our analysis, while coverage can help them to know how widely dispersed this idea was (e.g. did nearly everyone mention this, or was it a small group of participants).
Key Takeaways
- There are a wide variety of means by which you can deliver your qualitative research to the public. Choose one that takes into account the various considerations that we have discussed above and also honors the ethical commitments that we outlined early in this chapter.
- Presenting qualitative research requires some amount of creativity. Utilize the building blocks discussed in this chapter to help you consider how to most authentically and effectively convey your message to a wider audience.
Exercises
- What means of delivery will you be choosing for your dissemination plan?
- What building blocks will best convey your qualitative results to your audience?
- National Association of Social Workers. (2017). NASW code of ethics. Retrieved from https://www.socialworkers.org/About/Ethics/Code-of-Ethics/Code-of-Ethics-English ↵
- Kohli, R., & Pizarro, M. (2016). Fighting to educate our own: Teachers of Color, relational accountability, and the struggle for racial justice. Equity & Excellence in Education, 49(1), 72-84. ↵
- Karabanow, J., Gurman, E., & Naylor, T. (2012). Street youth labor as an Expression of survival and self-worth. Critical Social Work, 13(2). ↵
- Noonan, B. M., Gallor, S. M., Hensler-McGinnis, N. F., Fassinger, R. E., Wang, S., & Goodman, J. (2004). Challenge and success: A Qualitative study of the career development of highly achieving women with physical and sensory disabilities. Journal of Counseling Psychology, 51(1), 68. ↵
- Ede, L., & Starrin, B. (2014). Unresolved conflicts and shaming processes: risk factors for long-term sick leave for mental-health reasons. Nordic Journal of Social Research, 5, 39-54. ↵
Conditions that are not directly observable and represent states of being, experiences, and ideas.
The extent to which a measure “covers” the construct of interest, i.e., it's comprehensiveness to measure the construct.
Chapter Outline
- Developing your theoretical framework
- Conceptual definitions
- Inductive & deductive reasoning
- Nomothetic causal explanations
Content warning: examples in this chapter include references to sexual harassment, domestic violence, gender-based violence, the child welfare system, substance use disorders, neonatal abstinence syndrome, child abuse, racism, and sexism.
11.1 Developing your theoretical framework
Learning Objectives
Learners will be able to...
- Differentiate between theories that explain specific parts of the social world versus those that are more broad and sweeping in their conclusions
- Identify the theoretical perspectives that are relevant to your project and inform your thinking about it
- Define key concepts in your working question and develop a theoretical framework for how you understand your topic.
Theories provide a way of looking at the world and of understanding human interaction. Paradigms are grounded in big assumptions about the world—what is real, how do we create knowledge—whereas theories describe more specific phenomena. Well, we are still oversimplifying a bit. Some theories try to explain the whole world, while others only try to explain a small part. Some theories can be grouped together based on common ideas but retain their own individual and unique features. Our goal is to help you find a theoretical framework that helps you understand your topic more deeply and answer your working question.
Theories: Big and small
In your human behavior and the social environment (HBSE) class, you were introduced to the major theoretical perspectives that are commonly used in social work. These are what we like to call big-T 'T'heories. When you read about systems theory, you are actually reading a synthesis of decades of distinct, overlapping, and conflicting theories that can be broadly classified within systems theory. For example, within systems theory, some approaches focus more on family systems while others focus on environmental systems, though the core concepts remain similar.
Different theorists define concepts in their own way, and as a result, their theories may explore different relationships with those concepts. For example, Deci and Ryan's (1985)[1] self-determination theory discusses motivation and establishes that it is contingent on meeting one's needs for autonomy, competency, and relatedness. By contrast, ecological self-determination theory, as written by Abery & Stancliffe (1996),[2] argues that self-determination is the amount of control exercised by an individual over aspects of their lives they deem important across the micro, meso, and macro levels. If self-determination were an important concept in your study, you would need to figure out which of the many theories related to self-determination helps you address your working question.
Theories can provide a broad perspective on the key concepts and relationships in the world or more specific and applied concepts and perspectives. Table 7.2 summarizes two commonly used lists of big-T Theoretical perspectives in social work. See if you can locate some of the theories that might inform your project.
| Payne's (2014)[3] practice theories | Hutchison's (2014)[4] theoretical perspectives |
| Psychodynamic | Systems |
| Crisis and task-centered | Conflict |
| Cognitive-behavioral | Exchange and choice |
| Systems/ecological | Social constructionist |
| Macro practice/social development/social pedagogy | Psychodynamic |
| Strengths/solution/narrative | Developmental |
| Humanistic/existential/spiritual | Social behavioral |
| Critical | Humanistic |
| Feminist | |
| Anti-discriminatory/multi-cultural sensitivity |
Competing theoretical explanations
Within each area of specialization in social work, there are many other theories that aim to explain more specific types of interactions. For example, within the study of sexual harassment, different theories posit different explanations for why harassment occurs.
One theory, first developed by criminologists, is called routine activities theory. It posits that sexual harassment is most likely to occur when a workplace lacks unified groups and when potentially vulnerable targets and motivated offenders are both present (DeCoster, Estes, & Mueller, 1999).[5]
Other theories of sexual harassment, called relational theories, suggest that one's existing relationships are the key to understanding why and how workplace sexual harassment occurs and how people will respond when it does occur (Morgan, 1999).[6] Relational theories focus on the power that different social relationships provide (e.g., married people who have supportive partners at home might be more likely than those who lack support at home to report sexual harassment when it occurs).
Finally, feminist theories of sexual harassment take a different stance. These theories posit that the organization of our current gender system, wherein those who are the most masculine have the most power, best explains the occurrence of workplace sexual harassment (MacKinnon, 1979).[7] As you might imagine, which theory a researcher uses to examine the topic of sexual harassment will shape the questions asked about harassment. It will also shape the explanations the researcher provides for why harassment occurs.
For a graduate student beginning their study of a new topic, it may be intimidating to learn that there are so many theories beyond what you’ve learned in your theory classes. What’s worse is that there is no central database of theories on your topic. However, as you review the literature in your area, you will learn more about the theories scientists have created to explain how your topic works in the real world. There are other good sources for theories, in addition to journal articles. Books often contain works of theoretical and philosophical importance that are beyond the scope of an academic journal. Do a search in your university library for books on your topic, and you are likely to find theorists talking about how to make sense of your topic. You don't necessarily have to agree with the prevailing theories about your topic, but you do need to be aware of them so you can apply theoretical ideas to your project.
Applying big-T theories to your topic
The key to applying theories to your topic is learning the key concepts associated with that theory and the relationships between those concepts, or propositions. Again, your HBSE class should have prepared you with some of the most important concepts from the theoretical perspectives listed in Table 7.2. For example, the conflict perspective sees the world as divided into dominant and oppressed groups who engage in conflict over resources. If you were applying these theoretical ideas to your project, you would need to identify which groups in your project are considered dominant or oppressed groups, and which resources they were struggling over. This is a very general example. Challenge yourself to find small-t theories about your topic that will help you understand it in much greater detail and specificity. If you have chosen a topic that is relevant to your life and future practice, you will be doing valuable work shaping your ideas towards social work practice.
Integrating theory into your project can be easy, or it can take a bit more effort. Some people have a strong and explicit theoretical perspective that they carry with them at all times. For me, you'll probably see my work drawing from exchange and choice, social constructionist, and critical theory. Maybe you have theoretical perspectives you naturally employ, like Afrocentric theory or person-centered practice. If so, that's a great place to start since you might already be using that theory (even subconsciously) to inform your understanding of your topic. But if you aren't aware of whether you are using a theoretical perspective when you think about your topic, try writing a paragraph off the top of your head or talking with a friend explaining what you think about that topic. Try matching it with some of the ideas from the broad theoretical perspectives from Table 7.2. This can ground you as you search for more specific theories. Some studies are designed to test whether theories apply the real world while others are designed to create new theories or variations on existing theories. Consider which feels more appropriate for your project and what you want to know.
Another way to easily identify the theories associated with your topic is to look at the concepts in your working question. Are these concepts commonly found in any of the theoretical perspectives in Table 7.2? Take a look at the Payne and Hutchison texts and see if any of those look like the concepts and relationships in your working question or if any of them match with how you think about your topic. Even if they don't possess the exact same wording, similar theories can help serve as a starting point to finding other theories that can inform your project. Remember, HBSE textbooks will give you not only the broad statements of theories but also sources from specific theorists and sub-theories that might be more applicable to your topic. Skim the references and suggestions for further reading once you find something that applies well.
Exercises
Choose a theoretical perspective from Hutchison, Payne, or another theory textbook that is relevant to your project. Using their textbooks or other reputable sources, identify :
- At least five important concepts from the theory
- What relationships the theory establishes between these important concepts (e.g., as x increases, the y decreases)
- How you can use this theory to better understand the concepts and variables in your project?
Developing your own theoretical framework
Hutchison's and Payne's frameworks are helpful for surveying the whole body of literature relevant to social work, which is why they are so widely used. They are one framework, or way of thinking, about all of the theories social workers will encounter that are relevant to practice. Social work researchers should delve further and develop a theoretical or conceptual framework of their own based on their reading of the literature. In Chapter 8, we will develop your theoretical framework further, identifying the cause-and-effect relationships that answer your working question. Developing a theoretical framework is also instructive for revising and clarifying your working question and identifying concepts that serve as keywords for additional literature searching. The greater clarity you have with your theoretical perspective, the easier each subsequent step in the research process will be.
Getting acquainted with the important theoretical concepts in a new area can be challenging. While social work education provides a broad overview of social theory, you will find much greater fulfillment out of reading about the theories related to your topic area. We discussed some strategies for finding theoretical information in Chapter 3 as part of literature searching. To extend that conversation a bit, some strategies for searching for theories in the literature include:
- Using keywords like "theory," "conceptual," or "framework" in queries to better target the search at sources that talk about theory.
- Consider searching for these keywords in the title or abstract, specifically
- Looking at the references and cited by links within theoretical articles and textbooks
- Looking at books, edited volumes, and textbooks that discuss theory
- Talking with a scholar on your topic, or asking a professor if they can help connect you to someone
- Looking at how researchers use theory in their research projects
- Nice authors are clear about how they use theory to inform their research project, usually in the introduction and discussion section.
- Starting with a Big-T Theory and looking for sub-theories or specific theorists that directly address your topic area
- For example, from the broad umbrella of systems theory, you might pick out family systems theory if you want to understand the effectiveness of a family counseling program.
It's important to remember that knowledge arises within disciplines, and that disciplines have different theoretical frameworks for explaining the same topic. While it is certainly important for the social work perspective to be a part of your analysis, social workers benefit from searching across disciplines to come to a more comprehensive understanding of the topic. Reaching across disciplines can provide uncommon insights during conceptualization, and once the study is completed, a multidisciplinary researcher will be able to share results in a way that speaks to a variety of audiences. A study by An and colleagues (2015)[8] uses game theory from the discipline of economics to understand problems in the Temporary Assistance for Needy Families (TANF) program. In order to receive TANF benefits, mothers must cooperate with paternity and child support requirements unless they have "good cause," as in cases of domestic violence, in which providing that information would put the mother at greater risk of violence. Game theory can help us understand how TANF recipients and caseworkers respond to the incentives in their environment, and highlight why the design of the "good cause" waiver program may not achieve its intended outcome of increasing access to benefits for survivors of family abuse.
Of course, there are natural limits on the depth with which student researchers can and should engage in a search for theory about their topic. At minimum, you should be able to draw connections across studies and be able to assess the relative importance of each theory within the literature. Just because you found one article applying your theory (like game theory, in our example above) does not mean it is important or often used in the domestic violence literature. Indeed, it would be much more common in the family violence literature to find psychological theories of trauma, feminist theories of power and control, and similar theoretical perspectives used to inform research projects rather than game theory, which is equally applicable to survivors of family violence as workers and bosses at a corporation. Consider using the Cited By feature to identify articles, books, and other sources of theoretical information that are seminal or well-cited in the literature. Similarly, by using the name of a theory in the keywords of a search query (along with keywords related to your topic), you can get a sense of how often the theory is used in your topic area. You should have a sense of what theories are commonly used to analyze your topic, even if you end up choosing a different one to inform your project.
Theories that are not cited or used as often are still immensely valuable. As we saw before with TANF and "good cause" waivers, using theories from other disciplines can produce uncommon insights and help you make a new contribution to the social work literature. Given the privileged position that the social work curriculum places on theories developed by white men, students may want to explore Afrocentricity as a social work practice theory (Pellebon, 2007)[9] or abolitionist social work (Jacobs et al., 2021)[10] when deciding on a theoretical framework for their research project that addresses concepts of racial justice. Start with your working question, and explain how each theory helps you answer your question. Some explanations are going to feel right, and some concepts will feel more salient to you than others. Keep in mind that this is an iterative process. Your theoretical framework will likely change as you continue to conceptualize your research project, revise your research question, and design your study.
By trying on many different theoretical explanations for your topic area, you can better clarify your own theoretical framework. Some of you may be fortunate enough to find theories that match perfectly with how you think about your topic, are used often in the literature, and are therefore relatively straightforward to apply. However, many of you may find that a combination of theoretical perspectives is most helpful for you to investigate your project. For example, maybe the group counseling program for which you are evaluating client outcomes draws from both motivational interviewing and cognitive behavioral therapy. In order to understand the change happening in the client population, you would need to know each theory separately as well as how they work in tandem with one another. Because theoretical explanations and even the definitions of concepts are debated by scientists, it may be helpful to find a specific social scientist or group of scientists whose perspective on the topic you find matches with your understanding of the topic. Of course, it is also perfectly acceptable to develop your own theoretical framework, though you should be able to articulate how your framework fills a gap within the literature.
If you are adapting theoretical perspectives in your study, it is important to clarify the original authors' definitions of each concept. Jabareen (2009)[11] offers that conceptual frameworks are not merely collections of concepts but, rather, constructs in which each concept plays an integral role.[12] A conceptual framework is a network of linked concepts that together provide a comprehensive understanding of a phenomenon. Each concept in a conceptual framework plays an ontological or epistemological role in the framework, and it is important to assess whether the concepts and relationships in your framework make sense together. As your framework takes shape, you will find yourself integrating and grouping together concepts, thinking about the most important or least important concepts, and how each concept is causally related to others.
Much like paradigm, theory plays a supporting role for the conceptualization of your research project. Recall the ice float from Figure 7.1. Theoretical explanations support the design and methods you use to answer your research question. In student projects that lack a theoretical framework, I often see the biases and errors in reasoning that we discussed in Chapter 1 that get in the way of good social science. That's because theories mark which concepts are important, provide a framework for understanding them, and measure their interrelationships. If you are missing this foundation, you will operate on informal observation, messages from authority, and other forms of unsystematic and unscientific thinking we reviewed in Chapter 1.
Theory-informed inquiry is incredibly helpful for identifying key concepts and how to measure them in your research project, but there is a risk in aligning research too closely with theory. The theory-ladenness of facts and observations produced by social science research means that we may be making our ideas real through research. This is a potential source of confirmation bias in social science. Moreover, as Tan (2016)[13] demonstrates, social science often proceeds by adopting as true the perspective of Western and Global North countries, and cross-cultural research is often when ethnocentric and biased ideas are most visible. In her example, a researcher from the West studying teacher-centric classrooms in China that rely partially on rote memorization may view them as less advanced than student-centered classrooms developed in a Western country simply because of Western philosophical assumptions about the importance of individualism and self-determination. Developing a clear theoretical framework is a way to guard against biased research, and it will establish a firm foundation on which you will develop the design and methods for your study.
Key Takeaways
- Just as empirical evidence is important for conceptualizing a research project, so too are the key concepts and relationships identified by social work theory.
- Using theory your theory textbook will provide you with a sense of the broad theoretical perspectives in social work that might be relevant to your project.
- Try to find small-t theories that are more specific to your topic area and relevant to your working question.
Exercises
- In Chapter 2, you developed a concept map for your proposal. Take a moment to revisit your concept map now as your theoretical framework is taking shape. Make any updates to the key concepts and relationships in your concept map.
. If you need a refresher, we have embedded a short how-to video from the University of Guelph Library (CC-BY-NC-SA 4.0) that we also used in Chapter 2.
11.2 Conceptual definitions
Learning Objectives
Learners will be able to...
- Define measurement and conceptualization
- Apply Kaplan’s three categories to determine the complexity of measuring a given variable
- Identify the role previous research and theory play in defining concepts
- Distinguish between unidimensional and multidimensional concepts
- Critically apply reification to how you conceptualize the key variables in your research project
In social science, when we use the term measurement, we mean the process by which we describe and ascribe meaning to the key facts, concepts, or other phenomena that we are investigating. At its core, measurement is about defining one’s terms in as clear and precise a way as possible. Of course, measurement in social science isn’t quite as simple as using a measuring cup or spoon, but there are some basic tenets on which most social scientists agree when it comes to measurement. We’ll explore those, as well as some of the ways that measurement might vary depending on your unique approach to the study of your topic.
An important point here is that measurement does not require any particular instruments or procedures. What it does require is a systematic procedure for assigning scores, meanings, and descriptions to individuals or objects so that those scores represent the characteristic of interest. You can measure phenomena in many different ways, but you must be sure that how you choose to measure gives you information and data that lets you answer your research question. If you're looking for information about a person's income, but your main points of measurement have to do with the money they have in the bank, you're not really going to find the information you're looking for!
The question of what social scientists measure can be answered by asking yourself what social scientists study. Think about the topics you’ve learned about in other social work classes you’ve taken or the topics you’ve considered investigating yourself. Let’s consider Melissa Milkie and Catharine Warner’s study (2011)[14] of first graders’ mental health. In order to conduct that study, Milkie and Warner needed to have some idea about how they were going to measure mental health. What does mental health mean, exactly? And how do we know when we’re observing someone whose mental health is good and when we see someone whose mental health is compromised? Understanding how measurement works in research methods helps us answer these sorts of questions.
As you might have guessed, social scientists will measure just about anything that they have an interest in investigating. For example, those who are interested in learning something about the correlation between social class and levels of happiness must develop some way to measure both social class and happiness. Those who wish to understand how well immigrants cope in their new locations must measure immigrant status and coping. Those who wish to understand how a person’s gender shapes their workplace experiences must measure gender and workplace experiences (and get more specific about which experiences are under examination). You get the idea. Social scientists can and do measure just about anything you can imagine observing or wanting to study. Of course, some things are easier to observe or measure than others.
Observing your variables
In 1964, philosopher Abraham Kaplan (1964)[15] wrote The Conduct of Inquiry, which has since become a classic work in research methodology (Babbie, 2010).[16] In his text, Kaplan describes different categories of things that behavioral scientists observe. One of those categories, which Kaplan called “observational terms,” is probably the simplest to measure in social science. Observational terms are the sorts of things that we can see with the naked eye simply by looking at them. Kaplan roughly defines them as conditions that are easy to identify and verify through direct observation. If, for example, we wanted to know how the conditions of playgrounds differ across different neighborhoods, we could directly observe the variety, amount, and condition of equipment at various playgrounds.
Indirect observables, on the other hand, are less straightforward to assess. In Kaplan's framework, they are conditions that are subtle and complex that we must use existing knowledge and intuition to define. If we conducted a study for which we wished to know a person’s income, we’d probably have to ask them their income, perhaps in an interview or a survey. Thus, we have observed income, even if it has only been observed indirectly. Birthplace might be another indirect observable. We can ask study participants where they were born, but chances are good we won’t have directly observed any of those people being born in the locations they report.
Sometimes the measures that we are interested in are more complex and more abstract than observational terms or indirect observables. Think about some of the concepts you’ve learned about in other social work classes—for example, ethnocentrism. What is ethnocentrism? Well, from completing an introduction to social work class you might know that it has something to do with the way a person judges another’s culture. But how would you measure it? Here’s another construct: bureaucracy. We know this term has something to do with organizations and how they operate but measuring such a construct is trickier than measuring something like a person’s income. The theoretical concepts of ethnocentrism and bureaucracy represent ideas whose meanings we have come to agree on. Though we may not be able to observe these abstractions directly, we can observe their components.
Kaplan referred to these more abstract things that behavioral scientists measure as constructs. Constructs are “not observational either directly or indirectly” (Kaplan, 1964, p. 55),[17] but they can be defined based on observables. For example, the construct of bureaucracy could be measured by counting the number of supervisors that need to approve routine spending by public administrators. The greater the number of administrators that must sign off on routine matters, the greater the degree of bureaucracy. Similarly, we might be able to ask a person the degree to which they trust people from different cultures around the world and then assess the ethnocentrism inherent in their answers. We can measure constructs like bureaucracy and ethnocentrism by defining them in terms of what we can observe.[18]
The idea of coming up with your own measurement tool might sound pretty intimidating at this point. The good news is that if you find something in the literature that works for you, you can use it (with proper attribution, of course). If there are only pieces of it that you like, you can reuse those pieces (with proper attribution and describing/justifying any changes). You don't always have to start from scratch!
Exercises
Look at the variables in your research question.
- Classify them as direct observables, indirect observables, or constructs.
- Do you think measuring them will be easy or hard?
- What are your first thoughts about how to measure each variable? No wrong answers here, just write down a thought about each variable.
Measurement starts with conceptualization
In order to measure the concepts in your research question, we first have to understand what we think about them. As an aside, the word concept has come up quite a bit, and it is important to be sure we have a shared understanding of that term. A concept is the notion or image that we conjure up when we think of some cluster of related observations or ideas. For example, masculinity is a concept. What do you think of when you hear that word? Presumably, you imagine some set of behaviors and perhaps even a particular style of self-presentation. Of course, we can’t necessarily assume that everyone conjures up the same set of ideas or images when they hear the word masculinity. While there are many possible ways to define the term and some may be more common or have more support than others, there is no universal definition of masculinity. What counts as masculine may shift over time, from culture to culture, and even from individual to individual (Kimmel, 2008). This is why defining our concepts is so important.\
Not all researchers clearly explain their theoretical or conceptual framework for their study, but they should! Without understanding how a researcher has defined their key concepts, it would be nearly impossible to understand the meaning of that researcher’s findings and conclusions. Back in Chapter 7, you developed a theoretical framework for your study based on a survey of the theoretical literature in your topic area. If you haven't done that yet, consider flipping back to that section to familiarize yourself with some of the techniques for finding and using theories relevant to your research question. Continuing with our example on masculinity, we would need to survey the literature on theories of masculinity. After a few queries on masculinity, I found a wonderful article by Wong (2010)[19] that analyzed eight years of the journal Psychology of Men & Masculinity and analyzed how often different theories of masculinity were used. Not only can I get a sense of which theories are more accepted and which are more marginal in the social science on masculinity, I am able to identify a range of options from which I can find the theory or theories that will inform my project.
Exercises
Identify a specific theory (or more than one theory) and how it helps you understand...
- Your independent variable(s).
- Your dependent variable(s).
- The relationship between your independent and dependent variables.
Rather than completing this exercise from scratch, build from your theoretical or conceptual framework developed in previous chapters.
In quantitative methods, conceptualization involves writing out clear, concise definitions for our key concepts. These are the kind of definitions you are used to, like the ones in a dictionary. A conceptual definition involves defining a concept in terms of other concepts, usually by making reference to how other social scientists and theorists have defined those concepts in the past. Of course, new conceptual definitions are created all the time because our conceptual understanding of the world is always evolving.
Conceptualization is deceptively challenging—spelling out exactly what the concepts in your research question mean to you. Following along with our example, think about what comes to mind when you read the term masculinity. How do you know masculinity when you see it? Does it have something to do with men or with social norms? If so, perhaps we could define masculinity as the social norms that men are expected to follow. That seems like a reasonable start, and at this early stage of conceptualization, brainstorming about the images conjured up by concepts and playing around with possible definitions is appropriate. However, this is just the first step. At this point, you should be beyond brainstorming for your key variables because you have read a good amount of research about them
In addition, we should consult previous research and theory to understand the definitions that other scholars have already given for the concepts we are interested in. This doesn’t mean we must use their definitions, but understanding how concepts have been defined in the past will help us to compare our conceptualizations with how other scholars define and relate concepts. Understanding prior definitions of our key concepts will also help us decide whether we plan to challenge those conceptualizations or rely on them for our own work. Finally, working on conceptualization is likely to help in the process of refining your research question to one that is specific and clear in what it asks. Conceptualization and operationalization (next section) are where "the rubber meets the road," so to speak, and you have to specify what you mean by the question you are asking. As your conceptualization deepens, you will often find that your research question becomes more specific and clear.
If we turn to the literature on masculinity, we will surely come across work by Michael Kimmel, one of the preeminent masculinity scholars in the United States. After consulting Kimmel’s prior work (2000; 2008),[20] we might tweak our initial definition of masculinity. Rather than defining masculinity as “the social norms that men are expected to follow,” perhaps instead we’ll define it as “the social roles, behaviors, and meanings prescribed for men in any given society at any one time” (Kimmel & Aronson, 2004, p. 503).[21] Our revised definition is more precise and complex because it goes beyond addressing one aspect of men’s lives (norms), and addresses three aspects: roles, behaviors, and meanings. It also implies that roles, behaviors, and meanings may vary across societies and over time. Using definitions developed by theorists and scholars is a good idea, though you may find that you want to define things your own way.
As you can see, conceptualization isn’t as simple as applying any random definition that we come up with to a term. Defining our terms may involve some brainstorming at the very beginning. But conceptualization must go beyond that, to engage with or critique existing definitions and conceptualizations in the literature. Once we’ve brainstormed about the images associated with a particular word, we should also consult prior work to understand how others define the term in question. After we’ve identified a clear definition that we’re happy with, we should make sure that every term used in our definition will make sense to others. Are there terms used within our definition that also need to be defined? If so, our conceptualization is not yet complete. Our definition includes the concept of "social roles," so we should have a definition for what those mean and become familiar with role theory to help us with our conceptualization. If we don't know what roles are, how can we study them?
Let's say we do all of that. We have a clear definition of the term masculinity with reference to previous literature and we also have a good understanding of the terms in our conceptual definition...then we're done, right? Not so fast. You’ve likely met more than one man in your life, and you’ve probably noticed that they are not the same, even if they live in the same society during the same historical time period. This could mean there are dimensions of masculinity. In terms of social scientific measurement, concepts can be said to have multiple dimensions when there are multiple elements that make up a single concept. With respect to the term masculinity, dimensions could based on gender identity, gender performance, sexual orientation, etc.. In any of these cases, the concept of masculinity would be considered to have multiple dimensions.
While you do not need to spell out every possible dimension of the concepts you wish to measure, it is important to identify whether your concepts are unidimensional (and therefore relatively easy to define and measure) or multidimensional (and therefore require multi-part definitions and measures). In this way, how you conceptualize your variables determines how you will measure them in your study. Unidimensional concepts are those that are expected to have a single underlying dimension. These concepts can be measured using a single measure or test. Examples include simple concepts such as a person’s weight, time spent sleeping, and so forth.
One frustrating this is that there is no clear demarcation between concepts that are inherently unidimensional or multidimensional. Even something as simple as age could be broken down into multiple dimensions including mental age and chronological age, so where does conceptualization stop? How far down the dimensional rabbit hole do we have to go? Researchers should consider two things. First, how important is this variable in your study? If age is not important in your study (maybe it is a control variable), it seems like a waste of time to do a lot of work drawing from developmental theory to conceptualize this variable. A unidimensional measure from zero to dead is all the detail we need. On the other hand, if we were measuring the impact of age on masculinity, conceptualizing our independent variable (age) as multidimensional may provide a richer understanding of its impact on masculinity. Finally, your conceptualization will lead directly to your operationalization of the variable, and once your operationalization is complete, make sure someone reading your study could follow how your conceptual definitions informed the measures you chose for your variables.
Exercises
Write a conceptual definition for your independent and dependent variables.
- Cite and attribute definitions to other scholars, if you use their words.
- Describe how your definitions are informed by your theoretical framework.
- Place your definition in conversation with other theories and conceptual definitions commonly used in the literature.
- Are there multiple dimensions of your variables?
- Are any of these dimensions important for you to measure?
Do researchers actually know what we're talking about?
Conceptualization proceeds differently in qualitative research compared to quantitative research. Since qualitative researchers are interested in the understandings and experiences of their participants, it is less important for them to find one fixed definition for a concept before starting to interview or interact with participants. The researcher’s job is to accurately and completely represent how their participants understand a concept, not to test their own definition of that concept.
If you were conducting qualitative research on masculinity, you would likely consult previous literature like Kimmel’s work mentioned above. From your literature review, you may come up with a working definition for the terms you plan to use in your study, which can change over the course of the investigation. However, the definition that matters is the definition that your participants share during data collection. A working definition is merely a place to start, and researchers should take care not to think it is the only or best definition out there.
In qualitative inquiry, your participants are the experts (sound familiar, social workers?) on the concepts that arise during the research study. Your job as the researcher is to accurately and reliably collect and interpret their understanding of the concepts they describe while answering your questions. Conceptualization of concepts is likely to change over the course of qualitative inquiry, as you learn more information from your participants. Indeed, getting participants to comment on, extend, or challenge the definitions and understandings of other participants is a hallmark of qualitative research. This is the opposite of quantitative research, in which definitions must be completely set in stone before the inquiry can begin.
The contrast between qualitative and quantitative conceptualization is instructive for understanding how quantitative methods (and positivist research in general) privilege the knowledge of the researcher over the knowledge of study participants and community members. Positivism holds that the researcher is the "expert," and can define concepts based on their expert knowledge of the scientific literature. This knowledge is in contrast to the lived experience that participants possess from experiencing the topic under examination day-in, day-out. For this reason, it would be wise to remind ourselves not to take our definitions too seriously and be critical about the limitations of our knowledge.
Conceptualization must be open to revisions, even radical revisions, as scientific knowledge progresses. While I’ve suggested consulting prior scholarly definitions of our concepts, you should not assume that prior, scholarly definitions are more real than the definitions we create. Likewise, we should not think that our own made-up definitions are any more real than any other definition. It would also be wrong to assume that just because definitions exist for some concept that the concept itself exists beyond some abstract idea in our heads. Building on the paradigmatic ideas behind interpretivism and the critical paradigm, researchers call the assumption that our abstract concepts exist in some concrete, tangible way is known as reification. It explores the power dynamics behind how we can create reality by how we define it.
Returning again to our example of masculinity. Think about our how our notions of masculinity have developed over the past few decades, and how different and yet so similar they are to patriarchal definitions throughout history. Conceptual definitions become more or less popular based on the power arrangements inside of social science the broader world. Western knowledge systems are privileged, while others are viewed as unscientific and marginal. The historical domination of social science by white men from WEIRD countries meant that definitions of masculinity were imbued their cultural biases and were designed explicitly and implicitly to preserve their power. This has inspired movements for cognitive justice as we seek to use social science to achieve global development.
Key Takeaways
- Measurement is the process by which we describe and ascribe meaning to the key facts, concepts, or other phenomena that we are investigating.
- Kaplan identified three categories of things that social scientists measure including observational terms, indirect observables, and constructs.
- Some concepts have multiple elements or dimensions.
- Researchers often use measures previously developed and studied by other researchers.
- Conceptualization is a process that involves coming up with clear, concise definitions.
- Conceptual definitions are based on the theoretical framework you are using for your study (and the paradigmatic assumptions underlying those theories).
- Whether your conceptual definitions come from your own ideas or the literature, you should be able to situate them in terms of other commonly used conceptual definitions.
- Researchers should acknowledge the limited explanatory power of their definitions for concepts and how oppression can shape what explanations are considered true or scientific.
Exercises
Think historically about the variables in your research question.
- How has our conceptual definition of your topic changed over time?
- What scholars or social forces were responsible for this change?
Take a critical look at your conceptual definitions.
- How participants might define terms for themselves differently, in terms of their daily experience?
- On what cultural assumptions are your conceptual definitions based?
- Are your conceptual definitions applicable across all cultures that will be represented in your sample?
11.3 Inductive and deductive reasoning
Learning Objectives
Learners will be able to...
- Describe inductive and deductive reasoning and provide examples of each
- Identify how inductive and deductive reasoning are complementary
Congratulations! You survived the chapter on theories and paradigms. My experience has been that many students have a difficult time thinking about theories and paradigms because they perceive them as "intangible" and thereby hard to connect to social work research. I even had one student who said she got frustrated just reading the word "philosophy."
Rest assured, you do not need to become a theorist or philosopher to be an effective social worker or researcher. However, you should have a good sense of what theory or theories will be relevant to your project, as well as how this theory, along with your working question, fit within the three broad research paradigms we reviewed. If you don't have a good idea about those at this point, it may be a good opportunity to pause and read more about the theories related to your topic area.
Theories structure and inform social work research. The converse is also true: research can structure and inform theory. The reciprocal relationship between theory and research often becomes evident to students when they consider the relationships between theory and research in inductive and deductive approaches to research. In both cases, theory is crucial. But the relationship between theory and research differs for each approach.
While inductive and deductive approaches to research are quite different, they can also be complementary. Let’s start by looking at each one and how they differ from one another. Then we’ll move on to thinking about how they complement one another.
Inductive reasoning
A researcher using inductive reasoning begins by collecting data that is relevant to their topic of interest. Once a substantial amount of data have been collected, the researcher will then step back from data collection to get a bird’s eye view of their data. At this stage, the researcher looks for patterns in the data, working to develop a theory that could explain those patterns. Thus, when researchers take an inductive approach, they start with a particular set of observations and move to a more general set of propositions about those experiences. In other words, they move from data to theory, or from the specific to the general. Figure 8.1 outlines the steps involved with an inductive approach to research.
There are many good examples of inductive research, but we’ll look at just a few here. One fascinating study in which the researchers took an inductive approach is Katherine Allen, Christine Kaestle, and Abbie Goldberg’s (2011)[22] study of how boys and young men learn about menstruation. To understand this process, Allen and her colleagues analyzed the written narratives of 23 young cisgender men in which the men described how they learned about menstruation, what they thought of it when they first learned about it, and what they think of it now. By looking for patterns across all 23 cisgender men’s narratives, the researchers were able to develop a general theory of how boys and young men learn about this aspect of girls’ and women’s biology. They conclude that sisters play an important role in boys’ early understanding of menstruation, that menstruation makes boys feel somewhat separated from girls, and that as they enter young adulthood and form romantic relationships, young men develop more mature attitudes about menstruation. Note how this study began with the data—men’s narratives of learning about menstruation—and worked to develop a theory.
In another inductive study, Kristin Ferguson and colleagues (Ferguson, Kim, & McCoy, 2011)[23] analyzed empirical data to better understand how to meet the needs of young people who are homeless. The authors analyzed focus group data from 20 youth at a homeless shelter. From these data they developed a set of recommendations for those interested in applied interventions that serve homeless youth. The researchers also developed hypotheses for others who might wish to conduct further investigation of the topic. Though Ferguson and her colleagues did not test their hypotheses, their study ends where most deductive investigations begin: with a theory and a hypothesis derived from that theory. Section 8.4 discusses the use of mixed methods research as a way for researchers to test hypotheses created in a previous component of the same research project.
You will notice from both of these examples that inductive reasoning is most commonly found in studies using qualitative methods, such as focus groups and interviews. Because inductive reasoning involves the creation of a new theory, researchers need very nuanced data on how the key concepts in their working question operate in the real world. Qualitative data is often drawn from lengthy interactions and observations with the individuals and phenomena under examination. For this reason, inductive reasoning is most often associated with qualitative methods, though it is used in both quantitative and qualitative research.
Deductive reasoning
If inductive reasoning is about creating theories from raw data, deductive reasoning is about testing theories using data. Researchers using deductive reasoning take the steps described earlier for inductive research and reverse their order. They start with a compelling social theory, create a hypothesis about how the world should work, collect raw data, and analyze whether their hypothesis was confirmed or not. That is, deductive approaches move from a more general level (theory) to a more specific (data); whereas inductive approaches move from the specific (data) to general (theory).
A deductive approach to research is the one that people typically associate with scientific investigation. Students in English-dominant countries that may be confused by inductive vs. deductive research can rest part of the blame on Sir Arthur Conan Doyle, creator of the Sherlock Holmes character. As Craig Vasey points out in his breezy introduction to logic book chapter, Sherlock Holmes more often used inductive rather than deductive reasoning (despite claiming to use the powers of deduction to solve crimes). By noticing subtle details in how people act, behave, and dress, Holmes finds patterns that others miss. Using those patterns, he creates a theory of how the crime occurred, dramatically revealed to the authorities just in time to arrest the suspect. Indeed, it is these flashes of insight into the patterns of data that make Holmes such a keen inductive reasoner. In social work practice, rather than detective work, inductive reasoning is supported by the intuitions and practice wisdom of social workers, just as Holmes' reasoning is sharpened by his experience as a detective.
So, if deductive reasoning isn't Sherlock Holmes' observation and pattern-finding, how does it work? It starts with what you have already done in Chapters 3 and 4, reading and evaluating what others have done to study your topic. It continued with Chapter 5, discovering what theories already try to explain how the concepts in your working question operate in the real world. Tapping into this foundation of knowledge on their topic, the researcher studies what others have done, reads existing theories of whatever phenomenon they are studying, and then tests hypotheses that emerge from those theories. Figure 8.2 outlines the steps involved with a deductive approach to research.
While not all researchers follow a deductive approach, many do. We’ll now take a look at a couple excellentrecent examples of deductive research.
In a study of US law enforcement responses to hate crimes, Ryan King and colleagues (King, Messner, & Baller, 2009)[24] hypothesized that law enforcement’s response would be less vigorous in areas of the country that had a stronger history of racial violence. The authors developed their hypothesis from prior research and theories on the topic. They tested the hypothesis by analyzing data on states’ lynching histories and hate crime responses. Overall, the authors found support for their hypothesis and illustrated an important application of critical race theory.
In another recent deductive study, Melissa Milkie and Catharine Warner (2011)[25] studied the effects of different classroom environments on first graders’ mental health. Based on prior research and theory, Milkie and Warner hypothesized that negative classroom features, such as a lack of basic supplies and heat, would be associated with emotional and behavioral problems in children. One might associate this research with Maslow's hierarchy of needs or systems theory. The researchers found support for their hypothesis, demonstrating that policymakers should be paying more attention to the mental health outcomes of children’s school experiences, just as they track academic outcomes (American Sociological Association, 2011).[26]
Complementary approaches
While inductive and deductive approaches to research seem quite different, they can actually be rather complementary. In some cases, researchers will plan for their study to include multiple components, one inductive and the other deductive. In other cases, a researcher might begin a study with the plan to conduct either inductive or deductive research, but then discovers along the way that the other approach is needed to help illuminate findings. Here is an example of each such case.
Dr. Amy Blackstone (n.d.), author of Principles of sociological inquiry: Qualitative and quantitative methods, relates a story about her mixed methods research on sexual harassment.
We began the study knowing that we would like to take both a deductive and an inductive approach in our work. We therefore administered a quantitative survey, the responses to which we could analyze in order to test hypotheses, and also conducted qualitative interviews with a number of the survey participants. The survey data were well suited to a deductive approach; we could analyze those data to test hypotheses that were generated based on theories of harassment. The interview data were well suited to an inductive approach; we looked for patterns across the interviews and then tried to make sense of those patterns by theorizing about them.
For one paper (Uggen & Blackstone, 2004)[27], we began with a prominent feminist theory of the sexual harassment of adult women and developed a set of hypotheses outlining how we expected the theory to apply in the case of younger women’s and men’s harassment experiences. We then tested our hypotheses by analyzing the survey data. In general, we found support for the theory that posited that the current gender system, in which heteronormative men wield the most power in the workplace, explained workplace sexual harassment—not just of adult women but of younger women and men as well. In a more recent paper (Blackstone, Houle, & Uggen, 2006),[28] we did not hypothesize about what we might find but instead inductively analyzed interview data, looking for patterns that might tell us something about how or whether workers’ perceptions of harassment change as they age and gain workplace experience. From this analysis, we determined that workers’ perceptions of harassment did indeed shift as they gained experience and that their later definitions of harassment were more stringent than those they held during adolescence. Overall, our desire to understand young workers’ harassment experiences fully—in terms of their objective workplace experiences, their perceptions of those experiences, and their stories of their experiences—led us to adopt both deductive and inductive approaches in the work. (Blackstone, n.d., p. 21)[29]
Researchers may not always set out to employ both approaches in their work but sometimes find that their use of one approach leads them to the other. One such example is described eloquently in Russell Schutt’s Investigating the Social World (2006).[30] As Schutt describes, researchers Sherman and Berk (1984)[31] conducted an experiment to test two competing theories of the effects of punishment on deterring deviance (in this case, domestic violence).Specifically, Sherman and Berk hypothesized that deterrence theory (see Williams, 2005[32] for more information on that theory) would provide a better explanation of the effects of arresting accused batterers than labeling theory. Deterrence theory predicts that arresting an accused spouse batterer will reduce future incidents of violence. Conversely, labeling theory predicts that arresting accused spouse batterers will increase future incidents (see Policastro & Payne, 2013[33] for more information on that theory). Figure 8.3 summarizes the two competing theories and the hypotheses Sherman and Berk set out to test.
Research from these follow-up studies were mixed. In some cases, arrest deterred future incidents of violence. In other cases, it did not. This left the researchers with new data that they needed to explain. The researchers therefore took an inductive approach in an effort to make sense of their latest empirical observations. The new studies revealed that arrest seemed to have a deterrent effect for those who were married and employed, but that it led to increased offenses for those who were unmarried and unemployed. Researchers thus turned to control theory, which posits that having some stake in conformity through the social ties provided by marriage and employment, as the better explanation (see Davis et al., 2000[35] for more information on this theory).
What the original Sherman and Berk study, along with the follow-up studies, show us is that we might start with a deductive approach to research, but then, if confronted by new data we must make sense of, we may move to an inductive approach. We will expand on these possibilities in section 8.4 when we discuss mixed methods research.
Ethical and critical considerations
Deductive and inductive reasoning, just like other components of the research process comes with ethical and cultural considerations for researchers. Specifically, deductive research is limited by existing theory. Because scientific inquiry has been shaped by oppressive forces such as sexism, racism, and colonialism, what is considered theory is largely based in Western, white-male-dominant culture. Thus, researchers doing deductive research may artificially limit themselves to ideas that were derived from this context. Non-Western researchers, international social workers, and practitioners working with non-dominant groups may find deductive reasoning of limited help if theories do not adequately describe other cultures.
While these flaws in deductive research may make inductive reasoning seem more appealing, on closer inspection you'll find similar issues apply. A researcher using inductive reasoning applies their intuition and lived experience when analyzing participant data. They will take note of particular themes, conceptualize their definition, and frame the project using their unique psychology. Since everyone's internal world is shaped by their cultural and environmental context, inductive reasoning conducted by Western researchers may unintentionally reinforcing lines of inquiry that derive from cultural oppression.
Inductive reasoning is also shaped by those invited to provide the data to be analyzed. For example, I recently worked with a student who wanted to understand the impact of child welfare supervision on children born dependent on opiates and methamphetamine. Due to the potential harm that could come from interviewing families and children who are in foster care or under child welfare supervision, the researcher decided to use inductive reasoning and to only interview child welfare workers.
Talking to practitioners is a good idea for feasibility, as they are less vulnerable than clients. However, any theory that emerges out of these observations will be substantially limited, as it would be devoid of the perspectives of parents, children, and other community members who could provide a more comprehensive picture of the impact of child welfare involvement on children. Notice that each of these groups has less power than child welfare workers in the service relationship. Attending to which groups were used to inform the creation of a theory and the power of those groups is an important critical consideration for social work researchers.
As you can see, when researchers apply theory to research they must wrestle with the history and hierarchy around knowledge creation in that area. In deductive studies, the researcher is positioned as the expert, similar to the positivist paradigm presented in Chapter 5. We've discussed a few of the limitations on the knowledge of researchers in this subsection, but the position of the "researcher as expert" is inherently problematic. However, it should also not be taken to an extreme. A researcher who approaches inductive inquiry as a naïve learner is also inherently problematic. Just as competence in social work practice requires a baseline of knowledge prior to entering practice, so does competence in social work research. Because a truly naïve intellectual position is impossible—we all have preexisting ways we view the world and are not fully aware of how they may impact our thoughts—researchers should be well-read in the topic area of their research study but humble enough to know that there is always much more to learn.
Key Takeaways
- Inductive reasoning begins with a set of empirical observations, seeking patterns in those observations, and then theorizing about those patterns.
- Deductive reasoning begins with a theory, developing hypotheses from that theory, and then collecting and analyzing data to test the truth of those hypotheses.
- Inductive and deductive reasoning can be employed together for a more complete understanding of the research topic.
- Though researchers don’t always set out to use both inductive and deductive reasoning in their work, they sometimes find that new questions arise in the course of an investigation that can best be answered by employing both approaches.
Exercises
- Identify one theory and how it helps you understand your topic and working question.
I encourage you to find a specific theory from your topic area, rather than relying only on the broad theoretical perspectives like systems theory or the strengths perspective. Those broad theoretical perspectives are okay...but I promise that searching for theories about your topic will help you conceptualize and design your research project.
- Using the theory you identified, describe what you expect the answer to be to your working question.
11.4
Learning Objectives
Learners will be able to...
- Define and provide an example of idiographic causal relationships
- Describe the role of causality in quantitative research as compared to qualitative research
- Identify, define, and describe each of the main criteria for nomothetic causal relationships
- Describe the difference between and provide examples of independent, dependent, and control variables
- Define hypothesis, state a clear hypothesis, and discuss the respective roles of quantitative and qualitative research when it comes to hypotheses
Causality refers to the idea that one event, behavior, or belief will result in the occurrence of another, subsequent event, behavior, or belief. In other words, it is about cause and effect. It seems simple, but you may be surprised to learn there is more than one way to explain how one thing causes another. How can that be? How could there be many ways to understand causality?
Think back to our discussion in Section 5.3 on paradigms [insert chapter link plus link to section 1.2]. You’ll remember the positivist paradigm as the one that believes in objectivity. Positivists look for causal explanations that are universally true for everyone, everywhere because they seek objective truth. Interpretivists, on the other hand, look for causal explanations that are true for individuals or groups in a specific time and place because they seek subjective truths. Remember that for interpretivists, there is not one singular truth that is true for everyone, but many truths created and shared by others.
"Are you trying to generalize or nah?"
One of my favorite classroom moments occurred in the early days of my teaching career. Students were providing peer feedback on their working questions. I overheard one group who was helping someone rephrase their research question. A student asked, “Are you trying to generalize or nah?” Teaching is full of fun moments like that one. Answering that one question can help you understand how to conceptualize and design your research project.
Nomothetic causal explanations are incredibly powerful. They allow scientists to make predictions about what will happen in the future, with a certain margin of error. Moreover, they allow scientists to generalize—that is, make claims about a large population based on a smaller sample of people or items. Generalizing is important. We clearly do not have time to ask everyone their opinion on a topic or test a new intervention on every person. We need a type of causal explanation that helps us predict and estimate truth in all situations.
Generally, nomothetic causal relationships work best for explanatory research projects [INSERT SECTION LINK]. They also tend to use quantitative research: by boiling things down to numbers, one can use the universal language of mathematics to use statistics to explore those relationships. On the other hand, descriptive and exploratory projects often fit better with idiographic causality. These projects do not usually try to generalize, but instead investigate what is true for individuals, small groups, or communities at a specific point in time. You will learn about this type of causality in the next section. Here, we will assume you have an explanatory working question. For example, you may want to know about the risk and protective factors for a specific diagnosis or how a specific therapy impacts client outcomes.
What do nomothetic causal explanations look like?
Nomothetic causal explanations express relationships between variables. The term variable has a scientific definition. This one from Gillespie & Wagner (2018) "a logical grouping of attributes that can be observed and measured and is expected to vary from person to person in a population" (p. 9).[36] More practically, variables are the key concepts in your working question. You know, the things you plan to observe when you actually do your research project, conduct your surveys, complete your interviews, etc. These things have two key properties. First, they vary, as in they do not remain constant. "Age" varies by number. "Gender" varies by category. But they both vary. Second, they have attributes. So the variable "health professions" has attributes or categories, such as social worker, nurse, counselor, etc.
It's also worth reviewing what is not a variable. Well, things that don't change (or vary) aren't variables. If you planned to do a study on how gender impacts earnings but your study only contained women, that concept would not vary. Instead, it would be a constant. Another common mistake I see in students' explanatory questions is mistaking an attribute for a variable. "Men" is not a variable. "Gender" is a variable. "Virginia" is not a variable. The variable is the "state or territory" in which someone or something is physically located.
When one variable causes another, we have what researchers call independent and dependent variables. For example, in a study investigating the impact of spanking on aggressive behavior, spanking would be the independent variable and aggressive behavior would be the dependent variable. An independent variable is the cause, and a dependent variable is the effect. Why are they called that? Dependent variables depend on independent variables. If all of that gets confusing, just remember the graphical relationship in Figure 8.5.
Exercises
Write out your working question, as it exists now. As we said previously in the subsection, we assume you have an explanatory research question for learning this section.
- Write out a diagram similar to Figure 8.5.
- Put your independent variable on the left and the dependent variable on the right.
Check:
- Can your variables vary?
- Do they have different attributes or categories that vary from person to person?
- How does the theory you identified in section 8.1 help you understand this causal relationship?
If the theory you've identified isn't much help to you or seems unrelated, it's a good indication that you need to read more literature about the theories related to your topic.
For some students, your working question may not be specific enough to list an independent or dependent variable clearly. You may have "risk factors" in place of an independent variable, for example. Or "effects" as a dependent variable. If that applies to your research question, get specific for a minute even if you have to revise this later. Think about which specific risk factors or effects you are interested in. Consider a few options for your independent and dependent variable and create diagrams similar to Figure 8.5.
Finally, you are likely to revisit your working question so you may have to come back to this exercise to clarify the causal relationship you want to investigate.
For a ten-cent word like "nomothetic," these causal relationships should look pretty basic to you. They should look like "x causes y." Indeed, you may be looking at your causal explanation and thinking, "wow, there are so many other things I'm missing in here." In fact, maybe my dependent variable sometimes causes changes in my independent variable! For example, a working question asking about poverty and education might ask how poverty makes it more difficult to graduate college or how high college debt impacts income inequality after graduation. Nomothetic causal relationships are slices of reality. They boil things down to two (or often more) key variables and assert a one-way causal explanation between them. This is by design, as they are trying to generalize across all people to all situations. The more complicated, circular, and often contradictory causal explanations are idiographic, which we will cover in the next section of this chapter.
Developing a hypothesis
A hypothesis is a statement describing a researcher’s expectation regarding what they anticipate finding. Hypotheses in quantitative research are a nomothetic causal relationship that the researcher expects to determine is true or false. A hypothesis is written to describe the expected relationship between the independent and dependent variables. In other words, write the answer to your working question using your variables. That's your hypothesis! Make sure you haven't introduced new variables into your hypothesis that are not in your research question. If you have, write out your hypothesis as in Figure 8.5.
A good hypothesis should be testable using social science research methods. That is, you can use a social science research project (like a survey or experiment) to test whether it is true or not. A good hypothesis is also specific about the relationship it explores. For example, a student project that hypothesizes, "families involved with child welfare agencies will benefit from Early Intervention programs," is not specific about what benefits it plans to investigate. For this student, I advised her to take a look at the empirical literature and theory about Early Intervention and see what outcomes are associated with these programs. This way, she could more clearly state the dependent variable in her hypothesis, perhaps looking at reunification, attachment, or developmental milestone achievement in children and families under child welfare supervision.
Your hypothesis should be an informed prediction based on a theory or model of the social world. For example, you may hypothesize that treating mental health clients with warmth and positive regard is likely to help them achieve their therapeutic goals. That hypothesis would be based on the humanistic practice models of Carl Rogers. Using previous theories to generate hypotheses is an example of deductive research. If Rogers’ theory of unconditional positive regard is accurate, a study comparing clinicians who used it versus those who did not would show more favorable treatment outcomes for clients receiving unconditional positive regard.
Let’s consider a couple of examples. In research on sexual harassment (Uggen & Blackstone, 2004),[37] one might hypothesize, based on feminist theories of sexual harassment, that more females than males will experience specific sexually harassing behaviors. What is the causal relationship being predicted here? Which is the independent and which is the dependent variable? In this case, researchers hypothesized that a person’s sex (independent variable) would predict their likelihood to experience sexual harassment (dependent variable).
Sometimes researchers will hypothesize that a relationship will take a specific direction. As a result, an increase or decrease in one area might be said to cause an increase or decrease in another. For example, you might choose to study the relationship between age and support for legalization of marijuana. Perhaps you’ve taken a sociology class and, based on the theories you’ve read, you hypothesize that age is negatively related to support for marijuana legalization.[38] What have you just hypothesized?
You have hypothesized that as people get older, the likelihood of their supporting marijuana legalization decreases. Thus, as age (your independent variable) moves in one direction (up), support for marijuana legalization (your dependent variable) moves in another direction (down). So, a direct relationship (or positive correlation) involve two variables going in the same direction and an inverse relationship (or negative correlation) involve two variables going in opposite directions. If writing hypotheses feels tricky, it is sometimes helpful to draw them out and depict each of the two hypotheses we have just discussed.
It’s important to note that once a study starts, it is unethical to change your hypothesis to match the data you find. For example, what happens if you conduct a study to test the hypothesis from Figure 8.7 on support for marijuana legalization, but you find no relationship between age and support for legalization? It means that your hypothesis was incorrect, but that’s still valuable information. It would challenge what the existing literature says on your topic, demonstrating that more research needs to be done to figure out the factors that impact support for marijuana legalization. Don’t be embarrassed by negative results, and definitely don’t change your hypothesis to make it appear correct all along!
Criteria for establishing a nomothetic causal relationship
Let’s say you conduct your study and you find evidence that supports your hypothesis, as age increases, support for marijuana legalization decreases. Success! Causal explanation complete, right? Not quite.
You’ve only established one of the criteria for causality. The criteria for causality must include all of the following: covariation, plausibility, temporality, and nonspuriousness. In our example from Figure 8.7, we have established only one criteria—covariation. When variables covary, they vary together. Both age and support for marijuana legalization vary in our study. Our sample contains people of varying ages and varying levels of support for marijuana legalization. If, for example, we only included 16-year-olds in our study, age would be a constant, not a variable.
Just because there might be some correlation between two variables does not mean that a causal relationship between the two is really plausible. Plausibility means that in order to make the claim that one event, behavior, or belief causes another, the claim has to make sense. It makes sense that people from previous generations would have different attitudes towards marijuana than younger generations. People who grew up in the time of Reefer Madness or the hippies may hold different views than those raised in an era of legalized medicinal and recreational use of marijuana. Plausibility is of course helped by basing your causal explanation in existing theoretical and empirical findings.
Once we’ve established that there is a plausible relationship between the two variables, we also need to establish whether the cause occurred before the effect, the criterion of temporality. A person’s age is a quality that appears long before any opinions on drug policy, so temporally the cause comes before the effect. It wouldn’t make any sense to say that support for marijuana legalization makes a person’s age increase. Even if you could predict someone’s age based on their support for marijuana legalization, you couldn’t say someone’s age was caused by their support for legalization of marijuana.
Finally, scientists must establish nonspuriousness. A spurious relationship is one in which an association between two variables appears to be causal but can in fact be explained by some third variable. This third variable is often called a confound or confounding variable because it clouds and confuses the relationship between your independent and dependent variable, making it difficult to discern the true causal relationship is.
Continuing with our example, we could point to the fact that older adults are less likely to have used marijuana recreationally. Maybe it is actually recreational use of marijuana that leads people to be more open to legalization, not their age. In this case, our confounding variable would be recreational marijuana use. Perhaps the relationship between age and attitudes towards legalization is a spurious relationship that is accounted for by previous use. This is also referred to as the third variable problem, where a seemingly true causal relationship is actually caused by a third variable not in the hypothesis. In this example, the relationship between age and support for legalization could be more about having tried marijuana than the age of the person.
Quantitative researchers are sensitive to the effects of potentially spurious relationships. As a result, they will often measure these third variables in their study, so they can control for their effects in their statistical analysis. These are called control variables, and they refer to potentially confounding variables whose effects are controlled for mathematically in the data analysis process. Control variables can be a bit confusing, and we will discuss them more in Chapter 10, but think about it as an argument between you, the researcher, and a critic.
Researcher: “The older a person is, the less likely they are to support marijuana legalization.”
Critic: “Actually, it’s more about whether a person has used marijuana before. That is what truly determines whether someone supports marijuana legalization.”
Researcher: “Well, I measured previous marijuana use in my study and mathematically controlled for its effects in my analysis. Age explains most of the variation in attitudes towards marijuana legalization.”
Let’s consider a few additional, real-world examples of spuriousness. Did you know, for example, that high rates of ice cream sales have been shown to cause drowning? Of course, that’s not really true, but there is a positive relationship between the two. In this case, the third variable that causes both high ice cream sales and increased deaths by drowning is time of year, as the summer season sees increases in both (Babbie, 2010).[39]
Here’s another good one: it is true that as the salaries of Presbyterian ministers in Massachusetts rise, so too does the price of rum in Havana, Cuba. Well, duh, you might be saying to yourself. Everyone knows how much ministers in Massachusetts love their rum, right? Not so fast. Both salaries and rum prices have increased, true, but so has the price of just about everything else (Huff & Geis, 1993).[40]
Finally, research shows that the more firefighters present at a fire, the more damage is done at the scene. What this statement leaves out, of course, is that as the size of a fire increases so too does the amount of damage caused as does the number of firefighters called on to help (Frankfort-Nachmias & Leon-Guerrero, 2011).[41] In each of these examples, it is the presence of a confounding variable that explains the apparent relationship between the two original variables.
In sum, the following criteria must be met for a nomothetic causal relationship:
- The two variables must vary together.
- The relationship must be plausible.
- The cause must precede the effect in time.
- The relationship must be nonspurious (not due to a confounding variable).
The hypothetico-dedutive method
The primary way that researchers in the positivist paradigm use theories is sometimes called the hypothetico-deductive method (although this term is much more likely to be used by philosophers of science than by scientists themselves). Researchers choose an existing theory. Then, they make a prediction about some new phenomenon that should be observed if the theory is correct. Again, this prediction is called a hypothesis. The researchers then conduct an empirical study to test the hypothesis. Finally, they reevaluate the theory in light of the new results and revise it if necessary.
This process is usually conceptualized as a cycle because the researchers can then derive a new hypothesis from the revised theory, conduct a new empirical study to test the hypothesis, and so on. As Figure 8.8 shows, this approach meshes nicely with the process of conducting a research project—creating a more detailed model of “theoretically motivated” or “theory-driven” research. Together, they form a model of theoretically motivated research.
Keep in mind the hypothetico-deductive method is only one way of using social theory to inform social science research. It starts with describing one or more existing theories, deriving a hypothesis from one of those theories, testing your hypothesis in a new study, and finally reevaluating the theory based on the results data analyses. This format works well when there is an existing theory that addresses the research question—especially if the resulting hypothesis is surprising or conflicts with a hypothesis derived from a different theory.
But what if your research question is more interpretive? What if it is less about theory-testing and more about theory-building? This is what our next chapters will cover: the process of inductively deriving theory from people's stories and experiences. This process looks different than that depicted in Figure 8.8. It still starts with your research question and answering that question by conducting a research study. But instead of testing a hypothesis you created based on a theory, you will create a theory of your own that explain the data you collected. This format works well for qualitative research questions and for research questions that existing theories do not address.
Key Takeaways
- In positivist and quantitative studies, the goal is often to understand the more general causes of some phenomenon rather than the idiosyncrasies of one particular instance, as in an idiographic causal relationship.
- Nomothetic causal explanations focus on objectivity, prediction, and generalization.
- Criteria for nomothetic causal relationships require the relationship be plausible and nonspurious; and that the cause must precede the effect in time.
- In a nomothetic causal relationship, the independent variable causes changes in the dependent variable.
- Hypotheses are statements, drawn from theory, which describe a researcher’s expectation about a relationship between two or more variables.
Exercises
- Write out your working question and hypothesis.
- Defend your hypothesis in a short paragraph, using arguments based on the theory you identified in section 8.1.
- Review the criteria for a nomothetic causal relationship. Critique your short paragraph about your hypothesis using these criteria.
- Are there potentially confounding variables, issues with time order, or other problems you can identify in your reasoning?
Inductive & deductive (deductive focus)
Nomothetic causal explanations
Positivism
Chapter Outline
- Operational definitions (36 minute read)
- Writing effective questions and questionnaires (38 minute read)
- Measurement quality (21 minute read)
Content warning: examples in this chapter contain references to ethnocentrism, toxic masculinity, racism in science, drug use, mental health and depression, psychiatric inpatient care, poverty and basic needs insecurity, pregnancy, and racism and sexism in the workplace and higher education.
11.1 Operational definitions
Learning Objectives
Learners will be able to...
- Define and give an example of indicators and attributes for a variable
- Apply the three components of an operational definition to a variable
- Distinguish between levels of measurement for a variable and how those differences relate to measurement
- Describe the purpose of composite measures like scales and indices
Last chapter, we discussed conceptualizing your project. Conceptual definitions are like dictionary definitions. They tell you what a concept means by defining it using other concepts. In this section we will move from the abstract realm (conceptualization) to the real world (measurement).
Operationalization is the process by which researchers spell out precisely how a concept will be measured in their study. It involves identifying the specific research procedures we will use to gather data about our concepts. If conceptually defining your terms means looking at theory, how do you operationally define your terms? By looking for indicators of when your variable is present or not, more or less intense, and so forth. Operationalization is probably the most challenging part of quantitative research, but once it's done, the design and implementation of your study will be straightforward.
Indicators
Operationalization works by identifying specific indicators that will be taken to represent the ideas we are interested in studying. If we are interested in studying masculinity, then the indicators for that concept might include some of the social roles prescribed to men in society such as breadwinning or fatherhood. Being a breadwinner or a father might therefore be considered indicators of a person’s masculinity. The extent to which a man fulfills either, or both, of these roles might be understood as clues (or indicators) about the extent to which he is viewed as masculine.
Let’s look at another example of indicators. Each day, Gallup researchers poll 1,000 randomly selected Americans to ask them about their well-being. To measure well-being, Gallup asks these people to respond to questions covering six broad areas: physical health, emotional health, work environment, life evaluation, healthy behaviors, and access to basic necessities. Gallup uses these six factors as indicators of the concept that they are really interested in, which is well-being.
Identifying indicators can be even simpler than the examples described thus far. Political party affiliation is another relatively easy concept for which to identify indicators. If you asked a person what party they voted for in the last national election (or gained access to their voting records), you would get a good indication of their party affiliation. Of course, some voters split tickets between multiple parties when they vote and others swing from party to party each election, so our indicator is not perfect. Indeed, if our study were about political identity as a key concept, operationalizing it solely in terms of who they voted for in the previous election leaves out a lot of information about identity that is relevant to that concept. Nevertheless, it's a pretty good indicator of political party affiliation.
Choosing indicators is not an arbitrary process. As described earlier, utilizing prior theoretical and empirical work in your area of interest is a great way to identify indicators in a scholarly manner. And you conceptual definitions will point you in the direction of relevant indicators. Empirical work will give you some very specific examples of how the important concepts in an area have been measured in the past and what sorts of indicators have been used. Often, it makes sense to use the same indicators as previous researchers; however, you may find that some previous measures have potential weaknesses that your own study will improve upon.
All of the examples in this chapter have dealt with questions you might ask a research participant on a survey or in a quantitative interview. If you plan to collect data from other sources, such as through direct observation or the analysis of available records, think practically about what the design of your study might look like and how you can collect data on various indicators feasibly. If your study asks about whether the participant regularly changes the oil in their car, you will likely not observe them directly doing so. Instead, you will likely need to rely on a survey question that asks them the frequency with which they change their oil or ask to see their car maintenance records.
Exercises
- What indicators are commonly used to measure the variables in your research question?
- How can you feasibly collect data on these indicators?
- Are you planning to collect your own data using a questionnaire or interview? Or are you planning to analyze available data like client files or raw data shared from another researcher's project?
Remember, you need raw data. You research project cannot rely solely on the results reported by other researchers or the arguments you read in the literature. A literature review is only the first part of a research project, and your review of the literature should inform the indicators you end up choosing when you measure the variables in your research question.
Unlike conceptual definitions which contain other concepts, operational definition consists of the following components: (1) the variable being measured and its attributes, (2) the measure you will use, (3) how you plan to interpret the data collected from that measure to draw conclusions about the variable you are measuring.
Step 1: Specifying variables and attributes
The first component, the variable, should be the easiest part. At this point in quantitative research, you should have a research question that has at least one independent and at least one dependent variable. Remember that variables must be able to vary. For example, the United States is not a variable. Country of residence is a variable, as is patriotism. Similarly, if your sample only includes men, gender is a constant in your study, not a variable. A constant is a characteristic that does not change in your study.
When social scientists measure concepts, they sometimes use the language of variables and attributes. A variable refers to a quality or quantity that varies across people or situations. Attributes are the characteristics that make up a variable. For example, the variable hair color would contain attributes like blonde, brown, black, red, gray, etc. A variable’s attributes determine its level of measurement. There are four possible levels of measurement: nominal, ordinal, interval, and ratio. The first two levels of measurement are categorical, meaning their attributes are categories rather than numbers. The latter two levels of measurement are continuous, meaning their attributes are numbers.
Levels of measurement
Hair color is an example of a nominal level of measurement. Nominal measures are categorical, and those categories cannot be mathematically ranked. As a brown-haired person (with some gray), I can’t say for sure that brown-haired people are better than blonde-haired people. As with all nominal levels of measurement, there is no ranking order between hair colors; they are simply different. That is what constitutes a nominal level of gender and race are also measured at the nominal level.
What attributes are contained in the variable hair color? While blonde, brown, black, and red are common colors, some people may not fit into these categories if we only list these attributes. My wife, who currently has purple hair, wouldn’t fit anywhere. This means that our attributes were not exhaustive. Exhaustiveness means that all possible attributes are listed. We may have to list a lot of colors before we can meet the criteria of exhaustiveness. Clearly, there is a point at which exhaustiveness has been reasonably met. If a person insists that their hair color is light burnt sienna, it is not your responsibility to list that as an option. Rather, that person would reasonably be described as brown-haired. Perhaps listing a category for other color would suffice to make our list of colors exhaustive.
What about a person who has multiple hair colors at the same time, such as red and black? They would fall into multiple attributes. This violates the rule of mutual exclusivity, in which a person cannot fall into two different attributes. Instead of listing all of the possible combinations of colors, perhaps you might include a multi-color attribute to describe people with more than one hair color.
Making sure researchers provide mutually exclusive and exhaustive is about making sure all people are represented in the data record. For many years, the attributes for gender were only male or female. Now, our understanding of gender has evolved to encompass more attributes that better reflect the diversity in the world. Children of parents from different races were often classified as one race or another, even if they identified with both cultures. The option for bi-racial or multi-racial on a survey not only more accurately reflects the racial diversity in the real world but validates and acknowledges people who identify in that manner. If we did not measure race in this way, we would leave empty the data record for people who identify as biracial or multiracial, impairing our search for truth.
Unlike nominal-level measures, attributes at the ordinal level can be rank ordered. For example, someone’s degree of satisfaction in their romantic relationship can be ordered by rank. That is, you could say you are not at all satisfied, a little satisfied, moderately satisfied, or highly satisfied. Note that even though these have a rank order to them (not at all satisfied is certainly worse than highly satisfied), we cannot calculate a mathematical distance between those attributes. We can simply say that one attribute of an ordinal-level variable is more or less than another attribute.
This can get a little confusing when using rating scales. If you have ever taken a customer satisfaction survey or completed a course evaluation for school, you are familiar with rating scales. “On a scale of 1-5, with 1 being the lowest and 5 being the highest, how likely are you to recommend our company to other people?” That surely sounds familiar. Rating scales use numbers, but only as a shorthand, to indicate what attribute (highly likely, somewhat likely, etc.) the person feels describes them best. You wouldn’t say you are “2” likely to recommend the company, but you would say you are not very likely to recommend the company. Ordinal-level attributes must also be exhaustive and mutually exclusive, as with nominal-level variables.
At the interval level, attributes must also be exhaustive and mutually exclusive and there is equal distance between attributes. Interval measures are also continuous, meaning their attributes are numbers, rather than categories. IQ scores are interval level, as are temperatures in Fahrenheit and Celsius. Their defining characteristic is that we can say how much more or less one attribute differs from another. We cannot, however, say with certainty what the ratio of one attribute is in comparison to another. For example, it would not make sense to say that a person with an IQ score of 140 has twice the IQ of a person with a score of 70. However, the difference between IQ scores of 80 and 100 is the same as the difference between IQ scores of 120 and 140.
While we cannot say that someone with an IQ of 140 is twice as intelligent as someone with an IQ of 70 because IQ is measured at the interval level, we can say that someone with six siblings has twice as many as someone with three because number of siblings is measured at the ratio level. Finally, at the ratio level, attributes are mutually exclusive and exhaustive, attributes can be rank ordered, the distance between attributes is equal, and attributes have a true zero point. Thus, with these variables, we can say what the ratio of one attribute is in comparison to another. Examples of ratio-level variables include age and years of education. We know that a person who is 12 years old is twice as old as someone who is 6 years old. Height measured in meters and weight measured in kilograms are good examples. So are counts of discrete objects or events such as the number of siblings one has or the number of questions a student answers correctly on an exam. The differences between each level of measurement are visualized in Table 11.1.
| Nominal | Ordinal | Interval | Ratio | |
| Exhaustive | X | X | X | X |
| Mutually exclusive | X | X | X | X |
| Rank-ordered | X | X | X | |
| Equal distance between attributes | X | X | ||
| True zero point | X |
Levels of measurement=levels of specificity
We have spent time learning how to determine our data's level of measurement. Now what? How could we use this information to help us as we measure concepts and develop measurement tools? First, the types of statistical tests that we are able to use are dependent on our data's level of measurement. With nominal-level measurement, for example, the only available measure of central tendency is the mode. With ordinal-level measurement, the median or mode can be used as indicators of central tendency. Interval and ratio-level measurement are typically considered the most desirable because they permit for any indicators of central tendency to be computed (i.e., mean, median, or mode). Also, ratio-level measurement is the only level that allows meaningful statements about ratios of scores. The higher the level of measurement, the more complex statistical tests we are able to conduct. This knowledge may help us decide what kind of data we need to gather, and how.
That said, we have to balance this knowledge with the understanding that sometimes, collecting data at a higher level of measurement could negatively impact our studies. For instance, sometimes providing answers in ranges may make prospective participants feel more comfortable responding to sensitive items. Imagine that you were interested in collecting information on topics such as income, number of sexual partners, number of times someone used illicit drugs, etc. You would have to think about the sensitivity of these items and determine if it would make more sense to collect some data at a lower level of measurement (e.g., asking if they are sexually active or not (nominal) versus their total number of sexual partners (ratio).
Finally, sometimes when analyzing data, researchers find a need to change a data's level of measurement. For example, a few years ago, a student was interested in studying the relationship between mental health and life satisfaction. This student used a variety of measures. One item asked about the number of mental health symptoms, reported as the actual number. When analyzing data, my student examined the mental health symptom variable and noticed that she had two groups, those with none or one symptoms and those with many symptoms. Instead of using the ratio level data (actual number of mental health symptoms), she collapsed her cases into two categories, few and many. She decided to use this variable in her analyses. It is important to note that you can move a higher level of data to a lower level of data; however, you are unable to move a lower level to a higher level.
Exercises
- Check that the variables in your research question can vary...and that they are not constants or one of many potential attributes of a variable.
- Think about the attributes your variables have. Are they categorical or continuous? What level of measurement seems most appropriate?
Step 2: Specifying measures for each variable
Let’s pick a social work research question and walk through the process of operationalizing variables to see how specific we need to get. I’m going to hypothesize that residents of a psychiatric unit who are more depressed are less likely to be satisfied with care. Remember, this would be a inverse relationship—as depression increases, satisfaction decreases. In this question, depression is my independent variable (the cause) and satisfaction with care is my dependent variable (the effect). Now we have identified our variables, their attributes, and levels of measurement, we move onto the second component: the measure itself.
So, how would you measure my key variables: depression and satisfaction? What indicators would you look for? Some students might say that depression could be measured by observing a participant’s body language. They may also say that a depressed person will often express feelings of sadness or hopelessness. In addition, a satisfied person might be happy around service providers and often express gratitude. While these factors may indicate that the variables are present, they lack coherence. Unfortunately, what this “measure” is actually saying is that “I know depression and satisfaction when I see them.” While you are likely a decent judge of depression and satisfaction, you need to provide more information in a research study for how you plan to measure your variables. Your judgment is subjective, based on your own idiosyncratic experiences with depression and satisfaction. They couldn’t be replicated by another researcher. They also can’t be done consistently for a large group of people. Operationalization requires that you come up with a specific and rigorous measure for seeing who is depressed or satisfied.
Finding a good measure for your variable depends on the kind of variable it is. Variables that are directly observable don't come up very often in my students' classroom projects, but they might include things like taking someone's blood pressure, marking attendance or participation in a group, and so forth. To measure an indirectly observable variable like age, you would probably put a question on a survey that asked, “How old are you?” Measuring a variable like income might require some more thought, though. Are you interested in this person’s individual income or the income of their family unit? This might matter if your participant does not work or is dependent on other family members for income. Do you count income from social welfare programs? Are you interested in their income per month or per year? Even though indirect observables are relatively easy to measure, the measures you use must be clear in what they are asking, and operationalization is all about figuring out the specifics of what you want to know. For more complicated constructs, you will need compound measures (that use multiple indicators to measure a single variable).
How you plan to collect your data also influences how you will measure your variables. For social work researchers using secondary data like client records as a data source, you are limited by what information is in the data sources you can access. If your organization uses a given measurement for a mental health outcome, that is the one you will use in your study. Similarly, if you plan to study how long a client was housed after an intervention using client visit records, you are limited by how their caseworker recorded their housing status in the chart. One of the benefits of collecting your own data is being able to select the measures you feel best exemplify your understanding of the topic.
Measuring unidimensional concepts
The previous section mentioned two important considerations: how complicated the variable is and how you plan to collect your data. With these in hand, we can use the level of measurement to further specify how you will measure your variables and consider specialized rating scales developed by social science researchers.
Measurement at each level
Nominal measures assess categorical variables. These measures are used for variables or indicators that have mutually exclusive attributes, but that cannot be rank-ordered. Nominal measures ask about the variable and provide names or labels for different attribute values like social work, counseling, and nursing for the variable profession. Nominal measures are relatively straightforward.
Ordinal measures often use a rating scale. It is an ordered set of responses that participants must choose from. Figure 11.1 shows several examples. The number of response options on a typical rating scale is usualy five or seven, though it can range from three to 11. Five-point scales are best for unipolar scales where only one construct is tested, such as frequency (Never, Rarely, Sometimes, Often, Always). Seven-point scales are best for bipolar scales where there is a dichotomous spectrum, such as liking (Like very much, Like somewhat, Like slightly, Neither like nor dislike, Dislike slightly, Dislike somewhat, Dislike very much). For bipolar questions, it is useful to offer an earlier question that branches them into an area of the scale; if asking about liking ice cream, first ask “Do you generally like or dislike ice cream?” Once the respondent chooses like or dislike, refine it by offering them relevant choices from the seven-point scale. Branching improves both reliability and validity (Krosnick & Berent, 1993).[42] Although you often see scales with numerical labels, it is best to only present verbal labels to the respondents but convert them to numerical values in the analyses. Avoid partial labels or length or overly specific labels. In some cases, the verbal labels can be supplemented with (or even replaced by) meaningful graphics. The last rating scale shown in Figure 11.1 is a visual-analog scale, on which participants make a mark somewhere along the horizontal line to indicate the magnitude of their response.
Interval measures are those where the values measured are not only rank-ordered, but are also equidistant from adjacent attributes. For example, the temperature scale (in Fahrenheit or Celsius), where the difference between 30 and 40 degree Fahrenheit is the same as that between 80 and 90 degree Fahrenheit. Likewise, if you have a scale that asks respondents’ annual income using the following attributes (ranges): $0 to 10,000, $10,000 to 20,000, $20,000 to 30,000, and so forth, this is also an interval measure, because the mid-point of each range (i.e., $5,000, $15,000, $25,000, etc.) are equidistant from each other. The intelligence quotient (IQ) scale is also an interval measure, because the measure is designed such that the difference between IQ scores 100 and 110 is supposed to be the same as between 110 and 120 (although we do not really know whether that is truly the case). Interval measures allow us to examine “how much more” is one attribute when compared to another, which is not possible with nominal or ordinal measures. You may find researchers who “pretend” (incorrectly) that ordinal rating scales are actually interval measures so that we can use different statistical techniques for analyzing them. As we will discuss in the latter part of the chapter, this is a mistake because there is no way to know whether the difference between a 3 and a 4 on a rating scale is the same as the difference between a 2 and a 3. Those numbers are just placeholders for categories.
Ratio measures are those that have all the qualities of nominal, ordinal, and interval scales, and in addition, also have a “true zero” point (where the value zero implies lack or non-availability of the underlying construct). Think about how to measure the number of people working in human resources at a social work agency. It could be one, several, or none (if the company contracts out for those services). Measuring interval and ratio data is relatively easy, as people either select or input a number for their answer. If you ask a person how many eggs they purchased last week, they can simply tell you they purchased `a dozen eggs at the store, two at breakfast on Wednesday, or none at all.
Commonly used rating scales in questionnaires
The level of measurement will give you the basic information you need, but social scientists have developed specialized instruments for use in questionnaires, a common tool used in quantitative research. As we mentioned before, if you plan to source your data from client files or previously published results
Although Likert scale is a term colloquially used to refer to almost any rating scale (e.g., a 0-to-10 life satisfaction scale), it has a much more precise meaning. In the 1930s, researcher Rensis Likert (pronounced LICK-ert) created a new approach for measuring people’s attitudes (Likert, 1932).[43] It involves presenting people with several statements—including both favorable and unfavorable statements—about some person, group, or idea. Respondents then express their agreement or disagreement with each statement on a 5-point scale: Strongly Agree, Agree, Neither Agree nor Disagree, Disagree, Strongly Disagree. Numbers are assigned to each response and then summed across all items to produce a score representing the attitude toward the person, group, or idea. For items that are phrased in an opposite direction (e.g., negatively worded statements instead of positively worded statements), reverse coding is used so that the numerical scoring of statements also runs in the opposite direction. The entire set of items came to be called a Likert scale, as indicated in Table 11.2 below.
Unless you are measuring people’s attitude toward something by assessing their level of agreement with several statements about it, it is best to avoid calling it a Likert scale. You are probably just using a rating scale. Likert scales allow for more granularity (more finely tuned response) than yes/no items, including whether respondents are neutral to the statement. Below is an example of how we might use a Likert scale to assess your attitudes about research as you work your way through this textbook.
| Strongly agree | Agree | Neutral | Disagree | Strongly disagree | |
| I like research more now than when I started reading this book. | |||||
| This textbook is easy to use. | |||||
| I feel confident about how well I understand levels of measurement. | |||||
| This textbook is helping me plan my research proposal. |
Semantic differential scales are composite (multi-item) scales in which respondents are asked to indicate their opinions or feelings toward a single statement using different pairs of adjectives framed as polar opposites. Whereas in the above Likert scale, the participant is asked how much they agree or disagree with a statement, in a semantic differential scale the participant is asked to indicate how they feel about a specific item. This makes the semantic differential scale an excellent technique for measuring people’s attitudes or feelings toward objects, events, or behaviors. Table 11.3 is an example of a semantic differential scale that was created to assess participants' feelings about this textbook.
| 1) How would you rate your opinions toward this textbook? | ||||||
| Very much | Somewhat | Neither | Somewhat | Very much | ||
| Boring | Exciting | |||||
| Useless | Useful | |||||
| Hard | Easy | |||||
| Irrelevant | Applicable | |||||
This composite scale was designed by Louis Guttman and uses a series of items arranged in increasing order of intensity (least intense to most intense) of the concept. This type of scale allows us to understand the intensity of beliefs or feelings. Each item in the above Guttman scale has a weight (this is not indicated on the tool) which varies with the intensity of that item, and the weighted combination of each response is used as an aggregate measure of an observation.
Example Guttman Scale Items
- I often felt the material was not engaging Yes/No
- I was often thinking about other things in class Yes/No
- I was often working on other tasks during class Yes/No
- I will work to abolish research from the curriculum Yes/No
Notice how the items move from lower intensity to higher intensity. A researcher reviews the yes answers and creates a score for each participant.
Composite measures: Scales and indices
Depending on your research design, your measure may be something you put on a survey or pre/post-test that you give to your participants. For a variable like age or income, one well-worded question may suffice. Unfortunately, most variables in the social world are not so simple. Depression and satisfaction are multidimensional concepts. Relying on a single indicator like a question that asks "Yes or no, are you depressed?” does not encompass the complexity of depression, including issues with mood, sleeping, eating, relationships, and happiness. There is no easy way to delineate between multidimensional and unidimensional concepts, as its all in how you think about your variable. Satisfaction could be validly measured using a unidimensional ordinal rating scale. However, if satisfaction were a key variable in our study, we would need a theoretical framework and conceptual definition for it. That means we'd probably have more indicators to ask about like timeliness, respect, sensitivity, and many others, and we would want our study to say something about what satisfaction truly means in terms of our other key variables. However, if satisfaction is not a key variable in your conceptual framework, it makes sense to operationalize it as a unidimensional concept.
For more complicated measures, researchers use scales and indices (sometimes called indexes) to measure their variables because they assess multiple indicators to develop a composite (or total) score. Composite scores provide a much greater understanding of concepts than a single item could. Although we won't delve too deeply into the process of scale development, we will cover some important topics for you to understand how scales and indices developed by other researchers can be used in your project.
Although they exhibit differences (which will later be discussed) the two have in common various factors.
- Both are ordinal measures of variables.
- Both can order the units of analysis in terms of specific variables.
- Both are composite measures.
Scales
The previous section discussed how to measure respondents’ responses to predesigned items or indicators belonging to an underlying construct. But how do we create the indicators themselves? The process of creating the indicators is called scaling. More formally, scaling is a branch of measurement that involves the construction of measures by associating qualitative judgments about unobservable constructs with quantitative, measurable metric units. Stevens (1946)[44] said, “Scaling is the assignment of objects to numbers according to a rule.” This process of measuring abstract concepts in concrete terms remains one of the most difficult tasks in empirical social science research.
The outcome of a scaling process is a scale, which is an empirical structure for measuring items or indicators of a given construct. Understand that multidimensional “scales”, as discussed in this section, are a little different from “rating scales” discussed in the previous section. A rating scale is used to capture the respondents’ reactions to a given item on a questionnaire. For example, an ordinally scaled item captures a value between “strongly disagree” to “strongly agree.” Attaching a rating scale to a statement or instrument is not scaling. Rather, scaling is the formal process of developing scale items, before rating scales can be attached to those items.
If creating your own scale sounds painful, don’t worry! For most multidimensional variables, you would likely be duplicating work that has already been done by other researchers. Specifically, this is a branch of science called psychometrics. You do not need to create a scale for depression because scales such as the Patient Health Questionnaire (PHQ-9), the Center for Epidemiologic Studies Depression Scale (CES-D), and Beck’s Depression Inventory (BDI) have been developed and refined over dozens of years to measure variables like depression. Similarly, scales such as the Patient Satisfaction Questionnaire (PSQ-18) have been developed to measure satisfaction with medical care. As we will discuss in the next section, these scales have been shown to be reliable and valid. While you could create a new scale to measure depression or satisfaction, a study with rigor would pilot test and refine that new scale over time to make sure it measures the concept accurately and consistently. This high level of rigor is often unachievable in student research projects because of the cost and time involved in pilot testing and validating, so using existing scales is recommended.
Unfortunately, there is no good one-stop=shop for psychometric scales. The Mental Measurements Yearbook provides a searchable database of measures for social science variables, though it woefully incomplete and often does not contain the full documentation for scales in its database. You can access it from a university library’s list of databases. If you can’t find anything in there, your next stop should be the methods section of the articles in your literature review. The methods section of each article will detail how the researchers measured their variables, and often the results section is instructive for understanding more about measures. In a quantitative study, researchers may have used a scale to measure key variables and will provide a brief description of that scale, its names, and maybe a few example questions. If you need more information, look at the results section and tables discussing the scale to get a better idea of how the measure works. Looking beyond the articles in your literature review, searching Google Scholar using queries like “depression scale” or “satisfaction scale” should also provide some relevant results. For example, searching for documentation for the Rosenberg Self-Esteem Scale (which we will discuss in the next section), I found this report from researchers investigating acceptance and commitment therapy which details this scale and many others used to assess mental health outcomes. If you find the name of the scale somewhere but cannot find the documentation (all questions and answers plus how to interpret the scale), a general web search with the name of the scale and ".pdf" may bring you to what you need. Or, to get professional help with finding information, always ask a librarian!
Unfortunately, these approaches do not guarantee that you will be able to view the scale itself or get information on how it is interpreted. Many scales cost money to use and may require training to properly administer. You may also find scales that are related to your variable but would need to be slightly modified to match your study’s needs. You could adapt a scale to fit your study, however changing even small parts of a scale can influence its accuracy and consistency. While it is perfectly acceptable in student projects to adapt a scale without testing it first (time may not allow you to do so), pilot testing is always recommended for adapted scales, and researchers seeking to draw valid conclusions and publish their results must take this additional step.
Indices
An index is a composite score derived from aggregating measures of multiple concepts (called components) using a set of rules and formulas. It is different from a scale. Scales also aggregate measures; however, these measures examine different dimensions or the same dimension of a single construct. A well-known example of an index is the consumer price index (CPI), which is computed every month by the Bureau of Labor Statistics of the U.S. Department of Labor. The CPI is a measure of how much consumers have to pay for goods and services (in general) and is divided into eight major categories (food and beverages, housing, apparel, transportation, healthcare, recreation, education and communication, and “other goods and services”), which are further subdivided into more than 200 smaller items. Each month, government employees call all over the country to get the current prices of more than 80,000 items. Using a complicated weighting scheme that takes into account the location and probability of purchase for each item, analysts then combine these prices into an overall index score using a series of formulas and rules.
Another example of an index is the Duncan Socioeconomic Index (SEI). This index is used to quantify a person's socioeconomic status (SES) and is a combination of three concepts: income, education, and occupation. Income is measured in dollars, education in years or degrees achieved, and occupation is classified into categories or levels by status. These very different measures are combined to create an overall SES index score. However, SES index measurement has generated a lot of controversy and disagreement among researchers.
The process of creating an index is similar to that of a scale. First, conceptualize (define) the index and its constituent components. Though this appears simple, there may be a lot of disagreement on what components (concepts/constructs) should be included or excluded from an index. For instance, in the SES index, isn’t income correlated with education and occupation? And if so, should we include one component only or all three components? Reviewing the literature, using theories, and/or interviewing experts or key stakeholders may help resolve this issue. Second, operationalize and measure each component. For instance, how will you categorize occupations, particularly since some occupations may have changed with time (e.g., there were no Web developers before the Internet)? As we will see in step three below, researchers must create a rule or formula for calculating the index score. Again, this process may involve a lot of subjectivity, so validating the index score using existing or new data is important.
Scale and index development at often taught in their own course in doctoral education, so it is unreasonable for you to expect to develop a consistently accurate measure within the span of a week or two. Using available indices and scales is recommended for this reason.
Differences between scales and indices
Though indices and scales yield a single numerical score or value representing a concept of interest, they are different in many ways. First, indices often comprise components that are very different from each other (e.g., income, education, and occupation in the SES index) and are measured in different ways. Conversely, scales typically involve a set of similar items that use the same rating scale (such as a five-point Likert scale about customer satisfaction).
Second, indices often combine objectively measurable values such as prices or income, while scales are designed to assess subjective or judgmental constructs such as attitude, prejudice, or self-esteem. Some argue that the sophistication of the scaling methodology makes scales different from indexes, while others suggest that indexing methodology can be equally sophisticated. Nevertheless, indexes and scales are both essential tools in social science research.
Scales and indices seem like clean, convenient ways to measure different phenomena in social science, but just like with a lot of research, we have to be mindful of the assumptions and biases underneath. What if a scale or an index was developed using only White women as research participants? Is it going to be useful for other groups? It very well might be, but when using a scale or index on a group for whom it hasn't been tested, it will be very important to evaluate the validity and reliability of the instrument, which we address in the rest of the chapter.
Finally, it's important to note that while scales and indices are often made up of nominal or ordinal variables, when we analyze them into composite scores, we will treat them as interval/ratio variables.
Exercises
- Look back to your work from the previous section, are your variables unidimensional or multidimensional?
- Describe the specific measures you will use (actual questions and response options you will use with participants) for each variable in your research question.
- If you are using a measure developed by another researcher but do not have all of the questions, response options, and instructions needed to implement it, put it on your to-do list to get them.
Step 3: How you will interpret your measures
The final stage of operationalization involves setting the rules for how the measure works and how the researcher should interpret the results. Sometimes, interpreting a measure can be incredibly easy. If you ask someone their age, you’ll probably interpret the results by noting the raw number (e.g., 22) someone provides and that it is lower or higher than other people's ages. However, you could also recode that person into age categories (e.g., under 25, 20-29-years-old, generation Z, etc.). Even scales may be simple to interpret. If there is a scale of problem behaviors, one might simply add up the number of behaviors checked off–with a range from 1-5 indicating low risk of delinquent behavior, 6-10 indicating the student is moderate risk, etc. How you choose to interpret your measures should be guided by how they were designed, how you conceptualize your variables, the data sources you used, and your plan for analyzing your data statistically. Whatever measure you use, you need a set of rules for how to take any valid answer a respondent provides to your measure and interpret it in terms of the variable being measured.
For more complicated measures like scales, refer to the information provided by the author for how to interpret the scale. If you can’t find enough information from the scale’s creator, look at how the results of that scale are reported in the results section of research articles. For example, Beck’s Depression Inventory (BDI-II) uses 21 statements to measure depression and respondents rate their level of agreement on a scale of 0-3. The results for each question are added up, and the respondent is put into one of three categories: low levels of depression (1-16), moderate levels of depression (17-30), or severe levels of depression (31 and over).
One common mistake I see often is that students will introduce another variable into their operational definition. This is incorrect. Your operational definition should mention only one variable—the variable being defined. While your study will certainly draw conclusions about the relationships between variables, that's not what operationalization is. Operationalization specifies what instrument you will use to measure your variable and how you plan to interpret the data collected using that measure.
Operationalization is probably the trickiest component of basic research methods, so please don’t get frustrated if it takes a few drafts and a lot of feedback to get to a workable definition. At the time of this writing, I am in the process of operationalizing the concept of “attitudes towards research methods.” Originally, I thought that I could gauge students’ attitudes toward research methods by looking at their end-of-semester course evaluations. As I became aware of the potential methodological issues with student course evaluations, I opted to use focus groups of students to measure their common beliefs about research. You may recall some of these opinions from Chapter 1, such as the common beliefs that research is boring, useless, and too difficult. After the focus group, I created a scale based on the opinions I gathered, and I plan to pilot test it with another group of students. After the pilot test, I expect that I will have to revise the scale again before I can implement the measure in a real social work research project. At the time I’m writing this, I’m still not completely done operationalizing this concept.
Key Takeaways
- Operationalization involves spelling out precisely how a concept will be measured.
- Operational definitions must include the variable, the measure, and how you plan to interpret the measure.
- There are four different levels of measurement: nominal, ordinal, interval, and ratio (in increasing order of specificity).
- Scales and indices are common ways to collect information and involve using multiple indicators in measurement.
- A key difference between a scale and an index is that a scale contains multiple indicators for one concept, whereas an indicator examines multiple concepts (components).
- Using scales developed and refined by other researchers can improve the rigor of a quantitative study.
Exercises
Use the research question that you developed in the previous chapters and find a related scale or index that researchers have used. If you have trouble finding the exact phenomenon you want to study, get as close as you can.
- What is the level of measurement for each item on each tool? Take a second and think about why the tool's creator decided to include these levels of measurement. Identify any levels of measurement you would change and why.
- If these tools don't exist for what you are interested in studying, why do you think that is?
12.3 Writing effective questions and questionnaires
Learning Objectives
Learners will be able to...
- Describe some of the ways that survey questions might confuse respondents and how to word questions and responses clearly
- Create mutually exclusive, exhaustive, and balanced response options
- Define fence-sitting and floating
- Describe the considerations involved in constructing a well-designed questionnaire
- Discuss why pilot testing is important
In the previous section, we reviewed how researchers collect data using surveys. Guided by their sampling approach and research context, researchers should choose the survey approach that provides the most favorable tradeoffs in strengths and challenges. With this information in hand, researchers need to write their questionnaire and revise it before beginning data collection. Each method of delivery requires a questionnaire, but they vary a bit based on how they will be used by the researcher. Since phone surveys are read aloud, researchers will pay more attention to how the questionnaire sounds than how it looks. Online surveys can use advanced tools to require the completion of certain questions, present interactive questions and answers, and otherwise afford greater flexibility in how questionnaires are designed. As you read this section, consider how your method of delivery impacts the type of questionnaire you will design. Because most student projects use paper or online surveys, this section will detail how to construct self-administered questionnaires to minimize the potential for bias and error.
Start with operationalization
The first thing you need to do to write effective survey questions is identify what exactly you wish to know. As silly as it sounds to state what seems so completely obvious, we can’t stress enough how easy it is to forget to include important questions when designing a survey. Begin by looking at your research question and refreshing your memory of the operational definitions you developed for those variables from Chapter 11. You should have a pretty firm grasp of your operational definitions before starting the process of questionnaire design. You may have taken those operational definitions from other researchers' methods, found established scales and indices for your measures, or created your own questions and answer options.
Exercises
STOP! Make sure you have a complete operational definition for the dependent and independent variables in your research question. A complete operational definition contains the variable being measured, the measure used, and how the researcher interprets the measure. Let's make sure you have what you need from Chapter 11 to begin writing your questionnaire.
List all of the dependent and independent variables in your research question.
- It's normal to have one dependent or independent variable. It's also normal to have more than one of either.
- Make sure that your research question (and this list) contain all of the variables in your hypothesis. Your hypothesis should only include variables from you research question.
For each variable in your list:
- Write out the measure you will use (the specific questions and answers) for each variable.
- If you don't have questions and answers finalized yet, write a first draft and revise it based on what you read in this section.
- If you are using a measure from another researcher, you should be able to write out all of the questions and answers associated with that measure. If you only have the name of a scale or a few questions, you need to access to the full text and some documentation on how to administer and interpret it before you can finish your questionnaire.
- Describe how you will use each measure draw conclusions about the variable in the operational definition.
- For example, an interpretation might be "there are five 7-point Likert scale questions...point values are added across all five items for each participant...and scores below 10 indicate the participant has low self-esteem"
- Don't introduce other variables into the mix here. All we are concerned with is how you will measure each variable by itself. The connection between variables is done using statistical tests, not operational definitions.
- Detail any validity or reliability issues uncovered by previous researchers using the same measures. If you have concerns about validity and reliability, note them, as well.
If you completed the exercise above and listed out all of the questions and answer choices you will use to measure the variables in your research question, you have already produced a pretty solid first draft of your questionnaire! Congrats! In essence, questionnaires are all of the self-report measures in your operational definitions for the independent, dependent, and control variables in your study arranged into one document and administered to participants. There are a few questions on a questionnaire (like name or ID#) that are not associated with the measurement of variables. These are the exception, and it's useful to think of a questionnaire as a list of measures for variables. Of course, researchers often use more than one measure of a variable (i.e., triangulation) so they can more confidently assert that their findings are true. A questionnaire should contain all of the measures researchers plan to collect about their variables by asking participants to self-report. As we will discuss in the final section of this chapter, triangulating across data sources (e.g., measuring variables using client files or student records) can avoid some of the common sources of bias in survey research.
Sticking close to your operational definitions is important because it helps you avoid an everything-but-the-kitchen-sink approach that includes every possible question that occurs to you. Doing so puts an unnecessary burden on your survey respondents. Remember that you have asked your participants to give you their time and attention and to take care in responding to your questions; show them your respect by only asking questions that you actually plan to use in your analysis. For each question in your questionnaire, ask yourself how this question measures a variable in your study. An operational definition should contain the questions, response options, and how the researcher will draw conclusions about the variable based on participants' responses.
Writing questions
So, almost all of the questions on a questionnaire are measuring some variable. For many variables, researchers will create their own questions rather than using one from another researcher. This section will provide some tips on how to create good questions to accurately measure variables in your study. First, questions should be as clear and to the point as possible. This is not the time to show off your creative writing skills; a survey is a technical instrument and should be written in a way that is as direct and concise as possible. As I’ve mentioned earlier, your survey respondents have agreed to give their time and attention to your survey. The best way to show your appreciation for their time is to not waste it. Ensuring that your questions are clear and concise will go a long way toward showing your respondents the gratitude they deserve. Pilot testing the questionnaire with friends or colleagues can help identify these issues. This process is commonly called pretesting, but to avoid any confusion with pretesting in experimental design, we refer to it as pilot testing.
Related to the point about not wasting respondents’ time, make sure that every question you pose will be relevant to every person you ask to complete it. This means two things: first, that respondents have knowledge about whatever topic you are asking them about, and second, that respondents have experienced the events, behaviors, or feelings you are asking them to report. If you are asking participants for second-hand knowledge—asking clinicians about clients' feelings, asking teachers about students' feelings, and so forth—you may want to clarify that the variable you are asking about is the key informant's perception of what is happening in the target population. A well-planned sampling approach ensures that participants are the most knowledgeable population to complete your survey.
If you decide that you do wish to include questions about matters with which only a portion of respondents will have had experience, make sure you know why you are doing so. For example, if you are asking about MSW student study patterns, and you decide to include a question on studying for the social work licensing exam, you may only have a small subset of participants who have begun studying for the graduate exam or took the bachelor's-level exam. If you decide to include this question that speaks to a minority of participants' experiences, think about why you are including it. Are you interested in how studying for class and studying for licensure differ? Are you trying to triangulate study skills measures? Researchers should carefully consider whether questions relevant to only a subset of participants is likely to produce enough valid responses for quantitative analysis.
Many times, questions that are relevant to a subsample of participants are conditional on an answer to a previous question. A participant might select that they rent their home, and as a result, you might ask whether they carry renter's insurance. That question is not relevant to homeowners, so it would be wise not to ask them to respond to it. In that case, the question of whether someone rents or owns their home is a filter question, designed to identify some subset of survey respondents who are asked additional questions that are not relevant to the entire sample. Figure 12.1 presents an example of how to accomplish this on a paper survey by adding instructions to the participant that indicate what question to proceed to next based on their response to the first one. Using online survey tools, researchers can use filter questions to only present relevant questions to participants.
Researchers should eliminate questions that ask about things participants don't know to minimize confusion. Assuming the question is relevant to the participant, other sources of confusion come from how the question is worded. The use of negative wording can be a source of potential confusion. Taking the question from Figure 12.1 about drinking as our example, what if we had instead asked, “Did you not abstain from drinking during your first semester of college?” This is a double negative, and it's not clear how to answer the question accurately. It is a good idea to avoid negative phrasing, when possible. For example, "did you not drink alcohol during your first semester of college?" is less clear than "did you drink alcohol your first semester of college?"
You should also avoid using terms or phrases that may be regionally or culturally specific (unless you are absolutely certain all your respondents come from the region or culture whose terms you are using). When I first moved to southwest Virginia, I didn’t know what a holler was. Where I grew up in New Jersey, to holler means to yell. Even then, in New Jersey, we shouted and screamed, but we didn’t holler much. In southwest Virginia, my home at the time, a holler also means a small valley in between the mountains. If I used holler in that way on my survey, people who live near me may understand, but almost everyone else would be totally confused. A similar issue arises when you use jargon, or technical language, that people do not commonly know. For example, if you asked adolescents how they experience imaginary audience, they would find it difficult to link those words to the concepts from David Elkind’s theory. The words you use in your questions must be understandable to your participants. If you find yourself using jargon or slang, break it down into terms that are more universal and easier to understand.
Asking multiple questions as though they are a single question can also confuse survey respondents. There’s a specific term for this sort of question; it is called a double-barreled question. Figure 12.2 shows a double-barreled question. Do you see what makes the question double-barreled? How would someone respond if they felt their college classes were more demanding but also more boring than their high school classes? Or less demanding but more interesting? Because the question combines “demanding” and “interesting,” there is no way to respond yes to one criterion but no to the other.
Another thing to avoid when constructing survey questions is the problem of social desirability. We all want to look good, right? And we all probably know the politically correct response to a variety of questions whether we agree with the politically correct response or not. In survey research, social desirability refers to the idea that respondents will try to answer questions in a way that will present them in a favorable light. (You may recall we covered social desirability bias in Chapter 11.)
Perhaps we decide that to understand the transition to college, we need to know whether respondents ever cheated on an exam in high school or college for our research project. We all know that cheating on exams is generally frowned upon (at least I hope we all know this). So, it may be difficult to get people to admit to cheating on a survey. But if you can guarantee respondents’ confidentiality, or even better, their anonymity, chances are much better that they will be honest about having engaged in this socially undesirable behavior. Another way to avoid problems of social desirability is to try to phrase difficult questions in the most benign way possible. Earl Babbie (2010) [45] offers a useful suggestion for helping you do this—simply imagine how you would feel responding to your survey questions. If you would be uncomfortable, chances are others would as well.
Exercises
Try to step outside your role as researcher for a second, and imagine you were one of your participants. Evaluate the following:
- Is the question too general? Sometimes, questions that are too general may not accurately convey respondents’ perceptions. If you asked someone how they liked a certain book and provide a response scale ranging from “not at all” to “extremely well”, and if that person selected “extremely well," what do they mean? Instead, ask more specific behavioral questions, such as "Will you recommend this book to others?" or "Do you plan to read other books by the same author?"
- Is the question too detailed? Avoid unnecessarily detailed questions that serve no specific research purpose. For instance, do you need the age of each child in a household or is just the number of children in the household acceptable? However, if unsure, it is better to err on the side of details than generality.
- Is the question presumptuous? Does your question make assumptions? For instance, if you ask, "what do you think the benefits of a tax cut would be?" you are presuming that the participant sees the tax cut as beneficial. But many people may not view tax cuts as beneficial. Some might see tax cuts as a precursor to less funding for public schools and fewer public services such as police, ambulance, and fire department. Avoid questions with built-in presumptions.
- Does the question ask the participant to imagine something? Is the question imaginary? A popular question on many television game shows is “if you won a million dollars on this show, how will you plan to spend it?” Most participants have never been faced with this large amount of money and have never thought about this scenario. In fact, most don’t even know that after taxes, the value of the million dollars will be greatly reduced. In addition, some game shows spread the amount over a 20-year period. Without understanding this "imaginary" situation, participants may not have the background information necessary to provide a meaningful response.
Finally, it is important to get feedback on your survey questions from as many people as possible, especially people who are like those in your sample. Now is not the time to be shy. Ask your friends for help, ask your mentors for feedback, ask your family to take a look at your survey as well. The more feedback you can get on your survey questions, the better the chances that you will come up with a set of questions that are understandable to a wide variety of people and, most importantly, to those in your sample.
In sum, in order to pose effective survey questions, researchers should do the following:
- Identify how each question measures an independent, dependent, or control variable in their study.
- Keep questions clear and succinct.
- Make sure respondents have relevant lived experience to provide informed answers to your questions.
- Use filter questions to avoid getting answers from uninformed participants.
- Avoid questions that are likely to confuse respondents—including those that use double negatives, use culturally specific terms or jargon, and pose more than one question at a time.
- Imagine how respondents would feel responding to questions.
- Get feedback, especially from people who resemble those in the researcher’s sample.
Exercises
Let's complete a first draft of your questions. In the previous exercise, you listed all of the questions and answers you will use to measure the variables in your research question.
- In the previous exercise, you wrote out the questions and answers for each measure of your independent and dependent variables. Evaluate each question using the criteria listed above on effective survey questions.
- Type out questions for your control variables and evaluate them, as well. Consider what response options you want to offer participants.
Now, let's revise any questions that do not meet your standards!
- Use the BRUSO model in Table 12.2 for an illustration of how to address deficits in question wording. Keep in mind that you are writing a first draft in this exercise, and it will take a few drafts and revisions before your questions are ready to distribute to participants.
| Criterion | Poor | Effective |
| B- Brief | “Are you now or have you ever been the possessor of a firearm?” | Have you ever possessed a firearm? |
| R- Relevant | "Who did you vote for in the last election?" | Note: Only include items that are relevant to your study. |
| U- Unambiguous | “Are you a gun person?” | Do you currently own a gun?” |
| S- Specific | How much have you read about the new gun control measure and sales tax?” | “How much have you read about the new sales tax on firearm purchases?” |
| O- Objective | “How much do you support the beneficial new gun control measure?” | “What is your view of the new gun control measure?” |
Writing response options
While posing clear and understandable questions in your survey is certainly important, so too is providing respondents with unambiguous response options. Response options are the answers that you provide to the people completing your questionnaire. Generally, respondents will be asked to choose a single (or best) response to each question you pose. We call questions in which the researcher provides all of the response options closed-ended questions. Keep in mind, closed-ended questions can also instruct respondents to choose multiple response options, rank response options against one another, or assign a percentage to each response option. But be cautious when experimenting with different response options! Accepting multiple responses to a single question may add complexity when it comes to quantitatively analyzing and interpreting your data.
Surveys need not be limited to closed-ended questions. Sometimes survey researchers include open-ended questions in their survey instruments as a way to gather additional details from respondents. An open-ended question does not include response options; instead, respondents are asked to reply to the question in their own way, using their own words. These questions are generally used to find out more about a survey participant’s experiences or feelings about whatever they are being asked to report in the survey. If, for example, a survey includes closed-ended questions asking respondents to report on their involvement in extracurricular activities during college, an open-ended question could ask respondents why they participated in those activities or what they gained from their participation. While responses to such questions may also be captured using a closed-ended format, allowing participants to share some of their responses in their own words can make the experience of completing the survey more satisfying to respondents and can also reveal new motivations or explanations that had not occurred to the researcher. This is particularly important for mixed-methods research. It is possible to analyze open-ended response options quantitatively using content analysis (i.e., counting how often a theme is represented in a transcript looking for statistical patterns). However, for most researchers, qualitative data analysis will be needed to analyze open-ended questions, and researchers need to think through how they will analyze any open-ended questions as part of their data analysis plan. We will address qualitative data analysis in greater detail in Chapter 19.
To keep things simple, we encourage you to use only closed-ended response options in your study. While open-ended questions are not wrong, they are often a sign in our classrooms that students have not thought through all the way how to operationally define and measure their key variables. Open-ended questions cannot be operationally defined because you don't know what responses you will get. Instead, you will need to analyze the qualitative data using one of the techniques we discuss in Chapter 19 to interpret your participants' responses.
To write an effective response options for closed-ended questions, there are a couple of guidelines worth following. First, be sure that your response options are mutually exclusive. Look back at Figure 12.1, which contains questions about how often and how many drinks respondents consumed. Do you notice that there are no overlapping categories in the response options for these questions? This is another one of those points about question construction that seems fairly obvious but that can be easily overlooked. Response options should also be exhaustive. In other words, every possible response should be covered in the set of response options that you provide. For example, note that in question 10a in Figure 12.1, we have covered all possibilities—those who drank, say, an average of once per month can choose the first response option (“less than one time per week”) while those who drank multiple times a day each day of the week can choose the last response option (“7+”). All the possibilities in between these two extremes are covered by the middle three response options, and every respondent fits into one of the response options we provided.
Earlier in this section, we discussed double-barreled questions. Response options can also be double barreled, and this should be avoided. Figure 12.3 is an example of a question that uses double-barreled response options. Other tips about questions are also relevant to response options, including that participants should be knowledgeable enough to select or decline a response option as well as avoiding jargon and cultural idioms.
Even if you phrase questions and response options clearly, participants are influenced by how many response options are presented on the questionnaire. For Likert scales, five or seven response options generally allow about as much precision as respondents are capable of. However, numerical scales with more options can sometimes be appropriate. For dimensions such as attractiveness, pain, and likelihood, a 0-to-10 scale will be familiar to many respondents and easy for them to use. Regardless of the number of response options, the most extreme ones should generally be “balanced” around a neutral or modal midpoint. An example of an unbalanced rating scale measuring perceived likelihood might look like this:
Unlikely | Somewhat Likely | Likely | Very Likely | Extremely Likely
Because we have four rankings of likely and only one ranking of unlikely, the scale is unbalanced and most responses will be biased toward "likely" rather than "unlikely." A balanced version might look like this:
Extremely Unlikely | Somewhat Unlikely | As Likely as Not | Somewhat Likely |Extremely Likely
In this example, the midpoint is halfway between likely and unlikely. Of course, a middle or neutral response option does not have to be included. Researchers sometimes choose to leave it out because they want to encourage respondents to think more deeply about their response and not simply choose the middle option by default. Fence-sitters are respondents who choose neutral response options, even if they have an opinion. Some people will be drawn to respond, “no opinion” even if they have an opinion, particularly if their true opinion is the not a socially desirable opinion. Floaters, on the other hand, are those that choose a substantive answer to a question when really, they don’t understand the question or don’t have an opinion.
As you can see, floating is the flip side of fence-sitting. Thus, the solution to one problem is often the cause of the other. How you decide which approach to take depends on the goals of your research. Sometimes researchers specifically want to learn something about people who claim to have no opinion. In this case, allowing for fence-sitting would be necessary. Other times researchers feel confident their respondents will all be familiar with every topic in their survey. In this case, perhaps it is okay to force respondents to choose one side or another (e.g., agree or disagree) without a middle option (e.g., neither agree nor disagree) or to not include an option like "don't know enough to say" or "not applicable." There is no always-correct solution to either problem. But in general, including middle option in a response set provides a more exhaustive set of response options than one that excludes one.
The most important check before your finalize your response options is to align them with your operational definitions. As we've discussed before, your operational definitions include your measures (questions and responses options) as well as how to interpret those measures in terms of the variable being measured. In particular, you should be able to interpret all response options to a question based on your operational definition of the variable it measures. If you wanted to measure the variable "social class," you might ask one question about a participant's annual income and another about family size. Your operational definition would need to provide clear instructions on how to interpret response options. Your operational definition is basically like this social class calculator from Pew Research, though they include a few more questions in their definition.
To drill down a bit more, as Pew specifies in the section titled "how the income calculator works," the interval/ratio data respondents enter is interpreted using a formula combining a participant's four responses to the questions posed by Pew categorizing their household into three categories—upper, middle, or lower class. So, the operational definition includes the four questions comprising the measure and the formula or interpretation which converts responses into the three final categories that we are familiar with: lower, middle, and upper class.
It is interesting to note that even though participants inis an ordinal level of measurement. Whereas, Pew asks four questions that use an interval or ratio level of measurement (depending on the question). This means that respondents provide numerical responses, rather than choosing categories like lower, middle, and upper class. It's perfectly normal for operational definitions to change levels of measurement, and it's also perfectly normal for the level of measurement to stay the same. The important thing is that each response option a participant can provide is accounted for by the operational definition. Throw any combination of family size, location, or income at the Pew calculator, and it will define you into one of those three social class categories.
Unlike Pew's definition, the operational definitions in your study may not need their own webpage to define and describe. For many questions and answers, interpreting response options is easy. If you were measuring "income" instead of "social class," you could simply operationalize the term by asking people to list their total household income before taxes are taken out. Higher values indicate higher income, and lower values indicate lower income. Easy. Regardless of whether your operational definitions are simple or more complex, every response option to every question on your survey (with a few exceptions) should be interpretable using an operational definition of a variable. Just like we want to avoid an everything-but-the-kitchen-sink approach to questions on our questionnaire, you want to make sure your final questionnaire only contains response options that you will use in your study.
One note of caution on interpretation (sorry for repeating this). We want to remind you again that an operational definition should not mention more than one variable. In our example above, your operational definition could not say "a family of three making under $50,000 is lower class; therefore, they are more likely to experience food insecurity." That last clause about food insecurity may well be true, but it's not a part of the operational definition for social class. Each variable (food insecurity and class) should have its own operational definition. If you are talking about how to interpret the relationship between two variables, you are talking about your data analysis plan. We will discuss how to create your data analysis plan beginning in Chapter 14. For now, one consideration is that depending on the statistical test you use to test relationships between variables, you may need nominal, ordinal, or interval/ratio data. Your questions and response options should match the level of measurement you need with the requirements of the specific statistical tests in your data analysis plan. Once you finalize your data analysis plan, return to your questionnaire to match the level of measurement matches with the statistical test you've chosen.
In summary, to write effective response options researchers should do the following:
- Avoid wording that is likely to confuse respondents—including double negatives, use culturally specific terms or jargon, and double-barreled response options.
- Ensure response options are relevant to participants' knowledge and experience so they can make an informed and accurate choice.
- Present mutually exclusive and exhaustive response options.
- Consider fence-sitters and floaters, and the use of neutral or "not applicable" response options.
- Define how response options are interpreted as part of an operational definition of a variable.
- Check level of measurement matches operational definitions and the statistical tests in the data analysis plan (once you develop one in the future)
Exercises
Look back at the response options you drafted in the previous exercise. Make sure you have a first draft of response options for each closed-ended question on your questionnaire.
- Using the criteria above, evaluate the wording of the response options for each question on your questionnaire.
- Revise your questions and response options until you have a complete first draft.
- Do your first read-through and provide a dummy answer to each question. Make sure you can link each response option and each question to an operational definition.
- Look ahead to Chapter 14 and consider how each item on your questionnaire will inform your data analysis plan.
From this discussion, we hope it is clear why researchers using quantitative methods spell out all of their plans ahead of time. Ultimately, there should be a straight line from operational definition through measures on your questionnaire to the data analysis plan. If your questionnaire includes response options that are not aligned with operational definitions or not included in the data analysis plan, the responses you receive back from participants won't fit with your conceptualization of the key variables in your study. If you do not fix these errors and proceed with collecting unstructured data, you will lose out on many of the benefits of survey research and face overwhelming challenges in answering your research question.
Designing questionnaires
Based on your work in the previous section, you should have a first draft of the questions and response options for the key variables in your study. Now, you’ll also need to think about how to present your written questions and response options to survey respondents. It's time to write a final draft of your questionnaire and make it look nice. Designing questionnaires takes some thought. First, consider the route of administration for your survey. What we cover in this section will apply equally to paper and online surveys, but if you are planning to use online survey software, you should watch tutorial videos and explore the features of of the survey software you will use.
Informed consent & instructions
Writing effective items is only one part of constructing a survey. For one thing, every survey should have a written or spoken introduction that serves two basic functions (Peterson, 2000).[46] One is to encourage respondents to participate in the survey. In many types of research, such encouragement is not necessary either because participants do not know they are in a study (as in naturalistic observation) or because they are part of a subject pool and have already shown their willingness to participate by signing up and showing up for the study. Survey research usually catches respondents by surprise when they answer their phone, go to their mailbox, or check their e-mail—and the researcher must make a good case for why they should agree to participate. Thus, the introduction should briefly explain the purpose of the survey and its importance, provide information about the sponsor of the survey (university-based surveys tend to generate higher response rates), acknowledge the importance of the respondent’s participation, and describe any incentives for participating.
The second function of the introduction is to establish informed consent. Remember that this involves describing to respondents everything that might affect their decision to participate. This includes the topics covered by the survey, the amount of time it is likely to take, the respondent’s option to withdraw at any time, confidentiality issues, and other ethical considerations we covered in Chapter 6. Written consent forms are not always used in survey research (when the research is of minimal risk and completion of the survey instrument is often accepted by the IRB as evidence of consent to participate), so it is important that this part of the introduction be well documented and presented clearly and in its entirety to every respondent.
Organizing items to be easy and intuitive to follow
The introduction should be followed by the substantive questionnaire items. But first, it is important to present clear instructions for completing the questionnaire, including examples of how to use any unusual response scales. Remember that the introduction is the point at which respondents are usually most interested and least fatigued, so it is good practice to start with the most important items for purposes of the research and proceed to less important items. Items should also be grouped by topic or by type. For example, items using the same rating scale (e.g., a 5-point agreement scale) should be grouped together if possible to make things faster and easier for respondents. Demographic items are often presented last because they are least interesting to participants but also easy to answer in the event respondents have become tired or bored. Of course, any survey should end with an expression of appreciation to the respondent.
Questions are often organized thematically. If our survey were measuring social class, perhaps we’d have a few questions asking about employment, others focused on education, and still others on housing and community resources. Those may be the themes around which we organize our questions. Or perhaps it would make more sense to present any questions we had about parents' income and then present a series of questions about estimated future income. Grouping by theme is one way to be deliberate about how you present your questions. Keep in mind that you are surveying people, and these people will be trying to follow the logic in your questionnaire. Jumping from topic to topic can give people a bit of whiplash and may make participants less likely to complete it.
Using a matrix is a nice way of streamlining response options for similar questions. A matrix is a question type that that lists a set of questions for which the answer categories are all the same. If you have a set of questions for which the response options are the same, it may make sense to create a matrix rather than posing each question and its response options individually. Not only will this save you some space in your survey but it will also help respondents progress through your survey more easily. A sample matrix can be seen in Figure 12.4.
Once you have grouped similar questions together, you’ll need to think about the order in which to present those question groups. Most survey researchers agree that it is best to begin a survey with questions that will want to make respondents continue (Babbie, 2010; Dillman, 2000; Neuman, 2003).[47] In other words, don’t bore respondents, but don’t scare them away either. There’s some disagreement over where on a survey to place demographic questions, such as those about a person’s age, gender, and race. On the one hand, placing them at the beginning of the questionnaire may lead respondents to think the survey is boring, unimportant, and not something they want to bother completing. On the other hand, if your survey deals with some very sensitive topic, such as child sexual abuse or criminal convictions, you don’t want to scare respondents away or shock them by beginning with your most intrusive questions.
Your participants are human. They will react emotionally to questionnaire items, and they will also try to uncover your research questions and hypotheses. In truth, the order in which you present questions on a survey is best determined by the unique characteristics of your research. When feasible, you should consult with key informants from your target population determine how best to order your questions. If it is not feasible to do so, think about the unique characteristics of your topic, your questions, and most importantly, your sample. Keeping in mind the characteristics and needs of the people you will ask to complete your survey should help guide you as you determine the most appropriate order in which to present your questions. None of your decisions will be perfect, and all studies have limitations.
Questionnaire length
You’ll also need to consider the time it will take respondents to complete your questionnaire. Surveys vary in length, from just a page or two to a dozen or more pages, which means they also vary in the time it takes to complete them. How long to make your survey depends on several factors. First, what is it that you wish to know? Wanting to understand how grades vary by gender and year in school certainly requires fewer questions than wanting to know how people’s experiences in college are shaped by demographic characteristics, college attended, housing situation, family background, college major, friendship networks, and extracurricular activities. Keep in mind that even if your research question requires a sizable number of questions be included in your questionnaire, do your best to keep the questionnaire as brief as possible. Any hint that you’ve thrown in a bunch of useless questions just for the sake of it will turn off respondents and may make them not want to complete your survey.
Second, and perhaps more important, how long are respondents likely to be willing to spend completing your questionnaire? If you are studying college students, asking them to use their very free time to complete your survey may mean they won’t want to spend more than a few minutes on it. But if you find ask them to complete your survey during down-time between classes and there is little work to be done, students may be willing to give you a bit more of their time. Think about places and times that your sampling frame naturally gathers and whether you would be able to either recruit participants or distribute a survey in that context. Estimate how long your participants would reasonably have to complete a survey presented to them during this time. The more you know about your population (such as what weeks have less work and more free time), the better you can target questionnaire length.
The time that survey researchers ask respondents to spend on questionnaires varies greatly. Some researchers advise that surveys should not take longer than about 15 minutes to complete (as cited in Babbie 2010),[48] whereas others suggest that up to 20 minutes is acceptable (Hopper, 2010).[49] As with question order, there is no clear-cut, always-correct answer about questionnaire length. The unique characteristics of your study and your sample should be considered to determine how long to make your questionnaire. For example, if you planned to distribute your questionnaire to students in between classes, you will need to make sure it is short enough to complete before the next class begins.
When designing a questionnaire, a researcher should consider:
- Weighing strengths and limitations of the method of delivery, including the advanced tools in online survey software or the simplicity of paper questionnaires.
- Grouping together items that ask about the same thing.
- Moving any questions about sensitive items to the end of the questionnaire, so as not to scare respondents off.
- Moving any questions that engage the respondent to answer the questionnaire at the beginning, so as not to bore them.
- Timing the length of the questionnaire with a reasonable length of time you can ask of your participants.
- Dedicating time to visual design and ensure the questionnaire looks professional.
Exercises
Type out a final draft of your questionnaire in a word processor or online survey tool.
- Evaluate your questionnaire using the guidelines above, revise it, and get it ready to share with other student researchers.
Pilot testing and revising questionnaires
A good way to estimate the time it will take respondents to complete your questionnaire (and other potential challenges) is through pilot testing. Pilot testing allows you to get feedback on your questionnaire so you can improve it before you actually administer it. It can be quite expensive and time consuming if you wish to pilot test your questionnaire on a large sample of people who very much resemble the sample to whom you will eventually administer the finalized version of your questionnaire. But you can learn a lot and make great improvements to your questionnaire simply by pilot testing with a small number of people to whom you have easy access (perhaps you have a few friends who owe you a favor). By pilot testing your questionnaire, you can find out how understandable your questions are, get feedback on question wording and order, find out whether any of your questions are boring or offensive, and learn whether there are places where you should have included filter questions. You can also time pilot testers as they take your survey. This will give you a good idea about the estimate to provide respondents when you administer your survey and whether you have some wiggle room to add additional items or need to cut a few items.
Perhaps this goes without saying, but your questionnaire should also have an attractive design. A messy presentation style can confuse respondents or, at the very least, annoy them. Be brief, to the point, and as clear as possible. Avoid cramming too much into a single page. Make your font size readable (at least 12 point or larger, depending on the characteristics of your sample), leave a reasonable amount of space between items, and make sure all instructions are exceptionally clear. If you are using an online survey, ensure that participants can complete it via mobile, computer, and tablet devices. Think about books, documents, articles, or web pages that you have read yourself—which were relatively easy to read and easy on the eyes and why? Try to mimic those features in the presentation of your survey questions. While online survey tools automate much of visual design, word processors are designed for writing all kinds of documents and may need more manual adjustment as part of visual design.
Realistically, your questionnaire will continue to evolve as you develop your data analysis plan over the next few chapters. By now, you should have a complete draft of your questionnaire grounded in an underlying logic that ties together each question and response option to a variable in your study. Once your questionnaire is finalized, you will need to submit it for ethical approval from your professor or the IRB. If your study requires IRB approval, it may be worthwhile to submit your proposal before your questionnaire is completely done. Revisions to IRB protocols are common and it takes less time to review a few changes to questions and answers than it does to review the entire study, so give them the whole study as soon as you can. Once the IRB approves your questionnaire, you cannot change it without their okay.
Key Takeaways
- A questionnaire is comprised of self-report measures of variables in a research study.
- Make sure your survey questions will be relevant to all respondents and that you use filter questions when necessary.
- Effective survey questions and responses take careful construction by researchers, as participants may be confused or otherwise influenced by how items are phrased.
- The questionnaire should start with informed consent and instructions, flow logically from one topic to the next, engage but not shock participants, and thank participants at the end.
- Pilot testing can help identify any issues in a questionnaire before distributing it to participants, including language or length issues.
Exercises
It's a myth that researchers work alone! Get together with a few of your fellow students and swap questionnaires for pilot testing.
- Use the criteria in each section above (questions, response options, questionnaires) and provide your peers with the strengths and weaknesses of their questionnaires.
- See if you can guess their research question and hypothesis based on the questionnaire alone.
11.3 Measurement quality
Learning Objectives
Learners will be able to...
- Define and describe the types of validity and reliability
- Assess for systematic error
The previous chapter provided insight into measuring concepts in social work research. We discussed the importance of identifying concepts and their corresponding indicators as a way to help us operationalize them. In essence, we now understand that when we think about our measurement process, we must be intentional and thoughtful in the choices that we make. This section is all about how to judge the quality of the measures you've chosen for the key variables in your research question.
Reliability
First, let’s say we’ve decided to measure alcoholism by asking people to respond to the following question: Have you ever had a problem with alcohol? If we measure alcoholism this way, then it is likely that anyone who identifies as an alcoholic would respond “yes.” This may seem like a good way to identify our group of interest, but think about how you and your peer group may respond to this question. Would participants respond differently after a wild night out, compared to any other night? Could an infrequent drinker’s current headache from last night’s glass of wine influence how they answer the question this morning? How would that same person respond to the question before consuming the wine? In each cases, the same person might respond differently to the same question at different points, so it is possible that our measure of alcoholism has a reliability problem. Reliability in measurement is about consistency.
One common problem of reliability with social scientific measures is memory. If we ask research participants to recall some aspect of their own past behavior, we should try to make the recollection process as simple and straightforward for them as possible. Sticking with the topic of alcohol intake, if we ask respondents how much wine, beer, and liquor they’ve consumed each day over the course of the past 3 months, how likely are we to get accurate responses? Unless a person keeps a journal documenting their intake, there will very likely be some inaccuracies in their responses. On the other hand, we might get more accurate responses if we ask a participant how many drinks of any kind they have consumed in the past week.
Reliability can be an issue even when we’re not reliant on others to accurately report their behaviors. Perhaps a researcher is interested in observing how alcohol intake influences interactions in public locations. They may decide to conduct observations at a local pub by noting how many drinks patrons consume and how their behavior changes as their intake changes. What if the researcher has to use the restroom, and the patron next to them takes three shots of tequila during the brief period the researcher is away from their seat? The reliability of this researcher’s measure of alcohol intake depends on their ability to physically observe every instance of patrons consuming drinks. If they are unlikely to be able to observe every such instance, then perhaps their mechanism for measuring this concept is not reliable.
The following subsections describe the types of reliability that are important for you to know about, but keep in mind that you may see other approaches to judging reliability mentioned in the empirical literature.
Test-retest reliability
When researchers measure a construct that they assume to be consistent across time, then the scores they obtain should also be consistent across time. Test-retest reliability is the extent to which this is actually the case. For example, intelligence is generally thought to be consistent across time. A person who is highly intelligent today will be highly intelligent next week. This means that any good measure of intelligence should produce roughly the same scores for this individual next week as it does today. Clearly, a measure that produces highly inconsistent scores over time cannot be a very good measure of a construct that is supposed to be consistent.
Assessing test-retest reliability requires using the measure on a group of people at one time, using it again on the same group of people at a later time. Unlike an experiment, you aren't giving participants an intervention but trying to establish a reliable baseline of the variable you are measuring. Once you have these two measurements, you then look at the correlation between the two sets of scores. This is typically done by graphing the data in a scatterplot and computing the correlation coefficient. Figure 11.2 shows the correlation between two sets of scores of several university students on the Rosenberg Self-Esteem Scale, administered two times, a week apart. The correlation coefficient for these data is +.95. In general, a test-retest correlation of +.80 or greater is considered to indicate good reliability.
Again, high test-retest correlations make sense when the construct being measured is assumed to be consistent over time, which is the case for intelligence, self-esteem, and the Big Five personality dimensions. But other constructs are not assumed to be stable over time. The very nature of mood, for example, is that it changes. So a measure of mood that produced a low test-retest correlation over a period of a month would not be a cause for concern.
Internal consistency
Another kind of reliability is internal consistency, which is the consistency of people’s responses across the items on a multiple-item measure. In general, all the items on such measures are supposed to reflect the same underlying construct, so people’s scores on those items should be correlated with each other. On the Rosenberg Self-Esteem Scale, people who agree that they are a person of worth should tend to agree that they have a number of good qualities. If people’s responses to the different items are not correlated with each other, then it would no longer make sense to claim that they are all measuring the same underlying construct. This is as true for behavioral and physiological measures as for self-report measures. For example, people might make a series of bets in a simulated game of roulette as a measure of their level of risk seeking. This measure would be internally consistent to the extent that individual participants’ bets were consistently high or low across trials. A specific statistical test known as Cronbach’s Alpha provides a way to measure how well each question of a scale is related to the others.
Interrater reliability
Many behavioral measures involve significant judgment on the part of an observer or a rater. Interrater reliability is the extent to which different observers are consistent in their judgments. For example, if you were interested in measuring university students’ social skills, you could make video recordings of them as they interacted with another student whom they are meeting for the first time. Then you could have two or more observers watch the videos and rate each student’s level of social skills. To the extent that each participant does, in fact, have some level of social skills that can be detected by an attentive observer, different observers’ ratings should be highly correlated with each other.
Validity
Validity, another key element of assessing measurement quality, is the extent to which the scores from a measure represent the variable they are intended to. But how do researchers make this judgment? We have already considered one factor that they take into account—reliability. When a measure has good test-retest reliability and internal consistency, researchers should be more confident that the scores represent what they are supposed to. There has to be more to it, however, because a measure can be extremely reliable but have no validity whatsoever. As an absurd example, imagine someone who believes that people’s index finger length reflects their self-esteem and therefore tries to measure self-esteem by holding a ruler up to people’s index fingers. Although this measure would have extremely good test-retest reliability, it would have absolutely no validity. The fact that one person’s index finger is a centimeter longer than another’s would indicate nothing about which one had higher self-esteem.
Discussions of validity usually divide it into several distinct “types.” But a good way to interpret these types is that they are other kinds of evidence—in addition to reliability—that should be taken into account when judging the validity of a measure.
Face validity
Face validity is the extent to which a measurement method appears “on its face” to measure the construct of interest. Most people would expect a self-esteem questionnaire to include items about whether they see themselves as a person of worth and whether they think they have good qualities. So a questionnaire that included these kinds of items would have good face validity. The finger-length method of measuring self-esteem, on the other hand, seems to have nothing to do with self-esteem and therefore has poor face validity. Although face validity can be assessed quantitatively—for example, by having a large sample of people rate a measure in terms of whether it appears to measure what it is intended to—it is usually assessed informally.
Face validity is at best a very weak kind of evidence that a measurement method is measuring what it is supposed to. One reason is that it is based on people’s intuitions about human behavior, which are frequently wrong. It is also the case that many established measures in psychology work quite well despite lacking face validity. The Minnesota Multiphasic Personality Inventory-2 (MMPI-2) measures many personality characteristics and disorders by having people decide whether each of over 567 different statements applies to them—where many of the statements do not have any obvious relationship to the construct that they measure. For example, the items “I enjoy detective or mystery stories” and “The sight of blood doesn’t frighten me or make me sick” both measure the suppression of aggression. In this case, it is not the participants’ literal answers to these questions that are of interest, but rather whether the pattern of the participants’ responses to a series of questions matches those of individuals who tend to suppress their aggression.
Content validity
Content validity is the extent to which a measure “covers” the construct of interest. For example, if a researcher conceptually defines test anxiety as involving both sympathetic nervous system activation (leading to nervous feelings) and negative thoughts, then his measure of test anxiety should include items about both nervous feelings and negative thoughts. Or consider that attitudes are usually defined as involving thoughts, feelings, and actions toward something. By this conceptual definition, a person has a positive attitude toward exercise to the extent that they think positive thoughts about exercising, feels good about exercising, and actually exercises. So to have good content validity, a measure of people’s attitudes toward exercise would have to reflect all three of these aspects. Like face validity, content validity is not usually assessed quantitatively. Instead, it is assessed by carefully checking the measurement method against the conceptual definition of the construct.
Criterion validity
Criterion validity is the extent to which people’s scores on a measure are correlated with other variables (known as criteria) that one would expect them to be correlated with. For example, people’s scores on a new measure of test anxiety should be negatively correlated with their performance on an important school exam. If it were found that people’s scores were in fact negatively correlated with their exam performance, then this would be a piece of evidence that these scores really represent people’s test anxiety. But if it were found that people scored equally well on the exam regardless of their test anxiety scores, then this would cast doubt on the validity of the measure.
A criterion can be any variable that one has reason to think should be correlated with the construct being measured, and there will usually be many of them. For example, one would expect test anxiety scores to be negatively correlated with exam performance and course grades and positively correlated with general anxiety and with blood pressure during an exam. Or imagine that a researcher develops a new measure of physical risk taking. People’s scores on this measure should be correlated with their participation in “extreme” activities such as snowboarding and rock climbing, the number of speeding tickets they have received, and even the number of broken bones they have had over the years. When the criterion is measured at the same time as the construct, criterion validity is referred to as concurrent validity; however, when the criterion is measured at some point in the future (after the construct has been measured), it is referred to as predictive validity (because scores on the measure have “predicted” a future outcome).
Discriminant validity
Discriminant validity, on the other hand, is the extent to which scores on a measure are not correlated with measures of variables that are conceptually distinct. For example, self-esteem is a general attitude toward the self that is fairly stable over time. It is not the same as mood, which is how good or bad one happens to be feeling right now. So people’s scores on a new measure of self-esteem should not be very highly correlated with their moods. If the new measure of self-esteem were highly correlated with a measure of mood, it could be argued that the new measure is not really measuring self-esteem; it is measuring mood instead.
Increasing the reliability and validity of measures
We have reviewed the types of errors and how to evaluate our measures based on reliability and validity considerations. However, what can we do while selecting or creating our tool so that we minimize the potential of errors? Many of our options were covered in our discussion about reliability and validity. Nevertheless, the following table provides a quick summary of things that you should do when creating or selecting a measurement tool. While not all of these will be feasible in your project, it is important to include easy-to-implement measures in your research context.
Make sure that you engage in a rigorous literature review so that you understand the concept that you are studying. This means understanding the different ways that your concept may manifest itself. This review should include a search for existing instruments.[50]
- Do you understand all the dimensions of your concept? Do you have a good understanding of the content dimensions of your concept(s)?
- What instruments exist? How many items are on the existing instruments? Are these instruments appropriate for your population?
- Are these instruments standardized? Note: If an instrument is standardized, that means it has been rigorously studied and tested.
Consult content experts to review your instrument. This is a good way to check the face validity of your items. Additionally, content experts can also help you understand the content validity.[51]
- Do you have access to a reasonable number of content experts? If not, how can you locate them?
- Did you provide a list of critical questions for your content reviewers to use in the reviewing process?
Pilot test your instrument on a sufficient number of people and get detailed feedback.[52] Ask your group to provide feedback on the wording and clarity of items. Keep detailed notes and make adjustments BEFORE you administer your final tool.
- How many people will you use in your pilot testing?
- How will you set up your pilot testing so that it mimics the actual process of administering your tool?
- How will you receive feedback from your pilot testing group? Have you provided a list of questions for your group to think about?
Provide training for anyone collecting data for your project.[53] You should provide those helping you with a written research protocol that explains all of the steps of the project. You should also problem solve and answer any questions that those helping you may have. This will increase the chances that your tool will be administered in a consistent manner.
- How will you conduct your orientation/training? How long will it be? What modality?
- How will you select those who will administer your tool? What qualifications do they need?
When thinking of items, use a higher level of measurement, if possible.[54] This will provide more information and you can always downgrade to a lower level of measurement later.
- Have you examined your items and the levels of measurement?
- Have you thought about whether you need to modify the type of data you are collecting? Specifically, are you asking for information that is too specific (at a higher level of measurement) which may reduce participants' willingness to participate?
Use multiple indicators for a variable.[55] Think about the number of items that you will include in your tool.
- Do you have enough items? Enough indicators? The correct indicators?
Conduct an item-by-item assessment of multiple-item measures.[56] When you do this assessment, think about each word and how it changes the meaning of your item.
- Are there items that are redundant? Do you need to modify, delete, or add items?
Types of error
As you can see, measures never perfectly describe what exists in the real world. Good measures demonstrate validity and reliability but will always have some degree of error. Systematic error (also called bias) causes our measures to consistently output incorrect data in one direction or another on a measure, usually due to an identifiable process. Imagine you created a measure of height, but you didn’t put an option for anyone over six feet tall. If you gave that measure to your local college or university, some of the taller students might not be measured accurately. In fact, you would be under the mistaken impression that the tallest person at your school was six feet tall, when in actuality there are likely people taller than six feet at your school. This error seems innocent, but if you were using that measure to help you build a new building, those people might hit their heads!
A less innocent form of error arises when researchers word questions in a way that might cause participants to think one answer choice is preferable to another. For example, if I were to ask you “Do you think global warming is caused by human activity?” you would probably feel comfortable answering honestly. But what if I asked you “Do you agree with 99% of scientists that global warming is caused by human activity?” Would you feel comfortable saying no, if that’s what you honestly felt? I doubt it. That is an example of a leading question, a question with wording that influences how a participant responds. We’ll discuss leading questions and other problems in question wording in greater detail in Chapter 12.
In addition to error created by the researcher, your participants can cause error in measurement. Some people will respond without fully understanding a question, particularly if the question is worded in a confusing way. Let’s consider another potential source or error. If we asked people if they always washed their hands after using the bathroom, would we expect people to be perfectly honest? Polling people about whether they wash their hands after using the bathroom might only elicit what people would like others to think they do, rather than what they actually do. This is an example of social desirability bias, in which participants in a research study want to present themselves in a positive, socially desirable way to the researcher. People in your study will want to seem tolerant, open-minded, and intelligent, but their true feelings may be closed-minded, simple, and biased. Participants may lie in this situation. This occurs often in political polling, which may show greater support for a candidate from a minority race, gender, or political party than actually exists in the electorate.
A related form of bias is called acquiescence bias, also known as “yea-saying.” It occurs when people say yes to whatever the researcher asks, even when doing so contradicts previous answers. For example, a person might say yes to both “I am a confident leader in group discussions” and “I feel anxious interacting in group discussions.” Those two responses are unlikely to both be true for the same person. Why would someone do this? Similar to social desirability, people want to be agreeable and nice to the researcher asking them questions or they might ignore contradictory feelings when responding to each question. You could interpret this as someone saying "yeah, I guess." Respondents may also act on cultural reasons, trying to “save face” for themselves or the person asking the questions. Regardless of the reason, the results of your measure don’t match what the person truly feels.
So far, we have discussed sources of error that come from choices made by respondents or researchers. Systematic errors will result in responses that are incorrect in one direction or another. For example, social desirability bias usually means that the number of people who say they will vote for a third party in an election is greater than the number of people who actually vote for that candidate. Systematic errors such as these can be reduced, but random error can never be eliminated. Unlike systematic error, which biases responses consistently in one direction or another, random error is unpredictable and does not consistently result in scores that are consistently higher or lower on a given measure. Instead, random error is more like statistical noise, which will likely average out across participants.
Random error is present in any measurement. If you’ve ever stepped on a bathroom scale twice and gotten two slightly different results, maybe a difference of a tenth of a pound, then you’ve experienced random error. Maybe you were standing slightly differently or had a fraction of your foot off of the scale the first time. If you were to take enough measures of your weight on the same scale, you’d be able to figure out your true weight. In social science, if you gave someone a scale measuring depression on a day after they lost their job, they would likely score differently than if they had just gotten a promotion and a raise. Even if the person were clinically depressed, our measure is subject to influence by the random occurrences of life. Thus, social scientists speak with humility about our measures. We are reasonably confident that what we found is true, but we must always acknowledge that our measures are only an approximation of reality.
Humility is important in scientific measurement, as errors can have real consequences. At the time I'm writing this, my wife and I are expecting our first child. Like most people, we used a pregnancy test from the pharmacy. If the test said my wife was pregnant when she was not pregnant, that would be a false positive. On the other hand, if the test indicated that she was not pregnant when she was in fact pregnant, that would be a false negative. Even if the test is 99% accurate, that means that one in a hundred women will get an erroneous result when they use a home pregnancy test. For us, a false positive would have been initially exciting, then devastating when we found out we were not having a child. A false negative would have been disappointing at first and then quite shocking when we found out we were indeed having a child. While both false positives and false negatives are not very likely for home pregnancy tests (when taken correctly), measurement error can have consequences for the people being measured.
Key Takeaways
- Reliability is a matter of consistency.
- Validity is a matter of accuracy.
- There are many types of validity and reliability.
- Systematic error may arise from the researcher, participant, or measurement instrument.
- Systematic error biases results in a particular direction, whereas random error can be in any direction.
- All measures are prone to error and should interpreted with humility.
Exercises
Use the measurement tools you located in the previous exercise. Evaluate the reliability and validity of these tools. Hint: You will need to go into the literature to "research" these tools.
- Provide a clear statement regarding the reliability and validity of these tools. What strengths did you notice? What were the limitations?
- Think about your target population. Are there changes that need to be made in order for one of these tools to be appropriate for your population?
- If you decide to create your own tool, how will you assess its validity and reliability?
approach to recruitment where participants are members of an organization or social group with identified membership
Chapter Outline
- Choosing a research topic (10 minute read)
- Your research proposal (14 minute read)
- Critical considerations (5 minute read)
- Evaluating online resources (11 minute read)
Content warning: Examples in this chapter discuss substance use disorders, mental health disorders and therapies, obesity, poverty, gun violence, gang violence, school discipline, racism and hate groups, domestic violence, trauma and triggers, incarceration, child neglect and abuse, bullying, self-harm and suicide, racial discrimination in housing, burnout in helping professions, and sex trafficking of indigenous women.
2.1 Choosing a research topic
Learning Objectives
Learners will be able to...
- Brainstorm topics you may want to investigate as part of a research project
- Explore your feelings and existing knowledge about the topic
- Develop a working question
Research methods is a unique class in that you get to decide what you want to learn about. Perhaps you came to your MSW program with a specific issue you were passionate about. In my MSW program, I wanted to learn about the best interventions to use with people who have substance use disorders. This was in line with my future career plans, which included working in a clinical setting with clients with co-occurring mental health and substance use issues. I suggest you start by thinking about your future practice goals and create a research project that addresses a topic that represents an area of social work you are passionate about.
For those of you without a specific direction, don't worry. Many people enter their MSW program without an exact topic in mind they want to study. Throughout the program, you will be exposed to different populations, theories, practice interventions, and policies that will spark your interest. Think back to papers you enjoyed researching and writing in other classes. You may want to continue studying the same topic. Research methods will enable you to gain a deeper, more nuanced understanding of a topic or issue. If you haven't found an interesting topic yet, here are some other suggestions for seeking inspiration for a research project:
- If you already have practice experience in social work through employment, an internship, or volunteer work, think about practice issues you noticed in the placement. Do you have any idea of how to better address client needs? Do you need to learn more about existing interventions or the programs that fund your agency? Use this class as an opportunity to engage with your previous field experience in greater detail. Begin with “what” and “why” questions and then expand on those. For example, what are the most effective methods of treating severe depression among a specific population? Or why are people receiving food assistance more likely to be obese?
- You could also ask a professor at your school about possible topics. Read departmental information on faculty research interests, which may surprise you. Most departmental websites post the curriculum vitae (CV) of faculty, which lists their publications, credentials, and interests. For those of you interested in doctoral study, this process is particularly important. Students often pick schools based on professors they want to learn from or research initiatives they want to join.
Once you have a potential idea, start reading! A simple web search should bring you some basic information about your topic. News articles can reveal new or controversial information. You may also want to identify and browse academic journals related to your research interests. Faculty and librarians can help you identify relevant journals in your field and specific areas of interest. We'll also review more detailed strategies for searching the literature in Chapter 3. As you read, look for what’s missing. These may be “gaps in the literature” that you might explore in your own study.
It's a good idea to keep it simple when you're starting your project. Choose a topic that can be easily defined and explored. Your study cannot focus on everything that is important about your topic. A study on gun violence might address only one system, for example schools, while only briefly mentioning other systems that impact gun violence. That doesn't mean it's a bad study! Every study presents only a small picture of a larger, more complex and multifaceted issue. The sooner you can arrive at something specific and clear that you want to study, the better off your project will be.
Writing a working question
There are lots of great research topics. Perhaps your topic is a client population—for example, youth who identify as LGBTQ+ or visitors to a local health clinic. In other cases, your topic may be a social problem, such as gang violence, or a social policy or program, such as zero-tolerance policies in schools. Alternately, maybe there are interventions such as dialectical behavioral therapy or applied behavior analysis that interest you.
Whatever your topic idea, begin to think about it in terms of a question. What do you really want to know about the topic? As a warm-up exercise, try dropping a possible topic idea into one of the blank spaces below. The questions may help bring your subject into sharper focus and bring you closer towards developing your topic.
- What does ___ mean?
- What are the causes of ___?
- What are the consequences of ___?
- What are the component parts of ___?
- How does ___ impact ___?
- What is it like to experience ___?
- What is the relationship between _____ and the outcome of ____?
- What case can be made for or against ___?
- What are the risk/protective factors for ___?
- How do people think about ___?
Take a minute right now and write down a question you want to answer. Even if it doesn’t seem perfect, it is important to start somewhere. Make sure your research topic is relevant to social work. You’d be surprised how much of the world that encompasses. It’s not just research on mental health treatment or child welfare services. Social workers can study things like the pollution of irrigation systems and entrepreneurship in women, among other topics. The only requirement is your research must inform action to fight social problems faced by target populations.
Because research is an iterative process, one that you will revise over and over, your question will continue to evolve. As you progress through this textbook, you’ll learn how to refine your question and include the necessary components for proper qualitative and quantitative research questions. Your question will also likely change as you engage with the literature on your topic. You will learn new and important concepts that may shift your focus or clarify your original ideas. Trust that a strong question will emerge from this process. A good researcher must be comfortable with altering their question as a result of scientific inquiry.
Very often, our students will email us in the first few weeks of class and ask if they have a good research topic. We love student emails! But just to reassure you if you're about to send a panicked email to your professor, as long as you are interested in dedicating a semester or two learning about your topic, it will make a good research topic. That's why we would advise you to focus on how much you like this topic, so that three months from now you are still motivated to complete your project. Your project should have meaning to you.
How do you feel about your topic?
Now that you have an idea of what you might want to study, it's time to consider what you think and feel about that topic. Your motivation for choosing a topic does not have to be objective. Because social work is a value-based profession, scholars often find themselves motivated to conduct research that furthers social justice or fights oppression. Just because you think a policy is wrong or a group is being marginalized, for example, does not mean that your research will be biased. It means you must understand what you feel, why you feel that way, and what would cause you to feel differently about your topic.
Start by asking yourself how you feel about your topic. Sometimes the best topics to research are those about which you feel strongly. What better way to stay engaged with your research project than to study something you are passionate about? However, you must be able to accept that people may have a different perspective, and you must represent their viewpoints fairly in the research report you produce. If you feel prepared to accept all findings, even those that may be unflattering or distinct from your personal perspective, then perhaps you should begin your research project by intentionally studying a topic about which you have strong feelings.
Kathleen Blee (2002)[57] has taken this route in her research. Blee studies groups whose racist ideologies may be different than her own. You can listen to her lecture Women in Organized Racism that details some of her findings. Her scientific research is so impactful because she was willing to report her findings and observations honestly, even those contrary to her beliefs and feelings. If you believe that you may have personal difficulty sharing findings with which you disagree, then you may want to study a different topic. Knowing your own hot-button issues is an important part of self-knowledge and reflection in social work, and there is nothing wrong with avoiding topics that are likely to cause you unnecessary stress.
Social workers often use personal experience as a starting point to identify topics of interest. As we’ve discussed here, personal experience can be a powerful motivator to learn more about a topic. However, social work researchers should be mindful of their own mental health during the research process. A social worker who has experienced a mental health crisis or traumatic event should approach researching related topics cautiously. There is no need to trigger yourself or jeopardize your mental health for a research project. For example, a student who has just experienced domestic violence may want to know about Eye Movement Desensitization and Reprocessing (EMDR) therapy. While the student might gain some knowledge about potential treatments for domestic violence, they will likely have to read through many stories and reports about domestic violence as part of the research process. Unless the student’s trauma has been processed in therapy, conducting a research project on this topic may negatively impact the student’s mental health.
What do you think about your topic?
Once you figure out what you feel about your topic, consider what you think about it. There are many ways we know what we know. Perhaps your mother told you something is so. Perhaps it came to you in a dream. Perhaps you took a class last semester and learned something about your topic there. Or you may have read something about your topic in your local newspaper. We discussed the strengths and weaknesses associated with some of these different sources of knowledge in Chapter 1, and we’ll talk about other scientific sources of knowledge in Chapter 3 and Chapter 4. For now, take some time to think of everything you know about your topic. Thinking about what you already know will help you identify any biases you may have, and it will help as you begin to frame a question about your topic.
You might consider creating a concept map, just to get your thoughts and ideas on paper and beginning to organize them. Consider this video from the University of Guelph Library (CC-BY-NC-SA 4.0).
Key Takeaways
- You should pick a topic for your research proposal that you are interested in, since you will be working with it for several months.
- Investigate your own feelings and thoughts about a topic, and make sure you can be objective and fair in your investigation.
- Research projects are guided by a working question that develops and changes as you learn more about your topic.
Exercises
Just as a reminder, exercises are designed to help you create your individual research proposal. We designed these activities to break down your proposal into small but manageable chunks. We suggest completing each exercise so you can apply what you are learning to your individual research project, as the exercises in each section and each chapter build on one another.
If you haven't done so already, you can create a document in a word processor on your computer or in a written notebook with your answers to each exercise.
-
Brainstorm at least 4-5 topics of interest to you and pick the one you think is the most promising for a research project.
- For your chosen topic, outline what you currently know about the topic and your feelings towards the topic. Make sure you are able to be objective and fair in your research.
- Formulate at least one working question to guide your inquiry. It is common for topics to change and develop over the first few weeks of a project, but think of your working question as a place to start. Use the 10 examples we provided in this chapter if you need some help getting started.
2.2 Your research proposal
Learning Objectives
Learners will be able to...
- Describe the stages of a research project
- Define your target population and describe how your study will impact that population
- Identify the aim of your study
- Classify your project as descriptive, exploratory, explanatory, or evaluative
Most research methods courses are designed to help students propose a research project. But what is a research project? Figure 2.1 indicates the steps of the research project. Right now, we are in the top right corner, using your informal observations from your practice experience and lived experience to form a working draft of your research question. In the next three chapters, you'll learn how to find and evaluate scholarly literature on your topic. After thoroughly evaluating the literature, you'll conceptualize an empirical study based on a research question you create. In many courses, students will have to carry out these designs and make a contribution to the research literature in their topic area.
This book uses a project-based approach because it mimics the real-world process of inquiry on a social work topic. In an introductory research methods course, students often have to create a research proposal followed by a more advanced research class in which they conduct quantitative and qualitative data analysis. The research proposal, is a document produced by researchers that reviews the literature relevant to their topic and describes the methods they will use to conduct their study. Part 1 of this textbook is designed to help you with your literature review. Part 2 is designed to help you figure out which methods you will use in your study.
Check with your professor on whether you are required to carry out the project you propose to do in your research proposal. Some of you may only need to propose a hypothetical project. If you are planning to complete your project, you will have to pay more attention to the practical and ethical considerations in this chapter.
A research proposal is focused on a question. Right now, this is your working question from Section 2.1. If you haven't created one yet, this is a good time to pause and complete the exercises from section 2.1.[58] It is likely you will revise your working question many times as you read more literature about your topic. Consider yourself in the cycle between (re)creating your research question and reviewing the research literature for Part 1 of the textbook.
Student research proposals
Student research projects are a big undertaking, but they are well within your capability as a graduate student. Let's start with the research proposal. Think about the research proposal as a communication device. You are telling the reader (your professor, usually) everything they need to know in order to understand your topic and the research study you plan to do. You are also demonstrating to the reader that you are competent and informed enough to conduct the study.
You can think of a research proposal like creating a recipe. If you are a chef trying to cook a new dish from scratch, you would probably start by looking at other recipes. You might cook a few of them and come up with ideas about how to create your own version of the dish. Writing your recipe is a process of trial and error, and you will likely revise your proposal many times over the course of the semester. This textbook and its exercises designed to get you working on your project little by little, so that by the time you turn in your final research proposal, you'll be confident it represents the best way to answer your question. Of course, like with any time I cook, you never quite know how it will turn out. What matters for scientists in the end isn't whether your data proves your ideas right or wrong or whether your data collection doesn't work as planned or goes off perfectly. Instead, what matters is that you report your results (warts and all) as honestly and openly as possible to inform others engaged in scholarly inquiry.
Is writing a research proposal a useful skill for a social worker? On one hand, you probably won't be writing research proposals for a living. But the same structure of a research proposal (literature review + methods) is used in grant applications. Writing grant proposals is often a part of practice, particularly in agency-based and policy practice. Instead of finding a gap in the literature to study, practitioners write grant proposals describing a program they will use to address an issue in their community, as well as the research methods they will use to evaluate whether it worked. Similarly, a policy advocate or public administrator might sketch out a proposed program and its evaluation as part of a proposal. Proposal writing may differ somewhat in practice, but the general idea is the same.
Focusing your project
Based on your work in Section 2.1, you should have a working question—a place to start. Think about what you hope to accomplish with your study. This is the aim of your research project. Often, social work researchers begin with a target population in mind. As you will recall from section 1.4, social work research is research for action. Social workers engage in research to help people. Think about your working question. Why do you want to answer it? What impact would answering your question have?
In my MSW program, I began my research by looking at ways to intervene with people who have substance use disorders. My foundation year placement was in an inpatient drug treatment facility that used 12-step facilitation as its primary treatment modality. I observed that this approach differed significantly from others I had been exposed to, especially the idea of powerlessness over drugs and drug use. My working question started as "what are the alternatives to 12-step treatment for people with substance use issues and are they more effective?" The aim of my project was to determine whether different treatment approaches might be more effective, and I suspected that self-determination and powerlessness were important.
It's important to note that my working question contained a target population—people with substance use disorders. A target population is the group of people that will benefit the most. I envisioned I would help the field of social work to think through how to better meet clients where they were at, specific to the problem of substance use. I was studying to be a clinical social worker, so naturally, I formulated a micro-level question. Yet, the question also has implications for meso- and macro-level practice. If other treatment methods are more effective than 12-step facilitation, then we should direct more public money towards providing more effective therapies for people who use substances. We may also need to train the substance use professionals to use new treatment methodologies.
Exercises
Think about your working question.
- Is it more oriented towards micro-, meso-, or macro-level practice?
- What implications would answering your question have at each level of the ecosystem?
Asking yourself whether your project is more micro, meso, or macro is a good check to see if your project is well-focused. A project that seems like it could be all of those might have too many components or try to study too much. Consider identifying one ecosystemic level your project will focus on, and you can interpret and contextualize your findings at the other levels of analysis.
Exploration, description, and explanation
Social science is a big place. Looking at the various empirical studies in the literature, there is a lot of diversity—from focus groups with clients and families to multivariate statistical analysis of large population surveys conducted online. Ultimately, all of social science can be described as one of three basic types of research studies. As you develop your research question, consider which of the following types of research studies fits best with what you want to learn about your topic. In subsequent chapters, we will use these broad frameworks to help craft your study's final research question and choose quantitative and qualitative research methods to answer it.
Exploratory research
Researchers conducting exploratory research are typically at the early stages of examining their topics. Exploratory research projects are carried out to test the feasibility of conducting a more extensive study and to figure out the “lay of the land” with respect to the particular topic. Usually, very little prior research has been conducted on this topic. For this reason, a researcher may wish to do some exploratory work to learn what method to use in collecting data, how best to approach research subjects, or even what sorts of questions are reasonable to ask.
Often, student projects begin as exploratory research. Because students don't know as much about the topic area yet, their working questions can be general and vague. That's a great place to start! An exploratory question is great for delving into the literature and learning more about your topic. For example, the question “what are common social work interventions for parents who neglect their children?” is a good place to start when looking at articles and textbooks to understand what interventions are commonly used with this population. However, it is important for a student research project to progress beyond exploration unless the topic truly has very little existing research.
In my classes, I often read papers where students say there is not a lot of literature on a topic, but a quick search of library databases shows a deep body of literature on the topic. The skills you develop in Chapter 3 and Chapter 4 should assist you with finding relevant research, and working with a librarian can definitely help with finding information for your research project. That said, there are a few students each year who pick a topic for which there is in fact little existing research. Perhaps, if you were looking at child neglect interventions for parents who identify as transgender or parents who are refugees from the Syrian civil war, less would be known about child neglect for those specific populations. In that case, an exploratory design would make sense as there is little, if any, literature about your specific topic.
Descriptive research
Another purpose of a research project is to describe or define a particular phenomenon. This is called descriptive research. For example, researchers at the Princeton Review conduct descriptive research each year when they set out to provide students and their parents with information about colleges and universities around the United States. They describe the social life at a school, the cost of admission, and student-to-faculty ratios (to name just a few of the categories reported). If our topic were child neglect, we might seek to know the number of people arrested for child neglect in our community and whether they are more likely to have other problems, such as poverty, mental health issues, or substance use.
Social workers often rely on descriptive research to tell them about their service area. Keeping track of the number of parents receiving child neglect interventions, their demographic makeup (e.g., race, sex, age), and length of time in care are excellent examples of descriptive research. On a more macro-level, the Centers for Disease Control provides a remarkable amount of descriptive research on mental and physical health conditions. In fact, descriptive research has many useful applications, and you probably rely on such findings without realizing you are reading descriptive research.
Explanatory research
Lastly, social work researchers often aim to explain why particular phenomena operate in the way that they do. Research that answers “why” questions is referred to as explanatory research. Asking "why" means the researcher is trying to identify cause-and-effect relationships in their topic. For example, explanatory research may try to identify risk and protective factors for parents who neglect their children. Explanatory research may attempt to understand how religious affiliation impacts views on immigration. All explanatory research tries to study cause-and-effect relationships between two or more variables. A specific offshoot of explanatory research that comes up often is evaluation research, which investigates the impact of an intervention, program, or policy on a group of people. Evaluation research is commonly practiced in agency-based social work settings, and later chapters will discusses some of the basics for conducting a program evaluation.
There are numerous examples of explanatory social scientific investigations. For example, Dominique Simons and Sandy Wurtele (2010)[59] sought to understand whether receiving corporal punishment from parents led children to turn to violence in solving their interpersonal conflicts with other children. In their study of 102 families with children between the ages of 3 and 7, the authors found that experiencing frequent spanking did in fact result in children being more likely to accept aggressive problem-solving techniques. Another example of explanatory research can be seen in Robert Faris and Diane Felmlee’s (2011)[60] research study on the connections between popularity and bullying. From their study of 8th, 9th, and 10th graders in nineteen North Carolina schools, they found that aggression increased as adolescents’ popularity increased.[61]
Exercises
- Think back to your working question from section 2.1. Which type of research—exploratory, descriptive, or explanatory—best describes your working question?
- Try writing a question about your topic that fits with each type of research.
Important things are more rewarding to do
Another consideration in starting a research project is whether the question is important enough to answer. For the researcher, answering the question should be important enough to put in the effort and time required to complete a research project. As we discussed in section 2.1, you should choose a topic that is important to you—one you wouldn’t mind learning about for at least a few months, if not a few years. Time is your most precious resource as a student. Make sure you dedicate it to topics and projects you consider genuinely important.
Your research question should also be contribute to the larger expanse of research in that area. For example, if your research question is "does cognitive behavioral therapy (CBT) effectively treat depression?" you are a few decades late to be asking that question. Hundreds of scientists have published articles demonstrating its effectiveness in treating depression. However, a student interested in learning more about CBT can still find new areas to research. Perhaps there is a new population—for example, older adults in a nursing home—or a new problem—like mobile phone addiction—for which there is little research on the impact of CBT.
Your research project should contribute something new to social science. It should address a gap in what we know and what is written in the literature. This can seem intimidating for students whose projects involve learning a totally new topic. How could I add something new when other researchers have studied this for decades? Trust us, by thoroughly reviewing the existing literature, you can find new and unresolved research questions to answer. Google Scholar’s motto at the bottom of their search page is “stand on the shoulders of giants.” Social science research rests on the work of previous scholars, and builds off of what they discovered to learn more about the social world. Ensure that your question will bring our scientific understanding of your topic to new heights.
Finally, your research question should be of import to the social world. Social workers conduct research on behalf of individuals, groups, and communities to promote change as part of their mission to advance human rights and further social and economic justice. Your research should matter to the people you are trying to help. Your research project should aim to improve the lives of people in your target population by helping the world understand their needs more holistically.
Research projects, obviously, do not need to address all aspects of a problem. As social workers, our goal in enacting social justice isn't to accomplish it all in one semester (or even one lifetime). Our goal is to move the world in the right direction and make small, incremental progress. We encourage all students to think about how they will make their work accessible and relevant to the broader public and use their results to promote change.
Key Takeaways
- Research exists in a cycle. Your research project will follow this cycle, beginning from reading literature (where you are now), to proposing a study, to completing a research project, and finally, to publishing the results.
- Social work researchers should identify a target population and understand how their project will impact them.
- Research projects can be exploratory, descriptive, evaluative, or a combination therein. While you are likely still exploring your topic, you may settle on another type of research, particularly if your topic has been previously addressed extensively in the literature.
- Your research project should be important to you, fill a gap or address a controversy in the scientific literature, and make a difference for your target population and broader society.
Exercises
- State why your working question is an important one to answer, keeping in mind that your statement should address the scientific literature, target population, and the social world.
2.3 Critical considerations
Learning Objectives
Learners will be able to...
- Critique the traditional role of researchers and identify how action research addresses these issues
So far in this chapter, we have presented the steps of student research projects as follows:
- Find a topic that is important to you and read about it.
- Pose a question that is important to the literature and to your community.
- Propose to use specific research methods to answer your question.
- Carry out your project and report the results.
These were depicted in Figure 2.1 earlier in this chapter. There are important limitations to this approach. This section examines those problems and how to address them.
Whose knowledge is privileged?
First, let's critically examine your role as the researcher. Following along with the steps in a research project, you start studying the literature your topic, find a place where you can add to scientific knowledge, and conduct your study. But why are you the person who gets to decide what is important? Just as clients are the experts on their lives, members of your target population are the experts on their lives. What does it mean for a group of people to be researched on, rather than researched with? How can we better respect the knowledge and self-determination of community members?
A different way of approaching your research project is to start by talking with members of the target population and those who are knowledgeable about that community. Perhaps there is a community-led organization you can partner with on a research project. The researcher's role in this case would be more similar to a consultant, someone with specialized knowledge about research who can help communities study problems they consider to be important. The social worker is a co-investigator, and community members are equal partners in the research project. Each has a type of knowledge—scientific expertise vs. lived experience—that should inform the research process.
The community focus highlights something important about student projects: they are localized. Student projects can dedicate themselves to issues at a single agency or within a service area. With a local scope, student researchers can bring about change in their community. This is the purpose behind action research.
Action research
Action research is research that is conducted for the purpose of creating social change. When engaging in action research, scholars collaborate with community stakeholders to conduct research that will be relevant to the community. Social workers who engage in action research don't just go it alone; instead, they collaborate with the people who are affected by the research at each stage in the process. Stakeholders, particularly those with the least power, should be consulted on the purpose of the research project, research questions, design, and reporting of results.
Action research also distinguishes itself from other research in that its purpose is to create change on an individual and community level. Kristin Esterberg puts it quite eloquently when she says, “At heart, all action researchers are concerned that research not simply contribute to knowledge but also lead to positive changes in people’s lives” (2002, p. 137).[62] Action research has multiple origins across the globe, including Kurt Lewin’s psychological experiments in the US and Paulo Friere’s literacy and education programs (Adelman, 1993; Reason, 1994).[63] Over the years, action research has become increasingly popular among scholars who wish for their work to have tangible outcomes that benefit the groups they study.
A traditional scientist might look at the literature or use their practice wisdom to formulate a question for quantitative or qualitative research, as we suggested earlier in this chapter. An action researcher, on the other hand, would consult with people in target population and community to see what they believe the most pressing issues are and what their proposed solutions may be. In this way, action research flips traditional research on its head. Scientists are not the experts on the research topic. Instead, they are more like consultants who provide the tools and resources necessary for a target population to achieve their goals and to address social problems using social science research.
According to Healy (2001),[64] the assumptions of participatory-action research are that (a) oppression is caused by macro-level structures such as patriarchy and capitalism; (b) research should expose and confront the powerful; (c) researcher and participant relationships should be equal, with equitable distribution of research tasks and roles; and (d) research should result in consciousness-raising and collective action. Consistent with social work values, action research supports the self-determination of oppressed groups and privileges their voice and understanding through the conceptualization, design, data collection, data analysis, and dissemination processes of research. We will return to similar ideas in Part 4 of the textbook when we discuss qualitative research methods, though action research can certainly be used with quantitative research methods, as well.
Student projects can make a difference!
One last thing. We've told you all to think small and simple with your projects. The adage that "a good project is a done project" is true. At the same time, this advice might unnecessarily limit an ambitious and diligent student who wanted to investigate something more complex. For example, here is a Vice News article about MSW student Christine Stark's work on sex trafficking of indigenous women. Student projects have the potential to address sensitive and politically charged topics. With support from faculty and community partners, student projects can become more comprehensive. The results of your project should accomplish something. Social work research is about creating change, and you will find the work of completing a research project more rewarding and engaging if you can envision the change your project will create.
In addition to broader community and agency impacts, student research projects can have an impact on a university or academic program. Consider this resource on how to research your institution by Rine Vieth. As a student, you are one of the groups on campus with the least power (others include custodial staff, administrative staff, contingent and adjunct faculty). It is often necessary that you organize within your cohort of MSW students for change within the program. Not only is it an excellent learning opportunity to practice your advocacy skills, you can use raw data that is publicly available (such as those linked in the guide) or create your own raw data to inform change. The collaborative and transformative focus of student research projects like these can be impactful learning experiences, and students should consider projects that will lead to some small change in both themselves and their communities.
Key Takeaways
- Traditionally, researchers did not consult target populations and communities prior to formulating a research question. Action research proposes a more community-engaged model in which researchers are consultants that help communities research topics of import to them.
- Just because we’ve advised you to keep your project simple and small doesn’t mean you must do so! There are excellent examples of student research projects that have created real change in the world.
Exercises
- Apply the key concepts of action research to your project. How might you incorporate the perspectives and expertise of community members in your project? How can your project create real change?
2.4 Evaluating internet resources
Learning Objectives
Learners will be able to...
- Apply the SIFT technique to find better coverage of scientific information and current events.
When first learning about a new topic, a natural first place to look is an internet search engine (e.g., Google, Bing, or DuckDuckGo). Before diving into the academic literature (which we will do in the next chapter), let's explore how to use research methods to find scientific information that is intended for non-expert audiences. Take some time to learn the basics of your topic before you dive into more advanced literature, which may present a more nuanced (or jargon-filled and confusing) study of the topic. Generally, scholarly literature is specialized, in that it does not try to provide a broad overview of a topic for a non-expert audience. When you are looking at journal articles, you are looking at literature intended for other scientists and researchers.
While you can be assured that articles in reputable journals have passed peer review, that does not always mean they contain accurate information. Articles are often debated on social media or in journalistic outlets. For example, here is a news story debunking a journal article which erroneously found Safe Consumption Sites for people who use drugs were moderately associated with crime increases. After multiple scholars evaluated the article's data, they realized there were flaws in the design and the conclusions were not supported, which led the journal to retract the article. You can find these controversies in the literature by using a Google Scholar feature we've talked about before—'Cited By'. Click the 'Cited By' link to see which articles cited the article you are evaluating. If you see critical commentary on the article by other scholars, it is likely an area of active scientific debate. You should investigate the controversy further prior to using the source in your literature review.
If your literature search contains sources other than academic journal articles (and almost all of them do), you'll need to do a bit more work to assess whether the source is reputable enough to include in your review. Let's say you find a report from a Google Scholar search or a Bing search. Without peer review or a journal's approval, how do you know the information you are reading is any good?
The SIFT method
Mike Caulfield, Washington State University digital literacy expert, has helpfully condensed key fact-checking strategies into a short list of four moves, or things to do to quickly make a decision about whether or not a source is worthy of your attention. It is referred to as the “SIFT” method, and it stands for Stop, Investigate the source, Find better coverage, and Trace claims claims, quotes, and media to the original context.
Stop
When you initially encounter a source of information and start to read it—stop. Ask yourself whether you know and trust the author, publisher, publication, or website. If you don’t, use the other fact-checking moves that follow, to get a better sense of what you’re looking at. In other words, don’t read, share, or use the source in your research until you know what it is, and you can verify it is reliable.
This is a particularly important step, considering what we know about the attention economy—social media, news organizations, and other digital platforms purposely promote sensational, divisive, and outrage-inducing content that emotionally hijacks our attention in order to keep us “engaged” with their sites (clicking, liking, commenting, sharing). Stop and check your emotions before engaging! What about this website is driving your engagement?
Investigate the sources
You don’t have to do a three-hour investigation into a source to determine its truth. But if you’re reading a piece on economics, and the author is a Nobel prize-winning economist, that would be useful information. Likewise, if you’re watching a video on the many benefits of milk consumption, you would want to be aware if the video was produced by the dairy industry. This doesn’t mean the Nobel economist will always be right and that the dairy industry can’t ever be trusted. But knowing the expertise and agenda of the person who created the source is crucial to your interpretation of the information provided.
When investigating a source, fact-checkers read “laterally” across many websites, rather than digging deep (reading “vertically”) into the one source they are evaluating. That is, they don’t spend much time on the source itself, but instead they quickly get off the page and see what others have said about the source. Indeed, one study cited in the video below found that academic historians are actually less able to tell the difference between reputable and bogus internet sources because they do not read laterally but instead check references and credentials. Those are certainly a good idea to check when reading a source in detail, but fact checkers instead ask what other sources on the web say about it rather than what the source says about itself. They open up many tabs in their browser, piecing together different bits of information from across the web to get a better picture of the source they’re investigating. Not only is this faster, but it harnesses the collected knowledge of the web to more accurately determine whether a source is reputable or not.
We recommend watching this short video [2:44] for a demonstration of how to investigate online sources. Pay particular attention to how Wikipedia can be used to quickly get useful information about publications, organizations, and authors. Note: Turn on closed captions with the “CC” button or use the text transcript if you prefer to read.
Find better coverage
What if the source you find is low-quality, or you can’t determine if it is reliable or not? Perhaps you don’t really care about the source—you care about the claim that source is making. You want to know if it is true or false. You want to know if it represents a consensus viewpoint, or if it is the subject of much disagreement. A common example of this is a meme you might encounter on social media. The random person or group who posted the meme may be less important than the quote or claim the meme makes.
Your best strategy in this case might actually be to find a better source altogether, to look for other coverage that includes trusted reporting or analysis on that same claim. Rather than relying on the source that you initially found, you can trade up for a higher quality source. The point is that you’re not wedded to using that initial source. We have the internet! You can go out and find a better source, and invest your time there.
We recommend watching this short video [4:10] that demonstrates how to find better coverage and notes how fact-checkers build a library of trusted sources they can rely on to provide better coverage. Note: Turn on closed captions with the “CC” button or use the text transcript if you prefer to read.
Trace claims, quotes, and media to the original context
Much of what we find on the internet has been stripped of context. Maybe there’s a video of a fight between two people with Person A as the aggressor. But what happened before that? What was clipped out of the video and what stayed in? Maybe there’s a picture that seems real but the caption could be misleading. Maybe a claim is made about a new medical treatment based on a research finding—but you’re not certain if the cited research paper actually said that. The people who re-report these stories either get things wrong by mistake, or, in some cases, they are intentionally misleading us.
In these cases you will want to trace the claim, quote, or media back to the source, so you can see it in its original context and get a sense of whether the version you saw was accurately presented. We will talk about this more in Chapter 3 when we distinguish between primary sources and secondary sources. Secondary and tertiary sources are great for getting started with a topic, but researchers want to rely on the most highly informed source to give us information about a topic. If you see a news article about a research study, look for the journal article written by the researchers who performed the study as citations for your paper rather than a journalist who is unaffiliated with the project.
We recommend watching this short video [1:33] that discusses re-reporting vs. original reporting and demonstrates a quick tip: going “upstream” to find the original reporting source. Researchers must follow the thread of information from where they first read it to where it originated in order to understand its truth and value. Social workers who fail to check their sources can spread misinformation within our practice context or come to ill-informed conclusions that hurt clients or communities.
Once you have limited your search to trustworthy sources, ask yourself the following questions when evaluating which of these sources to download:
- Does this source help me answer my working question?
- Does this source help me revise and focus my working question?
- Does this source help me address what my professor expects in a literature review?
- Is this the best source I can find? Is this a primary or secondary source?
- What is the original context of this information?
- Is there controversy surrounding this source?
- Are the publisher and author reputable and unbiased?
Reflect and plan for the future
As you look search the literature, you will learn more about your topic area. You will learn new concepts that become new keywords in new queries. You will continue to come up with search queries and download articles throughout the research process. While we present this material at the beginning of the textbook, that is a bit misleading. You will return to search the literature often during the research process. As such, it is important to keep notes about what you did at each stage. I usually keep a "working notes" document in the same folder as the PDFs of articles I download. I can write down which categories different articles fall into (e.g., theoretical articles, empirical articles), reflect on how my question may need to change, or highlight important unresolved questions or gaps revealed in my search.
Creating and refining your working question will help you identify the key concepts you study will address. Once you identify those concepts, you’ll need to decide how to define them and how to measure them when it comes time to collect your data. As you are reading articles, note how other researchers who study your topic define concepts theoretically in the introduction and measure them in their methods section. Tuck these notes away for the future, when you will have to define and measure these concepts.
You need to be able to speak intelligently about the target population you want to study, so finding literature about their strengths, challenges, and how they have been impacted by historical and cultural oppression is a good idea. Last, but certainly not least, you should consider any potential ethical concerns that could arise during the course of carrying out your research project. These concerns might come up during your data collection, but they may also arise when you get to the point of analyzing data or disseminating results.
Decisions about the various research components do not necessarily occur in sequential order. For example, you may have to think about potential ethical concerns before changing your working question. In summary, the following list shows some of the major components you’ll need to consider as you design your research project. Make sure you have information that will inform how you think about each component.
- Research question
- Literature review
- Theories and causal relationships
- Unit of analysis and unit of observation
- Key concepts (conceptual definitions and operational definitions)
- Method of data collection
- Research participants (sample and population)
- Ethical concerns
Carve some time out each week during the beginning of the research process to revisit your working question. As you write notes on the articles you find, reflect on how that knowledge would impact your working question and the purpose of your research. You still have some time to figure it out. We'll work on turning your working question into a full-fledged research question in Chapter 9.
Key Takeaways
- Research requires fact-checking. The SIFT technique is an easy approach to critically investigating internet resources about your topic.
- Investigate the source of the information you find on the web and look for better coverage.
- Search for internet resources that help you address your working question and write your research proposal.
Exercises
- Look at your professor's prompt for a literature review and sketch out how you might answer those questions using your present level of knowledge. Search for sources that support or challenge what you think is true about your topic.
- Find a news article reporting about topics similar to your working question. Identify whether it is a primary or secondary source. If it is a secondary source, trace any claims to their primary sources. Provide the URLs.
This resource draws from the existing body of open educational resources on social science research methods. This index provides information about which sources were adapted in each chapter. This index does not include attributions for images, which are provided in the Media Attributions section at the end of each chapter before the footnotes.
I should note that this textbook draws heavily from Scientific inquiry in social work by Matt DeCarlo and published by Open Social Work under a CC-BY-NC-SA 4.0 license which adapted a large chunk of content from Principles of sociological inquiry: Quantitative and qualitative methods by Amy Blackstone and published by the Saylor Foundation under a CC-BY-NC-SA license. In many ways, this project is indebted to Dr. Blackstone's seminal scholarly gift.
Chapter 1 adapted content from:
- Chapter 1 of Scientific inquiry in social work by Matt DeCarlo and published by Open Social Work under a CC-BY-NC-SA 4.0 license.
- Cultural Humility: People, Principles and Practices—Part 1 of 4 published by Vivian Chavez under a CC BY-NC-ND 3.0 license.
Chapter 2 adapted content from:
- Sections 2.1 and 15.1 of Scientific inquiry in social work by Matt DeCarlo and published by Open Social Work under a CC-BY-NC-SA 4.0 license.
- How to create a concept map published by the University of Guelph Library under a CC-BY 4.0 license.
Chapters 3, 4, and 5 adapted content from:
- Sections 2.2, 2.3, 3.1, and 3.2 of Scientific inquiry in social work by Matt DeCarlo and published by Open Social Work under a CC-BY-NC-SA 4.0 license.
- Bonanno & Veselak's article A matter of trust: Parents attitudes towards child mental health information sources and published in Advances in Social Work under a CC-BY 4.0 license.
- The section on SIFT from Introduction to College Research by Walter D. Butler, Aloha Sargent, and Kelsey Smith and published under a CC-BY 4.0 license.
- Information privilege by Steely Library NKU and published under a CC-BY 4.0 license.
Chapter 6 adapted content from:
- Chapter 5 of Scientific inquiry in social work by Matt DeCarlo and published by Open Social Work under a CC-BY-NC-SA 4.0 license.
Chapter 7 adapted content from:
- Section 6.2 of Scientific inquiry in social work by Matt DeCarlo CC-BY-NC-SA 4.0 published by Open Social Work under a CC-BY-NC-SA 4.0 license.
- The GO-GN research methods handbook by Farrow, R., Iniesto, F., Weller, M. & Pitt., R. published by the Global OER Graduate Network (GO-GN) under a CC-BY 4.0 license.
- A small amount of content from the Wikipedia entry for postpositivism published under a CC-BY-SA 3.0 license as well as Yosef Jabarin's article on conceptual frameworks published by the
International Journal of Qualitative Methods under a CC-BY 4.0 license.
Chapter 8 adapted content from:
- Sections 6.3, 7.2, and 7.3 of Scientific inquiry in social work by Matt DeCarlo CC-BY-NC-SA 4.0 published by Open Social Work which is adapted from Principles of sociological inquiry: Quantitative and qualitative methods by Amy Blackstone CC-BY-NC-SA published by the Saylor Foundation.
- Chapter 10 of Research methods in psychology (4th edition) by Rajiv S. Jhangiani, I-Chant A. Chiang, Carrie Cuttler, and Dana C. Leighton CC-BY-NC-SA 4.0 from Kwantlen Polytechnic University Library.
Chapter 9 adapted content from:
- Chapter 8 of Scientific inquiry in social work by Matt DeCarlo CC-BY-NC-SA 4.0 published by Open Social Work under a CC-BY-NC-SA 4.0 license.
- Many figures and ideas from Principles of sociological inquiry: Quantitative and qualitative methods by Amy Blackstone CC-BY-NC-SA published by the Saylor Foundation.
Chapter 10 adapted content from:
- Chapter 10 of Scientific inquiry in social work by Matt DeCarlo and published by Open Social Work under a CC-BY-NC-SA 4.0 license.
Chapter 11 adapted content from:
- Chapter 9 of Scientific inquiry in social work by Matt DeCarlo CC-BY-NC-SA 4.0 published by Open Social Work under a CC-BY-NC-SA 4.0 license.
- Chapter 6 of Social Science Research: Principles, Methods, and Practices published by Anol Bhattacherjee under a CC-BY-NC-SA 3.0 Unported license.
- Chapter 35 of Research methods in psychology (4th edition) by Rajiv S. Jhangiani, I-Chant A. Chiang, Carrie Cuttler, and Dana C. Leighton CC-BY-NC-SA 4.0 from Kwantlen Polytechnic University Library.
Chapter 12 adapted content from:
- Chapter 35 of Research methods in psychology (4th edition) by Rajiv S. Jhangiani, I-Chant A. Chiang, Carrie Cuttler, and Dana C. Leighton CC-BY-NC-SA 4.0 from Kwantlen Polytechnic University Library.
- Chapter 9 of Social Science Research: Principles, Methods, and Practices published by Anol Bhattacherjee under a CC-BY-NC-SA 3.0 Unported license.
- The final section on cultural bias adapts a large amount of content from Navigating cross-cultural research: methodological and ethical considerations, an open access article by Tanya Broesch, Alyssa N. Crittenden, Bret A. Beheim, Aaron D. Blackwell, John A. Bunce, Heidi Colleran, Kristin Hagel, Michelle Kline, Richard McElreath, Robin G. Nelson, Anne C. Pisor, Sean Prall, Ilaria Pretelli, Benjamin Purzycki, Elizabeth A. Quinn, Cody Ross, Brooke Scelza, Kathrine Starkweather, Jonathan Stieglitz and Monique Borgerhoff Mulder published by Proceedings of the Royal Academy of Biological Sciences under a CC-BY 4.0 license.
Chapter 13 adapted content from:
- Chapter 7 of Research methods in psychology (3rd American edition) by Paul C. Price, Rajiv S. Jhangiani, I-Chant A. Chiang, Dana C. Leighton, and Carrie Cuttler published by Washington State University Press under a CC-BY-NC-SA 4.0 license.
Chapter 15 adapted content from:
- Chapter 12 of Research methods in psychology (3rd American edition) by Paul C. Price, Rajiv S. Jhangiani, I-Chant A. Chiang, Dana C. Leighton, and Carrie Cuttler CC-BY-NC-SA 4.0 published by Washington State University Press under a CC-BY-NC-SA 4.0 license.
Chapter 19 adapted content from:
- Figures 19.4 and 19.5 are from Growing up in New York City: A Generational Memoir (1941–1960). by Howard R. Wolf and published in American Studies Journal under a CC-BY-SA 4.0 license.
Chapter 21 adapted content from:
- Figure 21.1 is from Street youth labor as an expression of survival and self-worth by Karabanow, J & Gurman, E. and published in Critical Social Work under a CC-BY-NC-ND 4.0 license.
Chapter 24 adapted content from:
- Chapter 16 of Scientific inquiry in social work by Matt DeCarlo CC-BY-NC-SA 4.0 published by Open Social Work under a CC-BY-NC-SA 4.0 license.
Chapter Outline
- The scholarly literature (26 minute read)
- Identifying your information needs (13 minute read)
- Searching basics (23 minute read)
Content warning: examples in this chapter contain references to school discipline, mental health, gender-based discrimination, police shootings, ableism, autism and anti-vaccination conspiracy theories, children’s mental health, child abuse, poverty, substance use disorders and parenting/pregnancy, and tobacco use.
3.1 The scholarly literature
Learning Objectives
Learners will be able to...
- Define what we mean by "the literature"
- Differentiate between primary and secondary sources and summarize why researchers use primary sources
- Distinguish between different types of resources and how they may be useful for a research project
- Identify different types of journal articles
In the last chapter, we discussed how looking at the literature on your topic is the first thing you will need to do as part of a student research project. A literature review makes up the first half of a research proposal. But what do we mean by "the literature?" The literature consists of the published works that document a scholarly conversation on a specific topic within and between disciplines. You will find in “the literature” documents that track the progression of the research and background of your topic. You will also find controversies and unresolved questions that can inspire your own project. By this point in your social work academic career, you’ve probably heard that you need to get “peer-reviewed journal articles.” But what are those exactly? How do they differ from news articles or encyclopedias? That is the focus of this section of the text—different types of literature.
Disciplines
Information does not exist in the environment like some kind of raw material. It is produced by individuals working within a particular field of knowledge who use specific methods for generating new information, or a discipline. Disciplines consume, produce, and share knowledge. Looking through a university’s course catalog gives clues to disciplinary structure. Fields such as political science, biology, history, and mathematics are unique disciplines, as is social work. We have things we study often, like social welfare policy or mental health interventions, and a particular theoretical and ethical framework through which we view the world.
Social work researchers must take an interdisciplinary perspective, engaging with not just social work literature but literature from many disciplines to gain a comprehensive and accurate understanding of a topic. Consider how nursing and social work would differ in studying sports in schools. A nursing researcher might study how sports affect individuals’ health and well-being, how to assess and treat sports injuries, or the physical conditioning required for athletics. A social work researcher might study how schools privilege or punish student athletes, how athletics impact social relationships and hierarchies, or the differences in participation in organized sports between children of different genders. People in some disciplines are much more likely to publish on sports in schools than those in others, like political science or religious studies. I'm sure you can find a bit of scholarship on sports in schools in most disciplines, though you will want to target disciplines that are likely to address the topic often as part of the core purpose of the profession (like sports medicine or sports social work). Sometimes disciplines overlap in their focus, as clinical social work and counseling psychology or urban planning and public administration.
You will need to become comfortable with identifying the disciplines that might contribute information to any literature search. When you do this, you will also learn how to decode the way people talk about a topic within a discipline. This will be useful to you when you begin a review of the literature in your area of study.
Exercises
Think about your research topic and working question. What disciplines are likely to publish about your topic?
For each discipline, consider:
- What is important about the topic to scholars in that discipline?
- What is most likely to be the focus of their study about the topic?
- What perspective are they most likely to have on the topic?
Periodicals
First, let’s discuss periodicals. Periodicals include trade publications, magazines, and newspapers. While they may appear similar, particularly those that are found online, each of these periodicals has unique features designed for a specific purpose. Magazine and newspaper articles are usually written by journalists, are intended to be short and understandable for the average adult, contain color images and advertisements, and are designed as commodities to be sold to an audience. An example of a useful magazine for social workers is the New Social Worker.
Primary sources
Periodicals may contain primary or secondary literature depending on the article in question. An article that is a primary source gathers information as an event happens (e.g. an interview with a victim of a local fire), or it may relay original research reported on by the journalists (e.g. the Guardian newspaper’s The Counted webpage which tracked how many people were killed by police officers in the United States from 2015-2016).[65] Primary sources are based on raw data, which we discussed in Chapter 2.
Is it okay to use a magazine or newspaper as a source in your research methods class? If you were in my class, the answer is “probably not.” There are some exceptions, such as the Guardian page mentioned above (which is a great example of data journalism) or breaking news about a policy or community, but most of what newspapers and magazines publish is secondary information. Researchers should look for primary sources—the original source of information—whenever possible.
Secondary sources
Secondary sources interpret, discuss, and summarize primary sources. Examples of secondary sources include literature reviews written by researchers, book reviews, as well as news articles or videos about recently published research. Your job in this course is to read the original source of the information, the primary source. When you read a secondary source, you are relying on another's interpretation of the primary source. That person might be wrong in how they interpret the primary source, or they may simply focus on a small part of what the primary source says.
Most often, I see the distinction between primary and secondary sources in the references pages students submit for the first few assignments in my research class. Students are used to citing the summaries of scientific findings from periodicals like the New York Times or Washington Post. Instead, students should read the journal article written by the researchers, as it is the primary source. Journalists are not scientists. If you have seen articles about how chocolate cures cancer or how drinking whiskey can extend your life, you should understand how journalists can exaggerate or misinterpret scientific results. See this cartoon from Saturday Morning Breakfast Cereal for a hilarious illustration of sensationalizing findings from scientific research in journalism. Even when news outlets do a very good job of interpreting scientific findings, and that is certainly what most journalistic outlets strive for, you are still likely to only get a short summary of the key results.
When you are getting started with research, newspaper and magazine articles are excellent places to commence your journey into the literature, as they do not require specialized knowledge and may inspire deeper inquiry. Just make sure you are reading news, not opinion pieces, which may exclude facts that contradict the author's viewpoint. As we will talk about in the next chapter, the one secondary source that is highly valuable for student researchers are review articles published in academic journals, as they summarize much of the relevant literature on a topic.
Trade publications
Unlike magazines and newspapers, trade publications may take some specialized knowledge to understand. Trade publications or trade journals are periodicals directed to members of a specific profession. They often include information about industry trends and practical information for people working in the field. Because of this, trade publications are somewhat more reputable than newspapers or magazines, as the authors are specialists in their field.
NASW News, published by the National Association of Social Workers (NASW), is a good example of a trade publication in social work. Its intended audience is social work practitioners who want to learn about important practice issues. They report news and trends in a particular field but not scholarly research. They may also provide product or service reviews, job listings, and advertisements.
So, can you use trade publications in a formal research proposal? Again, if you’re in my class, the answer would be “probably not.” A main shortcoming of trade publications is the lack of peer review. Peer review refers to a formal process in which other esteemed researchers and experts ensure your work meets the standards and expectations of the professional field. Peer review is part of the cycle of publication for journal articles and provides a gatekeeper service, so to speak, which (in theory) ensures that only top-quality articles are published.
In reality, there are a number of problems with our present system of peer review. Briefly, critiques of the current peer review system include unreliable feedback across reviewers; inconsistent error and fraud detection; prolonged evaluation periods; and biases against non-dominant identities, perspectives, and methods. It can take years for a manuscript to complete peer review, likely with one or more rounds of revisions between the author and the reviewers or editors at the journal. See Kelly and colleagues (2014)[66] for an overview of the strengths and limitations of peer review.
A trade publication isn't as reputable as a journal article because it does not include expert peer review. While trade publications do contain a staff of editors, the level of review is not as stringent as with academic journals. On the other hand, if you are completing a study about practitioners, then trade publications may be highly relevant sources of information for your proposal.
In summary, newspapers and other popular press publications are useful for getting general topic ideas. Trade publications are useful for practical application in a profession and may also be a good source of keywords for future searching.
Journal articles
As you’ve probably heard by now, academic journal articles are considered to be the most reputable sources of information, particularly in research methods courses. Journal articles are written by scholars with the intended audience of other scholars (like you!). The articles are often long and contain extensive references that support the author's arguments. The journals themselves are often dedicated to a single topic, like violence or child welfare, and include articles that seek to advance the body of knowledge about a particular topic. We included a list of the 25 most popular social work journals in Chapter 1.
Peer review
Most journals are peer-reviewed (also known as refereed), which means a panel of scholars reviews the articles to provide a recommendation on whether they should be accepted into that specific journal. Scholarly journals make available articles of interest to experts or researchers. An editorial board of respected scholars (i.e., peers) reviews all articles submitted to a journal. Editors and volunteer reviewers decide if the article provides a noteworthy contribution to the field and should be published. For this reason, journal articles are the main source of information for researchers as well as for literature reviews. Usually, peer review is done confidentially, with the reviewer's name hidden from the author and vice versa. This confidentiality is mediated by the editorial staff at an academic journal and is termed blinded review, though this ableist language should be revised to confidential review (Ades, 2020).[67]
You can usually tell whether a journal is peer reviewed by going to its website. Under the “About Us” or "Author Submission" sections, the website should list the procedures for peer review. If a journal does not provide such information, you may have found a “predatory journal.” You may want to check with your professor or a librarian, as the websites for some reputable journals are not straightforward. It is important not to use articles from predatory journals. These journals will publish any article—no matter how bad it is—as long as the author pays them. If a journal appears suspicious and may be a predatory publication, search for it in the COPE member list. Another predatory publishing resource, Beall's List, has been critiqued as discriminatory towards some open access publishing models and non-Western publishers so we do not recommend its use (Berger & Cirasella, 2015).[68]
Even reputable, non-predatory journals publish articles that are later shown to be incorrect, unethically manipulated, or otherwise faulty. In Chapter 1, we talked about Andrew Wakefield's fabricated data linking autism and vaccines. His study was published in The Lancet, one of the most reputable journals in medicine, who formally retracted the article years later. Retraction Watch is a scholarly project dedicated to highlighting when scientific papers are withdrawn and why. It is important to note that peer review is not a perfect process, and that it is subject to error. There is a growing movement to make the peer review process faster and more transparent. For example, 17 life science journals moved to a model of publishing the unreviewed draft, the reviewer's comments, and how they informed the final draft (Brainard, 2019).[69]
Seminal articles
Peers don't just review work prior to publication. The whole point of publishing a journal article is to ensure that your peers will read it and use it in their work. A seminal article is “a classic work of research literature that is more than 5 years old and is marked by its uniqueness and contribution to professional knowledge” (Houser, 2018, p. 112).[70] Basically, it is a really important article. This article by Hodge, Lacasse, and Benson (2012)[71] reviews the most cited social work publications from the previous decade. These would all be considered seminal articles.
There are far more than just these 100 seminal articles in the literature. How do you know if you are looking at a seminal article? Seminal articles are cited a lot in the literature. You can see how many authors have cited an article using Google Scholar’s citation count feature when you search for the article, as depicted in Figure 3.1. Generally speaking, articles that have been cited more often are considered more reputable. There is nothing wrong with citing an article with a low citation count, but it is an indicator that not many other scholars have found the source to be useful or important, at least not yet.
Figure 3.1 shows a citation count for an article that is obviously seminal, with over 4,000 citations. There is no exact number of citations at which an article is considered seminal. A low citation count for an article published last year is not a reliable indicator of the importance of this article to the literature, as it takes time for people to read articles and incorporate them into their scholarship. For example, if you are looking at an article from five years ago with 20 citations, that's still a good quality resource to use. The purpose here is to recognize when articles are clearly influential in the literature, and are therefore more likely to be important to your review of the literature in that topic area.
Empirical articles
Journal articles can fall into one of several categories. Empirical articles report the results of a quantitative or qualitative analysis of raw data conducted by the author. Empirical articles also integrate theory and previous research. However, just because an article includes quantitative or qualitative results does not necessarily mean it is an empirical journal article. Since most articles contain a literature review with findings from empirical studies, you need to make sure the findings reported in the study stem from the author’s analysis of raw data.
Fortunately, empirical articles follow a similar structure, and include the following sections: introduction, methods, results, and discussion—in that order. While the exact wording in the headings may differ slightly from publication to publication and other sections may be added, this general structure applies to nearly all empirical journal articles. If you see this general structure, you are reading an empirical journal article. You can see this structure in a recent article from the open access social work journal, Advances in Social Work. Bonnano & Vesalak (2019)[72] conducted interviews of parents of children who are diagnosed with mental health issues, uncovering that trust was central to which sources parents used to inform themselves about their child's diagnosis. While there is no "Introduction" heading, the article begins on page 397 with the abstract and introduction. The "Methods" heading appears on page 400, after the introduction ends. The "Results" section begins on page 402. Because this is a qualitative empirical article, the results section is full of themes and quotes based on what participants said in their interviews with researchers. If it were a quantitative empirical article, like this one from Kelly, Kubart, and Freed (2020)[73] evaluating the impact of a social work intervention on depression in teenagers, the results section would be full of statistical data, tables, and figures that indicated mathematical relationships between the key variables in the research question. The "Discussion" for the Bonnano & Vesalak begins on page 407, and it contextualizes the findings in their "Results" section within the scholarly literature.
While this article uses the traditional headings, you may see other journal articles that use headings like "Background" instead of "Introduction" or "Findings" instead of "Results". You may also find headings like "Conclusions", "Implications", "Limitations", and so forth in other journal articles. Regardless of the specific wording, empirical journal articles follow the structure of introduction, methods, results, and discussion. If it has a methods and results section, it's almost certainly an empirical journal article. However, if the article does not have these specific sections, it's likely you've found a non-empirical article because the author did not analyze any raw data.
Review articles
Along with empirical articles, review articles are the most important type of article you will read as part of conducting your review of the literature. We will discuss their importance more in Chapter 4, but briefly, review articles are journal articles that summarize the findings other researchers and establish the state of the literature in a given topic area. They are so helpful because they synthesize a wide body of research information in a (relatively) short journal article. Review articles include literature reviews, like this one from Reynolds & Bacon (2018)[74] which summarizes the literature on integrating refugee children in schools. You may find literature reviews with a special focus like a critical review of the literature, which may apply a perspective like critical theory or feminist theory while surveying the scholarly literature on a given topic.
Review articles also include systematic reviews, which are like literature reviews but more clearly specify the criteria by which the authors search for and include scholarly literature in the review (Uman, 2011).[75] For example, this recent systematic review by Leonard, Hafford-Letchfield, and Couchman (2018)[76] discusses how to use strategies from arts education to inform social work education. As you can see in Figure 1 of this article, systematic reviews are very specific about which articles they include in their analysis. Because systematic reviews try to address all scholarly literature published on a given topic, researchers specify how the literature search was conducted, how many articles were included or excluded, and the reasoning for their decisions. This way, researchers make sure there is no relevant source excluded from their analysis.
The two final kinds of review articles, meta-analysis and meta-synthesis go even further than systematic reviews in that they analyze the raw data from all of the articles published in a given topic area, not just the published results. A meta-analysis is a study that combines raw data from multiple sources and analyzes the pooled data using statistics. Meta-analyses are particularly helpful for intervention studies, as it will pool together the raw data from multiple samples and studies to create a super-study of thousands of people, which has greater explanatory power. For example, this recent meta-analysis by Kennedy and colleagues (2016)[77] analyzes pooled data from six separate studies on parent-child interaction therapy to see if it is effective.
A meta-synthesis is similar to a meta-analysis but it pools qualitative results from multiple studies for analysis. For example, this meta-synthesis by Hodge and Horvath (2011)[78] pooled data from 11 qualitative studies that addressed spiritual needs in healthcare settings. While meta-analyses and meta-syntheses are the most methodologically robust type of review article, any recently published review article that is highly relevant to your topic area is a good place to start reading literature. Because review articles synthesize the results of multiple studies, they can give you a broad sense of the overall literature on your topic. We'll review a few kinds of non-empirical articles next.
Theoretical articles
Theoretical articles discuss a theory, conceptual model, or framework for understanding a problem. They may delve into philosophical or ethical analysis as well. For example, this theoretical article by Carrillo & Grady (2018)[79] discusses structural social work theory and anti-oppressive practice approaches. While most students know they need to have statistics to back up what they think is true, it's also important to use theory to inform what you think about your topic. Theoretical articles can help you understand how to think about a topic and may help you make sense of the results of empirical studies.
Practical articles
Practical articles describe “how things are done" from the perspective of on-the-ground practitioners. They are usually shorter than other types of articles and are intended to inform practitioners of a discipline on current issues. They may also reflect on a “hot topic” in the practice domain, a complex client situation, or an issue that may affect the profession as a whole. This practical article by Curry-Stevens and colleagues (2019)[80] was written by social work faculty for other social work faculty. It helps faculty learn how to increase the number of graduate social work students concentrating in macro practice, based on what faculty in one program did.
| Type of journal article | How do you know if you're looking at one? | Why is this type of article useful? |
| Peer-reviewed | Go to the journal's website, and look for information that describes peer review. This may be in the instructions for authors or about the journal sections. | To ensure other respected scholars have reviewed, provided feedback, and approved this article as containing honest and important scientific scholarship. |
| Article in a predatory journal | Use the SIFT method. Search for the publisher and journal on Wikipedia and in a search engine. Check the COPE member list to see if your journal appears there. | They are not useful. Articles in predatory journals are published for a fee and have not undergone serious review. They should not be cited. |
| *Not an journal article* | Does not provide the name of the journal it is published in. You may want to google the name of the article or report and its author to see if you can find a published version. | Journal articles aren't everything, but if your instructor asks for one, make sure you haven't mistakenly use dissertations, theses, and government reports. |
| Empirical article | Most likely, if it has a methods and results section, it is an empirical article. | Once you have a more specific idea in mind for your study, you can adapt the measures, design, and sampling approach of empirical articles (quantitative or qualitative). The introduction and results section can also provide important information for your literature review. Make sure not to rely too heavily on a single empirical study, though, and try to build connections across studies you read. |
| Quantitative article | The methods section contains numerical measures and instruments, and the results section provides outcomes of statistical analyses. | |
| Qualitative article | The methods section describes interviews, focus groups, or other qualitative techniques. The results section has a lot of quotes from study participants. | |
| Review article | The title or abstract includes terms like literature review, systematic review, meta-analysis, or meta-synthesis. | Getting a comprehensive but generalized understanding of the topic and identifying common and controversial findings across studies. |
| Theoretical article | The articles does not have a methods or results section. It talks about theory a lot. | Developing your theoretical understanding of your topic. |
| Practical articles | Usually in a section at the front or back of the journal, separate from full articles. It addresses hot topics in the profession from the standpoint of a practitioner. | Identifying how practitioners in the real world think about your topic. |
| Book reviews, notes, and commentary | Much shorter, usually only a page or two. Should state clearly what kind of note or review they are in the title, abstract, or keywords. | Can point towards sources that provide more substance on the topic you are studying or point out emerging or controversial aspects of your topic. |
What kind of articles should you read?
No one type of article is better than the other, as each serves a different purpose. Seminal articles relevant to your topic area are important to read because the ideas in them are deeply influential in the literature. Theoretical articles will help you understand the social theory behind your topic. Empirical articles should test those theories quantitatively or create those theories qualitatively, a process we will discuss in greater detail in Chapter 8. Review articles help you get a broad survey of the literature by summarizing the results of multiple studies, which is particularly important at the beginning of a literature search. And finally, practical articles will help you understand a practitioner’s on-the-ground experience. Pick the kind of article that gives you the kind of information you need.
Other sources of information
As I mentioned previously, newspaper and magazine articles are good places to start your search (though they should not be the end of your search!). Another source students go to almost immediately is Wikipedia. Wikipedia is a marvel of human knowledge. It is a digital encyclopedia to which anyone can contribute. The entries for each Wikipedia article are overseen by skilled and specialized editors who volunteer their time and knowledge to making sure their articles are correct and up to date. That said, Wikipedia has been criticized for gender and racial bias as well as inequities in the treatment of the Global South (Montgomery et al., 2021).[81]
Encyclopedias
Wikipedia is an example of a tertiary source. We reviewed primary and secondary sources in the previous section. Tertiary sources review primary and secondary sources. Examples of tertiary sources include encyclopedias, directories, dictionaries, and textbooks like this one. Tertiary sources are an excellent place to start (but are not a good place to end) your search. A student might consult Wikipedia, the Open Education Sociology Dictionary, or the Encyclopedia of Social Work to get a general idea of a social work topic. Encyclopedias can also be excellent resources for their citations, and often the citations will be to seminal articles or important primary sources.
The difference between secondary and tertiary sources is not exact, and as we’ve discussed, using one or both at the beginning of a project is a good idea. As your study of the topic progresses, you will naturally have to transition away from secondary and tertiary sources and towards primary sources. We’ve already talked about one particular kind of primary source—empirical journal articles. We will spend more time on this primary source than any other in this textbook. However, it is important to understand how other types of sources can be used as well.
Books
Books contain important scholarly information. They are particularly helpful for theoretical, philosophical, and historical inquiry. For example, in my research on self-determination for individuals with intellectual and developmental disabilities, I needed to define and explore the concept of self-determination. I learned how to define it from the philosophical literature on self-determination and the advocacy literature contained in books. You can use books to learn key concepts, theories, and keywords you can use to find more up-to-date sources. They may also help you understand the scope and foundations of a topic and how it has changed over time.
Some books contain chapters that look like academic journal articles. These are called edited volumes. They contain chapters similar to journal articles, but the content is reviewed by an editor rather than peer reviewers. Edited volumes are considered somewhat less reputable than journal articles, as confidential peer review is still considered to be more objective than editorial review. However, articles in social science journals will often include references to books and edited volumes.
Conference proceedings and presentations
Conferences provide a great source of information. At conferences such as the Council on Social Work Education’s (CSWE) Annual Program Meeting (APM) or your state’s NASW conference, researchers present papers on their most recent research. The papers presented at conferences are sometimes published in a volume called a conference proceeding, though these are not common in social work. Conference proceedings highlight the current discourse in a discipline and can lead you to scholars who are interested in specific research areas. For a list of social work conferences, with a focus on the United States, see Chapter 24.
A word about conference papers: several factors may render these documents difficult to find. It is not unusual that papers delivered at professional conferences are not published in print or electronic form, although an abstract may be available. In these cases, the full paper may only be available from the author or authors. The most important thing to remember is that if you have any difficulty finding a conference proceeding or paper, ask a librarian for assistance.
Gray literature
Another source of information is the gray literature, which consist of research reports released by non-commercial publishers, such as government agencies, policy organizations, and think tanks. If you have already taken a policy class, perhaps you’ve come across the Center on Budget and Policy Priorities (CBPP). The CBPP is a think tank—a group of scholars that conducts research and advocates for political, economic, and social change. Similarly, students often find the Centers for Disease Control website helpful for understanding the prevalence of social problems like mental illness and child abuse. If you find fact sheets, news items, blog posts, or other content from organizations like think tanks or advocacy organizations in your topic area, look at the references cited to find the primary source, rather than citing the secondary source (the fact sheet). While you might be persuaded by their arguments, you want to make sure to account for the author's potential bias.
Think tanks and policy organizations often have a specific viewpoint they support. For example, there are conservative, liberal, and libertarian think tanks. Policy organizations may be funded by donors to push a given message to the public. Government agencies are generally more objective, though they may be less critical of government programs than other sources might be. Similarly, agencies may be less critical in the reports they share with the public. The main shortcoming of gray literature is the lack of peer review that is found in academic journal articles. Government reports are often cited in research proposals, along with some reports from policy organizations.
Dissertations and theses
Dissertations and theses can be rich sources of information and include extensive reference lists you can use to scan for resources. However, dissertations and theses are considered gray literature because they are not peer reviewed. The accuracy and validity of the document itself may depend on the school that awarded the doctoral or master’s degree to the author. Having completed a dissertation, I can attest to the length of time they take to write as well as read. If you come across a relevant dissertation, it is a good idea to read the literature review and note the sources the author used so you can incorporate them into your literature review. However, the data analysis found in these sources is considered less reputable as it has not passed through peer review yet. Dissertations are approved by committees of scholars, so they are considered more reputable than sources that lack review by experts prior to publication. Consider searching for journal articles by the dissertation's author to see if any of the results have been peer reviewed and published. You will also be thankful because journal articles are much shorter than dissertations and theses!
Webpages
The final source of information we are going to talk about is webpages. My graduate research focused on substance abuse and drugs, and I was fond of reading Drug War Rant, a blog about drug policy. It provided me with breaking news about drug policy and editorial opinion about the drug war. I still do the same thing with Twitter, as I follow policy organizations and noted scholars to learn about developments in my interest areas. Podcasts (like the Social Work Podcast or Doin' the Work) are also great sources of news, interviews, and long-form journalism. I would almost never cite a tweet or podcast in a research proposal, but they are excellent places to find primary sources I can read and cite.
Webpages will also help you locate professional organizations and human service agencies that address your problem. Looking at their social media feeds, reports, publications, or “news” sections on an organization’s webpage can clue you into important topics to study. Because anyone can develop their own webpage, they are usually not considered scholarly sources to use in formal writing, but may be useful as you are first learning about a topic. Additionally, many advocacy webpages will provide references for the facts they site, providing you with the primary source. Keep in mind too that webpages can be tricky to quote and cite, as they can be updated or moved at any time. While some reputable sources will include editing notes, others may not. Webpages can be updated without notice.
Use quality sources
We will discuss more specific criteria for evaluating the quality of the sources you choose in section 4.1. For now, as you access and read each source you encounter, remember:
All information sources are not created equal. Sources can vary greatly in terms of how carefully they are researched, written, edited, and reviewed for accuracy. Common sense will help you identify obviously questionable sources, such as tabloids that feature tales of alien abductions, or personal websites with glaring typos. Sometimes, however, a source’s reliability—or lack of it—is not so obvious. You should consider criteria such as the type of source, its intended purpose and audience, the author’s (or authors’) qualifications, the publication’s reputation, any indications of bias or hidden agendas, how current the source is, and the overall quality of the writing, thinking, and design. (Writing for Success, 2015, p. 448).[82]
While each of these sources provides an important facet of our learning process, your research should focus on finding academic journal articles about your topic. These are the primary sources of the research world. While it may be acceptable and necessary to use other primary sources—like books, government reports, or an investigative article by a newspaper or magazine—academic journal articles are preferred. Finding these journal articles is the topic of the next section.
Key Takeaways
- News articles, Wikipedia, and webpages are a good place to start learning about a new topic, but you should rely on more reputable and peer-reviewed sources when reviewing “the literature” on your topic.
- Researchers should read and cite primary sources, tracing a fact or idea to its original source. If you read a news article or webpage that talks about a study or report (i.e., a secondary source), read the primary source to get all the facts.
- Empirical articles report quantitative or qualitative analysis completed by the author on raw data.
- You must critically assess the quality and bias of any source you read. Just because it’s published doesn’t mean it’s true.
Exercises
Conduct a general Google search on your topic.
- Identify whether the first 10 sources that come up are primary or secondary sources.
- Optionally, ask a friend to conduct the same Google search and compare your results. Google's search engine customizes the results you see based on your browsing history and other data the company has collected about you. This is called a filter bubble, and you can learn more about it in this Ted Talk about filter bubbles.
Find a secondary or tertiary source about your topic.
- Trace one to the original, primary source.
Find a review article in your topic area.
- Identify which kind it is (literature review, systematic review, meta-analysis, etc.)
Find a think tank, advocacy organization, or website that addresses your topic.
- Using their information, identify primary sources that might be of use to you. Consider adding them to your social media feed or joining an email newsletter.
3.2 Identifying your information needs
Learning Objectives
Learners will be able to...
- Identify a a need for information knowledge about a subject area
- Identify a search topic question and define it using simple terminology
- Articulate current knowledge on a topic
- Recognize a need for information and data to achieve a specific end and define limits to the information need
- Identify the key concepts relevant to your research question and inquiry into the literature
Scenario
Norm Allknow was having trouble. He had been using computers since he was five years old and thought he knew all there was to know about them. So, when he was given an assignment to write about the impact of the Internet on society, he thought it would be a breeze. He would just write what he knew, and in no time the paper would be finished. In fact, Norm thought the paper would probably be much longer than the required ten pages. He spent a few minutes imagining how impressed his teacher was going to be, and then sat down to start writing.
He wrote about how the Internet had helped him to play online games with his friends, and to keep in touch with distant relatives, and even to do some homework once in a while. Soon he leaned back in his chair and looked over what he had written. It was just half a page long and he was out of ideas.
Identifying a Personal Need for Information
One of the first things you need to do when beginning any information-based project is to identify your personal need for information. This may seem obvious, but it is something many of us take for granted. We may mistakenly assume, as Norm did in the above example, that we already know enough to proceed. Such an assumption can lead us to waste valuable time working with incomplete or outdated information. Information literacy addresses a number of abilities and concepts that can help us to determine exactly what our information needs are in various circumstances. These are discussed below, and are followed by exercises to help develop your fluency in this area.
Information literacy skills are necessary because the landscape of scientific information is constantly evolving. New information is always added to what is known about your topic. Trained experts, informed amateurs, and opinionated laypeople are publishing in traditional and emerging formats; there is always something new to find out. The scale of information available varies according to topic, but in general it’s safe to say that there is more information accessible now than ever before.
Due to the extensive amount of information available, part of becoming more information literate is developing habits of mind and of practice that enable you to continually seek new information and to adapt your understanding of topics according to what you find. Because of the widely varying quality of new information, evaluation is also a key element of information literacy, and will be addressed in the next chapter of this book.
Finally, while you are busy searching for information on your current topic, be sure to keep your mind open for new avenues or angles of research that you haven’t yet considered. Often the information you found for your initial need will turn out to be the pathway to a rich vein of information that can serve as raw material for many subsequent projects.
When you understand the information environment where your information need is situated, you can begin to define the topic more clearly and you can begin to understand where your research fits in with related work that precedes it. Your information literacy skills will develop against this changing background as you use the same underlying principles to do research on a variety of topics.
What do you already know?
Part of identifying your own information need is giving yourself credit for what you already know about your topic. It can be helpful to visualize your information need using a chart. Construct a chart using the following format to list whatever you already know about the topic.
A simple tracking chart can be seen in Figure 3.2. In the first column, list what you know about your topic. In the second column, briefly explain how you know this (heard it from the professor, read it in the textbook, saw it on a blog, etc.). In the last column, rate your confidence in that knowledge. Are you 100% sure of this bit of knowledge, or did you just hear it somewhere and assume it was right?
When you’ve looked at everything you think you know about the topic and why, step back and look at the chart as a whole. How much do you know about the topic, and how confident are you about it? You may be surprised at how little or how much you already know, but either way you will be aware of your own background on the topic. This self-awareness is key to becoming more information literate.
This exercise gives you a simple way to gauge your starting point, and may help you identify specific gaps in your knowledge of your topic that you will need to fill as you proceed with your research. It can also be useful to revisit the chart as you work on your project to see how far you’ve progressed, as well as to double check that you haven’t forgotten an area of weakness.
Once you’ve clearly stated what you do know, it should be easier to state what you don’t know. Keep in mind that you are not attempting to state everything you don’t know. You are only stating what you don’t know in terms of your current information need. This is where you define the limits of what you are searching for. These limits enable you to meet both size requirements and time deadlines for a project. If you state them clearly, they can help to keep you on track as you proceed with your research. You can learn more about this in the Scope chapter of this book.
One useful way to keep your research on track is with a “KWHL” chart. This type of chart enables you to state both what you know and what you want to know, as well as providing space where you can track your planning, searching and evaluation progress. For now, just fill out the first column, but start thinking about the gaps in your knowledge and how they might inform your research questions. You will learn more about developing these questions and the research activities that follow from them as you work through this book.
Revising your working question
Once you have identified your own need for knowledge, investigated the existing information on the topic, and set some limits on your research based on your current information need, write out your research question or state your thesis. The next exercise will help you conceptualize your question and the potential answers you are considering.
You’ll find that it’s not uncommon to revise your question or thesis statement several times in the course of a research project. As you become more and more knowledgeable about the topic, you will be able to state your ideas more clearly and precisely, until they almost perfectly reflect the information you have found.
Exercises
- State your working question.
- Write your proposed answer to that question...as best you know right now.
- It doesn’t have to be perfect at this point, but based on your current understanding of your topic and what you expect or hope to find is the answer to the question you asked.
Look at your question and your thesis/hypothesis, and make a list of the terms common to both lists (excluding “the,” “and,” “a,” etc.). These common terms are likely the important concepts that you will need to research to support your thesis/hypothesis. They may be the most useful search terms overall or they may only be a starting point.
If none of the terms from your question and thesis/hypothesis lists overlap at all, you might want to take a closer look and see if your thesis/hypothesis really answers your research question. If not, you may have arrived at your first opportunity for revision. If your question asks about self-esteem and your hypothesis talks about self-concept, the answers you seek will not address your working question.
- Does your question really ask what you’re trying to find out?
- Does your proposed answer really answer that question?
You may find that you need to change one or both, or to add something to one or both to really get at what you’re interested in. This is part of the process, and you will likely discover that as you gather more information about your topic, you will find other ways that you want to change your question or thesis to align with the facts, even if they are different from what you hoped.
Defining a research question can be more difficult than it seems. Your initial questions may be too broad or too narrow. You may not be familiar with specialized terminology used in the field you are researching. You may not know if your question is worth investigating at all.
These problems can often be solved by a preliminary investigation of scholarly information on the topic. As previously discussed, gaining a general understanding of the information environment helps you to situate your information need in the relevant context and can also make you aware of possible alternative directions for your research. On a more practical note, however, reading through some of the existing information can also provide you with commonly used terminology, which you can then use to state your own research question, as well as in searches for additional information. Don’t try to reinvent the wheel, but rely on the experts who have laid the groundwork for you to build upon. Find good, argumentative writing from scholars, experts, and people with lived experience that you find insightful.
Key Takeaways
- Researchers continually revise their working question and the answers they are considering throughout the literature search process.
- Your existing knowledge on a topic is a resource you can rely upon, but you will be required as social work practitioners to show evidence that what you say is scientifically true.
- As you learn more about your topic (theories, key terms, history, etc.), you should revise your working question and the answers you are considering to it.
3.3 Searching basics
Learning Objectives
Learners will be able to...
- Brainstorm potential keywords and search queries for your topic
- Apply basic Boolean search operators to your search queries
- Identify relevant databases in which to search for journal articles
- Target your search queries in databases to the most relevant results
One of the drawbacks (or joys, depending on your perspective) of being a researcher in the 21st century is that we can do much of our work without ever leaving the comfort of our recliners. This is certainly true of familiarizing yourself with the literature. Most libraries offer incredible online search options and access to all of the literature you'll ever need on your topic.
Unfortunately, even though many college classes require students to find and use academic journal articles, there is little training for students on how to do so. The purpose of this section and the rest of part 1 of the textbook is to build information literacy skills, which are the basic building blocks of any research project. While we describe the process here as linear, with one step following neatly after another, in reality, you will likely go back to Step 1 often as your topic gets more specific and your working question becomes clearer. Recall our diagram in the last chapter, Figure 2.1. You are cycling between reformulating your working question then searching for and reading the literature on your topic.
The process of searching databases and revising your working question can be time-consuming, so it is a good idea to write notes to yourself. Remembering why you included certain search terms, searched different databases, or other decisions in the literature search process can help you keep track of what you've done. You can then recreate searches to download articles you may have missed or remember reflections you had weeks ago when you last conducted a literature search. When I search the literature, I create a "working document" in a word processor where I write notes and reflections.
Step 1: Build search queries using keywords
What do you type when you are searching for something on in a search engine like Google or Bing? Are you a question-asker? Do you type in full sentences or just a few keywords? What you type into a database or search engine is called a query. Well-constructed queries get you to the information you need more quickly, while unclear queries will force you to sift through dozens of irrelevant articles before you find the ones you want. The danger in being a question-asker or typing like you talk is that including too many extraneous words will confuse the search engine. In this section, you will learn to use keywords rather than natural language to find scholarly information.
Keywords are the words or phrases you use in your search query, and they inform the relevance and quantity of results you get. Unfortunately, different studies often use different terms to mean the same thing. A study may describe its topic as substance abuse, rather than addiction (for example). Think about what keywords are important to answering your working question. Are there other ways of expressing the same idea?
Often in social work research, there is a bit of jargon to become familiar with when crafting your search queries. If you wanted to learn more about individuals of low SES who do not have access to a bank account, you may need to learn the term “unbanked,” which refers to people without bank accounts, and thereby include the word “unbanked” in your search query. If you wanted to learn about children who take on parental roles in families, you may need to include the word “parentification” as part of your search query.
As graduate researchers, you are not expected to know these terms ahead of time. Instead, start with the keywords you already know. As you read more about your topic and become familiar with some of these words, begin including these keywords into your search results, as they may return more relevant hits.
Exercises
- List all of the keywords you can think of that are relevant to your working question. Use your list of keywords from the previous section as a starting point.
- Add new keywords to this list as you search the literature and learn more about your topic.
- Start a document in which you can put your keywords as well as your notes and reflections on the search process.
Alternate keywords
think of synonyms or related terms for each concept. If you do this, you will have more flexibility when searching in case your first search term doesn’t produce any or enough results. This may sound strange, since if you are looking for information using a Web search engine, you almost always get too many results. Databases, however, contain fewer items, and having alternative search terms may lead you to useful sources. Even in a search engine like Google, having terms you can combine thoughtfully will yield better results.
The following worksheet is an example of a process you can use to come up with search terms. It illustrates how you might think about the topic of violence in high schools. Notice that this exact phrase is not what will be used for the search. Rather, it is a starting point for identifying the terms that will eventually be used.
Now, use a clean copy of the same worksheet to think about the topic Sarah’s team is working on. How might you divide their topic into concepts and then search terms? Keep in mind that the number of concepts will depend on what you are searching for. And that the search terms may be synonyms or narrower terms. Occasionally, you may be searching for something very specific, and in those cases, you may need to use broader terms as well. Jot down your ideas then compare what you have written to the information on the second, completed worksheet and identify 3 differences.
Boolean searching: Learn to talk like a robot
Google is a “natural language” search engine, which means it tries to use its knowledge of how people talk to better understand your query. Google’s academic database, Google Scholar, utilizes that same approach. However, other databases that are important for social work research—such as Academic Search Complete, PsycINFO, and PubMed—will not return useful results if you ask a question or type a sentence or phrase as your search query. Instead, these databases are best used by typing in keywords. Instead of typing “the effects of cocaine addiction on the quality of parenting,” you might type in “cocaine AND parenting” or “addiction AND child development.” [Note: you would not actually use the quotation marks in your search query for the examples in this subsection].
These operators (AND, OR, NOT) are part of what is called Boolean searching. Boolean searching works like a simple computer program. Your search query is made up of words connected by operators. Searching for “cocaine AND parenting” returns articles that mention both cocaine and parenting. There are lots of articles on cocaine and lots of articles on parenting, but fewer articles on that address both of those topics. In this way, the AND operator reduces the number of results you will get from your search query because both terms must be present.
The NOT operator also reduces the number of results you get from your query. For example, perhaps you wanted to exclude issues related to pregnancy. Searching for “cocaine AND parenting NOT pregnancy” would exclude articles that mentioned pregnancy from your results. Conversely, the OR operator would increase the number of results you get from your query. For example, searching for “cocaine OR parenting” would return not only articles that mentioned both words but also those that mentioned only one of your two search terms. This relationship is visualized in Figure 3.2 below.
Exercises
- Build a simple Boolean search query, appropriately using AND, OR, and NOT.
- Experiment with different Boolean search queries by swapping in different keywords.
- Write down which Boolean queries you create in your notes.
Step 2: Find the right databases
Queries are entered into a database or a searchable collection of information. In research methods, we are usually looking for databases that contain academic journal articles. Each database has a unique focus and presents distinct advantages or disadvantages. It is important to try out multiple databases, and experiment with different keywords and Boolean operators to find the most relevant results for your working question.
Google Scholar
The most commonly used public database of scholarly literature is Google Scholar, and there are many benefits to using it. Most importantly, Google Scholar is free and is the only database to which you will have access after you graduate. Building competence at using it now will help you as you engage in the evidence-based practice process post-graduation. Because Google Scholar is a natural language search engine, it can seem less intimidating. However, Google Scholar also understands Boolean search terms and contains some advanced searching features like searching within one specific author's body of work, one journal, or searching for keywords in the title of the journal. We have previously mentioned the 'Cited By' field which counts the number of other publications that cite the article and a link to explore and search within those citing articles. This can be helpful when you need to find more recent articles about the same topic.
I recommend linking your personal Google account and your university login which provides one-click access to journal articles available at your institution. I also often use their citation generator, though I have to fix a few things on most entries before using them in my references. You can also set email alerts for a specific search query, which can help you stay current on the literature over time, and organize journal articles into folders. If these advanced features sound useful to you, consult Google Scholar Search Tips for a walkthrough.
While these features are great, there are some limitations to Google Scholar. The sources in Google Scholar are not curated by a professional body like the American Psychological Association (who curates PsycINFO). Instead, Google crawls the internet for what it thinks are scholarly works and indexes them in their database. When you search in Google Scholar, the sources will be not only journal articles, but books, government and foundation reports, gray literature, graduate theses, and articles that have not undergone peer review. There is no way to select for a specific type of source, and you need to look clearly and identify what kind of source you are reading. Not every source on Google Scholar is reputable or will match what you need for an assignment.
With that broad focus, Google Scholar ranks the results from your query by the number of times each source was cited by other scholars and the number of times your keywords are mentioned in the source. This biases the search results towards older, more seminal articles. So, you should consider limiting your results to only those published the last few years. Unfortunately, Google Scholar also lacks advanced searching features of other databases, like searching by subject area or within the abstract of an article.
Exercises
- Type your queries into Google Scholar. See which queries provide you with the most relevant results.
- Look for new keywords in the results you find, particularly any jargon used in that topic area you may not have known before.
- Write down notes on which queries and keywords work best. Reflect on what you read in the abstracts and titles of the search results in Google Scholar.
Databases accessed via an institution
Although learning Google Scholar is important, you should take advantage of the databases to which your institution purchases access. The first place to look for social work scholarship are two databases: Social Service Abstracts and Social Work Abstracts, as these databases are indexed specifically for the discipline of social work. As you are aware, social work is interdisciplinary by nature, and it is a good idea not to limit yourself solely to the social work literature on your topic. If your study requires information from the disciplines of psychology, education, or medicine, you will likely find the databases PsycINFO, ERIC, and PubMed (respectively) to be helpful in your search.
There are also less specialized databases that contain articles from across social work and other disciplines. Academic Search Complete is similar to Google Scholar in its breadth, as it contains a number of smaller databases from a variety of social science disciplines. Within Academic Search Complete, click Choose Databases to select databases that apply to different disciplines or topic areas.
It is worth mentioning that many university libraries have a meta-search engine which searches some of the databases to which they have access, including both disciplinary and interdisciplinary databases. Meta-search engines are fantastic, but you must be careful to narrow your results down to the type of resource you are looking for, as they will include all types of media (e.g., books, movies, games). Unfortunately, not every database is included in these meta-search engines, so you will want to try other databases, as well.
You can find the databases we mentioned here and others on the Databases page of your university library (at my institution, the page is titled A-Z Databases). A university login is required to for you to access these databases because you pay for access as part of the tuition and fees at your university. This means that sometime after graduation, you will lose access to these databases and would need to physically travel to a university library to access them. Make the most of your database access during your graduate degree program. We will review in Chapter 4 how to get around paywalls after you graduate.
The databases we described here can be a bit more intimidating, as they have limited natural language support and rely mostly on Boolean searching. However, they contain more advanced search features which can help you narrow down your search to only the most relevant sources, our next step in the research process.
Exercises
- Explore different databases you can access via your university's library website and searching using your keywords.
- Write notes on which databases and keywords provide you with the most relevant results and the disciplines you are likely to find in each database.
- Look for any new keywords in your results that will help you target your search, and experiment with new search queries.
Step 3: Reduce irrelevant search results
At this point, you should have worked on a few different search queries, swapped in and out different keywords, and explored a few different databases of academic journal articles relevant to your topic. Next, you have to deal with the most common frustration for students: narrowing down your search and reducing the number of results you have to look at. I'm sure at some point you've typed a query into Google Scholar or your library's search engine and seen hundreds of thousands or millions of sources in your search results. Don’t worry! You don’t have to read all those articles to know enough about your topic area to produce a good research project.
While reading millions of articles is clearly ridiculous, there is no magic number of search results to reach. A good search will return results which are highly relevant. In my experience, student search queries tend to be too general (and have too many irrelevant results), rather than too specific (and have too few relevant results). Don't worry! You will not need to read all of the articles you find, but reducing the number of results will save you time by eliminating irrelevant articles.
You should have two goals in reducing the number of results:
- Reduce the number of sources until you could reasonably skim through the title and abstract of each source. Generally, you want a hundred or a thousand results, rather than a hundred thousand results.
- Reduce the number of irrelevant sources in your search results until you are much more likely to encounter relevant, rather than irrelevant articles. If only one of every ten results in your search is relevant to your topic for example, you are wasting time. You would be better served by using the tips below to better target your search query.
Here are some tips for reducing the number of irrelevant sources when you search in databases.
- Use quotation marks to indicate exact phrases, like “mental health” or “substance abuse.”
- Search for your keywords in the ABSTRACT. A lot of your results may be from articles about irrelevant topics that mention your search term only once. If your topic isn’t in the abstract, chances are the article isn’t relevant. You can be even more restrictive and search for your keywords in the TITLE, but that may be too restrictive, and exclude relevant articles with titles that use slightly different phrasing. Academic databases provide these options in their advanced search tools.
- Use SUBJECT headings to find relevant articles. These tags are added by the organization that provides the database, like the American Psychological Association who curates PyscINFO, to help classify and categorize information for easier browsing.
- Narrow down the years of publication. Unless you are gathering historical information about a topic, you are unlikely to find articles older than 10-15 years to be useful. They no longer tell you the current knowledge on a topic. All databases have options to narrow your results down by year. Check with your professor to see if they have specific guidelines for when an article is "too old."
- Talk to a librarian. They are professional knowledge-gatherers, and there is often a librarian assigned to your department. Their job is to help you find what you need to know, and they are extensively trained in how to help you!
- Talk to your stakeholders or someone knowledgeable about your target population. They have lived experience with your topic which can help you understand the literature through a different lens. Moreover, the words they use to describe their experiences can also be useful sources of keywords, theories, studies, or jargon.
Exercises
- Using the techniques described in this subsection, reduce the number of irrelevant results in the database queries you have performed so far. Pay particular attention to searching within the abstract, using quotation marks to indicate keyword phrases, and exploring subject headings relevant to your topic.
- Reduce your database query results down to a number (a) where you could reasonably skim through the titles and abstracts to identify relevant articles and (b) where you are much more likely to encounter relevant, rather than irrelevant articles. Repeat this process for your search of each database relevant to your project.
- Write down your queries or save them, so you can recreate them later if you need to return to them. Also, write down any reflections you have on the search process.
- Look for any new keywords in your results that will help you target your search better, and experiment with new search queries.
Step 4: Conduct targeted searches
Another way to save time when searching for literature is to look for articles that synthesize the results of other articles: review articles. Systematic reviews provide a summary of the existing literature on a topic. If you find one on your topic, you will be able to read one person’s summary of the literature and go deeper by reviewing and reading articles they cited in their references. Other types of reviews such as critical reviews may offer a viewpoint along with a birds-eye view of the literature.
Similarly, meta-analyses and meta-syntheses have long reference lists that are useful for finding additional sources on a topic. They use data from each article to run their own quantitative or qualitative data analysis. In this way, meta-analyses and meta-syntheses provide a more comprehensive overview of a topic. To find these kinds of articles, include the term “meta-analysis,” “meta-synthesis,” or “systematic review” as keywords in your search queries.
My advice is to read a review article first, before any type of article. The purpose of review articles is to summarize an entire body of literature is as short an article as possible. This is exactly what students who are learning a new area need. Time is a precious resource for graduate students, and review articles provide the most knowledge in the shortest time. Even if they are a few years old, they should help you understand what else you need to find in your literature search.
As you look through abstracts of articles in your search results, you should begin to notice that certain authors keep appearing. If you find an author that has written multiple articles on your topic, consider searching the AUTHOR field for that particular author. You can also search the web for that author’s Curriculum Vitae or CV (an academic resume) that will list their publications. Many authors maintain personal websites or host their CV on their university department’s webpage. Just type in their name and “CV” into a search engine. For example, you may find Michael Sherraden’s name often if your search query is about assets and poverty. You can find Michael Sherraden's CV on the Washington University of St. Louis website.
A similar process can be used for journal names. As you are going through search results, you may also notice that many of the articles you’ve skimmed come from the same journals. Searching with that journal name in the JOURNAL field will allow you to narrow down your results to just that journal. For example, if you are searching for articles related to values and ethics in social work, you might want to search for articles in the Journal of Social Work Values and Ethics. You can do so within any database that indexes the journal, like Google Scholar, or through the search feature journal’s webpage. Browse the abstracts and download the full-text PDF of any article you think is relevant.
Step 5: Explore references and citations
The last thing you'll need to do in order to make sure you didn't miss anything is to look at the references in each article you cite. If there are articles that seem relevant to you, you may want to download them. Unfortunately, references may contain out-of-date material and cannot be updated after publication. As a result, a reference section for an article published in 2014 will only include references from pre-2014.
To find research published after 2014 on the same topic, you can use Google Scholar’s 'Cited By' feature to do a future-looking search. Look up your article on Google Scholar and click the 'Cited By' link. The results are a list of all the sources that cite the article you just read. Google Scholar also allows you to search within these articles—check the box below the search bar to do so—and it will search within the 'Cited By' articles for your keywords.
Exercises
- Examine the search results from various databases and look for authors and journals that publish often on your topic. Conduct targeted searches of these authors and journals to make sure you are not missing any relevant sources.
- Identify at least one relevant article, preferably a review article. Look in the references for any other sources relevant to your working question. Search for the article in Google Scholar and use the the 'Cited By' feature to look for additional sources relevant to your working question.
- Search for review articles that will help you get a broad sense of the literature. Depending on your topic, you may also want to look for other types of articles as well.
- Look for any new keywords in your results that will help you target your search, and experiment with new search queries.
- Write notes on your targeted searches, so you can recreate them later and remember your reflections on the search process.
Getting the right results
I read a lot of literature review drafts of that say "this topic has not been studied in detail." Before you come to that conclusion, make an appointment with a librarian. It is their job to help you find material that is relevant to your academic interests. My first project as a graduate research assistant involved writing a literature review for a professor, and the time I spent working with a librarian on that project brought home how important and impactful they can be. But there is a lot you can do on your own!
Let’s walk through an example. Imagine a local university wherein smoking was recently banned, much to the chagrin of a group of student smokers. Students were so upset by the idea that they would no longer be allowed to smoke on university grounds that they staged several smoke-outs during which they gathered in populated areas around campus and enjoyed a puff or two. Their activism was surprising. They were advocating for the freedoms of people committing a deviant act—smoking—that is highly disapproved of. Were the protesters otherwise politically active? How much effort and coordination had it taken to organize the smoke-outs?
The student researcher began their research by attempting to familiarize themselves with the literature. They started by looking to see if there were any other student smoke-outs using a simple Google search. When they turned to the scholarly literature, their search in Google Scholar for “college student activist smoke-outs,” yielded no results. Concluding there was no prior research on their topic, they informed the professor that they would not be able to write the required literature review since there was no literature for them to review. What went wrong here?
The student had been too narrow in their search for articles in their first attempt. They went back to Google Scholar and searched again using queries with different combinations of keywords. Rather than searching for “college student activist smoke-outs” they tried, “college student activism," "smoke-out," "protest," and other keywords. This time their search yielded many articles. Of course, they were not all focused on pro-smoking activist efforts, but the results addressed their population of interest, college students, and on their broad topic of interest, activism. Experimenting with different keywords across different databases helped them get a comprehensive and multi-disciplinary view of the topic.
Reading articles on college student activism might give them some idea about what other researchers have found in terms of what motivates college students to become involved in activist efforts. They could also play around with their search terms and look for research on activism centered on other sorts of activities that are perceived by some to be deviant, such as marijuana use or veganism. In other words, they needed to broaden their search about college student activism to include more than just smoke-outs and understand the theory and empirical evidence around student activism.
Once they searched for literature about student activism, they could link it to their specific research focus: smoke-outs organized by college student activists. What is true about activism broadly is not necessarily true of smoke-outs, as advocacy for smokers is unique from advocacy on other topics like environmental conservation. Engaging with the sociological literature about their target population, college students who smoke cigarettes, for example, helped to link the broader literature on advocacy to their specific topic area.
Revise your working question often
Throughout the process of creating and refining search queries, it is important to revisit your working question. In this example, trying to understand how and why the smoke-out took place is a good start to a working question, but it will likely need to be developed further to be more specific and concrete. Perhaps the researcher wants to understand how the protest was organized using social media and how social media impacts how students perceived the protests when they happened. This is a more specific question than "how and why did the smoke-out take place?" though you can see how the researcher started with a broad question and made it more specific by identifying one aspect of the topic to investigate in detail. You should find your working question shifting as you learn more about your topic and read more of the literature. This is an important sign that you are critically engaging with the literature and making progress. Though it can often feel like you are going in circles, there is no shortcut to figuring out what you need to know to study what you want to study.
Key Takeaways
- The quality of your search query determines the quality of your search results. If you don't work on your queries, you’ll get a million results, only a small percentage of which are relevant to your project.
- Keywords and queries should be updated as you learn more about your topic.
- Your two goals in targeting your database searches should be reducing the number of results for each search query to a number you can feasibly skim and making those results more relevant to your project.
- Techniques to increase the number of relevant results in a search include using Boolean operators, quotation marks, searching in the abstract or title, and using subject headings.
- Use the databases that are most relevant to your topic. For general searches, Google Scholar has some strengths (ease of use, Cited By) and limitations (cannot search in abstract, includes resources other than journal articles). You can access other databases through your university's library.
- Don’t be afraid to reach out to a librarian or your professor for help with searching. It is their job to help you.
Exercises
- Reflect on your working question. Consider changes that would make it clearer and more specific based on the literature you have skimmed during your search.
- Describe how your search queries (across different databases) address your working question and provide a comprehensive view of the topic.
