28
Matthew DeCarlo
Chapter Outline
- Inductive and deductive reasoning (15 minute read)
- Nomothetic causal relationships (17 minute read)
- Idiographic causal relationships (12 minute read)
- Mixed methods research (8 minute read)
Content warning: examples in this chapter include references to sexual harassment, domestic violence, gender-based violence, the child welfare system, substance use disorders, neonatal abstinence syndrome, child abuse, racism, and sexism.
8.1 Inductive and deductive reasoning
Learning Objectives
Learners will be able to…
- Describe inductive and deductive reasoning and provide examples of each
- Identify how inductive and deductive reasoning are complementary
Congratulations! You survived the chapter on theories and paradigms. My experience has been that many students have a difficult time thinking about theories and paradigms because they perceive them as “intangible” and thereby hard to connect to social work research. I even had one student who said she got frustrated just reading the word “philosophy.”
Rest assured, you do not need to become a theorist or philosopher to be an effective social worker or researcher. However, you should have a good sense of what theory or theories will be relevant to your project, as well as how this theory, along with your working question, fit within the three broad research paradigms we reviewed. If you don’t have a good idea about those at this point, it may be a good opportunity to pause and read more about the theories related to your topic area.
Theories structure and inform social work research. The converse is also true: research can structure and inform theory. The reciprocal relationship between theory and research often becomes evident to students when they consider the relationships between theory and research in inductive and deductive approaches to research. In both cases, theory is crucial. But the relationship between theory and research differs for each approach.
While inductive and deductive approaches to research are quite different, they can also be complementary. Let’s start by looking at each one and how they differ from one another. Then we’ll move on to thinking about how they complement one another.
Inductive reasoning
A researcher using inductive reasoning begins by collecting data that is relevant to their topic of interest. Once a substantial amount of data have been collected, the researcher will then step back from data collection to get a bird’s eye view of their data. At this stage, the researcher looks for patterns in the data, working to develop a theory that could explain those patterns. Thus, when researchers take an inductive approach, they start with a particular set of observations and move to a more general set of propositions about those experiences. In other words, they move from data to theory, or from the specific to the general. Figure 8.1 outlines the steps involved with an inductive approach to research.
There are many good examples of inductive research, but we’ll look at just a few here. One fascinating study in which the researchers took an inductive approach is Katherine Allen, Christine Kaestle, and Abbie Goldberg’s (2011)[1] study of how boys and young men learn about menstruation. To understand this process, Allen and her colleagues analyzed the written narratives of 23 young cisgender men in which the men described how they learned about menstruation, what they thought of it when they first learned about it, and what they think of it now. By looking for patterns across all 23 cisgender men’s narratives, the researchers were able to develop a general theory of how boys and young men learn about this aspect of girls’ and women’s biology. They conclude that sisters play an important role in boys’ early understanding of menstruation, that menstruation makes boys feel somewhat separated from girls, and that as they enter young adulthood and form romantic relationships, young men develop more mature attitudes about menstruation. Note how this study began with the data—men’s narratives of learning about menstruation—and worked to develop a theory.
In another inductive study, Kristin Ferguson and colleagues (Ferguson, Kim, & McCoy, 2011)[2] analyzed empirical data to better understand how to meet the needs of young people who are homeless. The authors analyzed focus group data from 20 youth at a homeless shelter. From these data they developed a set of recommendations for those interested in applied interventions that serve homeless youth. The researchers also developed hypotheses for others who might wish to conduct further investigation of the topic. Though Ferguson and her colleagues did not test their hypotheses, their study ends where most deductive investigations begin: with a theory and a hypothesis derived from that theory. Section 8.4 discusses the use of mixed methods research as a way for researchers to test hypotheses created in a previous component of the same research project.
You will notice from both of these examples that inductive reasoning is most commonly found in studies using qualitative methods, such as focus groups and interviews. Because inductive reasoning involves the creation of a new theory, researchers need very nuanced data on how the key concepts in their working question operate in the real world. Qualitative data is often drawn from lengthy interactions and observations with the individuals and phenomena under examination. For this reason, inductive reasoning is most often associated with qualitative methods, though it is used in both quantitative and qualitative research.
Deductive reasoning
If inductive reasoning is about creating theories from raw data, deductive reasoning is about testing theories using data. Researchers using deductive reasoning take the steps described earlier for inductive research and reverse their order. They start with a compelling social theory, create a hypothesis about how the world should work, collect raw data, and analyze whether their hypothesis was confirmed or not. That is, deductive approaches move from a more general level (theory) to a more specific (data); whereas inductive approaches move from the specific (data) to general (theory).
A deductive approach to research is the one that people typically associate with scientific investigation. Students in English-dominant countries that may be confused by inductive vs. deductive research can rest part of the blame on Sir Arthur Conan Doyle, creator of the Sherlock Holmes character. As Craig Vasey points out in his breezy introduction to logic book chapter, Sherlock Holmes more often used inductive rather than deductive reasoning (despite claiming to use the powers of deduction to solve crimes). By noticing subtle details in how people act, behave, and dress, Holmes finds patterns that others miss. Using those patterns, he creates a theory of how the crime occurred, dramatically revealed to the authorities just in time to arrest the suspect. Indeed, it is these flashes of insight into the patterns of data that make Holmes such a keen inductive reasoner. In social work practice, rather than detective work, inductive reasoning is supported by the intuitions and practice wisdom of social workers, just as Holmes’ reasoning is sharpened by his experience as a detective.
So, if deductive reasoning isn’t Sherlock Holmes’ observation and pattern-finding, how does it work? It starts with what you have already done in Chapters 3 and 4, reading and evaluating what others have done to study your topic. It continued with Chapter 5, discovering what theories already try to explain how the concepts in your working question operate in the real world. Tapping into this foundation of knowledge on their topic, the researcher studies what others have done, reads existing theories of whatever phenomenon they are studying, and then tests hypotheses that emerge from those theories. Figure 8.2 outlines the steps involved with a deductive approach to research.
While not all researchers follow a deductive approach, many do. We’ll now take a look at a couple excellentrecent examples of deductive research.
In a study of US law enforcement responses to hate crimes, Ryan King and colleagues (King, Messner, & Baller, 2009)[3] hypothesized that law enforcement’s response would be less vigorous in areas of the country that had a stronger history of racial violence. The authors developed their hypothesis from prior research and theories on the topic. They tested the hypothesis by analyzing data on states’ lynching histories and hate crime responses. Overall, the authors found support for their hypothesis and illustrated an important application of critical race theory.
In another recent deductive study, Melissa Milkie and Catharine Warner (2011)[4] studied the effects of different classroom environments on first graders’ mental health. Based on prior research and theory, Milkie and Warner hypothesized that negative classroom features, such as a lack of basic supplies and heat, would be associated with emotional and behavioral problems in children. One might associate this research with Maslow’s hierarchy of needs or systems theory. The researchers found support for their hypothesis, demonstrating that policymakers should be paying more attention to the mental health outcomes of children’s school experiences, just as they track academic outcomes (American Sociological Association, 2011).[5]
Complementary approaches
While inductive and deductive approaches to research seem quite different, they can actually be rather complementary. In some cases, researchers will plan for their study to include multiple components, one inductive and the other deductive. In other cases, a researcher might begin a study with the plan to conduct either inductive or deductive research, but then discovers along the way that the other approach is needed to help illuminate findings. Here is an example of each such case.
Dr. Amy Blackstone (n.d.), author of Principles of sociological inquiry: Qualitative and quantitative methods, relates a story about her mixed methods research on sexual harassment.
We began the study knowing that we would like to take both a deductive and an inductive approach in our work. We therefore administered a quantitative survey, the responses to which we could analyze in order to test hypotheses, and also conducted qualitative interviews with a number of the survey participants. The survey data were well suited to a deductive approach; we could analyze those data to test hypotheses that were generated based on theories of harassment. The interview data were well suited to an inductive approach; we looked for patterns across the interviews and then tried to make sense of those patterns by theorizing about them.
For one paper (Uggen & Blackstone, 2004)[6], we began with a prominent feminist theory of the sexual harassment of adult women and developed a set of hypotheses outlining how we expected the theory to apply in the case of younger women’s and men’s harassment experiences. We then tested our hypotheses by analyzing the survey data. In general, we found support for the theory that posited that the current gender system, in which heteronormative men wield the most power in the workplace, explained workplace sexual harassment—not just of adult women but of younger women and men as well. In a more recent paper (Blackstone, Houle, & Uggen, 2006),[7] we did not hypothesize about what we might find but instead inductively analyzed interview data, looking for patterns that might tell us something about how or whether workers’ perceptions of harassment change as they age and gain workplace experience. From this analysis, we determined that workers’ perceptions of harassment did indeed shift as they gained experience and that their later definitions of harassment were more stringent than those they held during adolescence. Overall, our desire to understand young workers’ harassment experiences fully—in terms of their objective workplace experiences, their perceptions of those experiences, and their stories of their experiences—led us to adopt both deductive and inductive approaches in the work. (Blackstone, n.d., p. 21)[8]
Researchers may not always set out to employ both approaches in their work but sometimes find that their use of one approach leads them to the other. One such example is described eloquently in Russell Schutt’s Investigating the Social World (2006).[9] As Schutt describes, researchers Sherman and Berk (1984)[10] conducted an experiment to test two competing theories of the effects of punishment on deterring deviance (in this case, domestic violence).Specifically, Sherman and Berk hypothesized that deterrence theory (see Williams, 2005[11] for more information on that theory) would provide a better explanation of the effects of arresting accused batterers than labeling theory. Deterrence theory predicts that arresting an accused spouse batterer will reduce future incidents of violence. Conversely, labeling theory predicts that arresting accused spouse batterers will increase future incidents (see Policastro & Payne, 2013[12] for more information on that theory). Figure 8.3 summarizes the two competing theories and the hypotheses Sherman and Berk set out to test.
What the original Sherman and Berk study, along with the follow-up studies, show us is that we might start with a deductive approach to research, but then, if confronted by new data we must make sense of, we may move to an inductive approach. We will expand on these possibilities in section 8.4 when we discuss mixed methods research.
Ethical and critical considerations
Deductive and inductive reasoning, just like other components of the research process comes with ethical and cultural considerations for researchers. Specifically, deductive research is limited by existing theory. Because scientific inquiry has been shaped by oppressive forces such as sexism, racism, and colonialism, what is considered theory is largely based in Western, white-male-dominant culture. Thus, researchers doing deductive research may artificially limit themselves to ideas that were derived from this context. Non-Western researchers, international social workers, and practitioners working with non-dominant groups may find deductive reasoning of limited help if theories do not adequately describe other cultures.
While these flaws in deductive research may make inductive reasoning seem more appealing, on closer inspection you’ll find similar issues apply. A researcher using inductive reasoning applies their intuition and lived experience when analyzing participant data. They will take note of particular themes, conceptualize their definition, and frame the project using their unique psychology. Since everyone’s internal world is shaped by their cultural and environmental context, inductive reasoning conducted by Western researchers may unintentionally reinforcing lines of inquiry that derive from cultural oppression.
Inductive reasoning is also shaped by those invited to provide the data to be analyzed. For example, I recently worked with a student who wanted to understand the impact of child welfare supervision on children born dependent on opiates and methamphetamine. Due to the potential harm that could come from interviewing families and children who are in foster care or under child welfare supervision, the researcher decided to use inductive reasoning and to only interview child welfare workers.
Talking to practitioners is a good idea for feasibility, as they are less vulnerable than clients. However, any theory that emerges out of these observations will be substantially limited, as it would be devoid of the perspectives of parents, children, and other community members who could provide a more comprehensive picture of the impact of child welfare involvement on children. Notice that each of these groups has less power than child welfare workers in the service relationship. Attending to which groups were used to inform the creation of a theory and the power of those groups is an important critical consideration for social work researchers.
As you can see, when researchers apply theory to research they must wrestle with the history and hierarchy around knowledge creation in that area. In deductive studies, the researcher is positioned as the expert, similar to the positivist paradigm presented in Chapter 5. We’ve discussed a few of the limitations on the knowledge of researchers in this subsection, but the position of the “researcher as expert” is inherently problematic. However, it should also not be taken to an extreme. A researcher who approaches inductive inquiry as a naïve learner is also inherently problematic. Just as competence in social work practice requires a baseline of knowledge prior to entering practice, so does competence in social work research. Because a truly naïve intellectual position is impossible—we all have preexisting ways we view the world and are not fully aware of how they may impact our thoughts—researchers should be well-read in the topic area of their research study but humble enough to know that there is always much more to learn.
Key Takeaways
- Inductive reasoning begins with a set of empirical observations, seeking patterns in those observations, and then theorizing about those patterns.
- Deductive reasoning begins with a theory, developing hypotheses from that theory, and then collecting and analyzing data to test the truth of those hypotheses.
- Inductive and deductive reasoning can be employed together for a more complete understanding of the research topic.
- Though researchers don’t always set out to use both inductive and deductive reasoning in their work, they sometimes find that new questions arise in the course of an investigation that can best be answered by employing both approaches.
Exercises
- Identify one theory and how it helps you understand your topic and working question.
I encourage you to find a specific theory from your topic area, rather than relying only on the broad theoretical perspectives like systems theory or the strengths perspective. Those broad theoretical perspectives are okay…but I promise that searching for theories about your topic will help you conceptualize and design your research project.
- Using the theory you identified, describe what you expect the answer to be to your working question.
8.2 Nomothetic causal explanations
Learning Objectives
Learners will be able to…
- Define and provide an example of idiographic causal relationships
- Describe the role of causality in quantitative research as compared to qualitative research
- Identify, define, and describe each of the main criteria for nomothetic causal relationships
- Describe the difference between and provide examples of independent, dependent, and control variables
- Define hypothesis, state a clear hypothesis, and discuss the respective roles of quantitative and qualitative research when it comes to hypotheses
Causality refers to the idea that one event, behavior, or belief will result in the occurrence of another, subsequent event, behavior, or belief. In other words, it is about cause and effect. It seems simple, but you may be surprised to learn there is more than one way to explain how one thing causes another. How can that be? How could there be many ways to understand causality?
Think back to our discussion in Section 5.3 on paradigms [insert chapter link plus link to section 1.2]. You’ll remember the positivist paradigm as the one that believes in objectivity. Positivists look for causal explanations that are universally true for everyone, everywhere because they seek objective truth. Interpretivists, on the other hand, look for causal explanations that are true for individuals or groups in a specific time and place because they seek subjective truths. Remember that for interpretivists, there is not one singular truth that is true for everyone, but many truths created and shared by others.
“Are you trying to generalize or nah?”
One of my favorite classroom moments occurred in the early days of my teaching career. Students were providing peer feedback on their working questions. I overheard one group who was helping someone rephrase their research question. A student asked, “Are you trying to generalize or nah?” Teaching is full of fun moments like that one. Answering that one question can help you understand how to conceptualize and design your research project.
Nomothetic causal explanations are incredibly powerful. They allow scientists to make predictions about what will happen in the future, with a certain margin of error. Moreover, they allow scientists to generalize—that is, make claims about a large population based on a smaller sample of people or items. Generalizing is important. We clearly do not have time to ask everyone their opinion on a topic or test a new intervention on every person. We need a type of causal explanation that helps us predict and estimate truth in all situations.
Generally, nomothetic causal relationships work best for explanatory research projects [INSERT SECTION LINK]. They also tend to use quantitative research: by boiling things down to numbers, one can use the universal language of mathematics to use statistics to explore those relationships. On the other hand, descriptive and exploratory projects often fit better with idiographic causality. These projects do not usually try to generalize, but instead investigate what is true for individuals, small groups, or communities at a specific point in time. You will learn about this type of causality in the next section. Here, we will assume you have an explanatory working question. For example, you may want to know about the risk and protective factors for a specific diagnosis or how a specific therapy impacts client outcomes.
What do nomothetic causal explanations look like?
Nomothetic causal explanations express relationships between variables. The term variable has a scientific definition. This one from Gillespie & Wagner (2018) “a logical grouping of attributes that can be observed and measured and is expected to vary from person to person in a population” (p. 9).[15] More practically, variables are the key concepts in your working question. You know, the things you plan to observe when you actually do your research project, conduct your surveys, complete your interviews, etc. These things have two key properties. First, they vary, as in they do not remain constant. “Age” varies by number. “Gender” varies by category. But they both vary. Second, they have attributes. So the variable “health professions” has attributes or categories, such as social worker, nurse, counselor, etc.
It’s also worth reviewing what is not a variable. Well, things that don’t change (or vary) aren’t variables. If you planned to do a study on how gender impacts earnings but your study only contained women, that concept would not vary. Instead, it would be a constant. Another common mistake I see in students’ explanatory questions is mistaking an attribute for a variable. “Men” is not a variable. “Gender” is a variable. “Virginia” is not a variable. The variable is the “state or territory” in which someone or something is physically located.
When one variable causes another, we have what researchers call independent and dependent variables. For example, in a study investigating the impact of spanking on aggressive behavior, spanking would be the independent variable and aggressive behavior would be the dependent variable. An independent variable is the cause, and a dependent variable is the effect. Why are they called that? Dependent variables depend on independent variables. If all of that gets confusing, just remember the graphical relationship in Figure 8.5.
Exercises
Write out your working question, as it exists now. As we said previously in the subsection, we assume you have an explanatory research question for learning this section.
- Write out a diagram similar to Figure 8.5.
- Put your independent variable on the left and the dependent variable on the right.
Check:
- Can your variables vary?
- Do they have different attributes or categories that vary from person to person?
- How does the theory you identified in section 8.1 help you understand this causal relationship?
If the theory you’ve identified isn’t much help to you or seems unrelated, it’s a good indication that you need to read more literature about the theories related to your topic.
For some students, your working question may not be specific enough to list an independent or dependent variable clearly. You may have “risk factors” in place of an independent variable, for example. Or “effects” as a dependent variable. If that applies to your research question, get specific for a minute even if you have to revise this later. Think about which specific risk factors or effects you are interested in. Consider a few options for your independent and dependent variable and create diagrams similar to Figure 8.5.
Finally, you are likely to revisit your working question so you may have to come back to this exercise to clarify the causal relationship you want to investigate.
For a ten-cent word like “nomothetic,” these causal relationships should look pretty basic to you. They should look like “x causes y.” Indeed, you may be looking at your causal explanation and thinking, “wow, there are so many other things I’m missing in here.” In fact, maybe my dependent variable sometimes causes changes in my independent variable! For example, a working question asking about poverty and education might ask how poverty makes it more difficult to graduate college or how high college debt impacts income inequality after graduation. Nomothetic causal relationships are slices of reality. They boil things down to two (or often more) key variables and assert a one-way causal explanation between them. This is by design, as they are trying to generalize across all people to all situations. The more complicated, circular, and often contradictory causal explanations are idiographic, which we will cover in the next section of this chapter.
Developing a hypothesis
A hypothesis is a statement describing a researcher’s expectation regarding what they anticipate finding. Hypotheses in quantitative research are a nomothetic causal relationship that the researcher expects to determine is true or false. A hypothesis is written to describe the expected relationship between the independent and dependent variables. In other words, write the answer to your working question using your variables. That’s your hypothesis! Make sure you haven’t introduced new variables into your hypothesis that are not in your research question. If you have, write out your hypothesis as in Figure 8.5.
A good hypothesis should be testable using social science research methods. That is, you can use a social science research project (like a survey or experiment) to test whether it is true or not. A good hypothesis is also specific about the relationship it explores. For example, a student project that hypothesizes, “families involved with child welfare agencies will benefit from Early Intervention programs,” is not specific about what benefits it plans to investigate. For this student, I advised her to take a look at the empirical literature and theory about Early Intervention and see what outcomes are associated with these programs. This way, she could more clearly state the dependent variable in her hypothesis, perhaps looking at reunification, attachment, or developmental milestone achievement in children and families under child welfare supervision.
Your hypothesis should be an informed prediction based on a theory or model of the social world. For example, you may hypothesize that treating mental health clients with warmth and positive regard is likely to help them achieve their therapeutic goals. That hypothesis would be based on the humanistic practice models of Carl Rogers. Using previous theories to generate hypotheses is an example of deductive research. If Rogers’ theory of unconditional positive regard is accurate, a study comparing clinicians who used it versus those who did not would show more favorable treatment outcomes for clients receiving unconditional positive regard.
Let’s consider a couple of examples. In research on sexual harassment (Uggen & Blackstone, 2004),[16] one might hypothesize, based on feminist theories of sexual harassment, that more females than males will experience specific sexually harassing behaviors. What is the causal relationship being predicted here? Which is the independent and which is the dependent variable? In this case, researchers hypothesized that a person’s sex (independent variable) would predict their likelihood to experience sexual harassment (dependent variable).
Sometimes researchers will hypothesize that a relationship will take a specific direction. As a result, an increase or decrease in one area might be said to cause an increase or decrease in another. For example, you might choose to study the relationship between age and support for legalization of marijuana. Perhaps you’ve taken a sociology class and, based on the theories you’ve read, you hypothesize that age is negatively related to support for marijuana legalization.[17] What have you just hypothesized?
You have hypothesized that as people get older, the likelihood of their supporting marijuana legalization decreases. Thus, as age (your independent variable) moves in one direction (up), support for marijuana legalization (your dependent variable) moves in another direction (down). So, a direct relationship (or positive correlation) involve two variables going in the same direction and an inverse relationship (or negative correlation) involve two variables going in opposite directions. If writing hypotheses feels tricky, it is sometimes helpful to draw them out and depict each of the two hypotheses we have just discussed.
It’s important to note that once a study starts, it is unethical to change your hypothesis to match the data you find. For example, what happens if you conduct a study to test the hypothesis from Figure 8.7 on support for marijuana legalization, but you find no relationship between age and support for legalization? It means that your hypothesis was incorrect, but that’s still valuable information. It would challenge what the existing literature says on your topic, demonstrating that more research needs to be done to figure out the factors that impact support for marijuana legalization. Don’t be embarrassed by negative results, and definitely don’t change your hypothesis to make it appear correct all along!
Criteria for establishing a nomothetic causal relationship
Let’s say you conduct your study and you find evidence that supports your hypothesis, as age increases, support for marijuana legalization decreases. Success! Causal explanation complete, right? Not quite.
You’ve only established one of the criteria for causality. The criteria for causality must include all of the following: covariation, plausibility, temporality, and nonspuriousness. In our example from Figure 8.7, we have established only one criteria—covariation. When variables covary, they vary together. Both age and support for marijuana legalization vary in our study. Our sample contains people of varying ages and varying levels of support for marijuana legalization. If, for example, we only included 16-year-olds in our study, age would be a constant, not a variable.
Just because there might be some correlation between two variables does not mean that a causal relationship between the two is really plausible. Plausibility means that in order to make the claim that one event, behavior, or belief causes another, the claim has to make sense. It makes sense that people from previous generations would have different attitudes towards marijuana than younger generations. People who grew up in the time of Reefer Madness or the hippies may hold different views than those raised in an era of legalized medicinal and recreational use of marijuana. Plausibility is of course helped by basing your causal explanation in existing theoretical and empirical findings.
Once we’ve established that there is a plausible relationship between the two variables, we also need to establish whether the cause occurred before the effect, the criterion of temporality. A person’s age is a quality that appears long before any opinions on drug policy, so temporally the cause comes before the effect. It wouldn’t make any sense to say that support for marijuana legalization makes a person’s age increase. Even if you could predict someone’s age based on their support for marijuana legalization, you couldn’t say someone’s age was caused by their support for legalization of marijuana.
Finally, scientists must establish nonspuriousness. A spurious relationship is one in which an association between two variables appears to be causal but can in fact be explained by some third variable. This third variable is often called a confound or confounding variable because it clouds and confuses the relationship between your independent and dependent variable, making it difficult to discern the true causal relationship is.
Continuing with our example, we could point to the fact that older adults are less likely to have used marijuana recreationally. Maybe it is actually recreational use of marijuana that leads people to be more open to legalization, not their age. In this case, our confounding variable would be recreational marijuana use. Perhaps the relationship between age and attitudes towards legalization is a spurious relationship that is accounted for by previous use. This is also referred to as the third variable problem, where a seemingly true causal relationship is actually caused by a third variable not in the hypothesis. In this example, the relationship between age and support for legalization could be more about having tried marijuana than the age of the person.
Quantitative researchers are sensitive to the effects of potentially spurious relationships. As a result, they will often measure these third variables in their study, so they can control for their effects in their statistical analysis. These are called control variables, and they refer to potentially confounding variables whose effects are controlled for mathematically in the data analysis process. Control variables can be a bit confusing, and we will discuss them more in Chapter 10, but think about it as an argument between you, the researcher, and a critic.
Researcher: “The older a person is, the less likely they are to support marijuana legalization.”
Critic: “Actually, it’s more about whether a person has used marijuana before. That is what truly determines whether someone supports marijuana legalization.”
Researcher: “Well, I measured previous marijuana use in my study and mathematically controlled for its effects in my analysis. Age explains most of the variation in attitudes towards marijuana legalization.”
Let’s consider a few additional, real-world examples of spuriousness. Did you know, for example, that high rates of ice cream sales have been shown to cause drowning? Of course, that’s not really true, but there is a positive relationship between the two. In this case, the third variable that causes both high ice cream sales and increased deaths by drowning is time of year, as the summer season sees increases in both (Babbie, 2010).[18]
Here’s another good one: it is true that as the salaries of Presbyterian ministers in Massachusetts rise, so too does the price of rum in Havana, Cuba. Well, duh, you might be saying to yourself. Everyone knows how much ministers in Massachusetts love their rum, right? Not so fast. Both salaries and rum prices have increased, true, but so has the price of just about everything else (Huff & Geis, 1993).[19]
Finally, research shows that the more firefighters present at a fire, the more damage is done at the scene. What this statement leaves out, of course, is that as the size of a fire increases so too does the amount of damage caused as does the number of firefighters called on to help (Frankfort-Nachmias & Leon-Guerrero, 2011).[20] In each of these examples, it is the presence of a confounding variable that explains the apparent relationship between the two original variables.
In sum, the following criteria must be met for a nomothetic causal relationship:
- The two variables must vary together.
- The relationship must be plausible.
- The cause must precede the effect in time.
- The relationship must be nonspurious (not due to a confounding variable).
The hypothetico-dedutive method
The primary way that researchers in the positivist paradigm use theories is sometimes called the hypothetico-deductive method (although this term is much more likely to be used by philosophers of science than by scientists themselves). Researchers choose an existing theory. Then, they make a prediction about some new phenomenon that should be observed if the theory is correct. Again, this prediction is called a hypothesis. The researchers then conduct an empirical study to test the hypothesis. Finally, they reevaluate the theory in light of the new results and revise it if necessary.
This process is usually conceptualized as a cycle because the researchers can then derive a new hypothesis from the revised theory, conduct a new empirical study to test the hypothesis, and so on. As Figure 8.8 shows, this approach meshes nicely with the process of conducting a research project—creating a more detailed model of “theoretically motivated” or “theory-driven” research. Together, they form a model of theoretically motivated research.
Keep in mind the hypothetico-deductive method is only one way of using social theory to inform social science research. It starts with describing one or more existing theories, deriving a hypothesis from one of those theories, testing your hypothesis in a new study, and finally reevaluating the theory based on the results data analyses. This format works well when there is an existing theory that addresses the research question—especially if the resulting hypothesis is surprising or conflicts with a hypothesis derived from a different theory.
But what if your research question is more interpretive? What if it is less about theory-testing and more about theory-building? This is what our next chapter covers: the process of inductively deriving theory from people’s stories and experiences. This process looks different than that depicted in Figure 8.8. It still starts with your research question and answering that question by conducting a research study. But instead of testing a hypothesis you created based on a theory, you will create a theory of your own that explain the data you collected. This format works well for qualitative research questions and for research questions that existing theories do not address.
Key Takeaways
- In positivist and quantitative studies, the goal is often to understand the more general causes of some phenomenon rather than the idiosyncrasies of one particular instance, as in an idiographic causal relationship.
- Nomothetic causal explanations focus on objectivity, prediction, and generalization.
- Criteria for nomothetic causal relationships require the relationship be plausible and nonspurious; and that the cause must precede the effect in time.
- In a nomothetic causal relationship, the independent variable causes changes in the dependent variable.
- Hypotheses are statements, drawn from theory, which describe a researcher’s expectation about a relationship between two or more variables.
Exercises
- Write out your working question and hypothesis.
- Defend your hypothesis in a short paragraph, using arguments based on the theory you identified in section 8.1.
- Review the criteria for a nomothetic causal relationship. Critique your short paragraph about your hypothesis using these criteria.
- Are there potentially confounding variables, issues with time order, or other problems you can identify in your reasoning?
8.3 Idiographic causal relationships
Learning Objectives
Learners will be able to…
- Define and provide an example of an idiographic causal explanation
- Differentiate between idiographic and nomothetic causal relationships
- Link idiographic and nomothetic causal relationships with the process of theory building and theory testing
- Describe how idiographic and nomothetic causal explanations can be complementary
We began the previous section with a definition of causality, or the idea that “one event, behavior, or belief will result in the occurrence of another, subsequent event, behavior, or belief.” Then, we described one kind of causality: a simple cause-and-effect relationship supported by existing theory and research on the topic, also known as a nomothetic causal relationship. But what if there is not a lot of literature on your topic? What if your question is more exploratory than explanatory? Then, you need a different kind of causal explanation, one that accounts for the complexity of human interactions.
How can we build causal relationships if we are just describing or exploring a topic? Recall the definitions of exploratory research, descriptive research[/pb_glossary], and [pb_glossary id="1030"]explanatory research from Chapter 2. Wouldn’t we need to do explanatory research to build any kind of causal explanation? Explanatory research attempts to establish nomothetic causal relationships: an independent variable is demonstrated to cause change in a dependent variable. Exploratory and descriptive qualitative research contains some causal relationships, but they are actually descriptions of the causal relationships established by the study participants.
What do idiographic causal explanations look like?
An idiographic causal relationship tries to identify the many, interrelated causes that account for the phenomenon the researcher is investigating. So, if idiographic causal explanations do not look like Figure 8.5, 8.6, or 8.7 what do they look like? Instead of saying “x causes y,” your participants will describe their experiences with “x,” which they will tell you was caused and influenced by a variety of other factors, as interpreted through their unique perspective, time, and environment. As we stated before, idiographic causal explanations are messy. Your job as a social science researcher is to accurately describe the patterns in what your participants tell you.
Let's think about this using an example. If I asked you why you decided to become a social worker, what might you say? For me, I would say that I wanted to be a mental health clinician since I was in high school. I was interested in how people thought, and I was privileged enough to have psychology courses at my local high school. I thought I wanted to be a psychologist, but at my second internship in my undergraduate program, my supervisors advised me to become a social worker because the license provided greater authority for independent practice and flexibility for career change. Once I found out social workers were like psychologists who also raised trouble about social justice, I was hooked.
That’s not a simple explanation at all! But it's definitely a causal explanation. It is my individual, subjective truth of a complex process. If we were to ask multiple social workers the same question, we might find out that many social workers begin their careers based on factors like personal experience with a disability or social injustice, positive experiences with social workers, or a desire to help others. No one factor is the “most important factor,” like with nomothetic causal relationships. Instead, a complex web of factors, contingent on context, emerge when you interpret what people tell you about their lives.
Understanding "why?"
In creating an idiographic explanation, you are still asking "why?" But the answer is going to be more complex. Those complexities are described in Table 8.1 as well as this short video comparing nomothetic and idiographic relationships.
Nomothetic causal relationships | Idiographic causal relationships | |
---|---|---|
Paradigm | Positivist | Interpretivist |
Purpose of research | Prediction & generalization | Understanding & particularity |
Reasoning | Deductive | Inductive |
Purpose of research | Explanatory | Exploratory or descriptive |
Research methods | Quantitative | Qualitative |
Causality | Simple: cause and effect | Complex: context-dependent, sometimes circular or contradictory |
Role of theory | Theory testing | Theory building |
Remember our question from the last section, “Are you trying to generalize or nah?” If you answered nah (or no, like a normal person), you are trying to establish an idiographic causal explanation. The purpose of that explanation isn't to predict the future or generalize to larger populations, but to describe the here-and-now as it is experienced by individuals within small groups and communities. Idiographic explanations are focused less on what is generally experienced by all people but more on the particularities of what specific individuals in a unique time and place experience.
Researchers seeking idiographic causal relationships are not trying to generalize or predict, so they have no need to reduce phenomena to mathematics. In fact, only examining things that can be counted can rob a causal relationship of its meaning and context. Instead, the goal of idiographic causal relationships is understanding, rather than prediction. Idiographic causal relationships are formed by interpreting people’s stories and experiences. Usually, these are expressed through words. Not all qualitative studies use word data, as some can use interpretations of visual or performance art. However, the vast majority of qualitative studies do use word data, like the transcripts from interviews and focus groups or documents like journal entries or meeting notes. Your participants are the experts on their lives—much like in social work practice—and as in practice, people's experiences are embedded in their cultural, historical, and environmental context.
Idiographic causal explanations are powerful because they can describe the complicated and interconnected nature of human life. Nomothetic causal explanations, by comparison, are simplistic. Think about if someone asked you why you wanted to be a social worker. Your story might include a couple of vignettes from your education and early employment. It might include personal experience with the social welfare system or family traditions. Maybe you decided on a whim to enroll in a social work course during your graduate program. The impact of each of these events on your career is unique to you.
Idiographic causal explanations are concerned with individual stories, their idiosyncrasies, and the patterns that emerge when you collect and analyze multiple people's stories. This is the inductive reasoning we discussed at the beginning of this chapter. Often, idiographic causal explanations begin by collecting a lot of qualitative data, whether though interviews, focus groups, or looking at available documents or cultural artifacts. Next, the researcher looks for patterns in the data and arrives at a tentative theory for how the key ideas in people's stories are causally related.
Unlike nomothetic causal relationships, there are no formal criteria (e.g., covariation) for establishing causality in idiographic causal relationships. In fact, some criteria like temporality and nonspuriousness may be violated. For example, if an adolescent client says, “It’s hard for me to tell whether my depression began before my drinking, but both got worse when I was expelled from my first high school,” they are recognizing that it may not so simple that one thing causes another. Sometimes, there is a reciprocal relationship where one variable (depression) impacts another (alcohol abuse), which then feeds back into the first variable (depression) and into other variables as well (school). Other criteria, such as covariation and plausibility, still make sense, as the relationships you highlight as part of your idiographic causal explanation should still be plausible and its elements should vary together.
Theory building and theory testing
As we learned in the previous section, nomothetic causal explanations are created by researchers applying deductive reasoning to their topic and creating hypotheses using social science theories. Much of what we think of as social science is based on this hypothetico-deductive method, but this leaves out the other half of the equation. Where do theories come from? Are they all just revisions of one another? How do any new ideas enter social science?
Through inductive reasoning and idiographic causal explanations!
Let's consider a social work example. If you plan to study domestic and sexual violence, you will likely encounter the Power and Control Wheel, also known as the Duluth Model (Figure 8.9). The wheel is a model designed to depict the process of domestic violence. The wheel was developed based on qualitative focus groups conducted by sexual and domestic violence advocates in Duluth, MN. This video explains more about the Duluth Model of domestic abuse.
The Power and Control Wheel is an example of what an idiographic causal relationship looks like. By contrast, look back at the previous section's Figure 8.5, 8.6, and 8.7 on nomothetic causal relationships between independent and dependent variables. See how much more complex idiographic causal explanations are?! They are complex, but not difficult to understand. At the center of domestic abuse is power and control, and while not every abuser would say that is what they were doing, that is the understanding of the survivors who informed this theoretical model. Their power and control is maintained through a variety of abusive tactics from social isolation to use of privilege to avoid consequences.
What about the role of hypotheses in idiographic causal explanations? In nomothetic causal explanations, researchers create hypotheses using existing theory and then test them for accuracy. Hypotheses in idiographic causality are much more tentative, and are probably best considered as "hunches" about what they think might be true. Importantly, they might indicate the researcher's prior knowledge and biases before the project begins, but the goal of idiographic research is to let your participants guide you rather than existing social work knowledge. Continuing with our Duluth Model example, advocates likely had some tentative hypotheses about what was important in a relationship with domestic violence. After all, they worked with this population for years prior to the creation of the model. However, it was the stories of the participants in these focus groups that led the Power and Control Wheel explanation for domestic abuse.
As qualitative inquiry unfolds, hypotheses and hunches are likely to emerge and shift as researchers learn from what their participants share. Because the participants are the experts in idiographic causal relationships, a researcher should be open to emerging topics and shift their research questions and hypotheses accordingly. This is in contrast to hypotheses in quantitative research, which remain constant throughout the study and are shown to be true or false.
Over time, as more qualitative studies are done and patterns emerge across different studies and locations, more sophisticated theories emerge that explain phenomena across multiple contexts. Once a theory is developed from qualitative studies, a quantitative researcher can seek to test that theory. For example, a quantitative researcher may hypothesize that men who hold traditional gender roles are more likely to engage in domestic violence. That would make sense based on the Power and Control Wheel model, as the category of “using male privilege” speaks to this relationship. In this way, qualitatively-derived theory can inspire a hypothesis for a quantitative research project, as we will explore in the next section.
Complementary approaches
If idiographic and nomothetic still seem like obscure philosophy terms, let’s consider another example. Imagine you are working for a community-based non-profit agency serving people with disabilities. You are putting together a report to lobby the state government for additional funding for community support programs. As part of that lobbying, you are likely to rely on both nomothetic and idiographic causal relationships.
If you looked at nomothetic causal relationships, you might learn how previous studies have shown that, in general, community-based programs like yours are linked with better health and employment outcomes for people with disabilities. Nomothetic causal explanations seek to establish that community-based programs are better for everyone with disabilities, including people in your community.
If you looked at idiographic causal explanations, you would use stories and experiences of people in community-based programs. These individual stories are full of detail about the lived experience of being in a community-based program. You might use one story from a client in your lobbying campaign, so policymakers can understand the lived experience of what it’s like to be a person with a disability in this program. For example, a client who said “I feel at home when I’m at this agency because they treat me like a family member,” or “this is the agency that helped me get my first paycheck,” can communicate richer, more complex causal relationships.
Neither kind of causal explanation is better than the other. A decision to seek idiographic causal explanations means that you will attempt to explain or describe your phenomenon exhaustively, attending to cultural context and subjective interpretations. A decision to seek nomothetic causal explanations, on the other hand, means that you will try to explain what is true for everyone and predict what will be true in the future. In short, idiographic explanations have greater depth, and nomothetic explanations have greater breadth.
Most importantly, social workers understand the value of both approaches to understanding the social world. A social worker helping a client with substance abuse issues seeks idiographic explanations when they ask about that client’s life story, investigate their unique physical environment, or probe how their family relationships. At the same time, a social worker also uses nomothetic explanations to guide their interventions. Nomothetic explanations may help guide them to minimize risk factors and maximize protective factors or use an evidence-based therapy, relying on knowledge about what in general helps people with substance abuse issues.
So, which approach speaks to you? Are you interested in learning about (a) a few people's experiences in a great deal of depth, or (b) a lot of people's experiences more superficially, while also hoping your findings can be generalized to a greater number of people? The answer to this question will drive your research question and project. These approaches provide different types of information and both types are valuable.
Key Takeaways
- Idiographic causal explanations focus on subjectivity, context, and meaning.
- Idiographic causal explanations are best suited to exploratory research questions and qualitative methods.
- Idiographic causal explanations are used to create new theories in social science.
Exercises
- Explore the literature on the theory you identified in section 8.1.
- Read about the origins of your theory. Who developed it and from what data?
- See if you can find a figure like Figure 8.9 in an article or book chapter that depicts the key concepts in your theory and how those concepts are related to one another causally. Write out a short statement on the causal relationships contained in the figure.
8.4 Mixed methods research
Learning Objectives
Learners will be able to...
- Define sequence and emphasis and describe how they work in qualitative research
- List five reasons why researchers use mixed methods
As we discussed in the previous sections, while we contrast idiographic vs. nomothetic causality or inductive vs. deductive reasoning, the truth is that researchers combine both of these approaches when they conduct research. While these processes can occur in any kind of study, mixed methods research is an excellent example of how researchers use both approaches to logic and reasoning to improve understanding of a given topic.
So far in this textbook, we have talked about quantitative and qualitative methods as an either/or choice—you can choose quantitative methods or qualitative methods. However, researchers often use both quantitative methods inside of their research projects. This is called mixed methods research.
For example, I recently completed a study with administrators of state-level services for people with intellectual and developmental disabilities. They implemented a program called self-direction, which allows people with disabilities greater self-determination over their supports. In this study, my research partners and I used a mixed methods approach to describe the implementation of self-direction across the United States. We distributed a short, electronic questionnaire and conducted phone interviews with program administrators. While we could have just sent out a questionnaire that asked states to provide basic information on their program (size, qualifications, services offered, etc.), that would not provide us much information about some of the issues administrators faced during program implementation. Similarly, we could have interviewed program administrators without the questionnaire, but then we wouldn’t know enough about the programs to ask good questions. Instead, we chose to use both qualitative and quantitative methods to provide a more comprehensive picture of program implementation.
Sequence and emphasis
There are many different mixed methods designs, each with their own strengths and limitations (see Creswell & Clarke, 2017[21] for a more thorough introduction). However, a more simplified synthesis of mixed methods approaches is provided by Engel and Schutt (2016)[22] using two key terms. Sequence refers to the order that each method is used. Researchers can use both methods at the same time or concurrently. Or, they can use one and then the other, or sequentially.
For our study of self-direction, we used a sequential design by sending out a questionnaire first, conducting some analysis, and then conducting the interview. We used the quantitative questionnaire to gather basic information about the programs before we began the interviews, so our questions were specific to the features of each program. If we wanted to use a concurrent design for some reason, we could have asked quantitative questions during the interview. However, we felt this would waste the administrators’ time looking up information and would break up rhythm of the interviews.
The other key term in mixed methods research is emphasis. In our mixed methods study, the qualitative data was the most important data. The quantitative data was mainly used to provide background information for the qualitative interviews, and our study write up focused mostly on the qualitative information. Thus, qualitative methods were prioritized in our study. Of course, many other studies emphasize quantitative methods over qualitative methods. In these studies, qualitative data is used mainly to provide context for the quantitative findings.
For example, demonstrating quantitatively that a particular therapy works is important. By adding a qualitative component, researchers could find out how the participants experienced the intervention, how clients understood the therapy's effects, and the meaning the therapy had on their lives. This data would add depth and context to the findings and allow researchers to improve the therapeutic technique in the future.
A similar practice is when researchers use qualitative methods to solicit feedback on a quantitative scale or measure. The experiences of individuals allow researchers to refine the measure before they conduct the quantitative component of their study. Finally, it is possible that researchers are equally interested in qualitative and quantitative information. In studies of equal emphasis, researchers consider both methods as the focus of the research project.
Why researchers use mixed methods
Mixed methods research is more than just sticking an open-ended question at the end of a quantitative survey. Mixed methods researchers use mixed methods for both pragmatic and synergistic reasons. That is, they use both methods because it makes sense with their research questions and because they will get the answers they want by combining the two approaches.
Mixed methods also allow you to use both inductive and deductive reasoning. As we’ve discussed, qualitative research follows inductive logic, moving from data to empirical generalizations or theory. In a mixed methods study, a researcher could use the results from a qualitative component to create a theory that could be tested in a subsequent quantitative component. The quantitative component would use deductive logic, using the theory derived from qualitative data to create and test a hypothesis. In this way, mixed methods use the strengths of both inductive and deductive reasoning. Quantitative allows the researcher to test existing ideas. Qualitative allows the researcher to create new ideas.
With these two concepts in mind, we can start to see why researchers use mixed methods in the real world. I mentioned previously that our research project used a sequential design because we wanted to use our quantitative data to shape what qualitative questions we asked our participants. Mixed methods are often used this way, to initiate ideas with one method to study with another. For example, researchers could begin a mixed methods project by using qualitative methods to interview or conduct a focus group with participants. Based on their responses, the researchers could then formulate a survey to give out to a larger group of people to see how common the themes from the focus groups were. This is the inverse of what we did in our project, which was to use a quantitative survey to inform a more detailed qualitative interview.
In addition to providing information for subsequent investigation, using both quantitative and qualitative information provides additional context for the data. For example, in our questionnaire for the study on self-direction, we asked participants to list what services people could purchase. The qualitative data followed up on that answer by asking whether the administrators had added or taken away any services, how they decided that these services would be covered and not others, and what problems that arose around providing these services. With that information, we could analyze what services were offered, why they were offered, and how administrators made those decisions. In this way, we learned the lived experience of program administrators, not just the basic information about their programs.
Finally, another purpose of mixed methods research is to corroborate data from both quantitative and qualitative sources. Ideally, your qualitative and quantitative results should support each other. For example, if interviews with participants showed a relationship between two concepts, that relationship should also be present in the qualitative data you collected. Differences between quantitative and qualitative data require an explanation. Perhaps there are outliers or extreme cases that pushed your data in one direction, for example.
In summary, these are a few of the many reasons researchers use mixed methods. They are summarized below:
- Triangulation, or convergence on the same phenomenon to improve validity
- Complementarity, which aims to get at related but different facets of a phenomenon
- Development, or the use of results from one phase or a study to develop another phase
- Initiation, or the intentional analysis of inconsistent qualitative and quantitative findings to derive new insights
- Expansion, or using multiple components to extend the scope of a study (Burnett, 2012, p. 77).[23]
A word of caution
The use of mixed methods has many advantages. However, undergraduate researchers should approach mixed methods with caution. Conducting a mixed methods study may mean doubling or even tripling your work. You must conceptualize a component using quantitative methods, another using qualitative methods, and think about how they fit together. This may mean creating a questionnaire, then writing an interview guide, and thinking through how the data on each measure relate to one another—more work than using one quantitative or qualitative method alone. Similarly, in sequential studies, the researcher must collect and analyze data from one component and then conceptualize and conduct the second component. This may also impact how long a project may take. Before beginning a mixed methods project, you should have a clear vision for what the project will entail and how each methodology will contribute to that vision. Always remember that you should make your project feasible enough for you to conduct with the time, money, and other resources you have at your disposal right now.
Key Takeaways
- Mixed methods studies vary in sequence and emphasis.
- Mixed methods allow the research to corroborate findings, provide context, follow up on ideas, and use the strengths of each method.
Exercises
- Look at the literature on your topic, and see if you can find a study that uses mixed methods.
- Describe the sequence and importance that the researchers places on the quantitative and qualitative components.
- Identify why the researchers used mixed methods and how the project would have been different had researchers used only one (qualitative or quantitative) component.
- Allen, K. R., Kaestle, C. E., & Goldberg, A. E. (2011). More than just a punctuation mark: How boys and young men learn about menstruation. Journal of Family Issues, 32, 129–156. ↵
- Ferguson, K. M., Kim, M. A., & McCoy, S. (2011). Enhancing empowerment and leadership among homeless youth in agency and community settings: A grounded theory approach. Child and Adolescent Social Work Journal, 28, 1–22 ↵
- King, R. D., Messner, S. F., & Baller, R. D. (2009). Contemporary hate crimes, law enforcement, and the legacy of racial violence. American Sociological Review, 74, 291–315. ↵
- Milkie, M. A., & Warner, C. H. (2011). Classroom learning environments and the mental health of first grade children. Journal of Health and Social Behavior, 52, 4–22. ↵
- The American Sociological Association wrote a press release on Milkie and Warner’s findings: American Sociological Association. (2011). Study: Negative classroom environment adversely affects children’s mental health. Retrieved from: https://www.sciencedaily.com/releases/2011/03/110309073717.htm ↵
- Uggen, C., & Blackstone, A. (2004). Sexual harassment as a gendered expression of power. American Sociological Review, 69, 64–92. ↵
- Blackstone, A., Houle, J., & Uggen, C. “At the time I thought it was great”: Age, experience, and workers’ perceptions of sexual harassment. Presented at the 2006 meetings of the American Sociological Association. ↵
- Blackstone, A. (2012). Inductive or deductive? Two different approaches. Principles of sociological inquiry: Qualitative and quantitative methods. Saylor Foundation. ↵
- Schutt, R. K. (2006). Investigating the social world: The process and practice of research. Thousand Oaks, CA: Pine Forge Press. ↵
- Sherman, L. W., & Berk, R. A. (1984). The specific deterrent effects of arrest for domestic assault. American Sociological Review, 49, 261–272. ↵
- Williams, K. R. (2005). Arrest and intimate partner violence: Toward a more complete application of deterrence theory. Aggression and Violent Behavior, 10(6), 660-679. ↵
- Policastro, C., & Payne, B. K. (2013). The blameworthy victim: Domestic violence myths and the criminalization of victimhood. Journal of Aggression, Maltreatment & Trauma, 22(4), 329-347. ↵
- Berk, R., Campbell, A., Klap, R., & Western, B. (1992). The deterrent effect of arrest in incidents of domestic violence: A Bayesian analysis of four field experiments. American Sociological Review, 57, 698–708; Pate, A., & Hamilton, E. (1992). Formal and informal deterrents to domestic violence: The Dade county spouse assault experiment. American Sociological Review, 57, 691–697; Sherman, L., & Smith, D. (1992). Crime, punishment, and stake in conformity: Legal and informal control of domestic violence. American Sociological Review, 57, 680–690. ↵
- Taylor, B. G., Davis, R. C., & Maxwell, C. D. (2001). The effects of a group batterer treatment program: A randomized experiment in Brooklyn. Justice Quarterly, 18(1), 171-201. ↵
- Wagner III, W. E., & Gillespie, B. J. (2018). Using and interpreting statistics in the social, behavioral, and health sciences. SAGE Publications. ↵
- Uggen, C., & Blackstone, A. (2004). Sexual harassment as a gendered expression of power. American Sociological Review, 69, 64–92. ↵
- In fact, there are empirical data that support this hypothesis. Gallup has conducted research on this very question since the 1960s. For more on their findings, see Carroll, J. (2005). Who supports marijuana legalization? Retrieved from http://www.gallup.com/poll/19561/who-supports-marijuana-legalization.aspx ↵
- Babbie, E. (2010). The practice of social research (12th ed.). Belmont, CA: Wadsworth. ↵
- Huff, D. & Geis, I. (1993). How to lie with statistics. New York, NY: W. W. Norton & Co. ↵
- Frankfort-Nachmias, C. & Leon-Guerrero, A. (2011). Social statistics for a diverse society. Washington, DC: Pine Forge Press. ↵
- Creswell, J. W., & Clark, V. L. P. (2017). Designing and conducting mixed methods research. Sage publications. ↵
- Engel, R. J. & Schutt, R. K. (2016). The practice of research in social work (4th ed.). Washington, DC: SAGE Publishing. ↵
- Burnett, D. (2012). Inscribing knowledge: Writing research in social work. In W. Green & B. L. Simon (Eds.), The Columbia guide to social work writing (pp. 65-82). New York, NY: Columbia University Press. ↵
When a participant's answer to a question is altered due to the way in which a question is written. In essence, the question leads the participant to answer in a specific way.
In a measure, when people say yes to whatever the researcher asks, even when doing so contradicts previous answers.
.toc__author {
display: none;
}
a.glossary-term {
font-weight: bold !important;
}
a statement describing a researcher’s expectation regarding what they anticipate finding
a sample that looks like the population from which it was selected in all respects that are potentially relevant to the study
claims about the world that appear scientific but are incompatible with the values and practices of science
whether a research result is applicable in another context or situation
describes an individual’s world view and the position they adopt about a research task and its social and political context
In nonequivalent comparison group designs, the process in which researchers match the population profile of the comparison and experimental groups.
The bias that occurs when those who respond to your request to participate in a study are different from those who do not respond to you request to participate in a study.
Chapter Outline
- Ethical responsibility and cultural respectfulness (8 minute read)
- Critical considerations (5 minute read)
- Informing your dissemination plan (11 minute read)
- Final product taking shape (10 minute read)
Content warning: Examples in this chapter contain references to research as a potential tool to stigmatize or oppress vulnerable groups, mistreatment and inequalities experienced by Native American tribes, sibling relationships, caregiving, child welfare, criminal justice and recidivism, first generation college students, Covid-19, school culture and race, health (in)equity, physical and sensory abilities, and transgender youth.
Your sweat and hard work has paid off! You've planned your study, collected your data, and completed your analysis. But alas, no rest for the weary student researcher. Now you need to share your findings. As researchers, we generally have some ideas where and with whom we desire to share our findings, but these plans may evolve and change during our research process. Communicating our findings with a broader audience is a critical step in the research process, so make sure not to treat this like an afterthought. Remember, research is about making a contribution to collective knowledge-building in the area of study that you are interested in. Indeed, research is of no value if there is no audience to receive it. You worked hard...get those findings out there!
In planning for this phase of research, we can consider a variety of methods for sharing our study findings. Among other options, we may choose to write our findings up as an article in a professional journal, provide a report to an organization, give testimony to a legislative group, or create a presentation for a community event. We will explore these options in a bit more detail below in section 21.4 where we talk more about different types of qualitative research products. We also want to think about our intended audience.
Exercises
For your research, answer these two key questions as you are planning for dissemination:
- Who are you targeting to communicate your findings to? In other words, who needs to hear the results of your study?
- What do you hope your audience will take away after learning about your study?
Of course, we can't know how our findings will be received, but we will want to work to anticipate what expectations our target audience might have and use this information to tailor our message in a way that is clear, thorough and consistent with our data (keep it honest!). We will further discuss different audiences you might want to reach with your research and a few considerations for each type of audience that may shape your dissemination plan. Before we tackle these sections, we are going to take some time to think through ethical and critical considerations that should be at the forefront of your mind as you plan for the dissemination of your research. Remember, your research is NOT a static, inanimate object. It is a reflection of your participants. Furthermore, others will review, interpret and potentially form opinions based on your research Your research has real implications for individuals, groups, communities, agencies, programs, etc.
21.1 Ethical responsibility and cultural respectfulness
Learning Objectives
Learners will be able to...
- Identify key ethical considerations in developing their qualitative research dissemination plan
- Conceptualize how research dissemination may impact diverse groups, both presently and into the future
Have you ever been misrepresented or portrayed in a negative light? It doesn't feel good. It especially doesn't feel good when the person portraying us has power, control and influence. While you might not feel powerful, research can be a powerful tool, and can be used and abused for many ends. Once research is out in the world, it is largely out of our control, so we need to approach dissemination with care. Be thoughtful about how you represent your work and take time to think through the potential implications it may have, both intended and unintended, for the people it represents.
As alluded to in the paragraph above, research comes with hefty responsibilities. You aren't off the hook if you are conducting quantitative research. While quantitative research deals with numbers, these numbers still represent people and their relationships to social problems. However, with qualitative research, we are often dealing with a smaller sample and trying to learn more from them. As such, our job often carries additional weight as we think about how we will represent our findings and the people they reflect. Furthermore, we probably hope that our research has an impact; that in some way, it leads to change around some issue. This is especially true as social work researchers. Our research often deals with oppressed groups, social problems, and inequality. However, it's hard to predict the implications that our research may have. This suggests that we need to be especially thoughtful about how we present our research to others.
Two of the core values of social work involve respecting the inherent dignity and worth of each person, practicing with integrity, and behaving in a trustworthy manner.[1] As social work researchers, to uphold these values, we need to consider how we are representing the people we are researching. Our work needs to honestly and accurately reflect our findings, but it also needs to be sensitive and respectful to the people it represents. In Chapter 6 we discussed research ethics and introduced the concept of beneficence or the idea that research needs to support the welfare of participants. Beneficence is particularly important as we think about our findings becoming public and how the public will receive, interpret and use this information. Thus, both as social workers and researchers, we need to be conscientious of how dissemination of our findings takes place.
Exercises
As you think about the people in your sample and the communities or groups to which they belong, consider some of these questions:
- How are participants being portrayed in my research?
- What characteristics or findings are being shared or highlighted in my research that may directly or indirectly be associated with participants?
- Have the groups that I am researching been stigmatized, stereotyped, and/or misrepresented in the past? If so, how does my research potentially reinforce or challenge these representations?
- How might my research be perceived or interpreted by members of the community or group it represents?
- In what ways does my research honor the dignity and worth of participants?
Qualitative research often has a voyeuristic quality to it, as we are seeking a window into participants' lives by exploring their experiences, beliefs, and values. As qualitative researchers, we have a role as stewards or caretakers of data. We need to be mindful of how data are gathered, maintained, and most germane to our conversation here, how data are used. We need to craft research products that honor and respect individual participants (micro), our collective sample as a whole (meso), and the communities that our research may represent (macro).
As we prepare to disseminate our findings, our ethical responsibilities as researchers also involve honoring the commitments we have made during the research process. We need to think back to our early phases of the research process, including our initial conversations with research partners and other stakeholders who helped us to coordinate our research activities. If we made any promises along the way about how the findings would be presented or used, we need to uphold them here. Additionally, we need to abide by what we committed to in our informed consent. Part of our informed consent involves letting participants know how findings may be used. We need to present our findings according to these commitments. We of course also have a commitment to represent our research honestly.
As an extension of our ethical responsibilities as researchers, we need to consider the impact that our findings may have, as well as our need to be socially conscientious researchers. As scouts, we were taught to leave our campsite in a better state than when we arrived. I think it is helpful to think of research in these terms. Think about the group(s) that may be represented by your research; what impact might your findings have for the lives of members of this group? Will it leave their lives in a better state than before you conducted your research? As a responsible researcher, you need to be thoughtful, aware and realistic about how your research findings might be interpreted and used by others. As social workers, while we hope that findings will be used to improve the lives of our clients, we can't ignore that findings can also be used to further oppress or stigmatize vulnerable groups; research is not apolitical and we should not be naive about this. It is worth mentioning the concept of sustainable research here. Sustainable research involves conducting research projects that have a long-term, sustainable impact for the social groups we work with. As researchers, this means that we need to actively plan for how our research will continue to benefit the communities we work with into the future. This can be supported by staying involved with these communities, routinely checking-in and seeking input from community members, and making sure to share our findings in ways that community members can access, understand, and utilize them. Nate Olson provides a very inspiring Ted Talk about the importance of building resilient communities. As you consider your research project, think about it in these terms.
Key Takeaways
- As you think about how best to share your qualitative findings, remember that these findings represent people. As such, we have a responsibility as social work researchers to ensure that our findings are presented in honest, respectful, and culturally sensitive ways.
- Since this phase of research deals with how we are going to share our findings with the public, we need to actively consider the potential implications of our research and how it may be interpreted and used.
Exercises
- Is your work, in some way, helping to contribute to a resilient and sustainable community? It may not be a big tangible project as described in Olson's Ted Talk, but is it providing a resource for change and growth to a group of people, either directly or indirectly? Does it promote sustainability amongst the social networks that might be impacted by the research you are conducting?
21.2 Critical considerations
Learning Objectives
Learners will be able to...
- Identify how issues of power and control are present in the dissemination of qualitative research findings
- Begin to examine and account for their own role in the qualitative research process, and address this in their findings
This is the part of our research that is shared with the public and because of this, issues like reciprocity, ownership, and transparency are relevant. We need to think about who will have access to the tangible products of our research and how that research will get used. As researchers, we likely benefit directly from research products; perhaps it helps us to advance our career, obtain a good grade, or secure funding. Our research participants often benefit indirectly by advancing knowledge about a topic that may be relevant or important to them, but often don't experience the same direct tangible benefits that we do. However, a participatory perspective challenges us to involve community members from the outset in discussions about what changes would be most meaningful to their communities and what research products would be most helpful in accomplishing those changes. This is especially important as it relates to the role of research as a tool to support empowerment.
Ownership of research products is also important as an issue of power and control. We will discuss a range of venues for presenting your qualitative research, some of which are more amenable to shared ownership than others. For instance, if you are publishing your findings in an academic journal, you will need to sign an agreement with that publisher about how the information in that article can be used and who has access to it. Similarly, if you are presenting findings at a national conference, travel and other conference-related expenses and requirements may make access to these research products prohibitive. In these instances, the researcher and the organization(s) they negotiate with (e.g. the publishing company, the conference organizing body) share control. However, disseminating qualitative findings in a public space, public record, or community-owned resource means that more equitable ownership might be negotiated. An equitable or reciprocal arrangement might not always be able to be reached, however. Transparency about who owns the products of research is important if you are working with community partners. To support this, establishing a Memorandum Of Understanding (MOU) or Memorandum of Agreement (MOA) early in the research process is important. This document should clearly articulate roles, responsibilities, and a number of other details, such as ownership of research products between the researcher and the partnering group(s).
Resources for learning more about MOUs and MOAs
Center for Community Health and Development, University of Kansas. (n.d.). Section 9. Understanding and writing contracts and memoranda of agreement.
Collaborative Center for Health Equity, University of Wisconson Madison. (n.d.). Standard agreement for research with community organizations.
Office of Research, UC Davis. (n.d.). Research MOUs.
Office of Research, The University of Texas at Dallas. (n.d.). Types of agreements.
In our discussion about qualitative research, we have also frequently identified the need for the qualitative researcher to account for their role throughout the research process. Part of this accounting can specifically apply to qualitative research products. This is our opportunity to demonstrate to our audience that we have been reflective throughout the course of the study and how this has influenced the work we did. Some qualitative research studies include a positionality statement within the final product. This is often toward the beginning of the report or the presentation and includes information about the researcher(s)'s identity and worldview, particularly details relevant to the topic being studied. This can include why you are invested in the study, what experiences have shaped how you have come to think about the topic, and any positions or assumptions you make with respect to the topic. This is another way to encourage transparency. It can also be a means of relegating or at least acknowledging some of our power in the research process, as it can provide one modest way for us, as the researcher, to be a bit more exposed or vulnerable, although this is a far cry from making the risks of research equitable between the researcher and the researched. However, the positionality statement can be a place to integrate our identities, who we are as an individual, a researcher, and a social work practitioner. Granted, for some of us that might be volumes, but we need to condense this down to a brief but informative statement—don't let it eclipse the research! It should just be enough to inform the audience and allow them to draw their own conclusions about who is telling the story of this research and how well they can be trusted. This student provides a helpful discussion of the positionality statement that she developed for her study. Reviewing your reflexive journal (discussed in Chapter 20 as a tool to enhance qualitative rigor) can help in identifying underlying assumptions and positions you might have grounded in your reactions throughout the research process. These insights can be integrated into your positionality statement. Please take a few minutes to watch this informative video of a student further explaining what a positionality statement is and providing a good example of one.
Key Takeaways
- The products of qualitative research often benefit the researcher disproportionately when compared to research participants or the communities they represent. Whenever possible, we can seek out ways to disseminate research in ways that addresses this imbalance and supports more tangible and direct benefits to community members.
- Openly positioning ourselves in our dissemination plans can be an important way for qualitative researchers to be transparent and account for our role.
21.3 Informing your dissemination plan
Learning Objectives
Learners will be able to...
- Appraise important dimensions of planning that will inform their research dissemination plan, including: audience, purpose, context and content
- Apply this appraisal to key decisions they will need to make when designing their qualitative research product(s)
This section will offer you a general overview of points to consider as you form the dissemination plan for your research. We will start with considerations regarding your audience, then turn our attention to the purpose of your research, and finally consider the importance of attending to both content and context as you plan for your final research product(s).
Audience
Perhaps the most important consideration you have as you plan how to present your work is your audience. Research is a product that is meant to be consumed, and because of this, we need to be conscious of our consumers. We will speak more extensively about knowing your audience in Chapter 24, devoted to both sharing and consuming research. Regardless of who your audience is (e.g. community members, classmates, research colleagues, practicing social workers, state legislator), there will be common elements that will be important to convey. While the way you present them will vary greatly according to who is listening, Table 21.1 offers a brief review of the elements that you will want your audience to leave with.
Element | Purpose |
Aim | What did my research aim to accomplish and why is it important |
Process | How did I go about conducting my research and what did I do to ensure quality |
Concepts | What are the main ideas that someone needs to know to make sense of my topic and how are they integrated into my research plan |
Findings | What were my results and under what circumstances are they valid |
Connection | How are my findings connected to what else we know about this topic and why are they important |
Once we determine who our audience is, we can further tailor our dissemination plan to that specific group. Of course, we may be presenting our findings in more than one venue, and in that case, we will have multiple plans that will meet the needs of each specific audience.
It's a good idea to pitch your plan first. However you plan to present your findings, you will want to have someone preview before you share with a wider audience. Ideally, whoever previews will be a person from your target audience or at least someone who knows them well. Getting feedback can go a long way in helping us with the clarity with which we convey our ideas and the impact they have on our audience. This might involve giving a practice speech, having someone review your article or report, or practice discussing your research one-on-one, as you would with a poster presentation. Let's talk about some specific audiences that you may be targeting and their unique needs or expectations.
Below I will go through some brief considerations for each of these different audiences. I have tried to focus this discussion on elements that are relevant specific to qualitative studies since we do revisit this topic in Chapter 24.
Research community
When presenting your findings to an academic audience or other research-related community, it is probably safe to a make a few assumptions. This audience is likely to have a general understanding of the research process and what it entails. For this reason, you will have to do less explaining of research-related terms and concepts. However, compared to other audiences, you will probably have to provide a bit more detail about what steps you took in your research process, especially as they relate to qualitative rigor, because this group will want to know about how your research was carried out and how you arrived at your decisions throughout the research process. Additionally, you will want to make a clear connection between which qualitative design you chose and your research question; a methodological justification. Researchers will also want to have a good idea about how your study fits within the wider body of scientific knowledge that it is related to and what future studies you feel are needed based on your findings. You are likely to encounter this audience if you are disseminating through a peer-reviewed journal article, presenting at a research conference, or giving an invited talk in an academic setting.
Professional community
We often find ourselves presenting our research to other professionals, such as social workers in the field. While this group may have some working knowledge of research, they are likely to be much more focused on how your research is related to the work they do and the clients they serve. While you will need to convey your design accurately, this audience is most likely to be invested in what you learned and what it means (especially for practice). You will want to set the stage for the discussion by doing a good job expressing your connection to and passion for the topic (a positionality statement might be particularly helpful here), what we know about the issue, and why it is important to their professional lives. You will want to give good contextual information for your qualitative findings so that practitioners can know if these findings might apply to people they work with. Also, as since social work practitioners generally place emphasis on person-centered practice, hearing the direct words of participants (quotes) whenever possible, is likely to be impactful as we present qualitative results. Where academics and researchers will want to know about implications for future research, professionals will want to know about implications for how this information could help transform services in the future or understand the clients they serve.
Lay community
The lay community are people who don't necessarily have specialized training or knowledge of the subject, but may be interested or invested for some other reason; perhaps the issue you are studying affects them or a loved one. Since this is the general public, you should expect to spend the most time explaining scientific knowledge and research processes and terminology in accessible terms. Furthermore, you will want to invest some time establishing a personal connection to the topic (like I talked about for the professional community). They will likely want to know why you are interested and why you are a credible source for this information. While this group may not be experts on research, as potential members of the group(s) that you may be researching, you do want to remember that they are experts in their own community. As such, you will want to be especially mindful of approaching how you present findings with a sense of cultural humility (although hopefully you have this in mind across all audiences). It will be good to discuss what steps you took to ensure that your findings accurately reflect what participants shared with you (rigor). You will want to be most clear with this group about what they should take away, without overstating your findings.
Regardless of who your audience is, remember that you are an ambassador. You may represent a topic, a population, an organization, or the whole institution of research, or any combination of these. Make sure to present your findings honestly, ethically, and clearly. Furthermore, I'm assuming that the research you are conducting is important because you have spent a lot of time and energy to arrive at your findings. Make sure that this importance comes through in your dissemination. Tell a compelling story with your research!
Exercises
Who needs to hear the message of your qualitative research?
What will they know in advance?
- Example. If you are presenting your research about caregiver fatigue to a caregiver support group, you won't need to spend time describing the role of caregivers because your audience will have lived experience.
What terms will they likely already be familiar with and which ones will require additional explanation?
- Example. If you are presenting your research findings to a group of academics, you wouldn't have to explain what a sampling frame is, but if you are sharing it with a group of community members from a local housing coalition, you will need to help them understand what this is (or maybe use a phrase that is more meaningful to them).
What will your audience be looking for?
- Example. If you are speaking to a group of child welfare workers about your study examining trauma-informed communication strategies, they are probably going to want to know how these strategies might impact the work that they do.
What is most impactful or valued by your audience?
- Example. If you are sharing your findings at a meeting with a council member, it may be especially meaningful to share direct quotes from constituents.
Purpose
Being clear about the purpose of your research from the outset is immeasurably helpful. What are you hoping to accomplish with your study? We can certainly look to the overarching purpose of qualitative research, that being to develop/expand/challenge/explore understanding of some topic. But, what are you specifically attempting to accomplish with your study? Two of the main reasons we conduct research are to raise awareness about a topic and to create change around some issue. Let's say you are conducting a study to better understand the experience of recidivism in the criminal justice system. This is an example of a study whose main purpose is to better understand and raise awareness around a particular social phenomenon (recidivism). On the other hand, you could also conduct a study that examines the use of strengths-based strategies by probation officers to reduce recidivism. This would fall into the category of research promoting a specific change (the use of strengths-based strategies among probation officers). I would wager that your research topic falls into one of these two very broad categories. If this is the case, how would you answer the corresponding questions below?
Exercises
Are you seeking to raise awareness of a particular issue with your research? If so,
- Whose awareness needs raising?
- What will "speak" most effectively to this group?
- How can you frame your research so that it has the most impact?
Are you seeking to create a specific change with your research? If so,
- What will that change look like?
- How can your research best support that change occurring?
- Who has the power to create that change and what will be most compelling in reaching them?
How you answer these questions will help to inform your dissemination plan. For instance, your dissemination plan will likely look very different if you are trying to persuade a group of legislators to pass a bill versus trying to share a new model or theory with academic colleagues. Considering your purposes will help you to convey the message of your research most effectively and efficiently. We invest a lot of ourselves in our research, so make sure to keep your sights focused on what you hope to accomplish with it!
Content and context
As a reminder, qualitative research often has a dual responsibility for conveying both content and context. You can think of content as the actual data that is shared with us or that we obtain, while context is the circumstances under which that data sharing occurs. Content conveys the message and context provides us the clues with which we can decode and make sense of that message.
While quantitative research may provide some contextual information, especially in regards to describing its sample, it rarely receives as much attention or detail as it does in qualitative studies. Because of this, you will want to plan for how you will attend to both the content and context of your study in planning for your dissemination.
Key Takeaways
- Research is an intentional act; you are trying to accomplish something with it. To be successful, you need to approach dissemination planfully.
- Planning the most effective way of sharing our qualitative findings requires looking beyond what is convenient or even conventional, and requires us to consider a number of factors, including our audience, the purpose or intent of our research and the nature of both the content and the context that we are trying to convey.
21.4 Final product taking shape
Learning Objectives
Learners will be able to...
- Evaluate the various means of disseminating research and consider their applicability for your research project
- Determine appropriate building blocks for designing your qualitative research product
As we have discussed, qualitative research takes many forms. It should then come as no surprise that qualitative research products also come in many different packages. To help guide you as the final products of your research take shape, we will discuss some of the building blocks or elements that you are likely to include as tools in sharing your qualitative findings. These are the elements that will allow you to flesh out the details of your dissemination plan.
Building blocks
There are many building blocks that are at our disposal as we formulate our qualitative research product(s). Quantitative researchers have charts, graphs, tables, and narrative descriptions of numerical output. These tools allow the quantitative researcher to tell the story of their research with numbers. As qualitative researchers, we are tasked with telling the story of our research findings as well, but our tools look different. While this isn't an exhaustive list of tools that are at our disposal as qualitative researchers, a number of commonly used elements in sharing qualitative findings are discussed here. Depending on your study design and the type of data you are working with, you may use one or some combination of the building blocks discussed below.
Themes
Themes are a very common element when presenting qualitative research findings. They may be called themes, but they may also go by other names: categories, dimensions, main ideas, etc. Themes offer the qualitative researcher a way to share ideas that emerged from your analysis that were shared by multiple participants or across multiple sources of data. They help us to distill the large amounts of qualitative data that we might be working with into more concise and manageable pieces of information that are more consumable for our audience. When integrating themes into your qualitative research product, you will want to offer your audience: the title of the theme (try to make this as specific/meaningful as possible), a brief description or definition of the theme, any accompanying dimensions or sub-themes that may be relevant, and examples (when appropriate).
Quotes
Quotes offer you the opportunity to share participants' exact words with your audience. Of course, we can't only rely on quotes, because we need to knit the information that is shared into one cohesive description of our findings and an endless list of quotes is unlikely to support this. Because of this, you will want to be judicious in selecting your quotes. Choose quotes that can stand on their own, best reflect the sentiment that is being captured by the theme or category of findings that you are discussing, and are likely to speak to and be understood by your audience. Quotes are a great way to help your findings come alive or to give them greater depth and significance. If you are using quotes, be sure to do so in a balanced manner—don't only use them in some sections but not others, or use a large number to support one theme and only one or two for another. Finally, we often provide some brief demographic information in a parenthetical reference following a quote so our reader knows a little bit about the person who shared the information. This helps to provide some context for the quote.
Kohli and Pizarro (2016)[2] provide a good example of a qualitative study using quotes to exemplify their themes. In their study, they gathered data through short-answer questionnaires and in-depth interviews from racial-justice oriented teachers of Color. Their study explored the experiences and motivations of these teachers and the environments in which they worked. As you might guess, the words of the teacher-participants were especially powerful and the quotes provided in the results section were very informative and important in helping to fulfill the aim of the research study. Take a few minutes to review this article. Note how the authors provide a good amount of detail as to what each of the themes meant and how they used the quotes to demonstrate and support each theme. The quotes help bring the themes to life and anchor the results in the actual words of the participants (suggesting greater trustworthiness in the findings).
Figure 21.1 below offers a more extensive example of a theme being reported along with supporting quotes from a study conducted by Karabanow, Gurman, and Naylor (2012).[3] This study focused on the role of work activities in the lives of "Guatemalan street youth". One of the important themes had to do with intersection of work and identity for this group. In this example, brief quotes are used within the body of the description of the theme, and also longer quotes (full sentence(s)) to demonstrate important aspects of the description.
Work-based Identity: Creativity and Self-WorthWork, be it formal or informal, is beneficial for street youth not only for its financial benefits; but it helps youth to develop and rebuild their sense of self, to break away from destructive patterns, and ultimately contributes to any goals of exiting street life. Although many of the participants were aware that society viewed them as “lazy” or “useless,” they tended to see themselves as contributing members of society earning a valid and honest living. One participant said, “Well, a lot of people say, right? ‘The kid doesn’t want to do anything. Lazy kid’ right? And I wouldn’t like for people to say that about me, I’d rather do something so that they don’t say that I’m lazy. I want to be someone important in life.” This youth makes an interesting and important connection in this statement: he intrinsically associates “being someone” with “doing something” – he accepts the work-based identity that characterizes much of contemporary capitalist society. Many of the interviews subtly enforced this idea that in the informal economy, as in the formal economy, “who one is” is largely dependent on “what one does.” This demonstrates two important ideas: that street youth working in the informal sector are surprisingly ‘mainstream’ in their underlying beliefs and ambitions, and that work – be it formal or informal – plays a crucial role in allowing street youth, who have often dealt with trauma, isolation and low self-esteem, to rebuild a sense of self-worth. Many of the youth involved in this study dream of futures that echo traditional ideals: “to have my family all together…to have a home, or rather to have a nice house…to have a good job.” Several explained that this future is unattainable without hard work; many viewed those who “do nothing” as people who “waste their time” and think that “your life isn’t important to you.” On the other hand, those who value their lives and wish to attain a future of “peace and tranquility” must “look for work, that’s what needs to be done to have that future because if God allows it, in the future maybe you can find a partner, form a family and live peacefully.” For these youth, working – be it in the formal or informal sector – is essential to a feeling of “moving forward (seguir adelante).” This movement forward begins with self-esteem. Although the focus of this study was not the troubled pasts of the participants, many alluded to the difficulties they have faced and the various traumas that forced them onto the streets. Several of the youth noted that working was a catalyst in rebuilding positive feelings about oneself: one explained, “[When I’m working,] I feel happy, powerful…Sometimes when I go out to sell, I feel happy.” Another said: For me, when I’m working I feel free because I know that I’m earning my money in an honest way, not stealing right. Because when you’re stealing, you don’t feel free, right? Now when you’re working, you’re free, they can’t arrest you or anything because you’re selling. Now if you’re stealing and everything, you don’t feel free. But when you’re selling you feel free, out of danger. This feeling of being “free” or “powerful” rests on the idea that money is “earned” and not stolen; being able to earn money is associated with being “someone,” with being a valid and contributing member of society. In addition, work helps street youth to break away from destructive patterns. One participant spoke of her experience working full time at a café: For me, working means to be busy, to not just be there….It helps us meet other people, like new people and not to be always in the same scene. Because if you’re not busy, you feel really bored and you might want to, I don’t know, go back to the same thing you were in before…you even forget your problems because you’re keeping busy, you’re talking to other people, people who don’t know you. For this participant, a formal job was beneficial in that it supplied her with a daily routine and allowed her to interact with non-street people – these factors helped to separate her from the destructive lifestyle of the street, and helped her to “move forward.” Although these benefits are indeed most obvious with formal employment, many participants spoke of the positive effects of informal work as well, although to varying degrees. In Guatemala, since the informal economy accounts for over half of the country’s GNP, there is a wide range of under-the-table informal work available. These jobs frequently bring youth out of the street context and, therefore, provide similar benefits to a formal job, as described by the above participant. As to informal work that takes place on the street, such as hawking or car watching, the benefits of work are present, although to a different degree. Even hawking, for example, gives young workers a routine and a chance to interact with non-street people. As one young man continuously emphasized throughout his interview, “work helps you to keep your mind busy, to be in another mind-set, right? To not be thinking the same thing all the time: ‘Oh, drugs, drugs, drugs…’” As explained earlier, the code of the hawking world dictates that vendors cannot sell while high – just like a formal job, hawking helps to distance youth workers from some of their destructive street habits. However, as one participant thoughtfully noted, it is difficult to break these habits when one is still highly embroiled in street culture; “it depended on who was around me because if they were in the same problems as I was, I stopped working and I started doing the same as they did. And if I was surrounded by serious people, then I got my act together.” While certain types of informal work, like cleaning or waitressing, can help youth to distance themselves from destructive patterns, others, such as car watching and selling, may not do enough to separate youth from their peers. While the routine and activity do have positive effects, they often are not sufficient. Among some of the participants, there was the sentiment that informal work could function as a transition stage towards exiting the street; it could “change your life.” One participant said “there are lots of vendors who’ve gotten off the streets, if you make an effort, you go out to sell, you can get off the street. Like myself, when I was selling, I mean working, I got off the street, I went home and I managed to stay there quite a long time.” One might credit this success to several factors: first, the money the seller may have been able to save and accumulate; second, the routine of selling may have helped the seller to break from destructive patterns, such as drug use, and also prepared the seller for the demands of formal sector employment; and, thirdly, selling may have enabled the seller to develop the necessary confidence and sense of self to attempt exiting the street. |
Pictures or videos
If our data collection involves the use of photographs, drawings, videos or other artistic expression of participants or collection of artifacts, we may very well include selections of these in our dissemination of qualitative findings. In fact, if we failed to include these, it would seem a bit inauthentic. For the same reason we include quotes as direct representations of participants' contributions, it is a good idea to provide direct reference to other visual forms of data that support or demonstrate our findings. We might incorporate narrative descriptions of these elements or quotes from participants that help to interpret their meaning. Integrating pictures and quotes is especially common if we are conducting a study using a Photovoice approach, as we discussed in Chapter 17, where a main goal of the research technique is to bring together participant generated visuals with collaborative interpretation.
Take some time to explore the website linked here. It is the webpage for The Philidelphia Collaborative for Health Equity's PhotoVoice Exhibit Gallery and offers a good demonstration of research that brings together pictures and text.
Graphic or figure
Qualitative researchers will often create a graphic or figure to visually reflect how the various pieces of your findings come together or relate to each other. Using a visual representation can be especially compelling for people who are visual learners. When you are using a visual representation, you will want to: label all elements clearly; include all the components or themes that are part of your findings; pay close attention to where you place and how you orient each element (as their spatial arrangement carries meaning); and finally, offer a brief but informative explanation that helps your reader to interpret your representation. A special subcategory of visual representation is process. These are especially helpful to lay out a sequential relationship within your findings or a model that has emerged out of your analysis. A process or model will show the 'flow' of ideas or knowledge in our findings, the logic of how one concept proceeds to the next and what each step of the model entails.
Noonan and colleagues (2004)[4] conducted a qualitative study that examined the career development of high achieving women with physical and sensory disabilities. Through the analysis of their interviews, they built a model of career development based on these women's experiences with a figure that helps to conceptually illustrate the model. They place the 'dyanmic self' in the center, surrounded by a dotted (permeable) line, with a number of influences outside the line (i.e. family influences, disability impact, career attitudes and behaviors, sociopoltical context, developmental opportunities and social support) and arrows directed inward and outward between each influence and the dynamic self to demonstrate mutual influence/exchange between them. The image is included in the results section of their study and brings together "core categories" and demonstrates how they work together in the emergent theory or how they relate to each other. Because so many of our findings are dynamic, like Noonan and colleagues, showing interaction and exchange between ideas, figures can be especially helpful in conveying this as we share our results.
Composites
Going one step further than the graphic or figure discussed above, qualitative researchers may decide to combine and synthesize findings into one integrated representation. In the case of the graphic or figure, the individual elements still maintain their distinctiveness, but are brought together to reflect how they are related. In a composite however, rather than just showing that they are related (static), the audience actually gets to 'see' the elements interacting (dynamic). The integrated and interactive findings of a composite can take many forms. It might be a written narrative, such as a fictionalized case study that reflects of highlights the many aspects that emerged during analysis. It could be a poem, dance, painting or any other performance or medium. Ultimately, a composite offers an audience a meaningful and comprehensive expression of our findings. If you are choosing to utilize a composite, there is an underlying assumption that is conveyed: you are suggesting that the findings of your study are best understood holistically. By discussing each finding individually, they lose some of their potency or significance, so a composite is required to bring them together. As an example of a composite, consider that you are conducting research with a number of First Nations Peoples in Canada. After consulting with a number of Elders and learning about the importance of oral traditions and the significance of storytelling, you collaboratively determine that the best way to disseminate your findings will be to create and share a story as a means of presenting your research findings. The use of composites also assumes that the 'truths' revealed in our data can take many forms. The Transgender Youth Project hosted by the Mandala Center for Change, is an example of legislative theatre combining research, artistic expression, and political advocacy and a good example of action-oriented research.
Counts
While you haven't heard much about numbers in our qualitative chapters, I'm going to break with tradition and speak briefly about them here. For many qualitative projects we do include some numeric information in our final product(s), mostly in the way of counts. Counts usually show up in the way of frequency of demographic characteristics of our sample or characteristics regarding our artifacts, if they aren't people. These may be included as a table or they may be integrated into the narrative we provide, but in either case, our goal in including this information is to offer the reader information so they can better understand who or what our sample is representing. The other time we sometimes include count information is in respect to the frequency and coverage of the themes or categories that are represented in our data. Frequency information about a theme can help the reader to know how often an idea came up in our analysis, while coverage can help them to know how widely dispersed this idea was (e.g. did nearly everyone mention this, or was it a small group of participants).
Key Takeaways
- There are a wide variety of means by which you can deliver your qualitative research to the public. Choose one that takes into account the various considerations that we have discussed above and also honors the ethical commitments that we outlined early in this chapter.
- Presenting qualitative research requires some amount of creativity. Utilize the building blocks discussed in this chapter to help you consider how to most authentically and effectively convey your message to a wider audience.
Exercises
- What means of delivery will you be choosing for your dissemination plan?
- What building blocks will best convey your qualitative results to your audience?
uses the scientific method to analyze and improve the scientific production of knowledge (e.g., systematic review, meta-analysis)
Chapter Outline
- How do social workers know what to do? (12 minute read)
- The scientific method (16 minute read)
- Evidence-based practice (11 minute read + 4 minute video)
- Creating a question to examine scientific evidence (9 minute video)
Content warning: Examples in this chapter contain references to school discipline, child abuse, food insecurity, homelessness, poverty and anti-poverty stigma, anti-vaccination pseudoscience, autism, trauma and PTSD, mental health stigma, susto and culture-bound syndromes, gender-based discrimination at work, homelessness, psychiatric hospitalizations, substance use, and mandatory treatment.
1.1 How do social workers know what to do?
Learning Objectives
Learners will be able to...
- Reflect on how we, as social workers, make decisions
- Differentiate between micro-, meso-, and macro-level analysis
- Describe the concept of intuition, its purpose in social work, and its limitations
- Identify specific errors in thinking and reasoning
What would you do?
Case 1: Imagine you are a clinical social worker at a children’s mental health agency. One day, you receive a referral from your town’s middle school about a client who often skips school, gets into fights, and is disruptive in class. The school has suspended him and met with the parents on multiple occasions, who say they practice strict discipline at home. Yet the client’s behavior has worsened. When you arrive at the school to meet with your client, who is also a gifted artist, you notice he seems to have bruises on his legs, has difficulty maintaining eye contact, and appears distracted. Despite this, he spends the hour painting and drawing, during which time you are able to observe him.
- Given your observations of your client's strengths and challenges, what intervention would you select, and how could you determine its effectiveness?
Case 2: Imagine you are a social worker working in the midst of an urban food desert (a geographic area in which there is no grocery store that sells fresh food). As a result, many of your low-income clients either eat takeout, or rely on food from the dollar store or a convenience store. You are becoming concerned about your clients’ health, as many of them are obese and say they are unable to buy fresh food. Your clients tell you that they have to rely on food pantries because convenience stores are expensive and often don't have the right kinds of food for their families. You have spent the past month building a coalition of community members to lobby your city council. The coalition includes individuals from non-profit agencies, religious groups, and healthcare workers.
- How should this group address the impact of food deserts in your community? What intervention(s) do you suggest? How would you determine whether your intervention was effective?
Case 3: You are a social worker working at a public policy center whose work focuses on the issue of homelessness. Your city is seeking a large federal grant to address this growing problem and has hired you as a consultant to work on the grant proposal. After interviewing individuals who are homeless and conducting a needs assessment in collaboration with local social service agencies, you meet with city council members to talk about potential opportunities for intervention. Local agencies want to spend the money to increase the capacity of existing shelters in the community. In addition, they want to create a transitional housing program at an unused apartment complex where people can reside upon leaving the shelter, and where they can gain independent living skills. On the other hand, homeless individuals you interview indicate that they would prefer to receive housing vouchers to rent an apartment in the community. They also fear the agencies running the shelter and transitional housing program would impose restrictions and unnecessary rules and regulations, thereby curbing their ability to freely live their lives. When you ask the agencies about these client concerns, they state that these clients need the structure and supervision provided by agency support workers.
- Which kind of program should your city choose to implement? Which is most likely to be effective and why?
Assuming you’ve taken a social work course before, you will notice that these case studies cover different levels of analysis in the social ecosystem—micro, meso, and macro. At the micro-level, social workers examine the smallest levels of interaction; in some cases, just “the self” alone (e.g. the child in case one).
When social workers investigate groups and communities, such as our food desert in case 2, their inquiry is at the meso-level.
At the macro-level, social workers examine social structures and institutions. Research at the macro-level examines large-scale patterns, including culture and government policy.
These three domains interact with one another, and it is common for a research project to address more than one level of analysis. For example, you may have a study about individuals at a case management agency (a micro-level study) that impacts the organization as a whole (meso-level) and incorporates policies and cultural issues (macro-level). Moreover, research that occurs on one level is likely to have multiple implications across domains.
How do social workers know what to do?
Welcome to social work research. This chapter begins with three problems that social workers might face in practice, and three questions about what a social worker should do next. If you haven’t already, spend a minute or two thinking about the three aforementioned cases and jot down some notes. How might you respond to each of these cases?
We assume it is unlikely you are an expert in the areas of children’s mental health, community responses to food deserts, and homelessness policy. Don’t worry, we're not either. In fact, for many of you, this textbook will likely come at an early point in your graduate social work education, so it may seem unfair for us to ask you what the 'right' answers are. And to disappoint you further, this course will not teach you the 'right' answer to these questions. It will, however, teach you how to answer these questions for yourself, and to find the 'right' answer that works best in each unique situation.
Assuming you are not an experienced practitioner in the areas described above, you likely used intuition (Cheung, 2016).[6] when thinking about what you would do in each of these scenarios. Intuition is a "gut feeling" about what to think about and do, often based on personal experience. What we experience influences how we perceive the world. For example, if you've witnessed representations of trauma in your practice, personal life, or in movies or television, you may have perceived that the child in case one was being physically abused and that his behavior was a sign of trauma. As you think about problems such as those described above, you find that certain details stay with you and influence your thinking to a greater degree than others. Using past experiences, you apply seemingly relevant knowledge and make predictions about what might be true.
Over a social worker's career, intuition evolves into practice wisdom. Practice wisdom is the “learning by doing” that develops as a result of practice experience. For example, a clinical social worker may have a "feel" for why certain clients would be a good fit to join a particular therapy group. This idea may be informed by direct experience with similar situations, reflections on previous experiences, and any consultation they receive from colleagues and/or supervisors. This "feel" that social workers get for their practice is a useful and valid source of knowledge and decision-making—do not discount it.
On the other hand, intuitive thinking can be prone to a number of errors. We are all limited in terms of what we know and experience. One's economic, social, and cultural background will shape intuition, and acting on your intuition may not work in a different sociocultural context. Because you cannot learn everything there is to know before you start your career as a social worker, it is important to learn how to understand and use social science to help you make sense of the world and to help you make sound, reasoned, and well-thought out decisions.
Social workers must learn how to take their intuition and deepen or challenge it by engaging with scientific literature. Similarly, social work researchers engage in research to make certain their interventions are effective and efficient (see section 1.4 for more information). Both of these processes—consuming and producing research—inform the social justice mission of social work. That's why the Council on Social Work Education (CSWE), who accredits the MSW program you are in, requires that you engage in social science.
Competency 4: Engage In Practice-informed Research and Research-informed Practice Social workers understand quantitative and qualitative research methods and their respective roles in advancing a science of social work and in evaluating their practice. Social workers know the principles of logic, scientific inquiry, and culturally informed and ethical approaches to building knowledge. Social workers understand that evidence that informs practice derives from multi-disciplinary sources and multiple ways of knowing. They also understand the processes for translating research findings into effective practice. Social workers: • use practice experience and theory to inform scientific inquiry and research; • apply critical thinking to engage in analysis of quantitative and qualitative research methods and research findings; and • use and translate research evidence to inform and improve practice, policy, and service delivery (CSWE, 2015).[7]
Errors in thinking
We all rely on mental shortcuts to help us figure out what to do in a practice situation. All people, including you and me, must train our minds to be aware of predictable flaws in thinking, termed cognitive biases. Here is a link to the Wikipedia entry on cognitive biases, as well as an interactive list. As you can see, there are many types of biases that can result in irrational conclusions.
The most important error in thinking for social scientists to be aware of is confirmation bias. Confirmation bias involves observing and analyzing information in a way that confirms what you already believe to be true. We all arrive at each moment with a set of personal beliefs, experiences, and worldviews that have been developed and ingrained over time. These patterns of thought inform our intuitions, primarily in an unconscious manner. Confirmation bias occurs when our mind ignores or manipulates information to avoid challenging what we already believe to be true.
In our second case study, we are trying to figure out how to help people who receive SNAP (sometimes referred to as Food Stamps) who live in a food desert. Let’s say we have arrived at a policy solution and are now lobbying the city council to implement it. There are many who have negative beliefs about people who are “on welfare.” These people may believe individuals who receive social welfare benefits spend their money irresponsibly, are too lazy to get a job, and manipulate the system to maintain or increase their government payout.
Those espousing this belief may point to an example such as Louis Cuff, who bought steak and lobster with his SNAP benefits and resold them for a profit. However, they are falling prey to assuming that one person's bad behavior reflects upon an entire group of people. City council members who hold these beliefs may ignore the truth about the client population—that people experiencing poverty usually spend their money responsibly and that they genuinely need help accessing fresh and healthy food. In this way, confirmation bias often makes people less capable of empathizing with one another because they have difficulty accepting alternative perspectives.
Errors in reasoning
Because the human mind is prone to errors, when anyone makes a statement about what is true or what should be done in a given situation, errors in logic may emerge. Think back to the case studies at the beginning of this section. You most likely had some ideas about what to do in each case, but where did those ideas come from. Below are some of the most common logical fallacies and the ways in which they may negatively influence a social worker. Consider how some of these might apply to your thinking about the practice situations in this chapter.
- Making hasty generalization: when a person draws conclusions before having enough information. A social worker may apply lessons from a handful of clients to an entire population of people (see Louis Cuff, above). It is important to examine the scientific literature in order to avoid this.
- Confusing correlation with causation: when one concludes that because two things are correlated (as one changes, the other changes), they must be causally related. As an example, a social worker might observe both an increase in the minimum wage and higher unemployment in certain areas of the city. However, just because two things changed at the same time does not mean they are causally related. Social workers should explore other factors that might impact causality.
- Going down a slippery slope: when a person concludes that we should not do something because something far worse will happen if we do so. For example, a social worker may seek to increase a client's opportunity to choose their own activities, but face opposition from those who believe it will lead to clients making unreasonable demands. Clearly, this is nonsense. Changes that foster self-determination are unlikely to result in client revolt. Social workers should be skeptical of arguments opposing small changes because one argues that radical changes are inevitable.
- Appealing to authority: when a person draws a conclusion by appealing to the authority of an expert or reputable individual, rather than through the strength of the claim. You have likely encountered individuals who believe they are correct because another in a position of authority told them so. Instead, we should work to build a reflective and critical approach to practice that questions authority.
- Hopping on the bandwagon: when a person draws a conclusion consistent with popular belief. Just because something is popular does not mean it is correct. Fashionable ideas come and go. Social workers should engage with trendy ideas but must ground their work in scientific evidence rather than popular opinion.
- Using a straw man: when a person does not represent their opponent's position fairly or with sufficient depth. For example, a social worker advocating for a new group home may depict homeowners that are opposed to clients living in their neighborhood as individuals concerned only with their property values. However, this may not be the case. Social workers should instead engage deeply with all sides of an issue and represent them accurately.
Key Takeaways
- Social work research occurs at the micro-, meso-, and macro-level.
- Intuition is a powerful, though limited, source of information when making decisions.
- All human thought is subject to errors in thinking and reasoning.
- Scientific inquiry accounts for cognitive biases by applying an organized, logical way of observing and theorizing about the world.
Exercises
- Think about a social work topic you might want to study this semester as part of a research project. How do individuals commit specific errors in logic or reasoning when discussing a specific topic (e.g. Louis Cuff)? How can using scientific evidence help you combat popular myths about your topic that are based on erroneous thinking?
- Reflect on the strengths and limitations of your personal experiences as a way to guide your work with diverse populations. Describe an instance when your intuition may have resulted in biased or misguided thinking or behavior in a social work practice situation.
1.2 The scientific method
Learning Objectives
Learners will be able to...
- Define science and social science
- Describe the differences between objective truth and subjective truths
- Identify how qualitative and quantitative methods differ and how they can be used together
- Delineate the features of science that distinguish it from pseudoscience
If we asked you to draw a picture of science, what would you draw? Our guess is it would be something from a chemistry or biology classroom, like a microscope or a beaker. Maybe something from a science fiction movie. All social workers use scientific thinking in their practice. However, social workers have a unique understanding of what science means, one that is (not surprisingly) more open to the unexpected and human side of the social world.
Science and not-science
In social work, science is a way of 'knowing' that attempts to systematically collect and categorize facts or truths. A key word here is systematically—conducting science is a deliberate process. Scientists gather information about facts in a way that is organized and intentional, and usually follows a set of predetermined steps. Social work is not a science, but social work is informed by social science—the science of humanity, social interactions, and social structures. In other words, social work research uses organized and intentional procedures to uncover facts or truths about the social world. And social workers rely on social scientific research to promote change.
Science can also be thought of in terms of its impostor, pseudoscience. Pseudoscience refers to beliefs about the social world that are unsupported by scientific evidence. These claims are often presented as though they are based on science. But once researchers test them scientifically, they are demonstrated to be false. A scientifically uninformed social work practitioner using pseudoscience may recommend any number of ineffective, misguided, or harmful interventions. Pseudoscience often relies on information and scholarship that has not been reviewed by experts (i.e., peer review) or offers a selective and biased reading of reviewed literature.
An example of pseudoscience comes from anti-vaccination activists. Despite overwhelming scientific consensus that vaccines do not cause autism, a very vocal minority of people continue to believe that they do. Anti-vaccination advocates present their information as based in science, as seen here at Green Med Info. The author of this website shares real abstracts from scientific journal articles and studies but will only provide information on articles that show the potential dangers of vaccines, without showing any research that prevents the positive and safe side of vaccines. Green Med Info is an example of confirmation bias, as all data presented on the website supports what the pseudo-scientific researcher believes to be true. For more information on assessing causal relationships, consult Chapter 8, where we discuss causality in detail.
The values and practices associated with the scientific method work to overcome common errors in thinking (such as confirmation bias). First, the scientific method uses established techniques from the literature to determine the likelihood of something being true or false. The research process often cites these techniques, reasons for their use, and how researchers came to the decision to use said techniques. However, each technique comes with its own strengths and limitations. Rigorous science is about making the best choice, being open about your process, and allowing others to check your work. It is important to remember that there is no "perfect" study—all research has limitations because all scientific methods come with limitations.
Skepticism and debate
Unfortunately, the "perfect" researcher does not exist. Scientists are human, so they are subject to error and bias, such as gravitating toward fashionable ideas and thinking their work is more important than others' work. Theories and concepts fade in and out of use and may be tossed aside when new evidence challenges their truth. Part of the challenge in your research projects will be finding what you believe about an issue, rather than summarizing what others think about the topic. Good science, just like good social work practice, is authentic. When we see students present their research projects, those that are the strongest deliver both passionate and informed arguments about their topic area.
Good science is also open to ongoing questioning. Scientists are fundamentally skeptical. As such, they are likely to pursue alternative explanations. They might question the design of a study or replicate it to see if it works in another context. Scientists debate what is true until they arrive at a majority consensus. If you've ever heard that 97% of climate scientists agree that global warming is due to human activity[8] or that 99% of economists agree that tariffs make the economy worse,[9] you are seeing this sociology of science in action. This skepticism will help to catch situations in which scientists who make the oh-so-human mistakes in thinking and reasoning reviewed in Section 1.1.
Skepticism also helps to identify unethical scientists, as with Andrew Wakefield's study linking the MMR vaccination and autism. When other researchers looked at his data, they found that he had altered the data to match his own conclusions and sought to benefit financially from the ensuing panic about vaccination (Godlee, Smith, & Marcovitch, 2011).[10] This highlights another key value in science: openness.
Openness
Through the use of publications and presentations, scientists share the methods used to gather and analyze data. The trend towards open science has also prompted researchers to share data as well. This in turn enables other researchers to re-run, replicate, and validate analyses and results. A major barrier to openness in science is the paywall. When you've searched online for a journal article (we will review search techniques in Chapter 3), you have likely run into the $25-$50 price tag. Don't despair—your university should subscribe to these journals. However, the push towards openness in science means that more researchers are sharing their work in open access journals, which are free for people to access (like this textbook!). These open access journals do not require a university subscription to view.
Openness also means engaging the broader public about your study. Social work researchers conduct studies to help people, and part of scientific work is making sure your study has an impact. For example, it is likely that many of the authors publishing in scientific journals are on Twitter or other social media platforms, relaying the importance of study findings. They may create content for popular media, including newspapers, websites, blogs, or podcasts. It may lead to training for agency workers or public administrators. Regrettably, academic researchers have a reputation for being aloof and disengaged from the public conversation. However, this reputation is slowly changing with the trend towards public scholarship and engagement. For example, see this recent section of the Journal of the Society of Social Work and Research on public impact scholarship.
Science supported by empirical data
Pseudoscience is often doctored up to look like science, but the surety with which its advocates speak is not backed up by empirical data. Empirical data refers to information about the social world gathered and analyzed through scientific observation or experimentation. Theory is also an important part of science, as we will discuss in Chapter 7. However, theories must be supported by empirical data—evidence that what we think is true really exists in the world.
There are two types of empirical data that social workers should become familiar with. Quantitative data refers to numbers and qualitative data usually refers to word data (like a transcript of an interview) but can also refer to pictures, performances, and other means of expressing oneself. Researchers use specific methods designed to analyze each type of data. Together, these are known as research methods, or the methods researchers use to examine empirical data.
Objective truth
In our vaccine example, scientists have conducted many studies tracking children who were vaccinated to look for future diagnoses of autism (see Taylor et al. 2014 for a review). This is an example of using quantitative data to determine whether there is a causal relationship between vaccination and autism. By examining the number of people who develop autism after vaccinations and controlling for all of the other possible causes, researchers can determine the likelihood of whether vaccinations cause changes in the brain that are eventually diagnosed as autism.
In this case, the use of quantitative data is a good fit for disproving myths about the dangers of vaccination. When researchers analyze quantitative data, they are trying to establish an objective truth. An objective truth is always true, regardless of context. Generally speaking, researchers seeking to establish objective truth tend to use quantitative data because they believe numbers don't lie. If repeated statistical analyses don't show a relationship between two variables, like vaccines and autism, that relationship almost certainly does not exist. By boiling everything down to numbers, we can minimize the biases and logical errors that human researchers bring to the scientific process. That said, the interpretation of those numbers is always up for debate.
This approach to finding truth probably sounds similar to something you heard in your middle school science classes. When you learned about gravitational force or the mitochondria of a cell, you were learning about the theories and observations that make up our understanding of the physical world. We assume that gravity is real and that the mitochondria of a cell are real. Mitochondria are easy to spot with a powerful microscope, and we can observe and theorize about their function in a cell. The gravitational force is invisible but clearly apparent from observable facts, such as watching an apple fall. If we were unable to perceive mitochondria or gravity, they would still be there, doing their thing, because they exist independent of our observation of them.
Let’s consider a social work example. Scientific research has established that children who are subjected to severely traumatic experiences are more likely to be diagnosed with a mental health disorder (e.g., Mahoney, Karatzias, & Hutton, 2019).[11] A diagnosis of post-traumatic stress disorder (PTSD) is considered objective, and may refer to a mental health issue that exists independent of the individual observing it and is highly similar in its presentation across clients. The Diagnostic and Statistical Manual of Mental Disorders (DSM-5, 2017)[12] identifies a group of criteria which is based on unbiased, neutral client observations. These criteria are based in research, and render an objective diagnosis more likely to be valid and reliable. Through the clinician's observations and the client’s description of their symptoms, an objective determination of a mental health diagnosis can be made.
Subjective truth(s)
For those of you are skeptical, you may ask yourself: does a diagnosis tell a client's whole story? No. It does not tell you what the client thinks and feels about their diagnosis, for example. Receiving a diagnosis of PTSD may be a relief for a client. The diagnosis may suggest the words to describe their experiences. In addition, this diagnosis may provide a direction for therapeutic work, as there are evidence-based interventions clinicians can use with each diagnosis. On the other hand, a client may feel shame and view the diagnosis as a label, defining them in a negative way and limiting their potential (Barsky, 2015).[13]
Imagine if we surveyed people with PTSD to see how they interpreted their diagnosis. Objectively, we could determine whether more people said the diagnosis was, overall, a positive or negative event for them. However, it is unlikely that the experience of receiving a diagnosis was either completely positive or completely negative. In social work, we know that a client's thoughts and emotions are rarely binary, either/or situations. Clients likely feel a mix of positive and negative thoughts and emotions during the diagnostic process. How they incorporate a diagnosis into their life story is unique. These messy bits are subjective truths, or the thoughts and feelings that arise as people interpret and make meaning of situations. Importantly, looking for subjective truths can help us see the contradictory and multi-faceted nature of people's thoughts, and qualitative data allows us to avoid oversimplifying them into negative and positive feelings that could be counted, as in quantitative data. It is the role of a researcher, just like a practitioner, to seek to understand things from the perspective of the client. Unlike with objective truth, this will not lead to a general sense of what is true for everyone, but rather what is true in a particular time and place.
Subjective truths are best expressed through qualitative data, like conversations with a client or looking at their social media posts or journal entries. As a researcher, we might invite a client to tell us how they felt after they were first diagnosed, after they spoke with family, and over the course of the therapeutic process. While it may look different from what we normally think of as science (e.g. pharmaceutical studies), these stories are indeed a rich source of data for scientific analysis. However, it is impossible to analyze what this client said without also considering the sociocultural context in which they live. For example, the concept of PTSD is generated from Western thought and philosophy. How might people from other cultures understand trauma differently?
In the DSM-5 classification of mental health disorders, there is a list of culture-bound syndromes which appear only in certain cultures. For example, susto describes a unique cluster of symptoms experienced by Latin Americans after a traumatic event (Nogueira, Mari, & Razzouk, 2015).[14] Susto involves more physical symptoms than a traditional PTSD diagnosis. Indeed, many of these syndromes do not fit within a Western conceptualization of mental health because they differentiate less between the mind and body. To a Western scientist, susto may seem less real than PTSD. To someone from Latin America, their symptoms may not fit neatly into the PTSD framework developed in Western nations. Science has historically privileged knowledge from the United States and other nations in the West and Global North, marking them as objectively true. The objectivity of Western science as universally applicable to all cultures has been increasingly called into question as science has become less dominated by white males, and interaction between cultures and groups becomes broadly more democratic. Clearly, what is true depends in part on the context in which it is observed.
In this way, social scientists have a unique task. People are both objects and subjects. Objectively, you could quantify how tall a person is, what car they drive, how many adverse childhood experiences they had, or their score on a PTSD checklist. Subjectively, you could understand how a person made sense of a traumatic incident or how it contributed to certain patterns in thinking, negative feelings, or opportunities for growth, for example. It is this added dimension that renders social science unique to natural science (like biology or physics), which focuses almost exclusively on quantitative data and objective truth. For this reason, this book is divided between projects using qualitative methods and quantitative methods.
There is no "better" or "more true" way of approaching social science. Instead, the methods a researcher chooses should match the question they ask. If you want to answer, "do vaccines cause autism?" you should choose methods appropriate to answer that question. It seeks an objective truth—one that is true for everyone, regardless of context. Studies like these use quantitative data and statistical analyses to test mathematical relationships between variables. If, on the other hand, you wanted to know "what does a diagnosis of PTSD mean to clients?" you should collect qualitative data and seek subjective truths. You will gather stories and experiences from clients and interpret them in a way that best represents their unique and shared truths. Where there is consensus, you will report that. Where there is contradiction, you will report that as well.
Mixed methods
In this textbook, we will treat quantitative and qualitative research methods separately. However, it is important to remember that a project can include both approaches. A mixed methods study, which we will discuss more in Chapter 8, requires thinking through a more complicated project that includes at least one quantitative component, one qualitative component, and a plan to incorporate both approaches together. As a result, mixed methods projects may require more time for conceptualization, data collection, and analysis.
Finding patterns
Regardless of whether you are seeking objective or subjective truths, research and scientific inquiry aim to find and explain patterns. Most of the time, a pattern will not explain every single person’s experience, a fact about social science that is both fascinating and frustrating. Even individuals who do not know each other can create patterns that persist over time. Those new to social science may find these patterns frustrating because they may believe that the patterns describing their sex, age, or some other facet of their lives don’t represent their experience. It’s true. A pattern can exist among your cohort without your individual participation in it. There is diversity within diversity.
Let's consider some specific examples. You probably wouldn’t be surprised to learn that a person’s social class background has an impact on their educational attainment and achievement. You may be surprised to learn that people select romantic partners that have similar educational attainment, which in turn, impacts their children's educational attainment (Eika, Mogstad, & Zafar, 2019).[15] People who have graduated college pair off with other college graduates, as so forth. This, in turn, reinforces existing inequalities, stratifying society by those who have the opportunity to complete college and those who don't.
People who object to these findings tend to cite evidence from their own personal experience. However, the problem with this response is that objecting to a social pattern on the grounds that it doesn’t match one’s individual experience misses the point about patterns. Patterns don’t perfectly predict what will happen to an individual person. Yet, they are a reasonable guide that, when systematically observed, can help guide social work thought and action. When we don't investigate these patterns scientifically, we are more likely to act on stereotypes, biases, and other harmful beliefs.
A final note on qualitative and quantitative methods
There is not one superior way to find patterns that help us understand the world. As we will learn about in Chapter 7, there are multiple philosophical, theoretical, and methodological ways to approach scientific truth. Qualitative methods aim to provide an in-depth understanding of a relatively small number of cases. They also provide a voice for the client. Quantitative methods offer less depth on each case but can say more about broad patterns because they typically focus on a much larger number of cases. A researcher should approach the process of scientific inquiry by formulating a clear research question and using the methodological tools best suited to that question.
Believe it or not, there are still significant methodological battles being waged in the academic literature on objective vs. subjective social science. Usually, quantitative methods are viewed as “more scientific” and qualitative methods are viewed as “less scientific.” Part of this battle is historical. As the social sciences developed, they were compared with the natural sciences, especially physics, which rely on mathematics and statistics to come to a truth. It is a hotly debated topic whether social science should adopt the philosophical assumptions of the natural sciences—with its emphasis on prediction, mathematics, and objectivity—or use a different set of tools—contextual understanding, language, and subjectivity—to find scientific truth.
You are fortunate to be in a profession that values multiple scientific ways of knowing. The qualitative/quantitative debate is fueled by researchers who may prefer one approach over another, either because their own research questions are better suited to one particular approach or because they happened to have been trained in one specific method. In this textbook, we’ll operate from the perspective that qualitative and quantitative methods are complementary rather than competing. While these two methodological approaches certainly differ, the main point is that they simply have different goals, strengths, and weaknesses. A social work researcher should select the method(s) that best match(es) the question they are asking.
Key Takeaways
- Social work is informed by science.
- Social science is concerned with both objective and subjective knowledge.
- Social science research aims to understand patterns in the social world.
- Social scientists use both qualitative and quantitative methods, which, while different, are often complementary.
Exercises
-
Examine a pseudoscientific claim you've heard on the news or in conversation with others. Why do you consider it to be pseudoscientific? What empirical data can you find from a quick internet search that would demonstrate it lacks truth?
- Consider a topic you might want to study this semester as part of a research project. Provide a few examples of objective and subjective truths about the topic, even if you aren't completely certain they are correct. Identify how objective and subjective truths differ.
1.3 Evidence-based practice
Learning Objectives
Learners will be able to...
- Explain how social workers produce and consume research as part of practice
- Review the process of evidence-based practice and how social workers apply research knowledge with clients and groups
“Why am I in this class?”
“When will I ever use this information?"
While students aren't always so direct, we would wager a guess that these questions are on the mind of almost every student in a research methods class. And they are valid and important questions to ask! While it may seem strange, the answer is that you will probably use these skills often. Social workers engage with research on a daily basis by consuming it through popular media, social work education, and advanced training. They also often contribute to research projects, adding new scientific information to what we know. As professors, we also sometimes hear from field supervisors who say that research competencies are unimportant in their setting. One might wonder how these organizations measure program outcomes, report the impact of their program to board members or funding agencies, or create new interventions grounded in social theory and empirical evidence.
Social workers as research consumers
Whether you know it or not, your life is impacted by research every day. Many of our laws, social policies, and court proceedings are grounded in some degree of empirical research and evidence (Jenkins & Kroll-Smith, 1996).[16] That’s not to say that all laws and social policies are good or make sense. But you can’t have an informed opinion about any of them without understanding where they come from, how they were formed, and what their evidence base is. In order to be effective practitioners across micro, meso, and macro domains, social workers need to understand the root causes and policy solutions to social problems their clients are experiencing.
A recent lawsuit against Walmart provides an example of social science research in action. A sociologist named Professor William Bielby was enlisted by plaintiffs to conduct an analysis of Walmart’s personnel policies in order to support their claim that Walmart engages in gender discriminatory practices. Bielby’s analysis shows that Walmart’s compensation and promotion decisions may indeed have been vulnerable to gender bias. In June 2011, the United States Supreme Court decided against allowing the case to proceed as a class-action lawsuit (Wal-Mart Stores, Inc. v. Dukes, 2011).[17] While a class-action suit was not pursued in this case, consider the impact that such a suit against one of our nation’s largest employers could have had on companies, their employees, and even consumers around the country.[18]
A social worker might learn about this lawsuit through popular media, news media websites or television programs. Social science knowledge allows a social worker to apply a critical eye towards new information, regardless of the source. Unfortunately, popular media does not always report on scientific findings accurately. A social worker armed with scientific knowledge would be able to search for, read, and interpret the original study as well as other information that might challenge or support the study. Chapters 3, 4, and 5 of this textbook focus on information literacy, or how to understand what we already know about a topic and contribute to that body of knowledge.
When social workers consume research, they are usually doing so to inform their practice. Clinical social workers are required by a state licensing board to complete continuing education classes in order to remain informed on the latest information in their field. On the macro side, social workers at public policy think tanks consume information to inform advocacy and public awareness campaigns. Regardless of the role of the social worker, practice must be informed by research.
Evidence-based practice
Consuming research is the first component of evidence-based practice (EBP). Drisko and Grady (2015)[19] present EBP as a process composed of "four equally weighted parts: 1) client needs and current situation, (2) the best relevant research evidence, (3) client values and preferences, and (4) the clinician’s expertise" (p. 275). It is not simply “doing what the literature says,” but is rather a process by which practitioners examine the literature, client, self, and context to inform interventions with clients and systems (McNeese & Thyer, 2004).[20] It is a collaboration between social worker, client, and context. As we discussed in section 1.2, the patterns discovered by scientific research are not applicable to all situations. Instead, we rely on our critical thinking skills to apply scientific knowledge to real-world situations.
The bedrock of EBP is a proper assessment of the client or client system. Once we have a solid understanding of what the issue is, we can evaluate the literature to determine whether there are any interventions that have been shown to treat the issue, and if so, which have been shown to be the most effective. You will learn those skills in the next few chapters. Once we know what our options are, we should be upfront with clients about each option, what the interventions look like, and what the expected outcome will be. Once we have client feedback, we use our expertise and practice wisdom to make an informed decision about how to move forward.
If this sounds familiar, it's the same approach a doctor, physical therapist, or other health professional would use. This highlights a common critique of EBP: it is too focused on micro-level, clinical social work practice. Not every social worker is a clinical social worker. While there is a large body of literature on EBP for clinical practice, the same concepts apply to other social work roles as well. A social work manager should endeavor to be familiar with evidence-based management styles, and a social work policy advocate should argue for evidence-based policies.
In agency-based social work practice, EBP can take on a different role due to the complexities of the grant funding process. Funders naturally require agencies to demonstrate that their practice is effective. Agencies are almost always required to document that they are achieving the outcomes they intended. However, funders sometimes require agencies to choose from a limited list of interventions determined to be evidence-based practices. Funders want to direct their money to treatments that are proven to work, but by excluding funding for alternative approaches and limiting the degree to which clinicians can customize and localize interventions, financial incentives can bias the EBP process away from what is best for the client and community.
Not included in this model are clinical expertise, client values, or community context —key components of EBP and the therapeutic process. According to some funders, EBP is not a process conducted by a practitioner but instead consists of a list of interventions. Similar dynamics are at play in private clinical practice, in which insurance companies may specify the modality of therapy offered. For example, insurance companies may favor short-term, solution-focused therapy which minimizes cost. But what happens when someone has an idea for a new kind of intervention? How do new approaches get "on the list" of EBPs of grant funders?
Social workers as research producers
Innovation in social work is incredibly important. Social workers work on wicked problems for their careers. For those of you who have practice experience, you may have had an idea of how to better approach a practice situation. That is another reason you are here in a research methods class. You (really!) will have bright ideas about what to do in practice. Sam Tsemberis relates an “Aha!” moment from his practice in this Ted talk on homelessness. While a faculty member at the New York University School of Medicine, he noticed a problem with people cycling in and out of the local psychiatric hospital wards. Clients would arrive in psychiatric crisis, stabilize under medical supervision in the hospital, and end up back at the hospital in psychiatric crisis shortly after discharge.
When he asked the clients what their issues were, they said they were unable to participate in homelessness programs because they were not always compliant with medication for their mental health diagnosis and they continued to use drugs and alcohol. The housing supports offered by the city government required abstinence and medication compliance before one was deemed "ready" for housing. For these clients, the problem was a homelessness service system that was unable to meet clients where they were—ready for housing, but not ready for abstinence and psychiatric medication. As a result, chronically homeless clients were cycling in and out of psychiatric crises, moving back and forth from the hospital to the street.
The solution that Sam Tsemberis implemented and popularized is called Housing First—an approach to homelessness prevention that starts by, you guessed it, providing people with housing first and foremost. Tsemberis's model addresses chronic homelessness in people with co-occurring disorders (those who have a diagnosis of a substance use and mental health disorder). The Housing First model states that housing is a human right: clients should not be denied their right to housing based on substance use or mental health diagnoses.
In Housing First programs, clients are provided housing as soon as possible. The Housing First agency provides wraparound treatment from an interdisciplinary team, including social workers, nurses, psychiatrists, and former clients who are in recovery. Over the past few decades, this program has gone from a single program in New York City to the program of choice for federal, state, and local governments seeking to address homelessness in their communities.
The main idea behind Housing First is that once clients have a residence of their own, they are better able to engage in mental health and substance use treatment. While this approach may seem logical to you, it is the opposite of the traditional homelessness treatment model. The traditional approach began with the client abstaining from drug and alcohol use and taking prescribed medication. Only after clients achieved these goals were they offered group housing. If the client remained sober and medication compliant, they could then graduate towards less restrictive individual housing.
Conducting and disseminating research allows practitioners to establish an evidence base for their innovation or intervention, and to argue that it is more effective than the alternatives, and should therefore be implemented more broadly. For example, by comparing clients who were served through Housing First with those receiving traditional services, Tsemberis could establish that Housing First was more effective at keeping people housed and at addressing mental health and substance use goals. Starting first with smaller studies and graduating to larger ones, Housing First built a reputation as an effective approach to addressing homelessness. When President Bush created the Collaborative Initiative to Help End Chronic Homelessness in 2003, Housing First was used in a majority of the interventions and its effectiveness was demonstrated on a national scale. In 2007, it was acknowledged as an evidence-based practice in the Substance Abuse and Mental Health Services Administration’s (SAMHSA) EBP resource center.[21]
We suggest browsing around the SAMHSA EBP Resource Center and looking for interventions on topics that interest you. Other sources of evidence-based practices include the Cochrane Reviews digital library and Campbell Collaboration. In the next few chapters, we will talk more about how to search for and locate literature about clinical interventions. The use of systematic reviews, meta-analyses, and randomized controlled trials are particularly important in this regard, types of research we will describe more in Chapter 3 and Chapter 4.
So why share the story of Housing First? Well, we want you to think about what you hope to contribute to our knowledge of social work practice. What is your bright idea and how can it change the world? Practitioners innovate all the time, often incorporating those innovations into their agency’s approach and mission. Using scientific research methods, agency-based social workers can demonstrate to policymakers and other social workers that their innovations should be more widely used. Without this wellspring of new ideas, social services would not be able to adapt to the changing needs of their communities. Social workers in agency-based practice may also participate in research projects taking place at their agency. Partnerships between schools of social work and agencies are a common way of testing and implementing innovations in social work. In such a case, all parties receive an advantage: clinicians receive specialized training, clients receive additional services, agencies gain prestige, and researchers can illustrate the effectiveness of an intervention.
Evidence-based practice highlights the unique perspective that social work brings to research. Social work both "holds" and critiques evidence. With regard to the former, "holding" evidence refers to the fact that the field of social work values scientific information. The Housing First example demonstrates how this interplay between valuing and critiquing science works—first by critiquing existing research and conducting research to establish a new approach to a problem. It also demonstrates the importance of listening to your target population and privileging their understanding and perception of the issue. While their understanding is not the result of scientific inquiry, it is deeply informed through years of direct experience with the issue and embedded within the relevant cultural and historical context. Although science often searches for the "one true answer," social work researchers must remain humble about the degree to which we can really know, and must begin to engage with other ways of knowing that may originate from clients and communities.
See the video on cultural humility in healthcare settings (CC-BY-NC-ND 3.0) embedded below for an example of how "one true answer" about a population can often oversimplify things and overstate how much we know about how to intervene in a given situation.
Key Takeaways
While you may not become a scientist in the sense of wearing a lab coat and using a microscope, social workers must understand science in order to engage in ethical practice. In this section, we reviewed ways in which research is a part of social work practice, including:
- Determining the best intervention for a client or system
- Ensuring existing services are accomplishing their goals
- Satisfying requirements to receive funding from private agencies and government grants
- Testing a new idea and demonstrating that it should be more widely implemented
Exercises
-
Using a social work practice situation that you have experienced, walk through the four steps of the evidence-based practice process and how they informed your decision-making. Reflect on some of the difficulties applying EBP in the real world.
- Talk with a social worker about how they produce and consume research as part of practice. Consider asking them about the articles, books, and other scholarship that changed their practice or helped them think about a problem in a new way. You might also ask them about how they stay current with the literature as a practitioner. Reflect on your personal career goals and how research will fit into your future practice.
1.4 Creating a question to examine scientific evidence
Learning Objectives
Learners will be able to...
- Identify the common elements of evidence-based practice research question
- Apply the acronym PICO to begin inquiry into a social work research topic
What is the PICO Model?
The PICO Model is a format to help define your information need into a clinical question. By organizing a clinical question using PICO, the searcher can use the specific terms to aid in finding clinically relevant evidence in the literature. PubMed alone has over 34 million citations to search through so being able to reference a defined clinical question when reviewing title/abstracts will help filter the irrelevant materials out of the search results.
The PICO Model for Clinical Questions
P |
Patient, Population, or Problem |
How would I describe a group of patients similar to mine? |
I |
Intervention, Prognostic Factor, or Exposure |
Which main intervention, prognostic factor, or exposure am I considering? |
C |
Comparison or Intervention (if appropriate) |
What is the main alternative or gold standard to compare with the intervention? |
O |
Outcome you would like to measure or achieve |
What can I hope to accomplish, measure, improve, or affect for my patient or population? |
Add additional elements, if you need to specify further.
T |
What Type of clinical question are you asking? |
Diagnosis, Etiology/Harm, Therapy, Prognosis, Prevention categories |
T |
Is Time important to your search? |
Duration of data collection, duration of treatment, time to follow-up |
S |
What Study type do you want to find? |
What study design/methodology will address the clinical question according to the evidence hierarchy? |
Let's watch some examples on how to use PICO when approaching scientific literature. Please view the video below from the University of Binghamton Libraries on how to create a question to guide inquiry into evidence-based practice and other scientific questions about social work practice.
Exercises
- Create at least two questions using the PICO framework that are interesting enough for you to read about.
Chapter Outline
- Choosing a research topic (10 minute read)
- Your research proposal (14 minute read)
- Critical considerations (5 minute read)
- Evaluating online resources (11 minute read)
Content warning: Examples in this chapter discuss substance use disorders, mental health disorders and therapies, obesity, poverty, gun violence, gang violence, school discipline, racism and hate groups, domestic violence, trauma and triggers, incarceration, child neglect and abuse, bullying, self-harm and suicide, racial discrimination in housing, burnout in helping professions, and sex trafficking of indigenous women.
2.1 Choosing a research topic
Learning Objectives
Learners will be able to...
- Brainstorm topics you may want to investigate as part of a research project
- Explore your feelings and existing knowledge about the topic
- Develop a working question
Research methods is a unique class in that you get to decide what you want to learn about. Perhaps you came to your MSW program with a specific issue you were passionate about. In my MSW program, I wanted to learn about the best interventions to use with people who have substance use disorders. This was in line with my future career plans, which included working in a clinical setting with clients with co-occurring mental health and substance use issues. I suggest you start by thinking about your future practice goals and create a research project that addresses a topic that represents an area of social work you are passionate about.
For those of you without a specific direction, don't worry. Many people enter their MSW program without an exact topic in mind they want to study. Throughout the program, you will be exposed to different populations, theories, practice interventions, and policies that will spark your interest. Think back to papers you enjoyed researching and writing in other classes. You may want to continue studying the same topic. Research methods will enable you to gain a deeper, more nuanced understanding of a topic or issue. If you haven't found an interesting topic yet, here are some other suggestions for seeking inspiration for a research project:
- If you already have practice experience in social work through employment, an internship, or volunteer work, think about practice issues you noticed in the placement. Do you have any idea of how to better address client needs? Do you need to learn more about existing interventions or the programs that fund your agency? Use this class as an opportunity to engage with your previous field experience in greater detail. Begin with “what” and “why” questions and then expand on those. For example, what are the most effective methods of treating severe depression among a specific population? Or why are people receiving food assistance more likely to be obese?
- You could also ask a professor at your school about possible topics. Read departmental information on faculty research interests, which may surprise you. Most departmental websites post the curriculum vitae (CV) of faculty, which lists their publications, credentials, and interests. For those of you interested in doctoral study, this process is particularly important. Students often pick schools based on professors they want to learn from or research initiatives they want to join.
Once you have a potential idea, start reading! A simple web search should bring you some basic information about your topic. News articles can reveal new or controversial information. You may also want to identify and browse academic journals related to your research interests. Faculty and librarians can help you identify relevant journals in your field and specific areas of interest. We'll also review more detailed strategies for searching the literature in Chapter 3. As you read, look for what’s missing. These may be “gaps in the literature” that you might explore in your own study.
It's a good idea to keep it simple when you're starting your project. Choose a topic that can be easily defined and explored. Your study cannot focus on everything that is important about your topic. A study on gun violence might address only one system, for example schools, while only briefly mentioning other systems that impact gun violence. That doesn't mean it's a bad study! Every study presents only a small picture of a larger, more complex and multifaceted issue. The sooner you can arrive at something specific and clear that you want to study, the better off your project will be.
Writing a working question
There are lots of great research topics. Perhaps your topic is a client population—for example, youth who identify as LGBTQ+ or visitors to a local health clinic. In other cases, your topic may be a social problem, such as gang violence, or a social policy or program, such as zero-tolerance policies in schools. Alternately, maybe there are interventions such as dialectical behavioral therapy or applied behavior analysis that interest you.
Whatever your topic idea, begin to think about it in terms of a question. What do you really want to know about the topic? As a warm-up exercise, try dropping a possible topic idea into one of the blank spaces below. The questions may help bring your subject into sharper focus and bring you closer towards developing your topic.
- What does ___ mean?
- What are the causes of ___?
- What are the consequences of ___?
- What are the component parts of ___?
- How does ___ impact ___?
- What is it like to experience ___?
- What is the relationship between _____ and the outcome of ____?
- What case can be made for or against ___?
- What are the risk/protective factors for ___?
- How do people think about ___?
Take a minute right now and write down a question you want to answer. Even if it doesn’t seem perfect, it is important to start somewhere. Make sure your research topic is relevant to social work. You’d be surprised how much of the world that encompasses. It’s not just research on mental health treatment or child welfare services. Social workers can study things like the pollution of irrigation systems and entrepreneurship in women, among other topics. The only requirement is your research must inform action to fight social problems faced by target populations.
Because research is an iterative process, one that you will revise over and over, your question will continue to evolve. As you progress through this textbook, you’ll learn how to refine your question and include the necessary components for proper qualitative and quantitative research questions. Your question will also likely change as you engage with the literature on your topic. You will learn new and important concepts that may shift your focus or clarify your original ideas. Trust that a strong question will emerge from this process. A good researcher must be comfortable with altering their question as a result of scientific inquiry.
Very often, our students will email us in the first few weeks of class and ask if they have a good research topic. We love student emails! But just to reassure you if you're about to send a panicked email to your professor, as long as you are interested in dedicating a semester or two learning about your topic, it will make a good research topic. That's why we would advise you to focus on how much you like this topic, so that three months from now you are still motivated to complete your project. Your project should have meaning to you.
How do you feel about your topic?
Now that you have an idea of what you might want to study, it's time to consider what you think and feel about that topic. Your motivation for choosing a topic does not have to be objective. Because social work is a value-based profession, scholars often find themselves motivated to conduct research that furthers social justice or fights oppression. Just because you think a policy is wrong or a group is being marginalized, for example, does not mean that your research will be biased. It means you must understand what you feel, why you feel that way, and what would cause you to feel differently about your topic.
Start by asking yourself how you feel about your topic. Sometimes the best topics to research are those about which you feel strongly. What better way to stay engaged with your research project than to study something you are passionate about? However, you must be able to accept that people may have a different perspective, and you must represent their viewpoints fairly in the research report you produce. If you feel prepared to accept all findings, even those that may be unflattering or distinct from your personal perspective, then perhaps you should begin your research project by intentionally studying a topic about which you have strong feelings.
Kathleen Blee (2002)[22] has taken this route in her research. Blee studies groups whose racist ideologies may be different than her own. You can listen to her lecture Women in Organized Racism that details some of her findings. Her scientific research is so impactful because she was willing to report her findings and observations honestly, even those contrary to her beliefs and feelings. If you believe that you may have personal difficulty sharing findings with which you disagree, then you may want to study a different topic. Knowing your own hot-button issues is an important part of self-knowledge and reflection in social work, and there is nothing wrong with avoiding topics that are likely to cause you unnecessary stress.
Social workers often use personal experience as a starting point to identify topics of interest. As we’ve discussed here, personal experience can be a powerful motivator to learn more about a topic. However, social work researchers should be mindful of their own mental health during the research process. A social worker who has experienced a mental health crisis or traumatic event should approach researching related topics cautiously. There is no need to trigger yourself or jeopardize your mental health for a research project. For example, a student who has just experienced domestic violence may want to know about Eye Movement Desensitization and Reprocessing (EMDR) therapy. While the student might gain some knowledge about potential treatments for domestic violence, they will likely have to read through many stories and reports about domestic violence as part of the research process. Unless the student’s trauma has been processed in therapy, conducting a research project on this topic may negatively impact the student’s mental health.
What do you think about your topic?
Once you figure out what you feel about your topic, consider what you think about it. There are many ways we know what we know. Perhaps your mother told you something is so. Perhaps it came to you in a dream. Perhaps you took a class last semester and learned something about your topic there. Or you may have read something about your topic in your local newspaper. We discussed the strengths and weaknesses associated with some of these different sources of knowledge in Chapter 1, and we’ll talk about other scientific sources of knowledge in Chapter 3 and Chapter 4. For now, take some time to think of everything you know about your topic. Thinking about what you already know will help you identify any biases you may have, and it will help as you begin to frame a question about your topic.
You might consider creating a concept map, just to get your thoughts and ideas on paper and beginning to organize them. Consider this video from the University of Guelph Library (CC-BY-NC-SA 4.0).
Key Takeaways
- You should pick a topic for your research proposal that you are interested in, since you will be working with it for several months.
- Investigate your own feelings and thoughts about a topic, and make sure you can be objective and fair in your investigation.
- Research projects are guided by a working question that develops and changes as you learn more about your topic.
Exercises
Just as a reminder, exercises are designed to help you create your individual research proposal. We designed these activities to break down your proposal into small but manageable chunks. We suggest completing each exercise so you can apply what you are learning to your individual research project, as the exercises in each section and each chapter build on one another.
If you haven't done so already, you can create a document in a word processor on your computer or in a written notebook with your answers to each exercise.
-
Brainstorm at least 4-5 topics of interest to you and pick the one you think is the most promising for a research project.
- For your chosen topic, outline what you currently know about the topic and your feelings towards the topic. Make sure you are able to be objective and fair in your research.
- Formulate at least one working question to guide your inquiry. It is common for topics to change and develop over the first few weeks of a project, but think of your working question as a place to start. Use the 10 examples we provided in this chapter if you need some help getting started.
2.2 Your research proposal
Learning Objectives
Learners will be able to...
- Describe the stages of a research project
- Define your target population and describe how your study will impact that population
- Identify the aim of your study
- Classify your project as descriptive, exploratory, explanatory, or evaluative
Most research methods courses are designed to help students propose a research project. But what is a research project? Figure 2.1 indicates the steps of the research project. Right now, we are in the top right corner, using your informal observations from your practice experience and lived experience to form a working draft of your research question. In the next three chapters, you'll learn how to find and evaluate scholarly literature on your topic. After thoroughly evaluating the literature, you'll conceptualize an empirical study based on a research question you create. In many courses, students will have to carry out these designs and make a contribution to the research literature in their topic area.
This book uses a project-based approach because it mimics the real-world process of inquiry on a social work topic. In an introductory research methods course, students often have to create a research proposal followed by a more advanced research class in which they conduct quantitative and qualitative data analysis. The research proposal, is a document produced by researchers that reviews the literature relevant to their topic and describes the methods they will use to conduct their study. Part 1 of this textbook is designed to help you with your literature review. Part 2 is designed to help you figure out which methods you will use in your study.
Check with your professor on whether you are required to carry out the project you propose to do in your research proposal. Some of you may only need to propose a hypothetical project. If you are planning to complete your project, you will have to pay more attention to the practical and ethical considerations in this chapter.
A research proposal is focused on a question. Right now, this is your working question from Section 2.1. If you haven't created one yet, this is a good time to pause and complete the exercises from section 2.1.[23] It is likely you will revise your working question many times as you read more literature about your topic. Consider yourself in the cycle between (re)creating your research question and reviewing the research literature for Part 1 of the textbook.
Student research proposals
Student research projects are a big undertaking, but they are well within your capability as a graduate student. Let's start with the research proposal. Think about the research proposal as a communication device. You are telling the reader (your professor, usually) everything they need to know in order to understand your topic and the research study you plan to do. You are also demonstrating to the reader that you are competent and informed enough to conduct the study.
You can think of a research proposal like creating a recipe. If you are a chef trying to cook a new dish from scratch, you would probably start by looking at other recipes. You might cook a few of them and come up with ideas about how to create your own version of the dish. Writing your recipe is a process of trial and error, and you will likely revise your proposal many times over the course of the semester. This textbook and its exercises designed to get you working on your project little by little, so that by the time you turn in your final research proposal, you'll be confident it represents the best way to answer your question. Of course, like with any time I cook, you never quite know how it will turn out. What matters for scientists in the end isn't whether your data proves your ideas right or wrong or whether your data collection doesn't work as planned or goes off perfectly. Instead, what matters is that you report your results (warts and all) as honestly and openly as possible to inform others engaged in scholarly inquiry.
Is writing a research proposal a useful skill for a social worker? On one hand, you probably won't be writing research proposals for a living. But the same structure of a research proposal (literature review + methods) is used in grant applications. Writing grant proposals is often a part of practice, particularly in agency-based and policy practice. Instead of finding a gap in the literature to study, practitioners write grant proposals describing a program they will use to address an issue in their community, as well as the research methods they will use to evaluate whether it worked. Similarly, a policy advocate or public administrator might sketch out a proposed program and its evaluation as part of a proposal. Proposal writing may differ somewhat in practice, but the general idea is the same.
Focusing your project
Based on your work in Section 2.1, you should have a working question—a place to start. Think about what you hope to accomplish with your study. This is the aim of your research project. Often, social work researchers begin with a target population in mind. As you will recall from section 1.4, social work research is research for action. Social workers engage in research to help people. Think about your working question. Why do you want to answer it? What impact would answering your question have?
In my MSW program, I began my research by looking at ways to intervene with people who have substance use disorders. My foundation year placement was in an inpatient drug treatment facility that used 12-step facilitation as its primary treatment modality. I observed that this approach differed significantly from others I had been exposed to, especially the idea of powerlessness over drugs and drug use. My working question started as "what are the alternatives to 12-step treatment for people with substance use issues and are they more effective?" The aim of my project was to determine whether different treatment approaches might be more effective, and I suspected that self-determination and powerlessness were important.
It's important to note that my working question contained a target population—people with substance use disorders. A target population is the group of people that will benefit the most. I envisioned I would help the field of social work to think through how to better meet clients where they were at, specific to the problem of substance use. I was studying to be a clinical social worker, so naturally, I formulated a micro-level question. Yet, the question also has implications for meso- and macro-level practice. If other treatment methods are more effective than 12-step facilitation, then we should direct more public money towards providing more effective therapies for people who use substances. We may also need to train the substance use professionals to use new treatment methodologies.
Exercises
Think about your working question.
- Is it more oriented towards micro-, meso-, or macro-level practice?
- What implications would answering your question have at each level of the ecosystem?
Asking yourself whether your project is more micro, meso, or macro is a good check to see if your project is well-focused. A project that seems like it could be all of those might have too many components or try to study too much. Consider identifying one ecosystemic level your project will focus on, and you can interpret and contextualize your findings at the other levels of analysis.
Exploration, description, and explanation
Social science is a big place. Looking at the various empirical studies in the literature, there is a lot of diversity—from focus groups with clients and families to multivariate statistical analysis of large population surveys conducted online. Ultimately, all of social science can be described as one of three basic types of research studies. As you develop your research question, consider which of the following types of research studies fits best with what you want to learn about your topic. In subsequent chapters, we will use these broad frameworks to help craft your study's final research question and choose quantitative and qualitative research methods to answer it.
Exploratory research
Researchers conducting exploratory research are typically at the early stages of examining their topics. Exploratory research projects are carried out to test the feasibility of conducting a more extensive study and to figure out the “lay of the land” with respect to the particular topic. Usually, very little prior research has been conducted on this topic. For this reason, a researcher may wish to do some exploratory work to learn what method to use in collecting data, how best to approach research subjects, or even what sorts of questions are reasonable to ask.
Often, student projects begin as exploratory research. Because students don't know as much about the topic area yet, their working questions can be general and vague. That's a great place to start! An exploratory question is great for delving into the literature and learning more about your topic. For example, the question “what are common social work interventions for parents who neglect their children?” is a good place to start when looking at articles and textbooks to understand what interventions are commonly used with this population. However, it is important for a student research project to progress beyond exploration unless the topic truly has very little existing research.
In my classes, I often read papers where students say there is not a lot of literature on a topic, but a quick search of library databases shows a deep body of literature on the topic. The skills you develop in Chapter 3 and Chapter 4 should assist you with finding relevant research, and working with a librarian can definitely help with finding information for your research project. That said, there are a few students each year who pick a topic for which there is in fact little existing research. Perhaps, if you were looking at child neglect interventions for parents who identify as transgender or parents who are refugees from the Syrian civil war, less would be known about child neglect for those specific populations. In that case, an exploratory design would make sense as there is little, if any, literature about your specific topic.
Descriptive research
Another purpose of a research project is to describe or define a particular phenomenon. This is called descriptive research. For example, researchers at the Princeton Review conduct descriptive research each year when they set out to provide students and their parents with information about colleges and universities around the United States. They describe the social life at a school, the cost of admission, and student-to-faculty ratios (to name just a few of the categories reported). If our topic were child neglect, we might seek to know the number of people arrested for child neglect in our community and whether they are more likely to have other problems, such as poverty, mental health issues, or substance use.
Social workers often rely on descriptive research to tell them about their service area. Keeping track of the number of parents receiving child neglect interventions, their demographic makeup (e.g., race, sex, age), and length of time in care are excellent examples of descriptive research. On a more macro-level, the Centers for Disease Control provides a remarkable amount of descriptive research on mental and physical health conditions. In fact, descriptive research has many useful applications, and you probably rely on such findings without realizing you are reading descriptive research.
Explanatory research
Lastly, social work researchers often aim to explain why particular phenomena operate in the way that they do. Research that answers “why” questions is referred to as explanatory research. Asking "why" means the researcher is trying to identify cause-and-effect relationships in their topic. For example, explanatory research may try to identify risk and protective factors for parents who neglect their children. Explanatory research may attempt to understand how religious affiliation impacts views on immigration. All explanatory research tries to study cause-and-effect relationships between two or more variables. A specific offshoot of explanatory research that comes up often is evaluation research, which investigates the impact of an intervention, program, or policy on a group of people. Evaluation research is commonly practiced in agency-based social work settings, and later chapters will discusses some of the basics for conducting a program evaluation.
There are numerous examples of explanatory social scientific investigations. For example, Dominique Simons and Sandy Wurtele (2010)[24] sought to understand whether receiving corporal punishment from parents led children to turn to violence in solving their interpersonal conflicts with other children. In their study of 102 families with children between the ages of 3 and 7, the authors found that experiencing frequent spanking did in fact result in children being more likely to accept aggressive problem-solving techniques. Another example of explanatory research can be seen in Robert Faris and Diane Felmlee’s (2011)[25] research study on the connections between popularity and bullying. From their study of 8th, 9th, and 10th graders in nineteen North Carolina schools, they found that aggression increased as adolescents’ popularity increased.[26]
Exercises
- Think back to your working question from section 2.1. Which type of research—exploratory, descriptive, or explanatory—best describes your working question?
- Try writing a question about your topic that fits with each type of research.
Important things are more rewarding to do
Another consideration in starting a research project is whether the question is important enough to answer. For the researcher, answering the question should be important enough to put in the effort and time required to complete a research project. As we discussed in section 2.1, you should choose a topic that is important to you—one you wouldn’t mind learning about for at least a few months, if not a few years. Time is your most precious resource as a student. Make sure you dedicate it to topics and projects you consider genuinely important.
Your research question should also be contribute to the larger expanse of research in that area. For example, if your research question is "does cognitive behavioral therapy (CBT) effectively treat depression?" you are a few decades late to be asking that question. Hundreds of scientists have published articles demonstrating its effectiveness in treating depression. However, a student interested in learning more about CBT can still find new areas to research. Perhaps there is a new population—for example, older adults in a nursing home—or a new problem—like mobile phone addiction—for which there is little research on the impact of CBT.
Your research project should contribute something new to social science. It should address a gap in what we know and what is written in the literature. This can seem intimidating for students whose projects involve learning a totally new topic. How could I add something new when other researchers have studied this for decades? Trust us, by thoroughly reviewing the existing literature, you can find new and unresolved research questions to answer. Google Scholar’s motto at the bottom of their search page is “stand on the shoulders of giants.” Social science research rests on the work of previous scholars, and builds off of what they discovered to learn more about the social world. Ensure that your question will bring our scientific understanding of your topic to new heights.
Finally, your research question should be of import to the social world. Social workers conduct research on behalf of individuals, groups, and communities to promote change as part of their mission to advance human rights and further social and economic justice. Your research should matter to the people you are trying to help. Your research project should aim to improve the lives of people in your target population by helping the world understand their needs more holistically.
Research projects, obviously, do not need to address all aspects of a problem. As social workers, our goal in enacting social justice isn't to accomplish it all in one semester (or even one lifetime). Our goal is to move the world in the right direction and make small, incremental progress. We encourage all students to think about how they will make their work accessible and relevant to the broader public and use their results to promote change.
Key Takeaways
- Research exists in a cycle. Your research project will follow this cycle, beginning from reading literature (where you are now), to proposing a study, to completing a research project, and finally, to publishing the results.
- Social work researchers should identify a target population and understand how their project will impact them.
- Research projects can be exploratory, descriptive, evaluative, or a combination therein. While you are likely still exploring your topic, you may settle on another type of research, particularly if your topic has been previously addressed extensively in the literature.
- Your research project should be important to you, fill a gap or address a controversy in the scientific literature, and make a difference for your target population and broader society.
Exercises
- State why your working question is an important one to answer, keeping in mind that your statement should address the scientific literature, target population, and the social world.
2.3 Critical considerations
Learning Objectives
Learners will be able to...
- Critique the traditional role of researchers and identify how action research addresses these issues
So far in this chapter, we have presented the steps of student research projects as follows:
- Find a topic that is important to you and read about it.
- Pose a question that is important to the literature and to your community.
- Propose to use specific research methods to answer your question.
- Carry out your project and report the results.
These were depicted in Figure 2.1 earlier in this chapter. There are important limitations to this approach. This section examines those problems and how to address them.
Whose knowledge is privileged?
First, let's critically examine your role as the researcher. Following along with the steps in a research project, you start studying the literature your topic, find a place where you can add to scientific knowledge, and conduct your study. But why are you the person who gets to decide what is important? Just as clients are the experts on their lives, members of your target population are the experts on their lives. What does it mean for a group of people to be researched on, rather than researched with? How can we better respect the knowledge and self-determination of community members?
A different way of approaching your research project is to start by talking with members of the target population and those who are knowledgeable about that community. Perhaps there is a community-led organization you can partner with on a research project. The researcher's role in this case would be more similar to a consultant, someone with specialized knowledge about research who can help communities study problems they consider to be important. The social worker is a co-investigator, and community members are equal partners in the research project. Each has a type of knowledge—scientific expertise vs. lived experience—that should inform the research process.
The community focus highlights something important about student projects: they are localized. Student projects can dedicate themselves to issues at a single agency or within a service area. With a local scope, student researchers can bring about change in their community. This is the purpose behind action research.
Action research
Action research is research that is conducted for the purpose of creating social change. When engaging in action research, scholars collaborate with community stakeholders to conduct research that will be relevant to the community. Social workers who engage in action research don't just go it alone; instead, they collaborate with the people who are affected by the research at each stage in the process. Stakeholders, particularly those with the least power, should be consulted on the purpose of the research project, research questions, design, and reporting of results.
Action research also distinguishes itself from other research in that its purpose is to create change on an individual and community level. Kristin Esterberg puts it quite eloquently when she says, “At heart, all action researchers are concerned that research not simply contribute to knowledge but also lead to positive changes in people’s lives” (2002, p. 137).[27] Action research has multiple origins across the globe, including Kurt Lewin’s psychological experiments in the US and Paulo Friere’s literacy and education programs (Adelman, 1993; Reason, 1994).[28] Over the years, action research has become increasingly popular among scholars who wish for their work to have tangible outcomes that benefit the groups they study.
A traditional scientist might look at the literature or use their practice wisdom to formulate a question for quantitative or qualitative research, as we suggested earlier in this chapter. An action researcher, on the other hand, would consult with people in target population and community to see what they believe the most pressing issues are and what their proposed solutions may be. In this way, action research flips traditional research on its head. Scientists are not the experts on the research topic. Instead, they are more like consultants who provide the tools and resources necessary for a target population to achieve their goals and to address social problems using social science research.
According to Healy (2001),[29] the assumptions of participatory-action research are that (a) oppression is caused by macro-level structures such as patriarchy and capitalism; (b) research should expose and confront the powerful; (c) researcher and participant relationships should be equal, with equitable distribution of research tasks and roles; and (d) research should result in consciousness-raising and collective action. Consistent with social work values, action research supports the self-determination of oppressed groups and privileges their voice and understanding through the conceptualization, design, data collection, data analysis, and dissemination processes of research. We will return to similar ideas in Part 4 of the textbook when we discuss qualitative research methods, though action research can certainly be used with quantitative research methods, as well.
Student projects can make a difference!
One last thing. We've told you all to think small and simple with your projects. The adage that "a good project is a done project" is true. At the same time, this advice might unnecessarily limit an ambitious and diligent student who wanted to investigate something more complex. For example, here is a Vice News article about MSW student Christine Stark's work on sex trafficking of indigenous women. Student projects have the potential to address sensitive and politically charged topics. With support from faculty and community partners, student projects can become more comprehensive. The results of your project should accomplish something. Social work research is about creating change, and you will find the work of completing a research project more rewarding and engaging if you can envision the change your project will create.
In addition to broader community and agency impacts, student research projects can have an impact on a university or academic program. Consider this resource on how to research your institution by Rine Vieth. As a student, you are one of the groups on campus with the least power (others include custodial staff, administrative staff, contingent and adjunct faculty). It is often necessary that you organize within your cohort of MSW students for change within the program. Not only is it an excellent learning opportunity to practice your advocacy skills, you can use raw data that is publicly available (such as those linked in the guide) or create your own raw data to inform change. The collaborative and transformative focus of student research projects like these can be impactful learning experiences, and students should consider projects that will lead to some small change in both themselves and their communities.
Key Takeaways
- Traditionally, researchers did not consult target populations and communities prior to formulating a research question. Action research proposes a more community-engaged model in which researchers are consultants that help communities research topics of import to them.
- Just because we’ve advised you to keep your project simple and small doesn’t mean you must do so! There are excellent examples of student research projects that have created real change in the world.
Exercises
- Apply the key concepts of action research to your project. How might you incorporate the perspectives and expertise of community members in your project? How can your project create real change?
2.4 Evaluating internet resources
Learning Objectives
Learners will be able to...
- Apply the SIFT technique to find better coverage of scientific information and current events.
When first learning about a new topic, a natural first place to look is an internet search engine (e.g., Google, Bing, or DuckDuckGo). Before diving into the academic literature (which we will do in the next chapter), let's explore how to use research methods to find scientific information that is intended for non-expert audiences. Take some time to learn the basics of your topic before you dive into more advanced literature, which may present a more nuanced (or jargon-filled and confusing) study of the topic. Generally, scholarly literature is specialized, in that it does not try to provide a broad overview of a topic for a non-expert audience. When you are looking at journal articles, you are looking at literature intended for other scientists and researchers.
While you can be assured that articles in reputable journals have passed peer review, that does not always mean they contain accurate information. Articles are often debated on social media or in journalistic outlets. For example, here is a news story debunking a journal article which erroneously found Safe Consumption Sites for people who use drugs were moderately associated with crime increases. After multiple scholars evaluated the article's data, they realized there were flaws in the design and the conclusions were not supported, which led the journal to retract the article. You can find these controversies in the literature by using a Google Scholar feature we've talked about before—'Cited By'. Click the 'Cited By' link to see which articles cited the article you are evaluating. If you see critical commentary on the article by other scholars, it is likely an area of active scientific debate. You should investigate the controversy further prior to using the source in your literature review.
If your literature search contains sources other than academic journal articles (and almost all of them do), you'll need to do a bit more work to assess whether the source is reputable enough to include in your review. Let's say you find a report from a Google Scholar search or a Bing search. Without peer review or a journal's approval, how do you know the information you are reading is any good?
The SIFT method
Mike Caulfield, Washington State University digital literacy expert, has helpfully condensed key fact-checking strategies into a short list of four moves, or things to do to quickly make a decision about whether or not a source is worthy of your attention. It is referred to as the “SIFT” method, and it stands for Stop, Investigate the source, Find better coverage, and Trace claims claims, quotes, and media to the original context.
Stop
When you initially encounter a source of information and start to read it—stop. Ask yourself whether you know and trust the author, publisher, publication, or website. If you don’t, use the other fact-checking moves that follow, to get a better sense of what you’re looking at. In other words, don’t read, share, or use the source in your research until you know what it is, and you can verify it is reliable.
This is a particularly important step, considering what we know about the attention economy—social media, news organizations, and other digital platforms purposely promote sensational, divisive, and outrage-inducing content that emotionally hijacks our attention in order to keep us “engaged” with their sites (clicking, liking, commenting, sharing). Stop and check your emotions before engaging! What about this website is driving your engagement?
Investigate the sources
You don’t have to do a three-hour investigation into a source to determine its truth. But if you’re reading a piece on economics, and the author is a Nobel prize-winning economist, that would be useful information. Likewise, if you’re watching a video on the many benefits of milk consumption, you would want to be aware if the video was produced by the dairy industry. This doesn’t mean the Nobel economist will always be right and that the dairy industry can’t ever be trusted. But knowing the expertise and agenda of the person who created the source is crucial to your interpretation of the information provided.
When investigating a source, fact-checkers read “laterally” across many websites, rather than digging deep (reading “vertically”) into the one source they are evaluating. That is, they don’t spend much time on the source itself, but instead they quickly get off the page and see what others have said about the source. Indeed, one study cited in the video below found that academic historians are actually less able to tell the difference between reputable and bogus internet sources because they do not read laterally but instead check references and credentials. Those are certainly a good idea to check when reading a source in detail, but fact checkers instead ask what other sources on the web say about it rather than what the source says about itself. They open up many tabs in their browser, piecing together different bits of information from across the web to get a better picture of the source they’re investigating. Not only is this faster, but it harnesses the collected knowledge of the web to more accurately determine whether a source is reputable or not.
We recommend watching this short video [2:44] for a demonstration of how to investigate online sources. Pay particular attention to how Wikipedia can be used to quickly get useful information about publications, organizations, and authors. Note: Turn on closed captions with the “CC” button or use the text transcript if you prefer to read.
Find better coverage
What if the source you find is low-quality, or you can’t determine if it is reliable or not? Perhaps you don’t really care about the source—you care about the claim that source is making. You want to know if it is true or false. You want to know if it represents a consensus viewpoint, or if it is the subject of much disagreement. A common example of this is a meme you might encounter on social media. The random person or group who posted the meme may be less important than the quote or claim the meme makes.
Your best strategy in this case might actually be to find a better source altogether, to look for other coverage that includes trusted reporting or analysis on that same claim. Rather than relying on the source that you initially found, you can trade up for a higher quality source. The point is that you’re not wedded to using that initial source. We have the internet! You can go out and find a better source, and invest your time there.
We recommend watching this short video [4:10] that demonstrates how to find better coverage and notes how fact-checkers build a library of trusted sources they can rely on to provide better coverage. Note: Turn on closed captions with the “CC” button or use the text transcript if you prefer to read.
Trace claims, quotes, and media to the original context
Much of what we find on the internet has been stripped of context. Maybe there’s a video of a fight between two people with Person A as the aggressor. But what happened before that? What was clipped out of the video and what stayed in? Maybe there’s a picture that seems real but the caption could be misleading. Maybe a claim is made about a new medical treatment based on a research finding—but you’re not certain if the cited research paper actually said that. The people who re-report these stories either get things wrong by mistake, or, in some cases, they are intentionally misleading us.
In these cases you will want to trace the claim, quote, or media back to the source, so you can see it in its original context and get a sense of whether the version you saw was accurately presented. We will talk about this more in Chapter 3 when we distinguish between primary sources and secondary sources. Secondary and tertiary sources are great for getting started with a topic, but researchers want to rely on the most highly informed source to give us information about a topic. If you see a news article about a research study, look for the journal article written by the researchers who performed the study as citations for your paper rather than a journalist who is unaffiliated with the project.
We recommend watching this short video [1:33] that discusses re-reporting vs. original reporting and demonstrates a quick tip: going “upstream” to find the original reporting source. Researchers must follow the thread of information from where they first read it to where it originated in order to understand its truth and value. Social workers who fail to check their sources can spread misinformation within our practice context or come to ill-informed conclusions that hurt clients or communities.
Once you have limited your search to trustworthy sources, ask yourself the following questions when evaluating which of these sources to download:
- Does this source help me answer my working question?
- Does this source help me revise and focus my working question?
- Does this source help me address what my professor expects in a literature review?
- Is this the best source I can find? Is this a primary or secondary source?
- What is the original context of this information?
- Is there controversy surrounding this source?
- Are the publisher and author reputable and unbiased?
Reflect and plan for the future
As you look search the literature, you will learn more about your topic area. You will learn new concepts that become new keywords in new queries. You will continue to come up with search queries and download articles throughout the research process. While we present this material at the beginning of the textbook, that is a bit misleading. You will return to search the literature often during the research process. As such, it is important to keep notes about what you did at each stage. I usually keep a "working notes" document in the same folder as the PDFs of articles I download. I can write down which categories different articles fall into (e.g., theoretical articles, empirical articles), reflect on how my question may need to change, or highlight important unresolved questions or gaps revealed in my search.
Creating and refining your working question will help you identify the key concepts you study will address. Once you identify those concepts, you’ll need to decide how to define them and how to measure them when it comes time to collect your data. As you are reading articles, note how other researchers who study your topic define concepts theoretically in the introduction and measure them in their methods section. Tuck these notes away for the future, when you will have to define and measure these concepts.
You need to be able to speak intelligently about the target population you want to study, so finding literature about their strengths, challenges, and how they have been impacted by historical and cultural oppression is a good idea. Last, but certainly not least, you should consider any potential ethical concerns that could arise during the course of carrying out your research project. These concerns might come up during your data collection, but they may also arise when you get to the point of analyzing data or disseminating results.
Decisions about the various research components do not necessarily occur in sequential order. For example, you may have to think about potential ethical concerns before changing your working question. In summary, the following list shows some of the major components you’ll need to consider as you design your research project. Make sure you have information that will inform how you think about each component.
- Research question
- Literature review
- Theories and causal relationships
- Unit of analysis and unit of observation
- Key concepts (conceptual definitions and operational definitions)
- Method of data collection
- Research participants (sample and population)
- Ethical concerns
Carve some time out each week during the beginning of the research process to revisit your working question. As you write notes on the articles you find, reflect on how that knowledge would impact your working question and the purpose of your research. You still have some time to figure it out. We'll work on turning your working question into a full-fledged research question in Chapter 9.
Key Takeaways
- Research requires fact-checking. The SIFT technique is an easy approach to critically investigating internet resources about your topic.
- Investigate the source of the information you find on the web and look for better coverage.
- Search for internet resources that help you address your working question and write your research proposal.
Exercises
- Look at your professor's prompt for a literature review and sketch out how you might answer those questions using your present level of knowledge. Search for sources that support or challenge what you think is true about your topic.
- Find a news article reporting about topics similar to your working question. Identify whether it is a primary or secondary source. If it is a secondary source, trace any claims to their primary sources. Provide the URLs.
Chapter Outline
- The scholarly literature (26 minute read)
- Identifying your information needs (13 minute read)
- Searching basics (23 minute read)
Content warning: examples in this chapter contain references to school discipline, mental health, gender-based discrimination, police shootings, ableism, autism and anti-vaccination conspiracy theories, children’s mental health, child abuse, poverty, substance use disorders and parenting/pregnancy, and tobacco use.
3.1 The scholarly literature
Learning Objectives
Learners will be able to...
- Define what we mean by "the literature"
- Differentiate between primary and secondary sources and summarize why researchers use primary sources
- Distinguish between different types of resources and how they may be useful for a research project
- Identify different types of journal articles
In the last chapter, we discussed how looking at the literature on your topic is the first thing you will need to do as part of a student research project. A literature review makes up the first half of a research proposal. But what do we mean by "the literature?" The literature consists of the published works that document a scholarly conversation on a specific topic within and between disciplines. You will find in “the literature” documents that track the progression of the research and background of your topic. You will also find controversies and unresolved questions that can inspire your own project. By this point in your social work academic career, you’ve probably heard that you need to get “peer-reviewed journal articles.” But what are those exactly? How do they differ from news articles or encyclopedias? That is the focus of this section of the text—different types of literature.
Disciplines
Information does not exist in the environment like some kind of raw material. It is produced by individuals working within a particular field of knowledge who use specific methods for generating new information, or a discipline. Disciplines consume, produce, and share knowledge. Looking through a university’s course catalog gives clues to disciplinary structure. Fields such as political science, biology, history, and mathematics are unique disciplines, as is social work. We have things we study often, like social welfare policy or mental health interventions, and a particular theoretical and ethical framework through which we view the world.
Social work researchers must take an interdisciplinary perspective, engaging with not just social work literature but literature from many disciplines to gain a comprehensive and accurate understanding of a topic. Consider how nursing and social work would differ in studying sports in schools. A nursing researcher might study how sports affect individuals’ health and well-being, how to assess and treat sports injuries, or the physical conditioning required for athletics. A social work researcher might study how schools privilege or punish student athletes, how athletics impact social relationships and hierarchies, or the differences in participation in organized sports between children of different genders. People in some disciplines are much more likely to publish on sports in schools than those in others, like political science or religious studies. I'm sure you can find a bit of scholarship on sports in schools in most disciplines, though you will want to target disciplines that are likely to address the topic often as part of the core purpose of the profession (like sports medicine or sports social work). Sometimes disciplines overlap in their focus, as clinical social work and counseling psychology or urban planning and public administration.
You will need to become comfortable with identifying the disciplines that might contribute information to any literature search. When you do this, you will also learn how to decode the way people talk about a topic within a discipline. This will be useful to you when you begin a review of the literature in your area of study.
Exercises
Think about your research topic and working question. What disciplines are likely to publish about your topic?
For each discipline, consider:
- What is important about the topic to scholars in that discipline?
- What is most likely to be the focus of their study about the topic?
- What perspective are they most likely to have on the topic?
Periodicals
First, let’s discuss periodicals. Periodicals include trade publications, magazines, and newspapers. While they may appear similar, particularly those that are found online, each of these periodicals has unique features designed for a specific purpose. Magazine and newspaper articles are usually written by journalists, are intended to be short and understandable for the average adult, contain color images and advertisements, and are designed as commodities to be sold to an audience. An example of a useful magazine for social workers is the New Social Worker.
Primary sources
Periodicals may contain primary or secondary literature depending on the article in question. An article that is a primary source gathers information as an event happens (e.g. an interview with a victim of a local fire), or it may relay original research reported on by the journalists (e.g. the Guardian newspaper’s The Counted webpage which tracked how many people were killed by police officers in the United States from 2015-2016).[30] Primary sources are based on raw data, which we discussed in Chapter 2.
Is it okay to use a magazine or newspaper as a source in your research methods class? If you were in my class, the answer is “probably not.” There are some exceptions, such as the Guardian page mentioned above (which is a great example of data journalism) or breaking news about a policy or community, but most of what newspapers and magazines publish is secondary information. Researchers should look for primary sources—the original source of information—whenever possible.
Secondary sources
Secondary sources interpret, discuss, and summarize primary sources. Examples of secondary sources include literature reviews written by researchers, book reviews, as well as news articles or videos about recently published research. Your job in this course is to read the original source of the information, the primary source. When you read a secondary source, you are relying on another's interpretation of the primary source. That person might be wrong in how they interpret the primary source, or they may simply focus on a small part of what the primary source says.
Most often, I see the distinction between primary and secondary sources in the references pages students submit for the first few assignments in my research class. Students are used to citing the summaries of scientific findings from periodicals like the New York Times or Washington Post. Instead, students should read the journal article written by the researchers, as it is the primary source. Journalists are not scientists. If you have seen articles about how chocolate cures cancer or how drinking whiskey can extend your life, you should understand how journalists can exaggerate or misinterpret scientific results. See this cartoon from Saturday Morning Breakfast Cereal for a hilarious illustration of sensationalizing findings from scientific research in journalism. Even when news outlets do a very good job of interpreting scientific findings, and that is certainly what most journalistic outlets strive for, you are still likely to only get a short summary of the key results.
When you are getting started with research, newspaper and magazine articles are excellent places to commence your journey into the literature, as they do not require specialized knowledge and may inspire deeper inquiry. Just make sure you are reading news, not opinion pieces, which may exclude facts that contradict the author's viewpoint. As we will talk about in the next chapter, the one secondary source that is highly valuable for student researchers are review articles published in academic journals, as they summarize much of the relevant literature on a topic.
Trade publications
Unlike magazines and newspapers, trade publications may take some specialized knowledge to understand. Trade publications or trade journals are periodicals directed to members of a specific profession. They often include information about industry trends and practical information for people working in the field. Because of this, trade publications are somewhat more reputable than newspapers or magazines, as the authors are specialists in their field.
NASW News, published by the National Association of Social Workers (NASW), is a good example of a trade publication in social work. Its intended audience is social work practitioners who want to learn about important practice issues. They report news and trends in a particular field but not scholarly research. They may also provide product or service reviews, job listings, and advertisements.
So, can you use trade publications in a formal research proposal? Again, if you’re in my class, the answer would be “probably not.” A main shortcoming of trade publications is the lack of peer review. Peer review refers to a formal process in which other esteemed researchers and experts ensure your work meets the standards and expectations of the professional field. Peer review is part of the cycle of publication for journal articles and provides a gatekeeper service, so to speak, which (in theory) ensures that only top-quality articles are published.
In reality, there are a number of problems with our present system of peer review. Briefly, critiques of the current peer review system include unreliable feedback across reviewers; inconsistent error and fraud detection; prolonged evaluation periods; and biases against non-dominant identities, perspectives, and methods. It can take years for a manuscript to complete peer review, likely with one or more rounds of revisions between the author and the reviewers or editors at the journal. See Kelly and colleagues (2014)[31] for an overview of the strengths and limitations of peer review.
A trade publication isn't as reputable as a journal article because it does not include expert peer review. While trade publications do contain a staff of editors, the level of review is not as stringent as with academic journals. On the other hand, if you are completing a study about practitioners, then trade publications may be highly relevant sources of information for your proposal.
In summary, newspapers and other popular press publications are useful for getting general topic ideas. Trade publications are useful for practical application in a profession and may also be a good source of keywords for future searching.
Journal articles
As you’ve probably heard by now, academic journal articles are considered to be the most reputable sources of information, particularly in research methods courses. Journal articles are written by scholars with the intended audience of other scholars (like you!). The articles are often long and contain extensive references that support the author's arguments. The journals themselves are often dedicated to a single topic, like violence or child welfare, and include articles that seek to advance the body of knowledge about a particular topic. We included a list of the 25 most popular social work journals in Chapter 1.
Peer review
Most journals are peer-reviewed (also known as refereed), which means a panel of scholars reviews the articles to provide a recommendation on whether they should be accepted into that specific journal. Scholarly journals make available articles of interest to experts or researchers. An editorial board of respected scholars (i.e., peers) reviews all articles submitted to a journal. Editors and volunteer reviewers decide if the article provides a noteworthy contribution to the field and should be published. For this reason, journal articles are the main source of information for researchers as well as for literature reviews. Usually, peer review is done confidentially, with the reviewer's name hidden from the author and vice versa. This confidentiality is mediated by the editorial staff at an academic journal and is termed blinded review, though this ableist language should be revised to confidential review (Ades, 2020).[32]
You can usually tell whether a journal is peer reviewed by going to its website. Under the “About Us” or "Author Submission" sections, the website should list the procedures for peer review. If a journal does not provide such information, you may have found a “predatory journal.” You may want to check with your professor or a librarian, as the websites for some reputable journals are not straightforward. It is important not to use articles from predatory journals. These journals will publish any article—no matter how bad it is—as long as the author pays them. If a journal appears suspicious and may be a predatory publication, search for it in the COPE member list. Another predatory publishing resource, Beall's List, has been critiqued as discriminatory towards some open access publishing models and non-Western publishers so we do not recommend its use (Berger & Cirasella, 2015).[33]
Even reputable, non-predatory journals publish articles that are later shown to be incorrect, unethically manipulated, or otherwise faulty. In Chapter 1, we talked about Andrew Wakefield's fabricated data linking autism and vaccines. His study was published in The Lancet, one of the most reputable journals in medicine, who formally retracted the article years later. Retraction Watch is a scholarly project dedicated to highlighting when scientific papers are withdrawn and why. It is important to note that peer review is not a perfect process, and that it is subject to error. There is a growing movement to make the peer review process faster and more transparent. For example, 17 life science journals moved to a model of publishing the unreviewed draft, the reviewer's comments, and how they informed the final draft (Brainard, 2019).[34]
Seminal articles
Peers don't just review work prior to publication. The whole point of publishing a journal article is to ensure that your peers will read it and use it in their work. A seminal article is “a classic work of research literature that is more than 5 years old and is marked by its uniqueness and contribution to professional knowledge” (Houser, 2018, p. 112).[35] Basically, it is a really important article. This article by Hodge, Lacasse, and Benson (2012)[36] reviews the most cited social work publications from the previous decade. These would all be considered seminal articles.
There are far more than just these 100 seminal articles in the literature. How do you know if you are looking at a seminal article? Seminal articles are cited a lot in the literature. You can see how many authors have cited an article using Google Scholar’s citation count feature when you search for the article, as depicted in Figure 3.1. Generally speaking, articles that have been cited more often are considered more reputable. There is nothing wrong with citing an article with a low citation count, but it is an indicator that not many other scholars have found the source to be useful or important, at least not yet.
Figure 3.1 shows a citation count for an article that is obviously seminal, with over 4,000 citations. There is no exact number of citations at which an article is considered seminal. A low citation count for an article published last year is not a reliable indicator of the importance of this article to the literature, as it takes time for people to read articles and incorporate them into their scholarship. For example, if you are looking at an article from five years ago with 20 citations, that's still a good quality resource to use. The purpose here is to recognize when articles are clearly influential in the literature, and are therefore more likely to be important to your review of the literature in that topic area.
Empirical articles
Journal articles can fall into one of several categories. Empirical articles report the results of a quantitative or qualitative analysis of raw data conducted by the author. Empirical articles also integrate theory and previous research. However, just because an article includes quantitative or qualitative results does not necessarily mean it is an empirical journal article. Since most articles contain a literature review with findings from empirical studies, you need to make sure the findings reported in the study stem from the author’s analysis of raw data.
Fortunately, empirical articles follow a similar structure, and include the following sections: introduction, methods, results, and discussion—in that order. While the exact wording in the headings may differ slightly from publication to publication and other sections may be added, this general structure applies to nearly all empirical journal articles. If you see this general structure, you are reading an empirical journal article. You can see this structure in a recent article from the open access social work journal, Advances in Social Work. Bonnano & Vesalak (2019)[37] conducted interviews of parents of children who are diagnosed with mental health issues, uncovering that trust was central to which sources parents used to inform themselves about their child's diagnosis. While there is no "Introduction" heading, the article begins on page 397 with the abstract and introduction. The "Methods" heading appears on page 400, after the introduction ends. The "Results" section begins on page 402. Because this is a qualitative empirical article, the results section is full of themes and quotes based on what participants said in their interviews with researchers. If it were a quantitative empirical article, like this one from Kelly, Kubart, and Freed (2020)[38] evaluating the impact of a social work intervention on depression in teenagers, the results section would be full of statistical data, tables, and figures that indicated mathematical relationships between the key variables in the research question. The "Discussion" for the Bonnano & Vesalak begins on page 407, and it contextualizes the findings in their "Results" section within the scholarly literature.
While this article uses the traditional headings, you may see other journal articles that use headings like "Background" instead of "Introduction" or "Findings" instead of "Results". You may also find headings like "Conclusions", "Implications", "Limitations", and so forth in other journal articles. Regardless of the specific wording, empirical journal articles follow the structure of introduction, methods, results, and discussion. If it has a methods and results section, it's almost certainly an empirical journal article. However, if the article does not have these specific sections, it's likely you've found a non-empirical article because the author did not analyze any raw data.
Review articles
Along with empirical articles, review articles are the most important type of article you will read as part of conducting your review of the literature. We will discuss their importance more in Chapter 4, but briefly, review articles are journal articles that summarize the findings other researchers and establish the state of the literature in a given topic area. They are so helpful because they synthesize a wide body of research information in a (relatively) short journal article. Review articles include literature reviews, like this one from Reynolds & Bacon (2018)[39] which summarizes the literature on integrating refugee children in schools. You may find literature reviews with a special focus like a critical review of the literature, which may apply a perspective like critical theory or feminist theory while surveying the scholarly literature on a given topic.
Review articles also include systematic reviews, which are like literature reviews but more clearly specify the criteria by which the authors search for and include scholarly literature in the review (Uman, 2011).[40] For example, this recent systematic review by Leonard, Hafford-Letchfield, and Couchman (2018)[41] discusses how to use strategies from arts education to inform social work education. As you can see in Figure 1 of this article, systematic reviews are very specific about which articles they include in their analysis. Because systematic reviews try to address all scholarly literature published on a given topic, researchers specify how the literature search was conducted, how many articles were included or excluded, and the reasoning for their decisions. This way, researchers make sure there is no relevant source excluded from their analysis.
The two final kinds of review articles, meta-analysis and meta-synthesis go even further than systematic reviews in that they analyze the raw data from all of the articles published in a given topic area, not just the published results. A meta-analysis is a study that combines raw data from multiple sources and analyzes the pooled data using statistics. Meta-analyses are particularly helpful for intervention studies, as it will pool together the raw data from multiple samples and studies to create a super-study of thousands of people, which has greater explanatory power. For example, this recent meta-analysis by Kennedy and colleagues (2016)[42] analyzes pooled data from six separate studies on parent-child interaction therapy to see if it is effective.
A meta-synthesis is similar to a meta-analysis but it pools qualitative results from multiple studies for analysis. For example, this meta-synthesis by Hodge and Horvath (2011)[43] pooled data from 11 qualitative studies that addressed spiritual needs in healthcare settings. While meta-analyses and meta-syntheses are the most methodologically robust type of review article, any recently published review article that is highly relevant to your topic area is a good place to start reading literature. Because review articles synthesize the results of multiple studies, they can give you a broad sense of the overall literature on your topic. We'll review a few kinds of non-empirical articles next.
Theoretical articles
Theoretical articles discuss a theory, conceptual model, or framework for understanding a problem. They may delve into philosophical or ethical analysis as well. For example, this theoretical article by Carrillo & Grady (2018)[44] discusses structural social work theory and anti-oppressive practice approaches. While most students know they need to have statistics to back up what they think is true, it's also important to use theory to inform what you think about your topic. Theoretical articles can help you understand how to think about a topic and may help you make sense of the results of empirical studies.
Practical articles
Practical articles describe “how things are done" from the perspective of on-the-ground practitioners. They are usually shorter than other types of articles and are intended to inform practitioners of a discipline on current issues. They may also reflect on a “hot topic” in the practice domain, a complex client situation, or an issue that may affect the profession as a whole. This practical article by Curry-Stevens and colleagues (2019)[45] was written by social work faculty for other social work faculty. It helps faculty learn how to increase the number of graduate social work students concentrating in macro practice, based on what faculty in one program did.
Type of journal article | How do you know if you're looking at one? | Why is this type of article useful? |
Peer-reviewed | Go to the journal's website, and look for information that describes peer review. This may be in the instructions for authors or about the journal sections. | To ensure other respected scholars have reviewed, provided feedback, and approved this article as containing honest and important scientific scholarship. |
Article in a predatory journal | Use the SIFT method. Search for the publisher and journal on Wikipedia and in a search engine. Check the COPE member list to see if your journal appears there. | They are not useful. Articles in predatory journals are published for a fee and have not undergone serious review. They should not be cited. |
*Not an journal article* | Does not provide the name of the journal it is published in. You may want to google the name of the article or report and its author to see if you can find a published version. | Journal articles aren't everything, but if your instructor asks for one, make sure you haven't mistakenly use dissertations, theses, and government reports. |
Empirical article | Most likely, if it has a methods and results section, it is an empirical article. | Once you have a more specific idea in mind for your study, you can adapt the measures, design, and sampling approach of empirical articles (quantitative or qualitative). The introduction and results section can also provide important information for your literature review. Make sure not to rely too heavily on a single empirical study, though, and try to build connections across studies you read. |
Quantitative article | The methods section contains numerical measures and instruments, and the results section provides outcomes of statistical analyses. | |
Qualitative article | The methods section describes interviews, focus groups, or other qualitative techniques. The results section has a lot of quotes from study participants. | |
Review article | The title or abstract includes terms like literature review, systematic review, meta-analysis, or meta-synthesis. | Getting a comprehensive but generalized understanding of the topic and identifying common and controversial findings across studies. |
Theoretical article | The articles does not have a methods or results section. It talks about theory a lot. | Developing your theoretical understanding of your topic. |
Practical articles | Usually in a section at the front or back of the journal, separate from full articles. It addresses hot topics in the profession from the standpoint of a practitioner. | Identifying how practitioners in the real world think about your topic. |
Book reviews, notes, and commentary | Much shorter, usually only a page or two. Should state clearly what kind of note or review they are in the title, abstract, or keywords. | Can point towards sources that provide more substance on the topic you are studying or point out emerging or controversial aspects of your topic. |
What kind of articles should you read?
No one type of article is better than the other, as each serves a different purpose. Seminal articles relevant to your topic area are important to read because the ideas in them are deeply influential in the literature. Theoretical articles will help you understand the social theory behind your topic. Empirical articles should test those theories quantitatively or create those theories qualitatively, a process we will discuss in greater detail in Chapter 8. Review articles help you get a broad survey of the literature by summarizing the results of multiple studies, which is particularly important at the beginning of a literature search. And finally, practical articles will help you understand a practitioner’s on-the-ground experience. Pick the kind of article that gives you the kind of information you need.
Other sources of information
As I mentioned previously, newspaper and magazine articles are good places to start your search (though they should not be the end of your search!). Another source students go to almost immediately is Wikipedia. Wikipedia is a marvel of human knowledge. It is a digital encyclopedia to which anyone can contribute. The entries for each Wikipedia article are overseen by skilled and specialized editors who volunteer their time and knowledge to making sure their articles are correct and up to date. That said, Wikipedia has been criticized for gender and racial bias as well as inequities in the treatment of the Global South (Montgomery et al., 2021).[46]
Encyclopedias
Wikipedia is an example of a tertiary source. We reviewed primary and secondary sources in the previous section. Tertiary sources review primary and secondary sources. Examples of tertiary sources include encyclopedias, directories, dictionaries, and textbooks like this one. Tertiary sources are an excellent place to start (but are not a good place to end) your search. A student might consult Wikipedia, the Open Education Sociology Dictionary, or the Encyclopedia of Social Work to get a general idea of a social work topic. Encyclopedias can also be excellent resources for their citations, and often the citations will be to seminal articles or important primary sources.
The difference between secondary and tertiary sources is not exact, and as we’ve discussed, using one or both at the beginning of a project is a good idea. As your study of the topic progresses, you will naturally have to transition away from secondary and tertiary sources and towards primary sources. We’ve already talked about one particular kind of primary source—empirical journal articles. We will spend more time on this primary source than any other in this textbook. However, it is important to understand how other types of sources can be used as well.
Books
Books contain important scholarly information. They are particularly helpful for theoretical, philosophical, and historical inquiry. For example, in my research on self-determination for individuals with intellectual and developmental disabilities, I needed to define and explore the concept of self-determination. I learned how to define it from the philosophical literature on self-determination and the advocacy literature contained in books. You can use books to learn key concepts, theories, and keywords you can use to find more up-to-date sources. They may also help you understand the scope and foundations of a topic and how it has changed over time.
Some books contain chapters that look like academic journal articles. These are called edited volumes. They contain chapters similar to journal articles, but the content is reviewed by an editor rather than peer reviewers. Edited volumes are considered somewhat less reputable than journal articles, as confidential peer review is still considered to be more objective than editorial review. However, articles in social science journals will often include references to books and edited volumes.
Conference proceedings and presentations
Conferences provide a great source of information. At conferences such as the Council on Social Work Education’s (CSWE) Annual Program Meeting (APM) or your state’s NASW conference, researchers present papers on their most recent research. The papers presented at conferences are sometimes published in a volume called a conference proceeding, though these are not common in social work. Conference proceedings highlight the current discourse in a discipline and can lead you to scholars who are interested in specific research areas. For a list of social work conferences, with a focus on the United States, see Chapter 24.
A word about conference papers: several factors may render these documents difficult to find. It is not unusual that papers delivered at professional conferences are not published in print or electronic form, although an abstract may be available. In these cases, the full paper may only be available from the author or authors. The most important thing to remember is that if you have any difficulty finding a conference proceeding or paper, ask a librarian for assistance.
Gray literature
Another source of information is the gray literature, which consist of research reports released by non-commercial publishers, such as government agencies, policy organizations, and think tanks. If you have already taken a policy class, perhaps you’ve come across the Center on Budget and Policy Priorities (CBPP). The CBPP is a think tank—a group of scholars that conducts research and advocates for political, economic, and social change. Similarly, students often find the Centers for Disease Control website helpful for understanding the prevalence of social problems like mental illness and child abuse. If you find fact sheets, news items, blog posts, or other content from organizations like think tanks or advocacy organizations in your topic area, look at the references cited to find the primary source, rather than citing the secondary source (the fact sheet). While you might be persuaded by their arguments, you want to make sure to account for the author's potential bias.
Think tanks and policy organizations often have a specific viewpoint they support. For example, there are conservative, liberal, and libertarian think tanks. Policy organizations may be funded by donors to push a given message to the public. Government agencies are generally more objective, though they may be less critical of government programs than other sources might be. Similarly, agencies may be less critical in the reports they share with the public. The main shortcoming of gray literature is the lack of peer review that is found in academic journal articles. Government reports are often cited in research proposals, along with some reports from policy organizations.
Dissertations and theses
Dissertations and theses can be rich sources of information and include extensive reference lists you can use to scan for resources. However, dissertations and theses are considered gray literature because they are not peer reviewed. The accuracy and validity of the document itself may depend on the school that awarded the doctoral or master’s degree to the author. Having completed a dissertation, I can attest to the length of time they take to write as well as read. If you come across a relevant dissertation, it is a good idea to read the literature review and note the sources the author used so you can incorporate them into your literature review. However, the data analysis found in these sources is considered less reputable as it has not passed through peer review yet. Dissertations are approved by committees of scholars, so they are considered more reputable than sources that lack review by experts prior to publication. Consider searching for journal articles by the dissertation's author to see if any of the results have been peer reviewed and published. You will also be thankful because journal articles are much shorter than dissertations and theses!
Webpages
The final source of information we are going to talk about is webpages. My graduate research focused on substance abuse and drugs, and I was fond of reading Drug War Rant, a blog about drug policy. It provided me with breaking news about drug policy and editorial opinion about the drug war. I still do the same thing with Twitter, as I follow policy organizations and noted scholars to learn about developments in my interest areas. Podcasts (like the Social Work Podcast or Doin' the Work) are also great sources of news, interviews, and long-form journalism. I would almost never cite a tweet or podcast in a research proposal, but they are excellent places to find primary sources I can read and cite.
Webpages will also help you locate professional organizations and human service agencies that address your problem. Looking at their social media feeds, reports, publications, or “news” sections on an organization’s webpage can clue you into important topics to study. Because anyone can develop their own webpage, they are usually not considered scholarly sources to use in formal writing, but may be useful as you are first learning about a topic. Additionally, many advocacy webpages will provide references for the facts they site, providing you with the primary source. Keep in mind too that webpages can be tricky to quote and cite, as they can be updated or moved at any time. While some reputable sources will include editing notes, others may not. Webpages can be updated without notice.
Use quality sources
We will discuss more specific criteria for evaluating the quality of the sources you choose in section 4.1. For now, as you access and read each source you encounter, remember:
All information sources are not created equal. Sources can vary greatly in terms of how carefully they are researched, written, edited, and reviewed for accuracy. Common sense will help you identify obviously questionable sources, such as tabloids that feature tales of alien abductions, or personal websites with glaring typos. Sometimes, however, a source’s reliability—or lack of it—is not so obvious. You should consider criteria such as the type of source, its intended purpose and audience, the author’s (or authors’) qualifications, the publication’s reputation, any indications of bias or hidden agendas, how current the source is, and the overall quality of the writing, thinking, and design. (Writing for Success, 2015, p. 448).[47]
While each of these sources provides an important facet of our learning process, your research should focus on finding academic journal articles about your topic. These are the primary sources of the research world. While it may be acceptable and necessary to use other primary sources—like books, government reports, or an investigative article by a newspaper or magazine—academic journal articles are preferred. Finding these journal articles is the topic of the next section.
Key Takeaways
- News articles, Wikipedia, and webpages are a good place to start learning about a new topic, but you should rely on more reputable and peer-reviewed sources when reviewing “the literature” on your topic.
- Researchers should read and cite primary sources, tracing a fact or idea to its original source. If you read a news article or webpage that talks about a study or report (i.e., a secondary source), read the primary source to get all the facts.
- Empirical articles report quantitative or qualitative analysis completed by the author on raw data.
- You must critically assess the quality and bias of any source you read. Just because it’s published doesn’t mean it’s true.
Exercises
Conduct a general Google search on your topic.
- Identify whether the first 10 sources that come up are primary or secondary sources.
- Optionally, ask a friend to conduct the same Google search and compare your results. Google's search engine customizes the results you see based on your browsing history and other data the company has collected about you. This is called a filter bubble, and you can learn more about it in this Ted Talk about filter bubbles.
Find a secondary or tertiary source about your topic.
- Trace one to the original, primary source.
Find a review article in your topic area.
- Identify which kind it is (literature review, systematic review, meta-analysis, etc.)
Find a think tank, advocacy organization, or website that addresses your topic.
- Using their information, identify primary sources that might be of use to you. Consider adding them to your social media feed or joining an email newsletter.
3.2 Identifying your information needs
Learning Objectives
Learners will be able to...
- Identify a a need for information knowledge about a subject area
- Identify a search topic question and define it using simple terminology
- Articulate current knowledge on a topic
- Recognize a need for information and data to achieve a specific end and define limits to the information need
- Identify the key concepts relevant to your research question and inquiry into the literature
Scenario
Norm Allknow was having trouble. He had been using computers since he was five years old and thought he knew all there was to know about them. So, when he was given an assignment to write about the impact of the Internet on society, he thought it would be a breeze. He would just write what he knew, and in no time the paper would be finished. In fact, Norm thought the paper would probably be much longer than the required ten pages. He spent a few minutes imagining how impressed his teacher was going to be, and then sat down to start writing.
He wrote about how the Internet had helped him to play online games with his friends, and to keep in touch with distant relatives, and even to do some homework once in a while. Soon he leaned back in his chair and looked over what he had written. It was just half a page long and he was out of ideas.
Identifying a Personal Need for Information
One of the first things you need to do when beginning any information-based project is to identify your personal need for information. This may seem obvious, but it is something many of us take for granted. We may mistakenly assume, as Norm did in the above example, that we already know enough to proceed. Such an assumption can lead us to waste valuable time working with incomplete or outdated information. Information literacy addresses a number of abilities and concepts that can help us to determine exactly what our information needs are in various circumstances. These are discussed below, and are followed by exercises to help develop your fluency in this area.
Information literacy skills are necessary because the landscape of scientific information is constantly evolving. New information is always added to what is known about your topic. Trained experts, informed amateurs, and opinionated laypeople are publishing in traditional and emerging formats; there is always something new to find out. The scale of information available varies according to topic, but in general it’s safe to say that there is more information accessible now than ever before.
Due to the extensive amount of information available, part of becoming more information literate is developing habits of mind and of practice that enable you to continually seek new information and to adapt your understanding of topics according to what you find. Because of the widely varying quality of new information, evaluation is also a key element of information literacy, and will be addressed in the next chapter of this book.
Finally, while you are busy searching for information on your current topic, be sure to keep your mind open for new avenues or angles of research that you haven’t yet considered. Often the information you found for your initial need will turn out to be the pathway to a rich vein of information that can serve as raw material for many subsequent projects.
When you understand the information environment where your information need is situated, you can begin to define the topic more clearly and you can begin to understand where your research fits in with related work that precedes it. Your information literacy skills will develop against this changing background as you use the same underlying principles to do research on a variety of topics.
What do you already know?
Part of identifying your own information need is giving yourself credit for what you already know about your topic. It can be helpful to visualize your information need using a chart. Construct a chart using the following format to list whatever you already know about the topic.
A simple tracking chart can be seen in Figure 3.2. In the first column, list what you know about your topic. In the second column, briefly explain how you know this (heard it from the professor, read it in the textbook, saw it on a blog, etc.). In the last column, rate your confidence in that knowledge. Are you 100% sure of this bit of knowledge, or did you just hear it somewhere and assume it was right?
When you’ve looked at everything you think you know about the topic and why, step back and look at the chart as a whole. How much do you know about the topic, and how confident are you about it? You may be surprised at how little or how much you already know, but either way you will be aware of your own background on the topic. This self-awareness is key to becoming more information literate.
This exercise gives you a simple way to gauge your starting point, and may help you identify specific gaps in your knowledge of your topic that you will need to fill as you proceed with your research. It can also be useful to revisit the chart as you work on your project to see how far you’ve progressed, as well as to double check that you haven’t forgotten an area of weakness.
Once you’ve clearly stated what you do know, it should be easier to state what you don’t know. Keep in mind that you are not attempting to state everything you don’t know. You are only stating what you don’t know in terms of your current information need. This is where you define the limits of what you are searching for. These limits enable you to meet both size requirements and time deadlines for a project. If you state them clearly, they can help to keep you on track as you proceed with your research. You can learn more about this in the Scope chapter of this book.
One useful way to keep your research on track is with a “KWHL” chart. This type of chart enables you to state both what you know and what you want to know, as well as providing space where you can track your planning, searching and evaluation progress. For now, just fill out the first column, but start thinking about the gaps in your knowledge and how they might inform your research questions. You will learn more about developing these questions and the research activities that follow from them as you work through this book.
Revising your working question
Once you have identified your own need for knowledge, investigated the existing information on the topic, and set some limits on your research based on your current information need, write out your research question or state your thesis. The next exercise will help you conceptualize your question and the potential answers you are considering.
You’ll find that it’s not uncommon to revise your question or thesis statement several times in the course of a research project. As you become more and more knowledgeable about the topic, you will be able to state your ideas more clearly and precisely, until they almost perfectly reflect the information you have found.
Exercises
- State your working question.
- Write your proposed answer to that question...as best you know right now.
- It doesn’t have to be perfect at this point, but based on your current understanding of your topic and what you expect or hope to find is the answer to the question you asked.
Look at your question and your thesis/hypothesis, and make a list of the terms common to both lists (excluding “the,” “and,” “a,” etc.). These common terms are likely the important concepts that you will need to research to support your thesis/hypothesis. They may be the most useful search terms overall or they may only be a starting point.
If none of the terms from your question and thesis/hypothesis lists overlap at all, you might want to take a closer look and see if your thesis/hypothesis really answers your research question. If not, you may have arrived at your first opportunity for revision. If your question asks about self-esteem and your hypothesis talks about self-concept, the answers you seek will not address your working question.
- Does your question really ask what you’re trying to find out?
- Does your proposed answer really answer that question?
You may find that you need to change one or both, or to add something to one or both to really get at what you’re interested in. This is part of the process, and you will likely discover that as you gather more information about your topic, you will find other ways that you want to change your question or thesis to align with the facts, even if they are different from what you hoped.
Defining a research question can be more difficult than it seems. Your initial questions may be too broad or too narrow. You may not be familiar with specialized terminology used in the field you are researching. You may not know if your question is worth investigating at all.
These problems can often be solved by a preliminary investigation of scholarly information on the topic. As previously discussed, gaining a general understanding of the information environment helps you to situate your information need in the relevant context and can also make you aware of possible alternative directions for your research. On a more practical note, however, reading through some of the existing information can also provide you with commonly used terminology, which you can then use to state your own research question, as well as in searches for additional information. Don’t try to reinvent the wheel, but rely on the experts who have laid the groundwork for you to build upon. Find good, argumentative writing from scholars, experts, and people with lived experience that you find insightful.
Key Takeaways
- Researchers continually revise their working question and the answers they are considering throughout the literature search process.
- Your existing knowledge on a topic is a resource you can rely upon, but you will be required as social work practitioners to show evidence that what you say is scientifically true.
- As you learn more about your topic (theories, key terms, history, etc.), you should revise your working question and the answers you are considering to it.
3.3 Searching basics
Learning Objectives
Learners will be able to...
- Brainstorm potential keywords and search queries for your topic
- Apply basic Boolean search operators to your search queries
- Identify relevant databases in which to search for journal articles
- Target your search queries in databases to the most relevant results
One of the drawbacks (or joys, depending on your perspective) of being a researcher in the 21st century is that we can do much of our work without ever leaving the comfort of our recliners. This is certainly true of familiarizing yourself with the literature. Most libraries offer incredible online search options and access to all of the literature you'll ever need on your topic.
Unfortunately, even though many college classes require students to find and use academic journal articles, there is little training for students on how to do so. The purpose of this section and the rest of part 1 of the textbook is to build information literacy skills, which are the basic building blocks of any research project. While we describe the process here as linear, with one step following neatly after another, in reality, you will likely go back to Step 1 often as your topic gets more specific and your working question becomes clearer. Recall our diagram in the last chapter, Figure 2.1. You are cycling between reformulating your working question then searching for and reading the literature on your topic.
The process of searching databases and revising your working question can be time-consuming, so it is a good idea to write notes to yourself. Remembering why you included certain search terms, searched different databases, or other decisions in the literature search process can help you keep track of what you've done. You can then recreate searches to download articles you may have missed or remember reflections you had weeks ago when you last conducted a literature search. When I search the literature, I create a "working document" in a word processor where I write notes and reflections.
Step 1: Build search queries using keywords
What do you type when you are searching for something on in a search engine like Google or Bing? Are you a question-asker? Do you type in full sentences or just a few keywords? What you type into a database or search engine is called a query. Well-constructed queries get you to the information you need more quickly, while unclear queries will force you to sift through dozens of irrelevant articles before you find the ones you want. The danger in being a question-asker or typing like you talk is that including too many extraneous words will confuse the search engine. In this section, you will learn to use keywords rather than natural language to find scholarly information.
Keywords are the words or phrases you use in your search query, and they inform the relevance and quantity of results you get. Unfortunately, different studies often use different terms to mean the same thing. A study may describe its topic as substance abuse, rather than addiction (for example). Think about what keywords are important to answering your working question. Are there other ways of expressing the same idea?
Often in social work research, there is a bit of jargon to become familiar with when crafting your search queries. If you wanted to learn more about individuals of low SES who do not have access to a bank account, you may need to learn the term “unbanked,” which refers to people without bank accounts, and thereby include the word “unbanked” in your search query. If you wanted to learn about children who take on parental roles in families, you may need to include the word “parentification” as part of your search query.
As graduate researchers, you are not expected to know these terms ahead of time. Instead, start with the keywords you already know. As you read more about your topic and become familiar with some of these words, begin including these keywords into your search results, as they may return more relevant hits.
Exercises
- List all of the keywords you can think of that are relevant to your working question. Use your list of keywords from the previous section as a starting point.
- Add new keywords to this list as you search the literature and learn more about your topic.
- Start a document in which you can put your keywords as well as your notes and reflections on the search process.
Alternate keywords
think of synonyms or related terms for each concept. If you do this, you will have more flexibility when searching in case your first search term doesn’t produce any or enough results. This may sound strange, since if you are looking for information using a Web search engine, you almost always get too many results. Databases, however, contain fewer items, and having alternative search terms may lead you to useful sources. Even in a search engine like Google, having terms you can combine thoughtfully will yield better results.
The following worksheet is an example of a process you can use to come up with search terms. It illustrates how you might think about the topic of violence in high schools. Notice that this exact phrase is not what will be used for the search. Rather, it is a starting point for identifying the terms that will eventually be used.
Now, use a clean copy of the same worksheet to think about the topic Sarah’s team is working on. How might you divide their topic into concepts and then search terms? Keep in mind that the number of concepts will depend on what you are searching for. And that the search terms may be synonyms or narrower terms. Occasionally, you may be searching for something very specific, and in those cases, you may need to use broader terms as well. Jot down your ideas then compare what you have written to the information on the second, completed worksheet and identify 3 differences.
Boolean searching: Learn to talk like a robot
Google is a “natural language” search engine, which means it tries to use its knowledge of how people talk to better understand your query. Google’s academic database, Google Scholar, utilizes that same approach. However, other databases that are important for social work research—such as Academic Search Complete, PsycINFO, and PubMed—will not return useful results if you ask a question or type a sentence or phrase as your search query. Instead, these databases are best used by typing in keywords. Instead of typing “the effects of cocaine addiction on the quality of parenting,” you might type in “cocaine AND parenting” or “addiction AND child development.” [Note: you would not actually use the quotation marks in your search query for the examples in this subsection].
These operators (AND, OR, NOT) are part of what is called Boolean searching. Boolean searching works like a simple computer program. Your search query is made up of words connected by operators. Searching for “cocaine AND parenting” returns articles that mention both cocaine and parenting. There are lots of articles on cocaine and lots of articles on parenting, but fewer articles on that address both of those topics. In this way, the AND operator reduces the number of results you will get from your search query because both terms must be present.
The NOT operator also reduces the number of results you get from your query. For example, perhaps you wanted to exclude issues related to pregnancy. Searching for “cocaine AND parenting NOT pregnancy” would exclude articles that mentioned pregnancy from your results. Conversely, the OR operator would increase the number of results you get from your query. For example, searching for “cocaine OR parenting” would return not only articles that mentioned both words but also those that mentioned only one of your two search terms. This relationship is visualized in Figure 3.2 below.
Exercises
- Build a simple Boolean search query, appropriately using AND, OR, and NOT.
- Experiment with different Boolean search queries by swapping in different keywords.
- Write down which Boolean queries you create in your notes.
Step 2: Find the right databases
Queries are entered into a database or a searchable collection of information. In research methods, we are usually looking for databases that contain academic journal articles. Each database has a unique focus and presents distinct advantages or disadvantages. It is important to try out multiple databases, and experiment with different keywords and Boolean operators to find the most relevant results for your working question.
Google Scholar
The most commonly used public database of scholarly literature is Google Scholar, and there are many benefits to using it. Most importantly, Google Scholar is free and is the only database to which you will have access after you graduate. Building competence at using it now will help you as you engage in the evidence-based practice process post-graduation. Because Google Scholar is a natural language search engine, it can seem less intimidating. However, Google Scholar also understands Boolean search terms and contains some advanced searching features like searching within one specific author's body of work, one journal, or searching for keywords in the title of the journal. We have previously mentioned the 'Cited By' field which counts the number of other publications that cite the article and a link to explore and search within those citing articles. This can be helpful when you need to find more recent articles about the same topic.
I recommend linking your personal Google account and your university login which provides one-click access to journal articles available at your institution. I also often use their citation generator, though I have to fix a few things on most entries before using them in my references. You can also set email alerts for a specific search query, which can help you stay current on the literature over time, and organize journal articles into folders. If these advanced features sound useful to you, consult Google Scholar Search Tips for a walkthrough.
While these features are great, there are some limitations to Google Scholar. The sources in Google Scholar are not curated by a professional body like the American Psychological Association (who curates PsycINFO). Instead, Google crawls the internet for what it thinks are scholarly works and indexes them in their database. When you search in Google Scholar, the sources will be not only journal articles, but books, government and foundation reports, gray literature, graduate theses, and articles that have not undergone peer review. There is no way to select for a specific type of source, and you need to look clearly and identify what kind of source you are reading. Not every source on Google Scholar is reputable or will match what you need for an assignment.
With that broad focus, Google Scholar ranks the results from your query by the number of times each source was cited by other scholars and the number of times your keywords are mentioned in the source. This biases the search results towards older, more seminal articles. So, you should consider limiting your results to only those published the last few years. Unfortunately, Google Scholar also lacks advanced searching features of other databases, like searching by subject area or within the abstract of an article.
Exercises
- Type your queries into Google Scholar. See which queries provide you with the most relevant results.
- Look for new keywords in the results you find, particularly any jargon used in that topic area you may not have known before.
- Write down notes on which queries and keywords work best. Reflect on what you read in the abstracts and titles of the search results in Google Scholar.
Databases accessed via an institution
Although learning Google Scholar is important, you should take advantage of the databases to which your institution purchases access. The first place to look for social work scholarship are two databases: Social Service Abstracts and Social Work Abstracts, as these databases are indexed specifically for the discipline of social work. As you are aware, social work is interdisciplinary by nature, and it is a good idea not to limit yourself solely to the social work literature on your topic. If your study requires information from the disciplines of psychology, education, or medicine, you will likely find the databases PsycINFO, ERIC, and PubMed (respectively) to be helpful in your search.
There are also less specialized databases that contain articles from across social work and other disciplines. Academic Search Complete is similar to Google Scholar in its breadth, as it contains a number of smaller databases from a variety of social science disciplines. Within Academic Search Complete, click Choose Databases to select databases that apply to different disciplines or topic areas.
It is worth mentioning that many university libraries have a meta-search engine which searches some of the databases to which they have access, including both disciplinary and interdisciplinary databases. Meta-search engines are fantastic, but you must be careful to narrow your results down to the type of resource you are looking for, as they will include all types of media (e.g., books, movies, games). Unfortunately, not every database is included in these meta-search engines, so you will want to try other databases, as well.
You can find the databases we mentioned here and others on the Databases page of your university library (at my institution, the page is titled A-Z Databases). A university login is required to for you to access these databases because you pay for access as part of the tuition and fees at your university. This means that sometime after graduation, you will lose access to these databases and would need to physically travel to a university library to access them. Make the most of your database access during your graduate degree program. We will review in Chapter 4 how to get around paywalls after you graduate.
The databases we described here can be a bit more intimidating, as they have limited natural language support and rely mostly on Boolean searching. However, they contain more advanced search features which can help you narrow down your search to only the most relevant sources, our next step in the research process.
Exercises
- Explore different databases you can access via your university's library website and searching using your keywords.
- Write notes on which databases and keywords provide you with the most relevant results and the disciplines you are likely to find in each database.
- Look for any new keywords in your results that will help you target your search, and experiment with new search queries.
Step 3: Reduce irrelevant search results
At this point, you should have worked on a few different search queries, swapped in and out different keywords, and explored a few different databases of academic journal articles relevant to your topic. Next, you have to deal with the most common frustration for students: narrowing down your search and reducing the number of results you have to look at. I'm sure at some point you've typed a query into Google Scholar or your library's search engine and seen hundreds of thousands or millions of sources in your search results. Don’t worry! You don’t have to read all those articles to know enough about your topic area to produce a good research project.
While reading millions of articles is clearly ridiculous, there is no magic number of search results to reach. A good search will return results which are highly relevant. In my experience, student search queries tend to be too general (and have too many irrelevant results), rather than too specific (and have too few relevant results). Don't worry! You will not need to read all of the articles you find, but reducing the number of results will save you time by eliminating irrelevant articles.
You should have two goals in reducing the number of results:
- Reduce the number of sources until you could reasonably skim through the title and abstract of each source. Generally, you want a hundred or a thousand results, rather than a hundred thousand results.
- Reduce the number of irrelevant sources in your search results until you are much more likely to encounter relevant, rather than irrelevant articles. If only one of every ten results in your search is relevant to your topic for example, you are wasting time. You would be better served by using the tips below to better target your search query.
Here are some tips for reducing the number of irrelevant sources when you search in databases.
- Use quotation marks to indicate exact phrases, like “mental health” or “substance abuse.”
- Search for your keywords in the ABSTRACT. A lot of your results may be from articles about irrelevant topics that mention your search term only once. If your topic isn’t in the abstract, chances are the article isn’t relevant. You can be even more restrictive and search for your keywords in the TITLE, but that may be too restrictive, and exclude relevant articles with titles that use slightly different phrasing. Academic databases provide these options in their advanced search tools.
- Use SUBJECT headings to find relevant articles. These tags are added by the organization that provides the database, like the American Psychological Association who curates PyscINFO, to help classify and categorize information for easier browsing.
- Narrow down the years of publication. Unless you are gathering historical information about a topic, you are unlikely to find articles older than 10-15 years to be useful. They no longer tell you the current knowledge on a topic. All databases have options to narrow your results down by year. Check with your professor to see if they have specific guidelines for when an article is "too old."
- Talk to a librarian. They are professional knowledge-gatherers, and there is often a librarian assigned to your department. Their job is to help you find what you need to know, and they are extensively trained in how to help you!
- Talk to your stakeholders or someone knowledgeable about your target population. They have lived experience with your topic which can help you understand the literature through a different lens. Moreover, the words they use to describe their experiences can also be useful sources of keywords, theories, studies, or jargon.
Exercises
- Using the techniques described in this subsection, reduce the number of irrelevant results in the database queries you have performed so far. Pay particular attention to searching within the abstract, using quotation marks to indicate keyword phrases, and exploring subject headings relevant to your topic.
- Reduce your database query results down to a number (a) where you could reasonably skim through the titles and abstracts to identify relevant articles and (b) where you are much more likely to encounter relevant, rather than irrelevant articles. Repeat this process for your search of each database relevant to your project.
- Write down your queries or save them, so you can recreate them later if you need to return to them. Also, write down any reflections you have on the search process.
- Look for any new keywords in your results that will help you target your search better, and experiment with new search queries.
Step 4: Conduct targeted searches
Another way to save time when searching for literature is to look for articles that synthesize the results of other articles: review articles. Systematic reviews provide a summary of the existing literature on a topic. If you find one on your topic, you will be able to read one person’s summary of the literature and go deeper by reviewing and reading articles they cited in their references. Other types of reviews such as critical reviews may offer a viewpoint along with a birds-eye view of the literature.
Similarly, meta-analyses and meta-syntheses have long reference lists that are useful for finding additional sources on a topic. They use data from each article to run their own quantitative or qualitative data analysis. In this way, meta-analyses and meta-syntheses provide a more comprehensive overview of a topic. To find these kinds of articles, include the term “meta-analysis,” “meta-synthesis,” or “systematic review” as keywords in your search queries.
My advice is to read a review article first, before any type of article. The purpose of review articles is to summarize an entire body of literature is as short an article as possible. This is exactly what students who are learning a new area need. Time is a precious resource for graduate students, and review articles provide the most knowledge in the shortest time. Even if they are a few years old, they should help you understand what else you need to find in your literature search.
As you look through abstracts of articles in your search results, you should begin to notice that certain authors keep appearing. If you find an author that has written multiple articles on your topic, consider searching the AUTHOR field for that particular author. You can also search the web for that author’s Curriculum Vitae or CV (an academic resume) that will list their publications. Many authors maintain personal websites or host their CV on their university department’s webpage. Just type in their name and “CV” into a search engine. For example, you may find Michael Sherraden’s name often if your search query is about assets and poverty. You can find Michael Sherraden's CV on the Washington University of St. Louis website.
A similar process can be used for journal names. As you are going through search results, you may also notice that many of the articles you’ve skimmed come from the same journals. Searching with that journal name in the JOURNAL field will allow you to narrow down your results to just that journal. For example, if you are searching for articles related to values and ethics in social work, you might want to search for articles in the Journal of Social Work Values and Ethics. You can do so within any database that indexes the journal, like Google Scholar, or through the search feature journal’s webpage. Browse the abstracts and download the full-text PDF of any article you think is relevant.
Step 5: Explore references and citations
The last thing you'll need to do in order to make sure you didn't miss anything is to look at the references in each article you cite. If there are articles that seem relevant to you, you may want to download them. Unfortunately, references may contain out-of-date material and cannot be updated after publication. As a result, a reference section for an article published in 2014 will only include references from pre-2014.
To find research published after 2014 on the same topic, you can use Google Scholar’s 'Cited By' feature to do a future-looking search. Look up your article on Google Scholar and click the 'Cited By' link. The results are a list of all the sources that cite the article you just read. Google Scholar also allows you to search within these articles—check the box below the search bar to do so—and it will search within the 'Cited By' articles for your keywords.
Exercises
- Examine the search results from various databases and look for authors and journals that publish often on your topic. Conduct targeted searches of these authors and journals to make sure you are not missing any relevant sources.
- Identify at least one relevant article, preferably a review article. Look in the references for any other sources relevant to your working question. Search for the article in Google Scholar and use the the 'Cited By' feature to look for additional sources relevant to your working question.
- Search for review articles that will help you get a broad sense of the literature. Depending on your topic, you may also want to look for other types of articles as well.
- Look for any new keywords in your results that will help you target your search, and experiment with new search queries.
- Write notes on your targeted searches, so you can recreate them later and remember your reflections on the search process.
Getting the right results
I read a lot of literature review drafts of that say "this topic has not been studied in detail." Before you come to that conclusion, make an appointment with a librarian. It is their job to help you find material that is relevant to your academic interests. My first project as a graduate research assistant involved writing a literature review for a professor, and the time I spent working with a librarian on that project brought home how important and impactful they can be. But there is a lot you can do on your own!
Let’s walk through an example. Imagine a local university wherein smoking was recently banned, much to the chagrin of a group of student smokers. Students were so upset by the idea that they would no longer be allowed to smoke on university grounds that they staged several smoke-outs during which they gathered in populated areas around campus and enjoyed a puff or two. Their activism was surprising. They were advocating for the freedoms of people committing a deviant act—smoking—that is highly disapproved of. Were the protesters otherwise politically active? How much effort and coordination had it taken to organize the smoke-outs?
The student researcher began their research by attempting to familiarize themselves with the literature. They started by looking to see if there were any other student smoke-outs using a simple Google search. When they turned to the scholarly literature, their search in Google Scholar for “college student activist smoke-outs,” yielded no results. Concluding there was no prior research on their topic, they informed the professor that they would not be able to write the required literature review since there was no literature for them to review. What went wrong here?
The student had been too narrow in their search for articles in their first attempt. They went back to Google Scholar and searched again using queries with different combinations of keywords. Rather than searching for “college student activist smoke-outs” they tried, “college student activism," "smoke-out," "protest," and other keywords. This time their search yielded many articles. Of course, they were not all focused on pro-smoking activist efforts, but the results addressed their population of interest, college students, and on their broad topic of interest, activism. Experimenting with different keywords across different databases helped them get a comprehensive and multi-disciplinary view of the topic.
Reading articles on college student activism might give them some idea about what other researchers have found in terms of what motivates college students to become involved in activist efforts. They could also play around with their search terms and look for research on activism centered on other sorts of activities that are perceived by some to be deviant, such as marijuana use or veganism. In other words, they needed to broaden their search about college student activism to include more than just smoke-outs and understand the theory and empirical evidence around student activism.
Once they searched for literature about student activism, they could link it to their specific research focus: smoke-outs organized by college student activists. What is true about activism broadly is not necessarily true of smoke-outs, as advocacy for smokers is unique from advocacy on other topics like environmental conservation. Engaging with the sociological literature about their target population, college students who smoke cigarettes, for example, helped to link the broader literature on advocacy to their specific topic area.
Revise your working question often
Throughout the process of creating and refining search queries, it is important to revisit your working question. In this example, trying to understand how and why the smoke-out took place is a good start to a working question, but it will likely need to be developed further to be more specific and concrete. Perhaps the researcher wants to understand how the protest was organized using social media and how social media impacts how students perceived the protests when they happened. This is a more specific question than "how and why did the smoke-out take place?" though you can see how the researcher started with a broad question and made it more specific by identifying one aspect of the topic to investigate in detail. You should find your working question shifting as you learn more about your topic and read more of the literature. This is an important sign that you are critically engaging with the literature and making progress. Though it can often feel like you are going in circles, there is no shortcut to figuring out what you need to know to study what you want to study.
Key Takeaways
- The quality of your search query determines the quality of your search results. If you don't work on your queries, you’ll get a million results, only a small percentage of which are relevant to your project.
- Keywords and queries should be updated as you learn more about your topic.
- Your two goals in targeting your database searches should be reducing the number of results for each search query to a number you can feasibly skim and making those results more relevant to your project.
- Techniques to increase the number of relevant results in a search include using Boolean operators, quotation marks, searching in the abstract or title, and using subject headings.
- Use the databases that are most relevant to your topic. For general searches, Google Scholar has some strengths (ease of use, Cited By) and limitations (cannot search in abstract, includes resources other than journal articles). You can access other databases through your university's library.
- Don’t be afraid to reach out to a librarian or your professor for help with searching. It is their job to help you.
Exercises
- Reflect on your working question. Consider changes that would make it clearer and more specific based on the literature you have skimmed during your search.
- Describe how your search queries (across different databases) address your working question and provide a comprehensive view of the topic.
Chapter Outline
- Evaluating sources (18 minute read)
- Organizing information (18 minute read)
- Critical information literacy (14 minute read)
Content warning: examples in this chapter contain references to school discipline, mental health, gender-based discrimination, police shootings, ableism, autism and anti-vaccination conspiracy theories, children’s mental health, child abuse, poverty, substance use disorders and parenting/pregnancy, tobacco use, neocolonialism and Western hegemony, and COVID-19.
4.1 Evaluating sources
Learning Objectives
Learners will be able to...
- Use skimming to identify which articles are most relevant to your topic
- Overcome paywalls that block access to scholarly information
- Assess the reputability of resources by looking for bias and rigor
- Identify why review articles are helpful to read at the beginning of a project
- Revise their working question and overall project based on knowledge they learn from the scholarly literature
In the previous section, we discussed how to formulate search queries to get the most relevant results. At this point, you should not be staring at a Google Scholar window with 1,000,000 search results. If you haven't played around in multiple databases and refined your queries using the suggestions in section 3.2, pause here and spend some time working on your search queries. Hopefully, you can find at least a few different queries that provide relevant resources to help you answer your working question or introduce new ideas that might revise or update your working question. Remember that your working question should be revised and updated as you learn more about your topic by searching in the literature.
Skim abstracts
All databases will give you access to an article's abstract. The abstract is a summary of the main points of an article. It will provide the purpose of the article and summarize the author's conclusions. Once you have a few good search queries, start skimming through abstracts and find the articles that are most relevant to your working question. Soon enough, you will find articles that are so relevant that you may decide to read the full text.
It is a good idea to cast a wide net at this point, since your project is just beginning. If you like the article, make sure to download the full text PDF to your computer so you can read it later. Save it in a new folder using a descriptive title. I like to include the first author's last name, year, and the first few words of the title since the file names assigned by journals are often unclear.
What do you do with all of those PDFs? I usually keep mine in folders on my cloud storage drive, arranged by topic. For those who are more ambitious, you may want to use a reference manager like Mendeley or Zotero, which can help keep your sources and notes organized. Both them can also help you build a proper bibliography and organize your notes on each article. At the very least, write some notes on paper or in a word processor reflecting on the articles you skim and how they might be of use in your study.
Getting through paywalls to the article
When students try to download the full text of an article, they will often hit an obstacle: the paywall. A paywall is when a publisher charges you to access a publication. Academic journal articles are expensive, but good news! Part of the tuition and fees your university charges you goes toward paying major publishers of academic journals for the privilege of accessing their articles. You should be able to access the full text for most of the journal articles you need using your library's website. Do not limit your literature search to the free articles or random PDFs you find on the internet!
Because journal publishers charge a lot of money for access to journals, your school likely does not pay for all the journals in the world. If your university does not already pay for access to the article, you still do not have to pay for it! You will need to use the interlibrary loan feature on your library's website. It is often listed under Services. Just enter the information for your article (e.g. author, publication year, title, DOI), and a librarian will get you the PDF of the article that you need from a school that does pay for access to that article.
After you no longer have a university library login, getting a PDF of a journal article becomes more challenging. You can subscribe to journals yourself, and many practitioners do that. However, you can always ask an author for a copy of their article. They will usually send it to you. Some journal articles are available completely for free. If you followed the links in section 3.1 to examples of different types of articles, all of these pointed to articles that were free to access, as the authors of this text cannot be sure that all readers of this textbook will have access to paywalled articles. Open access journal articles do not have a paywall so are open to the public, but less than 13% of journal articles are open access (Chimmer, Geschuhn, & Vogler, 2015).[48]
Another alternative way to access journal articles is if they are available in for-profit repositories like Social Science Research Network, Academia.edu, and ResearchGate. You will likely encounter results from these websites if you search for articles in a search engine like Google or Bing, as well as on Google Scholar. Sometimes, these websites require users to create an account in order to view content, and students should consider using a throwaway email to do so. ResearchGate has been sued by scholarly publishers for illegally hosting copyrighted and paywalled content (Mackenzie, 2018).[49] By contrast, non-profit and community-owned repositories like SocArXiv and university repositories (like Virginia Commonwealth University's Scholars Compass) allow authors to legally share a final draft of their work either prior to publication or after publication within the boundaries set out by publishers. This practice is called self-archiving or green open access—as contrasted with gold open access, in which the publisher's version of the journal article is freely available rather than just the author's draft. Self-archived papers look a lot like the papers you turn in for class. They are formatted according to the style guide predominant in the discipline and intended to be reviewed by experts as part of peer review.
We will address the social equity implications of paywalls and open access at the end of this chapter when we talk about information privilege. For now, we will pass along this advice adapted from the brilliant librarians at Virginia Tech on getting through paywalls. While you are in school, your library will likely pay for access to some of the articles you want. Search your library's collection, and if it does not have access to it right away, request it via inter-library loan, and a copy will come within a few days thanks to some dedicated librarians at your school. If you need the article right away or are not affiliated with a university library, the steps below might be helpful in finding your article.
- Check Google Scholar for alternative versions of the article you are trying to access. On the bottom right of each source in Google Scholar results is a link that says "all # versions." If the author has uploaded an open access version of their work, you may need to click this link to get to a free copy rather than the paywalled copy at the journal.
- Conduct a search for the article in a normal Google search (possibly adding ".pdf" to the query) to see if you can find a copy of the article without a paywall. Often times, you will find links to a paywalled journal article on a for-profit repository like Social Science Research Network, Acacdemia.edu, or ResearchGate. These are okay to download from, but make sure to keep a keen eye on suspicious websites that promise access to journal articles but actually try to infect your computer with malware. If your browser gives you a warning when you go to a page or when you try to download a PDF, don't risk it. Find another way to get the article.
- Go to unpaywall.org and install the Unpaywall extension for Firefox and Chrome. Once installed, a green icon of an open lock will appear on any scholarly resource for which there is a publicly accessible version without a paywall.
- Go to openaccessbutton.org and either install the Chrome extension or use the web portal to request that the author of an article send you a free copy. The purpose of this project is to make seeking free copies of scholarship as easy as possible, though it still relies on the author to send their article when you request it using this service.
- Go to oahelper.org and install their desktop or mobile app to search for unpaywalled and open access versions of publications.
Everyone can access the abstracts of research articles for free, as long as they have an internet connection. Abstracts are a great place to start, but you need to access the full text of articles for your student projects. I've often seen student papers where they do a simple Google Scholar search and only read articles for which there is free access, with the PDF link in the search results. This is a mistake because few journal articles are free to access (Chimmer, Geschuhn, & Vogler, 2015).[50] Use your library to gain access to the full text of each journal article you need for your search. While you are studying at your university, your login is the key to unlock the paywall and grant you access to the information you need.
Which articles do I download?
Keeping your working question in mind, you should look at your potential sources and evaluate whether they are relevant to your inquiry. To assess the relevance of a source, ask yourself if the source will help you answer or think more deeply about your working question. Does the information help you answer this question, challenge your assumptions, or connect your question with another topic? Does the information present an opposing point of view, so you can show that you have addressed all sides of the argument in your paper? Does the article just provide a broad overview of your topic or does it have a specific focus on what you want to study? If the article isn't helpful to you, it's okay not to read it. No matter how good your searching skills, some articles won't be relevant. You don't need to read and include everything you find!
You may also want to check the relevance of the source with your professor or course syllabus. In my class, I have specific questions I will ask students to address in their literature reviews. You may want to find sources that help you answer the questions in your professor's prompt for a literature review. For example, my prompt asks "how many people experience this problem or issue?" My students then seek out a source that helps them understand the prevalence or incidence of an issue.
In addition to relevance, you should ask yourself about the quality of the source you've found. Is the information outdated? Is the source more than 5-10 years old? If so, it will not provide what we currently know about the topic–just what we used to know. Older sources are helpful for historical information, such as how our understanding of a topic has changed over time or how the prevalence of an issue has increase or decreased. However, unless historical analysis is the focus of your literature review, try to limit your sources to those that are current. Doing so will also narrow down your list of results considerably in a database. Also, you may want to consult with your professor to see if they have additional guidelines on which articles are "too old" to include in a literature review.
Older sources may be of some importance, however, if they are seminal articles. As we learned in section 3.1, seminal articles are cited often in the literature. They are clearly important to a lot of scholars in the field. While not all articles are seminal, you can get a quick sense of how important an article is to the broader literature on a topic by looking at how many other sources cited it. If you search for the article on Google Scholar (see Figure 3.1 for an example of a search result from Google Scholar), you can see how many other sources cited this information. Generally, the higher the number of citations, the more important the article. Of course, articles that were recently published will likely have fewer citations than older articles, and the citation count is only one indicator of an article's importance.
What evidence is in each type of article?
Literature searching is about finding evidence to inform your ideas, supporting or refuting what you think about a topic. Each type of article (we reviewed some of them in section 3.1) provides different kinds of evidence. As we talked about in section 3.2, a good place to start your literature search is with review articles—meta-analyses, meta-syntheses, and systematic reviews. These types of articles give you a birds-eye view of the literature in a topic area, and with meta-analyses and meta-syntheses, conduct empirical analyses on enormous datasets comprised of the raw data from multiple studies. As a result, their conclusions represent what is broadly true about the topic area. They also have comprehensive reference lists that you can browse for sources relevant to your topic.
In my experience, students are often tempted to read short articles because they can complete assignments more quickly that way. This is a trap for students starting a literature review. Short articles take less time to read, sure. But this isn't about reading a certain number of articles, but finding the information you need to write your literature review as efficiently as possible. You'll save time by reading more relevant articles with lengthier and more comprehensive literature reviews—in particular, review articles.
Review articles are the best place to start for any literature search, but you should also look for specific types of articles based on your working question. If your working question asks about an intervention, like a therapeutic technique or program, randomized controlled trials (RCTs) provide the strongest evidence if a meta-analysis or systematic review of relevant RCTs is unavailable. Quasi-experimental designs are considered inferior because researchers have less control over the research process. You will want to avoid relying heavily on articles that use non-experimental designs and include words like "pilot study," "convenience sample," or "exploratory study" in their methods section. These are preliminary studies that are done prior to a more rigorous experiment like an RCT, and their conclusions are tentative and collected for the purpose of informing future inquiry, not establishing what is true for broader populations. We will discuss experimental design in Chapter 13, but for now, it's important to know that the purpose is often to establish the efficacy of an intervention and the truth value of the evidence contained in them varies based on the design of the experiment, with RCTs being the gold standard.
Experiments are one of two quantitative designs explored in this book. The other design is survey research. Looking at survey research is a good idea in any project, as it provides evidence about broader populations by generalizing from a smaller sample of people. For example, surveys can tell you about the risk and protective factors for a social problem by querying the people who are likely to experience that problem over time. Longitudinal surveys are often the most helpful in understanding causality because you have a record of how things have changed over time. Cross-sectional surveys are more limited in establishing causal relationships, as they only query people at one point in time. We discuss the differences between these types of surveys in Chapter 12. Both cross-sectional and longitudinal surveys are very commonly cited types of sources in social work literature reviews because their results are often applicable across broad populations. However, they are limited in the degree to which they can establish causality, as they lack the controlled environment of an experiment. As with experiments, students should be very cautious about using survey results that are "exploratory" or a "pilot study," as the purpose of those studies is to inform future research rather than understand what is true about broader populations.
The previous few paragraphs can be summarized in this hierarchy of evidence, as described by McNeese & Thyer (2004).[51] The higher a type of article on this list, the better they can reliably and directly inform evidence-based practice in social work.
- Systematic reviews and meta-analyses
- Randomized controlled trials
- Quasi-experimental studies
- Case-control and cohort studies
- Pre-experimental (or non-experimental) group studies
- Surveys
- Qualitative studies
The hierarchy of evidence is a useful heuristic, and it is based on sound reasoning. A systematic review or meta-analysis will provide you with a better picture of what is generally true for most people about a given topic than a qualitative study. However, generalizable objective truth is not the only thing researchers want to know. For example, if you wanted to research the impact of gentrification on a community, systematic reviews would not provide you the depth you need to understand the stories of people impacted, displaced, and discriminated against in housing policy. That's not what systematic reviews are designed to do.
To find evidence like that, the most relevant evidence is in qualitative studies, at the bottom of the hierarchy. For this reason, it is better to think of the hierarchy of evidence within the broader context of your working question and the knowledge you need to investigate it. Different types of sources are useful depending on what your question is. Table 4.1 contains a suggested starting point for evaluating what types of literature will provide the most relevant evidence for your project.
What you want know about: | The most relevant evidence will be found in a: |
General knowledge about a topic | Systematic review, meta-analysis literature review, textbook, encyclopedia |
Intervention (therapy, policy, program) | Randomized controlled trial (RCT), meta-analysis, systematic review, cohort study, case-control study, case series, clinical practice guidelines |
Lived experience & sociocultural context | Qualitative study, participatory and action research, humanities and cultural studies |
Theory and practice models | Theoretical and non-empirical article, textbook, manual for an evidence-based treatment, book or edited volume |
Prevalence of a diagnosis or social problem | Survey research, government and nonprofit reports |
How practitioners think about a topic | Practice note, survey or qualitative study of practitioners, reports from professional organizations |
Qualitative studies are not designed to provide information about a broader population. As a result, you should treat their results as related to the specific time and place in which they occurred. If a study's context is similar to the one you plan to research, then you might expect similar results to emerge in your research project. Qualitative studies provide the lived experience and personal reflections of people knowledgeable about your topic. These subjective truths provide evidence that is just as important as other studies in the hierarchy of evidence.
Key Takeaways
- Once you have a reasonable number of search results, you can start skimming abstracts. If an article is relevant to your project, download the full text PDF to a folder on your computer.
- Download strategically, based on what your professor expects in your literature review and what information you need to understand, revise, and answer your working question.
- Paywalls do not apply to students at a university. You have access to a lot of articles through your university's library or through interlibrary loan. Do not limit yourself to PDFs floating around on the internet.
- Write notes to yourself, so you can track how your project has developed over time.
Exercises
- Look at your professor's prompt for a literature review and sketch out how you might answer those questions using your present level of knowledge. Search for sources that support or challenge what you think is true about your topic.
- Find a news article reporting about topics similar to your working question. Identify whether it is a primary or secondary source. If it is a secondary source, trace any claims to their primary sources. Provide the URLs.
4.2 Organizing information
Learning Objectives
Learners will be able to...
- Describe how to use summary tables to organize information from empirical articles
- Describe how to use topical outlines to organize information from the literature reviews of articles you read
- Create a concept map that visualizes the key concepts and relationships relevant to your working question
- Use what you learn in the literature search to revise your working question
After working through the exercises Chapter 3, you should have a folder full of articles. For me, searching the literature is the fun part of a literature review. I have grand ideas about reading this article and that article and every article! Once I'm satisfied I have enough literature to speak intelligently on my topic, a creeping sense of dread kicks in. How will I ever find the time to read 60 journal articles?! This chapter is all about how to pick the most relevant articles, write notes about them, and incorporate the information relevant to your working question in your literature review.
Read literature reviews
At the beginning of a research project, you don't know a lot about your research topic. You don't even know what you don't know! That's why it's a good idea to get a broad, birds-eye view of the literature by looking at other literature reviews. Someone has gone through the trouble of reading a few dozen sources and telling you what's important about them. Get a broad sense of the literature and follow up on subtopics that interest you. As we discussed in Chapter 3, review articles are useful because they synthesize all the information on a given topic. By reading what another researcher thinks about the literature, you can get a more wide-ranging sense of it than reading the results of only one study.
As we discussed in the last chapter, review articles will often have "literature review," "systematic review," or other similar terms in the title. These articles are 100% literature review. The author's primary goal is to present a comprehensive and authoritative view of the important research in a particular topic area. Think of these as a way to engage with dozens of articles at the same time, and you can find a lot of relevant references from reading the article. For the same reason, meta-analyses and meta-syntheses are also excellent sources as you are starting a literature review.
Unfortunately, review articles do not exist for every topic. If you are unable to find a review article, try to find an empirical article that has a lengthy literature review. You are mostly reading to see what the author says about the literature on your topic in the introduction and discussion sections. In this way, you can use the author's literature search to inform your own. You will likely find new ideas and get a sense of the broader scientific literature just by reading what other researchers think about the literature.
This section will introduce you to three tools scholars use to organize and synthesize (i.e., weave together) information from multiple sources. First, we will discuss how to build a summary table containing information from empirical articles that are highly relevant—from literature review, to methods and results—to your entire research proposal. These are articles you will need to know the details of back-to-front because they are so highly related to your proposed study.
Second, we'll discuss what to do with the other articles you've downloaded. As we've discussed previously, you're not going to read most of the sources you download from start-to-finish. Instead, you'll look at the author's literature review, key ideas, and skim for any relevant passages for your project. As you do so, you should create a topical outline that organizes all relevant facts you might use in your literature that you've collected from the abstract, literature review, and conclusion of the articles you've found. Of course, it is important to note the original source of the information you are citing.
Finally, we will revisit concept mapping as a technique for visualizing the concepts in your study. Altogether, these techniques should help you create intermediary products—documents you are not likely to show to anyone or turn in for a grade—but that are vital steps to a final research proposal.
Organizing empirical articles using a summary table
Your research proposal is an empirical project. You will collect raw data and analyze it to answer your question. Over the next few weeks, identify about 10 articles that are empirically similar to the study you want to conduct. If you plan on conducting surveys of practitioners, it's a good idea for you to read in detail other articles that have used similar methods (sampling, measures, data analysis) and asked similar questions to your proposal. A summary table can help you organize these Top 10 articles: empirical articles that are highly relevant to your proposal and working question.
Using the annotations in Section 4.2 as a guide, create a spreadsheet or Word table with your annotation categories as columns and each source as new row. For example, I was searching for articles on using a specific educational technique in the literature. I wanted to know whether other researchers found positive results, how big their samples were, and whether they were conducted at a single school or across multiple schools. I looked through each empirical article on the topic and filled in a summary table. At the end, I could do an easy visual analysis and state that most studies revealed no significant results and that there were few multi-site studies. These arguments were then included in my literature review. These tables are similar to those you will find in a systematic review article.
A basic summary table is provided in Figure 5.1. A more detailed example is available from Elaine Gregersen's blog, and you can download an Excel template from Raul Pacheco-Vega's blog. Remember, although “the means of summarizing can vary, the key at this point is to make sure you understand what you’ve found and how it relates to your topic and research question” (Bennard et al., 2014, para. 10).[52] As you revisit and revise your working question over the next few weeks, think about those sources that are so relevant you need to understand every detail about them.
A good summary table will also ensure that when you cite these articles in your literature review, you are able to provide the necessary detail and context for readers to properly understand the results. For example, one of the common errors I see in student literature reviews is using a small, exploratory study to represent the truth about a larger population. You will also notice important differences in how variables are measured or how people are sampled, for instance, and these details are often the source of a good critical review of the literature.
Exercises
- Using your folder of article PDFs from you've downloaded in previous exercises, identify which articles are likely to be most relevant to your proposed study. This may change as you revise your working question and study design over the next few weeks. Create a list of 10 articles that are highly relevant to the extent that you will need to remember key details from each section of the article.
- Create a spreadsheet for your summary table and save it in your project folder on your hard drive. Using one of the templates linked in this chapter, fill in the columns of your spreadsheet. Enter the information from one of the articles you've read so far. As you finalize your research question over the next few weeks, fill in your summary table with the 5 most relevant empirical articles on your topic.
Synthesizing facts using a topical outline
If we're only reading 10 articles in detail, what do we do with the others? Raul Pacheco-Vega recommends using the AIC approach: read the abstract, introduction, and conclusion (and the discussion section, in empirical articles). For non-empirical articles, it's a little less clear but the first few pages and last few pages of an article usually contain the author's reading of the relevant literature and their principal conclusions. You may also want to skim the first and last sentence of each paragraph. Only read paragraphs in which you are likely to find information relevant to your working question. Skimming like this gives you the general point of the article, though you should read in detail the most valuable resource of all—another author's literature review.
It's impossible to read all of the literature about your topic. You will read about 10 articles in detail. For a few dozen more (there is no magic number), you will read the abstract, introduction, and conclusion, skim the rest of the article, but ultimately never read everything. Make the most out of the articles you do read by extracting as many facts as possible from each. You are starting your research project without a lot of knowledge of the topic you want to study, and by using the literature reviews provided in academic journal articles, you can gain a lot of knowledge about a topic in a short period of time. This way, by reading only a small number of articles, you are also reading their citations and synthesis of dozens of other articles as well.
As you read an article in detail, we suggest copying any facts you find relevant in a separate word processing document. Another idea is to copy anything you've annotated as background information in Section 4.2 into an outline. Copying and pasting from PDF to Word can be difficult because PDFs are image files, not documents. To make that easier, use the HTML version of the article, convert the PDF to Word in Adobe Acrobat or another PDF reader, or use the “paste special” command to paste the content into Word without formatting. If it’s an old PDF, you may have to simply type out the information you need. It can be a messy job, but having all of your facts in one place is very helpful when drafting your literature review.
You should copy and paste any fact or argument you consider important. Some good examples include definitions of concepts, statistics about the size of the social problem, and empirical evidence about the key variables in the research question, among countless others. It’s a good idea to consult with your professor and the course syllabus to understand what they are looking for when reading your literature review. Facts for your literature review are principally found in the introduction, results, and discussion section of an empirical article or at any point in a non-empirical article. Copy and paste into your notes anything you may want to use in your literature review.
Importantly, you must make sure you note the original source of each bit of information you copy. Nothing is worse than needing to track down a source for fact you read who-knows-where. If you found a statistic that the author used in the introduction, it almost certainly came from another source that the author cited in a footnote or internal citation. You will want to check the original source to make sure the author represented the information correctly. Moreover, you may want to read the original study to learn more about your topic and discover other sources relevant to your inquiry.
Assuming you have pulled all of the facts out of multiple articles, it’s time to start thinking about how these pieces of information relate to each other. Start grouping each fact into categories and subcategories as shown in Table 5.2. For example, a statistic stating that single adults who are homeless are more likely to be male may fit into a category of gender and homelessness. For each topic or subtopic you identify during your critical analysis of each paper, determine what those papers have in common. Likewise, determine which differ. If there are contradictory findings, you may be able to identify methodological or theoretical differences that could account for these contradictions. For example, one study may sample only high-income earners or those living in a rural area. Determine what general conclusions you can report about the topic or subtopic, based on all of the information you’ve found.
Create a separate document containing a topical outline that combines your facts from each source and organizes them by topic or category. As you include more facts and more sources in your topical outline, you will begin to see how each fact fits into a category and how categories are related to one another. Keep in mind that your category names may change over time, as may their definitions. This is a natural reflection of the learning you are doing.
Facts copied from an article | Topical outline: Facts organized by category |
|
|
A complete topical outline is a long list of facts arranged by category. As you step back from the outline, you should assess which topic areas for which you have enough research support to allow you to draw strong conclusions. You should also assess which areas you need to do more research in before you can write a robust literature review. The topical outline should serve as a transitional document between the notes you write on each source and the literature review you submit to your professor. It is important to note that they contain plagiarized information that is copied and pasted directly from the primary sources. In this case, it is not problematic because these are just notes and are not meant to be turned in as your own ideas. For your final literature review, you must paraphrase these sources to avoid plagiarism. More importantly, you should keep your voice and ideas front-and-center in what you write as this is your analysis of the literature. Make strong claims and support them thoroughly using facts you found in the literature. We will pick up the task of writing your literature review in section 5.3.
Exercises
- In your folder full of article PDFs, look for the most relevant review articles. If you don't have any, try to look for some. If there are none in your topic area, you can also use other non-empirical articles or empirical articles with long literature reviews (in the introduction and discussion sections).
- Create a word processing document for your topical outline and save it in your project folder on your hard drive. Using a review article, start copying facts you identified as Background Information or Results into your topical outline. Try to organize each fact by topic or theme. Make sure to copy the internal citation for the original source of each fact. For articles that do not use internal citations, create one using the information in the footnotes and references. As you finalize your research question over the next few weeks, skim the literature reviews of the articles you download for key facts and copy them into your topical outline.
Putting the pieces together: Building a concept map
Developing a concept map or mind map around your topic can be helpful in figuring out how the facts fit together. We talked about concept mapping briefly in Chapter 2, when we were first thinking about your topic and sketching out what you already know about it. Concept mapping during the literature review stage of a research project builds on this foundation of knowledge and aims to improve the “description of the breadth and depth of literature in a domain of inquiry. It also facilitates identification of the number and nature of studies underpinning mapped relationships among concepts, thus laying the groundwork for systematic research reviews and meta-analyses” (Lesley, Floyd, & Oermann, 2002, p. 229).[53] Its purpose, like other question refinement methods, is to help you organize, prioritize, and integrate material into a workable research area—one that is interesting, answerable, feasible, objective, scholarly, original, and clear.
Think about the topics you created in your topic outline. How do they relate to one another? Within each topic, how do facts relate to one another? As you write down what you have, think about what you already know. What other related concepts do you not yet have information about? What relationships do you need to investigate further? Building a conceptual map should help you understand what you already know, what you need to learn next, and how you can organize a literature review.
This technique is illustrated in this YouTube video about concept mapping. You may want to indicate which concepts and relationships you've already found in your review and which ones you think might be true but haven't found evidence of yet. Once you get a sense of how your concepts are related and which relationships are important to you, it's time to revise your working question.
Exercises
- Create a concept map using a pencil and paper.
- Identify the key ideas inside the literature, how they relate to one another, and the facts you know about them.
- Reflect on those areas you need to learn more about prior to writing your literature review.
- As you finalize your research question over the next few weeks, update your concept map and think about how you might organize it into a written literature review.
- Refer to the topics and headings you use in your topical outline and think about what literature you have that helps you understand each concept and relationship between them in your concept map.
Revising your working question
You should be revisiting your working question throughout the literature review process. As you continue to learn more about your topic, your question will become more specific and clearly worded. This is normal, and there is no way to shorten this process. Keep revising your question in order to ensure it will contribute something new to the literature on your topic, is relevant to your target population, and is feasible for you to conduct as a student project.
For example, perhaps your initial idea or interest is how to prevent obesity. After an initial search of the relevant literature, you realize the topic of obesity is too broad to adequately cover in the time you have to do your project. You decide to narrow your focus to causes of childhood obesity. After reading some articles on childhood obesity, you further narrow your search to the influence of family risk factors on overweight children. A potential research question might then be, “What maternal factors are associated with toddler obesity in the United States?” You would then need to return to the literature to find more specific studies related to the variables in this question (e.g. maternal factors, toddler, obesity, toddler obesity).
Similarly, after an initial literature search for a broad topic such as school performance or grades, examples of a narrow research question might be:
- “To what extent does parental involvement in children’s education relate to school performance over the course of the early grades?”
- “Do parental involvement levels differ by family social, demographic, and contextual characteristics?”
- “What forms of parent involvement are most highly correlated with children’s outcomes? What factors might influence the extent of parental involvement?” (Early Childhood Longitudinal Program, 2011).[54]
In either case, your literature search, working question, and understanding of the topic are constantly changing as your knowledge of the topic deepens. A literature review is an iterative process, one that stops, starts, and loops back on itself multiple times before completion. As research is a practice behavior of social workers, you should apply the same type of critical reflection to your inquiry as you would to your clinical or macro practice.
There are many ways to approach synthesizing literature. We’ve reviewed the following: summary tables, topical outlines, and concept maps. Other examples you may encounter include annotated bibliographies and synthesis matrices. As you are learning how to conduct research, find a method that works for you. Reviewing the literature is a core component of evidence-based practice in social work. See the resources below if you need some additional help:
Literature Reviews: Using a Matrix to Organize Research / Saint Mary’s University of Minnesota
Literature Review: Synthesizing Multiple Sources / Indiana University
Writing a Literature Review and Using a Synthesis Matrix / Florida International University
Sample Literature Reviews Grid / Complied by Lindsay Roberts
Literature review preparation: Creating a summary table. (Includes transcript) / Laura Killam
Key Takeaways
- You won’t read every article all the way through. For most articles, reading the abstract, introduction, and conclusion are enough to determine its relevance. It's expected that you skim or search for relevant sections of each article without reading the whole thing.
- For articles where everything seems relevant, use a summary table to keep track of details. These are particularly helpful with empirical articles.
- For articles with literature review content relevant to your topic, copy any relevant information into a topical outline, along with the original source of that information.
- Use a concept map to help you visualize the key concepts in your topic area and the relationships between them.
- Revise your working question regularly. As you do, you will likely need to revise your search queries and include new articles.
Exercises
- Look back at the working question for your topic and consider any necessary revisions. It is important that questions become clearer and more specific over time. It is also common that your working question shift over time, sometimes drastically, as you explore new lines of inquiry in the literature. Return to your working question regularly and make sure it reflects the focus of your inquiry. You will continue to revise your working question until we formalize it into a research question at the end of Part 2 of this textbook.
4.3 Critical information literacy
Learning Objectives
Learners will be able to...
- Define critical information literacy
- Explore the relationship between social work practice, consuming social science, and fighting for social justice
Ultimately, searching and reviewing the literature will be one of the most transferable skills from your research methods classes. All social workers have to consume research information as part of their jobs. In this closing section, we hope to ground this orientation toward scientific literature in your identity as a social worker, scholar, and social scientist. In one sense, to be a critical consumer of research is to be discerning about which sources of information you read and why. If you have followed along with the exercises in this chapter, you hopefully have about 30, 50, or even 100 PDFs of articles that are relevant to your project in a folder on your computer. You do not have to read every article or source that discusses your topic.
At some point, reading another article won't add anything new to your literature review. We've discussed in this chapter how to read sources strategically, getting the most out of each one without wasting your time. Ultimately, the number of sources you need should be guided, at a minimum, by your professor's expectations and what you need to provide a comprehensive review of the literature on your topic. The purpose of the literature review is to give an overview of the research relevant to your project—everything a reader would need to understand the importance, purpose, and thought behind your project—not for you to restate everything that has ever been studied about your topic.
Research is a human right
But when we talk about being critical in social work, we mean more than critical thinking. The core skill you are developing in Chapters 3-5 of this textbook is information literacy, or "a set of abilities requiring individuals to 'recognize when information is needed and have the ability to locate, evaluate, and use effectively the needed information" (American Library Association, 2020).[55] Information literacy is key to the lifelong learning process of social work practice, as you will have to continue to absorb new information after you leave school and progress as a social work practitioner. In the evidence-based practice process, information literacy is a key component in helping clients and communities achieve their goals.
However, social work researchers and librarians dedicated to social change embrace a more progressive conceptualization of this skill called critical information literacy. It is not enough to simply know how to use the tools we use to share knowledge. Instead, social work researchers should critically examine "the social, political, economic, and corporate systems that have power and influence over information production, dissemination, access, and consumption” (Gregory & Higgins, 2013, p. ix).[56] Just like all other social structures, those we use to create and share scientific knowledge are subject to the same oppressive forces we encounter in everyday life—racism, sexism, classism, and so forth. Critical information literacy combines both fine-tuned skills for finding and evaluating literature as well as an understanding of the cultural forces of oppression and domination that structure the availability of information. As Tewell (2016)[57] argues, "critical information literacy is an attempt to render visible the complex workings of information so that we may identify and act upon the power structures that shape our lives" (para. 6).
From critical information literacy, we get the term information privilege which is defined by Booth (2017)[58] as:
the accumulation of special rights and advantages not available to others in the area of information access. Individuals with the resources to access the information they need, are affiliated with research or academic institutions and libraries, or live close to a public library with access to resources and services such as free interlibrary loan are examples of those with information privilege. Those who are unable to access the information they need are information underprivileged or impoverished [emphasis in original]; this includes people who are incarcerated, poor, unaffiliated with a university or research institution, or live in rural areas distant from a public library.
How can you recognize information privilege? The video from the Steely Library at Northern Kentucky University below will help you.
It is important to recognize your own information privilege and use it to help enact social change on behalf of those who are oppressed within the system of scholarly knowledge production and sharing we currently have. We previously discussed the issue of paywalls in this chapter. Paywalls lock away access to knowledge to only those who can afford to pay, shutting out those with fewer resources. This effect of publishing only in journals that require a paywall is endemic to social work scholarship, as Pendell (2020, 2018)[59] describes, there are few social work open access journals and social work researchers do not post open access copies of their articles even when explicitly allowed by publishers to do so. The reasons for this are not financial, as authors do not profit from the sales of journal articles. Indeed, faculty who edit, review, and author articles are not paid for their work under the current model of commercial journal publication. Scholars do not have a meaningful share in the 31.3% profit margins of Elsevier, the largest commercial publisher of scholarly journals (RELX, 2019).[60] Libraries are struggling to keep up with annual price increases. This topsy-turvy world is best encapsulated in Figure 4.1, which summarizes the problems in scholarly publication succinctly.
As we discussed at the beginning of this chapter, paywalls effectively shut out social work students from accessing journal articles after they graduate.[61] However, paywalls also shut out community members, clients, and self-advocates from accessing the research information they need to effectively advocate for themselves. This is because paywalls are an obstruction to the basic human right to access and interact with the knowledge humanity creates.
As Willinsky (2006) states, "access to knowledge is a human right that is closely associated with the ability to defend, as well as to advocate for, other rights.” The open access movement is a human rights movement that seeks to secure the universal right to freely access information in order to produce social change. Information is a resource, and the current approach to sharing that resource—the key to human development—excludes many oppressed groups from accessing what they need to address matters of social justice. Appadurai (2006)[62] conceptualizes this as a "right to research...the right to the tools through which any citizen can systematically increase that stock of knowledge which they consider most vital to their survival as human beings and to their claims as citizens" (p. 168). From a human rights perspective, research is not something confined to professional researchers but "the capacity to systematically increase the horizons of one’s current knowledge, in relation to some task, goal or aspiration" (p. 173).
In Chapter 2, we discussed action research which includes community members as part of the research team. Action research addresses information privilege by addressing the power imbalance between academic researchers and community members, ensuring that the voice of the community is represented throughout the process of creating scientific knowledge. This equity-enhancing process provides community members with access to scientific knowledge in many ways: (1) access via academic to scholarly databases, (2) access to the training and human capital of the researcher, who embodies years of education in how to conduct social science research, and (3) access to the specialized resources like statistical or qualitative data analysis software required to conduct scientific research. Moreover, it highlights that while open access is important, it is only one half of the equation. Open access only addresses how people receive scholarly information. But what about how social science knowledge is created? Action research underscores that equity is not just about accessing scientific knowledge, but co-creating scientific knowledge as partners with community members. Critical information literacy critiques the current practices of scientific inquiry in social work and other disciplines as exclusionary, as they reinforce existing sources of privilege and oppression.
This imbalance between academic researchers and community members should be considered within an English-speaking, Western context. The widespread adoption of open access and open science in the developing world underscores the extreme imbalances that international researchers face in accessing scholarly information (Arunachalam, 2017).[63] Paywalls, barriers to international travel, and the tradition of in-person physical conferences structure the information sharing practices in social science in a manner that excludes participation from researchers who lack the financial and practical resources to access these exclusionary spaces (Bali & Caines, 2016; Eaves, 2020).[64]
In the United States, the federal government has increasingly adopted policies that facilitate open access, including the National Institute of Health's requirement that all publications from projects it funds must be open access within a year after publication in a commercial journal as well as President Obama's open data initiative. At the time of this writing, the Scholarly Publishing and Academic Resources Coalition (SPARC) is working with the Trump administration on a formal proposal to make all research articles published using federal grant funds accessible to the public as open access, though at the time of this writing there is no final agreement on the policy design or implementation. In Europe, Plan S is a multi-stakeholder social change coalition that is seeking structural change to scholarly publication and open access to research. Latin America and South America are far ahead of their Western counterparts in adopting open access, providing a view of what is possible when community and equity are centered in scholarly publication (Aguado-Lopez & Becerril-Garcia, 2019).[65]
The slow progress of social movement organizations to ensure free access to scientific knowledge has inspired radical action. Out of growing frustration with journal paywalls, Sci-Hub was created by a computer programmer from Kazakhstan, Alexandra Elbakyan. This resource is an illegal, pirate repository of about 85% of all paywalled journal articles in existence up to 2020, when it stopped uploading papers in compliance with an injunction from an Indian court (Himmelstein et al., 2018).[66] Sci-Hub is not legal under international copyright law,[67] yet researchers unaffiliated with a well-funded university in the developed world cannot conduct research without it. The site has moved multiple times as their domains are shut down commercial publishers whose content is illegally stored by Sci-Hub, and its founder hopes that by successfully arguing her case before and Indian court, she can continue to operate Sci-Hub legally under Indian law, and in effect, guarantee the human right to access research for free within India (Cummins, 2021).[68] Sci-Hub is a symptom of a broken system of scholarly publication that locks away scientific knowledge behind impossibly high textbook and journal paywalls. These are impossible for nearly all people to pay for in the course of a normal literature review, and thus, our broken system of scholarly publication necessitates that researchers use shadow libraries like Sci-Hub, Z-Library, and Library Genesis. It should not require mass-scale copyright infringement for the world to access scientific information.
The current model of scientific publishing privileges those already advantaged and raises important obstacles for those in oppressed groups to access and contribute to scientific knowledge or research-informed social change initiatives. For a more thorough, Black feminist perspective on scholarly publication see Eaves (2020).[69] As social workers, it is your responsibility to use your information privilege—the access you have right now to the world's knowledge and the skills you gain during your graduate education and post-graduate professional development—to fight for social change.
Being a responsible consumer of research
Social work often involves accessing, creating, and otherwise engaging with social science knowledge to create change. On the micro-level, critical information literacy is needed to inform evidence-based decision-making in clinical practice. On the meso- and macro-level, social workers can use information literacy skills to give a voice for community concerns, help communities access and create the knowledge they need to foster change, or evaluate how well existing programs serving the community achieve their goals. Now that you are familiar with how to conduct ethical and responsible research and how to read the results of others’ research, you have an obligation to use your information literacy skills to create social change. This is part of critical information literacy: using your information privilege to address social injustices.
Collecting, sharing, and creating research findings for social change requires you to take seriously your identity as a social scientist. Doing so is in part a matter of being able to distinguish what you do know based on research findings from what you do not know. It is also a matter of having some awareness about what you can and cannot reasonably know.
When assessing social scientific findings, think about the information provided to you. Social scientists should be transparent with how they collected their data, ran their analyses, and formed their conclusions. These methodological details can help you evaluate the researcher’s claims. If, on the other hand, you come across some discussion of social scientific research in a popular magazine or newspaper, chances are that you will not find the same level of detailed information that you would find in a scholarly journal article. With secondary sources like news articles, it is hard to know enough about the study to be an informed consumer of information. Always read the primary source when possible.
Additionally, take into account whatever information is provided about a study’s funding source. Most funders want, and in fact require, that recipients acknowledge them in publications. Keep in mind that some sources may not disclose who funds them. If this information is not provided in the source from which you learned about a study, it might behoove you to do a quick search on the internet to see if you can learn more about the funding. Findings that seem to support a particular political agenda, for example, might have more or less influence on your thinking about your topic once you know if the funding posed a potential conflict of interest.
There is some information that even the most responsible consumer of research cannot know. Because researchers are ethically bound to keep the identity of people in our study confidential, for example, we will never know exactly who participated in a given study or sometimes even in what location it was conducted. Awareness that you may never know everything about a study should provide some humility in terms of what we can “take away” from a given report of findings.
Unfortunately, researchers also sometimes engage in unethical behavior and do not disclose things that they should. For example, a recent study on COVID-19 (Bendavid et al., 2020)[70] did not disclose that it was funded by the chief executive of JetBlue, an airline company losing money from COVID-19 travel restrictions. It is alleged that this study was paid for by airline executives in order to provide scientific support for increasing the use of air travel and lifting travel restrictions. These conflicts of interest demonstrate that science is not distinct from other areas of social life that can be abused by those with more resources in order to benefit themselves.
COVID-19 is a particularly instructive case in spotting bad science. In a rapidly evolving information context, social workers and others were forced to make decisions quickly using imperfect information. The Lancet and the New England Journal of Medicine, two of the most prestigious journals in medicine, had to retract two studies due to irregularities missed by pre-publication peer review despite the fact that the results had already been used to inform policy changes (Joseph, 2020).[71] At the same time, President Trump's lies and misinformation about COVID-19 were a grave reminder of how information can be corrupted by those with power in the social and political arena (Paz, 2020).[72]
Figure 4.2 presents a rough guide to spotting bad science that is particularly useful when science has not had enough time for peer review and scholarly debate to thoroughly and systematically investigate a topic, as in the COVID-19 crisis. It is also a useful quick-reference guide for social work practitioners who encounter new information about their topics from the news media, social media, or other informal sources of information like friends, family, and colleagues.
Feminist perspectives on science generally support the framework in Figure 4.2, but consider them to be incomplete, as they do not critique the scientific process itself. By drawing a line between science vs. pseudoscience, scientific inquiry can further social justice by using data to battle misinformation and uncover social inequities. However, feminist perspectives also draw our attention to historically overlooked aspects of the scientific process. D'ignazio and Klein (2020)[73] instead offer seven intersectional feminist principles for equitable and actionable COVID-19 data, as visualized in Figure 4.3.
Information literacy beyond scholarly literature
The research process we described in this chapter will help you arrive at an understanding of the scientific literature. However, that is not the only literature of value for researchers. For those conducting action research that engages more with communities and target populations, researchers are responsible not only for reviewing the scientific literature on their topic but also the literature that communities find important. If there are local newspapers, television shows, religious services or events, community meetings, or other sources of knowledge that your target population finds important, you should engage with those resources as well. Understanding these information sources builds empathetic understanding of participating groups and can help inform your research study. Moreover, they are likely to contain knowledge that is not a part of the scientific literature but is nevertheless crucial for conducting scientific research appropriately and effectively in a community.
Key Takeaways
- At this point, you should have a folder full of articles (at least a few dozen) that are relevant to your topic. Don't worry! You won't read all of them, but you will skim most of them for the most important information.
- Social workers should develop critical information literacy. This helps social workers use social science knowledge for social change, and it critiques how current publishing models exclude and privilege different groups from accessing and creating knowledge.
- Social work involves helping others understand social science and applying it in your practice. You must learn how to discriminate between reliable and unreliable information and apply your scientific knowledge to benefit your clients and community.
Exercises
- Write a few sentences about what you think the answer to your working question might be.
- Identify at least five reputable sources of information from your literature search that provide evidence that what you wrote is true.
- What steps do you plan to take to demonstrate that you are helping others to understand social science and its relationship to your practice?
process by which researchers spell out precisely how a concept will be measured in their study
A research instrument consisting of a set of questions (items) intended to capture responses from participants in a standardized manner
Chapter Outline
- Reading the results of empirical studies (16 minute read)
- Annotating empirical journal articles (15 minute read)
- Generalizability and transferability of empirical results (15 minute read)
Content warning: examples in this chapter contain references to domestic violence and details on types of abuse, drug use, poverty, mental health, sexual harassment and details on harassing behaviors, children’s mental health, LGBTQ+ oppression and suicide, obesity, anti-poverty stigma, and psychotic disorders.
5.1 Reading the results of empirical studies
Learning Objectives
Learners will be able to...
- Describe how statistical significance and confidence intervals demonstrate which results are most important
- Differentiate between qualitative and quantitative results in an empirical journal article
If you recall from section 3.1, empirical journal articles are those that report the results of quantitative or qualitative data analyzed by the author. They follow a set structure—introduction, methods, results, discussion/conclusions. This section is about reading the most challenging section: results.
I want to normalize not understanding statistics terms and symbols. However, a basic understanding of a results section goes a very long way to understanding the key results in an article. This will take you beyond the two or three sentences in the abstract that summarize the study's results and into the nitty-gritty of what they found for each concept they studied.
Read beyond the abstract
At this point, I have read hundreds of literature reviews written by students. One of the challenges I have noted is that students will report the results as summarized in the abstract, rather than the detailed findings laid out in the results section of the article. This poses a problem when you are writing a literature review because you need to provide specific and clear facts that support your reading of the literature. The abstract may say something like: “we found that poverty is associated with mental health status.” For your literature review, you want the details, not the summary. In the results section of the article, you may find a sentence that states: “children living in households experiencing poverty are three times more likely to have a mental health diagnosis.” This more specific statistical information provides a stronger basis on which to build the arguments in your literature review.
Using the summarized results in an abstract is an understandable mistake to make. The results section often contains figures and tables that may be challenging to understand. Often, without having completed more advanced coursework on statistical or qualitative analysis, some of the terminology, symbols, or diagrams may be difficult to comprehend. This section is all about how to read and interpret the results of an empirical (quantitative or qualitative) journal article. Our discussion here will be basic, and in parts three and four of the textbook, you will learn more about how to interpret results from statistical tests and qualitative data analysis.
Remember, this section only addresses empirical articles. Non-empirical articles (e.g., theoretical articles, literature reviews) don't have results. They cite the analysis of raw data completed by other authors, not the person writing the journal article who is merely summarizing others' work.
Quantitative results
Quantitative articles often contain tables, and scanning them is a good way to begin reading the results. A table usually provides a quick, condensed summary of the report’s key findings. Tables are a concise way to report large amounts of data. Some tables present descriptive information about a researcher’s sample (often the first table in a results section). These tables will likely contain frequencies (N) and percentages (%). For example, if gender happened to be an important variable for the researcher’s analysis, a descriptive table would show how many and what percent of all study participants are of a particular gender. Frequencies or “how many” will probably be listed as N, while the percent symbol (%) might be used to indicate percentages.
In a table presenting a causal relationship, two sets of variables are represented. The independent variable, or cause, and the dependent variable, the effect. We will discuss these further when we review quantitative conceptualization and measurement. Independent variable attributes are typically presented in the table’s columns, while dependent variable attributes are presented in rows. This allows the reader to scan a table’s rows to see how values on the dependent variable change as the independent variable values change (i.e., changes in the dependent variable depend on changes in the independent variable). Tables displaying results of quantitative analysis will also likely include some information about which relationships are significant or not. We will discuss the details of significance and p-values later in this section.
Let’s look at a specific example: Table 5.1. It presents the causal relationship between gender and experiencing harassing behaviors at work. In this example, gender is the independent variable (the cause) and the harassing behaviors listed are the dependent variables (the effects).[74] Therefore, we place gender in the table’s columns and harassing behaviors in the table’s rows.
Reading across the table’s top row, we see that 2.9% of women in the sample reported experiencing subtle or obvious threats to their safety at work, while 4.7% of men in the sample reported the same. We can read across each of the rows of the table in this way. Reading across the bottom row, we see that 9.4% of women in the sample reported experiencing staring or invasion of their personal space at work while just 2.3% of men in the sample reported having the same experience. We’ll discuss p values later in this section.
Behavior experienced at work | Women | Men | p-value |
Subtle or obvious threats to your safety | 2.9% | 4.7% | 0.623 |
Being hit, pushed, or grabbed | 2.2% | 4.7% | 0.480 |
Comments or behaviors that demean your gender | 6.5% | 2.3% | 0.184 |
Comments or behaviors that demean your age | 13.8% | 9.3% | 0.407 |
Staring or invasion of your personal space | 9.4% | 2.3% | 0.039 |
Note: Sample size was 138 for women and 43 for men. |
While you can certainly scan tables for key results, they are often difficult to understand without reading the text of the article. The article and table were meant to complement each other, and the text should provide information on how the authors interpret their findings. The table is not redundant with the text of the results section. Additionally, the first table in most results sections is a summary of the study's sample, which provides more background information on the study than information about hypotheses and findings. It is also a good idea to look back at the methods section of the article as the data analysis plan the authors outline should walk you through the steps they took to analyze their data which will inform how they report them in the results section.
Statistical significance
The statistics reported in Table 5.1 represent what the researchers found in their sample. The purpose of statistical analysis is usually to generalize from a the small number of people in a study's sample to a larger population of people. Thus, the researchers intend to make causal arguments about harassing behaviors at workplaces beyond those covered in the sample.
Generalizing is key to understanding statistical significance. According to Cassidy and colleagues, (2019)[75] 89% of research methods textbooks in psychology define statistical significance incorrectly. This includes an early draft of this textbook which defined statistical significance as "the likelihood that the relationships we observe could be caused by something other than chance." If you have previously had a research methods class, this might sound familiar to you. It certainly did to me!
But statistical significance is less about "random chance" than more about the null hypothesis. Basically, at the beginning of a study a researcher develops a hypothesis about what they expect to find, usually that there is a statistical relationship between two or more variables. The null hypothesis is the opposite. It is the hypothesis that there is no relationship between the variables in a research study. Researchers then can hopefully reject the null hypothesis because they find a relationship between the variables.
For example, in Table 5.1 researchers were examining whether gender impacts harassment. Of course, researchers assumed that women were more likely to experience harassment than men. The null hypothesis, then, would be that gender has no impact on harassment. Once we conduct the study, our results will hopefully lead us to reject the null hypothesis because we find that gender impacts harassment. We would then generalize from our study's sample to the larger population of people in the workplace.
Statistical significance is calculated using a p-value which is obtained by comparing the statistical results with a hypothetical set of results if the researchers re-ran their study a large number of times. Keeping with our example, imagine we re-ran our study with different men and women from different workplaces hundreds and hundred of times and we assume that the null hypothesis is true that gender has no impact on harassment. If results like ours come up pretty often when the null hypothesis is true, our results probably don't mean much. "The smaller the p-value, the greater the statistical incompatibility with the null hypothesis" (Wasserstein & Lazar, 2016, p. 131).[76] Generally, researchers in the social sciences have used 0.05 as the value at which a result is significant (p is less than 0.05) or not significant (p is greater than 0.05). The p-value 0.05 refers to if 5% of those hypothetical results from re-running our study show the same or more extreme relationships when the null hypothesis is true. Researchers, however, may choose a stricter standard such as 0.01 in which only 1% of those hypothetical results are more extreme or a more lenient standard like 0.1 in which 10% of those hypothetical results are more extreme than what was found in the study.
Let's look back at Table 5.1. Which one of the relationships between gender and harassing behaviors is statistically significant? It's the last one in the table, "staring or invasion of personal space," whose p-value is 0.039 (under the p<0.05 standard to establish statistical significance). Again, this indicates that if we re-ran our study over and over again and gender did not impact staring/invasion of space (i.e., the null hypothesis was true), only 3.9% of the time would we find similar or more extreme differences between men and women than what we observed in our study. Thus, we conclude that for staring or invasion of space only, there is a statistically significant relationship.
For contrast, let's look at "being pushed, hit, or grabbed" and run through the same analysis to see if it is statistically significant. If we re-ran our study over and over again and the null hypothesis was true, 48% of the time (p=.48) we would find similar or more extreme differences between men and women. That means these results are not statistically significant.
This discussion should also highlight a point we discussed previously: that it is important to read the full results section, rather than simply relying on the summary in the abstract. If the abstract stated that most tests revealed no statistically significant relationships between gender and harassment, you would have missed the detail on which behaviors were and were not associated with gender. Read the full results section! And don't be afraid to ask for help from a professor in understanding what you are reading, as results sections are often not written to be easily understood.
Statistical significance and p-values have been critiqued recently for a number of reasons, including that they are misused and misinterpreted (Wasserstein & Lazar, 2016)[77], that researchers deliberately manipulate their analyses to have significant results (Head et al., 2015)[78], and factor into the difficulty scientists have today in reproducing many of the results of previous social science studies (Peng, 2015).[79] For this reason, we share these principles, adapted from those put forth by the American Statistical Association,[80] for understanding and using p-values in social science:
- P-values provide evidence against a null hypothesis.
- P-values do not indicate whether the results were produced by random chance alone or if the researcher's hypothesis is true, though both are common misconceptions.
- Statistical significance can be detected in minuscule differences that have very little effect on the real world.
- Nuance is needed to interpret scientific findings, as a conclusion does not become true or false when the p-value passes from p=0.051 to p=0.049.
- Real-world decision-making must use more than reported p-values. It's easy to run analyses of large datasets and only report the significant findings.
- Greater confidence can be placed in studies that pre-register their hypotheses and share their data and methods openly with the public.
- "By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis. For example, a p-value near 0.05 taken by itself offers only weak evidence against the null hypothesis. Likewise, a relatively large p-value does not imply evidence in favor of the null hypothesis; many other hypotheses may be equally or more consistent with the observed data" (Wasserstein & Lazar, 2016, p. 132).
Confidence intervals
Because of the limitations of p-values, scientists can use other methods to determine whether their models of the world are true. One common approach is to use a confidence interval, or a range of values in which the true value is likely to be found. Confidence intervals are helpful because, as principal #5 above points out, p-values do not measure the size of an effect (Greenland et al., 2016).[81] Remember, something that has very little impact on the world can be statistically significant, and the values in a confidence interval would be helpful. In our example from Table 5.1, imagine our analysis produced a confidence interval that women are 1.2-3.4x more likely to experience "staring or invasion of personal space" than men. As with p-values, calculation for a confidence interval compares what was found in one study with a hypothetical set of results if we repeated the study over and over again. If we calculated 95% confidence intervals for all of the hypothetical set of hundreds and hundreds of studies, that would be our confidence interval.
Confidence intervals are pretty intuitive. As of this writing, my wife and are expecting our second child. The doctor told us our due date was December 11th. But the doctor also told us that December 11th was only their best estimate. They were actually 95% sure our baby might be born any time in the 30-day period between November 27th and December 25th. Confidence intervals are often listed with a percentage, like 90% or 95%, and a range of values, such as between November 27th and December 25th. You can read that as: "we are 95% sure your baby will be born between November 27th and December 25th because we've studied hundreds of thousands of fetuses and mothers, and we're 95% sure your baby will be within these two dates."
Notice that we're hedging our bets here by using words like "best estimate." When testing hypotheses, social scientists generally phrase their findings in a tentative way, talking about what results "indicate" or "support," rather than making bold statements about what their results "prove." Social scientists have humility because they understand the limitations of their knowledge. In a literature review, using a single study or fact to "prove" an argument right or wrong is often a signal to the person reading your literature review (usually your professor) that you may not have appreciated the limitations of that study or its place in the broader literature on the topic. Strong arguments in a literature review include multiple facts and ideas that span across multiple studies.
You can learn more about creating tables, reading tables, and tests of statistical significance in a class focused exclusively on statistical analysis. We provide links to many free and openly licensed resources on statistics in Chapter 16. For now, we hope this brief introduction to reading tables will improve your confidence in reading and understanding the results sections in quantitative empirical articles.
Qualitative results
Quantitative articles will contain a lot of numbers and the results of statistical tests demonstrating associations between those numbers. Qualitative articles, on the other hand, will consist mostly of quotations from participants. For most qualitative articles, the authors want to put their results in the words of their participants, as they are the experts. Articles that lack quotations make it difficult to assess whether the researcher interpreted the data in a trustworthy, unbiased manner. These types of articles may also indicate how often particular themes or ideas came up in the data, potentially reflective of how important they were to participants.
Authors often organize qualitative results by themes and subthemes. For example, see this snippet from the results section in Bonanno and Veselak (2019)[82] discussion parents' attitudes towards child mental health information sources.
Data analysis revealed four themes related to participants’ abilities to access mental health help and information for their children, and parents’ levels of trust in these sources. These themes are: others’ firsthand experiences family and friends with professional experience, protecting privacy, and uncertainty about schools as information sources. Trust emerged as an overarching and unifying concept for all of these themes.
Others’ firsthand experiences. Several participants reported seeking information from other parents who had experienced mental health struggles similar to their own children. They often referenced friends or family members who had been or would be good sources of information due to their own personal experiences. The following quote from Adrienne demonstrates the importance of firsthand experience:
[I would only feel comfortable sharing concerns or asking for advice] if I knew that they had been in the same situation. (Adrienne)
Similarly, Michelle said: And I talked to a friend of mine who has kids who have IEPs in the district to see, kind of, how did she go about it. (Michelle)
...
Friends/family with professional experience. Several respondents referred to friends or family members who had professional experience with or knowledge of child mental health and suggested that these individuals would be good sources of information. For example, Hannah said:
Well, what happened with me was I have an uncle who’s a psychiatrist. Sometimes if he’s up in (a city to the north), he’s retired, I can call him sometimes and get information. (Hannah)
Michelle, who was in nursing school, echoed this sentiment: At this point, [if my child’s behavioral difficulties continued], I would probably call one of my [nursing] professors. That’s what I’ve done in the past when I’ve needed help with certain things...I have a professor who I would probably consider a friend who I would probably talk to first. She has a big adolescent practice. (Michelle) (p. 402-403)
The terms in bold above refer to the key themes (i.e., qualitative results) that were present in the data. Researchers will state the process by which they interpret each theme, providing a definition and usually some quotations from research participants. Researchers will also draw connections between themes, note consensus or conflict over themes, and situate the themes within the study context.
Qualitative results are specific to the time, place, and culture in which they arise, so you will have to use your best judgment to determine whether these results are relevant to your study. For example, students in my class at Radford University in Southwest Virginia may be studying rural populations. Would a study on group homes in a large urban city transfer well to group homes in a rural area?
Maybe. But even if you were using data from a qualitative study in another rural area, are all rural areas the same? How is the client population and sociocultural context in the article similar or different to the one in your study? Qualitative studies have tremendous depth, but researchers must be intentional about drawing conclusions about one context based on a study in another context. To make conclusions about how a study applies in another context, researchers need to examine each component of an empirical journal article--they need to annotate!
Key Takeaways
- The results section of empirical articles are often the most difficult to understand.
- To understand a quantitative results section, look for results that were statistically significant and examine the confidence interval, if provided.
- To understand a qualitative results section, look for definitions of themes or codes and use the quotations provided to understand the participants’ perspective.
Exercises
Select a quantitative empirical article related to your topic.
- Write down the results the authors identify as statistically significant in the results section.
- How do the authors interpret their results in the discussion section?
- Do the authors provide enough information in the introduction for you to understand their results?
Select a qualitative empirical article relevant to your topic.
- Write down the key themes the authors identify and how they were defined by the participants.
- How do the authors interpret their results in the discussion section?
- Do the authors provide enough information in the introduction for you to understand their results?
5.2 Annotating empirical journal articles
Learning Objectives
Learners will be able to...
- Define annotation and describe how to use it to identify, extract, and reflect on the information you need from an article
Annotation refers to the process of writing notes on an article. There are many ways to do this. The most basic technique is to print out the article and build a binder related to your topic. Raul Pacheco-Vega's excellent blog has a post on his approach to taking physical notes. Honestly, while you are there, browse around that website. It is full of amazing tips for students conducting a literature review and graduate research projects. I see a lot of benefits to the paper, pen, and highlighter approach to annotating articles. Personally though, I prefer to use a computer to write notes on an article because my handwriting is terrible and typing notes allows me search for keywords. For other students, electronic notes work best because they cannot afford to print every article that they will use in their paper. No matter what you use, the point is that you need to write notes when you're reading. Reading is research!
There are a number of free software tools you can use to help you annotate a journal article. Most PDF readers like Adobe Acrobat have a commenting and highlighting feature, though the PDF readers included with internet browsers like Google Chrome, Microsoft Edge, and Safari do not have this feature. The best approach may be to use a citation manager like Zotero. Using a citation manager, you can build a library of articles, save your annotations, and link annotations across PDFs using keywords. They also provide integration with word processing programs to help with citations in a reference list
Of course, I don't follow this advice because I have a system that works well for me. I have a PDF open in one computer window and a Word document open in a window next to it. I type notes and copy quotes, listing the page number for each note I take. It's a bit low-tech, but it does make my notes searchable. This way, when I am looking for a concept or quote, I can simply search my notes using the Find feature in Word and get to the information I need.
Annotation and reviewing literature does not have to be a solo project. If are working in a group, you can use the Hypothes.is web browser extension to annotate articles collaboratively. You can also use Google Docs to collaboratively annotate a shared PDF using the commenting feature and write collaborative notes in a shared document. By sharing your highlights and comments, you can split the work of getting the most out of each article you read and build off one another's ideas.
Common annotations
In this section, we present common annotations people make when reading journal articles. These annotations are adapted from Craig Whippo and Raul Pacheco-Vega. If you are annotating on paper, I suggest using different color highlighters for each type of annotation listed below. If you are annotating electronically, you can use the names below as tags to easily find information later. For example, if you are searching for definitions of key concepts, you can either click on the tag for [definitions] in your PDF reader or thumb through a printed copy of article for whatever color or tag you used to indicate definitions of key terms. Most of all, you want to avoid reading through all of your sources again just to find that one thing you know you read somewhere. Time is a graduate student's most valuable resource, so our goal here is to help you spend your time reading the literature wisely.
Personal reflections
Personal reflections are all about you. What do you think? Are there any areas you are confused about? Any new ideas or reflections come to mind while you're reading? Treat these annotations as a means of capturing your first reflections about an article. Write down any questions or thoughts that come to mind as you read. If you think the author says something inaccurate or unsubstantiated, write that down. If you don't understand something, make a note about it and ask your professor. Don't feel bad! Journal articles are hard to understand sometimes, even for professors. Your goal is to critically read the literature, so write down what you think while reading! Table 4.2 contains some questions that might stimulate your thoughts.
Report section | Questions worth asking |
Abstract | What are the key findings? How were those findings reached? How does the author frame their study? |
Acknowledgments | Who are this study’s major stakeholders? Who provided feedback? Who provided support in the form of funding or other resources? |
Problem statement (introduction) | How does the author frame the research focus? What other possible ways of framing the problem exist? Why might the author have chosen this particular way of framing the problem? |
Literature review (introduction) |
What are the major themes the author identifies in the literature? Are there any gaps in the literature? Does the author address challenges or limitations to the studies they cite? Is there enough literature to frame the rest of the article or do you have unanswered questions? Does the author provide conceptual definitions for important ideas or use a theoretical perspective to inform their analysis? |
Sample (methods) | Where was the data collected? Did the researchers provide enough information about the sample and sampling process for you to assess its quality? Did the researchers collect their own data or use someone else’s data? What population is the study trying to make claims about, and does the sample represent that population well? What are the sample’s major strengths and major weaknesses? |
Data collection (methods) | How were the data collected? What do you know about the relative strengths and weaknesses of the methods employed? What other methods of data collection might have been employed, and why was this particular method employed? What do you know about the data collection strategy and instruments (e.g., questions asked, locations observed)? What don’t you know about the data collection strategy and instruments? Look for appendixes and supplementary documents that provide details on measures. |
Data analysis (methods) | How were the data analyzed? Is there enough information provided for you to feel confident that the proper analytic procedures were employed accurately? How open are the data? Can you access the data in an open repository? Did the researchers register their hypotheses and methods prior to data collection? Is there a data disclosure statement available? |
Results | What are the study’s major findings? Are findings linked back to previously described research questions, objectives, hypotheses, and literature? Are sufficient amounts of data (e.g., quotes and observations in qualitative work, statistics in quantitative work) provided to support conclusions? Are tables readable? |
Discussion/conclusion | Does the author generalize to some population beyond the sample? How are these claims presented? Are claims supported by data provided in the results section (e.g., supporting quotes, statistical significance)? Have limitations of the study been fully disclosed and adequately addressed? Are implications sufficiently explored? |
Definitions
Note definitions of key terms for your topic. At minimum, you should include a scholarly definition for the concepts represented in your working question. If your working question asks about the process of leaving a relationship with domestic violence, your research proposal will have to explain how you define domestic violence, as well as how you define "leaving" an abusive relationship. While you may already know what you mean by domestic violence, the person reading your research proposal does not.
Annotating definitions also helps you engage with the scholarly debate around your topic. Definitions are often contested among scholars. Some definitions of domestic violence will be more comprehensive, including things such economic abuse or forcing the victim to problematically use substances. Other definitions will be less comprehensive, covering only physical, verbal, and sexual abuse. Often, how someone defines something conceptually is highly related to how they measure it in their study. Since you will have to do both of these things, find a definition that feels right to you or create your own, noting the ways in which it is similar or different from those in the literature.
Definitions are also an important way of dealing with jargon. Becoming familiar with a new content area involves learning the jargon experts use. For example, in the last paragraph I used the term economic abuse, but that's probably not a term you've heard before. If you were conducting a literature review on domestic violence, you would want to search for keywords like economic abuse if they are relevant to your working question. You will also want to know what they mean so you can use them appropriately in designing your study and writing your literature review.
Theoretical perspective
Noting the theoretical perspective of the article can help you interpret the data in the same manner as the author. For example, articles on supervised injection facilities for people who use intravenous drugs most likely come from a harm reduction perspective, and understanding the theory behind harm reduction is important to make sense of empirical results. Articles should be grounded in a theoretical perspective that helps the author conceptualize and understand the data. As we discussed in Chapter 3, some journal articles are entirely theoretical and help you understand the theories or conceptual models related to your topic. We will help you determine a theoretical perspective for your project in Chapter 7. For now, it's a good idea to note what theories authors mention when talking about your topic area. Some articles are better about this than others, and many authors make it a bit challenging to find theory (if mentioned at all). In other articles, it may help to note which social work theories are missing from the literature. For example, a study's findings might address issues of oppression and discrimination, but the authors may not use critical theory to make sense of what happened.
Background knowledge
It's a good idea to note any relevant information the author relies on for background. When an author cites facts or opinions from others, you are subsequently able to get information from multiple articles simultaneously. For example, if we were looking at this meta-analysis about domestic violence, in the introduction section, the authors provide facts from many other sources. These facts will likely be relevant to your inquiry on domestic violence, as well.
As you are looking at background information, you should also note any subtopics or concepts about which there is controversy or consensus. The author may present one viewpoint and then an opposing viewpoint, something you may do in your literature review as well. Similarly, they may present facts that scholars in the field have come to consensus on and describe the ways in which different sources support these conclusions.
Sources of interest
Note any relevant sources the author cites. If there is any background information you plan to use, note the original source of that information. When you write your literature review, cite the original source of a piece of information you are using, which may not be where you initially read it. Remember that you should read and refer to the primary source. If you are reading Article A and the author cites a fact from Article B, you should note Article B in your annotations and use Article B when you cite the fact in your paper. You should also make sure Article A interpreted Article B correctly and scan Article B for any other useful facts.
Research question/Purpose
Authors should be clear about the purpose of their article. Charitable authors will give you a sentence that starts with something like this:
- "The purpose of this research project was..."
- "Our research question was..."
- "The research project was designed to test the following hypothesis..."
Unfortunately, not all authors are so clear, and you may to hunt around for the research question or hypothesis. Generally, in an empirical article, the research question or hypothesis is at the end of the introduction. In non-empirical articles, the author will likely discuss the purpose of the article in the abstract or introduction.
Results
We will discuss in greater detail how to read the results of empirical articles in Chapter 5. For now, just know that you should highlight any of the key findings of an article. They will be described very briefly in the abstract, and in much more detail in the article itself. In an empirical article, you should look at both the 'Results' and 'Discussion' sections. For a non-empirical article, the key findings will likely be in the conclusion. You can also find them in the topic or concluding sentences in a paragraph within the body of the article.
Measures
How do researchers know something when they see it? Found in the 'Methods' section of empirical articles, the measures section is where researchers spell out the tools, or measures, they used to gather data. For quantitative studies, you will want to get familiar with the questions researchers typically use to measure key variables. For example, to measure domestic violence, researchers often use the Conflict Tactics Scale. The more frequently used and cited a measure is, the more we know about how well it works (or not). Qualitative studies will often provide at least some of the interview or focus group questions they used with research participants. They will also include information about how their inquiry and hypotheses may have evolved over time. Keep in mind however, sometimes important information is cut out of an article during editing. If you need more information, consider reaching out to the author directly. Before you do so, check if the author provided an appendix with the information you need or if the article links to a their data and measures as part open data sharing practices.
Sample
Who exactly were the study participants and how were they recruited? In quantitative studies, you will want to pay attention to the sample size. Generally, the larger the sample, the greater the study's explanatory power. Additionally, randomly drawn samples are desirable because they leave any variation up to chance. Samples that are conducted out of convenience can be biased and non-representative of the larger population. In qualitative studies, non-random sampling is appropriate but consider this: how well does what we find for this group of people transfer to the people who will be in your study? For qualitative studies and quantitative studies, look for how well the sample is described and whether there are important characteristics missing from the article that you would need to determine the quality of the sample.
Limitations
Honest authors will include these at the end of each article. But you should also note any additional limitations you find with their work as well.
Your annotations
These are just a few suggested annotations, but you can come up with your own. For example, maybe there are annotations you would use for different assignments or for the problem statement in your research proposal. If you have an argument or idea that keeps coming to mind when you read, consider creating an annotation for it so you can remember which part of each article supports your ideas. Whatever works for you. The goal with annotation is to extract as much information from each article while reading, so you don't have to go back through everything again. It's useless to read an article and forget most of what you read. Annotate!
Key Takeaways
- Begin your search by reading thorough and cohesive literature reviews. Review articles are great sources of information to get a broad perspective of your topic.
- Don’t read an article just to say you’ve read it. Annotate and take notes so you don’t have to re-read it later.
- Use software or paper-and-pencil approaches to write notes on articles.
- Annotation is best used when closely reading an empirical study highly similar to your research project.
Exercises
- Select an empirical article highly related to the study you would like to conduct.
- Annotate the article using the aforementioned annotations and create some of your own.
- Create the first draft of a summary table with key information from this empirical study that you would like to compare to other empirical studies you closely read.
5.3 Generalizability and transferability of empirical results
Learning Objectives
Learners will be able to...
- Define generalizability and transferability.
- Assess the generalizability and transferability to how researchers use the results from empirical research studies to make arguments about what is objectively true.
- Relate both concepts to the hierarchy of evidence and the types of articles in the scholarly literature
Now that you have read an empirical article in detail, it's important to put its results in conversation with the broader literature on your topic. In this chapter we discuss two important concepts--generalizability and transferability--and the interrelationship between the two. We also explain how these two properties of empirical data impact your literature review and evidence-based practice.
Generalizability
The figure below provides a common approach to assessing empirical evidence. As you move up the pyramid below, you can be more sure that the data contained in those studies generalizes to all people who experience the issue.
As we reviewed in Chapter 1, objective truth is true for everyone, regardless of context. In other words, objective truths generalize beyond the sample of people from whom data were collected to the larger population of people who experience the issue under examination. You can be much more sure that information from a systematic review or meta-analysis will generalize than something from a case study of a single person, pilot projects, and other studies that do not seek to establish generalizability.
The type of article listed here is also related to the types of research methods the authors used. While we cover many of these approaches in this textbook, some of them (like cohort studies) are somewhat less common in social work. Additionally, there is one important research method, survey design, that does not appear in this diagram. Finally, social work research uses many different types of qualitative research--some of which generates more generalizable data than others.
For a refresher on the different types of evidence available in each type of article, refer back to section 4.1. You'll recall the hierarchy of evidence as described by McNeese & Thyer (2004)[83]
- Systematic reviews and meta-analyses
- Randomized controlled trials
- Quasi-experimental studies
- Case-control and cohort studies
- Pre-experimental (or non-experimental) group studies
- Surveys
- Qualitative studies
Because there is further variation in the types of studies used by social work researchers, I expanded the hierarchy of evidence to cover a greater breadth of research methods in Figure 5.3.
Refined information from multiple sources
The top of the hierarchy represents refined scientific information or meta-research. Meta-research uses the scientific method to analyze and improve the scientific production of knowledge. For example, meta-analyses pull together samples of people from all high-quality studies on a given topic area creating a super-study with far more people than any single researcher could feasibly collect data from. Because scientists (and clinical experts) refine data across multiple studies, these represent the most generalizable research findings.
Of course, not all meta-analyses or systematic reviews are of good quality. As a peer reviewer for a scholarly journal, I have seen poor quality systematic reviews that make methodological mistakes—like not including relevant keywords—that lead to incorrect conclusions. Unfortunately, not all errors are caught in the peer review process, and not all limitations are acknowledged by the authors. Just because you are looking at a systematic review does not mean you are looking at THE OBJECTIVE TRUTH. Nevertheless, you can be pretty sure that results from these studies are generalizable to the population in the study’s research question.
A good way to visualize the process of sampling is by examining the procedure used for systematic reviews and meta-analyses to scientifically search for articles. In Figure 5.4 below, you can see how researchers conducting a systematic review identified a large pool of potentially relevant articles, downloaded and analyzed them for relevance, and in the end, analyzed only 71 articles in their systematic review out of a total of 1,589 potentially relevant articles. Because systematic reviews or meta-analyses are intended to make strong, generalizable conclusions, they often exclude studies that still contain good information.
In the process of selecting articles for a meta-analysis and systematic review, researchers may exclude articles with important information for a number of good reasons. No study is perfect, and all research methods decisions come with limitations--including meta-research. Authors conducting a meta-analysis cannot include a study unless researchers provide data for the authors to include in their meta-analysis, and many empirical journal articles do not make their data available. Additionally, a study’s intervention or measures may be a bit different than what researchers want to make conclusions about. This is a key truth applicable across all articles you read—who or what gets selected for analysis in a research project determines how well the project’s results generalize to everyone.
We will talk about this in future chapters as sampling, and in those chapters, we will learn which sampling approaches are intended to support generalizability and which are used for other purposes. For example, availability or convenience sampling is often used to get quick information while random sampling approaches are intended to support generalizability. It is impossible to know everything about your article right now, but by the end of this course, you will have the information you need to critically examine the generalizability of a sample.
Primary sources (empirical studies)
Because refined sources like systematic reviews exclude good studies, they are only a first step in getting to know a topic area. You will need to examine primary sources--the reports of researchers who conducted empirical studies--to make evidence-based conclusions about your topic. Figure 5.3 describes three different types of data and ranks them vertically based on how well you can be sure the information generalizes.
As we will discuss further in our chapter on causal explanations, a key factor in scientifically assessing what happened first. Researchers conducting intervention studies are causing change by providing therapy, housing, or whatever the intervention is and measuring the outcomes of that intervention after they happen. This is unlike survey researchers, who do not introduce an intervention but ask people to self-report information on a questionnaire. Longitudinal surveys are particularly helpful because they can provide a clearer picture of whether the cause came before the effect in a causal relationship, but because they are expensive and time-consuming to conduct, longitudinal studies are relatively rare in the literature and most surveys measure people at only one point in time. Thus, because researchers cannot tightly control the causal variable (an intervention, an experience of abuse, etc.) we can be somewhat less certain of the conclusions of surveys than experiments. At the same time, because surveys measure people in their naturalistic environment rather than in a laboratory or artificial setting, they may do a better job at reducing the potential for the researcher to influence the data a participant provides. Surveys also provide descriptive information--like the number of people with a diagnosis or risk factor--that experiments cannot provide.
Surveys and experiments are commonly used in social work, and we will describe the methods they use in future chapters. When assessing the generalizability of a given survey or experiment, you are looking at whether the methods used by the researchers improve generalizability (or, at least that those methods are intended to improve generalizability). Specifically, there are sampling, measurement, and design decisions that researchers make that can improve generalizability. And once the study is conducted, whether those methods worked as intended also impact generalizability.
We address sampling, measurement, and design in the coming chapters, and you will need more in-depth knowledge of research methods to assess the generalizability of the results you are reading. In the meantime, Figure 5.3 is organized by design, and this is a good starting point for your inquiry since it only requires you to identify the design in each empirical article--which should be included in the abstract and described in detail in the methods section. For more information on how to conduct sampling, measurement, and design in a way that maximizes generalizability, read Part 2 of this textbook.
When searching for design of a study, look for specific keywords that indicate the researcher used methods that do not generalize well like pilot study, pre-experiment, non-experiment, convenience sample, availability sample, and exploratory study. When researchers are seeking to perform a pilot study, they are optimizing for time, not generalizability. Their results may still be useful to you! But, you should not generalize from their study to all people with the issue under analysis without a lot of caution and additional supporting evidence. Instead, you should see whether the lessons from this study might transfer to the context in which you are researching--our next topic.
Qualitative studies use sampling, measures, and designs that do not try to optimize generalizability. Thus, if the results of a qualitative study indicate 10 out of 50 students who participated in the focus group found the mandatory training on harassment to be unhelpful, does that mean 20% of all college students at this university find it unhelpful? Because focus groups and interviews (and other qualitative methods we will discuss) use qualitative methods, they are not concerned with generalizability. It would not make sense to generalize from focus groups to all people in a population. Instead, focus groups methods optimize for trustworthy and authentic research projects that make sure, for example, all themes and quotes in the researcher's report are traceable to quotes from focus group participants. Instead of providing what is generally true, qualitative research provides a thick description of people's experiences so you can understand them. Subjective inquiry is less generalizable but provides greater depth in understanding people's feelings, beliefs, and decision-making processes within their context.
In Figure 5.3, you will note that some qualitative studies are ranked higher than others in terms of generalizability. Meta-syntheses are ranked highest because they are meta-research, pooling together the themes and raw data from multiple qualitative studies into a super-study. A meta-synthesis is the qualitative equivalent of a meta-analysis, which analyzes quantitative data. Because the researchers conducting the meta-syntheses aim to make more broad generalizations across research studies, even though generalizability is not strictly the goal. In a similar way, grounded theory studies (a type of qualitative design) aim to produce a testable hypothesis that could generalize. At the bottom of the hierarchy are individual case studies, which report what happens with a single person, organization, or event. It's best not to think too long about the generalizability of qualitative results. When examining qualitative articles, you should be examining their transferability, our topic for the next subsection.
Transferability
Generalizability asks one question: How well does the sample of people in this study represent everyone with this issue? If you read in a study that 50% of people in the sample experienced depression, does that mean 50% of everyone experiences depression? We previewed future discussions in this textbook that will discuss the specific quantitative research methods used to optimize the generalizability of results. By adhering strictly to best practices in sampling, measurement, and design, researchers can provide you with good evidence for the generalizability of their study's results.
Of course, generalizability is not the only question worth asking. Just because a study's sample represents a broader population does not mean it is helpful for making conclusions about your working question. In assessing a study's transferability, you are making a weaker but compelling argument that the conclusions of one study can be applied to understanding the people in your working question and research project. Generalizable results may be applicable because they are broadly transferable across situations, and you can be confident in that when they follow the best practices in this textbook for improving generalizability. However, there may be aspects of a study that make its results difficult to transfer to your topic area.
When evaluating the transferability of a research result to your working question, consider the sample, measures, and design. That is, how data was collected from individuals, who those individuals are, and what researchers did with them. You may find that the samples in generalizable studies do not talk about the specific ethnic, cultural, or geographic group that is in your working question. Similarly, studies that measure the outcomes of substance use treatment by measuring sobriety may not match your working question on moderation, medication adherence, or substitution as an outcome in substance use treatment. Evaluating the transferability of designs may help you identify whether the methods the authors used would be similar to those you might use if you were to conduct a study gathering and collecting your own raw data.
Assessing transferability is more subjective. You are using your knowledge of your topic area and research methods (which are always improving!) to make a reasonable argument about why a given piece of evidence from a primary source helps you understand something. Look back at Table 5.2, your annotations, and the researchers' sampling, data analysis, results, and design. Using your critical thinking (and the knowledge you can in Part 2 and Part 3 of this textbook) you will need to make a reasonable argument that these results transfer to the people, places, and culture that you are talking about in your working question.
In the final chapter of Part 1, we will discuss how to assemble the facts you have taken from journal articles into a literature review that represents what you think about the topic.
Key Takeaways
- Begin your search by reading thorough and cohesive literature reviews. Review articles are great sources of information to get a broad perspective of your topic.
- Don’t read an article just to say you’ve read it. Annotate and take notes so you don’t have to re-read it later.
- Use software or paper-and-pencil approaches to write notes on articles.
- Annotation is best used when closely reading an empirical study highly similar to your research project.
Exercises
- Select an empirical article highly related to the study you would like to conduct.
- Annotate the article using the aforementioned annotations and create some of your own.
- Create the first draft of a summary table with key information from this empirical study that you would like to compare to other empirical studies you closely read.
Chapter Outline
- Human subjects research (19 minute read)
- Specific ethical issues to consider (12 minute read)
- Benefits and harms of research across the ecosystem (7 minute read)
- Being an ethical researcher (8 minute read)
Content warning: examples in this chapter contain references to numerous incidents of unethical medical experimentation (e.g. intentionally injecting diseases into unknowing participants, withholding proven treatments), social experimentation under extreme conditions (e.g. being directed to deliver electric shocks to test obedience), violations of privacy, gender and racial inequality, research with people who are incarcerated or on parole, experimentation on animals, abuse of people with Autism, community interactions with law enforcement, WWII, the Holocaust, and Nazi activities (especially related to research on humans).
With your literature review underway, you are ready to begin thinking in more concrete terms about your research topic. Recall our discussion in Chapter 2 on practical and ethical considerations that emerge as part of the research process. In this chapter, we will expand on the ethical boundaries that social scientists must abide by when conducting human subjects research. As a result of reading this chapter, you should have a better sense of what is possible and ethical for the research project you create.
6.1 Human subjects research
Learning Objectives
Learners will be able to...
- Understand what we mean by ethical research and why it is important
- Understand some of the egregious ethical violations that have occurred throughout history
While all research comes with its own set of ethical concerns, those associated with research conducted on human subjects vary dramatically from those of research conducted on nonliving entities. The US Department of Health and Human Services (USDHHS) defines a human subject as “a living individual about whom an investigator (whether professional or student) conducting research obtains (1) data through intervention or interaction with the individual, or (2) identifiable private information” (USDHHS, 1993, para. 1).[84] Some researchers prefer the term "participants" to "subjects'" as it acknowledges the agency of people who participate in the study. For our purposes, we will use the two terms interchangeably.
In some states, human subjects also include deceased individuals and human fetal materials. Nonhuman research subjects, on the other hand, are objects or entities that investigators manipulate or analyze in the process of conducting research. Nonhuman research subjects typically include sources such as newspapers, historical documents, pieces of clothing, television shows, buildings, and even garbage (to name just a few), that are analyzed for unobtrusive research projects. Unsurprisingly, research on human subjects is regulated much more heavily than research on nonhuman subjects. This is why many student research projects use data that is publicly available, rather than recruiting their own study participants. However, there are ethical considerations that all researchers must take into account, regardless of their research subject. We’ll discuss those considerations in addition to concerns that are unique to human subject research.
Why do research participants need protection?
First and foremost, we are professionally bound to engage in the ethical practice of research. This chapter discusses ethical research and will show you how to engage in research that is consistent with the NASW Code of Ethics as well as national and international ethical standards all researchers are accountable to. Before we begin, we need to understand the historical occurrences that were the catalyst for the formation of the current ethical standards. This chapter will enable you to view ethics from a micro, mezzo, and macro perspective.
The research process has led to many life-changing discoveries; these have improved life expectancy, improved living conditions, and helped us understand what contributes to certain social problems. That said, not all research has been conducted in respectful, responsible, or humane ways. Unfortunately, some research projects have dramatically marginalized, oppressed, and harmed participants and whole communities.
Would you believe that the following actions have been carried out in the name of research? I realize there was a content warning at the beginning of the chapter, but it is worth mentioning that the list below of research atrocities may be particularly upsetting or triggering.
- intentionally froze healthy body parts of prisoners to see if they could develop a treatment for freezing[85]
- scaled the body parts of prisoners to how best to treat soldiers who had injuries from being exposed to high temperatures[86]
- intentionally infected healthy individuals to see if they could design effective methods of treatment for infections[87]
- gave healthy people TB to see if they could treat it[88]
- attempted to transplant limbs, bones, and muscles to another person to see if this was possible[89]
- castrated and irradiated genitals to see if they could develop a faster method of sterilization[90]
- starved people and only allowed them to drink seawater to see if they could make saline water drinkable[91]
- artificially inseminated women with animal sperm to see what would happen[92]
- gassed living people to document how they would die[93]
- conducted cruel experiments on people and if they did not die, would kill them so they could undergo an autopsy[94]
- refused to treat syphilis in African American men (when treatment was available) because they wanted to track the progression of the illness[95]
- vivisected humans without anesthesia to see how illnesses that researches gave prisoners impacted their bodies[96]
- intentionally tried to infect prisoners with the Bubonic Plague[97]
- intentionally infected prisoners, prostitutes, soldiers, and children with syphilis to study the disease's progression[98]
- performed gynecological experiments on female slaves without anesthesia to investigate new surgical methods[99]
The sad fact is that not only did all of these occur, in many instances, these travesties continued for years until exposed and halted. Additionally, these examples have contributed to the formation of a legacy of distrust toward research. Specifically, many underrepresented groups have a deep distrust of agencies that implement research and are often skeptical of research findings. This has made it difficult for groups to support and have confidence in medical treatments, advances in social service programs, and evidence-informed policy changes. While the aforementioned unethical examples may have ended, this deep and painful wound on the public's trust remains. Consequently, we must be vigilant in our commitment to ethical research.
Many of the situations described may seem like extreme historical cases of misuse of power as researchers. However, ethical problems in research don't just happen in these extreme occurrences. None of us are immune to making unethical choices and the ethical practice of research requires conscientious mindful attention to what we are asking of our research participants. A few examples of less noticeable ethical issues might include: failing to fully explain to someone in advance what their participation might involve because you are in a rush to recruit a large enough sample; or only presenting findings that support your ideas to help secure a grant that is relevant to your research area. Remember, any time research is conducted with human beings, there is the chance that ethical violations may occur that pose social, emotional, and even physical risks for groups, and this is especially true when vulnerable or oppressed groups are involved.
A brief history of unethical social science research
Research on humans hasn’t always been regulated in the way it is today. The earliest documented cases of research using human subjects are of medical vaccination trials (Rothman, 1987).[100] One such case took place in the late 1700s, when scientist Edward Jenner exposed an 8-year-old boy to smallpox in order to identify a vaccine for the devastating disease. Medical research on human subjects continued without much law or policy intervention until the mid-1900s when, at the end of World War II, a number of Nazi doctors and scientists were put on trial for conducting human experimentation during the course of which they tortured and murdered many concentration camp inmates (Faden & Beauchamp, 1986).[101] The trials, conducted in Nuremberg, Germany, resulted in the creation of the Nuremberg Code, a 10-point set of research principles designed to guide doctors and scientists who conduct research on human subjects. Today, the Nuremberg Code guides medical and other research conducted on human subjects, including social scientific research.
Medical scientists are not the only researchers who have conducted questionable research on humans. In the 1960s, psychologist Stanley Milgram (1974)[102] conducted a series of experiments designed to understand obedience to authority in which he tricked subjects into believing they were administering an electric shock to other subjects. In fact, the shocks weren’t real at all, but some, though not many, of Milgram’s research participants experienced extreme emotional distress after the experiment (Ogden, 2008).[103] A reaction of emotional distress is understandable. The realization that one is willing to administer painful shocks to another human being just because someone who looks authoritative has told you to do so might indeed be traumatizing—even if you later learn that the shocks weren’t real.
Around the same time that Milgram conducted his experiments, sociology graduate student Laud Humphreys (1970)[104] was collecting data for his dissertation on the tearoom trade, which was the practice of men engaging in anonymous sexual encounters in public restrooms. Humphreys wished to understand who these men were and why they participated in the trade. To conduct his research, Humphreys offered to serve as a “watch queen,” in a local park restroom where the tearoom trade was known to occur. His role would be to keep an eye out for police while also getting the benefit of being able to watch the sexual encounters. What Humphreys did not do was identify himself as a researcher to his research subjects. Instead, he watched his subjects for several months, getting to know several of them, learning more about the tearoom trade practice and, without the knowledge of his research subjects, jotting down their license plate numbers as they pulled into or out of the parking lot near the restroom.
Sometime after participating as a watch queen, with the help of several insiders who had access to motor vehicle registration information, Humphreys used those license plate numbers to obtain the names and home addresses of his research subjects. Then, disguised as a public health researcher, Humphreys visited his subjects in their homes and interviewed them about their lives and their health. Humphreys’ research dispelled a good number of myths and stereotypes about the tearoom trade and its participants. He learned, for example, that over half of his subjects were married to women and many of them did not identify as gay or bisexual.[105]
Once Humphreys’ work became public, there was some major controversy at his home university (e.g., the chancellor tried to have his degree revoked), among scientists in general, and among members of the public, as it raised public concerns about the purpose and conduct of social science research. In addition, the Washington Post journalist Nicholas von Hoffman wrote the following warning about “sociological snoopers”:
We’re so preoccupied with defending our privacy against insurance investigators, dope sleuths, counterespionage men, divorce detectives and credit checkers, that we overlook the social scientists behind the hunting blinds who’re also peeping into what we thought were our most private and secret lives. But they are there, studying us, taking notes, getting to know us, as indifferent as everybody else to the feeling that to be a complete human involves having an aspect of ourselves that’s unknown (von Hoffman, 1970).[106]
In the original version of his report, Humphreys defended the ethics of his actions. In 2008[107], years after Humphreys’ death, his book was reprinted with the addition of a retrospect on the ethical implications of his work. In his written reflections on his research and the fallout from it, Humphreys maintained that his tearoom observations constituted ethical research on the grounds that those interactions occurred in public places. But Humphreys added that he would conduct the second part of his research differently. Rather than trace license numbers and interview unwitting tearoom participants in their homes under the guise of public health research, Humphreys instead would spend more time in the field and work to cultivate a pool of informants. Those informants would know that he was a researcher and would be able to fully consent to being interviewed. In the end, Humphreys concluded “there is no reason to believe that any research subjects have suffered because of my efforts, or that the resultant demystification of impersonal sex has harmed society” (Humphreys, 2008, p. 231).[108]
Today, given increasing regulation of social scientific research, chances are slim that a researcher would be allowed to conduct a project similar to Humphreys’. Some argue that Humphreys’ research was deceptive, put his subjects at risk of losing their families and their positions in society, and was therefore unethical (Warwick, 1973; Warwick, 1982).[109] Others suggest that Humphreys’ research “did not violate any premise of either beneficence or the sociological interest in social justice” and that the benefits of Humphreys’ research, namely the dissolution of myths about the tearoom trade specifically and human sexual practice more generally, outweigh the potential risks associated with the work (Lenza, 2004, p. 23).[110] What do you think, and why?
These and other studies (Reverby, 2009)[111] led to increasing public awareness of and concern about research on human subjects. In 1974, the US Congress enacted the National Research Act, which created the National Commission for the Protection of Human Subjects in Biomedical and Behavioral Research. The commission produced The Belmont Report, a document outlining basic ethical principles for research on human subjects (National Commission for the Protection of Human Subjects in Biomedical and Behavioral Research, 1979).[112] The National Research Act (1974)[113] also required that all institutions receiving federal support establish institutional review boards (IRBs) to protect the rights of human research subjects. Since that time, many organizations that do not receive federal support but where research is conducted have also established review boards to evaluate the ethics of the research that they conduct. IRBs are overseen by the federal Office of Human Research Protections.
The Belmont Report
As mentioned above, The Belmont Report is a federal document that outlines the foundational principles that guide the ethical practice of research in the United States. These ethical principles include: respect for persons, beneficence, and justice. Each of these terms has specific implications as they are applied to the practice of research. These three principles arose as a response to many of the mistreatment and abuses that have been previously discussed and provide important guidance as researchers consider how they will construct and conduct their research studies. As you are crafting your research proposal, makes sure you are mindful of these important ethical guidelines.
Respect for Persons
As social workers, our professional code of ethics requires that we recognize and respect the "inherent dignity and worth of the person."[114] This is very similar to the ethical research principle of respect for persons. According to this principle, as researchers, we need to treat all research participants with respect, dignity and inherent autonomy. This is reflected by ensuring that participants have self-determination to make informed decisions about their participation in research, that they have a clear understanding of what they will be asked to do and any risks involved, and that their participation is voluntary and can be stopped at any time. Furthermore, for those persons who may have diminished autonomy (e.g. children, people who are incarcerated), extra protections must be built in to these research studies to ensure that respect for persons continues to be demonstrated towards these groups, as they may be especially vulnerable to exploitation and coercion through the research process. A critical tool in establishing respect for persons in your research is the informed consent process, which will be discussed in more detail below.
Beneficence
You may not be familiar with this word yet, but the concept is pretty straightforward. The main idea with beneficence is that the intent of research is to do good. As researchers, to accomplish this, we seek to maximize benefits and minimize risks. Benefits may be something good or advantageous directly received by the research participant, or they may represent a broader good to a wider group of people or the scientific community at large (such as increasing knowledge about the topic or social problem that you are studying). Risks are potential physical, social, or emotional harm that may come about as a response to participation in a study. These risks may be more immediate (e.g. risk of identifying information about a participant being shared, or a participant being upset or triggered by a particular question), or long-term (e.g. some aspect about the person could be shared that could lead to long-term stigmatization). As researchers, we need to think about risk that might be experienced by the individual, but also risks that might be directed towards the community or population(s) the individual may represent. For instance, if our study is specifically focused on surveying single parents, we need to consider how the sharing of our findings might impact this group and how they are perceived. It is a very rare study in which there is no risk to participants. However, a well-designed and ethically sound study will seek to minimize these risks, provide resources to anticipate and address them, and maximize the benefits that are gained through the study.
Justice
The final ethical principle we need to cover is justice. While you likely have some idea what justice is, for the purposes of research, justice is the idea that the benefits and the burdens of research are distributed fairly across populations and groups. To help illustrate the concept of justice in research, research in the area of mental health and psychology has historically been critiqued as failing to adequately represent women and people of diverse racial and ethnic groups in their samples (Cundiff, 2012).[115] This has created a body of knowledge that is overly representative of the white male experience, further reinforcing systems of power and privilege. In addition, consider the influence of language as it relates to research justice. If we create studies that only recruit participants fluent in English, which many studies do, we are often failing to satisfy the ethical principle of justice as it applies to people who don't speak English. It is unrealistic to think that we can represent all people in all studies. However, we do need to thoughtfully acknowledge voices that might not be reflected in our samples and attempt to recruit diverse and representative samples whenever possible.
These three principles provide the foundation for the oversight work that is carried out by Institutional Review Boards, our next topic.
Institutional review boards
Institutional review boards, or IRBs, are tasked with ensuring that the rights and welfare of human research subjects will be protected at all institutions, including universities, hospitals, nonprofit research institutions, and other organizations, that receive federal support for research. IRBs typically consist of members from a variety of disciplines, such as sociology, economics, education, social work, and communications (to name a few). Most IRBs also include representatives from the community in which they reside. For example, representatives from nearby prisons, hospitals, or treatment centers might sit on the IRBs of university campuses near them. The diversity of membership helps to ensure that the many and complex ethical issues that may arise from human subjects research will be considered fully and by a knowledgeable and experienced panel. Investigators conducting research on human subjects are required to submit proposals outlining their research plans to IRBs for review and approval prior to beginning their research. Even students who conduct research on human subjects must have their proposed work reviewed and approved by the IRB before beginning any research (though, on some campuses, exceptions are made for student projects that will not be shared outside of the classroom).
The IRB has three levels of review, defined in statute by the USDHHS.
Exempt
Exempt review is the lowest level of review. Studies that are considered exempt expose participants to the least potential for harm and often involve little participation by human subjects. In social work, exempt studies often examine data that is publicly available or secondary data from another researcher that has been de-identified by the person who collected it.
Expedited
Expedited review is the middle level of review. Studies considered under expedited review do not have to go before the full IRB board because they expose participants to minimal risk. However, the studies must be thoroughly reviewed by a member of the IRB committee. While there are many types of studies that qualify for expedited review, the most relevant to social workers include the use of existing medical records, recordings (such as interviews) gathered for research purposes, and research on individual group characteristics or behavior.
Full board
Finally, the highest level of review is called a full board review. A full board review will involve multiple members of the IRB evaluating your proposal. When researchers submit a proposal under full board review, the full IRB board will meet, discuss any questions or concerns with the study, invite the researcher to answer questions and defend their proposal, and vote to approve the study or send it back for revision. Full board proposals pose greater than minimal risk to participants. They may also involve the participation of vulnerable populations, or people who need additional protection from the IRB. Vulnerable populations include prisoners, children, people with cognitive impairments, people with physical disabilities, employees, and students. While some of these populations can fall under expedited review in some cases, they will often require the full IRB to approve their study.
It may surprise you to hear that IRBs are not always popular or appreciated by researchers. Who wouldn’t want to conduct ethical research, you ask? In some cases, the concern is that IRBs are most well-versed in reviewing biomedical and experimental research, neither of which is particularly common within social work. Much social work research, especially qualitative research, is open-ended in nature, a fact that can be problematic for IRBs. The members of IRBs often want to know in advance exactly who will be observed, where, when, and for how long, whether and how they will be approached, exactly what questions they will be asked, and what predictions the researcher has for their findings. Providing this level of detail for a year-long participant observation within an activist group of 200-plus members, for example, would be extraordinarily frustrating for the researcher in the best case and most likely would prove to be impossible. Of course, IRBs do not intend to have researchers avoid studying controversial topics or avoid using certain methodologically sound data collection techniques, but unfortunately, that is sometimes the result. The solution is not to eradicate review boards, which serve a necessary and important function, but instead to help educate IRB members about the variety of social scientific research methods and topics covered by social workers and other social scientists.
What we have provided here is only a short summary of federal regulations and international agreements that provide the boundaries between ethical and unethical research.
Here are a few more detailed guides for continued learning about research ethics and human research protections.
- University of California, San Francisco: Levels of IRB Review
- United States Department of Health and Human Services: The Belmont Report
- NIH, National Institute of Environmental Health Sciences: What is Ethics in Research & Why is it important
- NIH: Guiding Principles for Ethical Research
- Council on Social Work Education: National Statement on Research Integrity in Social Work
- Butler, I. (2002). A code of ethics for social work and social care research. British Journal of Social Work, 32(2), 239-248
Key Takeaways
- Research on human subjects presents a unique set of challenges and opportunities when it comes to conducting ethical research.
- Research on human subjects has not always been regulated to the extent that it is today.
- All institutions receiving federal support for research must have an IRB. Organizations that do not receive federal support but where research is conducted also often include IRBs as part of their organizational structure.
- Researchers submit studies for IRB review at one of three different levels, depending on the level of harm the study may cause.
Exercises
- Recall whether your project will gather data from human subjects and sketch out what the data collection process might look like.
- Identify which level of IRB review is most appropriate for your project.
- For many students, your professors may have existing agreements with your university's IRB that allow students to conduct research projects outside the supervision of the IRB. Make sure that your project falls squarely within those parameters. If you feel you may be outside of such an agreement, consult with your professor to see if you will need to submit your study for IRB review before starting your project.
6.2 Specific ethical issues to consider
Learning Objectives
Learners will be able to...
- Define informed consent, and describe how it works
- Identify the unique concerns related to the study of vulnerable populations
- Differentiate between anonymity and confidentiality
- Explain the ethical responsibilities of social workers conducting research
As should be clear by now, conducting research on humans presents a number of unique ethical considerations. Human research subjects must be given the opportunity to consent to their participation in research, and be fully informed of the study’s risks, benefits, and purpose. Further, subjects’ identities and the information they share should be protected by researchers. Of course, how consent and identity protection are defined may vary by individual researcher, institution, or academic discipline. In this section, we’ll take a look at a few specific topics that individual researchers must consider before embarking on research with human subjects.
Informed consent
An expectation of voluntary participation is presumed in all social work research projects. In other words, we cannot force anyone to participate in our research without that person’s knowledge or consent. Researchers must therefore design procedures to obtain subjects’ informed consent to participate in their research. This specifically relates back to the ethical principle of respect for persons outlined in The Belmont Report. Informed consent is defined as a subject’s voluntary agreement to participate in research based on a full understanding of the research and of the possible risks and benefits involved. Although it sounds simple, ensuring that one has actually obtained informed consent is a much more complex process than you might initially presume.
The first requirement is that, in giving their informed consent, subjects may neither waive nor even appear to waive any of their legal rights. Subjects also cannot release a researcher, her sponsor, or institution from any legal liability should something go wrong during the course of their participation in the research (USDHHS,2009).[116] Because social work research does not typically involve asking subjects to place themselves at risk of physical harm by, for example, taking untested drugs or consenting to new medical procedures, social work researchers do not often worry about potential liability associated with their research projects. However, their research may involve other types of risks.
For example, what if a social work researcher fails to sufficiently conceal the identity of a subject who admits to participating in a local swinger’s club? In this case, a violation of confidentiality may negatively affect the participant’s social standing, marriage, custody rights, or employment. Social work research may also involve asking about intimately personal topics that may be difficult for participants to discuss, such as trauma or suicide. Participants may re-experience traumatic events and symptoms when they participate in your study. Even if you are careful to fully inform your participants of all risks before they consent to the research process, I’m sure you can empathize with thinking you could bear talking about a difficult topic and then finding it too overwhelming once you start. In cases like these, it is important for a social work researcher to have a plan to provide supports. This may mean providing referrals to counseling supports in the community or even calling the police if the participant is an imminent danger to himself or others.
It is vital that social work researchers explain their mandatory reporting duties in the consent form and ensure participants understand them before they participate. Researchers should also emphasize to participants that they can stop the research process at any time or decide to withdraw from the research study for any reason. Importantly, it is not the job of the social work researcher to act as a clinician to the participant. While a supportive role is certainly appropriate for someone experiencing a mental health crisis, social workers must ethically avoid dual roles. Referring a participant in crisis to other mental health professionals who may be better able to help them is the expectation.
Beyond the legal issues, most IRBs require researchers to share some details about the purpose of the research, possible benefits of participation, and, most importantly, possible risks associated with participating in that research with their subjects. In addition, researchers must describe how they will protect subjects’ identities, how, where, and for how long any data collected will be stored, how findings may be shared, and whom to contact for additional information about the study or about subjects’ rights. All this information is typically shared in an informed consent form that researchers provide to subjects. In some cases, subjects are asked to sign the consent form indicating that they have read it and fully understand its contents. In other cases, subjects are simply provided a copy of the consent form and researchers are responsible for making sure that subjects have read and understand the form before proceeding with any kind of data collection. Your IRB will often provide guidance or even templates for what they expect to see included in an informed consent form. This is a document that they will inspect very closely. Table 6.1 outlines elements to include in your informed consent. While these offer a guideline for you, you should always visit your schools, IRB website to see what guidance they offer. They often provide a template that they prefer researchers to use. Using these templates ensures that you are using the language that the IRB reviewers expect to see and this can also save you time.
Elements | Brief description |
Welcome | A greeting for your participants, a few words about who you/your team are, the aim of your study |
Procedures | What your participants are being asked to do throughout the entire research process |
Risks | Any potential risks associated with your study (this is very rarely none!); also, make sure to provide resources that address or mitigate the risks (e.g. counseling services, hotlines, EAP) |
Benefits | Any potential benefits, either direct to participant or more broadly (indirect) to community or group; include any compensation here, as well |
Privacy | Brief explanation of steps taken to protect privacy.; address confidentiality or anonymity (whichever applies); also address how the results of the study may be used/disseminated |
Voluntary Nature | It is important to emphasize that participation is voluntary and can be stopped at any time |
Contact Information | You will provide your contact information as the researcher and often the contact of the IRB that is providing approval for the study |
Signatures | We will usually seek the signature and date of participant and researcher on these forms (unless otherwise specified and approved in your IRB application) |
One last point to consider when preparing to obtain informed consent is that not all potential research subjects are considered equally competent or legally allowed to consent to participate in research. Subjects from vulnerable populations may be at risk of experiencing undue influence or coercion (USDHHS, 2009).[117] The rules for consent are more stringent for vulnerable populations. For example, minors must have the consent of a legal guardian in order to participate in research. In some cases, the minors themselves are also asked to participate in the consent process by signing special, age-appropriate assent forms designed specifically for them. Prisoners and parolees also qualify as vulnerable populations. Concern about the vulnerability of these subjects comes from the very real possibility that prisoners and parolees could perceive that they will receive some highly desired reward, such as early release, if they participate in research or that there could be punitive consequences if they choose not to participate. When a participant faces undue or excess pressure to participate by either favorable or unfavorable means, this is known as coercion and must be avoided by researchers.
Another potential concern regarding vulnerable populations is that they may be underrepresented or left out of research opportunities, specifically because of concerns about their ability to consent. So, on the one hand, researchers must take extra care to ensure that their procedures for obtaining consent from vulnerable populations are not coercive. The procedures for receiving approval to conduct research with these groups may be more rigorous than that for non-vulnerable populations. On the other hand, researchers must work to avoid excluding members of vulnerable populations from participation simply on the grounds that they are vulnerable or that obtaining their consent may be more complex. While there is no easy solution to this ethical research dilemma, an awareness of the potential concerns associated with research on vulnerable populations is important for identifying whatever solution is most appropriate for a specific case.
Protection of identities
As mentioned earlier, the informed consent process includes the requirement that researchers outline how they will protect the identities of subjects. This aspect of the research process, however, is one of the most commonly misunderstood. Furthermore, failing to protect identities is one of the greatest risks to participants in social work research studies.
In protecting subjects’ identities, researchers typically promise to maintain either the anonymity or confidentiality of their research subjects. These are two distinctly different terms and they are NOT interchangeable. Anonymity is the more stringent of the two and is very hard to guarantee in most research studies. When a researcher promises anonymity to participants, not even the researcher is able to link participants’ data with their identities. Anonymity may be impossible for some social work researchers to promise due to the modes of data collection many social workers employ. Face-to-face interviewing means that subjects will be visible to researchers and will hold a conversation, making anonymity impossible. In other cases, the researcher may have a signed consent form or obtain personal information on a survey and will therefore know the identities of their research participants. In these cases, a researcher should be able to at least promise confidentiality to participants.
Offering confidentiality means that some identifying information is known at some time by the research team, but only the research team has access to this identifying information and this information will not be linked with their data in any publicly accessible way. Confidentiality in research is quite similar to confidentiality in clinical practice. You know who your clients are, but others do not. You agree to keep their information and identity private. As you can see under the “Risks” section of the consent form in Figure 5.1, sometimes it is not even possible to promise that a subject’s confidentiality will be maintained. This is the case if data are collected in public or in the presence of other research participants in the course of a focus group, for example. Participants who social work researchers deem to be of imminent danger to self or others or those that disclose abuse of children and other vulnerable populations fall under a social worker’s duty to report. Researchers must then violate confidentiality to fulfill their legal obligations.
There are a number of steps that researchers can take to protect the identities of research participants. These include, but are not limited to:
- Collecting data in private spaces
- Not requesting information that will uniquely identify or "out" that person as a participant
- Assigning study identification codes or pseudonyms
- Keeping signed informed consent forms separate from other data provided by the participant
- Making sure that physical data is kept in a locked and secured location, and the virtual data is encrypted or password-protected
- Reporting data in aggregate (only discussing the data collectively, not by individual responses)
Protecting research participants’ identities is not always a simple prospect, especially for those conducting research on stigmatized groups or illegal behaviors. Sociologist Scott DeMuth learned that all too well when conducting his dissertation research on a group of animal rights activists. As a participant observer, DeMuth knew the identities of his research subjects. So when some of his research subjects vandalized facilities and removed animals from several research labs at the University of Iowa, a grand jury called on Mr. DeMuth to reveal the identities of the participants in the raid. When DeMuth refused to do so, he was jailed briefly and then charged with conspiracy to commit animal enterprise terrorism and cause damage to the animal enterprise (Jaschik, 2009).[118]
Publicly, DeMuth’s case raised many of the same questions as Laud Humphreys’ work 40 years earlier. What do social scientists owe the public? Is DeMuth, by protecting his research subjects, harming those whose labs were vandalized? Is he harming the taxpayers who funded those labs? Or is it more important that DeMuth emphasize what he owes his research subjects, who were told their identities would be protected? DeMuth’s case also sparked controversy among academics, some of whom thought that as an academic himself, DeMuth should have been more sympathetic to the plight of the faculty and students who lost years of research as a result of the attack on their labs. Many others stood by DeMuth, arguing that the personal and academic freedom of scholars must be protected whether we support their research topics and subjects or not. DeMuth’s academic adviser even created a new group, Scholars for Academic Justice, to support DeMuth and other academics who face persecution or prosecution as a result of the research they conduct. What do you think? Should DeMuth have revealed the identities of his research subjects? Why or why not?
Discipline-specific considerations
Often times, specific disciplines will provide their own set of guidelines for protecting research subjects and, more generally, for conducting ethical research. For social workers, the National Association of Social Workers (NASW) Code of Ethics section 5.02 describes the responsibilities of social workers in conducting research. Summarized below, these responsibilities are framed as part of a social worker’s responsibility to the profession. As representative of the social work profession, it is your responsibility to conduct and use research in an ethical manner.
A social worker should:
- Monitor and evaluate policies, programs, and practice interventions
- Contribute to the development of knowledge through research
- Keep current with the best available research evidence to inform practice
- Ensure voluntary and fully informed consent of all participants
- Not engage in any deception in the research process
- Allow participants to withdraw from the study at any time
- Provide access to appropriate supportive services for participants
- Protect research participants from harm
- Maintain confidentiality
- Report findings accurately
- Disclose any conflicts of interest
Key Takeaways
- Researchers must obtain the informed consent of research participants.
- Social workers must take steps to minimize the harms that could arise during the research process.
- If anonymity is promised, individual participants cannot be linked with their data.
- If confidentiality is promised, the identities of research participants cannot be revealed, even if individual participants can be linked with their data.
- The NASW Code of Ethics includes specific responsibilities for social work researchers.
Exercises
- Talk with your professor to see if an informed consent form is required for your research project. If documentation is required, customize the template provided by your professor or the IRB at your school, using the details of your study. If documentation on consent is not required, for example if consent is given verbally, use the templates as guides to create a guide for what you will say to participants regarding informed consent.
- Identify whether your data will be confidential or anonymous. Describe the measures you will take to protect the identities of individuals in your study. How will you store the data? How will you ensure that no one can identify participants based on what you report in papers and presentations? Be sure to think carefully. People can be identified by characteristics such as age, gender, disability status, location, etc.
6.3 Benefits and harms of research across the ecosystem
Learning Objectives
Learners will be able to...
- Identify and distinguish between micro-, mezzo-, and macro-level considerations with respect to the ethical conduct of social scientific research
This chapter began with a long list of harmful acts that researchers engaged in while conducting studies on human subjects. Indeed, even the last section on informed consent and protection of confidential information can be seen in light of minimizing harm and maximizing benefits. The benefits of your study should be greater than the harms. But who benefits from your research study, and who might be harmed? The first person who benefits is, most clearly, you as the researcher. You need a project to complete, be it for a grade, a grant, an academic responsibility, etc. However you need to make sure that your benefit does not come at the expense of harming others. Furthermore, research requires resources, including resources from the communities we work with. Part of being good stewards of these resources as social work researchers means that we need to engage in research that benefits the people we serve in meaningful and relevant ways. We need to consider how others are impacted by our research.
Micro-, mezzo-, and macro-level concerns
One useful way to think about the breadth of ethical questions that might arise out of any research project is to think about potential issues from the perspective of different analytical levels that are familiar to us as social workers. In Chapter 1, you learned about the micro-, mezzo-, and macro-levels of inquiry and how a researcher’s specific point of focus might vary depending on her level of inquiry. Here we’ll apply this ecological framework to a discussion of research ethics. Within most research projects, there are specific questions that arise for researchers at each of these three levels.
At the micro-level, researchers must consider their own conduct and the impact on individual research participants. For example, did Stanley Milgram behave ethically when he allowed research participants to think that they were administering electric shocks to fellow participants? Did Laud Humphreys behave ethically when he deceived his research subjects about his own identity? Were the rights of individuals in these studies protected? How did these participants benefit themselves from the research that was conducted? While not social workers by trade, would the actions of these two researchers hold up against our professional NASW Code of Ethics? The questions posed here are the sort that you will want to ask yourself as a researcher when considering ethics at the micro-level.
At the mezzo-level, researchers should think about their duty to the community. How will the results of your study impact your target population? Ideally, your results will benefit your target population by identifying important areas for social workers to intervene and to better understand the experiences of the communities they serve. However, it is possible that your study may perpetuate negative stereotypes about your target population or damage its reputation. Indigenous people in particular have highlighted how historically social science has furthered marginalization of indigenous peoples (Smith, 2013).[119] Mezzo-level concerns should also address other groups or organizations that are connected to your target population. This may include the human service agencies with whom you've partnered for your study as well as the communities and peoples they serve. If your study reflected negatively on a particular housing project in your area, for example, will community members seek to remove it from their community? Or might it draw increased law enforcement presence that is unwanted by participants or community members? Research is a powerful tool and can be used for many purposes, not all of them altruistic. In addition, research findings can have many implications, intended and unintended. As responsible researchers, we need to do our best to thoughtfully anticipate these consequences.
Finally, at the macro-level, a researcher should consider duty to, and the expectations of, society. Perhaps the most high-profile case involving macro-level questions of research ethics comes from debates over whether to use data gathered by, or cite published studies based on data gathered from, the Nazis in the course of their unethical and horrendous experiments on humans during World War II (Moe, 1984).[120] Some argue that because the data were gathered in such an unquestionably unethical manner, they should never be used. The data, say these people, are neither valid nor reliable and should therefore not be used in any current scientific investigation (Berger, 1990).[121]
On the other hand, some people argue that data themselves are neutral; that “information gathered is independent of the ethics of the methods and that the two are not linked together” (Pozos, 1992, p. 104).[122] Others point out that not using the data could inadvertently strengthen the claims of those who deny that the Holocaust ever happened. In his striking statement in support of publishing the data, medical ethics professor Velvl Greene (1992) says,
Instead of banning the Nazi data or assigning it to some archivist or custodial committee, I maintain that it be exhumed, printed, and disseminated to every medical school in the world along with the details of methodology and the names of the doctors who did it, whether or not they were indicted, acquitted, or hanged.…Let the students and the residents and the young doctors know that this was not ancient history or an episode from a horror movie where the actors get up after filming and prepare for another role. It was real. It happened yesterday (p. 169–170).[123]
While debates about the use of data collected by the Nazis are typically centered on medical scientists’ use of them, there are conceivable circumstances under which these data might be used by social scientists. Perhaps, for example, a social scientist might wish to examine contemporary reactions to the experiments. Or perhaps the data could be used in a study of the sociology of science. What do you think? Should data gathered by the Nazis be used or cited today? What arguments can you make in support of your position, and how would you respond to those who disagree?
Additionally at the macro-level, you must also consider your responsibilities to the profession of social work. When you engage in social work research, you stand on the reputation the profession has built for over a century. Since research is public-facing, meaning that research findings are intended to be shared publicly, you are an ambassador for the profession. How you conduct yourself as a social work researcher has potential implications for how the public perceives both social work and research. As a social worker, you have a responsibility to work towards greater social, environmental, and economic justice and human rights. Your research should reflect this responsibility. Attending to research ethics helps to fulfill your responsibilities to the profession, in addition to your target population.
Table 6.2 summarizes the key questions that researchers might ask themselves about the ethics of their research at each level of inquiry.
Level of inquiry | Focus | Key ethics questions for researchers to ask themselves |
Micro-level | Individual | Does my research interfere with the individual’s right to privacy? |
Could my research offend subjects in any way, either the collection of data or the sharing of findings? | ||
Could my research cause emotional distress to any of my subjects?
In what ways does my research benefit me? In what ways does my research benefit participants? |
||
Has my own conduct been ethical throughout the research process? | ||
Mezzo-level | Group | How does my research portray my target population? |
Could my research positively or negatively impact various communities and the systems they are connected to?
How do community members perceive my research? |
||
Have I met my duty to those who funded my research?
What are potential ripple effects for my target population by conducting this research? |
||
Macro-level | Society | Does my research meet the societal expectations of social research?
What is the historical, political, social context of my research topic? |
Have I met my social responsibilities as a researcher and as a social worker?
Does my research follow the ethical guidelines of my profession and discipline? How does my research advance social, environmental or economic justice and/or human rights? How does my research reinforce or challenge systems of power, control and structural oppression? |
Key Takeaways
- At the micro-level, researchers should consider their own conduct and the rights of individual research participants.
- At the mezzo-level, researchers should consider the expectations of their profession, any organizations that may have funded their research, and the communities affected by their research.
- At the macro-level, researchers should consider their duty to and the expectations of society with respect to social science research.
Exercises
- Summarize the benefits and harms at the micro-, mezzo-, and macro-level of inquiry. At which level of inquiry is your research project?
- In a few sentences, identify whether the benefits of your study outweigh the potential harms.
6.4 Being an ethical researcher
Learning Objectives
Learners will be able to...
- Identify why researchers must provide a detailed description of methodology
- Describe what it means to use science in an ethical way
Research ethics has to do with both how research is conducted and how findings from that research are used. In this section, we’ll consider research ethics from both angles.
Doing science the ethical way
As you should now be aware, researchers must consider their own personal ethical principles in addition to following those of their institution, their discipline, and their community. We’ve already considered many of the ways that social workers strive to ensure the ethical practice of research, such as informing and protecting subjects. But the practice of ethical research doesn’t end once subjects have been identified and data have been collected. Social workers must also fully disclose their research procedures and findings. This means being honest about how research subjects were identified and recruited, how exactly data were collected and analyzed, and ultimately, what findings were reached.
If researchers fully disclose how they conducted their research, then those who use their work to build research projects, create social policies, or make treatment decisions can have greater confidence in the work. By sharing how research was conducted, a researcher helps assure readers they have conducted legitimate research and didn’t simply come to whatever conclusions they wanted to find. A description or presentation of research findings that is not accompanied by information about research methodology is missing relevant information. Sometimes methodological details are left out because there isn’t time or space to share them. This is often the case with news reports of research findings. Other times, there may be a more insidious reason that important information is missing. This may be the case if sharing methodological details would call the legitimacy of a study into question. As researchers, it is our ethical responsibility to fully disclose our research procedures. As consumers of research, it is our ethical responsibility to pay attention to such details. We’ll discuss this more in the next section.
There’s a New Yorker cartoon that depicts a set of filing cabinets that aptly demonstrates what we don’t want to see happen with research. Each filing cabinet drawer in the cartoon is labeled differently. The labels include such headings as, “Our Facts,” “Their Facts,” “Neutral Facts,” “Disputable Facts,” “Absolute Facts,” “Bare Facts,” “Unsubstantiated Facts,” and “Indisputable Facts.” The implication of this cartoon is that one might just choose to open the file drawer of her choice and pick whichever facts one likes best. While this may occur if we use some of the unscientific ways of knowing described in Chapter 1, it is fortunately not how the discovery of knowledge in social work, or in any other science for that matter, takes place. There actually is a method to this madness we call research. At its best, research reflects a systematic, transparent, informative process.
Honesty in research is facilitated by the scientific principle of replication. Ideally, this means that one scientist could repeat another’s study with relative ease. By replicating a study, we may become more (or less) confident in the original study’s findings. Replication is far more difficult (perhaps impossible) to achieve in the case of many qualitative studies, as our purpose is often a deep understanding of very specific circumstances, rather than the broad, generalizable knowledge we traditionally seek in quantitative studies. Nevertheless, transparency in the research process is an important standard for all social scientific researchers—that we provide as much detail as possible about the processes by which we reach our conclusions. This allows the quality of our research to be evaluated. Along with replication, peer review is another important principle of the scientific process. Peer review involves other knowledgeable researchers in our field of study to evaluate our research and to determine if it is of sufficient quality to share with the public. There are valid critiques of the peer review process: that it is biased towards studies with positive findings, that it may reinforce systemic barriers to oppressed groups accessing and leveraging knowledge, that it is far more subjective and/or unreliable than it claims to be. Despite these critiques, peer review remains a foundational concept for how scientific knowledge is generated.
Full disclosure also includes the need to be honest about a study’s strengths and weaknesses, both with oneself and with others. Being aware of the strengths and weaknesses of your own work can help a researcher make reasonable recommendations about the next steps other researchers might consider taking in their inquiries. Awareness and disclosure of a study’s strengths and weaknesses can also help highlight the theoretical or policy implications of one’s work. In addition, openness about strengths and weaknesses helps those reading the research better evaluate the work and decide for themselves how or whether to rely on its findings. Finally, openness about a study’s sponsors is crucial. How can we effectively evaluate research without knowing who paid the bills? This allows us to assess for potential conflicts of interest that may compromise the integrity of the research.
The standard of replicability, the peer-review process, and openness about a study’s strengths, weaknesses, and funding sources enables those who read the research to evaluate it fairly and completely. Knowledge of funding sources is often raised as an issue in medical research. Understandably, independent studies of new drugs may be more compelling to the Food and Drug Administration (FDA) than studies touting the virtues of a new drug that happen to have been funded by the company who created that drug. But medical researchers aren’t the only ones who need to be honest about their funding. If we know, for example, that a political think tank with ties to a particular party has funded some research, we can take that knowledge into consideration when reviewing the study’s findings and stated policy implications. Lastly, and related to this point, we must consider how, by whom, and for what purpose research may be used.
Using science the ethical way
Science has many uses. By “use” I mean the ways that science is understood and applied (as opposed to the way it is conducted). Some use science to create laws and social policies; others use it to understand themselves and those around them. Some people rely on science to improve their life conditions or those of other people, while still others use it to improve their businesses or other undertakings. In each case, the most ethical way for us to use science is to educate ourselves about the design and purpose of any studies we may wish to use. This helps us to more adequately critique the value of this research, to recognize its strengths and limitations.
As part of my research course, students are asked to critique a research article. I often find in this assignment that students often have very lofty expectations for everything that 'should' be included in the journal article they are reviewing. While I appreciate the high standards, I often give them feedback that it is perhaps unrealistic (even unattainable) for a research study to be perfectly designed and described for public consumption. All research has limitations; this may be a consequence of limited resources, issues related to feasibility, and unanticipated roadblocks or problems as we are carrying out our research. Furthermore, the ways we disseminate or share our research often has restrictions on what and how we can share our findings. This doesn't mean that a study with limitations has no value—every study has limitations! However, as we are reviewing research, we should look for an open discussion about methodology, strengths, and weaknesses of the study that helps us to interpret what took place and in what ways it may be important.
For instance, this can be especially important to think about in terms of a study's sample. It can be challenging to recruit a diverse and representative sample for your study (however, that doesn't mean we shouldn't try!). The next time you are reading research studies that were used to help establish an evidence based practice (EBP), make sure to look at the description of the sample. We cannot assume that what works for one group of people will uniformly work with all groups of people with very different life experiences; however, historically much of our intervention repertoire has been both created by and evaluated on white men. If research studies don't obtain a diverse sample, for whatever reason, we would expect that the authors would identify this as a limitation and an area requiring further study. We need to challenge our profession to provide practices, strategies, models, interventions, and policies that have been evaluated and tested for their efficacy with the diverse range of people that we work with as social workers.
Social scientists who conduct research on behalf of organizations and agencies may face additional ethical questions about the use of their research, particularly when the organization for which a study is conducted controls the final report and the publicity it receives. There is a potential conflict of interest for evaluation researchers who are employees of the agency being evaluated. A similar conflict of interest might exist between independent researchers whose work is being funded by some government agency or private foundation.
So who decides what constitutes ethical conduct or use of research? Perhaps we all do. What qualifies as ethical research may shift over time and across cultures as individual researchers, disciplinary organizations, members of society, and regulatory entities, such as institutional review boards, courts, and lawmakers, all work to define the boundaries between ethical and unethical research.
Key Takeaways
- Conducting research ethically requires that researchers be ethical not only in their data collection procedures but also in reporting their methods and findings.
- The ethical use of research requires an effort to understand research, an awareness of your own limitations in terms of knowledge and understanding, and the honest application of research findings.
Exercises
- Think about your research hypothesis at this point. What would happen if your results revealed information that could harm the population you are studying? What are your ethical responsibilities as far as reporting about your research?
- Ultimately, we cannot control how others will use the results of our research. What are the implications of this for how you report on your research?
when a measure indicates the presence of a phenomenon, when in reality it is not present