Knowing What to Look For
Improve Your Observations by Asking Better Questions
So acquire the habit of being present at this activity of the material and moral universe. Learn to look; compare what is before you with familiar or secret ideas. Do not see in a town merely houses, but human life and history.
If you cannot look thus, you will become, or be, a man of only commonplace mind. A thinker is like a filter, in which truths as they pass through leave their best substance behind.
- Sertillanges
Imagine with me for a moment, that we are members of a team of researchers tasked with traveling through time to collect first hand anthropological research on ancient cultures. However, on our latest assignment the time machine malfunctions. We crash land in an unknown time and location. We are confused and directionless.
This is how thinking and learning often feels. We know that the pursuit of knowledge, and the thinking that follows should begin with inquiry (research). But the landscape of that research is often vast,confusing, and directionless. Like our hypothetical team of time-travelers, we have a hard time figuring out where we are, when we are, and what we need to know in order to get back home.
In this post, I want to propose that successful navigation through the ambiguous terrain of inquiry depends on our ability to make good observations. The quality of those observations is directly related to the quality of the questions we ask in the research process. Below, I elaborate on this important relationship, and provide the three step guide I use when formulating my own research questions.
Observations Start With Questions
In a previous post, I described observation as the foundational skill we need in thinking and learning. Observation is the skill of noticing. It is the distinct act of calling something to attention and of moving information from the background into the foreground. When you highlight a sentence in a book or write something down that you heard in a lecture, you are making an observation. You are making an intentional choice to call something out from the background of the page or speech, and put it into the foreground of your notebook.
For our team of stranded time-travelers, they will need to make excellent observations using some of the expertise and training they have (memory), to map out where and when the time machine has crash landed. The quality of their observations will directly depend on the quality of the questions they ask. For example:
Low Quality Question: What time period are we in?
High Quality Question: What can the building materials/methods of the civilization tell me about this time period?
Crafting good research questions is a lot like making a map. We embark on the turbulent journey of research because there are things that we want to know. When we set out on the journey to find the answers, we collect observations (data). Those observations are then collected, organized (taxonomy), and stored (memory). From there, they can be called upon in order to form conclusions and solutions.
In the pursuit of excellent thinking, the observations we collect are directly related to the questions we ask. Often, when research feels confusing and directionless, it is a result of having not asked the right questions. Asking good research questions is a craft and the most important part of the research journey. After all, one should not set out on an expedition without a good map.
Making A Map With Questions
Step 1: Make a Pirate Map
By “Pirate Map,” I am simply referring to the silly cartoon maps we were familiar with as children. They feature big dotted lines, an “X” that marks the spot, and general directions based on landmarks (e.g. you will know you’re close to the treasure after you pass the mountains).
When we start our research journey we need a similar approach. We need to know what the big landmarks are that we are looking for. I often refer to this as the “research domain.” If I find myself in a domain that doesn’t have the big landmarks, then I’ve drifted off course and need to correct.
Your research domain will often consist of the big question and purpose of your research (what are you wanting to find out?). For example, if I am doing research on AI policy in Higher Education, then I might build a pirate map like this:
What are the policies of other liberal arts/humanities based colleges?
How have policies evolved since Fall 2022?
What has been the impact or effectiveness of these policies?
These three questions are still fairly broad. Still, they form a pirate map that tells me the landmarks that I need to pay attention to in my research: (1) policies of liberal arts/humanities colleges; (2) articles written after fall 2022; (3) articles that primarily evaluate or assess policy effectiveness. If I find myself reading articles or asking questions outside of this domain, then I need to course correct.
Step 2: Chart the Territory
Once we have a general idea of where we are going (purpose) and the landmarks we are looking for (big questions), we need to chart the territory by identifying key themes, variables, or potential gaps in existing literature.
With the key landmarks in mind, we will need to do an initial search and make observations along the way. Try to get into the habit of collecting the research questions that others are asking about your topic. This is usually one of the first things I look for when reading an article or chapter. The research questions I observe from other literature will not only help me read more closely, but it will also help me craft my own questions along the way. You also want to pay attention to the methods that authors use when pursuing the answers to these questions (more on this below).
Here I offer three suggestions for where you might look for some of this background information:
Popular News/Media: These are some of the most helpful resources for getting the pulse on an issue or topic. Popular media can tell us a lot about the general perception of something and provide us with some of the contextual relevance we might need to strengthen our argument. These sources should mainly be limited to setting the context and illustrating points. Popular media (typically) should not be used to support the main points of our arguments.
Personal Experience: In a similar way, personal experience is generally not recommended to strengthen our arguments (i.e. hasty generalization). But we can use personal experience to help us understand why we are interested in the research.
Existing Literature: Most of the time when we engage in research, we will be consulting the existing literature on a topic. These are published (often peer-reviewed) scholarly articles, books, chapters, presentations, etc.
It is from these common sources that we get a feel for the territory we are working in and can finally move forward with the process of crafting our own research questions. I should also add that the observations we make as we engage with the research is also a source from which we gain further territorial insight.
SIDEBAR: Students often express frustration that they “can’t find anything about (or against) their topic.” However, this is rarely the case. If relevant literature seems unavailable, the issue is likely with the framing of the research question rather than the topic itself. Finding useful sources often requires refining your question, exploring related concepts, and engaging more deeply with existing literature. You will always find what you’re looking for - so long as you ask the right questions.
Step 3: Plan the Route
Here is where we finally arrive at the step of asking our research questions. This is where we go from pirate map to aerial view.
Hopefully, a thorough reading of the literature has equipped us with everything we need to formulate our research questions. J.T. Dillon (1984) identified 17 different kinds of research questions that Alvesson & Sandberg summarize in the following four categories (*whispers 🤫* this reminds me of Aristotle’s four causes):
Descriptive Questions (what is it?) are aimed at knowledge concerning kind, components, parts, quality, quantity, etc.
Example: What policies currently regulate the use of AI in academic writing and research at universities?
Comparative Questions (what is it like, not like?) get at the heart of differences and similarities.
Example: In what ways do faculty perspectives on AI policy differ in the STEM fields versus the liberal arts and humanities?
Explanatory Questions (what are its causes and effects?) help us further understand the potential impact or effectiveness of a thing.
Example: What impact (if any) do strict AI restrictions have on student learning outcomes and academic integrity violations?
Normative Questions (what is its utility? what should be done?) are all framed around what “should” or “ought” to be done in light of the observations we make. These are difficult questions to answer apart from the previous three.
Example: What are the ethical principles that should guide AI policy development in higher education?
Now that we have four general categories of questions that we can ask, we need to consider the components that make up a good research question. Here are three qualities that every research question should have:
Good Research Questions are:
Focused: Our questions need to be specific to a given domain and the big questions we are asking. Further, they need to be restricted to only one kind of category (above), and should only be answerable by one type of observation.
Exploratory: At the same time, we want to ensure that we are asking questions in which we do not already assume or suppose we have the answers. Again, we are wanting to make observations and assuming we have the answer already puts us in the vulnerable state of bias and blindness.
Strategic: Lastly, we need to make sure that we are asking questions which tell us exactly what we need to look for (observations) in order to arrive at the answer to the questions. Knowing what kind of observations we need to make will help us know what tool to use in order to source those observations.
When I am setting up a large research project, I will typically begin with a Research Question Map similar to this (I typically give either each question or objective its own row):
Now that we have a map of where we are going, we are ready to head out on our research journey.
Looking Forward: The View from the Street
Writing research questions and making observations is an iterative process. We are always going back and forth between our research map and the view from the street to make sure we are headed in the right direction. The more observations we make from the street view, the more we might need to refine our research questions.
Often, seeing the details up close can help us get a better understanding of what it is we are looking for.
Are the questions you asked generating the specific observations you need to advance the purpose or objective of your research?
Are your research questions actually strategic or do they merely outline the procedure/step to take in your research? Tip: if you use a research question map like the one above, then you should likely avoid this error.
Can the questions you are asking be answered by the sources/methods you have access to and/or do your sources have access to the information?
Asking and answering these questions when we get stuck along the way will direct us back to the drawing board to refine our questions and ultimately make better observations.
The first step towards improving our observations often starts with asking better questions. This helps us know what to look for.
Super helpful framework that I plan to steal for researching articles and organizing reading projects around research questions.