What is this guide? This guide aims to outline my strategies to encourage idea generation, essential for long-term research progress, through a habit of regular writing.
Who is this guide for? While I have written this guide for my PhD students, it aims to help academics and working professions for whom creative problem solving is a central part of their responsibilities.
In my experience, research progress—at least in the sciences—can be grouped into two phases:
When in the milestone completion phase, it’s often fairly straightforward to understand how to make progress. By contrast, there is no formula for coming up with new ideas. Despite its importance, the amorphous nature of idea generation can make it feel frustrating; a lack of immediate progress may discourage, leading one to put idea generation on the backburner. Yet idea generation, broadly defined, is incredibly important for long-term progress.
Through my PhD and in my time since as an academic, I have found a number of strategies that have helped me to make progress towards new research ideas, advancing stuck projects, and writing grant proposals. This post is about the approach that I have found helps me make progress towards my open-ended research aims and to set a direction for my and my lab’s work; it outlines practices that I generally recommend my students follow.
Writing forces me to clarify my thoughts.
In research, the best ideas often begin with a problem to be solved. Too many times, I have thought I understood a problem until I tried to write it out in full detail, to explain it to someone else, or to withstand deep questions about it. When writing something down, I must face head-on the ambiguities, contradictions, or unresolved issues that my mind may have glossed over. When honing an idea, I try to be as precise and complete as possible, so as to leave little ambiguity about the problem I am trying to solve. When I write, I do so with the intention of better understanding a problem or, if I think I already understand the problem, coming up with possible remedies to the core issue or experiments to run that can help me better understand it. For problems of appreciable size or complexity, it becomes impossible to identify the root cause of a problem or what to do about it without writing it out in full.
For me, an idea generation “session” is an unbroken period of time devoted to writing, typically an hour in length. For each writing session, I to set a well-defined goal for what I want to accomplish. I usually target some sort of intermediate progress milestone—more about that in the next section—or, failing to come up with one, set a target word limit for my writing; even 300 words of clear prose can represent solid progress towards better understanding a difficult problem. These intermediate milestones help me to feel as if I am making progress even when I do not complete a high-level objective.
Many of my best ideas come to me when I least expect them: when going for a walk, taking a shower, or talking with friends and colleagues. This raises the question, why have dedicated writing sessions for idea generation if many of the best ideas come from elsewhere? These sorts of epiphanies only occur for problems that I’ve already thought deeply about and tried (and failed) to make solid progress towards. The goal of my writing sessions is therefore less to produce breakthroughs as it is about readying the mind to have such breakthroughs.
As such, most of my writing sessions are devoted to better understanding a problem. Rather than tell myself during today’s writing session, I am going to come up with a solution to a problem, my intention for a writing will often include goals to precisely (mathematically) define the problem I am trying to solve or to describe in detail a scenario where I expect the current state-of-the-art will fail: i.e., an example that motivates the need for a new approach that overcomes this issue. If I do not have a clear picture of what problem I’m solving, the writing session is devoted to more precisely defining it.
Over time, I have amassed quite a list of problems I think are interesting. I have devoted significant effort towards understanding why those problems exist and, often, to identifying commonalities between them in an effort to discover an underlying root cause. It is from this list—and the process of creating it—that many of my research ideas have come about. Once I feel that I may have a potential idea of how to address one or more of the problems I have written about, I will spend a writing session expanding upon it. The resulting ideas then form the backbone of new experiments to validate my hypothesis.
In my field, these proof-of-concept experiments are an essential component of this process. The successes and failures that result from experiments are needed to make progress.
In my field, these proof-of-concept experiments are an essential component of this process. The successes and failures that result from experiments are needed to make progress.
This attitude also makes it easier to set intermediate progress milestones. It can be discouraging to sit down with an intention to come up with a new idea and then fail to do so. Instead, when focusing on clearly describing a problem scenario or a potential solution, adding particular details becomes a target and it becomes far easier to conclude a writing session feeling as if progress has been made.
It is difficult, if not impossible, to force a breakthrough. On the whole, I would say that fewer than half of my leaps in progress towards my research happen during dedicated sessions to make progress towards them. The rest happen when I least expect it: when going for a walk, taking a shower, or cooking dinner. A deep understanding of a problem only happens if you think about it often; writing is the most effective way to ensure that thought is deep and purposeful.
Pushing myself during long writing sessions often becomes counterproductive. Concentrating all the writing for a project to a single writing binge can be somewhat useful when focusing on a deadline or trying to enumerate all the details for an experiment or already well-formed idea. Yet when trying to reach a not-yet-well-defined goal or overcome a tricky problem, I can easily become frustrated after a few hours of feeling as if I have made little progress.
However, if I’m in the midst of a very productive writing session, I won’t force myself to stop. Preferring shorter writing sessions is a guideline to avoid burnout and frustration, but should not be rigidly enforced at the expense of progress.
However, if I’m in the midst of a very productive writing session, I won’t force myself to stop. Preferring shorter writing sessions is a guideline to avoid burnout and frustration, but should not be rigidly enforced at the expense of progress.
I make an effort to write every (work) day for roughly an hour. This amount of time is usually long enough that I can feel as if I’ve made progress without tiring me out. Writing every day ensures that I am regularly thinking about the challenges that are holding back me or my lab.
Relatedly, as a regular habit is essential, procrastination is the enemy. I try to sit down to write even on days where I feel I have little to say; I am frequently rewarded for the effort.
Where do you write? At what time of day do you write most productively? Do you listen to music, prefer the hum of a busy cafe, or require silence? Do you prefer bright lights, sunlight, or absolute darkness? Do you prefer to type when you write or hand-write on paper or a tablet?
As a PhD student, I did not give much thought to these questions, yet I now appreciate how important setting is to my writing productivity.
My writing is most productive under two conditions: (i) in the morning at my local cafe, a cup of coffee in my hand, sitting near the window to get lots of light and so that I can stare aimlessly into the distance without unnerving nearby patrons and (ii) late at night in my bedroom with the lights dimmed to near total darkness. I always listen to music using my noise-canceling headphones.
I am a huge fan of the Anjunadeep record label, whose continuously-mixed deep house compilation albums have fueled many of my writing sessions and best ideas since graduate school.
I am a huge fan of the Anjunadeep record label, whose continuously-mixed deep house compilation albums have fueled many of my writing sessions and best ideas since graduate school.
While I don’t fully understand why these are the conditions under which I write the most effectively, it was a process of protracted trial-and-error to discover them. Critically, I rarely write in the middle of the day, when my efforts are best spent on different activities. In developing a regular writing habit, it is important to consider such details.
The time between writing sessions is just as important as the writing itself. As I mentioned above, writing is to stimulate the mind to think more deeply about a problem. In addition to writing regularly, my other habits and practices help me to make progress.
Conduct experiments. I work in an experimentally-driven field, and so the purpose of writing is to generate hypotheses of how to make progress; thus, experiments are of central importance. In addition to running large-scale experiments to explore every facet of a problem, targeted experiments meant to better understand small pieces of it can be just as productive and with a faster iteration time. Frequently, my writing will expose some lack of knowledge of the nature of the problem and I must run some small proof-of-concept experiments before I am ready to make more progress.
Talk to others. Find a handful of people who are somewhat knowledgeable about the problems you are trying to solve. Explaining the problem to them and your thoughts on how to solve it can be an incredibly useful exercise. Moreover, those people should be willing to challenge you on those ideas and push you to clarify what you mean or hone in on where a proposed solution might not work. These dialogues—if approached from a place of curiosity or shared purpose, rather than with adversarial intent—can be incredibly productive for all involved.
Relatedly, we should all strive to be idea confidants for others.
Relatedly, we should all strive to be idea confidants for others.
Talk to yourself. I regularly think out loud. Whether I’m writing or driving to work, trying to speak my thoughts serves a similar function to writing by forcing me to clarify my often-underdeveloped internal musings. This doesn’t work for all, but is incredibly useful for me.
Keep an open mind. Writing need not be so rigid and, in fact, I often ensure that every week or so, I devote a writing session to open-ended brainstorming about a challenging problem or idea. Research requires a certain amount of creativity and such sessions can help to produce new ideas or help me to see an old problem from a new angle.
Keep a list of thoughts to revisit later. Focus during a writing session is important. I am easily distracted, not just by noise and external distractions, but also by potentially promising ideas that deviate from my writing goal. Sometimes I’ll pursue these in the moment, but more often, it is best to add it to a list of ideas I want to revisit. Even outside of a writing session, I may have an idea to come back to later and so I have a note on my phone (always on hand) where I can keep such scribbles. I typically have one list for each project I am working towards.
Go outside. Take breaks. Eat well. Exercise. Sleep. Balance is important. The mind cannot operate most effectively without a healthy body. Sleep is particularly important. While I occasionally trade sleep in pursuit of short-term productivity, virtually none of my best ideas have come about when I have been sleep deprived.
Any writing habits of yours you would like to share? I welcome thoughts and feedback on Twitter.