Not that Kind of Letter: Tales of Rejection

Doing Research / Grants & Funding

I have been thinking about rejections in science. Rejections come in all shapes and sizes, from the grant you need to build your program, to an awesome rotation student picking another lab, to a manuscript rejection at yet another journal. While I definitely had my share of rejection emails as a graduate student and postdoc, there has been an exponential rise since starting my faculty appointment. So how is a new principal investigator (PI) to deal? As always these days, n=me.

The first rejection of any type hurts the most: There is no magic pill to lessen the blow of the first rejection. Going into my faculty position with my successful K99/R00, I thought securing my first R01 would be challenging, but not impossible. Needless to say, when my first R01 submission swung the other way and scored terribly, it was a brutal awakening. I have put in numerous grants now and while annoyed and concerned about my lack of R01 funding, my recovery time with each unfunded grant has decreased substantially. 

Rejections in science can feel personal: One of the more challenging aspects of rejections in science is that they can feel deeply personal. For example, in grants, not only are your ideas carefully (or perhaps brutally) dissected, but your entire training, publication, and funding history are examined and commented upon. While I do not believe most reviewers aim to be cruel, sometimes objective statements carelessly phrased cut deeply. Similarly, when a talented postdoc or rotation student picks a more established laboratory, it can feel like a judgement on your mentoring philosophy, potential, and research vision.

It is OK to be sad: Mourning the loss of possibilities arising from an unfunded grant, unsecured postdoc, or rejected manuscript is normal. I often mope for a day (or two) and then usually throw myself into something that reaffirms my skills in the area of rejection. For example, on grant rejections, I throw myself back into lab work and moving papers along or look at the holes in the data, which the reviewers pointed out.

Talk about it: Venting is good for the soul. Yes, the reviewer completely missed the mark and how dare they not see the brilliance of my research proposal? How could the rotation student pick that other lab and not see the vision of their thesis that you had outlined? Sometimes it is easier to see the validity of a criticism when you discuss it. Moreover, by talking about rejections we normalize them. Fifty percent of grants in a study section are not discussed! The majority of grants that are discussed are not funded! Rejection is the norm in this job. By talking about it, we accept that it is frequent, normal, and inevitable.

Make a plan and move forward: After you have gone through the five stages of grief, make a plan. How are you going to address reviewer comments on your triaged grant? Prepare to spend some time with those horrible pink sheets and figure out what really did not work. Not sure what went wrong? Ask for help from your peers, mentors, community members. Sometimes realizing that you are not competitive for a grant or your lab is not the right fit for graduate students is an important part of moving forward.

Find a community that celebrates wins: There are many more losses than wins in this job. It is simply the nature of our chosen profession, and one way to lessen the blows of these seemingly endless rejections is to find a community that celebrates each other’s victories, great and small. Be this Twitter, New PI Slack, your department, or a group of assistant professors, find your people. Knowing you did not get the grant, but one of your friends did makes the rejection sting less.

These are my approaches to dealing with rejection, and I am sure there are many more. Whatever your approach, do not let the fear of rejection keep you from applying for grants, awards, and recruiting talented trainees! The worst they can say is “no”. Stay tuned for more tales!

 

Still have questions? More confused than when you started? Need to vent about the process? Feel free to send some electrons my way in the comments, via Twitter @PipetteProtag, or through traditional electronic mail pipette.protagonist@gmail.com

More Resources

More Friendly Advice: When Rejection Isn’t Really Rejection

Fighting Rejection: Three Little Reviewers

Dealing with Rejection

 

 

Discover What’s Getting Funded with NIH Matchmaker

Grants & Funding

To compete well for grants, you need to know what’s getting funded.  Enter NIH Matchmaker.  The service takes text about your science and shows you the projects that are most like yours.  Reading the abstracts yields insight into the kinds of aims reviewers find attractive, and the study section and institute that most commonly fund your kind of science.

Matchmaker lives within the larger NIH Reporter site. You’ll see a text box to enter any content you want to match on. The character limit is 15,000, so not only could you use the abstract of a grant or conference presentation, but also almost an entire manuscript, a dissertation chapter, or similar. Some researchers hack the matchmaking algorithm by putting in a list of words related to their research, repeating the more important ones several times.  (I.e., “fibroids fibroids fibroids leiomyomata miscarriage pregnancy ultrasound ultrasound.”) Once you’ve entered your text, choose “Similar Projects.” (Can also check what comes up under “Similar Program Officials.”)

Matchmaker then returns up to 500 similar projects, listed in decreasing similarity (as indicated by the match score on the left). NIH provides info on how they calculate a match score (natural language processing using a combination of how frequently the term appears in the matched grant’s title, abstract, and aims, and how uncommon the word is across all grants). Click on “About RePORTER Data” and scroll to the bottom of that section.

Click on any project title to read the abstract, which typically provides a glimpse of the specific aims. Using the tabs at the top, you can see how much money the project received, which institute, program officer and study section it went through, and, if the project is far enough along, a list of papers linked to that particular grant.

Filter your results by any combination of institute, grant type (aka activity code), or study section using the bar graphs at the top of the results page. For example, in the results on the right, if you only wanted to see R01s funded by NICHD, you would click those two bars of the graphs. If you’re trying to decide between study sections to request, you can examine which grants on your match list have been funded by each of the study sections you’re considering by using the filters. (Guide to study section abbreviations.)

All of this information helps you reality check the attractiveness of the potential focus of your new grant and tailor your science and methods. This helps best align your grant application with the leading edge that NIH is funding right now–and when paylines are tight, you have to write what gets reviewers excited.

Additional Resources

Harness the Immense Power of Nosiness in NIH RePORTER

Using NIH RePORTER to Find Your Guide

Which Study Section Should I Pick? Try the Assisted Referral Tool!

Video: How to Use NIH Matchmaker

Mapping the Path for a K or R Submission

Grants & Funding

When submitting a grant, sixteen weeks out is the magic number for starting the process. (For the NIH deadlines, this means start working at the deadline before your target: For the June deadline, early February is the optimal time to begin, and so on.) Understanding the path and having a clear timeline is key to a successful grant submission that doesn’t leave you wrung out.

Dr. Carol Lorenz, aka Durango Kid, provides everything you need to pace your winter grant to successful submission.  With a history of managing more than 70 grant submissions in a year, Dr. Lorenz is a PhD scientist with guaranteed reality-based expertise in project management. Over the course of seven blog entries, she breaks down the huge endeavor of a grant submission into manageable chunks, and shows you how to create a customized grant timeline to ensure the smooth completion of your application.

Part 1: #*@*! Plan is Not a Four-Letter Word

Why should you do this? How can it help you? Assignment: Decide on which grant you want to apply for.

Part 2: Planning to Plan: Gathering Materials

How should you start your plan? Assignment: Assemble supplies for creating a paper timeline.

Part 3: Can You (Really) Do Your Proposed Study?

How to assess feasibility so you don’t bite off more than you can chew (or put into one application). Assignment: Create a project proposal.

Part 4: Researchers: Start Your Timelines

How much time do you really have, and how can you best break it down? Assignment: Add submission date and units of time to your timeline. For February 2022, we suggest these dates.

Part 5: 500 Mile(stones)

Milestones can tell you if you’re progressing at the right pace toward your submission due date. Assignment: Note important dates and transfer them to your timeline.

Part 6: Buckets of Fun (Work?)

How to break your work into packages so you can then define specific tasks. Assignment: Create your buckets.

Part 7: What’s In Your Bucket(s)?

Now that you have your buckets, what do you put in them? Assignment: Brainstorm tasks and fill your buckets, then your timeline.

An example of a completed timeline (click for full size).

More Resources

Ten Insider Tips: What Your Grants Manager Wishes You Knew

More Things I Wish I’d Known Before I Wrote My K

Three (Grant) Peeves in a Pod: Write Better

If They Can’t Trust You with Stats, They Won’t Trust You with Money

Grants & Funding

For most grants you write, here is all you need to do to get them funded:

1. Convince me that it will be a crime against humanity if the science doesn’t get done, and that

2. No one in the world besides you can do it.

Simple.

For grants with a clinical trial, though, there is one more thing that you have to do, which seems simple when I say it, but people still screw it up continuously.

3. Get the statistics and recruitment right.

For clinical trials, (1) is usually pretty easy – by the time you’re proposing a clinical trial, you presumably have a mountain of preclinical evidence on the efficacy of whatever you’re proposing in a variety of animal models, you’ve probably got some safety data, and you’ve got a molecular mechanism that’s compelling. Nine out of ten reviewers are going to agree with you that it’s time to bring it into humans.

And then you blow it by setting up your trial design and statistics in such a way that you convince reviewers that no matter what happens, you’re not going to end with meaningful data.  Your recruitment assumes literally everyone who comes through your clinic agrees to participate (or you don’t say how many people are coming through your clinic at all). Your recruitment requires you to be recruiting 24/7 – but you don’t even have a clinical coordinator, let alone a research nurse or two.  Are you gonna sleep in the ER?

You power it based on wildly optimistic assumptions of efficacy – one clinical grant I recently reviewed required tenfold higher effectiveness than any other drug ever tested in order for detectable effect. You ignore effects of other treatments, or variability in the etiology of the underlying disease, in your inclusion/exclusion criteria (shock from blunt trauma ain’t the same as shock from burns and shock from infection). You include subgroup analyses you’re really not powered for. You include large numbers of secondary endpoints without telling me how you’re correcting for multiple testing (I like lots of endpoints, but you need to be careful with them statistically).

Your statistics section was obviously stolen from some other grant, so it has statistics-sounding words, that don’t make sense in current context.

Many authors don’t get power right:

https://elifesciences.org/articles/34412/figures

Two big pieces of advice:

  • If you’re not expert on it yourself, get a biostatistician involved in your trial design right from the start. Don’t just get someone to give you their CV to include.
  • Get somebody who’s actually run a trial to help with the design and recruitment strategy. It’ll keep you from making the sorts of mistakes that call into question your ability to complete the study.

I just came off of a study section that was nothing but reviews of clinical trials. Essentially every one of them had these problems – and we all loved the premise and preliminary data on all of them. What that means is, get this piece right, and you’ve got a big jump on the competition.

Keeping Your Eye on the R01 Ball

Grants & Funding

This post condenses a Q&A led by Alex Smith, MD, MS, MPH and Katherine Hartmann, MD, PhD, on the topic of K to R conversion at the annual 2018 meeting of the Paul B. Beeson Emerging Leaders Career Development Awards in Aging. The round table included Beeson Scholars, who are K awardees, their mentors and other leaders from academic medicine and the NIH. The goals of the meeting include building the aging research community, encouraging dissemination of findings, and enhancing development of scholarship and leadership.

Q.  I’ve been advised to write an R03 to have more funding and show that I can achieve my own NIH funding. Is that a good starting place?

Several facets need consideration:

  • While having an R03 or R21 could provide a bit of a halo effect in subsequent reviews, other reviewers will wonder why a career development awardee did not get straight into the game.
  • Being a productive independent investigator requires substantial funding over extended periods. R01s and equivalent large grants are an accepted (if contentious) gold standard for achieving independence. Pragmatically, they also provide the funds needed to build and sustain a research team, which is a fundamental need for a successful transition.
  • R03s and R21s are simply not long enough and don’t carry enough funding to build a research enterprise. You are in essence an entrepreneur and must launch rapidly.
  • Hypothetically if you were ready on the first day of your career development award to submit a smaller grant, bear in mind that R03s and R21s are slightly harder to get than R01s. Per NIH’s Funding Facts, success rates over the past five fiscal years for R21s are around 13.5%, and R03s around 17%. The R01 success rate for most years is 18%. Study this data for your institute.
  • “High-risk, high-yield” and note of lower requirements for preliminary data for R21s does not equate to “perfect for early career researchers.” The majority of R21s go to established labs because they are a strong bet for the funding gamble to pay off.  Most often the award demonstrates novel techniques or tools within a thriving research portfolio.
  • R03s/R21 are equally as slow to obtain as R01s. Because resubmission is nearly guaranteed, time from first submission to receipt of funds can take 18 months or longer. You can’t wait that long to conduct research to support your first R01 submission. You need more immediate pilot funds if your K cannot support key research components.
  • It’s very difficult to generate data, get it published, and integrate it into an R01 application in the short time frame of these grants because of the delay built into receiving an award. In the session prior to this discussion we created timelines that demonstrated that K-awardees with a five year grant need to submit a first R01 no later than the middle of the third year of funding to allow for revision and resubmission. Optimally you’ll submit earlier (see discussion of overlap of K and R below).

Q. What is the place for R03 and R21 grants?

  • To branch out from your current line of research and build a base to move to a new area.
  • To validate a method, do a feasibility study, or pursue another set of experiments or analyses that aren’t enough to drive an entire R01.
  • To pull out a strong aim from an R01 application that got mixed reviews and turn the well-reviewed aim into an R03 while you regroup for the R01.

Q. I’ve been told I don’t want to get an R01 too early. Won’t I lose my K award?

  • Creating a timeline is very informative: map when you will finish crucial data collection, when you will submit key materials for publication, and when you anticipate being ready to submit an R01 award. Be sure not to think only in terms of scientific readiness. Allow 16-20 weeks to prepare all the components of the grant including details like staffing, subcontracts, letters of support, and locking in co-investigators. Then allow for the NIH review and resubmission dates, remembering to extend the horizon past review to dates of council and possible start of funding.
  • It is rare that K awardees are ready to play professional ball, aka submit an R01, in the first year or two of preparation. Few individuals need to worry about going too fast. Most need to worry (based on timeline above) about starting too late.
  • Career development awards pay for the scientist, not the science. Wouldn’t you rather have a larger budget for your science and for key personnel in your research group?
  • Federal career development awards allow for up to two years of overlap of a K award with an R01. This is magic because it preserves a high level of protected time (which you will never again receive on a single R award) to learn how to play and hire and coach your team on your way to complete independence and smooth function.
  • Anecdotally, a mentor in the group noted an extremely unusual K awardee who submitted an R01 around 8 months into her K award and was funded on the first submission. The hypothetical question: was she was worse off? Of course not; she still had two additional years of 75% protected time and $2.7 million to seriously drive her science and build a bigger team. It also came with the psychological advantage of not chasing independence close to tenure and promotion review.

Q. How much preliminary data do I need to support an R01?

  • It’s not just about data. You do need to support claims as strongly as possible – provide actual data in the grant. Do not expect reviewers to go look at your papers. They will, but for other reasons. They expect you to distill key information in the text.
  • Remember reviewers consider many categories of information to judge if you are prepared to conduct an R01-scope study. The rationale and scientific thought model must be airtight and clear and the approach must be attractive. Beyond that reviewers will also consider whether you have:
  1. Shown them that you and your team are capable of doing the technical aspects of the work. [feasibility]
  2. Published work that supports your hypotheses or led to your hypotheses. [evidence]
  3. Published work that demonstrates #1 above. [proof of technical ability]
  4. Embedded unpublished findings in your application that support your hypotheses or demonstrate feasibility.
  5. Provided publications, data, or letters from others that support hypotheses or feasibility.
  6. Argued effectively from analogy to similar model systems or study approaches that you are likely to succeed.
  7. Recruited similar participants (or acquired similar specimens) at the proposed pace. [proof of concept]
  8. Led studies of comparable design or complexity even if on another topic. [capability/experience]
  9. Completed key approvals that could slow or prevent the research such as IRB approval, protocol registration, or IND documentation. [administrative readiness]
  10. Research team of sufficient skill to do the described work.
  11. Have in place key infrastructure such as databases, electronically programmed and ready to administer instruments for human subjects, quality control procedures for assays, training manuals, etc. that will be required to function rapidly at the highest level. [functional readiness]
  12. Collaborators who are contributing their expertise to your work (as opposed to giving the impression you are continuing the work of a mentor).
  13. Calls from national organizations, NIH, and/or thought leaders for the work you are proposing, suggesting it is valid and important.
  • Ask yourself, have I provided what they need to know across my entire grant from the abstract to body, budget justification and biosketches that convinces them I am capable and ready to do this work?
  • Get experienced reviewers to candidly review your grant, asking specifically, have I convinced you this is important, well-founded, and do-able?

Q. Should I aim for a specific RFA? 

  • A PA – program announcement – yes. An RFA, generally, no.
  • Exceptions to avoiding RFAs for first grant applications would include calls seeking specific candidates such as under-represented minorities working in a narrow content area or instances in which your work or comparable work is cited in the RFA as the foundation for the focus of the RFA. Be sure experienced mentors read it the same way.
  • Remember many RFAs are “wired” to draw out particularly strong existing teams. Program officers are in an awkward situation of knowing this is typical but not being able to state that it is a pattern. You will go up against bigger dogs for limited resources.
  • If the RFA does not allow resubmission, aka is a one-time offer, be especially cautious.
  • Rarely, a team led by an early career investigator can justify responding to an RFA if it will drive them to move faster towards completing a research protocol and preparing a related grant. If you and collaborators would write exactly the same grant with or without the existence of the RFA it might be a more logical choice. But don’t lose sight of the timeline. Even a well-targeted diversion is still a diversion since you will not be able to submit the work as an investigator initiated RPG until the proposals solicited in the RFA are reviewed. And in some cases of uncertain score, awaiting a decision of NIH Council can further delay teams.

More Resources

BioScience Writers – Should I Apply for an R01, R03, or R21 NIH Grant Award?

NIAID – Comparing Popular Research Project Grants: R01, R03, or R21 (Key quote: “R21 projects should be distinct from those supported through the traditional R01 and are not intended as a bridge between training and obtaining an R01 grant.”)

Ten Insider Tips: What Your Grants Manager Wishes You Knew

Grants & Funding

Your grants manager is a key person in ensuring smooth submission of your grant. Here’s what every grants manager wants you to know:

1. Notify your grants manager at the very beginning of the grant writing process. This will help them schedule their time and give your grant the necessary attention.

2. Start working with your grants manager 6-8 weeks ahead of the submission date. Familiarize yourself with your institution’s specific guidelines for routing and submission.

3. Read the program announcement, or RFA, carefully and know the requirements and limitations. Make sure you are eligible to apply and the work you are proposing is doable with the resources available.

4. The grants manager should provide a list of requirements including who does what. If you do not receive this document, ask them to provide one.

5. For R01s and other non-career development grants, the budget is a key component of the grant and can be time consuming to build, especially if there are subcontracts. Start the budget early!

6. Make sure you are using the most recent and appropriate forms.

7. Ask people to be on the grant and have their permission to include them before giving their names to the grants manager.

8. Understand the different roles people can play on a grant: Co-PI, co-investigator, collaborator, consultant, and other significant contributor. (NIH definitions).

9. If you make changes to the title, budget, or other key components, let your grants manager know right away. Late changes, especially to the budget, may require modification of various other grant components.

10. Get to know your grants manager – they are dedicated to helping you succeed. And most of them like treats, especially chocolate.

Thank you to Pattie Council and Suellen Sachariat for their combined 20 years of grants management experience and sharing their expert knowledge with Edge for Scholars.

More Resources

Mapping the Path for K (or R) Submission

More Things I Wish I’d Known Before I Wrote My K

Try the NIH “All About Grants” Podcast

How to Get a K24

Grants & Funding

Four recipients of K24 Midcareer Investigator Award in Patient-Oriented Research awards shared their best advice for writing these grants that protect time for independent investigators at the Assistant or Associate Professor level to mentor. (More on why you should write a K24.)

Panelists at this Vanderbilt event:

Tina Hartert, MD, MPH
Lulu H. Owen Professor of Medicine
Lorraine Ware, MD
Professor of Medicine, Pathology, Microbiology & Immunology
Warren Taylor, MD, MSc
James G. Blakemore Professor of Psychiatry
Lisa Young, MD
Associate Professor of Pediatrics, Medicine & Cell Biology

What Is a K24, Why Do You Need One, and When Should You Apply?

Success rates for fiscal year 2021 across NIH institutes offering K24s. Data from NIH RePORT.

  • A K24 is a midcareer award that protects 25-50% of your time for mentoring and learning a new skill or expanding on an existing skill.
  • Eligible applicants are at the Assistant or Associate Professor level, R01 (or equivalent)-funded, and doing patient-oriented research.
  • Patient-oriented research involves direct contact with patients (not just the EMR or biospecimens).
  • PhDs and MDs are eligible.
  • Not all institutes offer one, and those that do have different rules about renewals, allowed effort, and other components.
  • Success rates are very high—typically over 45%, sometimes up to 100%. Many are funded on the first try.
  • Think about timing of a K24 submission in relationship to your own grant funding cycles and promotion timeline.
  • When you’re ready for the responsibility of mentoring postdocs or K scholars (which equals responsibility for someone else’s career!), this will protect your time to do it.

Do and Consider Before Writing

  • Search RePORTER for funded K24s in your institute—very helpful for getting a sense of the scope of projects proposed by others and how to uniquely position yourself.
  • Solicit advice from others with K24s.
  • Before you apply, honestly appraise your mentoring track record. Can you show mentoring success? What kind of impact have you had on people who trained with you even for a short time?
  • Know your local department and your field—how many people are like you?
  • Select the specific group of people you want to mentor; make sure it fills a need and fits with institution priorities. External mentorship counts—use professional organizations’ mentoring programs.
  • Make sure you have research projects available at the right level for the people you propose to train, from medical or graduate students to fellows to K scholars. You will need to show how your mentees can create a niche for themselves and propel their career, rather than simply extending your research program.
  • Strategically plan for what you will give up (or negotiate to give up) if your K24 is funded so that you have protected time for mentoring.
  • Get involved with national projects that show leadership and mentoring, such as organizing a Gordon conference, leading a workshop at the big meeting in your field, or blogging on Edge for Scholars. Not only can you highlight it in your application, you can also request letters of support from others involved.
  • Talk with your program officer.
  • Read institute goals and align your application with them. Make sure you read the most current. Institutes frequently revise these programs based on institutional priorities.
  • As with R01s, sometimes your grant might fit more than one institute, and you can tweak your application for the one with a better payline. Your K24 does not have to be through the same institute as your R01.

Shape Your Mentoring and Learning Strategies

  • Decide whether you want to and can best make a case for learning a new skill or technique vs. developing a logical extension of your current skillset. Make an argument about how this will drive your career.
  • Your proposed research must meet the definition of patient-oriented, but consider applying even if you have a track record in basic or translational research if you can make the case for how this will propel your career development and that of your mentees.
  • One panelist’s reviewers considered it a minor weakness to not list a senior mentor.
  • Choosing a primary mentor:
    • Choose someone not on your K23/K08.
    • Thematically look at investigators who can provide new ideas and new approaches to your research efforts.
    • Look at mentors who are successful.
    • Consider a mentoring committee rather than a single individual.
    • Consider external expertise for members of a committee.
  • Prior to writing, discuss with local training infrastructure leaders how to take advantage of both mentoring opportunities and supports available through your institution, as well as the resources your mentees can access.
  • Specify how you will find mentees. Think outside your own discipline, and even outside your institution.
  • Consider how mentees will be able to develop their own niche and eventually become independent.

Write the Application

  • Think of this grant as one big personal statement and resume. Find your story.
  • You might write this grant in a non-standard order. One panelist wrote most of the grant with a relatively underdeveloped Aims page, and then did a lot of Aims page work and realigned rather late in the grant writing process.
  • Consider using the instructions or even copying the required section headers into the application, as this grant is in a non-traditional format compared with what many reviewers are accustomed to reviewing.
  • Clarify what you will not do in order to focus on mentoring and research, and have your chair put this in writing in his or her letter.
  • Your mentoring history is really important—be sure to showcase trainee awards, funding, publications and current (hopefully academic/research) positions. Keep up with your mentees.
  • Have a clear vision of what is next. How will you use the skills you develop, and what is your next step as a mentor?
  • Some propose to write a T32 or K12 while on the K24; having a K24 in hand assures reviewers of training grants that you will have time to actually mentor those trainees.
  • The research plan needs to build upon your funded research program that is already in place. Describe your program, how the proposed research builds upon it, and specifically how it supports trainee projects and their careers.
  • If you elect to propose new research within the research plan, be aware that funds are limited and that a new project opens you up to critiques of the research project along with critiques of everything else in the application.
  • The budget allows salary support to protect your time for mentoring and use of remaining support for components of your research program that can help to support mentees.
  • Consider allocating budget for statistical support and research coordinator/nurse to support trainees especially. A K24 also reduces the salary support you must take on your R01(s), which frees up funds for trainees.

Do you have other advice for writers of K24s? Tell us in the comments below.

More Mentoring Ideas

Why You Want to Mentor via NRMN Guided Virtual Mentorships

It Is Never Too Early to Be a Mentor

Speed Mentoring: Seven Steps to a Successful Session

 

Even More Cranky

Grants & Funding

Since the holy trinity important things have come in threes—listen up. Not all beautiful things are functional or practical. Think white upholstery, stiletto heels in the lab, bilateral justification of your grants. Bilateral justification is from the devil. It’s a cognitive drain and subtle form of punishment for your reader.

Bilateral justification (both left and right margins are flush, meaning straight):

  1. Decreases reading speed/efficiency.

The eye uses the ragged right edge to scan written materials efficiently. If my subconscious is telling me I am reading slowly, it also tells me your text must be dense or dull.

  1. Impairs recall of written content.

You need your reviewers to have optimal comprehension. I will appreciate significance and innovation better, and think about your approach more lucidly, if I remember your prior material.

  1. Causes stilted writing.

You know you do it. You see a line or paragraph with unappealing spacing and change or add some words like changing “use” to “utilize”, or adding unnecessary connectors such as “therefore” to adjust the spacing. Switch to left justification and edit like a fiend. Your text will get leaner and more concise. I will be happier.

Quit ignoring the evidence about best practices for formatting (Table 3, below).

Don’t make my brain tell me I am slogging through your content. It makes me cranky.

Other resources:

Designing Instruction and Informational Text

How Document Readability and Legibility Are Affected by Text Justification

More crankiness: 

Three (Grant) Peeves in a Pod: Formatting

Three (Grant) Peeves in a Pod: Appearance Matters

Three (Grant) Peeves in a Pod: Write Better

Don’t Let Your Research Questions Go Out Without PICOTS

Doing Research / Grants & Funding

All the best aims are wearing PICOTS (pronounced “peacoats”). Specification of your PICOTS* is the minimum outerwear required to prevent your research question from being caught in a downpour of questions. Having these details tucked in gets you ready to have a meaningful conversation with colleagues, evaluate feasibility, brainstorm about how to get the best study done, and prepare to share your concept:

P = Population

I (E) = Intervention (Exposure)

C = Comparator

O = Outcome

T = Timing

S = Setting

Use PICOTS as a checklist for operationalizing research questions and probing how the research would shape up under different assumptions. Ask:

Population: 

  • What group of participants is ideal?
  • Whom does this imply we need to include or exclude?
  • How would we operationalize those criteria?
  • What influence will that have on ability to identify participants/recruit well?
  • Do we need to worry about proof of concept or generalizability more at this stage?

Intervention:

  • What will participants do/experience in the study that is being tested for its effects?
  • What dose, frequency, intensity will be tested?
  • Do we need to invoke a specific behavioral or causal model?

Or Exposure:

  • What is the behavior, biomarker, experience, metric for which we are interested in evaluating the effect?
  • How will it be measured?
  • How will we ensure quality of the measure?

Comparator:

  • What comparison provides the most relevant contrast (e.g. usual care, no intervention, placebo, etc)?
  • What analytic approach will best support the comparison?
  • Does this comparator help test our causal model or could it be stronger and more direct?

Outcome:

  • What is our measurable outcome?
  • How will measurement be operationalized?
  • Do we need primary and secondary outcomes?
  • Can we achieve adequate power to assess the outcome?
  • If there is loss to follow-up, do we have alternative ways of assessing outcomes?

Timing:

  • Over what time frame will participants be recruited?
  • What is the time period over which intervention will be conducted for an individual participant?
  • How long after completion of intervention will measures be collected?
  • When will outcomes measured? How wide is the tolerable window for measurement?

Setting:

  • Where will the research be conducted or participants be recruited (e.g. academic tertiary care center, network of health department clinics, community-based, etc.)?
  • What are the characteristics of that setting?
  • If extant data, what was the setting in which the data was developed?

Try it, you’ll like it. And it’s better than the alternative of getting soaked later by questions and requests for details needed to clarify your concept for the research.

Taking PICOTS for a Spin

For example, if you’re interested in asking: “Do community-based lifestyle interventions really work?” or “What determines who stays in community-based lifestyle interventions?” work the PICOTS:

Initial Question: “Do community-based lifestyle interventions really work?”

Goal: Pilot intervention study with a primary aim of determining if an intervention results in weight loss

In this case a pilot would be a typical approach for estimating the effect size, feasibility, participant satisfaction, loss to follow-up, and need for adjustments to inform design of a future definitive randomized trial. So we sketch a picture of what the study could look like:

P: Adult women with physician’s permission who are registered for the first session of the 12-week New Beginnings Program, and who speak English or Spanish.

I: Structured small group (n=5 to 8) coaching program with 1) specific weekly goal setting targets (eliminating sodas, understanding metabolic effects of exercise and tracking, counting carbohydrates, planning daily physical activity, enhancing sleep, writing an individual vision for one’s health, making a long term health contract with oneself, etc.) 2) three small group resistance and circuit training coached sessions each week, 3) social media peer connections, and 4) individualized exercise, diet and stress-reduction prescriptions.

C: Women who have applied for the program and are eligible but who are currently on the wait list with an anticipated wait time of 14 or more weeks.

O: Primary outcome will be weight loss, measured as difference between first measurement (in pounds to one decimal place on scale provided and calibrated by the study) at intake session and weight at the last group session. Outcomes will be grouped by completion status where completers attended ≥75% of schedule sessions and non-completers fewer sessions. Weight loss will also be described by group for each of the 12 weeks. For secular trend among those with an intention to lose weight the wait list comparison (secondary analysis) group weight will be collected from initial application (or as documented in physician’s permission letter in application) and weight at intake session, adjusted for elapsed time between application and start of program.

T: The intervention will last for 12 weeks of structured lifestyle and exercise coaching. Informal peer and social media networks established during the intervention will continue unsupervised after completion. Secondary outcome data will be collected at 3, 6, and 12 months after completion of the intervention.

S: Privately owned gym facility partnered with non-profit (501C3) to provide a comprehensive lifestyle intervention program to means-tested low income women, the majority of whom are age 40 and older, African American and weigh, on average, more than 200 pounds.

Initial Question: “What determines who stays in community-based lifestyle interventions?”

Goal: Observational study of whether baseline mental and physical health status, locus of control, and dispositional optimism are associated with completion of a community-based lifestyle intervention

P: Adult women with physician’s permission who are registered the 12-week New Beginnings Program which is a structured small group (n=5 to 8) coaching program with 1) specific weekly goal setting targets (eliminating sodas, understanding metabolic effects of exercise and tracking, counting carbohydrates, planning daily physical activity, enhancing sleep, writing an individual vision for one’s health, making a long term health contract with oneself, etc.) 2) three small group resistance and circuit training coached sessions each week, 3) social media peer connections, and 4) individualized exercise, diet and stress-reduction prescriptions.

E: Lower levels^ of physical and mental health as assessed by Short Form 36, lower self-efficacy (assessed by Generalized Self-efficacy Sale), and greater pessimism (assessed by the Revised Life Orientation Test) at baseline.

C: Higher levels^ of physical and mental health as assessed by Short Form 36, internal locus of control, and greater optimism at baseline incorporated into logistic regression models to assess association of characteristics with outcome.

^ Cut offs to be determined by distribution of traits in context of national normative reference data.

O: Program completers will be classified as those who attended ≥ 75% of scheduled sessions and non-completers fewer sessions. Will also capture week of attendance for secondary analysis in time-to-event analysis.

T: The assessment will be completed within 12 weeks.

S: Privately owned gym facility partnered with non-profit (501C3) to provide a comprehensive lifestyle intervention program to means-tested low income women, the majority of whom are age 40 and older, African American and weigh, on average, more than 200 pounds.

But I can’t possibly know these details when I first think the thought!?

True, but you can get much closer than you think. Start by daydreaming and then add parameters that are initially fantasy. The approach to shaping questions jumpstarts thinking that then leads to:

  • Productive generation and sifting of research ideas.
  • Greater focus for literature review.
  • Strategic thinking about multiple aspects of feasibility .
  • Weighing the best choices for measures of exposure, covariates, and outcomes.
  • Enhanced ability to rapidly gather input from others.

Related Posts:

Acing Your Observational Research Aims

All research proposals – grants, dissertations, internal funding – must ace the description of aims.  Many scientific questions are interesting.  Not all are useful.  You must persuade your readers that the proposed aims/hypotheses to be tested and the related analysis will fill gaps in scientific knowledge.

Don’t Crash on Approach

Getting the approach – the methods section of your grant –  fine-tuned is literally the heart of it all. You must land your science smoothly. Study section members know, and recent evidence confirms, your grant’s score is not an equal weighting of component scores. NIH criterion scores are for significance, innovation, approach, investigators, and environment.

* Gordon Guyatt initially described PICOTS in Guyatt G, Drummond R, Meade M, Cook D. The Evidence Based-Medicine Working Group Users’ Guides to the Medical Literature. 2nd edition. McGraw Hill; Chicago: 2008. Subsequently the framework became standard for formulating inclusion and exclusion criteria for conduct of systematic evidence reviews and meta-analyses of interventions.

Using NIH RePORTER to Find Your Guide

Grants & Funding

In much the same way the Assisted Referral Tool can help you pick a study section, the Program Official option for NIH’s Matchmaker tool provides insight into the Program Officer who works with the most projects that look like yours.

To use it, visit the Matchmaker portion of NIH RePORTER. (New to RePORTER? Here’s how to find out a wealth of information about grants on your campus and elsewhere.) In the text box, you can enter a paper or grant abstract, or any other text you want to search on up to 15,000 characters. Click the “Similar Program Officials” button.

In a few moments, the system will return a list of up to 175 program officers, starting with those whose grant portfolio most closely matches the text you put in. Click on a name to bring up a list of the active grants for whom that person is the PO. Are they in the same ballpark as your research? Then go back to your search results and grab the email address of that PO to start a conversation.

Why might you want to talk to your PO before submitting a grant? Several reasons:

  • To find out if the institute is enthusiastic about your research area or if there might be a better fit.
  • To get clarification on which study section is ideal.
  • To confirm the appropriate FOA.
  • To get suggestions for alternative programs, FOAs, or institutes.
  • If you have questions about your budget or scope of work.

Also of note: Matchmaker will give you graphs of how many POs at each institute work with grants that look like yours, as well as how many of which grant mechanisms (R01, R03, K08, etc.) they handle.

Additional Resources

Harness the Immense Power of Nosiness in NIH RePORTER

Which Study Section Should I Pick? Try the Assisted Referral Tool!

Video: How to Use NIH Matchmaker