Doing Research – Page 4 – Edge for Scholars at Vanderbilt

Not that Kind of Bias: Tales of Survivorship Bias

Doing Research February 9, 2020

I am in my fourth year as a faculty member and experiencing the typical “how do I get these grants funded” struggles that many, if not all, of us face. Over the past year, I have been given a truly staggering amount of conflicting advice from mentors. This has led me to thinking about survivorship bias in the advice we receive from well-meaning mentors, Twitter, blogs (ahem), and books. The irony is not lost on me: after writing advice/experience blogs on and off for almost three years, I am finally writing one on survivorship bias. This time, n= all of us.

Survivorship bias: Survivorship bias is bias that occurs when only survivors are examined. A classic example of this you will see come across Science Twitter is a graphic of an airplane with red dots, which is a reference to airplanes returning from combat during World War II with bullet holes on the wings. History goes, that in an attempt to minimize airplane loss, military intelligence suggested that they reinforce the areas that were targeted. However, Abraham Wald pointed out that it was the bombers that did not return that represented lethal strikes to the airplanes, and airplanes that had returned likely represented non-lethal strikes. Thus, it was more appropriate to reinforce the areas that never returned with bullet holes, like the pilot’s cabin and the engines. In medical research, this amounts to performing a clinical trial and only analyzing the patients who survived the intervention.

Consequences in an academic research career: There are two major points to consider relevant to survivorship bias in an academic research career. The first is perhaps the most important: we obsess over figuring out “what worked” for successful applicants (survivors). For example, to characterize what makes a successful faculty application, we examine postdocs who successfully transitioned into faculty appointments and characterize what their CVs looked like. Every single postdoc association is likely to have the local new PI come in and talk about how to get a tenure-track academic job. Is there value in these types of seminars? Of course, but we all should be disclosing that this is what worked for us, it may not work for others, and others have found success other ways. The other side of survivorship bias in academic research careers is we become surprisingly superstitious and convinced of the importance of certain things, like the number of aims in a grant or the font in which they are written or how the text was justified (or not). Again, each of us is strongly influenced by what has worked or not, no matter how tenuous the correlation.

Everyone in this career path has survivorship bias: Everyone in this career path, from graduate students through tenured faculty, is biased. Many of us will say, “I did X and it resulted in Y,” or “I do not do X because it did Y.” Advice on grant writing is particularly rife with these types of comments, which is unsurprising considering the paylines at some institutes. I have been told to submit something every grant cycle (or only when the grant is ready or more than one grant per cycle). I have been told to only focus on big grants (or only pilot grants or a mix of grants or only non-NIH grants). I have been told to get more papers out before submitting (or only include unpublished preliminary data or submit as soon as possible so paper expectations are low). Whose advice is the right advice?

How to synthesize less-biased advice: My best advice (ha) to even out the biased advice we get is to solicit input from many individuals on broad topics (like how to run a lab, how to organize time, etc.) and get advice from qualified individuals on specific topics (like how to make this grant on Y competitive at study section X). On broad topics, collecting advice is a good approach. For example, there are many ways to run a research program, and some may not be appealing to you. If you do not sample broadly, you might not discover what works for PIs with your management style. Conversely, some advice should come from specific individuals. Advice on grant content from a standing member of a study section from your institution is priceless. Advice on grant content from your fellow new PI with a grant from said study section a little less so, although it is useful to see the expectations for early stage investigators (ESI). Advice from a legend in the field who writes always-funded renewals, who could turn in a piece of paper with “I need $1.25 million for five years” in the spirit of Otto Warburg and still be funded, should probably be used for high-level conceptual advice.

There is one shining piece of good news when it comes to survivorship bias: most of the mistakes your mentors tell you they have made are likely to be survivable. So go forth, take a chance, and make some mistakes. But when you succeed and tell tales of your success, admit that you advice is entirely biased. Stay tuned for more tales!

Did I get it wrong? Need to vent? Feel free to send some electrons my way in the comments, via Twitter @PipetteProtag, or through traditional electronic mail pipette.protagonist@gmail.com

Not that Kind of Letter: Tales of Rejection

Doing Research / Grants & Funding January 11, 2020

I have been thinking about rejections in science. Rejections come in all shapes and sizes, from the grant you need to build your program, to an awesome rotation student picking another lab, to a manuscript rejection at yet another journal. While I definitely had my share of rejection emails as a graduate student and postdoc, there has been an exponential rise since starting my faculty appointment. So how is a new principal investigator (PI) to deal? As always these days, n=me.

The first rejection of any type hurts the most: There is no magic pill to lessen the blow of the first rejection. Going into my faculty position with my successful K99/R00, I thought securing my first R01 would be challenging, but not impossible. Needless to say, when my first R01 submission swung the other way and scored terribly, it was a brutal awakening. I have put in numerous grants now and while annoyed and concerned about my lack of R01 funding, my recovery time with each unfunded grant has decreased substantially.

Rejections in science can feel personal: One of the more challenging aspects of rejections in science is that they can feel deeply personal. For example, in grants, not only are your ideas carefully (or perhaps brutally) dissected, but your entire training, publication, and funding history are examined and commented upon. While I do not believe most reviewers aim to be cruel, sometimes objective statements carelessly phrased cut deeply. Similarly, when a talented postdoc or rotation student picks a more established laboratory, it can feel like a judgement on your mentoring philosophy, potential, and research vision.

It is OK to be sad: Mourning the loss of possibilities arising from an unfunded grant, unsecured postdoc, or rejected manuscript is normal. I often mope for a day (or two) and then usually throw myself into something that reaffirms my skills in the area of rejection. For example, on grant rejections, I throw myself back into lab work and moving papers along or look at the holes in the data, which the reviewers pointed out.

Talk about it: Venting is good for the soul. Yes, the reviewer completely missed the mark and how dare they not see the brilliance of my research proposal? How could the rotation student pick that other lab and not see the vision of their thesis that you had outlined? Sometimes it is easier to see the validity of a criticism when you discuss it. Moreover, by talking about rejections we normalize them. Fifty percent of grants in a study section are not discussed! The majority of grants that are discussed are not funded! Rejection is the norm in this job. By talking about it, we accept that it is frequent, normal, and inevitable.

Make a plan and move forward: After you have gone through the five stages of grief, make a plan. How are you going to address reviewer comments on your triaged grant? Prepare to spend some time with those horrible pink sheets and figure out what really did not work. Not sure what went wrong? Ask for help from your peers, mentors, community members. Sometimes realizing that you are not competitive for a grant or your lab is not the right fit for graduate students is an important part of moving forward.

Find a community that celebrates wins: There are many more losses than wins in this job. It is simply the nature of our chosen profession, and one way to lessen the blows of these seemingly endless rejections is to find a community that celebrates each other’s victories, great and small. Be this Twitter, New PI Slack, your department, or a group of assistant professors, find your people. Knowing you did not get the grant, but one of your friends did makes the rejection sting less.

These are my approaches to dealing with rejection, and I am sure there are many more. Whatever your approach, do not let the fear of rejection keep you from applying for grants, awards, and recruiting talented trainees! The worst they can say is “no”. Stay tuned for more tales!

Still have questions? More confused than when you started? Need to vent about the process? Feel free to send some electrons my way in the comments, via Twitter @PipetteProtag, or through traditional electronic mail pipette.protagonist@gmail.com

More Resources

More Friendly Advice: When Rejection Isn’t Really Rejection

Fighting Rejection: Three Little Reviewers

Dealing with Rejection

Thoughts from a Self-Proclaimed Interdisciplinarian

Doing Research July 28, 2019

I am scientist trained in sociology, social work and public health (behavioral sciences, to be exact), who did a postdoc in a Pharmacy department and is now a tenure-track Assistant Professor in a College of Nursing. There are pros, cons and lessons learned from my interdisciplinary and inter-professional journey worth sharing as you consider your next job or career move.

Pros

My training gives me an incredibly unique and holistic perspective that I bring to my research. My field of research is complex and multifactorial. Because I live between disciplines, I can borrow from a variety of disciplinary frameworks and approaches to address issues and collaborate with others.

I have developed broad professional networks across institutions, departments and disciplines. These scientific and professional networks have introduced me to numerous colleagues across the country who work in my field, and helped me secure professional development opportunities, such as conference travel awards (American Association for Cancer Research Travel Grants) and sponsored research training opportunities. These networks also allow me to be a resource to the students I mentor and to other faculty looking for various career and professional development resources.

My background also opens up new possibilities for funding and publishing. As a postdoc, I applied for and secured a fellowship that commonly targets pharmacy faculty. I am also able to frame my research findings for audiences across a broad variety of discipline specific journals.

Cons

Sometimes I feel like I don’t have a home. Many academic and research funding institutions are siloed by disciplines, diseases, or health professions. I have not pledged my devotion to a single discipline (although I do get very excited when I meet other social workers!).

In line with this, I have found it hard to find a professional society or organization that can best support my career and professional development. I have been a member of a number of organizations, but often don’t feel that they are meeting my needs or giving me service opportunities needed for tenure and promotion. Stay tuned because I will be attending a new conference in a few weeks that I think may be my unicorn!

Lessons Learned

Be open to the guidance of others. At every major turning point in my scientific training and career, there were colleagues and mentors who suggested I consider opportunities I would have never imagined. Ironically, I was not a good fit for some faculty positions I applied for in social work and public health departments. Yet I ended up in Nursing because a colleague urged me to apply and it’s been a perfect fit for me in so many ways.

Be more concerned with solving problems than with your disciplinary approach. My obsession with creating solutions to reduce health disparities has guided my research and career. The problems we face as scientists are increasingly more complex to understand and solve. I think it’s safe to say that most of these problems cannot be solved by a single discipline. By working with other scientists from different disciplines, we are more likely to reach new solutions. For example, I am currently working with a team of mentors (e.g., behavioral scientist, nurse scientist, epidemiologist) to address the age-old problem of “medication adherence” by developing a multi-level intervention to improve adherence to endocrine therapy among breast cancer survivors. We believe this new approach to an old problem is promising.

More Resources

It’s All About Teamwork: William Wester

How to PhD: 10 Tips from Hindsight

Resilience as a Common Trait in Researchers

Using Timelines to Diagnose Problems in Career Planning

Doing Research / Mentoring April 2, 2019

This post builds on a presentation at the 2019 CTSA Annual Meeting of Training Program Directors.

Reviewing intended career progression is a cornerstone of mentorship. Goals for milestones such as manuscript and grant submissions typically dominate discussion. Often goals are described in documents for mentor panel meetings, individual career development plans, or alluded to in career documents like the scholar’s CV or biosketch.

In our early career faculty development programs, we’ve become convinced that looking at something is better than talking (or reading) about it. We find translating intended goals into a single simple image as a timeline is a powerful tool. Preparing the timeline requires a scholar to more deeply examine feasibility of plans by requiring they take into account timing of built in delays such as those:

Between grant submission, review, resubmission, review, and funding.
Typical in the review-to-publication cycle for journals to which they intend to submit.
Allowing for submissions of manuscripts to multiple journals.
Caused by small grants and awards pulling attention away from building their core scientific mission and research team if not perfectly aligned.

Our program’s use of timelines began with a basic retrospective examination of outcomes. We saw more timelines like the one below than we hoped. [Note R01 is an oversimplified label in our examples for all large awards including VA Merit, U01, CDC contracts, and other funding at a value of ≥$250K direct for a minimum of three years.]

The majority of our scholars are tenure track when they start their career development award. Most awards were five years long (culminating at the yellow line). At our institution, fairly firm decisions are made about remaining tenure track or changing tracks at the seven-year mark (dark orange warning line).

This is an uncomfortable, risky and potentially costly small lag in funding:

The scholar has forfeited the benefits of being allowed up to two years of overlap of their K and first large-scale grant.
They have fallen off the K-cliff from 75%+ protected time to PI-level effort for a single grant which creates additional funding needs.
At the end of the K the individual does not know the score on their resubmission.
They need bridge funding at the end of the K to continue their work.
Delays in federal start dates, now common, can compound the pressure.
Bridge funding at the end of a K, when work is underway and related staff may have been hired, is costlier than “pre-bridging” by allowing time for pilot work to progress further before starting a K award.
Time to launch a new round of submissions is insufficient within the pre-tenure review window.
The faculty member’s posture with the department can change from achiever to “needs watching.”
If they were tenure track before receiving the K award, they are in a more profound bind.

To best advise mentees and mentors about ideal pace, we then asked what pace was implemented by K and other career development awardees who successfully transitioned from their CDA to R01 or comparable funding. We restricted to individuals who made this transition and then compared the timelines of those who made the transition within five years – the typical period of a career-development award – and those who achieved funding as an independent investigator but did so after more than five years had elapsed.

For simplicity in this aggregate analysis, we included scientific publications related to the content areas or methods of continued research and timing of submission and funding of proposals of varied types. The image below reflects the average timing of publications and R01 submissions within the two groups:

During comparison several elements of mentoring surfaced that required myth busting. Among those who took more than five years to transition many had been told:

Earlier than the third year was too early to submit. (Not all mentors were aware of allowance for overlap of K and R.)
Explicitly that the fourth year was fine for initial submission and some had been advised to pursue R03 or R21s first, delaying their receipt of funding sufficient to support a research team.
They needed more or higher impact publications before submitting.

Some had attempted to submit earlier and found they:

Received worse internal reviews than expected and needed to delay to fix flaws.
Underestimated the overall complexity of the process.
Did not have ideal data about feasibility and proof of concept, having focused on narrow bands of pilot data.
Underestimated power and needed to add sites or collaborators.
Underestimated costs, discovered this too close to intended submission and needed to re-scope their project.

Combined, these mentor and process factors were most related to delayed success. Very few faculty attributed delays to parenting or challenges outside of work. In aggregate most felt they had been naïve about the need to start earlier.

In response, we initiated grant pacing workshops to break down all the component parts of large grants and develop timelines for completion simultaneous with recommending (or requiring for some groups) that faculty scholars complete, review, and revise a timeline a minimum of twice a year for each mentor panel.

In reviewing these intended timelines (purple line marks when prepared), we have found them key for rapid diagnosis of concerns that are more difficult to detect in formal academic documents, bulleted lists of accomplishments, or text description of goals.

As an example, the faculty member below was rightly perceived by division chief and chair as highly successful, producing seven meaningful publications in solid journals, presenting at multiple national meetings including plenary sessions, and receiving a prestigious professional organization scientific award for $125K a year for two years. Everyone was happy, until they thought about the implications of this timeline submitted as part of an annual report. The individual had 18 months on tenure track prior to CDA. In this picture that means Year 5 is actually Year 7 on the tenure clock and Year 7 is Year 9, at which time they face an up-or-out promotion review without knowing until the final hour if they may qualify.

Ideally this would have been prevented by discussing the timeline in Year 1 or 2. Even as late as Year 3 of the CDA, we were able to dispel the idea that the individual could spend a year focusing predominantly on the professional association award. Mentors promoted tight integration of that award with preparation for an R that moved somewhat away from the K. This allowed earlier submission of the R, with less time elapsed before resubmission and, by very good fortune and good science, earlier receipt of funding by almost a full year.

Timelines for evaluations reveal key patterns that mentors and career development leadership can identify and disrupt to be sure timelines reflect the pace at which funding is needed and the milestones implied.

We see these as themes:

Distracted mentors losing track of time or providing poor advice that did not emphasize necessary pace for achieving first major grant. These scholars need a real timeline!
Over-alignment of K aims to R when some portion has not panned out or findings have proven unenticing. They need encouragement to deploy talents in related, higher value pursuits.
Distracted scholars chasing small shiny objects. Among those who submit 7 to 11 smaller value awards during their CDA, none achieved prompt transition to independent funding. Lower numbers are also concerning. They need an ultimatum as time gets short to keep the main thing the main thing.
Idea hamsters who submit multiple proposals with a wide range of different methods and hypotheses, and/or those who prepare publications on multiple topics that do not build to an acknowledged area of mastery. Same intervention as distracted scholars.
Trust fund recipients who were aggressively wooed and provided robust start-up funds. They are at risk of misperceiving this as a safety net that can help support future salary needs. They often under use the funds in launching the appropriate scale of hiring and research investments. They need coaching on using their resources to accelerate the work and produce a large scale grant sooner.
Comfort in poverty shows up as individuals who lack confidence in or do not have a personal vision for their career. They behave as if they are earlier in their careers and fail to exercise agency in decision making. They typically don’t prepare a timeline until pressed or they project too much time will elapse and ruminate on failure. They allow mentors and others to dictate what is needed rather than determining from their own goals. They need consistent encouragement to differentiate from mentors and clarify their passion for research.
Time management fails among individuals who do life with their “hair on fire” and have multiple explanations for why key pieces of scientific endeavors or grant preparation derail without acknowledging the lack of sufficient lead time and lack of use for resources like expert review of aims and internal study section that are associated with higher levels of success. These individuals push back their timeline each time they draw it. They need accountability.

The rarest are:

Failed scientific ideas when methods cannot be replicated or results of experiments derail intended direction.
Mentor departures that disrupt the progress of the early career faculty member by following the mentor or needing new mentoring contacts.

The latter two are unavoidable, even with timelines, and are worthy of other forms of consideration.

The good news is the majority of scholars who start to prepare realistic timelines around the time they decide to seek a K award can march out that plan. They find it helpful and anxiety reducing. Being oriented to what one needs to do and demystifying the steps is empowering.

Download career development timeline template.

Be Proud of Your Accent! Give Confident Conference Presentations

Doing Research / Faculty Life February 7, 2019

Is English not your first language?

Currently, and this could well change, most international research is communicated in English. For now at least, all international researchers need to become proficient at speaking English in public. Using interpreters at conferences is truly difficult and a great deal is lost in translation.

Now, it is well known that many people fear public speaking second only to death, so how much more frightening is it to have to give a formal presentation in a language other than your mother tongue? And how does this fear affect your speaking technique?

It’s not the accent, it’s the fear!

In recent years I have worked at Australian universities, training postgraduate students how to communicate their research. During this time, some PhD students undertook some special training for the 3MT© (Three Minute Thesis competition) and as part of this training, we had an expert give a workshop on voice projection and “presence.” This was fun to watch, and it was really all about learning to be comfortable with yourself and speaking out.

Some weeks after attending this workshop, one European student who had a very strong accent asked if she could present a 30-minute seminar to one of my workshops for new PhD students. I said yes, and she came and gave a powerful and exciting talk. All the new students admired her confidence and thought her presentation was great.

It was really enjoyable listening to her speaking now that she was no longer embarrassed by her accent and was projecting her voice fully. Her accent was still very strong, but her full voice projection fully engaged us. Her confidence and overall presence were now so powerful that her sometimes poor grammar, mispronunciation and misuse of words no longer mattered at all!

Fear makes many people swallow their words so that the audience can neither hear nor understand what they are saying. But if they can learn to speak out, whether or not the language and actual words are correct, the audience will still usually be able to understand and enjoy the talk. Communication is far more than words. Certainly, the actual words are very important whenever someone is trying to explain detailed and important research, but props like diagrams and figures can be used to fill in some of the details.

It is actually much more important to have great projection and appropriate body language than to have the exact words and pronunciation.

Can you be over-confident? Strong English accents, dialects and speed

Lacking confidence is, however, not the only problem in clear communication in English. Native English speakers who have strong dialects and accents are often extremely hard to understand. If you have a strong accent, even if English is your first language, many people will simply not understand you. You must slow down and if at all possible, reduce your accent.

People who speak too fast can also leave most of the audience bewildered. Since the speed of speech tends to be cultural, you might not be aware of your speed unless you really stop and take notice. Even if your accent is clear, if you speak too fast, you are likely to leave your audience behind!

Always remember that as well as hearing you speak, your audience needs time to process the information you are presenting. You are no doubt extremely familiar with your own work, but for most of the audience, it is the first time that they have heard it. Even the sharpest minds need a few seconds to process new information. Speaking too fast is a poor quality in any presenter.

Difficult, unusual or critical words need to be displayed visually

One very simple way to ensure that your audience understands your key message: Write it down or display it in some type of visual. You should adopt this habit regardless of whether English is your first language.

Don’t ever run the risk of your audience thinking that you are speaking about anything other than your key topic! You might be surprised how often people come away from a talk with completely the wrong message.

Make sure that this never happens to you.

I provide individualized speaker training at my site.

Don’t Let Your Research Questions Go Out Without PICOTS

Doing Research / Grants & Funding November 1, 2018

All the best aims are wearing PICOTS (pronounced “peacoats”). Specification of your PICOTS* is the minimum outerwear required to prevent your research question from being caught in a downpour of questions. Having these details tucked in gets you ready to have a meaningful conversation with colleagues, evaluate feasibility, brainstorm about how to get the best study done, and prepare to share your concept:

P = Population

I (E) = Intervention (Exposure)

C = Comparator

O = Outcome

T = Timing

S = Setting

Use PICOTS as a checklist for operationalizing research questions and probing how the research would shape up under different assumptions. Ask:

Population:

What group of participants is ideal?
Whom does this imply we need to include or exclude?
How would we operationalize those criteria?
What influence will that have on ability to identify participants/recruit well?
Do we need to worry about proof of concept or generalizability more at this stage?

Intervention:

What will participants do/experience in the study that is being tested for its effects?
What dose, frequency, intensity will be tested?
Do we need to invoke a specific behavioral or causal model?

Or Exposure:

What is the behavior, biomarker, experience, metric for which we are interested in evaluating the effect?
How will it be measured?
How will we ensure quality of the measure?

Comparator:

What comparison provides the most relevant contrast (e.g. usual care, no intervention, placebo, etc)?
What analytic approach will best support the comparison?
Does this comparator help test our causal model or could it be stronger and more direct?

Outcome:

What is our measurable outcome?
How will measurement be operationalized?
Do we need primary and secondary outcomes?
Can we achieve adequate power to assess the outcome?
If there is loss to follow-up, do we have alternative ways of assessing outcomes?

Timing:

Over what time frame will participants be recruited?
What is the time period over which intervention will be conducted for an individual participant?
How long after completion of intervention will measures be collected?
When will outcomes measured? How wide is the tolerable window for measurement?

Setting:

Where will the research be conducted or participants be recruited (e.g. academic tertiary care center, network of health department clinics, community-based, etc.)?
What are the characteristics of that setting?
If extant data, what was the setting in which the data was developed?

Try it, you’ll like it. And it’s better than the alternative of getting soaked later by questions and requests for details needed to clarify your concept for the research.

Taking PICOTS for a Spin

For example, if you’re interested in asking: “Do community-based lifestyle interventions really work?” or “What determines who stays in community-based lifestyle interventions?” work the PICOTS:

Initial Question: “Do community-based lifestyle interventions really work?”

Goal: Pilot intervention study with a primary aim of determining if an intervention results in weight loss

In this case a pilot would be a typical approach for estimating the effect size, feasibility, participant satisfaction, loss to follow-up, and need for adjustments to inform design of a future definitive randomized trial. So we sketch a picture of what the study could look like:

P: Adult women with physician’s permission who are registered for the first session of the 12-week New Beginnings Program, and who speak English or Spanish.

I: Structured small group (n=5 to 8) coaching program with 1) specific weekly goal setting targets (eliminating sodas, understanding metabolic effects of exercise and tracking, counting carbohydrates, planning daily physical activity, enhancing sleep, writing an individual vision for one’s health, making a long term health contract with oneself, etc.) 2) three small group resistance and circuit training coached sessions each week, 3) social media peer connections, and 4) individualized exercise, diet and stress-reduction prescriptions.

C: Women who have applied for the program and are eligible but who are currently on the wait list with an anticipated wait time of 14 or more weeks.

O: Primary outcome will be weight loss, measured as difference between first measurement (in pounds to one decimal place on scale provided and calibrated by the study) at intake session and weight at the last group session. Outcomes will be grouped by completion status where completers attended ≥75% of schedule sessions and non-completers fewer sessions. Weight loss will also be described by group for each of the 12 weeks. For secular trend among those with an intention to lose weight the wait list comparison (secondary analysis) group weight will be collected from initial application (or as documented in physician’s permission letter in application) and weight at intake session, adjusted for elapsed time between application and start of program.

T: The intervention will last for 12 weeks of structured lifestyle and exercise coaching. Informal peer and social media networks established during the intervention will continue unsupervised after completion. Secondary outcome data will be collected at 3, 6, and 12 months after completion of the intervention.

S: Privately owned gym facility partnered with non-profit (501C3) to provide a comprehensive lifestyle intervention program to means-tested low income women, the majority of whom are age 40 and older, African American and weigh, on average, more than 200 pounds.

Initial Question: “What determines who stays in community-based lifestyle interventions?”

Goal: Observational study of whether baseline mental and physical health status, locus of control, and dispositional optimism are associated with completion of a community-based lifestyle intervention

P: Adult women with physician’s permission who are registered the 12-week New Beginnings Program which is a structured small group (n=5 to 8) coaching program with 1) specific weekly goal setting targets (eliminating sodas, understanding metabolic effects of exercise and tracking, counting carbohydrates, planning daily physical activity, enhancing sleep, writing an individual vision for one’s health, making a long term health contract with oneself, etc.) 2) three small group resistance and circuit training coached sessions each week, 3) social media peer connections, and 4) individualized exercise, diet and stress-reduction prescriptions.

E: Lower levels^ of physical and mental health as assessed by Short Form 36, lower self-efficacy (assessed by Generalized Self-efficacy Sale), and greater pessimism (assessed by the Revised Life Orientation Test) at baseline.

C: Higher levels^ of physical and mental health as assessed by Short Form 36, internal locus of control, and greater optimism at baseline incorporated into logistic regression models to assess association of characteristics with outcome.

^ Cut offs to be determined by distribution of traits in context of national normative reference data.

O: Program completers will be classified as those who attended ≥ 75% of scheduled sessions and non-completers fewer sessions. Will also capture week of attendance for secondary analysis in time-to-event analysis.

T: The assessment will be completed within 12 weeks.

But I can’t possibly know these details when I first think the thought!?

True, but you can get much closer than you think. Start by daydreaming and then add parameters that are initially fantasy. The approach to shaping questions jumpstarts thinking that then leads to:

Productive generation and sifting of research ideas.
Greater focus for literature review.
Strategic thinking about multiple aspects of feasibility .
Weighing the best choices for measures of exposure, covariates, and outcomes.
Enhanced ability to rapidly gather input from others.

Acing Your Observational Research Aims

All research proposals – grants, dissertations, internal funding – must ace the description of aims. Many scientific questions are interesting. Not all are useful. You must persuade your readers that the proposed aims/hypotheses to be tested and the related analysis will fill gaps in scientific knowledge.

Don’t Crash on Approach

Getting the approach – the methods section of your grant – fine-tuned is literally the heart of it all. You must land your science smoothly. Study section members know, and recent evidence confirms, your grant’s score is not an equal weighting of component scores. NIH criterion scores are for significance, innovation, approach, investigators, and environment.

* Gordon Guyatt initially described PICOTS in Guyatt G, Drummond R, Meade M, Cook D. The Evidence Based-Medicine Working Group Users’ Guides to the Medical Literature. 2nd edition. McGraw Hill; Chicago: 2008. Subsequently the framework became standard for formulating inclusion and exclusion criteria for conduct of systematic evidence reviews and meta-analyses of interventions.

Publishing Null Results

Doing Research / Writing & Publishing August 19, 2018

Nearly every scientist has felt the frustration of pouring effort, money and (sometimes) tears into a project only to get null results. The elusive p<0.05 decides whether results get published or not—the oft-mentioned file drawer problem. Others have explained better than I can why this is a problem. Rather, I’d like to contribute some tips I’ve picked up on how to publish null results.

Combine results with a significant result. If the story of the paper allows it, consider adding the null results to another related analysis and publishing one big paper. I’ve had some success with this myself, where I decided to include three analyses of biomarkers that were non-significant with one that was.
Consider different journals. There is a bias to not even review null results. So consider other options such as open access journals or lower tier journals. The science still gets out there and a publication is better than no publication. Some journals specifically focus on null results such as PLOS One’s Missing Pieces and the Journal in Support of the Null Hypothesis. While I personally haven’t tried any journals focusing on null results, I have had some success looking beyond the top tier journals in my field and even considering journals in other fields.
Don’t give up easily, but know when to fold. One part of publishing null results is getting used to rejection, particularly not even having papers reviewed. If you know your methods are solid and your science strong, don’t give up. Human beings are prone to bias and this likely includes a bias against wanting to publish null results (particularly if it counters a favored hypothesis or theory). Be mindful, though, of how much time you have to spend reformatting for different journals compared to the time you need to spend on other responsibilities. There is no shame in giving up once you’ve made a good effort.
When designing your study or analysis, try to design it in such a way that even null results are interesting. This, obviously, won’t help after you’ve analyzed your data but you can still consider why the null results would be useful.
Add power analyses. I’ve tried this myself and it does seem to help. Showing that your study was powered to find a typical effect size for your field helps establish that the result wasn’t just because the study was too small or had too much error.

These strategies don’t negate the root cause of the file drawer problem, namely a publication system that wants flashy, highly citable papers, and null results just aren’t that. I don’t have any good suggestions for how to solve the systemic parts of the problem except what others have already said: more journals that specifically publish null results or having current journals commit to publishing more null results or abandoning p-values altogether. Hopefully these tips will help other new scientists until the field addresses the problem.

How to Protect Your Protected Time

Doing Research / Faculty Life / Productivity June 25, 2018

You’ve just gotten your K award—awesome! 75% of your professional effort is now protected to focus on your research and career development. But wait. What about that class you teach, or those days your department expects you to be in clinic, or the students whose dissertation committee you’re on, or…

Keeping 75% of your time protected can quickly get complicated. Compliance experts Tesha Garcia-Taylor, MBA, and Robert Dow, MBA, recently presented to a group of Vanderbilt career development awardees their top tips for keeping on the straight and narrow.

First, consider the pie. Pizza, if you like. The pie is all of the effort you give to your work in an average year, across research, clinical, teaching, and any other activities you probably wouldn’t do unless your institution was paying you. It includes everything from being in the OR to reading cell cultures to preparing a class syllabus to checking your work email. Of this entire pie, 75% (six slices of your standard eight-slice pizza) should be a.) your research, or b.) your career development, which can include writing grants, presenting your work at meetings, and other things that might not be specifically sitting at a bench/interviewing research subjects/analyzing data that we’ll get to in a moment.

“But I’m in clinic 15 hours a week. Isn’t that more than 25% of a week?” Well, what’s a normal work-week for you? More importantly, what’s a normative week for your profession? Specifically, for your specialty—surgeons are more likely to work 80-hour weeks on a standard basis than PhD scientists, for example. At Vanderbilt, a typical work-week for most, not all, of our faculty is right around 60 hours. So 15 hours a week of clinic would actually be exactly 25% of a week in that scenario. (This ignores the fact that you might also want/need to do things like go to grand rounds or complete compliance training, teach clinical trainees, etc., so best not to assume that 15 hours is all you’d be doing that isn’t research or career development.)

“Okay, but half the faculty in our tiny department just went on maternity leave, and I have to teach this and cover for that and all these other things. I can’t just say nope, sorry. What do I do?”

Option 1: Explode your workweek to 100 hours. Fit in 25 hours of teaching/clinic/other and focus on your career development for 75. We at Edge for Scholars (also anyone sane) do not recommend this option.

Option 2: Let the class/clinic/whatever take up 40% of your time this month or quarter, but devote 90% to your K work for the next month/quarter. Effort should average out over the year, not the day or even the month. That said, keeping effort balanced each quarter is preferable, because it’s easy to let things slide until there’s not enough time left in the year to get the right average.

Option 3: Say to your leadership, “I’m coming up on my annual progress report/I’m six months into my award/I’m [fill in appropriate time marker here], and I’m concerned that things aren’t squaring up with my effort on this K award.” It’s not an urban legend that institutions around the country have had to give back money to the feds because they didn’t let a K awardee have his or her protected time. Your department doesn’t want to give back grant money or be scrutinized by the compliance office or NIH, we promise.

However, the best time to discuss your effort with your boss is before you submit the grant proposal. Mutually decide what activities you will put down if you get the award, and get that agreement in writing in the letter of institutional support. When thinking about what you would drop to focus on your K award, consider a few things:

What can you really not miss? If everyone in your department including Professor Multimillion Dollar Lab goes to the department seminar, you’re going to the seminar. But that likely means you can miss journal club. Go to the things that are most relevant to you, not to every event.

As well, what national things should you keep attending? If you’re going to a meeting to network with potential collaborators and disseminate your research, of course keep going; this falls under your K effort. But if, for example, the society for your specialty has an annual meeting that’s mostly attended by clinicians in private practice, you don’t necessarily need to be the one who takes the residents there to present. Consider attending some meetings every other or every few years.

Does it overlap? Many activities that look like service may ultimately end up feeding into your research. Say you need to learn how to read a particular kind of PET scan or a how to perform a new microscopy technique, so you visit a colleague and learn it from them. In learning the new thing, you work on scans or samples from your colleague’s work that need to be read/analyzed. Does this help them get through a number of scans or samples? Sure, but it also helps you master these skills, so as far as the feds are concerned, that effort fits with your K. Similarly, teaching a student who’s working on your research may pay dividends for them in the form of a degree, but it also helps your research get done.

Get to know your financial officers. They want to help you understand and comply with regulations around effort. If you don’t already receive a monthly budget and effort report, ask for regular updates to make sure they line up with reality. This will become even more important as you move on to larger grants and run a bigger research team.

Protecting 75% of your time requires you to be proactive. Check in regularly with yourself to make sure you’re spending the right amount of effort on your K work, and follow the advice above if things start looking unbalanced.

More Resources

Designing Your Career

Not that Kind of Grant Application: Tales of Career Development Awards

More Things I Wish I’d Known Before I Wrote My K

Not that Kind of Year: Tales of Year 1 as a New PI

Doing Research December 20, 2017

It has been about a year since I started my new position as a principal investigator (PI) at Clinical Department in R1 University. It has been a challenge. Here are my suggestions on surviving the first year, based on areas where I succeed and failed. As always these days, n=me.

Find your people: The shock of loneliness in the first year is significant. You no longer have a built-in community of graduate students and postdocs with which to spend time. Your employees and your trainees cannot be your friends. You are their employer and/or mentor, and while this leaves abundant room for kindness, compassion, and well-wishes, the power differential is prohibitive for true friendship with your staff and trainees. The easiest way to deal with this loneliness is to find your community. Potential communities include other new PIs in your department, your institution, or your professional society. There are even peer-to-peer mentoring groups, like New PI Slack, which provide access to a community of new PIs who have gone, are going, or will go through similar trials and tribulations. However you decide to do it, make sure you find a community to support you and that you can support in turn.

Find balance with life: This has been a point of failure for me. I let the work take over this year. Do better than I did. Make sure you leave dedicated time for your hobbies, friends, and family. The administrative work will be there Monday morning too.

Find balance with collaborations: Find a balance in starting collaborations and be protective of your time. As a new PI you have a pot of unallocated money that you can invest in projects of interest. Established PIs know this as well, and you can find yourself paying for their pilot projects out of your start-up. As with all things, you need to carefully weigh whether this pilot study will translate into a grant or if it is a one-off experiment that will collect dust. You also do not want to work in a silo. It can be difficult to get that first senior author paper or grant, and by working with more seasoned investigators, you can build your publication record and grant support. My suggestion would be to find one or two PIs with whom you would be happy to collaborate and build those relationships. I have recently become very protective of my time and accepted that just because someone wants to work on my Terrible Disease of Interest, does not mean they have to work on it with me. I have instead cultivated two collaborations at my institution which have resulted in two grant submissions that were largely driven by my collaborators.

Find mentors: Even though you are a faculty member now, you still need mentors, and particularly mentors at your institution. While your mentors can provide scientific input, their roles are now focused on objectively evaluating your progress towards tenure. Some departments have this mentoring requirement formally established—others do not. Either way, establish a mentoring committee and make your semi-annual meetings a priority. It is much better to hear from your mentoring committee that you are not on track to achieve tenure versus hearing it from the Promotion and Tenure (P&T) committee after formal review. I meet with my committee every six months, and we discuss everything from funding to manuscripts to teaching and service.

Build good relationships with university officials: Get to know the people who make lab life possible. This includes individuals like the departmental administrator who reviews your grants before they are submitted to the Office of Research, individuals in the Finance Office, the Director of Facilities who provides building access and mediates installing big equipment, and the head of Procurement who signs off on big orders. In some departments, you might be able to delegate this to an administrative assistant or administrator. My approach has been to do it myself. It takes half the time and the next time I need something, like building access for an undergraduate student, I can take care of it quickly.

Write things down and get organized: This is the time to develop on-boarding documents for new hires, update lab protocols to reflect current equipment, generate lists of laboratory reagents (including how to order them), and develop lab policies (code of conduct, travel policy, etc.). Identify a way to have staff and trainees access and update documents through Google Drive or Box. Developing these resources and documents is quite a bit of work, but it lays the groundwork for the lab to grow and remain organized. One of the things that I did not do was develop written expectations for staff and trainees. This has resulted in some unpleasant back and forth with an undergrad about what my expectations were for them. It would have benefited us both to have a signed document for them to reference. Developing this document is currently on my priority list.

Practice saying no: You do not have to do all the things in your first year. Focus on getting the lab established and everything in place. Service on committees can surely wait until year 2 or 3. You do not have to work with everyone, you do not have to apply for every single grant mechanism available to you, and you surely do not have to agree to give every seminar or attend every conference in your field the first year. My department and institution have been reasonable in their expectations, but I did find myself saying yes to a lot of national conferences and seminars this year. While this is an excellent issue to have, my absence from the lab for such extended periods of time slowed research progress and impeded training. This, in turn, has delayed some of our grant submissions. Whether the national exposure versus manuscript/grant submission progress was a wise choice remains to be seen. On a personal note, all the travel did not contribute to work-life balance.

Celebrate the little things: We would all like to celebrate the big things, like papers in press or big grants awarded. While these are worthy goals for your first year, they can be difficult to achieve. Celebrate the little things, like the first experiment, the first figure for a paper, or the first lab poster. This has helped keep me motivated and lab members excited about progress while we work towards the big ticket items.

With these thoughts in mind, I go into the new year and into year 2. The upcoming year will be a busy one, with R01 submissions, dreams of two papers submitted, some teaching about Terrible Disease, and a couple new conferences to attend. Stay tuned for more tales!

Did I miss an important point? Do you have questions or concerns about the post? Or perhaps an anecdote to contribute! Feel free to send some electrons my way in the comments, via Twitter @PipetteProtag, or through traditional electronic mail pipette.protagonist@gmail.com

More Resources

Tune Your When, How Much, and What in Your Days

Not that Kind of Investment: Tales of Time Commitment

Staying Mentally Well in Academia is a Balancing Act

Not that Kind of Doctor: Tales of a PhD in a Clinical Department

Doing Research December 4, 2017

As long as I have been in science, my research has always been a mix of basic science and translational studies, nestled in a clinical department. This convergence of PhD-initiated basic science questions in a clinical setting has led to some fantastic collaborations and deep introspection about how we define ourselves as scientists and how we find our departmental fit. Views are my own, based on my experiences (n=me).

The perks: Working in a clinical department has many advantages. You are closer to the patients and the clinicians who treat them. This gives you access to expertise, tissues, and clinical data normally outside your reach. These patient interactions can translate into philanthropy for the department, which can fund small pilot projects, conference travel, and even new equipment. Clinical colleagues have herds of motivated medical students looking for small research projects that, if you are clever, you can dovetail into some of your own research. Your clinical colleagues also have a different skill set that you can leverage against a common problem of interest. In addition to the intellectual input your clinical colleagues provide, this can also include assistance with difficult small animal surgeries, pathology scoring, or institutional review board (IRB) preparation. For me, the most significant part of my clinical department appointment is that it keeps the disease research at the forefront of my mind. Even if the project is asking a very basic question, I am always thinking about what it could mean for understanding, detecting, or treating my Terrible Disease of Interest.

The challenges: There are, of course, challenges with working in a clinical department. Here are the challenges I have seen, and how I started to deal with them.

Patients come first: Patients will always come first, and they should. This means meetings get pushed back, happen at odd times, or get outright cancelled. I have accepted that larger meetings and conferences happen very early in the morning or late in the afternoon to accommodate more clinicians. For smaller meetings, for example with my Chair, I work closely with the administrative assistants to schedule meetings on lower volume clinic or administrative days. This does not always work out, but it increases the odds I will actually have my meeting.

Busy clinicians: My clinical colleagues wear many hats. Aside from their considerable clinical duties, they mentor residents and fellows, lead NIH panels, develop clinical guidelines, serve on study section, and some even run their own research programs. I have become more appreciative of the time they commit to research, and learned to accept that their time is structured differently than mine, especially if their protected time is limited. My first step in every collaboration is determining how much time they are able and/or willing to contribute. Is it a quick coffee chat about treatment strategies as a consultant? A plan for providing access to their database as a co-investigator? Or are they going to start attending our lab meetings and we will put in a grant together as collaborators?

Different lives: While there are a number of really talented physician-scientists who live both PhD and MD life, most clinical departments are staffed by MDs concerned with clinical practice and PhDs concerned with lab life. My clinical colleagues are concerned with their patients, their case load, residents, and publications/abstracts. Some do health services research. On the flip side, my PhD colleagues are worried about teaching, tenure, grants, and publications. Neither life is easy, even if it looks like it sometimes. It is best to accept that we have different concerns, not dwell on the details, and support each other as colleagues.

Small number of basic scientists: Clinical departments tend to have a smaller basic science contingent. This sometimes leads to seminars and retreats focused more on clinical agendas, rather than research. My solution for this has been to become a secondary faculty member in a basic science department. I identified a larger department that had a weekly seminar, an annual retreat, and a good amount of graduate students and postdocs. Joining this department has provided a much needed basic science department feel, where a good mechanism, even if it does not translate into the clinic, is appreciated. Moreover, this secondary department provides access to graduate students and a social home for postdocs. It has also exposed me to academic faculty life with committees, faculty meetings, and teaching opportunities. The most important part of my secondary appointment, however, is that I do not have an agenda for the department. I am not competing for space, resources, or tenure lines relative to my research interests. Instead, I enjoy departmental life, with a little bit of department service as a price.

In summary, the things that make being in a clinical department so valuable, can also make being a basic scientist in said clinical department a challenge. But it is worth it. Now, excuse mw while I get ready for this 7am meeting. Stay tuned for more tales!