Seven (more) Deadly Sins? Common reviewer criticisms (Part 2)

The second part of my list of the most common points of criticism from grant reviewers in the life sciences

I recently blogged about common reviewer criticisms and how to avoid them. This involved a trawl through grant reviews and panel-feedback letters, to draw up a list of recurring themes. And the list turned out to be quite long – longer in fact than could reasonably be covered in a single blog post. But given how often these criticisms crop up, it seems reasonable to draw attention to them and highlight strategies for making sure they never appear in feedback on your own proposals.

My last post on the subject offered seven commonly-seen criticisms from reviewers and grants panels. Without further ado and once again in no particular order, here are seven more:

1. “The level of project risk appears unacceptably high” / “Potential risks do not appear to have been adequately considered”

This criticism might be phrased in general terms as above, or it could focus on a specific point. Funders tend to be risk averse, and even those that claim to tolerate a higher-than-normal degree of scientific risk (i.e. high-risk, high-gain research) will balk at unnecessary project risks. We saw an important example of unacceptable project risk in the previous post – over-ambition and the associated risk that it won’t be possible to get everything done with the time and resources available.

Specific project risks can be diverse, but they’re frequently eventualities that, if they arose, could scupper the project. Perhaps it seems unlikely that you’ll be able to recruit enough participants or elicit enough responses. Maybe you’re too reliant on collaborators over whom you have little control, possibly because they’re overseas. Perhaps you’ve front-loaded the project with the riskiest work, which could bring down the whole show in year one if it fails. Maybe inter-dependencies between multiple work packages create prominent points of failure.

Avoiding this criticism

Reviewers will expect you to have considered objectively what might potentially go wrong. You should assess how likely it is to happen, quantify the severity of the impact on the project if it were to happen, and consider what you can do both to reduce the risk of it happening and to mitigate the consequences if it did. It’s common to use a red, amber green (RAG) grading scheme for risk likelihood and severity.

Be thorough and use a bit of imagination. All too often, I see very short paragraphs on risk that seem to suggest the applicant has led a very charmed life where nothing has ever gone wrong. This isn’t about listing all of the force majeure occurrences that could potentially scupper any project – “lab burns down”, “Co-I steps under bus” etc. It involves assessing risks that are specific to the project. Make sure that mitigation measures are specific, practicable and likely to be effective.

The best way to show you’ve done all this is to include a risk table or matrix in your application. If you don’t have the space, summarise key risks and mitigations in narrative form, and indicate that you’ve prepared a full risk table which is available on request.

Funders do realise that it wouldn’t be research if we knew exactly what will happen. They’re not looking for risk-free projects, but they do want reassurance of the likelihood that whatever happens, some good scientific progress will be made. It’s all about showing that you’re aware of the risks, have minimized them to reasonable levels, and have a sound Plan B.

There’s more on risk analysis and management in research projects in this article – Don't let your proposal fail due to a poor risk analysis.

2. “There is little or no evidence of training and career-development activities for research staff”

You may have noticed that funders are placing more emphasis on capacity building and skills development among the research workforce. Even if a grant is not specifically training focused (like a fellowship is), it represents a major training and career-development opportunity for all involved. Research staff in particular have an opportunity to acquire valuable new skills and experience upon which to build their future careers. The funder will expect you to maximise those opportunities, and if you don’t seem to be doing that then your proposal is likely to attract criticism.

Avoiding this criticism

Training and skills development activities can be formal or informal. When it comes to formal training, be aware of what opportunities exist for research staff within your institution, and indicate how you will support your researchers in identifying and accessing them. If a researcher will need to acquire specialist skills to undertake part of the work then make clear how this will happen, and how you and the project will support this process.

We all learn through doing, and working as a research assistant or postdoc offers valuable opportunities for experiential learning. Once again, describe how you’ll support the learning process, through things like mentoring and opportunities to take on meaningful tasks that will challenge and extend skillsets. These might include opportunities to attend and participate in conferences; write papers and grants; undertake public-engagement activities; learn specific techniques; participate in project management; and build network links. Look to provide constructive feedback, give credit when it’s deserved, and celebrate achievements collectively. Don’t underestimate the extent to which an early-career researcher or other team member could benefit from your particular knowledge, experience and insights, and actively look for ways to share your expertise throughout the lifetime of the project.

Your description of staff-training and development activities needn’t be lengthy, but it should be clear and specific. If you have to complete a justification of resources section then this can often be a good place to cover some of the specifics of staff-development activities and associated costs.

3. “Value for money is not obviously high”

The first thing to note is that the evaluator should not be complaining that the project is costly – i.e. a lot of money – per se. If the total sum requested is within the funder’s maximum limit then a meaty budget shouldn’t on its own make it uncompetitive. Reviewers are however frequently asked to comment on a proposed project’s value for money, and to do so they need to consider the cost of the project in the light of the possible benefits (including new knowledge) that would come from its completion.

In a world where financial resources aren’t infinite, if reviewers and/or grants panels feel that a research proposal doesn’t offer good value for money then it’s unlikely to be successful, regardless of its merits.

Avoiding this criticism

Justify everything in your budget – people’s time, equipment and consumables, travel and so on – explaining why you need it and how you arrived at the cost. I’ve previously written a blog post about justifying resources. Try to make sure there isn’t an obvious way of achieving the same or similar results while spending considerably less on resources.

Having satisfied the funder that they’re not over-paying for the project and its planned outcomes, you’ll need to convince them that those outcomes are worth the cost. Ideally, you’ll try to quantify the anticipated impact in financial terms. For example: “The NHS currently spends £55m annually treating 25,000 patients for this painful condition. Our proposed new care pathway would reduce the per-patient cost to less than £2,000, leading directly to savings of at least £5m per year.” If the project will cost £1.2m then this would seem to offer very good value for money.

For various reasons, it’s not always easy to quantify impact in financial terms. Many projects won’t on their own lead directly to clinical (or other health/wellbeing) impacts, and often those impacts don’t have an easily-calculated financial value. This shouldn’t stop us from seeking to spell out, in a realistic and proportionate manner, the nature of the intended impact in such a way that it can be viewed objectively alongside project costs. In general terms, medical, health or care research that isn’t obviously excessively priced will be deemed worthwhile if it will yield a modest benefit for a large number of people; or a substantial benefit for a smaller number.

4. “Claims about the impact of the research are insufficiently backed up”

“Impact claims are unsubstantiated.” Hmmm. When we’re really excited about a research project it can be hard to resist letting our enthusiasm run away with us. And it can be easy to slip into persuasive hyperbole when it comes to describing its importance and potential.

Who hasn’t at some time described the research need as “urgent”? Referred to the need for the research as “imperative”? Or claimed that the impact of their project will be “transformational”? Do such descriptions convince the evaluators, or do they just irritate them?

Imagine you’re browsing the shelves in a cookware shop. A sales-person approaches: “Sir, this air fryer will change your life! It’s a truly amazing product, and you’d be crazy not to buy it!”

This unevidenced hyperbole won’t convince me and will probably just annoy me. I’m quite capable of making my own mind up, I’m highly sceptical about ‘life-changing’ claims for products, and like everyone else I’m firmly convinced that it’s the world that’s crazy, not me. A different approach might have been more productive: “I think air fryers’ popularity is well deserved – they use up to 80% less oil than deep-fat frying, they can reduce harmful acrylamide production by as much as 90%, and they only use around half the energy of a standard electric oven. You can check those figures and find out more about their benefits on the AllFactsAboutAirFryers.com website if you’re interested.”

I think that, like me, reviewers and grants-panel members prefer to make up their own minds, based on the facts and evidence, rather than being told what they should think. Your idea of what constitutes ‘transformational’ may after all be very different to theirs.

Avoiding this criticism

The solution here is really simple – avoid hyperbole. Think twice before peppering your proposal with adjectives, and instead let the facts and figures do the talking. Have faith in the importance and relevance of your research. If you do make a bold claim that the research is transformative, urgent or whatever, then be sure to back that claim up very convincingly with objective evidence.

I previously wrote a blog post on this subject – the inadvisability of using adjectives and hype in research proposals.

5. “Impact plans are insufficiently detailed”

Impact plans (proposed impact activities) are the things that you and your project team will do to maximise a project’s potential for impact. They’re usually vital for ensuring that your project results – perhaps just some numbers in a spreadsheet – can escape from the confines of your laptop and be translated into changes that will ultimately benefit people ‘in the real world’.

If reviewers aren’t convinced that your claims for impact are likely to come about, because they just don’t understand how your results will ever have an influence on policy, practice, procedure, products, processes or whatever, then they’re unlikely to recommend funding. Remember – impact is usually the answer to the question “what’s the point of doing this research?”

Avoiding this criticism

Whole books have been written on impact, so what follows will just be a very brief overview. Impact commonly involves some sort of change, for example a change of government policy or the adoption of a new way of doing something. If your project results and outputs are going to trigger and inform such change, there are usually two key questions that you must answer:

1. Who are the key decision-makers who need to know about your results?
2. What is the best way to reach those individuals and groups with your results?

It’s often useful to do a stakeholder-analysis exercise to determine who the main interested parties are and formulate a communications and engagement plan. In many cases, it’s highly advantageous to bring some key stakeholder representatives on board from the outset, so that they’re close to and involved with the research. If for example influencing the way social workers operate will be key to impactful change, then it’s probably important to include some social workers and others involved with social-services policy on a project advisory board or similar.

As in all other aspects of bid writing, specificity is paramount. “We will engage with key stakeholders through a series of workshops to ensure impact” carries no real weight at all. “We will communicate our results and best-practice recommendations directly to care home managers through an article in Care Home Management (which has agreed in-principle to publish our findings), and provide a link to an interactive online decision tool that we will publish on both the University’s and funder’s websites” does a much better job.

If your impact plans depend on producing and distributing some sort of ‘toolkit’ then be specific about what this would look like (my default understanding of a toolkit is a bag full of hammers and spanners – will it look like that??) If the plan is to produce a ‘policy report’ or similar then be realistic about the chances of the right people reading and acting on it. I have heard it suggested that the shelves of government are lined with hundreds of unopened, unsolicited policy reports sent in by academic researchers, all of which are slowly gathering dust.

UCL has published a useful Guide to Creating Impact from Research.

6. “The proposal is not truly interdisciplinary”

This criticism can sometimes be a version of Point 4 above – you’ve made a claim, in this case about interdisciplinarity, but it isn’t supported by the facts. But funders are increasingly publishing grant calls that specifically require interdisciplinary approaches, so they’re on the lookout for applications that that don’t actually hit this key criterion.

Avoiding this criticism

Multi-disciplinarity is not interdisciplinarity, and a claim to the contrary will likely be met with the above criticism.

Imagine a bike-racing event that involves a 50-mile lap on the road, followed by a 30-mile lap on off-road trails. Competitors start the event on a skinny-tyred road bike, and then switch to a mountain bike for the off-road section. This is a multi-disciplinary event, involving two distinct cycling disciplines.

Now imagine an event comprising a single 80-mile circuit that intermittently takes in both lengthy tarmac sections and loose trails. Neither a mountain bike nor a road bike would be very suitable – the road bike would be a nightmare on the trails, while the mountain bike would be slow and hard work to ride on the road. Instead, a new type of bike is called for – a gravel bike. Gravel bikes share characteristics with both mountain bikes and road bikes, and are excellent for riding long distances over mixed terrain. A gravel-bike race borrows from and integrates two existing cycling disciplines and is truly interdisciplinary. Indeed, the popularity of gravel biking has seen it emerge as a new discipline in its own right.

As in cycling, so in research. A proposal that would see physicists working alongside biomedical researchers, with each completing a separate part of the project, is multi-disciplinary; but one that would involve researchers working at the interface between medical science and physics, integrating approaches from both disciplines and crossing disciplinary boundaries to prevent, diagnose or treat disease, is likely to be demonstrating an interdisciplinary approach. Biomedical physics has indeed emerged as a discipline, and involves the application of physics to medicine for diverse applications such as the study of biomolecular structure in disease states.

(Transdisciplinary research, by the way, is not synonymous with interdisciplinary research. Rather than sitting between two or more traditional disciplines it transcends them. It is holistic and sits apart from the disciplines from which it emerged, being much more than just the sum of their respective parts. The recently-emerged field of climate-change research is, for example, substantially transdisciplinary.)

To avoid receiving this particular criticism, it’s essential to understand what constitutes multi-disciplinary and interdisciplinary (and indeed transdisciplinary) research sufficiently clearly to judge which category your project falls into. If you’re still feel unsure about how the categories differ then this nice diagram may be helpful.

7. “The application is poorly written” / “The proposal shows a lack of attention to detail”

I don’t suppose that most reviewers consider reading proposals to be an unbridled joy. People volunteer for peer-review colleges and grants panels for a number of different reasons, but I suspect that pure pleasure isn’t usually one of them. So we should see it as our duty to take every possible step to prevent the whole experience being more unpleasurable than necessary.

Spelling mistakes, typographical errors, poor grammar, inconsistencies, strange punctuation and downright sloppiness will all impede the reader’s attempt to understand and appraise your proposal. Even though reviewers are never asked to evaluate the typographical quality of proposals, they’re very often unable to resist showing their frustration in their reviews. “There are spelling mistakes and other typographical errors, suggesting a lack of care on the part of the applicant”. The implication here of course is that someone who can’t devote due care and attention to their proposal might not be inclined to lavish much effort on delivering the project.

Avoiding this criticism

A little bit of me dies each time I see this criticism in a review. Regardless of how excellent, novel and relevant the research idea, the applicant has succeeded in irritating the reviewer to the extent that they are – in part at least – negatively disposed towards it. And it’s so easily avoided!

Leave enough time at the end of the writing process to read through everything properly and in its entirety, at least once. This should ideally be done with fresh eyes. Ask others to look through and proof-read drafts too. Be sure that all of the proof-reading tools in your word-processing software are switched on and properly configured, and use the spell-checker (and grammar-checker) tool liberally. Pay attention to consistency – if you used the UK English version of a word in one place, don’t then switch to the US version elsewhere; if you gave a word a capital letter in once place then do so throughout. Spell out acronyms and abbreviations in full on first use. Try to keep punctuation simple and unobtrusive. Aim to submit a ‘pretty’ document with good and consistent use of white space, consistent font type and size, helpful subheadings, and so on. While you should view the idea of letting ChatGPT and other generative AI loose on your research proposals with caution, AI tools are increasingly good at proof-reading and tidying up sloppy text if they’re properly supervised. Do check their output carefully though – don’t let ChatGPT be the last thing to edit your proposal.

The best of the rest (the rest of the worst?)

There are other criticisms that crop up quite regularly but didn’t make the cut here. ‘Plans for participant recruitment seem unrealistic’; ‘expertise required to complete the project appears to be missing’; ‘the applicant appears to have ignored [something of relevance] in the literature’; ‘there is a lack of clarity or missing information in describing [whatever]’. Happily, they’re all readily avoidable through legwork, attention to detail and thorough project planning.

The views and opinions expressed in this blog are mine alone and are in no way endorsed by my employer. Factual information and guidance are provided on a 'best-endeavour' basis and may become out of date over time. Web-links were correct at time of writing but commonly go out of date. No responsibility can be taken for any action or inaction taken or not in respect of the content of this blog. If you think you recognise your own panel rejection letter in the above list, please be assured that it is a composite of the main recurring themes taken from a large number of different proposal reviews