Education & Academic Research

Resolved: Academic journals should prioritize methodological soundness and reproducibility over novelty and perceived impact during peer review.

Core clash: Scientific Integrity vs. Innovation

Global (academic publishing) 2015 – present Sources last checked: 2026-10-31

Sources: 3 of 5 checked against the full text

2 abstract only

PF-style rendering of a BP motionOriginal: BIDO 2026, Round 4motion list

Original BP wording: “This House believes that academic journals should prioritize soundness over significance in peer review.”

The “Resolved:” wording is this site’s rendering; it was not the tournament’s text.

01Event Background

The real-world development that makes this motion relevant, and its current status.

Key research questions

Why do published findings sometimes fail to replicate?
How do publication incentives influence research quality?
Could prioritizing reproducibility discourage novel research?

Event or policy: The replication crisis and journals that review for soundness (Registered Reports, PLOS ONE)
Date or period: 2015 – present
Policy status: Varies by jurisdiction

In 2015 the Open Science Collaboration repeated 100 experiments published in 2008 in three leading psychology journals; depending on the criterion used, roughly one-third to one-half of the original findings were reproduced. The result focused attention on journal incentives: if editors favour novel, surprising, positive results, researchers have reasons to run small studies, analyse data flexibly and leave null results unpublished. Two kinds of reform now exist alongside traditional peer review. Registered Reports are reviewed and accepted on the strength of their question and methods before results are known; a 2021 comparison found that 44% of Registered Reports reported a positive result for their first hypothesis, compared with 96% of standard articles. Some journals, such as PLOS ONE, judge submissions only on technical soundness and publish valid negative results, without assessing perceived importance. The counter-concern is innovation: a study of 773,311 papers found that highly novel work is cited less at first and less likely to appear in high-impact journals, yet is much more likely to become a top-1% paper in the long run. Prioritising soundness could either correct a bias toward flashy results or deepen an existing bias against risky, original work.

2015-08
Reproducibility Project: Psychology
About a third to a half of 100 findings reproduced.
Estimating the reproducibility of psychological scienceOpen Science Collaboration (Science 349(6251), aac4716), 2015-08-28(abstract only)
2016
“Bias against novelty in science” (working paper)
Novel papers are cited late but more likely to become top-1% papers.
Bias against novelty in science: A cautionary tale for users of bibliometric indicatorsJian Wang, Reinhilde Veugelers, Paula Stephan (Research Policy 46(8), 1416–1436; NBER Working Paper 22180), 2017-10
2021-04
Registered Reports vs standard literature
44% vs 96% positive results.
An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered ReportsAnne M. Scheel, Mitchell R. M. J. Schijen, Daniël Lakens (Advances in Methods and Practices in Psychological Science 4(2)), 2021-04-16(abstract only)

02Motion Definitions

Resolved: Academic journals should prioritize methodological soundness and reproducibility over novelty and perceived impact during peer review.

Methodological soundness: Whether the design, analysis and reporting can support the conclusions: adequate sample size, appropriate controls, pre-specified analyses, transparent data.
Reproducibility: Whether the findings can be obtained again: by re-running the analysis on the same data (computational reproducibility) or by repeating the study with new data (replication).
Novelty and perceived impact: Whether a result is new, surprising or likely to be influential, as judged by editors and reviewers before publication.
Prioritize … during peer review: Make soundness the decisive acceptance criterion, with novelty or impact considered secondarily or not at all; for example, Registered Reports or soundness-only review.; The motion concerns editorial criteria, not banning novel research.

03Stakeholders

Researchers, especially early-career
Vulnerability: moderate
Journal criteria shape what research is rewarded and therefore what is done.
- Publications for careers
- Credible findings to build on
- Freedom to pursue risky ideas
Scientific Integrity Innovation
Journals and editors
Vulnerability: lower
Criteria determine their identity and impact; soundness-first journals may receive fewer citations.
- Reputation
- Citations
- Credibility
Accountability Transparency
Funders and policymakers
Vulnerability: lower
Base decisions on published research; unreliable findings waste money and mislead policy.
- Reliable evidence
- Breakthroughs
Efficiency Collective Welfare
The public and patients
Vulnerability: moderate
Bear the costs of false findings (wasted treatments, policies) and the benefits of discoveries.
- Trustworthy science
- Progress on problems
Collective Welfare Scientific Integrity

04Proposition Arguments

PropositionArgument 1
Rewarding novelty fills journals with results that do not hold up
Claim
When acceptance depends on striking results, the published record overstates positive findings, and many do not replicate.
Warrant
If journals publish what is surprising and positive, the literature is a filtered sample of results that is biased toward chance successes.
Mechanism
Researchers know positive, novel results are publishable. Small studies, flexible analysis and selective reporting raise the chance of a publishable result; null results go unpublished. The published literature therefore contains too many positives, and replications often fail.
Impact
Wasted research effort building on false findings, misleading evidence for policy and medicine, and eroding trust in science.
Value
Scientific integrity: the point of the scientific record is to be a reliable account of what is known.
Possible weaknesses
Replication failure has several causes (sample differences, low power in replications), not only editorial selection.
Evidence
Repeating 100 experiments from three leading psychology journals reproduced roughly one-third to one-half of the original findings, depending on the criterion. Verified fact
96% of standard psychology articles reported a positive result for their first hypothesis, compared with 44% of Registered Reports reviewed before results were known. Verified fact
Assumptions
Editorial selection for novelty is a major cause of false positive findings.
Counterevidence
The reproducibility estimate varies from one-third to one-half depending on how success is defined, showing that “failure to replicate” is partly a matter of criteria, not a simple count of false findings.
Scientific Integrity Accountability
PropositionArgument 2
Judging methods before results removes the incentive to manufacture positives
Claim
When journals accept papers on the quality of their question and design, researchers have no reason to chase significance.
Warrant
Removing a reward for an outcome removes the incentive to produce it by questionable means.
Mechanism
In Registered Reports, peer review happens before data collection and the journal commits to publish whatever the results. Null results become publishable, so the literature reflects what studies actually found.
Impact
A literature with a realistic rate of positive results and fewer false findings.
Value
Transparency and integrity: decisions about what counts as knowledge should not depend on whether the answer was exciting.
Possible weaknesses
Registered Reports may attract different types of studies, so part of the 96% vs 44% gap is not publication bias.
Evidence
96% of standard psychology articles reported a positive result for their first hypothesis, compared with 44% of Registered Reports reviewed before results were known. Verified fact
Assumptions
Researchers respond to publication criteria by changing practices.
Counterevidence
The authors of the Registered Reports comparison note the gap may be only partly due to reduced publication bias; differences in what is studied may also contribute.
Scientific Integrity Transparency
PropositionArgument 3
Editors are poor judges of impact anyway
Claim
Pre-publication judgments of significance are unreliable and themselves biased against the most original work, so dropping them costs little.
Warrant
A criterion that cannot be applied accurately should not decide publication.
Mechanism
Highly novel papers are cited late and are less likely to appear in high-impact journals, even though they are more likely to become top-1% papers in the long run. Impact screening at the point of review therefore misjudges exactly the work it claims to favour.
Impact
Soundness-only review, as at PLOS ONE, lets readers and time judge importance.
Value
Fairness and integrity: researchers should be judged on the quality of their work, not editors' guesses about influence.
Possible weaknesses
The same evidence can be read as showing that novel work needs protection, not that impact judgments should be dropped.
Evidence
Highly novel papers (11% of 773,311 articles) were cited less in early years and less likely to appear in high-impact journals, but much more likely to become top-1% cited papers in the long run. Verified fact
PLOS ONE's criteria judge technical and ethical soundness, publish valid negative and null results, and do not assess perceived importance. Verified fact
Assumptions
Readers, citations and time judge importance better than editors at the point of review.
Counterevidence
Novel papers are under-recognised by high-impact journals, so current impact-based selection does not reliably identify the most important work.
Scientific Integrity Academic Fairness

05Opposition Arguments

OppositionArgument 1
A soundness-first rule further disadvantages risky, original work
Claim
Novel research is already disadvantaged; making methodological conservatism the priority would push researchers toward safe, incremental questions.
Warrant
Original work often uses new methods, small exploratory samples or unconventional designs that look weaker by established standards.
Mechanism
Reviewers apply familiar standards most easily to familiar designs. If soundness is decisive, exploratory and boundary-crossing work is rejected more often, and researchers adapt by avoiding it.
Impact
Fewer breakthroughs, slower progress on new problems.
Value
Innovation: science advances through new ideas, which are uncertain when first proposed.
Possible weaknesses
Soundness is not the same as conventionality; a well-designed exploratory study can be sound.
Evidence
Highly novel papers (11% of 773,311 articles) were cited less in early years and less likely to appear in high-impact journals, but much more likely to become top-1% cited papers in the long run. Verified fact
Assumptions
Reviewers judge unconventional designs as less sound than conventional ones.
Counterevidence
The novelty-bias study finds novel work is already disadvantaged by impact-focused indicators, so the existing impact regime, not soundness review, is what disadvantages originality.
Innovation
OppositionArgument 2
Replication failure is not mainly an editorial problem
Claim
Many replication failures reflect differences between original and replication studies, small samples and genuine context-dependence, which changing journal priorities will not fix.
Warrant
A policy should target the cause of the problem.
Mechanism
Effects can depend on populations, settings and procedures; replications vary in power. Funders, training and data-sharing rules address these causes more directly than editorial criteria.
Impact
The motion's reform would carry costs to innovation without fixing much of the problem.
Value
Efficiency in reform: scarce reform effort should go where it works.
Possible weaknesses
The excess of positive results in standard papers points directly to publication decisions.
Evidence
Repeating 100 experiments from three leading psychology journals reproduced roughly one-third to one-half of the original findings, depending on the criterion. Verified fact
Assumptions
Most replication failures stem from causes outside journals' control.
Counterevidence
The reproducibility estimate varies from one-third to one-half depending on how success is defined, showing that “failure to replicate” is partly a matter of criteria, not a simple count of false findings.
Efficiency Scientific Integrity
OppositionArgument 3
Readers need curation as well as validity
Claim
With millions of papers a year, journals that select for importance help scientists and the public find what matters; soundness-only publishing shifts that burden onto readers.
Warrant
Selection is a service; without it, attention goes to whatever is most promoted, not most important.
Mechanism
Soundness-only venues publish all valid work, so readers must filter more; impact-selective journals concentrate attention on findings editors judge important.
Impact
Worse allocation of attention, and important findings lost in volume.
Value
Efficiency and collective welfare: scientific communication must help people use knowledge.
Possible weaknesses
Curation can happen after publication (reviews, recommendations), without making it a condition of publication.
Evidence
PLOS ONE's criteria judge technical and ethical soundness, publish valid negative and null results, and do not assess perceived importance. Verified fact
Assumptions
Pre-publication selection for importance is necessary for readers to find important work.
Counterevidence
Novel papers are under-recognised by high-impact journals, so current impact-based selection does not reliably identify the most important work.
Efficiency Collective Welfare

06Interactive Argument Map

Switch between cases, expand mechanisms to see evidence and assumptions, and select a node to follow what it supports or challenges.

React Flow

Inspect a node

Select any box to read its explanation, see its sources and follow the arguments it supports or attacks. Drag to pan; pinch, Ctrl + scroll or the buttons in the corner zoom.

Node types

Claim— What a side asserts.
Mechanism— How the claimed effect happens.
Evidence— Material that supports a step; linked to sources.
Assumption— What must be true for the reasoning to work.
Counterevidence— Material that challenges a step. Dashed outline.
Rebuttal— A response to an attack.
Impact— The consequence that follows.
Value— Why the consequence matters.
Weighing— How competing impacts are compared.

Relationships

supports— Makes the target more likely or more important.
challenges— Weakens or attacks the target.
depends on— Only works if the target is true.
qualifies— Limits the scope or strength of the target.
leads to— Causes or produces the target.
justifies— Explains why the target matters morally.
conflicts with— The two cannot both be fully honoured.

Lines labelled “(across sides)” connect a Proposition element to an Opposition element: these are the points where the two cases directly clash.

Open the map full width →

07Assumptions

What must be true for each argument to work. Empirical assumptions can be tested; normative ones must be argued for.

empirical assumptions

Proposition
Editorial selection for novelty is a major cause of false positive findings.
If other causes dominate, changing editorial priorities achieves less.
How to test: Compare replication rates of findings from journals with different selection criteria.
Proposition
Researchers respond to publication criteria by changing practices.
The incentive argument depends on behaviour responding to rewards.
Proposition
Readers, citations and time judge importance better than editors at the point of review.
If editors add real value in selection, dropping it is costly.
Opposition
Reviewers judge unconventional designs as less sound than conventional ones.
If soundness is judged fairly across designs, prioritising it need not penalise originality.
How to test: Experiments giving reviewers matched novel and conventional manuscripts.
Opposition
Most replication failures stem from causes outside journals' control.
Determines whether editorial reform targets the problem.
Opposition
Pre-publication selection for importance is necessary for readers to find important work.
Post-publication curation could substitute.

normative assumptions

Proposition
The reliability of the scientific record matters more than the speed of discovery.
This is the value judgment beneath the empirical dispute.

08Evidence

Raised by the Proposition

PropositionVerified factQuantitative
Repeating 100 experiments from three leading psychology journals reproduced roughly one-third to one-half of the original findings, depending on the criterion.
Supports: Rewarding novelty fills journals with results that do not hold up; Replication failure is not mainly an editorial problem
Limitations: One field; failure can reflect replication differences.
Estimating the reproducibility of psychological scienceOpen Science Collaboration (Science 349(6251), aac4716), 2015-08-28(abstract only)
PropositionVerified factQuantitative
96% of standard psychology articles reported a positive result for their first hypothesis, compared with 44% of Registered Reports reviewed before results were known.
Supports: Rewarding novelty fills journals with results that do not hold up; Judging methods before results removes the incentive to manufacture positives
Limitations: Different study types may self-select into Registered Reports.
An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered ReportsAnne M. Scheel, Mitchell R. M. J. Schijen, Daniël Lakens (Advances in Methods and Practices in Psychological Science 4(2)), 2021-04-16(abstract only)

Raised by the Opposition

OppositionVerified factQuantitative
Highly novel papers (11% of 773,311 articles) were cited less in early years and less likely to appear in high-impact journals, but much more likely to become top-1% cited papers in the long run.
Supports: A soundness-first rule further disadvantages risky, original work; Editors are poor judges of impact anyway
Limitations: Citations as proxy for value; checked against summary.
Bias against novelty in science: A cautionary tale for users of bibliometric indicatorsJian Wang, Reinhilde Veugelers, Paula Stephan (Research Policy 46(8), 1416–1436; NBER Working Paper 22180), 2017-10

Bearing on both sides

NeutralVerified fact
PLOS ONE's criteria judge technical and ethical soundness, publish valid negative and null results, and do not assess perceived importance.
Supports: Editors are poor judges of impact anyway; Readers need curation as well as validity
Limitations: Stated criteria, not measured outcomes.
PLOS ONE: Criteria for PublicationPLOS

09Counterevidence

Evidence that challenges an argument's empirical claims, and what it does and does not show.

Raised by the Proposition

PropositionCompeting interpretation
Novel papers are under-recognised by high-impact journals, so current impact-based selection does not reliably identify the most important work.
Challenges: Readers need curation as well as validity
Limitations: Bibliometric evidence only.
Bias against novelty in science: A cautionary tale for users of bibliometric indicatorsJian Wang, Reinhilde Veugelers, Paula Stephan (Research Policy 46(8), 1416–1436; NBER Working Paper 22180), 2017-10
PropositionCompeting interpretation
The novelty-bias study finds novel work is already disadvantaged by impact-focused indicators, so the existing impact regime, not soundness review, is what disadvantages originality.
Challenges: A soundness-first rule further disadvantages risky, original work
Limitations: The study did not test soundness-based review.
Bias against novelty in science: A cautionary tale for users of bibliometric indicatorsJian Wang, Reinhilde Veugelers, Paula Stephan (Research Policy 46(8), 1416–1436; NBER Working Paper 22180), 2017-10

Raised by the Opposition

OppositionVerified fact
The authors of the Registered Reports comparison note the gap may be only partly due to reduced publication bias; differences in what is studied may also contribute.
Challenges: Judging methods before results removes the incentive to manufacture positives
Limitations: From abstract wording (“at least partly”).
An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered ReportsAnne M. Scheel, Mitchell R. M. J. Schijen, Daniël Lakens (Advances in Methods and Practices in Psychological Science 4(2)), 2021-04-16(abstract only)

Bearing on both sides

NeutralCompeting interpretationQuantitative
The reproducibility estimate varies from one-third to one-half depending on how success is defined, showing that “failure to replicate” is partly a matter of criteria, not a simple count of false findings.
Challenges: Rewarding novelty fills journals with results that do not hold up; Replication failure is not mainly an editorial problem
Limitations: Interpretive.
Estimating the reproducibility of psychological scienceOpen Science Collaboration (Science 349(6251), aac4716), 2015-08-28(abstract only)

10Rebuttals and Main Clashes

Oppositionresponds toRewarding novelty fills journals with results that do not hold up
- Targets: Editorial selection for novelty is a major cause of false positive findings.
Low replication rates also reflect hidden moderators and underpowered replications; journal selection is only one cause.
Answer back: The excess of positive results in standard papers compared with Registered Reports points squarely at publication decisions.
Propositionresponds toA soundness-first rule further disadvantages risky, original work
- Targets: Reviewers judge unconventional designs as less sound than conventional ones.
Soundness is not conventionality: exploratory studies can be clearly labelled and rigorously reported. The evidence shows novelty is penalised by impact metrics, not by soundness review.
Answer back: In practice, reviewers' sense of rigour is shaped by familiar designs, so novel methods face more scepticism.
Propositionresponds toReaders need curation as well as validity
- Targets: Pre-publication selection for importance is necessary for readers to find important work.
Selection for importance can move after publication (reviews, commentaries, citations) without making it a gatekeeping criterion.
Answer back: Post-publication curation is patchy and favours those with platforms to promote their work.
Oppositionresponds toJudging methods before results removes the incentive to manufacture positives
- Targets: 96% of standard psychology articles reported a positive result for their first hypothesis, compared with 44% of Registered Reports reviewed before results were known.
The Registered Reports comparison mixes publication bias with differences in study type; its authors say the gap is at least partly due to bias, not wholly.
Answer back: Even a partial reduction in publication bias is a large improvement in a literature that reports 96% positive results.

Main argument clashes

Causality
How much of the replication problem is caused by journals' preference for novelty?
Proposition
A great deal: selection produces an excess of positive results.
Opposition
Part of it; many failures have other causes.
What would settle it
Replication rates for findings published under different review models.
Forecast
Would soundness-first review reduce innovative research?
Proposition
No: novelty is penalised by impact metrics, not by rigour.
Opposition
Yes: unconventional work struggles to meet established standards.
What would settle it
Track novelty of papers accepted under Registered Reports and soundness-only review.
Values
Is reliability or discovery the more important goal of a journal?
Proposition
Reliability: a journal certifies knowledge.
Opposition
Discovery: journals exist to advance knowledge.
What would settle it
A normative account of the role of journals.
Feasibility
Can journals assess soundness and reproducibility reliably in review?
Proposition
Pre-registration, data sharing and reporting standards make it feasible.
Opposition
Reproducibility can only be shown after publication, by replication.
What would settle it
Evidence on reviewer agreement about methods quality.

11Values & Clashes

From each side's position down to the values that make its impacts matter, then the clash between those values.

Proposition

Scientific Integrity Transparency Accountability

Opposition

Innovation Efficiency Collective Welfare

1
Layer 1 — Position
What specific policy or action does each side support?
Proposition
Peer review should make methodological soundness and reproducibility the decisive criteria, with novelty and impact secondary.
Opposition
Novelty and impact should remain central criteria alongside soundness; demoting them harms discovery.
2
Layer 2 — Mechanism
How does each side believe its position will create change?
Proposition
Removing the reward for striking results removes the incentive for questionable practices, makes null results publishable and stops editors misjudging impact.
Opposition
Novel work looks less rigorous by established standards; selection for importance helps readers; replication failures have other causes.

Layer 3 — Impact

Who experiences the consequences? Immediate or long-term? Reversible? Evenly distributed?

Impact	Side	Timeframe	Reversibility	Distribution
A more reliable scientific record with fewer false positives.	proposition	long-term	reversible	Benefits everyone who relies on research, especially patients and policymakers.
Fewer risky, original projects and slower breakthroughs.	opposition	long-term	partially reversible	Costs fall on future beneficiaries of discoveries and on researchers with unconventional ideas.

4
Layer 4 — Underlying value
What makes the outcome morally or politically important?
Proposition
Scientific integrity (a trustworthy record), transparency in how decisions are made, and accountability for claims.
Opposition
Innovation (the growth of knowledge), efficiency in directing attention, and collective welfare from discoveries.
5
Layer 5 — Value weighing
How should the competing values be compared, and on what grounds?
Both sides want reliable discoveries; the disagreement is about which error is worse: publishing false findings (type I) or failing to publish important but uncertain ones (type II). The Proposition's harm is measured; the Opposition's is forecast. But the Opposition can point to evidence that originality is already undervalued. The strongest Proposition position shows that soundness-first review does not penalise originality, only unreliable claims.

Deep value clash analysis

Scientific IntegrityversusInnovationmixed disagreement
A reliable record of what is known versus the pursuit of new and uncertain ideas.
Proposition
Every later study, policy and treatment builds on the published record; false findings compound, while true novel findings will still be published if sound.
Opposition
Progress depends on bold, uncertain ideas; a system optimised against error will also filter out discoveries.
Are the values genuinely in conflict?
Less than it appears. Soundness and novelty conflict only if novel work is less sound or is judged so; well-designed exploratory work can satisfy both.
What creates the conflict
Reviewers' familiarity with established methods; small samples in early-stage research; the cost of rigorous designs.
Could both be partly protected?
Labelled exploratory tracks, Registered Reports for confirmatory work, and soundness criteria that accept new methods with transparent reporting.
Policies that reduce the trade-off
- Separate exploratory and confirmatory article types
- Funding replication studies
- Data and code sharing requirements
- Post-publication review
Assumptions that decide the priority
- Whether soundness review penalises novel designs
- How costly false positives are relative to missed discoveries
Evidence that could change the judgment
- Novelty and long-run influence of papers accepted under soundness-first review
- Replication rates by review model
Empirical questions (evidence could settle)
- Do soundness-first journals publish less novel work?
- Do their findings replicate more often?
Normative questions (argument must settle)
- Which error should science be more willing to accept?
What would remain even if both sides accepted the same facts
Even with full information, people weigh false confidence and missed discovery differently.
How different ethical frameworks might judge it
Consequentialist
Compare expected costs of false findings with expected losses from missed discoveries across fields.
Virtue ethics
Intellectual honesty and humility favour soundness; intellectual courage favours novelty; good science needs both.
Contractualist
Rules for publication should be acceptable to both researchers and the public who rely on findings; reliability is harder to reasonably reject.
No framework is presented as correct; each makes different assumptions about what matters morally.

Explore a value

Choose a value to trace the arguments, stakeholders, evidence and assumptions connected to it.

Scientific Integrityis mainly invoked byPropositionGlossary entry →

Arguments that rely on it

PropositionRewarding novelty fills journals with results that do not hold up
PropositionJudging methods before results removes the incentive to manufacture positives
PropositionEditors are poor judges of impact anyway
OppositionReplication failure is not mainly an editorial problem

Stakeholder interests

Researchers, especially early-career: Journal criteria shape what research is rewarded and therefore what is done.
The public and patients: Bear the costs of false findings (wasted treatments, policies) and the benefits of discoveries.

Competing values

(mixed)
A reliable record of what is known versus the pursuit of new and uncertain ideas.

Evidence on the consequences

Repeating 100 experiments from three leading psychology journals reproduced roughly one-third to one-half of the original findings, depending on the criterion.
96% of standard psychology articles reported a positive result for their first hypothesis, compared with 44% of Registered Reports reviewed before results were known.
Highly novel papers (11% of 773,311 articles) were cited less in early years and less likely to appear in high-impact journals, but much more likely to become top-1% cited papers in the long run.
PLOS ONE's criteria judge technical and ethical soundness, publish valid negative and null results, and do not assess perceived importance.
The reproducibility estimate varies from one-third to one-half depending on how success is defined, showing that “failure to replicate” is partly a matter of criteria, not a simple count of false findings.
The authors of the Registered Reports comparison note the gap may be only partly due to reduced publication bias; differences in what is studied may also contribute.
Novel papers are under-recognised by high-impact journals, so current impact-based selection does not reliably identify the most important work.

Assumptions that decide the priority

Whether soundness review penalises novel designs
How costly false positives are relative to missed discoveries
The reliability of the scientific record matters more than the speed of discovery.

How it is weighed

LikelihoodPropositionThe reliability problem is measured; the innovation cost is predicted.
ScalePropositionFalse findings affect everyone who builds on them.
Less restrictive alternativesOppositionOther reforms target replication failures more directly.

12Weighing Mechanisms

Not every debate can be reduced to a numerical score. Here, weighing means comparing the likelihood, severity, distribution and ethical significance of competing outcomes. The goal is to make the reasoning behind different judgments explicit, not to calculate an objectively correct moral answer.

Likelihood
Proposition
The reliability problem is measured; the innovation cost is predicted.
Large replication projects and the positive-result gap are observed; reduced innovation is a forecast.
Long-term implications
Opposition
Breakthroughs have outsized long-run value.
Top-1% papers are disproportionately novel; losing even a few could outweigh many small reliability gains.
Scale
Proposition
False findings affect everyone who builds on them.
Errors propagate through citations, meta-analyses, policies and treatments.
Less restrictive alternatives
Opposition
Other reforms target replication failures more directly.
Funding replications, data sharing and training may improve reliability without changing what journals value.

Questions to test any weighing claim in this debate

Scale: How many people are affected?
Severity: How serious is the harm or benefit for each person?
Likelihood: How probable is the claimed outcome, given the evidence?
Distribution: Who bears the costs and who receives the benefits?
Reversibility: Can the outcome be undone if the policy is wrong?
Vulnerability: Are the affected stakeholders less able to protect themselves?
Rights & principled constraints: Does the policy cross a line that outcomes alone cannot justify?
Less restrictive alternatives: Could the same goal be reached with a smaller intrusion?
Long-term implications: What precedents or path dependencies does the choice create?
Fairness: Is the burden placed on people who can reasonably be asked to bear it?
Uncertain assumptions: Which empirical or normative assumption, if it turned out false, would reverse the comparison?

To see how numerical weights can change a conclusion, and why such models are only illustrative, try the multi-objective optimisation module or the illustrative weighing framework.

13Mathematical Connections

Statistical and causal reasoning errors that matter in this debate, each linked to the Math × Debate module that explains it.

Wide intervals around exciting results
Small studies produce imprecise estimates; a result that barely excludes zero is fragile.
Module 8: Confidence Intervals →
Small samples and extreme estimates
Small studies produce more extreme results by chance, which are the ones most likely to be published.
Module 9: Sample Size and Variability →
Regression on replication
Findings published because they were unusually large tend to shrink when repeated.
Module 10: Regression to the Mean →
How much should one study move you?
If positive results are common even for false hypotheses, a single positive finding is weak evidence.
Module 5: Bayes' Theorem →
Does the conclusion depend on one parameter?
Test which assumptions (effect size, replication power) drive the case.
Module 15: Sensitivity Analysis →

Questions for your own judgment

Should different fields (medicine, psychology, physics) apply this motion differently?
Can reviewers assess reproducibility before a study has been replicated?
Who should pay for replication studies?

14Research Sources

Known research gaps

Two academic sources (the replication project and the Registered Reports comparison) were checked against abstracts only.
No direct evidence was found on whether soundness-first journals publish less novel work.
A 2025 Nature item identified during research could not be accessed (login required) and is not cited.

Academic publicationVerification: Abstract checkedosc-2015-reproducibility
Estimating the reproducibility of psychological science
Open Science Collaboration (Science 349(6251), aac4716) · 2015-08-28
Key finding
Replications of 100 experiments published in 2008 in three high-ranking psychology journals reproduced roughly one-third to one-half of the original findings, depending on the criterion used.
Methodology
Coordinated direct replications, with several criteria for comparing each replication with its original.
Methodological limitations
One discipline and three journals; replications can fail for reasons other than a false original (differences in samples, procedures or power). Checked against the abstract (Crossref).
Last checked: 2026-10-31
Academic publicationVerification: Abstract checkedscheel-2021-registered-reports
An Excess of Positive Results: Comparing the Standard Psychology Literature With Registered Reports
Anne M. Scheel, Mitchell R. M. J. Schijen, Daniël Lakens (Advances in Methods and Practices in Psychological Science 4(2)) · 2021-04-16
Key finding
96% of 152 randomly sampled standard hypothesis-testing psychology articles reported a positive result for their first hypothesis, versus 44% of 71 Registered Reports, in which peer review and the publication decision happen before results are known.
Methodology
Comparison of published Registered Reports (to November 2018) with a random sample of standard articles, coding the first hypothesis of each.
Methodological limitations
Registered Reports may test different kinds of hypotheses (e.g. more replications), so the gap is not entirely due to publication bias, as the authors acknowledge. Checked against the abstract (Crossref).
Last checked: 2026-10-31
Academic publicationVerification: Full text checkedwang-2017-novelty-bias
Bias against novelty in science: A cautionary tale for users of bibliometric indicators
Jian Wang, Reinhilde Veugelers, Paula Stephan (Research Policy 46(8), 1416–1436; NBER Working Paper 22180) · 2017-10
Key finding
Among 773,311 Web of Science research articles from 2001, papers combining prior knowledge in unprecedented ways (11% of the sample) were cited less in their first years and less likely to appear in high-impact-factor journals, but were much more likely to become top-1% cited papers in the long run and to inspire important follow-up work.
Methodology
Bibliometric analysis; novelty measured as first-time combinations of referenced journals.
Methodological limitations
Citations are an imperfect proxy for scientific value; novelty measured from reference-list journal combinations (89% of papers make none, so about 11% are novel). Full NBER working paper checked.
Last checked: 2026-10-31
OtherVerification: Full text checkedplos-one-criteria
PLOS ONE: Criteria for Publication
PLOS · date not recorded
Key finding
PLOS ONE evaluates submissions on technical and ethical soundness: rigorous experiments with appropriate controls, conclusions supported by the data, adequate reporting and data availability. It aims to publish all valid research, including negative and null results, and its criteria do not include perceived importance or novelty.
Methodology
Journal editorial policy page.
Methodological limitations
Describes stated criteria, not how consistently reviewers apply them or the journal's influence.
Last checked: 2026-10-31
Tournament recordVerification: Full text checkedtab-bido-2026
BIDO 2026: motions
BIDO 2026 (Tabbycat) · date not recorded
Key finding
Lists the Round 1 (grade caps), Round 3 (consolidation of the luxury industry), Round 4 (soundness over significance in peer review) and Novice Finals (mandate prenups) motions.
Methodology
Official motion list published on the tournament's Tabbycat (calicotab) site.
Methodological limitations
Confirms the original BP wording and round only. The PF-style wording used on this site is an editorial rendering, not the tournament's text.
Last checked: 2026-10-31

Resolved: Academic journals should prioritize methodological soundness and reproducibility over novelty and perceived impact during peer review.

01Event Background

02Motion Definitions

03Stakeholders

Researchers, especially early-career

Journals and editors

Funders and policymakers

The public and patients

04Proposition Arguments

Rewarding novelty fills journals with results that do not hold up

Judging methods before results removes the incentive to manufacture positives

Editors are poor judges of impact anyway

05Opposition Arguments

A soundness-first rule further disadvantages risky, original work

Replication failure is not mainly an editorial problem

Readers need curation as well as validity

06Interactive Argument Map

07Assumptions

empirical assumptions

normative assumptions

08Evidence

Raised by the Proposition

Raised by the Opposition

Bearing on both sides

09Counterevidence

Raised by the Proposition

Raised by the Opposition

Bearing on both sides

10Rebuttals and Main Clashes

Main argument clashes

11Values & Clashes

Layer 1 — Position

Layer 2 — Mechanism

Layer 3 — Impact

Layer 4 — Underlying value

Layer 5 — Value weighing

Deep value clash analysis

Explore a value

Arguments that rely on it

Stakeholder interests

Competing values

Evidence on the consequences

Assumptions that decide the priority

How it is weighed

12Weighing Mechanisms

13Mathematical Connections

Questions for your own judgment

14Research Sources

Resolved: Academic journals should prioritize methodological soundness and reproducibility over novelty and perceived impact during peer review.

01Event Background

02Motion Definitions

03Stakeholders

Researchers, especially early-career

Journals and editors

Funders and policymakers

The public and patients

04Proposition Arguments

Rewarding novelty fills journals with results that do not hold up

Judging methods before results removes the incentive to manufacture positives

Editors are poor judges of impact anyway

05Opposition Arguments

A soundness-first rule further disadvantages risky, original work

Replication failure is not mainly an editorial problem

Readers need curation as well as validity

06Interactive Argument Map

07Assumptions

empirical assumptions

normative assumptions

08Evidence

Raised by the Proposition

Raised by the Opposition

Bearing on both sides

09Counterevidence

Raised by the Proposition

Raised by the Opposition

Bearing on both sides

10Rebuttals and Main Clashes

Main argument clashes

11Values & Clashes

Layer 1 — Position