Study consisting of 600 UK and US software engineers finds projects adopting Agile Manifesto practices are 268% more likely to fail than those which do the opposite. Research demonstrates how Agile software project failure rates can be cut 6.5x using a new Impact Engineering methodology. Adopting Impact Engineering could save $115bn USD on wasted R&D spending in the USA annually and British taxpayers could save an estimated £7bn GBP annually on failed government digital change projects. EDINBURGH & NEW YORK; 4th June 2024 - The Agile Manifesto has existed for over 21 years now, yet there remains a gap in empirical research as to the actual impact of its values on the industry despite recent research finding 81% of business decision-makers in the UK and 89% in the USA are concerned about on-time delivery of software projects in their organisations.
Today, new research conducted for a new book, Impact Engineering Impact Engineering approach detailed in a new book released today only failed 10% of the time.
The statistical significance of the study data showing that projects using Impact Engineering practices performed better than all others is so strong that the probability of the finding being incorrect is the equivalent of rolling a number six consecutively six times on a fair six-sided dice, on the first attempt.
The study, conducted by Junade Ali PhD CEng FIET and J.L. Partners , saw participation from 600 software engineers (250 in the UK and 350 in the USA). Fieldwork was conducted from 3rd to 7th May 2024. J.L. Partners is a member of the British Polling Council and abides by its rules.
Three of the four practices listed in the Agile Manifesto are "Working software over comprehensive documentation", "Customer collaboration over contract negotiation" and "Responding to change over following a plan". However, the new research has found that projects which had a specification or documented requirements before development started were 50% more likely to succeed than those which didn't, projects which had clear requirements before starting development were 97% more likely to succeed and projects which did not require making significant requirements changes late into the development process were 7% more likely to succeed.
Other practices also increased success. Projects in which the software engineer reported feeling psychologically safe to discuss and address problems quickly when they emerged were 87% more likely to succeed those which didn't. Projects where the requirements were accurately based on a real-world problem were 54% more likely to succeed than ones which didn't.
Interestingly, the study found no statistically significant difference between project success for those working on one project versus those working on multiple, despite reducing work-in-progress being a key tenent of the Lean software development methodology. However, previous research Dr Ali conducted has shown that 83% of software engineers report feeling burnout with "high workload" reported as the main reason.
The research comes in the wake of catastrophic software failures increasingly being present in the public consciousness. Dr Ali 's previous book "How to Protect Yourself from Killer Computers " investigated numerous cases of fatal software where the technical causes have been attributed to software design problems, including aircraft entering ‘death dives', fatal car crashes and killer radiation overdoses in hospitals.
Indeed, the Horizon IT system was one of the earliest large-scale projects to use an Agile methodology, namely Rapid Application Development, which has been condemned by Fujitsu engineering witnesses in the public inquiry (Terence Austin and the whistleblower David McDonnell ) as a cause of the technical problems due to the absence of a robust requirements engineering process. Charles Cipione , the technical expert witness to the inquiry, summarised simply, saying that "if you don't have a good design, it's not going to work properly." The failure to address these issues and cover-up attempts led to the Post Office scandal which has been described as the largest miscarriage of justice in British history, linked to multiple suicides with those wrongly imprisoned including a pregnant woman.
The research also found, disturbingly, that software engineers in the UK were 13% less likely to feel they were able to discuss and address problems than those in the US; the largest difference of all engineering practices between the two countries. The finding comes after November 2023 research by Engprax found that 75% of software engineers in the UK faced retaliation after reporting wrongdoing.
Author of Impact Engineering , Dr Junade Ali said: "With 65% of projects adopting Agile practices failing to be delivered on time, it's time to question Agile's cult following. Our research has shown that what matters when it comes to delivering high-quality software on time and within budget is a robust requirements engineering process and having the psychological safety to discuss and solve problems when they emerge, whilst taking steps to prevent developer burnout. This is fundamental to the philosophy of Impact Engineering ."
" Impact Engineering " is now available on Amazon in Kindle eBook and paperback formats. The business novel is based on real-world case studies of personal and organisational transformations using the Impact Engineering methodology and a newly developed psychological framework for achieving successful transformations, alongside having a chapter describing the underlying scientific basis of the methodology.
Ends
Update 17th September 2024
You can find various responses to claims made by Agile Manifesto co-authors regarding this research at the following links:
Note to Editors Media images can be found here , please attribute "Nicola Bald Photography" when using the pictures of Dr Ali. The book cover design is by Katarina Naskovski and can be attributed to her. The author, Dr Junade Ali MSc PhD CEng FIET is a British software engineer and computer scientist. "Junade" is pronounced Jun-aid (with a hard "J") - recorded pronunciations can be found on HowToPronounce and Dr Ali's LinkedIn profile .
For further information or for an advance copy of the book exclusively available to the media please contact the Engprax Ltd press office: b.mahmood@stockwoodstrategy.com .
About Engprax
Engprax is a Scottish company offering software audits, investigations and consultancy - minimising the risks associated with software whilst maximising rewards, to ensure all of society sees the benefits of technology. Our publications have included:
To learn more, please visit: www.engprax.com .
Research & Data Tables
From May 3rd to May 7th 2024, J.L. Partners solicited responses from 481 software engineers who last encountered a successful project and 119 who last encountered a failed project, to allow sufficient analysis of both groups.
The following software engineering practices saw the following increases in the success rates versus when they are not used:
Engineering practice
Increase in success when using vs not using practice (%)
The project requirements were clear before the software development process began
97%
Being able to discuss and address problems quickly
(I.e. "psychological safety")
87%
The project requirements were accurately based on the real-world problem
54%
The project had a complete specification or requirements document before the development started
50%
No significant changes were made to the requirements late in the development process
7%
The software engineer reported not having to work on more than one project at the same time
No statistically significant difference (-3%, p = 0.558).
The failure rates by methodology and the increases in the success rates when adopting the various methodologies are provided in the following table:
Engineering Methodology
Failure Rate (%)
Increase in failure when using vs not using methodology (%)
T-Statistic
P-Value
Agile Requirements Engineering
(Development starts before clear requirements, no complete specification, significant changes late in development.)
65%
268%
4.94
3.83E-5
Lean software development
(Only working on one project at once.)
21%
7%
0.59
0.55 (Not statistically significant)
Impact Engineering (Use of all engineering practices studied which increase success rates.)
10%
-56%
-4.15
4.11E-5
Practices used by prevalence in the UK and USA individually are presented in the following table:
Practice
Prevalence in UK (%)
Prevalence in USA (%)
Being able to discuss and address problems quickly
79%
90%
The software engineer reported not having to work on more than one project at the same time
61%
71%
Significant changes were made to the requirements late in the development process
65%
59%
The project had a complete specification or requirements document before the development started
88%
83%
The project requirements were clear before the software development process began
92%
88%
The project requirements were accurately based on the real-world problem
83%
86%
Statistic 89% of US business decision-makers and 81% of UK decision-makers are concerned about the on-time delivery of software from research published in December 2023 conducted by Dr Ali and J.L. Partners for Haystack Analytics.
Projects using Impact Engineering were 50% less likely to fail than a baseline of all projects in the dataset. CISQ (the Consortium for IT Software Quality) estimated that in 2020, unsuccessful software developments cost the US economy $260 billion, a 50% decrease in failure rate would amount to a $130bn USD annual saving. The UK House of Commons Committee of Public Accounts estimates that £20bn is spent annually by the UK Government on digital change; a 50% decrease of 70% failing projects would be an estimated £7bn annual saving.
Update 17th September 2024
You can find various responses to claims made by Agile Manifesto co-authors regarding this research at the following links:
