A recent Gartner report has revealed that most AI projects don’t meet expectations or fail, a trend mirrored in general software development. In industries like manufacturing, failure rates can be even higher. What causes these failures?

While many reasons for project failures are obvious to engineers and industry stakeholders, they evolve over time. Some previously common issues have diminished, while others have become more prominent. 

In this article, we’ll explore the current landscape of software development failures, identifying key factors that contribute to these failures and offering strategies to mitigate risks. 

Agile Methodology Shortcomings 

For a long time, stakeholders in tech have suspected that the Agile methodology might not be all it’s cracked up to be.

In fact, Dutch computer scientist Erik Meijer called agile “…a cancer that we have to eliminate from the industry» as far back as 2015 at a talk he gave at the Reaktor Dev Day in Finland.

The methodology started in the 90s as a lightweight alternative to the inflexible way software was being built at the time. However, it is increasingly scrutinized as a pitfall for the same development it’s meant to enhance. 

One Agile developer criticized the daily stand-up element of the principle, describing it as  «a feast of regurgitation» while insinuating that the constant stream of code patch contribution core to it contributes to the decline in code quality and features appearing unfinished or poorly thought out. 

A recent study seems to confirm this. According to tech consultancy firm Engprax, 65% of agile software projects fail to be delivered on time and within budget, to a high standard of quality. 

The survey seems to confirm the result of another conducted by BCG that suggested that despite the hype and widespread acceptance of agile software practices as an antidote to failing IT projects, there is no correlation between the methodology used to design and deliver programs and their success. 

According to BCG’s findings, although agile principles can underpin excellent outcomes, not consistently. In fact, 64% of respondents to the company’s survey reported that their IT teams already used some form of agile software development tools as the foundation for their primary program design efforts. Yet, despite this, the concerning trends for IT project failures persist.

Non-Memory-Safe Programming 

If you’ve read the news in the past few days, you might have heard about the big disruption to critical services that affected the likes of Microsoft, airlines, and several governmental agencies across the world. 

Here’s how Reuters described it: 

“A software update by global cybersecurity firm CrowdStrike (CRWD.O), one of the largest operators in the industry, triggered systems problems that grounded flights, forced broadcasters off air and left customers without access to services such as healthcare or banking. Global shipper FedEx (FDX.N), faced major disruptions and some moderators who police content on Meta’s Facebook were hit.

U.S. Customs and Border Protection [experienced] processing delays and [is] working to mitigate issues related to international trade and travel. The Dutch and United Arab Emirates’ foreign ministries also reported disruptions.

In Britain, booking systems used by doctors were offline, posts on X by medical officials said, while Sky News, one of the country’s major broadcasters, was taken off the air.

The governments of Australia, New Zealand, and a number of U.S. states are facing issues, while American Airlines (AAL.O), Delta Airlines (DAL.N), United Airlines (UAL.O), and Allegiant Air (ALGT.O) grounded flights citing communication problems.

Why does this happen? Why are we seeing an increase in disruptive bugs following software updates like in the case of CrowdStrike? 

Although CrowdStrike’s Falcon Sensor software which caused the recent global service outage was built in memory-safe Python language, The US Cybersecurity and Infrastructure Security Agency (CISA) has warned that critical software projects failing to use memory-safe programming languages that could help tighten up software security vulnerabilities could be a reason for the recent increase in software failures. 

According to itPro.com, the FBI, the Australian Signals Directorate’s Australian Cyber Security Centre, the Canadian Cyber Security Center, and CISA analyzed 172 open-source software projects from the OpenSSF list of critical projects.

They found that more than half (52%) of the projects contain code written in memory-unsafe languages requiring manual management of memory that increases the risk of errors that could lead to security vulnerabilities. 

Further, a dependency analysis of three projects written in memory-safe languages that the committee analyzed revealed that all of them depended on other components written in memory-unsafe languages.

According to a member of the committee, the Australian Cyber Security Centre, «Most critical open source projects, even those written in memory-safe languages, potentially contain memory safety vulnerabilities. Successful exploitation of these types of vulnerabilities, such as buffer overflows and ‘use after free’, may allow adversaries to take control of software, systems, and data. Continued diligent use of memory safe programming languages, secure coding practices, and security testing is imperative to help mitigate these, and other limitations.»

Delays and Budget Overruns

Another reason why IT projects fail is because they routinely suffer long development delays and expensive cost overruns. 

In the BCG survey cited above, nearly half of all the global C-suite executives who were respondents said that more than 30% of their organization’s IT projects were over budget and late. Further, nearly one in five executives expressed that less than satisfactory outcomes occur more than 50% of the time.

Reasons for IT failures and unsatisfactory outcomes, the survey finds, include:

1. Lack of alignment between the technology and business sides of organizations about operational objectives
2. Unrealistic timelines
3. Insufficient resources.

According to BCG, popular software development principles such as agile that implements progress tracking are pertinent but not sufficient to guarantee timely delivery.  

The survey found that “improving the outcomes of IT projects at most companies will depend squarely on technology leadership creating the proper requirements and culture for teams to be able to deliver in agile fashion. Up to now, this has proved to be difficult to achieve.”

Inadequate and Incomplete Specification 

Finally, perhaps undoubtedly the biggest reason why software projects fail in 2024 is Inadequate and or incomplete project specification, going by the top three reasons the survey found to be the culprit of chronic software project failures and unsatisfactory outcomes. 

According to the pollsters, “These three explanations for why IT projects languish point to organizational leadership problems on both the technology and business sides of a company. CIOs, CTOs, and the heads of IT departments (or whatever their title is at a given company) are all too often either left out of the overall decision-making process when a technology implementation is initiated, or they fail to plainly set expectations for what an investment in a new application will specifically deliver both in terms of a timeline and performance.”

Ultimately, overcoming the leadership challenge that causes Inadequate and incomplete project specification will be the biggest driver of software projects’ success. 

Where technology leaders are actively involved from the beginning in strategy development for a tech project, the study finds that the success rate was 154% higher compared to projects where they didn’t. 

As far as BCG’s survey pollsters are concerned, “Technology leaders speak a very different language than business department managers, who can’t evaluate technical paradigms and archetypes or AI algorithms and machine learning dictionaries. But these managers should be able to adequately describe which performance improvements they are seeking from a new application or program and what constitutes a positive return on investment. In turn, CIOs must take the business wish list and explain what is possible—and what resources are required to achieve the possible—in clear language that demystifies the technology.”

In other words, strategic dialogue between client and agency developers rather than managers should be the first step in the development process. This helps to avoid the failures that arise from inadequacy or incompleteness of project specifications that occur when IT leaders are not involved from the beginning.  

Do you have an important software task? Contact us today directly with the form below to request a free consultation session.