The Trouble With Software Estimation
Originally published 2011-01-21 The Trouble With Software Estimation In an effort to become more consistent with the way our company delivers estimates to clients and to each other, we decided to t...
Your browser is unable to display this site correctly. Please try an up-to-date version of Chrome or Firefox instead.
Customers of our mid-sized Canadian software consultancy are generally well-established companies who have experience hiring vendors to help them with large, complex software builds.
When engaging on a new development project, these customers can choose between the following 3 scenarios:
The relative costs of the first two choices were the subject of a prior article: "The Hidden Value of Software Consultants." However, if the choice to outsource the work has already been made (for various reasons also discussed in that article), companies must then additionally decide whether to go with an onshore provider or an offshore provider. This article aims to holistically contrast these two choices.
Hourly rates of a now-mature offshore software consulting industry have settled in at between 25% to 50% of an "equivalent" onshore development rate. For the purposes of this article, we will assume that the cost of an offshore hour is 30% of the onshore cost for each nominally equivalent role. Also, I will use the word "client" to represent a purchaser of software services, and the word "vendor" to represent an external seller of these services.
Some clients choose the "lowest blended hourly rate" as the most important cost metric, implicitly assuming the value of an offshore hour is approximately the same as the value of an onshore hour. Unfamiliarity with the software development process may lead to this assumption - that an hour of development effort is a commodity, the cost of which should be easily comparable between vendors. Another, more subtle reason is that the blended hourly rate (measured in dollars per hour) is simply easier to measure than the cost per unit value (often measured in dollars per "function point"). Because the latter measure is often unavailable or difficult to achieve, the "cost per hour" measure is adopted instead, by default.
From a business perspective, however, the "cost per unit value" measure is far more desirable. Businesses are charged with delivering business functions to their clients at minimal cost, as opposed to simply purchasing hours at a minimal cost. This is clear if the extreme case is contemplated, wherein effort hours are spent but no value is recognized. This is the case, for example, when a project is cancelled mid-stream, before any features can be delivered.
Stated another way, if Team A is 50% as efficient as Team B and costs 50% as much per hour, the overall cost of the project will be the same. We will thus use "efficiency" as the factor that allows an apples-to-apples comparison of effort hours.
Some of the more obvious differences between onshore and offshore teams are discussed in this section. Those who have run offshore projects before will likely recognize them.
Time zone differences are usually seen as distinct disadvantage, owing to the relatively small window of time each working day for synchronous communications between team members.
It should be noted that some view this as an advantage, since it allows a daily 16-24 hour work cycle as opposed to an 8-12 hour cycle. Formal documentation can help with this, and indeed, when responsibilities are very well understood and specified, the 24hr work cycle may be more of a benefit. This is why testing engagements (the work for which is usually well-specified and involves repetition) are seen as the best candidates for offshoring, as the development team can produce code in the first 12 hours of the day, and the testing team can test what they’ve produced during the next 12 hours, assuming the joint team can be locked into that schedule.
However, this doesn't decrease the number of hours of effort on the project - it only (potentially) compresses the schedule. Furthermore, maintaining reams of formal documentation to ensure the success of independent team members that depend on each other is an inefficiency in and of itself. In fact, it's not cost effective to do when creating a system from scratch, due to the rate of course changes, which is why software specs tend to be much looser these days, especially in the "agile story" world.
Furthermore, time zone differences become more detrimental as the amount of collaboration required between teams increases. Having to wait a day each time important information needs to be communicated (or each time a small question needs to be asked) can be absolutely maddening. And custom software projects are highly collaborative in nature. A tremendous amount of work product malleability is expected from the digital medium, which engenders the need for continuous communication between team members, which in turn works much less well with remote teams.
I’m assigning a 15% efficiency penalty for major time zone differences between the customer and the vendor.
On one occasion, some from our Toronto team engaged with a cultural consultant to help us understand some of the potential pitfalls of working on a joint onshore / offshore project with a development team from India. The discussion was fascinating, and the consultant’s assertions were subsequently largely borne out in our experience on the project. At the risk of over-generalizing, here are a few examples:
There were many other differences, and it’s not for this article to discuss which approaches work better overall. The point is that major differences exist, and expectations for team members are generally somewhat hidden, abstract, and implicit before projects begin. Communicating abstract goals, requirements, designs, and expectations in general to joint team members with different cultural norms can (and does) lead to unexpected results and inefficiencies.
I’ll assign an efficiency lift of 15% for onshore teams with similar cultural norms as the client.
Communication is difficult enough as it is between human beings having the same mother tongue. Everyone’s heard the running joke that there are 4 conversations going on whenever I communicate with someone else: what I said, what I think I said, what you heard, and what you think you heard (Not to mention "what I think you heard" etc etc).
This is exacerbated by the fact that communication in software development requires a high level of precision, so much so that clients are often annoyed by the way business analysts continuously press them for precise specifications ("that’ll never happen.", followed by "but what if it does?" etc). Codifying instructions to unthinking machines requires this level of precision, which has to be derived from the specifications or through conversation.
Additionally, the work product is very abstract. Much of it can’t really be "seen," making precision of communication even more important. Somewhat opaque specifications, technical models, and code comments are used to specify and communicate what should be produced, what will be produced, and what was produced. Sharp, clear communications are very important in this sphere, especially in the early stages before teams have explored and optimized their working relationships. Language barriers can be a significant issue even for teams that are co-located, but distance and time exacerbate these barriers significantly.
Since clients generally operate in different jurisdictions than their offshore vendor counterparts, and communication efficiency is so important on software projects, I’m assigning a 25% language barrier efficiency loss for projects with offshore vendors, relative to onshore ones.
We’ve all experienced the following things related to a lack of face-time:
The bandwidth of in-person communications is shockingly high relative to remote communications. Additionally, nothing establishes a connection with another person better than a face-to-face meeting, a meal or drink, or just some chit-chat between working sessions. Sharing personal stories help team members to let their guard down and engage more fully. Counterparts implicitly offer humanity and vulnerability when they meet face to face, in good faith. Asking "how are we doing?" gives the customer an opportunity to voice complaints without making a big deal out of it, lowering tensions for all. Face-to-face meetings promote trust and good relationships, and increase collaborative capacity dramatically.
It’s clear that onshore vendors will have a distinct face time advantage here: let’s say the ability to collaborate in person offers a 25% efficiency lift.
Most offshore vendors try to solve the problems associated with remote workers using a hybrid model, in which onshore leaders are placed onto the project (at the higher onshore hourly rates) and offshore workers are used to lower the blended cost per hour.
Onshore workers interact face-to-face with the clients and communicate long-distances with the offshore development team. The effectiveness of this strategy depends largely on how much interaction is required between the customer and the team members, and between the onshore leads and their team members. On custom development projects, this tends to be significant.
This strategy also introduces an extra layer of inefficiency due to "translation" of what the customer wants into instructions for the remote team, and it still depends on the onshore lead’s ability to successfully navigate the cultural and communications differences of the different parties.
The ratio of leaders to workers is important in this regard. Our company’s experimentation with this model led to the need to install a leader onshore and an offshore leader counterpart to get the most out of the remote team, with the attendant losses in efficiency. On a team of 8 with let’s say 2 onshore leaders at $125/hr plus 6 offshore team members at $33/hr, the blended rate is closer to $56/hr overall – a huge percentage increase!
The addition of each new team member carries with it an extra piece of overhead, owing to the communication and understanding that must be established in that person’s head in order to profitably contribute to a project. Additionally, there are the attendant overheads in reviewing the work of more people, collecting status from more people, ensuring that more people meet project standards like design conventions and code structure, and management overhead related to a higher rate of interpersonal issues (more intra-team relationships lead to more issues). Let’s say each new person added to a project gives us 80% of an FTE in terms of productive work (i.e. there is a 20% overhead associated with simply adding that person).
Offshore teams tend to be larger than the equivalent onshore teams. The rationale for the client goes something like this: "we will suffer some overheads and inefficiencies by offshoring this project, however we will make it up by being able to afford more people on the project due to the lower cost per hour."
This logic negates the concept that small teams are more efficient. At Jonah, we like to use 6-8 person teams for peak efficiency and risk management. Based on my experience, I assert that peak efficiency is achieved with a team of approximately this size. Going "too small" creates a risk of losing key resources, but going "too large" creates too much communications overhead. We need a swath of different capabilities on the team and a small amount of redundancy such that we don’t put all of our eggs in one basket.
If schedule and velocity are more important than efficiency, and the client is willing to live with the performance hit associated with a larger team, then the team can be made larger, though it’s usually broken up into sub-teams to keep things manageable.
If there’s likely a 20% (per team member) overhead for each new person added to a project team, and the team is (say) 50% larger than the equivalent onshore team, then this amounts to an overall 10% efficiency decrease due to team size overheads alone.
Here’s a summary of our efficiency penalties owing to hard factors alone, as described:
If we assume that the known factors are completely independent of one another, we would add up all of the efficiency penalties to bring us to a 90% total efficiency penalty factor. If, on the other hand, we assume that all effects multiply to produce a combined efficiency loss (e.g. if language barriers are exacerbated by cultural differences), then the penalty factor can be as high as 227% (1.15*1.15*1.25*1.25*1.1).
A 10,000-hour project onshore at a blended rate of $110/hr costs $1,100,000.
Applying a 90% offshore efficiency penalty means a 10,000 hour project onshore will require 19,000 hours. At $56 per hour (hybrid model), the total "offshore" cost is $1,064,000.
Applying an 227% offshore efficiency penalty means a 10,000 hour project onshore will require 32,700 hours. At $56 per hour, the total offshore cost is $1,831,200.
The reality is somewhere between these two numbers of course, but on hard factors alone, it seems that offshoring may not be as attractive as we once thought. And we haven’t yet considered soft factors.
Some of the more subtle differences between onshore and offshore teams relative to efficiency are discussed in this section. They are somewhat more tricky to quantify, but can have far-reaching effects.
One easy way to establish the cost per unit value of a project is to ask the vendor to provide a fixed-price quotation for the delivery of a set of software features. Assuming the number of function points represented by those features can be counted, the cost per function point can thus easily be established by dividing the total quoted project cost by the total number of function points. While a fixed-price quotation might seem to solve the problem from the client’s perspective, it pushes the responsibility for rigorously counting the number of function points to the vendor. And it begs the question "what assumptions were made when the estimate was created?" When vendors make different assumptions about the requirements (i.e. the scope and complexity of the system to be developed), it becomes difficult to compare their respective quotes.
On the other hand, on Time and Materials (T&M) style contracts, the client pays the vendor for each hour of effort (as opposed to each delivered function point ), thus assuming the risk of effort overruns.
Irrespective of the contractual style chosen, we can easily agree that knowing the cost per function point is an important metric to both client and vendor, and that assumptions about what is to be built will derail quote comparisons.
The question then becomes: do offshore vendors underestimate the number of function points relative to their onshore counterparts? On the face of it this seems like a silly assertion, until we understand vendor motivations, which I’ll discuss a little later on.
Much has been written about "Technical Debt," the hidden gremlins or ghosts in the machine that only surface long after a project has been delivered, cheques have been written, and the champagne from the delivery celebration has run dry. This is the spaghetti code that is brittle to change, is not properly factored, is difficult to operate and manage, is not commented well, contains few unit tests, does the job but performs poorly, and is a teeming heap of anti-patterns and poor design. Technical debt expresses itself over the long term, and the more that is tolerated in the short term, the worse the long-term pain. Debt also reduces the useful lifespan of a system, making extra capital cost expenditures necessary earlier on in the product lifecycle.
It’s easy to see that technical debt is bad, but it’s not as obvious that the amount of technical debt introduced into a system is a matter of choice. Specifically, it’s a matter of choice of the vendor’s development team. If the client team is rigorous enough to review the code, and experienced enough to detect debt and understand the impact of it, then they might also be in a position to help decide on the level of debt that can be tolerated. More often than not, however, companies hire vendors because they don’t have the specific skills to do the work in-house, much less evaluate the "internal quality" of the code for the presence of technical debt.
A good software vendor should aim to explain the costs and benefits with respect to the balance between debt reduction and the additional effort associated with attending to it regularly. To do this honestly, however, requires trust between the client and the vendor. And for new relationships, that trust would need to be developed to a large extent during the sales process. The associated higher anticipated short-term costs of addressing debt appropriately during the project may put the vendor’s quote at a risk of being rejected, introducing a conflict of interest. That’s why trust is so important at the point of sale.
Indeed, early-stage conversations between client and vendor may surface questions that may seem odd to the customer, like "how much quality do you want?" and "how long do you expect this system to be in place before it is retired and replaced?" Questions like this show that the vendor is being intentional about deciding how much effort will be spent reducing debt, and open about the fact that continuously addressing debt will be more costly in the short term than not addressing it, but will lead to much bigger savings in the long term.
It’s clear that technical debt needs to be managed properly, and tuned to the needs of the customer. It’s not as clear how this relates to the choices that an offshore vendor might make relative to an onshore vendor. Either type of vendor has the opportunity to treat it seriously or carelessly. Again, this relates to a discussion of vendor quality, brand, and motivations.
I’ve heard prospective clients muse aloud that "even if an offshore vendor doesn’t get it right the first time, we can fix the system and still spend less money than if we had gone with an onshore vendor." This may or may not be true, but it begs an interesting question: "What is the ROI of getting it right the first time?" It’s worth considering that a project’s ROI is not limited merely to the value of the finished system delivered in a vacuum.
Delivering fully-functioning projects on time and on budget ensures higher ROI by negating all the downstream crud that occurs when a vendor misses delivery targets, including:
Putting aside for a moment which vendors are better at "getting it right the first time", we can surely agree that there are significant costs associated with missed targets and unmet expectations - costs that are external to the features delivered with the system. Hand-waving wildly here, in considering an enterprise system that’s a part of a much larger value chain within the organization, I’ll quantify the cost of not "getting it right" the first time at around 50% of the project’s nominal value. Multiplying this by the rate of failure or challenged projects in IT helps us understand the ROI of right.
It’s easy to agree that lower costs per hour are generally better than higher costs per hour, just as easy as it is to agree that higher value or efficiency delivered per effort hour is more desirable than lower value per effort hour.
In marketing their services, all vendors will choose a brand that accentuates what they are most able to deliver. Offshore vendors will thus focus on lowering rates, as their cost structure per hour is much lower. Onshore or Near-shore vendors will focus on collaborative capacity, efficiency, and relationship, because they’re in a better position to deliver on these relative to their offshore competitors. Small vendors will pine about their nimbleness and how their customers are "all big fish", while large vendors will boast about the "big names on their logo field" and extol the virtues of stability and experience.
So as publicly positioned, brand promises are hard to validate. It should be noted, however, that a vendor’s external brand also translates into specific primary motivations for its employees. Brands are really just a set of stories told both externally (marketing) and internally (culture), both of which are rooted in the brand values that the company has chosen to express.
Indeed, in arguing that "effort hours are commodities," the lower-cost-per-hour brand is also telling this story to their own employees, with the following associated implicit messages:
What effect might these messages have on the vendor’s employees? It’s not hard to contemplate that a "low-cost" brand is expressed internally a cost-cutting culture. And while minimizing waste is a positive way to cut costs for any company, a negative way of doing the same thing is to externalize costs (e.g. via tolerating more technical debt), or to cut corners (e.g. via doing "just enough" to achieve acceptance of the deliverable). Remote vendors are also less directly accountable to the effects of corner cutting.
Additionally, a low-dollars-per-hour "commodity work" mindset tends to reduce engagement and enthusiasm in the minds of employees, which has an impact on the resulting level of service and work product. They naturally assume that creativity and craftsmanship are not what the customer is asking for.
Finally, when a "low-cost" brand encounters a pressure situation, its culture is arguably more likely to lead it to cut corners, relative to a local "high-value" brand, whose values, culture, and internal stories make it more likely to guard against making that same choice.
Conversely, in focusing on quality, efficiency, and partnership, the "high-quality" brand is telling a different story to its employees, with the following associated implicit messages:
A "high-quality" brand is thus expressed internally in such way as to act on behalf of the customer in producing high quality systems.
To be fair, "low-cost" vendors exist both onshore and offshore, so this is more of a question of evaluating the specific vendor you’re dealing with. However, it’s no secret that offshore vendors compete primarily on price.
Sales cycles for custom software projects are very long, and often this time is spent developing trust with a series of small steps over many meetings. As such, I would argue that the risk of missteps in any of the three categories of "soft factors" is higher when using an offshore vendor, simply because it’s harder for the customer to develop that trust over long distances. This means that at the point of sale, for remote vendors it will be more difficult to know:
Though it’s not particularly useful to attempt to quantify their effects here because of the wild assumptions I’d have to make, soft factors can have a massive impact on eventual project efficiencies and cost.
We must also consider the value system of the buyer, and whether or not they line up with the vendor’s. A penny-pinching customer might revel in the prospect of a low-cost purchases with a few rough edges, whereas a quality-conscious customer would rave about product quality, service, and experience. Where on this spectrum are you?
Finally, it should be clear that vendor due diligence is a major responsibility for the customer, especially with respect to the "soft factors" described above. Clients should be satisfied that value systems align, and that a comfortable level of trust has been developed with the vendor, irrespective of what time zone they are in.
It may seem intuitive that the choice to offshore work at a lower hourly rate would be a no-brainer, but things are often more complex in practice. This is not to say there have not been good experiences with offshore projects, especially for certain kinds of projects. However, others have chosen to remain with onshore vendors, or have tried the offshore route and decided that on-shoring or near-shoring is better for them in the long run.
More philosophically, in order to decide on the right path, one must not only consider the system to be built, but also answer questions like:
As mentioned in the prior article, some of the risk associated with outsourcing (whether it be on-shoring or offshoring) may be mitigated over the long term by cultivating a trust relationship between the client and their chosen vendor. This is dependent on how well the vendor delivers services their clients, and the recognition that good relationships are the shared interest of both parties. Indeed, development partners sometimes become so deeply embedded within the companies they service that they are viewed as indistinct from the company’s internal team. This is when the best work occurs. The ability of (onshore or offshore) vendors to do this is highly correlated with efficiency and overall success.
Comparisons between costs for onshore and offshore staff can be difficult. I hope this article lends some perspective to apples-and-oranges comparisons vis à vis hourly rates of vendor consultants, and the effective hourly rates of offshore development staff.
 Note that the "function points" mentioned here are loosely related to, but not the same as the "points" referenced in various agile software methodologies like Scrum. Agile "points" - normally assigned to features during informal planning poker sessions - are not as accurate as the "function point index," which is assigned to a system to be built using "function point analysis" a more rigorous analysis exercise during which UI fields, system interactions, algorithms, procedures, and data complexity are all counted and multiplied by standardized point values. Function point analysis can generally only work when systems are very well specified, and hence it is not a prevalent practice in industry, since few software projects match that description. Hence, there is a built-in complexity variance with software estimates, making it extremely difficult to compare effort estimates between vendors.
Jonah Group is a digital consultancy the designs and builds high-performance software applications for the enterprise. Our industry is constantly changing, so we help our clients keep pace by making them aware of the possibilities of digital technology as it relates to their business.