Estimation? It’s Complicated…

Paweł Brodziński January 28, 2015

We are a web software shop. We frequently work for startups, including those who are just starting. Inevitably we get asked to estimate batches of work.

I never have a quick and easy answer when it comes to estimations.

On one hand I perfectly understand the rationale behind requesting estimates. If I were starting my own project or even a company, I would like to know how much I need to bet in order to turn it into a business or at least validate the hypothesis that it can be turned into one in the first place. In fact, the scale of that bet would influence my decision.

Even if the information in terms of the expected cost wouldn’t make me turn away from the idea altogether it may influence decisions about who will build it for me and how I will structure the development process. It’s not that I think the expected cost should be the determining factor in terms of making the ultimate decision when it comes to building software. Pretty much the opposite. Though, it has to be taken into account.

In either case, an estimate provides some information which is often pretty important in deciding on a course of action.

What’s the problem with estimates then?

Estimates are most often interpreted as commitments, which they are not. Not only does it influence how people act in a situation when it turns out they were not accurate, it also changes the whole discussion about estimation. A thing I would frequently hear is that since a project is well defined providing a precise estimate in such a case should be easy.

It’s not for a number of reasons.

Precise Estimates

One thing is that the human brain is pretty much incapable of estimating in abstract metrics such as space and time. Not only that. It also doesn’t get better with simply more experience at estimation. Research conducted by Roger Buehler et al showed that even for tasks we are familiar with, and even when asked for an estimate that we’re 99% sure of, we’d get that estimate wrong more often than not.

planning fallacy chart

Let me rephrase, even when we play it super-safe the quality of our estimates is mediocre at best.

In software development, in the majority of cases, we simply can’t play it super-safe. A new project or a product is always new so familiarity with the subject matter is always limited and clients very, very rarely are willing to pay for confidence levels as high as 80%, let alone 99%.

The bottom line is that even for a fairly well known domain we can’t expect precise estimates.

Scope of Work

That’s not all though. Another argument in this discussion is what gets estimated. First we estimate a batch of work and then we never, never ever, in the end build exactly that. I’m yet to see a project where the scope doesn’t change over the course of building it.

There’s a number of good reasons for that.

  • As the project progresses and a client sees the parts that are already working they have more insight on what should and what should not be there.
  • As time passes clients learn more about their business environment and that results in changes in how they think about the product.
  • We are only human and simply forget stuff.
  • We have to take care of all the rework that happens once a decision maker changes their mind after something has already been built.

The worst thing that could happen is when a death cycle starts: we keep adding new features to a product, which makes it late and this means we have to add even more features to catch up with the market. This in turn makes the product even later, which forces us to add even more stuff… Now, go figure out what happens with reliability of estimates in this scenario.

The tricky part is that it’s much easier for us to add new stuff to the plate than to remove existing things. Most commonly we end up building more stuff than what was part of the initial scope. How does that work for estimates?

Knowledge Discovery

Another challenge is related to the learning process that is an inherent part of every project. We start a project with a bunch of assumptions. These assumptions will frequently get invalidated which, more often than not, means more work than expected.

A common answer to that is to add more details to specifications. That doesn’t really work. It would if we were talking about the known unknowns. In other words, we would know exactly what questions we should ask. The big risks are in the unknown unknowns – ones that we can’t predict or ask about up front. We will become aware of them eventually and they would affect the scope of work but the trigger is having parts of the app built and getting feedback from a client.

There’s another reason why going into more detail with specification is most often a bad idea. It doesn’t come for free. It means spending time and money on scoping out all the details and delays the start of actual work. The latter is frequently much more costly because of the Cost of Delay.

In fact, Douglas Hubbard argues that adding more and more details makes estimators more confident while the quality of their estimates gets worse.

When you look at a project from the perspective of a knowledge discovery process, the moment where you naturally know the least is before it commences. It is at this exact point where we are expected to prepare estimates.

Collaboration

Finally, there’s something that is almost never taken into account, despite the fact that this factor itself may be responsible for increasing the effort needed to build the same batch of work by a factor of two or more.

This magic factor is the quality of collaboration. It affects a team working on a project on many levels.

The first thing is the availability of the client. If we get all the information we need in a timely manner we go in with much fewer non-validated assumptions when we build specific features. This highly reduces the amount of tweaks needed to satisfy the needs of a client.

Then we have feedback. The faster we get feedback about the stuff we built the faster we can improve it and thus we shorten the cycle time of finalizing features and limit the number of tasks that are in progress. This results in less multitasking and a much better efficiency of work.

If that wasn’t enough we also have more psychological factors. If collaboration on an interpersonal level works well people tend to be much happier. Happy people also results in more efficient work. Not to mention that it also means that a team is much more likely to go the extra mile.

It all stacks up to a point where the quality of collaboration is typically the biggest leverage one can use to limit effort and the cost of building a project. At the same time we never know how this part will go until we start working together. On one hand it is a major factor that influences any estimate, on the other it can hardly be known at the time when estimates are prepared.

The Bottom Line

Let’s go back to what I started with. Estimates often have a crucial role to play in the decision making process. We can’t get them right though. Are we doomed then?

Not really. There are two basic lessons to learn here. One is that the reasons of commonly low quality estimates are not specific for us – these are general observations. That means that when someone provides an estimation with precision, they’re either unaware of all the uncertainty involved or they’re one of these “let’s throw an appealing number at a client and then we’ll figure something out” guys.

If the latter is true I’d challenge the idea that it’s wise to work with such a partner. It’s not going to turn into a trust relationship.

If the former is the case, there’s at least some hope. With some dose of luck the outcome of such a project can be OK and a reasonable estimate can make a decision about kicking the project off easier. There’s a downside too. Such an approach is reckless. It means a lack of awareness of risks and as a result no risk management. In the case where a budget is not sufficient to cover every single thing, which is a common situation, tradeoffs have to be made when we have the least options available.

I don’t think I need to mention that it is much better to actively manage risks, including those related to the scope of work, from the very beginning, as this universally yields better outcomes. For that a level of awareness of the challenges facing estimation is a hard prerequisite.

Fixed Budget

After sharing all that, one could get an impression that we wouldn’t consider working under fixed budget constraints. After all, we say that the estimates we provide have a high level of uncertainty which translates to a lack of hard commitment that we would finish a defined batch of work within a specified budget.

Such an impression would be wrong. We have nothing against working under a fixed budget. One thing that both parties need to be aware of is that in this scenario we don’t treat the scope as fixed.

There are consequences of that fact for both parties. One thing is that we need to work very closely with a client to define what the crucial parts of the app are and in what order features should be developed.

It doesn’t mean that we need to have all the decisions up front. In fact, my common advice is to define a set of features that the application must include in every single, even the most crazy scenario that can happen. I call it a Minimal Indispensable Feature Set. It definitely doesn’t mean that such a set of features would constitute an MVP. It shouldn’t. It shouldn’t be viable for release. At the same time it would be a pretty safe choice as these features are assumed to be ultimately part of the app in all possible cases.

Further decisions can be delayed till we get feedback from building this set of features, which provides a lot of value:

  • We will be further down the knowledge discovery process so we will know more about the work in general.
  • We will get some feedback from a client about the early releases.
  • We would uncover some of our assumptions – unknown unknowns – and move them to a more predictable domain, the one where we at least know what questions we should ask.
  • Based on the data points from the early stage of the project we will be able to tell much more and with much better precision what our pace of development is and thus vastly improve our estimates.

From that point on we can continue making decisions about the scope in a similar manner, deciding on fairly small chunks of work that we know we will need, and continuously iterate on that while learning more about the work we do. It also means that our understanding of what would fit the fixed budget will be improving continuously and thus we will inform the tradeoff decisions. We will suck the most out of the available budget.

By the way, we can use exactly the same approach in the case of where we have continuous funding for a project. Although, I rarely see enough determination and discipline on the side of the client in such scenarios to go for that.

Meaning of Estimates

My message here is twofold. The less important part is explaining why we do what we do with the estimates and why I would always start answering a question about estimates with “It’s complicated…”

The important part is that in order to have a meaningful discussion about the estimates we need to understand all the caveats behind the process. Otherwise, we simply go with the answer that is most comforting for us initially, which is likely also the one that would introduce a lot of troubles down the line.

It is very rarely, if ever, a sensible choice.

How many colors do you see?

Have you ever wondered how many colors a picture actually has? How many colors do you think are on the following image?

cat

Many people would say 2, but actually there are 1942.

It’s because of the fact that in order to make an image with smooth borders not all of the pixels are pure black but consist of many grayscale colors.

Existing solutions

I came across this interesting problem recently, as I needed to implement a validation of a color count for the image user uploads to our Rails application.

My first thought was that it should be trivial to do so by just using RMagick. Unfortunately, the color_histogram method returns all these grayscale colors too and I couldn’t find a way to exclude them using just RMagick. I’ve tried all possible options for image processing, trying to flatten the colors as much as possible to reduce the color count, but it seemed to be impossible to set such parameters that would work for all the possible images that people could upload.

I found some sort of a solution using a Python library – colorific, but I also wasn’t able to configure it in a way that it will work for all my sample images.

Algorithm

So I decided to change my approach completely. Using RMagick I got a color histogram, so I had all the existing colors with their occurrence count, and I was able to easily calculate the percentage of each particular color in the image.

Now I needed to decide which colors I should take into consideration and which to ignore. I didn’t want to just set an arbitrary percentage threshold. Imagine we set the threshold to 1%, so every color that doesn’t cover more than 1% is ignored. And now lets have an image 10 pixels wide and 10 pixels high consisting of 100 different colors, so each one covers 1% of the image. With such a threshold, all colors would be ignored, so the image would have 0 colors while it actually has 100 colors. Also setting the threshold too low would not work, as with other pictures it would calculate more colors than expected.

So I came across the idea to sort colors by percentage in descending order and take the first of them that sums to an arbitrary percentage – after experimenting with many samples I chose 98.1%.

But then I conducted an experiment on the image below:

Image with a colorful text saying 'colorful'

As you can see, there are gradients on every letter and here these gradients actually matter.

Take a look on magnified letters “o” and “r”:

'Colorful' image magnified detail

If you just take into consideration one of the green colors from “o”, it will make such a low percentage score that it will be ignored by the reduction algorithm.

I realized that I need to group similar colors before reducing. So firstly, I had to agree what does “similar” actually mean. I tried comparing two colors in different color spaces – RGB, YUV and Lab and the last turned out to be the most appropriate – it matches more accurately with how the human eye perceives color.

For performance reasons I also added a phase of color limiting if the amount of colors is so big that calculating groups would take too long – by default I take into consideration only the first 10000 colors with the highest percentage value.

Summary

So to sum up the phases go in this particular order: limiting colors, clustering and noise reduction in this particular order. The result is a Ruby hash with the main colors as keys and grouped colors as values.

You can use it to display color palette: A color palette After doing so much work on that I decided to make it publicly useful and extracted the code to a gem. I named it gauguin in honor of my favourite artist.

Recoloring

One of my Lunar colleagues, Hania Seweryn, came across a great idea – a feature that would take the image and it’s calculated palette and return a new image, colored only with the main colors. I loved the idea, so I implemented it in the gauguin gem.

For the above image it would be:

Modified 'Colorful' image

What can I use this for?

Color palette with paintbrush You can use palette method whenever you need to get a realistic color count of an image or want to create color palettes based on images. The recolor method can be used to reduce noise, for example before vectorising an image.

It was very useful in my original task – I needed to validate the color count of the images uploaded to our application because they were meant to be printed, and the price depended on the number of colors. Images were to be vectorised before printing, and thanks to the recolor feature I could show the user how the colors will be reduced when the image is printed.

For both retrieving palette and recoloring you can check out this demo site.

Software Carpentry in Krakow!

Anna Ślimak December 11, 2014

Aleksandra running bash workshop Software Carpentry is a non-profit organisation which runs workshops all over the world, provides open access teaching materials, and runs an instructor training program.

The main idea for these events is to help scientists to develop their programming skills so they can use them on a daily basis to automate their work.

The workshop program is usually fixed to the following areas:

  • the Unix shell (and how to automate repetitive tasks)
  • Python or R (and how to grow a program in a modular, testable way)
  • Git and GitHub (and how to track and share work efficiently)
  • SQL (and the difference between structured and unstructured data)

Attendees work on their own laptops so they have everything installed so as to be able to work on their own later on.

This weekend, thanks to Aleksandra Pawlik, we had the first such workshop in Cracow – Women in Science and Engineering. This particular event was targeted towards women who work as scientists and who could use programming skills to be more productive at their work. There are lots of events targeted only for women lately as they tend to be too shy to go for the usual events, so there is a strong community that tries to convince them that they can do anything they want.

I had a pleasure to help during the event – while Aleksandra and Paulina Lach were running the workshop – I along with Barbara Szczygieł, Iwona Grelowska and Patrycja Radaczyńska were ready to solve issues and answer any question from attendees. And one of my Lunar Logic collegues – Basia Madej – helped with organization.

Smart stickies system The event was, as usual, two days long.

On the first day Aleksandra ran a workshop on bash and version control.

It was very important to participants as they sometimes are required to login to some server without a graphical interface so they need to know how to use command line.

Also version control can be really useful, eg. while working with a colleague on a publication.

On the second day Paulina Lach was teaching Python. Attendees could learn how to replace their usual Excel documents with more efficient Python scripts, how to plot things, how to use numpy library to compute operations on matrixes and so on.

Last, but not least, participants learnt about the power of the databases and how they can use them in their work.

We agreed on a very smart system of red (or pink ;)) and green stickies – when an attendee finished a task without problems, she put a green sticky and any time she had an issue – she used a red sticky so people could react immediately to help.

It was really great to see all these women fascinated with the topics and willing to continue learning programming, feeling that it can be really useful for them.

There were lot’s of people interested in taking part, unfortunately they ended up on the waiting list, but I believe there will be more such events in the future.

I really like that the community is growing here!

Photo of women in Science and Engineering

How to prepare for the future of software testing?

Paulina Materna December 3, 2014

Vulcan salute illustration

Imagine when you go to your first Agile Testing conference. It’s the opening keynote. What do you see?

I saw Spock. On stage. With “Welcome to the future!” written behind him. And unicorns…

Yes, the main theme for this year’s Agile Testing Days’ 2014 was all about what’s on the horizon in software testing and some of the speakers took it quite seriously. The suggestions of how tomorrow’s world of software testing will look sneaked in to almost every talk. The conference started with Lisa Crispin and Janet Gregory, dressed up in Star Trek costumes, speaking on how we should ready ourselves.


The opening keynote inspired me to gather together everything that I thought would prepare me for my agile testing journey. So through the whole conference I was looking for ideas, that would give me clues on how to prepare for my own future adventure.

Here’s a list that I came up with:

1. Experiment and be creative!

The only thing that you can surely say about the future is that it brings change. There is no point in being stuck with only one correct style of testing (I don’t even believe that such a thing exists). To embrace everything that the future brings, You have to be creative and sometimes do the unexpected.

The need to experiment was first highlighted by Lisa Crispin and Janet Gregory who, during the opening keynote, showed that it is the best way to face complex problems and new unexpected things is to try fresh techniques.

The talk that inspired me the most in terms of creativity was Jan Jaap Cannegieters “Flexible testing and using different sides of your brain to optimize testing”. He proved that one way of testing is not good for all projects. Sometimes you have to follow your intuition, try new things with a goal in mind, question everything you do or just play with your work. To continue this path I went to „Growing into Exploratory Testing” by Peter Walen, where I learned that sometimes the best way of testing is by not creating a big plan, but by just letting your next move be determined by the previous one.

2. Communicate

There could be a whole book written on how communication helps an agile tester. Seriously. During the conference I realised, how much value our every-day conversations add to the work that we are doing together. Even when we don’t talk about code or features and just chat about our feelings, daily experience or problems, our conversations are fruitful, because we share and reflect our needs. This is what I got from Bob Marshalls keynote “The Antimatter Principle”. He assumes that the purpose of creating software is to “meet peoples’ needs”. And by people he not only means users and customers, but also programmers, managers, testers and everyone involved in the environment where “making software” takes place. And why would we need to know ours or others needs to create good software? Because people are able to self motivate and the thing you have to do to trigger intrinsic motivation is create the space where people’s needs are met and discussed.

3. Develop Your skills

lego “Testing is learning”. These are the words from Alessandra Moreiras presentation “The mindset of an agile tester.” You have to study the application you are working with all the time. You also have to know a lot about testing techniques to choose the one that fits the most, not the one that you feel the most comfortable working with.

During this year’s ATD there were many opportunities to develop testing skills. One of my favourites was Huib Schoots‘ workshop “Practical Agile Test Strategy with Heuristics” . It was really well driven. Explanation, theory and exercises were balanced and effective. It inspired me a lot (and I’m currently using heuristics that I’ve learned there).

Another great workshop was “LEGO TDD and refactoring” by Bryan Beecham and Mike Bowler. Who would have thought that playing with lego will explain to me the idea of refactoring, TDD and (what is the most important) words written in the Agile Manifesto. I still have in mind the moment of having an idea of an awesome giant lego sculpture and realising that there is not enough blocks left to prepare a shipping container to deliver it to the destination point. And sometimes I still get myself caught up in thinking about applications as being a lego structure :)

4. Fight against what limits you, embrace your talents

I couldn’t write about this years ATD2014 without mentioning Antony Marcanos keynote “Don’t put me inside the box”, which was a true enlightenment to me. It made me aware that calling people a “human resource” is always harmful. Each of us is a unique personality and has multiple talents, but we often forget about it and don’t use what’s best in us. We allow ourselves to feel limited by our job title, desk, workspace, mindset. But, guess what? We are not the machines, we have our special features and strengths, that always add value to the work we are doing. That’s why, just like for Antony, my favourite job title is my own name and my skills are as an equal part of my everyday work as my personality. Knowing whats the best in us, fighting against things that limit us makes us a better… whoever we need to be.

5. Don’t be afraid to… fail

Sometimes we have this awesome idea that would change everything. Our mini revolution. We force it to work, we try to share it with others and … it appears that the idea wasn’t that good at all. We get frustrated and forget about it. No one shares the stories of failure, nobody publishes research that finished without a clear result. Wrong!

On the second day keynote Alexander Schwartz and Fanny Pittack shared their stories of failure. Why? Because failure is not always the end, it can be the beginning of creating something better. During their presentation they showed that frustration (ours or our team members) is always important feedback and that we learn a lot from the ideas that didn’t work out.

Another important factor is creating a failure tolerant environment, which was mentioned in Roman Pichlers “Strategy testing” keynote. Life is too short for delivering things nobody needs. Examining and analyzing ideas at an early stage is very important. Accepting the fact that some of them might fail, can help you focus on really important problems.

My list is ready, when does the journey start?

Agile Testing Days stage “Your future is created by what you do today, not tomorrow.” said Richard Seidl in “5 habits to survive as a tester”. There is no starting point for the journey. My list will grow with each and every experience. There was a slide in Lisa Crispin and Janet Gregory presentation that said:

“As we continue to go where no man has gone before, can we really predict what will happen?”

I don’t know what the future will be like, but I can surely say – I’m ready for it!

Using guard gem in a project with multiple modules

Anna Ślimak November 26, 2014

Tests are important

guard illustration Regardless of what I work on – frontend or backend – the most important thing for me is to be sure that everything is working properly after my changes. That’s why, in my opinion, tests are essential.

In order to not have to remember about executing tests on every change, I use guard gem, which does it automatically. So I always add guard to every project.

Complicated project with lots of modules

Lets assume we have a project that consists of separate modules, each having its own test suite. Theoretically you could setup guard for every module separately but then you would have to monitor separate terminal tabs for each module to be sure that everything is green.

It’s easy to get lost in tabs, so I wanted to setup guard in the usual way, to have one terminal with the guard that will watch changes from all the modules.

I found the watchdir option, which I thought should do the job, but it didn’t work the way I expected. I touched base with the guard core team and it turned out that this option is just an optimization for really large projects in order to not kill your processor or hard drive.

New option in guard to the rescue

After discussing the issue with the guard team, I decided to implement a new option for that. The new option is called chdir. There are changes to both guard and guard-rspec, they have been merged into a master and are waiting for release.

Assuming you have your project setup like here, so you have all the usual tests in spec directory but you also have the moduleA directory with a separate test suite in moduleA/spec, you can use guard to monitor all your modules running the usual command bundle exec guard start and having Guardfile configured in this way:

There are 4 important things there:

  • chdir option passed to guard method
  • changes in cmdguard needs to change directory to execute your tests in moduleA
  • a prefix needs to be added to every pattern
  • everything is done in a loop for every test suite

Now you can run bundle exec guard start and check that it also monitors also tests in the moduleA directory.