Pragmatic “user testing” at SI

We’re not religous about user testing, and most of our smaller features only go review by one or two people before getting deployed. But larger features can get a special treatment, and here’s how we do it.

It depends on the size of a feature, but before shipping a major feature, we typically work like this:

  • First, every developer needs to give their feedback, and we fine tune until we’re happy
  • Then we show the feature to a customer success agent, watching their screen while they explore. We tend to show a feature to the most enthusiastic person first, take their feedback on board, show it to the next person, and so on
  • Once all SI staff are happy, we pick one or two customers and let them explore a new feature while we watch (screensharing again). This often reveals entirely unexpected problems or shortcomings. A favorite user testing session went like this: We had rewritten a very large admin screen from scratch, made it tons more useful, stable, faster and prettier, and we were really proud of it. Our tester, a long term SI customer, looked at the screen briefly, muttered “yep, looks like always, nothing special here.” (we were silently crying at this point). Then she said “dang, there is still no preview-feature for emails that I trigger, does it have to be this cumbersome?”. This was her one feedback – and we realised that we had missed an elephant in the room. Fortunately, she did like the overall screen (although she didn’t realise just how much worse the original version had been) and after adding said email preview feature, she was happy.
  • Once we’ve satisfied two or three key users, we enable a new feature in beta mode for select customers to use permanently. We ask them via mail or phone how things are going, and since each of our application pages has a prominent “give feedback” button, we do hear when there are problems
  • In the next phase, we might announce the beta over our newsletter or via in-application notification, and allow customers to enable a new feature if they like. This increases adoption and feedback again.
  • In the final step, we switch all customers to the new feature, while leaving a “keep the old version?” button in place for another month or two. Once nobody uses the old feature anymore, we entirely drop it, and all associated code.

Well, it’s no rocket science really, but many applicants have been burnt by much worse development processes, so we wanted to share this!

How we plan and prioritize our work

Inevitably, all good applicants ask us this question: “so, perks aside, how do you actually plan and prioritize your work at SI?”.

First of all, we need to consider our overall goal:

“We want to build an awesome product, and enjoy coding and supporting it.

This guides the first two stages of filtering:

Ensure ‘Philosophy match’
This is something we always have in the back of our head. If a feature would make the product less awesome, then we won’t build it. For instance, we often get asked about adding tons more ratings. We don’t think that’s useful, we’re a feedback tool and not a ratings too. So that feature won’t be built at all.

Ensure “Technology match”

Some features would be awesome to have, but terribly to code. Internationalization is such a feature. Sure, having great German or French or Spanish language support would be slick – but it simply makes coding less fun, and each feature much slower to implement. Demand exists, but it doesn’t outweigh the downsides by far. We’re not touching internationalization for the next couple of years.

Prioritization

Once we’ve removed all the feature requests we don’t want to build, there’s still a massive amount of things we can pick from.

Next up, here’s our general prioritization approach:

  • We favor bugfixes over new features. We hate answering the same support ticket over and over again, so it’s more efficient to squash bugs with a high priority.
  • We favor incremental improvements to existing features over entirely new features. It’s better to have fewer but more awesome features, and you only get there by iterating relentlessly. So we are careful in opening up new cans of worms. We’ll ensure that existing features are awesome first.
  • We clean up code debt all the time. This slows us down in the short term, but helps us move faster longer term, and makes bugfixing easier too. Also, if we didn’t clean up debt, we’d get buried after a while, hate our jobs, and quit.
  • We build new features as well. It’s most fun of course. 🙂 But we usually wait until no other pressing work needs to be done before we tackle entirely new features.

Having this basic ordering is good, but there’s still a large amount of features and improvements to choose from. We need to consider efficiency as well:

  • How many customers will like a feature (or improvements) vs how many customers will not even use it?
  • How risky/hard is it to build?
  • What developers is available to build it, will they be efficient at it, and are they keen on building it at all?

The first items are kind of obvious and probably commonplace at most companies. Items 3 is not quite as common though. We believe that everyone should enjoy their work, so even if our AngularJS expert Sebastian is available for building a new AngularJS screen, and would be super efficient – well, if he needs a break and wants to work on the Java side for a month, then the task might get postponed.

Urgency?
If you’ve read this far, you might have wondered about “but, what about urgency!?”.  And that’s the really great thing about SI. It’s almost never urgent to implement something. Sure, we do want to ship a lot of stuff, and do so as fast as we can. We’re not slackers. But working under extreme pressure only leads to problems in code quality and morale, so we try to avoid it. By always prioritizing bugs, by always cleaning up code when we see it, bugs rarely linger until they are really urgent.

Sure, sometimes unexpected problems crop up, sometimes issues are urgent. Then we work on them immediately – but it’s usually not a problem if they take more than a day. We have built a pretty large test system that covers the backend and the frontend, so a massive unexpected problem is rare, and we haven’t done a single night shift in the 4 years of SI history.

So, summing it up, we first filter loads of things that look good on paper but don’t match our philosophy, and then we prioritize with a bias for bugfixes and improvements of existing features, and also take into account what each developer wants to work on.

It’s no rocket science, but we get asked about this a lot by applicants, so we thought we’d share it!