Metrics vs Indicators

As I am currently preparing for a CBAP test and going through BABOK a lot of questions come up. I will use this space to post some of those questions and thoughts and findings I had. One particular question for me was the difference between Metrics and Indicators.

I realized I did not have a good understanding of the difference between the two. BABOK 3 defines a metric as “a quantifiable level of indicator organization uses to measure progress against objective”, an indicator “as result of analyzing of one or more measures for addressing a concern about a need, solution, etc.”. Now we come to a concept of measure that I don’t think BABOK clearly defines but I understand measure as “a quantity /value defined through some tool/approach”

None of those definitions were really clear to me.
After some digging, what seems to be the answer is that a metric is a measured state of some sort of a unit at a given point in time e.g. 10000 page views in a week while indicator is something that actually can be used to judge if you are moving in the right direction. If the objective is 1M page views by end of 2018 and current measure is 200K and weekly growth is 10% that means the objective should be hit as planned

At the same time not all metrics are indicators meaning that not everything we measure is really important.

A few links that helped to figure this out for anyone interested


Opportunity Assessment and Problem Pyramid: another way to look at a problem


In this post I will speak about two techniques to elicit a problem and present it. One is the problem pyramid and its light alterations that can be used to elicit a definition of a problem. Another is the opportunity canvas that in my mind can be used in addition to the problem pyramid or instead of it for the purpose of eliciting the problem definition as well as portraying it in a canvas like format similar to Product or Business Model Canvas.


Even though I have already been working with software requirements for over 5 years, I believe I still lack a well-formed and structured method of eliciting information about a problem and then describing it so that it can be addressed. I will be asking a lot of Whys and of course, in the beginning of the project I will utilize a well-known problem statement format to capture the definition problem being addressed. I find that very often it is difficult to put enough meaningful information into the template and it becomes too general and thus meaningless. Quite often, the problem ends up being put together as “legacy design that makes any changes to the software very costly to implement and support impacting level of customer satisfaction. A solution then is to reduce cost of making changes to the system”. Frequently the only solution option then becomes to replace existing software with new software as a way to address the problem. Other solution options are not being looked at period.

Couple years ago I have found out about the problem pyramid concept (term coined by Robin Goldsmith) that allows to present the problem very neatly.

I have used this tool retrospectively trying to understand how well was the original problem statement put together and was able to find flaws in my work. I have also used this inconsistently during the actual elicitation sessions when I was trying to grasp the definition of the problem as seen from customer’s point of view. There were some challenges with using this tool however because often my work starts not from review of a problem but from review of a solution presented as a problem. You might have encountered something similar in your career where you are given a specific project to work on that may sound like “Implement or replace system A with system B; Implement interface for communication between system A and B using API”. You have a few choices then: start doing exactly what you are told, trying to figure out functional/solution requirements in the process and communicating them to the team or question what exactly you need to do starting a series of conversations to get to the actual driver of the project and the problem it is addressing. When trying to reverse engineer the problem being addressed based on the incoming solution description, there is another tool that may come handy. If you have read the disclaimer, you might have guessed its name as Opportunity Canvas. If so, you are right, this is the tool I want to focus more on in the next few paragraphs. .

Let’s look at a couple of scenarios where these tools can apply.

Figuring out what the problem is using Problem Pyramid
Recently I have had an opportunity to elicit information about challenges with existing software we are enhancing faced by larger customers. The representatives of those customers were able to meet me in person so I organized an impromptu meeting with those people and included the product manager so he was aware of what was going on.

Initial issue brought up by the customers sounded like “inability to partition CAD incidents based on certain criteria”.

We had a whiteboard and I was standing in front of it doing my best to figure out what questions I actually need to ask. I could not come up with anything better other than asking them to tell me a story of what is happening today to understand the context in which the problem materializes. After understanding the context better, I focused on getting a better sense of the symptoms i.e. how the users were actually experiencing the problem. Next, I asked the question about the implications of the problem i.e. what is the outcome of the symptoms as they appear. Having got a better sense of that, I was then asking questions about what would it mean to address the problem i.e. what would success look like for those customers.  I also asked about any steps users do today to try and get the issue resolved – any software or manual processes involved.

After some discussion, it turned out we dealt with more than 1 problem. A specific issue that was elicited was that a CAD incident could disappear from the list of incidents visible to the reporter without informing the reporter that the incident was successfully transferred to someone else and was being worked on.  The implications were that the issue was causing fear among reporters of the incidents that the system did not keep track of the incident and hence it was not available for first responders (officers) to see. The implication of that was dispatch of the incident was essentially delayed and whoever needed 911 help might have received it too late. Successful resolution of the problem meant that at no point in time there would be a lost incident that no one in dispatch knew the status on.

My notes from the meeting look quite unstructured even though I was able to capture some of the key statements.

Meeting notes

The definition of the problem that we could then discuss with development team came down to include the following aspects:

  • Who is affected
  • What are the symptoms (scenarios when issue materializes, how often the issue occurs, how the issue is experienced by the user)
  • What are the outcomes (how does this affect the dispatch process)
  • Why is the problem happening – root cause
  • What is success (what would change if the problem goes away and how can that be measured)

This really looks a lot like an aforementioned Robin Goldsmith’s Problem Pyramid even though my notes are far from articulating anything as structured. Item in red is what I have failed to elicit during the face-to-face session with the customers and had to elicit via the phone prior to a meeting with the dev team.

The main difference with the problem pyramid is that the sections for solution options and actual scope of the solution were not actually filled in during the meeting with the customers. That was not done for a number of reasons – a) main goal was to first understand the actual problem b) doing both in a single meeting could well be an overload (similar to when I have tried doing as is and to be cjms in a single meeting).

The problem pyramid after the first meeting with the customers then looked like

Problem pyramid problem completed

To fill in items in red I needed an entirely different meeting where I actually presented the problem and then facilitated a silent brainstorming session with the dev team. In my opinion, it may be difficult and unnecessary to complete both the problem and solution parts together when dealing with a complex problem that is still ill-formed.

Where this tool fails in my opinion as well is capturing information about existing solutions and workarounds people do today because keeping that will be one of the solution options worth considering. Secondly, the problem pyramid is lacking context about the specific profiles of customers experiencing the problem and affected roles/personas. Something I refer to as the story unraveling which sets the context for the problem. That information can be captured however there is no immediate reminder about it in the technique’s skeleton. Nevertheless, it is a great tool that can be used during initial elicitation of the problem with the team or the customers.

Presenting the problem or opportunity using Opportunity Canvas

I have found out a term “opportunity assessment” from SVPG website. I have used that template on a project back in 2013 but can’t say it has helped me much then. Recently I read Jeff Patton’s book “User Story Mapping” that introduced a similar concept “Opportunity Canvas”. The purpose is to define the opportunity we are thinking about pursuing either by starting from the definition of the problem being addressed or the feature/product idea that might have come first before a problem was recognized.

I have used this technique twice in the past month and both times it worked well communicating what is known about a problem or opportunity so far. If we look at the structure of the canvas, we can see it is broken down to 10 sections, two of which are numbered exactly the same (Problems and Solutions). By the way there is an article explaining the opportunity canvas and specifics of using it in real projects that you can review as well. The author was also inspired by Jeff Patton’s book.

  • Problems or solutions (1, 1) – pain points or solutions to certain pain points
  • Users and customers (2) – who is affected by a certain pain point
  • Solutions today (3) – if there are ways to address the pain today, what are those
  • User value / How will users use your solution (4) – what is the value in case the change is made
  • User metrics (5) – what to track to know the impact of the change
  • Adoption strategy (6) – how customers will start using the solution and how they are going to be informed
  • Business problems (7) – what is the value to your business given that a solution is in place
  • Business metrics (8) – how would you measure success from standpoint of your organization
  • Budget (9) – how much money, time and resources will this opportunity require

When a colleague was looking at this tool he asked me the reason for having two sections numbered exactly the same. Frankly speaking, I don’t recall the official rationale for this but my thinking goes back to the book where Jeff mentions that often we are presented not with problems but with solutions that we then have to analyze to understand the actual need for the solution. In this case, the starting point for the opportunity canvas can either be the problem or the envisioned solution proposed by one of the stakeholders.

Both times I used this technique I used it posthumously (after the actual conversation). This is of course not the best and effective strategy since the initial conversation that does not follow the format can easily miss some of the key elements that you have to assemble later to complete the canvas. That makes the whole process relatively ineffective. Nevertheless, even using the tool after the initial conversation but before any go-no go decisions are made still has merit.

What feels better comparing to the Problem Pyramid is that not only there is a place to mention the user and/or customer, but also that besides customer the business is kept in mind. Also, there is a place for a budget that can be used as a restraining mechanism setting clear boundaries for the cost of the solution. Lastly, the tool allows to specify existing workarounds or solutions in place allowing to assess the priority of the issue being looked at.

What I still lack even with Opportunity Canvas is the context element to describe in which context the problem actually occurs. I have a feeling that both customer section and adoption strategy could be used for that purpose however it is not intuitive. I also lack ability to describe scenarios in which the problem occurs. Finally, it really is unclear how to handle a case with a few related symptoms that may point to the same or multiple problems. Should an opportunity canvas be put together for each of those problems or a single canvas can be used to aggregate many similar problems into one.

To me it means that preliminary work needs to be done to dissect some of the sections into more specific parts. It is important to do a draft runthrough on your own, putting together everything you know in a canvas-like template. That will help to identify questions you definitely will need to focus on during the canvas workshop.

opp canvas completed


Here is my interpretation of how Opportunity Canvas sections can correlate to a Problem Pyramid.

Comparing oopc and pp

Overall, opportunity assessment feels more like a tool to be used with a group of internal stakeholders (at least parts of it) when you already have a certain grasp of the problem to be addressed. At the same time, parts of it can be used together with problem pyramid when you are just collecting information about the problem you want to find solutions for. Also, the exercise of filling both problem pyramid and opportunity canvas can be split into 2 or even 3 parts: problem definition from customer’s perspective, problem definition from perspective of the business and solution definition if you don’t have the right audience for all parts at the same time. The problem pyramid can be enhanced with some of the elements of the opportunity canvas like Business challenges and Business Benefits to include the point of view of the business stakeholders onto the problem.

The resulting template for the customer/user facing problem definition would then include all of the items I mentioned above

  • Who is affected
  • What are the symptoms (scenarios when issue materializes, how often the issue occurs, how the issue is experienced by the user)
  • What are the outcomes today (how does this affect the dispatch process)
  • Why is the problem happening – root cause
  • What is success (what would change if the problem goes away and how can that be measured)
  • Solutions today  – existing workarounds

as well as the Solutions today section. In this case we should have enough information to put together a well-formed problem statement from customer’s point of view and later extend it with business perspective and then with solution options.

I look forward to trying this version of the tool and sharing what happens


Writing “Confessions of a dangerous mind”

I was thinking about “Confessions of a dangerous mind” with Sam Rockwell when writing the title. Truth is the post has nothing to do with the movie, so everyone interested in the movie – feel free to skip the article or search for better places.

Here is a copy of a poster for the movie fans not to feel bad. This was just to get your attention

sam rock

This year I was involved as a co-organizer of a Product Stream at IT Arena Lviv conference 2015 ( Prior to this I have organized just one event – Eleks Battle of the bands in 2013 – so I can’t say I have a wealth of experience in the area. My main responsibilities this year were to guarantee quality content i.e. find speakers and ensure they present at the event. I have found a few interesting folks (according to feedback I’ve heard) but now I realize how far the initial plan was from the actual execution. This post is a braindump of things I wanted to share after the conference. Some of ideas require a lot more elaboration and deserve separate posts. If you have enough patience to go through this – then maybe you can spend even more time and share your ideas and experience in event management via comments to the post

  • Design of the event
    • Waterfall is still alive. Almost any event is a great example of waterfall. This is particularly valid for conferences. This means that all phases preceding to the actual execution take 99% of time and therefore need to be thought through for the execution to go well
    • It is essential to define goals of the event. I knew goals of Eleks as the company I represented as a co-organizer of product stream but I did not know goals of the rest of key organizers. This made me narrow-minded in understanding the attendees of other streams
  • People buy tickets to the conference and not a certain stream
    • Attendees don’t distinguish Product stream or Business stream or whatever stream as different parts of the conference. For them this is a single experience for which they have paid money and they expect the same level of quality and experience at any speech of any stream they attend. They don’t know the conference streams may be organized by different people or teams or they don’t care about that. They care about 3 days they spend at the conference
    • This means there has to be consistent work among stream leaders to help one another with speakers, content, promotion and any questions that arise as the conference is a shared responsibility. Failure of one stream becomes a failure of the entire conference
  • Guidelines for presentations – how to define a high-quality presentation
    • What should be inside content-wise
      • Takeaways
      • Learning outcomes
      • Text + image combo or text limits
    • Font size
  • Presentation censorship
    • Should not be 1 person making a decision about speakers
    • Should be a group of 2 or 3 that review on a regular basis to ensure unbiased decision
    • Keep track of all speakers who have already participated
    • Introduce a mechanism for rating speakers to decide whom to invite
      • Past experience
      • Topic
      • Presentation skills
      • Past ratings
      • Personal knowledge of the speaker
      • Travel cost
      • Cost of stay
    • Work with speakers – iterative approach
      • Central idea
      • Main takeaways/conclusions
      • Slide titles
      • Actual meat
      • Visuals
      • Presentation
      • Define timeframe for each stage – should be completed well enough before conference
      • Train speakers to present or check their presentation skills / if a speaker doesn’t agree – let them go and don’t let them participate
    • Substitute speakers
      • High risk that should be considered
      • May be needed to find 2-4 substitute speakers
      • Should be given some form of presenting at the conference not to feel second best
    • Workshops
      • People need to be motivated to come to workshops
      • Workshops should be paid separately
      • Integrated environment – buy conference ticket, workshop ticket or combo
      • Prepare real cases for audience
      • Collect audience expectations
      • Promote workshops through opinion-makers
    • Work with top level management
      • People at the top are often there for a reason and could be strong proponents of ideas so they should at least be made aware of opportunities. From there they can figure out who to involve
      • Senior people should be kept in the loop from the very beginning as that helps to generate even more ideas
    • Food catering
      • Lines like they are don’t always work. Lines helped Tom Gilb to start speaking to Alex Skrypnyk but that could be a rare exception
      • Need to think of ways to get speakers holding the floor right after lunch to eat quickly
    • Inviting speakers
      • A formal letter is not really important. Time is money. It is more important to check speaker’s availability on time than to spend a couple weeks preparing an official letter to find out it is already too late to invite the speaker
      • Timeliness of the invitation is key (6-8 months in advance)
      • Graphics is important – visual part of the invitation should be there
    • Keynote speakers
      • Keynote should be someone known by everyone regardless of age. That person should be the candy everyone is looking for
      • Keynotes deserve to be paid
      • Keynotes should make the people want to attend everything at the conference
      • Keynotes should be fun
    • Conference language
      • There should be one language of the conference. Speakers should be checked against that language. Questionable statement
    • Conference timing
      • There should be breaks between presentations to give people enough time (5-10 minutes)
      • Don’t put a keynote for 9am on days after registration or promote the keynote
      • The number of presentations in one day should probably not be higher than 8 (considering 40 minute long or so)
    • App improvements
      • Engage attendees more into conference by introducing more features than attendify app (rebranded for it arena) has
      • Local Bas and UX experts can be involved into design and development of those extra features i.e. thing feature through and mock it up
        • Reminder to rate speaker at the end of the speech
        • Other ideas tbd
      • People should start using the app a lot earlier than the conference starts
    • Screens and places
      • If there are tv screens available they all can be used to show conference speakers or presentations in real time not like this time where a bunch of screens was showing a static image for 3 days of the conference
      • Adjustable screens or better / larger screens
      • Multiple working screens in rooms with columns
    • Speaker companions
      • There should be someone responsible for either a foreign speaker or a keynote speaker and helping them get in touch with key people from different companies
      • There should be a person who is able to lead networking and serve as the connector between speakers and different executives/key people present at the conference
    • Talk to other related conference leaders and get them onboard to promote the event
      • Not direct competitors i.e. not happening around the same time
      • In this case, invite leaders of conferences like UXPoland, Agilia to get advice from them before and after the event
    • Conference experts
      • In case the organizers (or one of them) is not a professional organizer, there should be a 3rd party consultant involved to help with pressing issues)
      • This will help to ensure many of items outlined above are at least known and considered
      • This is what of one of the speakers mentioned to me in a dialog. “Let’s say you have google, internet, scalpel and you need to do a surgery on a heart of a friend. What will you do? Call a professional or rely on youtube videos?” and this makes sense as there are professionals who have already done quite a few things to get a knack on what needs to be done to make a conference successful

Business Analysis Work Planning Mistakes

In this article the author investigates most common mistakes of a business analyst during product/project analysis phase

Main mistakes include:

  1. Lack of domain glossary needed for the project (half checkmark)
  2. Poor expectations management from analysis phase – not asking customer regarding what they expect to get as a result of analysis
  3. Not agree on templates/format in which analysis results are delivered (Checkmark)
  4. Not try to elicit and outline customer’s risks
  5. Don’t try to reuse past experience  – very challenging to do in reality
  6. Try to follow the process without understanding what to do
  7. Lack of clarity regarding what to do next (half checkmark). A lot of information was requested but it was not properly analyzed in fact

Conducting my first BA+UX workshop

The idea of the event was to investigate an internal process of business travel management and to identify existing pains and try to address them using

  • combination of business analysis and UX techniques
  • a lot of clever heads
  • prototype a possible solution

This is more of a followup to make sure the important items are not forgotten

What I did good:

  • game for group dynamics and to set the mood (marshmallow challenge with spaghetti)
  • eliciting expectations from everyone
  • getting everyone involved and active (I am particularly happy that some of the participants who I’ve not seen very active in the past felt involved and contributed)
  • got peopple interested – in the end a part of the group (3 people out of the original 6) were interested in further work and stayed after the expected closing time (6pm)
  • managed expectations more or less, 50-75% expectations were reached for a part of group (2 out of 3, 3 out of 4)
  • logistics (supplies, food, materials) – as there was a dedicated team member

What I did poorly:

  • i did not organize the knowledge exchange prior to the meeting
  • interviews were not monitored to ensure the questions were valid and to the point
  • facilitation during the training session was not quite there and there were a lot of distractions
  • managing scope of the event. the overall plan was too ambitious
  • defining the scope of responsibilities
    • there was a huge gap here as the other person who co-organized the event felt the overall idea has changed and everything should have been done differently
    • in practice very few of the original objectives have been reached need specifics here

general observations

  • this was not a workshop but more of an initiative/project ->format should have been different with the current team
    • one team doing everything end-to-end
    • many teams doing different projects
    • not one team working out using parts of the process
  • still, there are people who are interested in getting this done
  • i should have prepared better in terms of BA work
    • NOT all stakeholders have been identified
    • introduction on the matter should have been better
    • needed to focus on a smaller piece that could have been swallowed in one take
    • need to learn BPMN and EPC better to be able to quickly use those notation models
  • i need to build my team to do things like this better

Quick action list after Lviv IT Arena 2014

After the conference is over and I still remember the books/materials I should be reading soon, I’d better list them here

In terms of prioritizing the list, I think it would be as follows:

  • UML in 24h (Joseph Schmuller)
  • The mythical man month
  • The writer’s journey

then the rest