Tag Archives: Communication

Testers Cannot Be Perfect Proxies for End Users

The following is a snippet of a Skype conversation I was a part of recently. I wanted to share it, since I think it’s a perfect example of a perennial tester conversation. See if you recognize yourself in it. The participants’ names have been removed in the interest of privacy. Other minor changes were made for the sake of clarity.

[11:25:37 AM] Dev Mgr: Programmer– i really need you to look into that “flash wont resize without a reload” fact
[11:25:42 AM] Dev Mgr: its not being believed [by the CEO]
[11:25:50 AM] Dev Mgr: and quite honestly
[11:26:02 AM] Dev Mgr: i went to some other pages that clearly have graphs on them
[11:26:07 AM] Dev Mgr: that dont need to be reloaded
[11:26:29 AM] Dev Mgr: when they get resized
[11:26:29 AM] Programmer: i’ll read fusionchart docs
[11:26:32 AM] Dev Mgr: thats
[11:26:33 AM] Dev Mgr: thanks
[11:26:40 AM] Dev Mgr: also Tester 1/Tester 2
[11:26:49 AM] Dev Mgr: i’m not really sure how that was not part of a QA
[11:27:08 AM] Dev Mgr: when you are looking for UI stuff before a release
[11:27:16 AM] Dev Mgr: not everyone has a big ass monitor
[11:27:25 AM] Dev Mgr: please add qa on your laptop screen to the test
[11:27:49 AM] Dev Mgr: this is not a successful launch
[11:28:18 AM] Tester 1: I agree that it should be added as a regular part of our tests–NOW that we know it’s importantl.
[11:28:34 AM] Tester 1: There are any number of things that one could complain about with these pages.
[11:28:59 AM] Tester 1: I strongly suspect that, had Tester 2 or I complained about this to Programmer prior to the launch we would have been seen as horrible nit pickers.
[11:29:16 AM] Dev Mgr: What??????????????
[11:29:26 AM] Tester 1: “Why are you complaining about something so trivial when we’re trying to get this out?”
[11:29:28 AM] Dev Mgr: how can the UI not be improtant
[11:29:34 AM] Tester 1: I’m not saying it’s not.
[11:29:43 AM] Tester 1: I’m saying that there are MANY things that one could pick to complain about.
[11:29:49 AM] Tester 1: The colors are one.
[11:29:58 AM] Tester 1: The fact that many of the graphs don’t have proper labels is another.
[11:30:22 AM] Tester 1: I could come up with more.
[11:31:19 AM] Tester 1: Unfortunately, prior to today, we weren’t aware that resizing would be the important thing.
[11:32:46 AM] Programmer: i agree with Tester 1, on a scale of 1 to 10, resizing is low on the totem pole since there is an almost obvious solution vs. the colors
[11:33:09 AM] Dev Mgr: i strongly disagree
[11:34:38 AM] Tester 1: Dev Mgr, could you have predicted that CEO would be up-in-arms about resizing?
[11:34:53 AM] Tester 2: This was my responsibility, and I will take the hit for it, that’s fine. That being said, however, considering that there were over a dozen pieces to this [specification], and we, as a team, fixed things that were given to us incorrectly in the 1040 from CEO, I would consider this issue is not the barometer or whether or not this was a successful launch.
[11:34:53 AM] Dev Mgr: this isn’t about CEO
[11:35:02 AM] Dev Mgr: i went to BBW on my laptop
[11:35:06 AM] Dev Mgr: and i had to resize it
[11:35:11 AM] Dev Mgr: and it happened to me
[11:36:19 AM] Tester 1: No doubt. But the fact that it happened is distinct from whether or not it is an important enough issue to have said “Let’s hold off on release until this is resolved.”
[11:36:37 AM] Dev Mgr: i would not have released that
[11:36:51 AM] Dev Mgr: USER EXPERIENCE
[11:36:55 AM] Tester 1: I’m very glad that I, Tester 2, and Programmer now know this.
[11:37:05 AM] Dev Mgr: this makes us look bad
[11:37:07 AM] Dev Mgr: to others
[11:37:10 AM] Dev Mgr: its a window into us
[11:37:20 AM] Dev Mgr: people/ clients see this page
[11:37:21 AM] Dev Mgr: we’re a tech company
[11:37:28 AM] Dev Mgr: and we can’t get our allignment right?
[11:38:49 AM] Dev Mgr: you didn’t see it and decide whether or not to push it
[11:38:55 AM] Dev Mgr: you said you never tested for it
[11:39:07 AM] Dev Mgr: did you test in IE?
[11:39:26 AM] Tester 2: yes, but did not resize window from full laptop view
[11:39:37 AM] Tester 1: What I said was that I believe that, HAD we tested it, Programmer, Tester 2, and I would’ve all concluded that it wasn’t a big enough issue to stop the launch.
[11:39:59 AM] Dev Mgr: then we have a bigger issue to discuss separately
[11:40:14 AM] Dev Mgr: image is everything
[11:40:19 AM] Dev Mgr: we are part of a public company
[11:40:25 AM] Dev Mgr: this is not an internal tool
[11:40:37 AM] Dev Mgr: our visual output
[11:40:49 AM] Dev Mgr: is more important to some than the guts that go into it
[11:41:52 AM] Tester 1: I wonder if perhaps the deeper lesson learned here is that sometimes we *need* to get buy-in from CEO prior to going live with stuff.
[11:42:18 AM] Dev Mgr: this isnt’ about CEO
[11:42:29 AM] Dev Mgr: other than it sucks that he found it first
[11:42:54 AM] Tester 1: But many other people aside from CEO saw it and didn’t say anything about it being important.
[11:42:56 AM] Dev Mgr: if you have a question about a UI that you can decide you can use me as a gauge
[11:43:18 AM] Tester 1: My fundamental issue is that it’s very hard for us to be certain that we are truly aware of all the important things.
[11:43:20 AM] Dev Mgr: Tester 1…you all say this wasn’t tested
[11:43:25 AM] Dev Mgr: so that’s all that matters
[11:43:38 AM] Dev Mgr: not if you had what you would have done
[11:43:45 AM] Tester 1: Had we tested it and decided it wasn’t important we’d be in the same boat.
[11:44:07 AM] Tester 1: The fact that we didn’t test it probably means that if we had we wouldn’t have concluded it needed to be fixed prior to launch.
[11:44:49 AM] Tester 1: I’m trying to get at the deeper point:
[11:44:58 AM] Tester 1: We won’t always know what is important to the end users.
[11:49:26 AM] Dev Mgr: well if you dont think that this opens up some eyes to what needs to be accpetable to users than i will now have to approve everything with a UI component
[11:49:55 AM] Dev Mgr: and that is not the right answer…might i add
[11:53:17 AM] Tester 1: The best answer I can give you at this point: We have learned as a result of this. Our testing in the future will be better. Unfortunately we aren’t omniscient, and don’t have unlimited amounts of time, so I fear we will make other mistakes, which we will also learn from.

Irreverence Versus Arrogance

Everything sacred is a tie, a fetter.
— Max Stirner

I am an irreverent guy. I’m a fan of South Park and QA Hates You, for example. Furthermore, I think it’s important–nay, essential–for software testers to cultivate a healthy irreverence. Nothing should be beyond question or scrutiny. “Respecting” something as “off limits” (also known as dogmatism) is bound to lead to unexamined assumptions, which in turn can lead to missed bugs and lower quality software. If anything, I think testers should consider themselves akin to the licensed fools of the royal court: Able–and encouraged–to call things as they see them and, especially, to question authority.

Contrast that with arrogance–an attitude often confused with irreverence. The distinction between them may be subtle, but it is key. Irreverence and humility are not mutually exclusive, whereas arrogance involves a feeling of smug superiority; a sense that one is “right.” Arrogance thus contains a healthy dose of dogmatism. The irreverent, on the other hand, are comfortable with the possibility that they’re wrong. They question all beliefs, including their own. The arrogant only question the beliefs of others.

I pride myself (yes, I am being intentionally ironic, here) on knowing this difference. So, it pains me to share the following email with you. It’s an embarrassing example of a moment when I completely failed to keep the distinction in mind. Worse, I had to re-read it several times before I could finally see that my tone was indeed arrogant, not irreverent, as I intended it. I’ll spare you my explanations and rationalizations about how and why this happened (though I have a bunch, believe me!).

The email–reproduced here unmodified except for some re-arranging, to improve clarity–was meant only for the QA team, not the 3rd-party developer of the system. In a comedy of errors and laziness it ended up being sent to them anyway. Sadly, I think its tone ensured that none of the ideas for improvements were implemented.

After you’ve read the email, I invite you to share any thoughts you have about why it crosses the line from irreverence into arrogance. Naked taunts are probably appropriate, too. On the other hand, maybe you’ll want to tell me I’m wrong. It really isn’t arrogant! I won’t hold my breath.

Do you have any stories of your own where you crossed the line and regretted it later?

The user interface for OEP has lots of room for improvement (I’m trying to be kind).

Below are some of my immediate thoughts while looking at the OEP UI for the front page. (I’ll save thoughts on the other pages for later)

1. Why does the Order Reference Number field not allow wildcards? I think it should, especially since ORNs are such long numbers.

2. Why can you not simply click a date and see the orders created on that date? The search requires additional parameters. Why? (Especially if the ORN field doesn’t allow wildcards!)

3. Why, when I click a date in the calendar, does the entire screen refresh, but a search doesn’t actually happen? I have to click the Search button. This is inconsistent with the way the Process Queue drop down works. There, when I select a new queue, it shows me that instantly. I don’t have to click the “Get Orders” button.

5. What does “Contact Name” refer to? When is anyone going to search by “Contact Name”? I don’t even know what a Contact Name is! Is it the patient? Is it the OEP user???

Click for full size

Click for full size

4. In fact, I *never* have to click the Get Orders button. Why is it even there on the screen?

6. Why waste screen space with a “Select” column (with the word “Select” repeated over and over again–this is UGLY) when you could eliminate that column and make the Order Reference number clickable? That would conserve screen space.

7. Why does OEP restrict the display list to only 10 items? It would be better if it allowed longer lists, so that there wouldn’t need to be so much searching around.

8. Why are there “View Notes” links for every item, when most items don’t have any notes associated with them? It seems like the View Notes link should only appear for those records that actually have notes.

9. Same question as above, for “Show History Records”.

10. Also, why is it “Show History Records” instead of just “History”, which would be more elegant, given the width of the column?

11. Speaking of that, why not just have “History” and “Notes” as the column headers, and pleasant icons in those rows where History or Notes exist? That would be much more pleasing to the eye.

Click for full size

Click for full size

12. In the History section, you have a “Record Comment” column and an “Action Performed” column. You’ll notice that there is NEVER a situation where the “Action Performed” column shows any useful information beyond what you can read in the “Record Comment” field. Why include something on the screen if it’s not going to provide useful information to the user?

For example:

Record Comment: Order checked out by user -TSIAdmin-
Action Performed: CheckOut

That is redundant information.

In addition to that, in this example the Record Create User ID field says “TSIAdmin”. That’s more redundant information.

There must be some other useful information that can be put on this screen.

13. Why does the History list restrict the display to only 5 items? Why not 20 items? Why not give the user the option to “display all on one page”?

Click for full size

Click for full size

14. In Notes section of the screen, the column widths seem wrong. The Date and User ID columns are very wide, leaving lots of white space on the screen.


The Blackest of Boxes

At a recent gig I was one of several veteran testers brought in to shore up a team that… I don’t want to say “lacked experience.” Probably the best description is that they were “insular.” They were basically re-inventing the wheel.

I had a number of discussions with the team’s manager and he seemed receptive to my suggestions, up to a point. Past that, though, all I heard was “We don’t do that.” When I first heard this my mind was suddenly struck with visions of Nomad, the robot from Star Trek, whose favorite thing to say was “Non sequitur. Your facts are uncoordinated.”

Don’t worry. While I was tempted to, I didn’t actually say it out loud.

Let me back up a little. The job involved testing a system that is sandwiched between two others–a digital scanner and a database. The system is designed to take the scanned information and process it so that the right data can be updated and stored, with a minimum of human intervention. That last part is key. Essentially most of what happens is supposed to be automatic and invisible to the end user. You trust that checks are in place such that any question about data integrity is flagged and presented to a human.

Testers, however, are not end users. This is a complicated system, with a lot of moving parts (metaphorically speaking). When erroneous data ends up in the database it’s not a simple matter to figure out where the problem lies. When I asked for tools to help the team “look under the hood” (for example, at the original OCR data or the API calls sent to the database) and eliminate much of the overhead involved with some of the tests, I was told “But we are User Acceptance Testing”, as if that were somehow an argument.

Worse, the tool was not being developed in-house. This meant that our interaction with the programmers was essentially non-existant. We’d toss our bugs over a high wall and they’d toss their fixes back over it. The team was actively discouraged from asking questions to programmers directly (the programmers were in another state, so obviously face-to-face communication was not possible, but even emails and phone calls were verboten).

One thing I’ve learned over the years is that it’s never a good idea to separate programmers and testers. It fosters an us-versus-them mentality instead of a healthy one where the testers are viewed as helping the programmers. It lengthens the lifespan of bugs since a programmer doesn’t have the option to walk over to a tester and ask to be shown the issue. Testers are deprived of the opportunity to learn important information about how the system is built–stuff that’s not at all apparent from the user interface, like snippets of code used in several unrelated places. A change to one area could break things elsewhere and the testers might otherwise never think to look there.

So, when I hear “We don’t do that” it strikes me as a cop out. It’s an abdication of responsibility. A non sequitur.