'I Solved my Problem' button is ambiguous

This is a serious usability short-comings.

We should either find a label that is more clear or find a way that users can correct that mistakes.

Kiko's suggestions:

- I Solved my own problem

- I Solved My Problem On My Own

- I Solved the Problem Myself

I just tested this locally, and concluded that there is no obvious way to thank someone while also marking their answer as the best. People will naturally think it important to thank whoever answered the question, and they'll use the comment form to do that, at which point "I Solved [M]y Problem" is the most accurate button available.

After forming this hypothesis I looked at the examples, and sure enough, in all except one of them the questioner is thanking a previous commenter.

So tweaking "I Solved my Problem" to decrease the number of people clicking it probably wouldn't increase the number of satisfied people clicking "This Solved [M]y Problem". It might increase the number of satisfied people clicking "I'm Providing More Information", but that's still not what we want.

I think this problem would be solved by the interface I suggested originally:
* have a "Mark as best answer" checkbox, not a button, in each comment (except for comments made by the questioner)
* make the form buttons "Problem Solved" and "I Still Need an Answer"
* make either of those buttons submit any best-answer markings, as well as submitting the comment.

As I said at the time, this would mean that sometimes the best answer wouldn't be marked as such, but it's better to have no marking than a bad marking.

I would like to point out that if users click on the link included in the notification related to that answer, that page provide a text box where they can thanks the user. But I guess, that users don't use that link.

To implement Martin's plan, we need to update:

5 doc tests
2 pagestests
1 browser view (most of the work)
1 template

We need to add a test for the answer comment in pagetests. Comments are already supported for the answer in the code.

I think I can do this this week with the momentum I have right now.

Francis argues that the primary problem is that users are not seeing the 'This Solved my Problem button'. The secondary problem are the users who choose the 'I Solved my Problem' button because it has a comment field.

So our options are:

a) Change the button label (easy, low-cost, may not work).

b) Submit the comment box along the 'This solved my problem' button (complex, not obvious that the box is linked to these buttons).

c) Add a comment box to the 'This solved my problem' button:
    a) Display everything (complex, minor cost, interrupts reading).
    b) restrict the comment field to one line, expand it to 15 lines onfocus (complex, medium-cost, awkward).
    c) display the comment field only when the mouse is within the borders of the comment box. (complex, medium-cost, awkward/annoying).

d) Replace the 'This solved my problem' with radio buttons, Add 'Someone Solved my Problem' below the comment. (complex, high-cost, who is default?).

e) mpt's suggestion above (complex, high-cost, probably best UI).

Running the query https://devpad.canonical.com/~andrew/paste/fileMmcn0B.html on yesterday's database, I found out those very interesting numbers:

Out of 1262 questions that had a confirmed answer:
- 384 were confirmed using the 'This Solved my Problem' button (30%)
- 670 confirmed their own answer (53%)
- 208 used the link from the notification email to confirm the answer (16%)

So, we can say that nearly 50% (since some of the self-answers are legitimate) of users misuse/misunderstand the 'Best answer' feature. I can see two likely explanations for those numbers:

a) Users don't see/understand the 'This Solved my Problem' button.
b) Users know exactly what they are doing, and use the 'I Solved my Problem' hoping to get a karma bonus.

I think a) is a likely explanation because when users gets back to the question page, they want to provide feedback for the answer. So according to the 'do-it-now' principle, they'll jump right to the Message box because that's where they expect to provide the feedback.

If this is true, I don't think the checkbox design will be of any use, we'll just have as much self-answer by mistake (many people won't see the checkboxes, like they don't see the existing button.)

The question I ask myself is how come people use the main question link to provide feedback on the answer and do not not use the confirmation URL in the notification message. Is it because the first one is at the bottom of the email? Or is it because, they use the web interface to browse to the question (using the 'Created' or 'Need attention' report)?

Maybe we should just remove the 'Best answer' selection and revert to automatic heuristic like we had before
 (use latest ANSWER), which altough not always correct, was probably correct more often that the current situation.

Or we could allow rating individual answer. For example, in Jive, users can select two 'helpful' messages in addition to the 'this message solved my problem' award. The confirmation message would not select the best answer, this would be done separately.

Let's just change the button label for 1.0. I had provided some different wordings: "I Solved My Own Problem" or "I Solved The Problem Myself"; better wordings are of course welcome.

I don't think people avoid "This Solved My Problem" because "they don't see the ... button", I think they do see it but avoid it because it seems not to allow them to thank the person. And again, I think tweaking the "I Solved My Problem" label would have little effect, other than increasing mistaken "I'm Providing More Information" clicks.

So here's what I suggest.
* For 1.0, change the buttons to "Problem Solved" and "I Still Need An Answer", and remove all UI for marking a best answer. Don't try to guess it, either (that would be aggravating when the guess was wrong); give equal karma for all comments.
* After 1.0, introduce the checkboxes.

This approach has the benefit that there would have been only two sets of button labels over time: the set we have now, and the 1.0 set. The only difference between the 1.0 interface and the post-1.0 interface would be the presence of the checkboxes.

Matthew,
I really doubt that people don't use the 'This Solved My Problem' button because there is no obvious way to thank the user. I really think its a matter of not "seeing" (for your varying definition of see): the button doesn't really contrast with the background, it's not in a place that is expected for the task at end, etc.

Of course, without any empirical evidence (eye tracking, observation data, or even user interviews) your opinion is just as good as mine.

In the lack of evidence for one or the other theory though, I think it would be a little drastic to remove all the functionality related to the 'Best answer' now. An unfortunate side effect of that solution would be to introduce some inconsistencies in the data model. Now, all questions in the SOLVED status have their answerer and answer attribute set. It wouldn't be the case after the change.

I think a more conservative change, allowing us to gather some more data on the reason behind that behaviour before choosing the final solution is best for now.

We should first change the label to a very long and verbose, but very explicit 'I Solved the Problem on My Own'. As an additional safe-guard, we should add a message near the buttons like: "To confirm an answer, use the 'This Solved My Problem" button located at the bottom of the answer.'

We should revisit this issue two weeks after the change to measure if this change had an impact on the ratio of the different answers kind. We should probably also attempt to contact some of the users that didn't succeed in using the UI properly to try to understand what is the source of their mistake. We should take care in the way we formulate our question to minimize rationalization effect (see http://www.uxmatters.com/MT/archives/000183.php for some tips).

I think Francis' approach is worth a shot.

"...[T]hat solution would ... introduce some inconsistencies in the data model. Now, all questions in the SOLVED status have their answerer and answer attribute set. It wouldn't be the case after the change."

But you're going to need to change that *anyway*, sooner or later. Sometimes a problem goes away by itself; it's Solved, but nobody solved it. Other times, one person provides half the answer and another person provides the rest; the problem is Solved, but not by just one person. A UI that denies either of these possibilities will cause bad data and meta-noise until it's fixed, regardless of whether you manage to minimize other problems. That's the rationale for my proposal: fixing this problem in the short term, reducing work for you in the long term, and reducing relearning for repeat users. But I don't care that strongly about it, so I'll shut up now. :-)

Partial fix committed in RF 4164. The wording was revised and a note added to the bottom of the form. I'll keep this issue 'In Progress' until we evaluate how the patch changes user behavior. In the unlikely case that the problem becomes negligible, we can close it. I expect that we will want to revise the form's behaviour after we understand how users are reading it.

partial fix released on 2007-04-30 in version 4080.

In the absence of new evidence. I'm closing this since it ware released 3 weeks ago. If this is not correct, let's get some concrete reasons for why we have a problem.

Coincidentally, just before reading that comment I came across an example of the problem still happening. <https://answers.launchpad.net/launchpad/+question/6828>

I ran a similar query than before and it seems that the label change did improve thing somewhat. But I think we are still having a problem:

Out of 398 questions that came out since the fix was rolled out to production, there were:
- 193 were confirmed using the 'This Solved my Problem' button (48% vs 30% before the fix)
- 138 confirmed their own answer (34% vs 53% before the fix)
- 67 used the link from the notification email to confirm the answer (16% same as before)

So the label change had an impact on the use of the button, but I think 34% of self-answers is still high. From the initial exploration, 3 out of 26 solved questions (~11%) were real self-answers.

I think it's time to give serious considerations to MPT suggestion :-)

Have we seen evidence of this being marked incorrectly for Launchpad answers? It may be irrelevant, but I haven't a single recollection of that happening for answers I've looked at (and I get mailed all of them).

Francis, excellent use of statistics! (And I'm not saying that just because they supported my prediction, honest.:-)

Kiko, since the relabelling 25 "questions" have been asked about Launchpad projects, of which 2 were incorrectly marked by the questioner as being solved by themselves: question 7019, and question 6828 (which you answered). That's an 8% error rate, though obviously with a very wide confidence interval.

I can think of two reasons for Launchpad's 8% being lower than the average 34%. First, people dedicated enough to ask questions about how to use Launchpad -- rather than just not using it -- are likely those who have already adopted Launchpad, which means they're already good at dealing with non-obvious UI. Second, 8 of those 25 "questions" were actually requests for account removal, and once your account is removed you can't mark *anyone's* response as best.

Hi Martin. I'm putting thought to your 2007-04-23 remarks. I'm going to put in effort to solve this issue for good.

> Matthew Paul Thomas said on 2007-04-23:
> * make either of those buttons submit any best-answer markings, as well as submitting the comment.
Do you want this behaviour in case the user chooses the wrong button?

When a user chooses the 'Problem Solved' and there are no best answers marked, should I confirm this is what the user wants:

    You have not selected any best answers.
    If one or more answers were helpful in solving your problem
    please select them and choose 'Problem Solved'
    [ I will select some answers ] [ I solved my problem on my own ]

I feel this proposal is.. awkward. The existing UI has limitations (misleading for the solved my own problem case, only one best answer) but it's pretty simple and elegant. I'd want to make sure that the new UI is only slightly less so.

"When a user chooses the 'Problem Solved' and there are no best answers marked, should I confirm this is what the user wants"

No, please don't.

Kiko, before and after:
* "( This Solved My Problem )" becomes "[ ] Mark as best answer"
* "( I Solved the Problem on My Own )" becomes "( Problem Solved )"
* "( I'm Still Having This Problem )" becomes "( I Still Need an Answer )"

So the extra complexity in the database (multiple best answers) will actually make the UI slightly more simple, not less.

Matthew, I don't think anybody is going to see that checkbox. How can we cope with that? Currently the button is /very/ visible and I'm reluctant on giving that up (and creating a bigger problem in the process).

Kiko, I just checked the last 40 "Solved" Ubuntu questions. Of those 40, in 13 of them the questioner had incorrectly marked their own answer as best. It doesn't matter how visible the button is, as long as one third of best-answer markings are wrong, the best-answer system is effectively useless.

You're welcome to approve or to veto my proposal, but we've been discussing it on-and-off for eleven months now, so please forgive me for losing interest. Once it is implemented and the results measured, I will be glad to discuss any need for further tweaks.

Dare I say it, I'm undertaking this issue. I'm going to inch my way to Matthew's proposal. If UI gets awkward, I'll call for assistance.

The branch that addresses this is in review. With input from Matthew, Francis, and Kiko, I have implemented a compromise between Matthew's suggestion and Kikos.

The buttons are labeled as Matthew suggests:
* "( This Solved My Problem )" becomes "[ ] Mark as best answer"
* "( I Solved the Problem on My Own )" becomes "( Problem Solved )"
* "( I'm Still Having This Problem )" becomes "( I Still Need an Answer )"

The workflow now permits the question owner to confirm a best answer /after/ the question is in the SOLVED state. The questions owner can add a comment about the solution and choose the 'Problem Solved' button. A notification message suggests that the user choose a best answer if one is applicable. No karma points are awarded to the question owner for providing a self-solution ('Problem Solved ').

Switching the best answer button to a checkbox that permits the question owner choose multiple best answers was not done because it needs a spec to define how the karma will work.

We must revise https://help.launchpad.net/GettingSupportInLaunchpad to reflect the workflow change when this fix is released.

That's a good start. I have a couple of questions:

1. "The workflow now permits the question owner to confirm a best answer /after/ the question is in the SOLVED state." Why only after? If you have to mark the question as solved, and then mark a best answer separately, I predict that that will reduce the number of best-answer markings by at least 60%. (If anyone then tries to blame the reduction on using checkboxes rather than buttons, rather than on the extra page loads required, I will not be amused.)

2. "Switching the best answer button to a checkbox that permits the question owner choose multiple best answers was not done..." So how are you using checkboxes at all? What happens if the questioner checks more than one checkbox?

Sorry. I must clarify.

1. Confirms continue to happen /before/ a question reaches the solved state. Confirms may now may happen /after/ the question reaches the solved state. The workflow provides more flexibility to the user to confirm the answer. The assumption is that more answers will be confirmed when we allow the question owner to set the state to solved (and leave a message), then we followup with a notification about choosing a Best Answer if one is appropriate.

2. The [ Mark as best answer ] button is still a submit button. You cannot choose more than one. Questions that have many partial answers, or multiple correct answer are mired is ambiguity. I think this still needs to be solved, and spec may be required.

Fixed in RF 4532.

Curtis,

On staging the button now reads "Problem Solved" which is good but the text below reads:
"To confirm an answer, use the 'This Solved My Problem' button located at the bottom of the answer." which will never appear if you solved your own problem.

Perhaps that needs to be rewritten since it's confusing until you see the button appear next to someone's answer. My vote is to remove it entirely. If the buttons are obvious then we should rework the button.

If the buttons are NOT obvious then we should rework the button.

The reason i call this out is that the phrase is highlighted quite a bit more now with the font changes.

Opps. I will fix that text. The instruction should match the UI change.

Joey,

The message is about "confirming" an answer given by someone else. This problem was reported as bug 126432.

This is fixed in my 107810-this_solved_my_problem branch. Joey, I'm glad you caught this because you were actually looking at the wrong button, which was part of the ambiguity--This Solved My Problem -> Mark as best answer. You are looking at the button that is causing so much consternation.

Fix released in Launchpad 1.1.7.