User Bug Tracking
(This might be related to SCS_ISSUE_TRACKER_MAINTENANCE_R3 but I assume this will be a later revision; right now that task focuses on how developers work with issues, and this is mostly about how users deal with issues.)
My original document
This is basically an idea for how user bug tracking could work for Sophie 2, based on what happened with Sophie 1.
History
With Sophie 1, we were using Mantis for bug tracking; for a while we told users to post their own bug reports. This didn't work very well because there were a lot of categories that users didn't understand (bug severity, for example; what developers thought of as a "crash" was different from what users thought of as a "crash"). We ended up with a lot of bug reports that weren't very useful; there wasn't enough information for us to do anything about, and there was often no way of tracking down the user who'd filed them to get something better. A lot of the reason this didn't work very well was because we were counting on the user to copy and paste the crash log - which wasn't very easy to find in the Sophie app - into the bug report; Sophie overwrote this frequently, and if the user didn't get it immediately, there would be no way to get it.
At a certain point, a crash window was introduced inside the program; this automatically copied the crash log, and popped up a window asking the user for a name, email address, and some information about what happened when there had been a crash. When Okay was clicked, this was emailed to one of the developers, who would go through the reports and put them on Mantis. This worked a little bit better, but there were still problems: there was nothing to force the user to enter their name or email, so a huge percentage of these reports ended up being anonymous. (It didn't help that Sophie was crashing often enough that users just wanted to close the crash window and go on with whatever they were doing.) We could sometimes track down who was having problems because of directory information from the crash report (i.e., my Sophie.app was at /Users/Dan Visel/Desktop/Sophie.app) but we still got a lot of useless reports. Instructions on debugging on the website helped a little bit, but not so much. Another problem was that frequently we would need the crashed book to see the bug: we could sometimes, but not always, get the user to email the book, but often books were enormous, which made this a problem.
Suggestions
So: what I think we should be doing with Sophie 2.
First: we need a form inside Sophie to submit bugs; this would email us crash reports & a user-submitted description of the crash report when a crash happens. Users selecting Help > Report Bug would also be able to submit a bug in this way without having a crash. Ideally this would have some way of knowing the user's email address (for pre-release versions of Sophie); we'd need to get an email address the first time the user starts a pre-release version of Sophie. (Maybe this could just be kept in a preference file from the first time the user submits a bug - in Sophie 1, the user had to re-enter their name and email every time there was a crash, and people quickly got lazy. I don't know why we couldn't have remembered it.)
These bugs would be sent to a triage email address; triage would sort them, correspond if necessary, and enter them into the actual bug-tracking system on Trac. Triage wouldn't be done by a developer - triage would be done by someone like me, who can tell what's serious and what's not serious. The purpose of this is so that we don't unnecessarily waste developers' time dealing with users instead of dealing with bugs.
An internal form isn't the only way that we should deal with bugs; we also need to have a section of the user website where users can submit bugs. This is required as well as the internal system because there may be bugs which crash the application and the bug-submitting form entirely; the downside of this is that users may not be assiduous in attaching crash reports & we might need to correspond with them to get that. This method will require someone to do triage in exactly the same way; the form will email an address, and someone will sort the bugs, correspond with the users, and enter bugs in Trac. If the developers need more information about a bug in Trac, they could ask the triage person to get it from the user.
A second method: for more complicated problems where a response is desired, users should be encouraged to post in the forums at the dev site ( http://sophie2.org/trac/discussion/0 ); a specific area for bug reporting should be answered. This will work for more complicated bugs - I'm getting a crash, but I can't tell exactly what's making it happen. The benefits of this would be that submitters could take care of some of the triage by looking for previous forum topics that seem related. This method, however, would require someone to monitor the forum & do triage; for best results, we'd have to have the forums more integrated into the engineer's process than they currently are. Right now we have an RSS feed for the entire forum (need to check this); it would be helpful to have an RSS/email feed for just the bug-tracking forum (so the bug-fixing engineers could stay on top of this) and to have individual RSS/email feeds for particular threads in the forums (so users could stay on top of their bugs being fixed).
And a third method: one of the things that's worked well this go-round while working remotely is to have a persistent group Skype chat - if I'm having a problem I can go there & leave a message. More often than not, somebody is up and will respond; if not, I can usually leave the window open in the background & wait until something happens. We could do something like this for the public who'd like to work through bugs: give the address for a group Skype chat, which the bug-fixing engineers would also be members of. A benefit of this from our end is that Skype is already integrated into people's workflow; it wouldn't be hard to keep this open to monitor for bugs. The primary benefit of this is that users will feel like they are being attended to; I think this will translate into good will towards the project. Downside: might be a little complicated telling end users how to get into the Skype chat - need to look into this.
(A related idea: a lot of libraries now have instant messenger reference help, where you IM the library with questions. Something like this could be done - presumably the user could be IMing a group of respondents rather than a single person - though the Skype idea seems more appealing because it's already in use. AIM/GoogleTalk is probably easier for the enduser, but there might be some chaff; Skype also seems to keep a record of conversations better than AIM/GoogleTalk does.)
All of these things are related mostly to how the user sees bug-reporting; these shouldn't impact the way that the engineers are using the bug-reporting system. I think we'll be able to deal with bugs more efficiently if users are polluting Trac; I don't think we should hide it from them, but we should give them friendlier ways of dealing with bugs. It's worth noting that we will certainly need to put some effort into keeping the main bug-tracking database clean: especially when we have so many engineers working on the project, there are going to be a lot of duplicate reports, some malformed entries that will need to be cleaned up, etc. Probably worth assigning someone to this job so that someone's responsible for it; in Sophie 1, people would periodically clean out the bug-tracking database, but there was a lot of deep cleaning that was never done - bugs from pre-alpha that couldn't be replicated which were kept around because no one closed them.
All of these suggestions assume that we have someone - who could be me - who is a middleman between the users and the developers, converting user reports into properly formed bug reports by corresponding with the users. This won't be a very intensive job at first; it will probably become more intensive as work goes on, probably becoming approximately full time work as we get into the beta. It would be good to maintain a spreadsheet/database of people that we've talked to about bug fixing so we have some sense of what the users are trying to do & their past problems; this will make the project seem friendlier and more responsive to people outside.
Does this make sense? I figure it's best to have a jump on this now (before we need to) rather than when it actually hits us & we're surprised by it.
Boyan's comments
Here are my comments on bug tracking (and support in general). I'll try to be short in order not to bother you:
- The user should have the option to disable automatic bug reporting. Some users don't like to see an error report every time something bad happens.
- The user's email should be kept so that it doesn't have to be entered every time. We should not allow anonymous reports.
- The bug report form should be really simple.
- While providing ability to enter trac tickets, it is probably a good idea to have a simple form for bug reporting on the site as well. Regular users can be scared by the New Ticket form.
- We should provide anti-bot (anti-spam) filters in Trac. We already have some tickets added by bots.
- It is a good idea to have a Support discussion forum. However, the current forum should be improved a lot - it is now practically unusable.
- Some kind of live chat also seems reasonable, no matter whether we use Skype or not. However, Skype public chats are probably a good solution.
- We might want to provide the good old support@… mail for problems. I'm not sure whether this is actually a good idea.
Nick's concerns
Nick says: we can't put up a disclaimer saying that by downloading Sophie you're consenting to have your bugs tracked.
Synthesis
Areas where we're all in agreement
Areas that need to be worked out
Comments
(leave comments here)
