Design and Evaluation of a Command Recommendation System for Software Applications

Wei Li, Justin Matejka, Tovi Grossman, Joseph A. Konstan, George Fitzmaurice

July 2011 · ACM Trans. Comput.-Hum. Interact. (TOCHI)

Abstract

We examine the use of modern recommender system technology to aid command awareness in complex software applications. We first describe our adaptation of traditional recommender system algorithms to meet the unique requirements presented by the domain of software commands. A user study showed that our item-based collaborative filtering algorithm generates 2.1 times as many good suggestions as existing techniques. Motivated by these positive results, we propose a design space framework and its associated algorithms to support both global and contextual recommendations. To evaluate the algorithms, we developed the CommunityCommands plug-in for AutoCAD. This plug-in enabled us to perform a 6-week user study of real-time, within-application command recommendations in actual working environments. We report and visualize command usage behaviors during the study, and discuss how the recommendations affected users behaviors. In particular, we found that the plug-in successfully exposed users to new commands, as unique commands issued significantly increased.

DOI PDF

Figures

Fig. 1. Number of built-in commands in each AutoCAD yearly release.

Fig. 2. Histogram of the number of commands used by 4000 AutoCAD users in 6 months. The largest group of users use only between 31 and 40 commands.

Fig. 3. CommunityCommands system overview.

Fig. 4.Map of Good, Poor, and Unnecessary Recommendations.

Fig. 5. Cumulative percentage of command counts for AutoCAD commands.

Fig. 6.The active user is in red, and her expected frequency table will be compiled from her most similarneighbors (in yellow).

Fig. 7.Simpliﬁed example of user similarity. Alice and Bob are more similar than Bob and Cindy as theangle between their command vectors is smaller.

Fig. 8.k-Tail evaluation of a command sequence.

Fig. 9.Ofﬂine results showing RecallkT ail, the percentage of times the next new command was predicted ina list of 10 by each algorithm.

Fig. 10. Percentage of “good” suggestions by technique. Error bars show standard error.

Fig. 11. Percentage of “poor” suggestions. Error bars show standard error.

Fig. 12.Design space of command recommendation systems.

Fig. 13.Example user history of a typical user showing distinct sessions of activity with breaks in activityin between. Red lines indicate the intensity of user activity (darker red means more activit

Fig. 14.The recommendation process with the two decision points of using user histories broken up intosessions or users, and looking at the command data from the current session or the entire history

Fig. 15. Offline evaluation of 4 recommendation approaches described in Figure 12 (higher percentages are better).

Fig. 16.CommunityCommands palette docked in the bottom right corner beside the command line.

Fig. 17.Components of a suggestion button.

Fig. 18.User interface change between short-term and long-term modes.

Fig. 19.Visualization comparing short-term and-long term recommendations over a 48 hour time period.In the short-term mode the list of commands is changing often to match the user’s context, while in

Fig. 20.Overview visualization showing the command recommendations and interactions of a single userover a two week period.

Fig. 23.User history graph showing two “bursts” of interaction with the recommendation window.

Fig. 24. Number of commands dismissed by each user.

Fig. 25.Average number of previously unobserved commands used each day.

Fig. 26. Average number of new commands used in the last two weeks without, and the first two weeks with, the recommender system.

Fig. 27. Number of new commands actually used compared to the model.

Fig. 28.Average number of new commands used in weeks 5 and 6 split into categories.

Fig. 29. Average number of recommended commands used once and used multiple times for the short-term and long-term algorithms.

Fig. 30.User preference for recommendation mode.

BibTeX

@article{10.1145/1970378.1970380,
 abstract = {We examine the use of modern recommender system technology to aid command awareness in complex software applications. We first describe our adaptation of traditional recommender system algorithms to meet the unique requirements presented by the domain of software commands. A user study showed that our item-based collaborative filtering algorithm generates 2.1 times as many good suggestions as existing techniques. Motivated by these positive results, we propose a design space framework and its associated algorithms to support both global and contextual recommendations. To evaluate the algorithms, we developed the CommunityCommands plug-in for AutoCAD. This plug-in enabled us to perform a 6-week user study of real-time, within-application command recommendations in actual working environments. We report and visualize command usage behaviors during the study, and discuss how the recommendations affected users behaviors. In particular, we found that the plug-in successfully exposed users to new commands, as unique commands issued significantly increased.},
 address = {New York, NY, USA},
 articleno = {6},
 author = {Li, Wei and Matejka, Justin and Grossman, Tovi and Konstan, Joseph A. and Fitzmaurice, George},
 doi = {10.1145/1970378.1970380},
 issn = {1073-0516},
 issue_date = {June 2011},
 journal = {ACM Trans. Comput.-Hum. Interact.},
 keywords = {command recommender system, Collaborative filtering},
 month = {jul},
 number = {2},
 numpages = {35},
 publisher = {Association for Computing Machinery},
 title = {Design and Evaluation of a Command Recommendation System for Software Applications},
 url = {https://doi.org/10.1145/1970378.1970380},
 volume = {18},
 year = {2011}
}