Gamifault: A Gamified Motivational Framework for Software Debugging Process

Document Type : Research Article


1 Faculty of Computer Science and Engineering, Shahid Beheshti University, Tehran, Iran.

2 Department of Computer Engineering, Parand Branch, Islamic Azad University, Tehran, Iran.

3 Faculty of Education and Psychology, Shahid Beheshti University, Tehran, Iran.


The released software systems still involve some faults, for which debugging becomes necessary. On the one hand, manual software debugging remains an arduous, time-consuming, and expensive task. On the other hand, effective software debugging is organized around motivated and patient developers. In this paper, a novel approach, namely Gamifault, is provided to make debugging more attractive and enjoyable. Particularly, the objective of Gamifault is to make the developer more curious to proceed debugging, that is fault localization and program repair, enthusiastically. To achieve this objective, the concepts and potentials of gamification are adapted to the typical tasks of software debugging. In particular, Gamifault makes use of an existing fault localization technique to determine the likelihood to each statement may be faulty. Based on the likelihood, the developer then attempts to find the exact fault location and fix the fault. Next, Gamifault reacts to the developer with a gamified success rate. That is, it shows the number of test cases that have been passed on the modified program. This process is repeated until the program passes on every given test case. To evaluate Gamifault, a prototype web-based tool was implemented in Java that targets faulty software programs. Then, 16 developers were asked to employ gamified and non-gamified versions of the tool in their debugging activities on 46 subject programs taken from the Code4Bench suite of programs. Developers could successfully debug 7 and 95 faulty programs using the non-gamified and gamified tools, respectively. In addition, the gamified tool helped developers debug the faulty program in less than two minutes on average. These results suggest that Gamifault offers advantages over existing debugging systems.


Main Subjects

[1] R. Abraham and M. Erwig. Goal-directed debugging of spreadsheets. In 2005 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC'05), pages 37--44. IEEE, 2005. [ bib | DOI ]
[2] R. Abreu, P. Zoeteweij, and A. J. Van Gemund. On the Accuracy of Spectrum-based Fault Localization. In Testing: Academic and industrial conference practice and research techniques-MUTATION (TAICPART-MUTATION 2007), pages 89--98. IEEE, 2007. [ bib | DOI ]
[3] S. Amiri-Chimeh, H. Haghighi, M. Vahidi-Asl, K. Setayesh-Ghajar, and F. Gholami-Ghavamabad. Rings: A Game with a Purpose for Test Data Generation. Interacting with Computers, 30(1):1 -- 30, 2018. [ bib | DOI ]
[4] P. Ammann and J. Offutt. Introduction to software testing. Cambridge University Press, 2016. [ bib ]
[5] S. Anand, E. K. Burke, T. Y. Chen, J. Clark, M. B. Cohen, W. Grieskamp, M. Harman, M. J. Harrold, P. McMinn, and A. Bertolino. An orchestrated survey of methodologies for automated software test case generation. Journal of Systems and Software, 86(8):1978--2001, 2013. [ bib | DOI ]
[6] S. Arai, K. Sakamoto, H. Washizaki, and Y. Fukazawa. A Gamified Tool for Motivating Developers to Remove Warnings of Bug Pattern Tools. In 2014 6th International Workshop on Empirical Software Engineering in Practice, pages 37--42. IEEE, 2014. [ bib | DOI ]
[7] B. Ashok, J. Joy, H. Liang, S. K. Rajamani, G. Srinivasa, and V. Vangala. DebugAdvisor: A recommender system for debugging. In Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, pages 373--382. ACM, 2009. [ bib | DOI ]
[8] F. Y. Assiri and J. M. Bieman. Fault localization for automated program repair: effectiveness, performance, repair correctness. Software Quality Journal, 25:171–199, 2017. [ bib | DOI ]
[9] T. Barik, Y. Song, B. Johnson, and E. Murphy-Hill. From quick fixes to slow fixes: Reimagining static analysis resolutions to enable design space exploration. In 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 211--221. IEEE, 2016. [ bib | DOI ]
[10] K. A. Batterton and K. N. Hale. The Likert Scale What It Is and How To Use It. Phalanx, 50(2):32--39, 2017. [ bib | DOI ]
[11] M. Beller, N. Spruit, D. Spinellis, and A. Zaidman. On the dichotomy of debugging behavior among programmers. In Proceedings of the 40th International Conference on Software Engineering, pages 572--583. ACM, 2018. [ bib | DOI ]
[12] M. Brodie, S. Ma, G. Lohman, L. Mignet, N. Modani, M. Wilding, J. Champlin, and P. Sohn. Quickly Finding Known Software Problems via Automated Symptom Matching. In Second International Conference on Autonomic Computing (ICAC'05), pages 101--110. IEEE, 2005. [ bib | DOI ]
[13] G. Catolino, F. Palomba, A. Zaidman, and F. Ferrucci. Not all bugs are the same: Understanding, characterizing, and classifying bug types. Journal of Systems and Software, 152:165--181, 2019. [ bib | DOI ]
[14] S. Cooper, A. Treuille, J. Barbero, A. Leaver-Fay, K. Tuite, F. Khatib, A. C. Snyder, M. Beenen, D. Salesin, and D. Baker. The challenge of designing scientific discovery games. In Proceedings of the Fifth international Conference on the Foundations of Digital Games, page 40–47. ACM, 2010. [ bib | DOI ]
[15] A. Darejeh and S. S. Salim. Gamification Solutions to Enhance Software User Engagement—A Systematic Review. International Journal of Human-Computer Interaction, 32(8):613--642, 2016. [ bib | DOI ]
[16] G. M. de Jesus, F. C. Ferrari, L. N. Paschoal, Simone d. R. S. de Souza, de D. Paula Porto, and V. H. S. Durelli. Is It Worth Using Gamification on Software Testing Education? An Extended Experience Report in the Context of Undergraduate Students. Journal of Software Engineering Research and Development, 8:6--1, 2020. [ bib | DOI ]
[17] A. De la Croix and J. Skelton. The simulation game: an analysis of interactions between students and simulated patients. Medical Education, 47(1):49--58, 2013. [ bib | DOI ]
[18] D. de Paula Porto, G. M. de Jesus, F. C. Ferrari, and S. C. P. F. Fabbri. Initiatives and challenges of using gamification in software engineering: A Systematic Mapping. Journal of Systems and Software, 173:110870, 2021. [ bib | DOI ]
[19] F. Garcia, O. Pedreira, M. Piattini, A. Cerdeira-Pena, and M. Penabad. A framework for gamification in software engineering. Journal of Systems and Software, 132:21--40, 2017. [ bib | DOI ]
[20] L. Gazzola, D. Micucci, and L. Mariani. Automatic software repair: a survey. IEEE Transactions on Software Engineering, 45(1):34 -- 67, 2019. [ bib | DOI ]
[21] B. Hartmann, D. MacDougall, J. Brandt, and S. R. Klemmer. What would other programmers do: suggesting solutions to error messages. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 1019--1028. ACM, 2010. [ bib | DOI ]
[22] A. T. Jebb, V. Ng, and L. Tay. A review of key Likert scale development advances: 1995--2019. Frontiers in psychology, 12:637547, 2021. [ bib | DOI ]
[23] D. Jeffrey, M. Feng, N. Gupta, and R. Gupta. BugFix: A learning-based tool to assist developers in fixing bugs. In 2009 IEEE 17th International Conference on Program Comprehension, pages 70--79. IEEE, 2009. [ bib | DOI ]
[24] J. A. Jones, M. J. Harrold, and J. Stasko. Visualization of test information to assist fault localization. In Proceedings of the 24th International Conference on Software Engineering, page 467–477. ACM, 2002. [ bib | DOI ]
[25] J. A. Jones and M. J. Harrold. Empirical evaluation of the tarantula automatic fault-localization technique. In Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering, page 273–282. ACM, 2005. [ bib | DOI ]
[26] S. Kaleeswaran, V. Tulsian, A. Kanade, and A. Orso. MintHint: automated synthesis of repair hints. In Proceedings of the 36th International Conference on Software Engineering, page 266–276. ACM, 2014. [ bib | DOI ]
[27] J. M. Keller. Motivational design of instruction. Instructional design theories and models: An overview of their current status, 1(1983):383--434, 1983. [ bib | DOI ]
[28] A. Khalilian, A. Baraani-Dastjerdi, and B. Zamani. CGenProg: Adaptation of cartesian genetic programming with migration and opposite guesses for automatic repair of software regression faults. Expert Systems with Applications, 169:114503, 2021. [ bib | DOI ]
[29] C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer. GenProg: A Generic Method for Automatic Software Repair. IEEE Transactions on Software Engineering, 38(1):54 -- 72, 2012. [ bib | DOI ]
[30] P. S. Leitao-Junior, D. M. Freitas, S. R. Vergilio, C. G. Camilo-Junior, and R. Harrison. Search-based fault localisation: A systematic mapping study. Information and Software Technology, 123:106295, 2020. [ bib | DOI ]
[31] C. Liu, L. Fei, X. Yan, J. Han, and S. P. Midkiff. Statistical Debugging: A Hypothesis Testing-Based Approach. IEEE Transactions on Software Engineering, 32(10):831 -- 848, 2006. [ bib | DOI ]
[32] K. Liu, L. Li, A. Koyuncu, D. Kim, Z. Liu, J. Klein, and T. F. Bissyandé. A critical review on the evaluation of automated program repair systems. Journal of Systems and Software, 171:110817, 2021. [ bib | DOI ]
[33] P. Ma, H. Cheng, J. Zhang, and J. Xuan. Can this fault be detected: A study on fault detection via automated test generation. Journal of Systems and Software, 170:110769, 2020. [ bib | DOI ]
[34] A. Majd, M. Vahidi-Asl, A. Khalilian, A. Baraani-Dastjerdi, and B. Zamani. Code4Bench: A multidimensional benchmark of Codeforces data for different program analysis techniques. Journal of Computer Languages, 53:38--52, 2019. [ bib | DOI ]
[35] X. Mao, Y. Lei, Z. Dai, Y. Qi, and C. Wang. Slice-based statistical fault localization. Journal of Systems and Software, 89:51--62, 2014. [ bib | DOI ]
[36] J. McGonigal and R. Broken. Why Games Make Us Better and How They Can Change the World. 2011. US.: Penguin Group, 2011. [ bib | DOI ]
[37] M. Monperrus. Automatic Software Repair: A Bibliography. ACM Computing Surveys (CSUR), 51(1):1–24, 2019. [ bib | DOI ]
[38] B. Morschheuser, L. Hassan, K. Werder, and J. Hamari. How to design gamification? A method for engineering gamified software. Information and Software Technology, 95:219--237, 2018. [ bib | DOI ]
[39] A. Nath and P. Domingos. Learning tractable probabilistic models for fault localization. In Proceedings of the AAAI Conference on Artificial Intelligence, 2016. [ bib | DOI ]
[40] C. Parnin and A. Orso. Are automated debugging techniques actually helping programmers? In Proceedings of the 2011 International Symposium on Software Testing and Analysis, pages 199--209, 2011. [ bib | DOI ]
[41] G. Rodríguez-Pérez, G. Robles, A. Serebrenik, A. Zaidman, D. M. Germán, and J. M. Gonzalez-Barahona. How bugs are born: a model to identify how bugs are introduced in software components. Empirical Software Engineering, 25:1294--1340, 2018. [ bib | DOI ]
[42] C. Parnin and A. Orso. Code Defenders: Crowdsourcing Effective Tests and Subtle Mutants with a Mutation Testing Game. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), pages 677--688. IEEE, 2017. [ bib | DOI ]
[43] K. Seaborn and D. I. Fels. Gamification in theory and action: A survey. International Journal of Human-Computer Studies, 74:14--31, 2015. [ bib | DOI ]
[44] S. Stieglitz, C. Lattemann, S. Robra-Bissantz, R. Zarnekow, and T. Brockmann. Gamification Using Game Elements in Serious Contexts. Berlin: Springer, 2017. [ bib ]
[45] N. Tillmann, J. Bishop, N. Horspool, D. Perelman, and T. Xie. Code hunt: searching for secret code for fun. In Proceedings of the 7th International Workshop on Search-Based Software Testing, page 23–26. ACM, 2014. [ bib | DOI ]
[46] J. Tu, X. Xie, T. Y. Chen, and B. Xu. On the analysis of spectrum based fault localization using hitting sets. Journal of Systems and Software, 174:106--123, 2019. [ bib | DOI ]
[47] R. Van Rozen. Languages of Games and Play: A Systematic Mapping Study. ACM Computing Surveys (CSUR), 53(6):1–37, 2021. [ bib | DOI ]
[48] L. Von Ahn and L. Dabbish. Designing games with a purpose. Communications of the ACM, 51(8):58--67, 2008. [ bib | DOI ]
[49] W E. Wong, R. Gao, Y. Li, R. Abreu, and F. Wotawa. A Survey on Software Fault Localization. IEEE Transactions on Software Engineering, 42(8):707--740, 2016. [ bib | DOI ]
[50] Q. Wu, Y. Zhu, and Z. Luo. A gamification approach to getting students engaged in academic study. Bulletin of the IEEE Technical Committee on Learning Technology, 17(4):26--29, 2015. [ bib | DOI ]
[51] Y. Xiaobo, L. Bin, and W. Shihai. A Test Restoration Method based on Genetic Algorithm for effective fault localization in multiple-fault programs. Journal of Systems and Software, 172:110861, 2021. [ bib | DOI ]
[52] X. Xie, T. Y. Chen, F. Kuo, and B. Xu. A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization. ACM Transactions on software engineering and methodology (TOSEM), 22(4):1–40, 2013. [ bib | DOI ]
[53] J. Xuan, M. Martinez, F. Demarco, M. Clement, S. L. Marcote, T. Durieux, D. Le Berre, and M. Monperrus. Nopol: Automatic repair of conditional statement bugs in java programs. IEEE Transactions on Software Engineering, 43(1):34--55, 2016. [ bib | DOI ]
[54] A. Zakari, S. P. Lee, R. Abreu, B. H. Ahmed, and R. A. Rasheed. Multiple fault localization of software programs: A systematic literature review. Information and Software Technology, 124:106312, 2020. [ bib | DOI ]
[55] M. Zhang, Y. Li, X. Li, L. Chen, Y. Zhang, L. Zhang, and S. Khurshid. An Empirical Study of Boosting Spectrum-Based Fault Localization via PageRank. IEEE Transactions on Software Engineering, 47(6):1089 -- 1113, 2021. [ bib | DOI ]
[56] D. Zou, J. Liang, Y. Xiong, M. D. Ernst, and L. Zhang. An Empirical Study of Fault Localization Families and Their Combinations. IEEE Transactions on Software Engineering, 47(2):332--347, 2019. [ bib | DOI ]
[57] W. Weimer. Patches as better bug reports. In Proceedings of the 5th international conference on Generative programming and component engineering, page 181–190. ACM, 2006. [ bib | DOI ]