publications

2026

1. The Impact of Documentation on Test Engagement in Pull Requests in OSS

   Teal Amore, Nathan Berman, and Siyuan Jiang

   In Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering, Concordia University, Montreal, QC, Canada, 2026

   Abs DOI arXiv Bib

   Automated testing is crucial for maintaining open-source software quality. However, motivating contributors to include tests for code changes remains a challenge. While existing interventions, such as code coverage metrics and reviewer feedback, are often reactive and applied only after a pull request is opened, this study investigates whether documentation on testing can serve as a proactive measure to encourage testing behavior.In this work, we investigate the relationship between documentation on testing and contributor testing behavior. We introduce the Test Engagement Ratio (TER) to help understand testing frequency. Using data from 160 OSS repositories, we analyze the relationship between documentation comprehensiveness and TER. Our results show a weak but statistically significant positive correlation (ρ = 0.36, p < 0.001), which strengthens to a moderate relationship (ρ = 0.44) in repositories with higher pull request activity. Documentation categories such as How to Run Tests and How to Write Tests show the strongest correlation with testing engagement. Furthermore, TER is found to be moderately correlated (ρ = 0.52, p < 0.001) with Test Code Ratio, providing preliminary evidence of its validity. Our findings suggest that documentation on testing may be associated with increased testing engagement. Future work will explore causality, documentation quality at a granular level, and cross-repository exposure effects.

   @inproceedings{10.1145/3803437.3805555,
     author = {Amore, Teal and Berman, Nathan and Jiang, Siyuan},
     title = {The Impact of Documentation on Test Engagement in Pull Requests in OSS},
     year = {2026},
     isbn = {9798400726361},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     doi = {10.1145/3803437.3805555},
     booktitle = {Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering},
     pages = {1197-1201},
     numpages = {5},
     keywords = {contribution guidelines, documentation, software testing, pull requests},
     location = {Concordia University, Montreal, QC, Canada},
     series = {FSE Companion '26}
   }