Software QA FYI - SQAFYI

Quality Practices and Problems in Free Software Projects

By: Martin Michlmayr, Francis Hunt, David Probert

Abstract—Free software and open source projects are often perceived to be of high quality. It has been suggested that the high level of quality found in some free software projects is related to the open development model which promotes peer review. While the quality of some free software projects is comparable to, if not better than, that of closed source software, not all free software projects are successful and of high quality. Even mature and successful projects face quality problems; some of these are related to the unique characteristics of free software and open source as a distributed development model led primarily by volunteers. In exploratory interviews performed with free software and open source developers, several common quality practices as well as actual quality problems have been identified. The results of these interviews are presented in this paper in order to take stock of the current status of quality in free software projects and to act as a starting point for the implementation of quality process improvement strategies.

Index Terms—quality practices, quality assurance, quality improvement, free software, open source.

In recent years, the free software and open source model has established itself as a viable alternative to other development models. Many popular products, such as Linux, Apache and Samba, have been created according to open source practices, and the number of free software applications is increasing steadily. An empirical study of a large collection of free software conducted in 2001 found more than 55,000,000 lines of code, with an estimated value of almost US$1.9 billion according to the COCOMO model [4]; moreover, the system has more than doubled in size since then [5].

of significant pieces of software, but many of these applications show levels of quality comparable to or exceeding that of software developed in a closed and proprietary fashion [6], [14]. In his paper “The Cathedral and the Bazaar”, Raymond suggests that this high level of quality is partly due to the high degree of peer review and user involvement often found in free software projects [13]. This hypothesis is in line with findings from traditional software engineering research, which have shown that peer review has a significant impact on the quality of software [3]. While Raymond’s hypothesis sounds plausible, his model has yet to be validated and tested rigorously through empirical data analysis. In general, much of what has been said about quality in free software is based on anecdotal rather than empirical evidence. Given the recent success of many free software applications it can be assumed that, to some extent, the software fulfils the requirements and quality standards of users. Nevertheless, empirical work is necessary to study quality in free software and to contrast it with that of proprietary software. In particular, work on the identification of problems related to the free software model would be beneficial. Insights from such research can subsequently be used to propose process improvement strategies.

II. BACKGROUND Free software projects usually exhibit two characteristics that have important implications on quality assurance. First, they are distributed, in many cases all over the world. Second, the participants of free software projects are usually unpaid volunteers. The recent interest in free software, along with its deployment, clearly shows that successful and mature products for end-users can be built by volunteers in a distributed fashion. The free software development model offers certain advantages over proprietary models; these include the potential for substantial peer review and the attraction of excellent programmers from across the world. However, free software also faces certain challenges that are unique to this model. For example, due to the voluntary nature of free software projects, it is impossible to fully rely on project participants [11]. This issue is further complicated by the distributed nature because it makes it difficult to identify volunteers who are neglecting their duties, and to decide where more resources are needed [10]. While most research on open source has focused on popular and successful projects such as Apache [12] and GNOME [9], there is an increasing awareness that not all free software projects reach maturity, success or high quality. SourceForge, which is currently the most popular hosting site for free software and open source projects with over 95,000 projects, is not only a good resource to find well maintained free software applications – there are also a large number of abandoned projects and software with low quality [7]. Some of these abandoned projects may be explained in terms of a selection process given that more interesting projects with a higher potential will probably attract a larger number of volunteers, but it has also been suggested that project failures might be related to the lack of project management skills [15]. Nevertheless, large and successful projects also face important problems related to quality, such as the volatility of volunteer contributions [11], [10] and the inability of developers to keep up with bug reports [17].
In order to ensure that free software remains a viable model for the creation of mature and high quality software suitable for corporate and mission-critical use, free software quality assurance and project management have to take these challenges and other quality problems into account and find solutions to them. As a first step in the development of practices and processes to assure and further improve quality in free software projects, existing quality practices and quality problems have to be clearly identified. To date however, only a few initial surveys on quality related activities in free software projects have been conducted [19], [20]. The rest of this paper will present results from exploratory interviews conducted with free software developers from a diverse range of projects. The insights gained from these interviews lead to a more complete account of the current status of quality practices and problems.

III. METHODOLOGY For this study, exploratory interviews with seven free software and open source developers have been conducted. The projects were of a very diverse nature and it is believed that despite the relatively small sample size a fair representation of various kinds of free software projects is given. One project started at a university was aimed at giving undergraduate students the opportunity to get experience of a software development project. This project was later transformed into a community project in which students of this university participated with external contributors. Another project was part of research activities carried out by a large corporation, and one resulted from a university research project. The remaining projects originated as community projects. The size of the projects varied quite widely, from large projects such as Apache, GNOME and Debian, to smaller efforts such as VideoLAN and Dasher.

The interviews were of a fairly unstructured nature so as to allow a thorough exploration of quality in the projects in which the interviewees were involved. Questions, such as the following, were asked directly but the interviews remained open for the exploration of other topics: • Are there any quality issues in your project? If so, what are they and how do you deal with them?
• What techniques can be applied in free software and open source projects to ensure quality?
• What kind of facilities does your project have to ensure quality?
• How would you compare the quality of open source and proprietary software? When might the quality in one be higher than in the other?

Full article...

Other Resource

... to read more articles, visit

Quality Practices and Problems in Free Software Projects