Optimizing software crowdsourcing requirements design through machine learning
Crowdsourcing incentive design
Crowdsourcing is rapidly gaining ground in recent years with advances being made in technology. The modern world is aggressively embracing artificial intelligence with machine learning as the engine behind AI. Research has been conducted from different perspectives with regard to crowdsourcing and trying to unravel the inner working mechanisms. Scholars are eager to answer the question as to how people can effectively collaborate and work virtually. Worker motivation has been under study for a significant period for it has been found to have a higher stake in employee performance11. Worker motivation can be studied from different perspectives. Motivation can be classified as either extrinsic motivation or intrinsic. Intrinsic motivation occurs when a pool of experts is driven by curiosity to learn and know different aspects of things. In this study, participants were not motivated by intrinsic incentive when no extrinsic motivation was provided (no payment). Individuals are organically motivated to contribute and make some works in a crowdsourcing situation. Another sort of intrinsic drive is achievement motivation, which arises when participation in activity leads to a sense of accomplishment. According to Stol et al.12, individuals who have a sense of personal gratification when obtaining certain coding skills, for example, are intrinsically motivated and will have a pleasurable experience.
On the other hand, intrinsic motivation to experience stimulation arises when a person participates in the activity because it is exciting or enjoyable. Workers are paid for accomplishing needed activities in paid crowd work. TopCoder (the largest software crowdsourcing platform), Amazon Mechanical Turk, Upwork, Fiverr, Utest, and other crowdsourcing platforms provide workers with monetary incentives in exchange for the projects they handle.
According to Finnerty et al.13, there are different incentives provided with the aim of improving the quality of software development outcomes. For example, workers can be given a chance to evaluate their peers’ work, this makes people the workers to feel part of the system. Alelyani et al.7, on the other hand, argue that workers value quality evaluation feedback and that it can help them achieve better achievements. Despite their success in attracting talented individuals to take part and produce various products, studies show that crowdsourcing platforms face challenges in designing effective incentive systems for their various tasks. These challenges can be due to requesters’ constraints and the online market setting of a particular project. As a result, further research is needed to better understand crowd behavior and build appropriate mechanisms.
Crowdsourcing participants’ behavior
Participant behavior has gained traction among researchers in recent years. The scholars aim to find out the relationship between participant’s behavior and the quality of work. Prior work studied the dependability of crowd workers in selecting and handling various projects in the crowdsourcing platforms14. Workers with high-reliability ratings are more precise in the types of rewards they choose, whereas workers with low-reliability ratings are more general. According to the findings, the most dependable workers in the high-reliability group are more driven to swiftly register for a task and complete it in the shortest possible period.
The contest outcome on crowdsourcing was previously determined by a game theory model. The game theory model majorly focused on the outcome of a single contest by determining the reward value13. In another related scholarly work15, conducted research aimed at determining the effects of the reputation system being applied by TopCoder. The TopCoder reputation system touched on the behavior of the participants. Individual characteristics, as well as project payment and the number of project requirements, were found to be significant predictors of project quality in the study. In addition, the researchers discovered evidence of contestants’ strategic behavior. According to Moussawi and Koufaris16, the competitive round of the contest puts high-rated participants up against the tougher competition. To make things easier for themselves, they go first in the contest’s registration phase, registering for specific projects ahead of time.
According to the findings, TopCoder’s reputation methods improve the allocative efficiency of concurrent all-pay contests and should be evaluated for implementation in other crowdsourcing platforms. A recent study looked at the impact of developer behavior and community structure on software crowdsourcing methods using social network analysis. The study discovered that TopCoder users engage in temporal burst patterns of online interaction, which leads to a resemblance in the type and level of participation at TopCoder14.
Crowdsourcing task design
According to Goodman and Paolacci17, recent studies have conclusively agreed that the data collection interface design directly relates to the quality of the work to be delivered by the worker. The graphical user interfaces of any crowdsourcing platform are an important factor when it comes to quality assurance. Prior research used the task choice decision (TCD) model and found that there is a close correlation between task design and user productivity18. As one part of the study focused on determining the relationship between work deliverables and task design, the other focused was studying the effects such designs have on quality. The study concluded that there is a need to apply different designs specific to different tasks. However, the revelations from the above studies do not close the possibility of more studies being conducted on the stated topic. A study on crowdsourcing relatedness and growth took a different approach to problem design. Moussawi and Koufaris16, investigated how the kind and scale of the tasks influence the level of creativity, complexity, and skill necessary to perform them. A pay vs. time trade-off was determined to be the key determinant for selecting tasks to do in this study. More research on the types of incentive mechanisms that should be used when developing activities is needed, according to the findings. Another study looked at how task design, such as instructions and user interface, can influence workers’ perceptions of the job16. The difficulties of cognitive load theory and its importance for building interface systems were discussed by Vaz, Steinmacher, and Marczak19. When users interact with the interface, one part of cognitive stress, according to the study, can address the concept of memory burden. Considering users’ knowledge and minimizing redundant content is one strategy to reduce cognitive load. Sarma et al.20 defined task quality in crowdsourcing as a set of components that can be used to assess the quality of an assignment. Time spent on a task, job length, task resources, and price options is some of these elements. The findings of applying a regression model to the evaluated tasks reveal that the previously described criteria play a role in determining task quality.
Decision making and task uncertainty
Software crowdsourcing tasks are being misinterpreted by workers due to the emerging uncertainties caused during posting on the platforms. Crowdsourcing participants have exhibited risk behavior through unintentional deviations from quality assurance. According to Vaz et al.19, researchers are concerned about the participants’ risk behaviors emanating from poorly designed ‘requirements’ posting interfaces. The completion of the software crowdsourcing tasks is deemed uncertain if the task description is misinterpreted. The quality of the deliverables is significantly threatened by the participants’ risks. Crowdsourcing systems need workers to make hazardous judgments by selecting from a range of options, each of which has a chance of yielding related payoffs. The TopCoder model will be used to better describe the software crowdsourcing decision-making process.
In general, a developer can express interest in any task by signing up for a challenge, which indicates that the developer is competing with others on that assignment. Performance and skill ratings for the developer are also displayed. Multiple micro activities are typically chained together into workflows for quality assurance and to complete difficult jobs. In the “Task Preparation” phase, such workflows may divide bigger jobs into smaller subtasks, and then recompose subtask solutions into an overall work product. To avoid any ambiguity or confusion in the decomposed task, this part of the workflow should follow a methodical procedure. Failure to do so would jeopardize the development process, thus impacting the crowdsourcing experience. Existing platforms do not always follow a regular procedure for breaking down huge projects into smaller tasks, setting up requirements, pricing each task, or managing inter-dependencies21. Workers may have ambiguity because of these challenges when attempting to register for a task. Following the preparatory phase, tasks are made public and open to the public for registration and contribution. The perception of task design by crowd participants is a difficult subject to answer because there are always some differences in task design and preparation. Participants have a set amount of time to complete and submit projects after registering for them. A task is considered unsuccessful if it receives no submissions. The term “task hunger” refers to the lack of submissions for a task21.
Following submission, the peer-review process begins. All contributions to a challenge are vetted by review boards, which are made up of experienced. Individuals picked by the site. If the task’s score after the review phase was greater than 75%, it was deemed successful; otherwise, it was canceled. Developers are notified confidentially of their ratings and have the option to appeal if they are unhappy with their present results. It is important to note that this “appeal phase” can only be used once. Finally, all of the developers’ scores are made public, and the reviewers provide ratings. The above workflow demonstrates how crowdsourcing technologies allowed task requesters and workers to transcend geographical barriers, reducing expert shortages and scalability. Crowdsourcing can help to assign the appropriate assignment to the right people at the right price by utilizing the internet. Crowdsourcing platforms, on the other hand, are still grappling with how to better build a system that can reduce task ambiguity, which could compromise the quality of their decision-making5. Previous research looked at how crowdsourcing platforms might improve their present methods by collecting all components of crowdsourcing market decision-making. This entails developing several models that synthesize all aspects that can be combined to improve decision-making. With the development of various software crowdsourcing platforms, decision making has become more popular. Its use has spread across a variety of fields, including operations research, economics, and computer science. Each platform’s management can define the interaction model. Requestors can freely interact with available workers on some platforms. Other platforms, on the other hand, prohibit this type of connection between workers and requestors, making the platform the only route via which both sides must communicate to complete the task. As a result, various modeling challenges should be considered in this category. For example, the process of assigning workers to jobs based on factors such as worker reliability, feedback, experience, and talents14. One of the key roadblocks to software crowdsourcing platforms is the modeling of crowdsourcing incentives22. The task’s success may be jeopardized if the task’s price is overestimated or underestimated. As a result, various studies have attempted to solve this issue in the context of software crowdsourcing. Task requestors and workers are also concerned about performance optimization. Given the current market rivalry, platforms must maximize the utility of requestors to ensure their long-term viability. Workers’ management, on the other hand, is a difficult task because there are constantly issues about their dependability, motivation, and involvement.
Socio technical systems
According to the Socio-Technical System (STS) design model, the technical and social aspects of software design are key factors influencing positive productivity4. The interdependencies between the various components of a socio-technical system are primarily structured. The qualities and interactions of these two components should be considered in any effective system. As described by Sherief et al.23, any management information system design should be framed within the STS design methodology. The STS method applies to both current and new work systems. In a crowdsourcing context, a platform comprises two interconnected systems: social and technological.
Tasks, technology, and design are all part of the technical system, which transforms input into output. The social system comprises people, skills, beliefs, connections, and reward systems. Therefore, any system design or redesign must consider both systems holistically. According to Sherief et al.20, optimal fit is achieved through a design process aimed at the joint optimization of these subsystems, which is the cornerstone of the socio-technical system approach. In other words, an organizational system can only maximize performance if the interdependence of its subsystems is openly recognized. Prior research identifies software crowdsourcing as a socio-technical system, where individuals, technologies, and labor activities are interconnected within the platform7. The interaction among requesters, developers, and platforms is essential for system success, increasing the interdependence between system design and user engagement.
Prior research24 has studied the impact of five risk characteristics on crowdsourcing performance through self-reporting. Workers, requestors, relationships, requirements, and task complexity were all identified as potential sources of risk. Research investigating the impact of these components on the process is crucial yet surprisingly understudied, as system design might influence participant behavior. Existing studies indicate that social subsystem risks reduce crowdsourcing performance, suggesting that various forms of risk collectively affect performance. Many studies have discovered a link between crowdsourcing performance and factors such as task design, price, worker experience, skills, motivation, and other aspects of the crowdsourcing system. Crowdsourcing performance is measured by the degree of effective work completion14. Given the substantial technological and social uncertainties in crowdsourcing, this paper hypothesizes that crowdsourcing risk relates to several design features in task description, incentive, communication, and engagement tactics, based on past research. Prior research studied the impact of five risk characteristics on crowdsourcing performance in a self-reporting study. Workers, requestors, relationships, requirements, and task complexity were all mentioned as possible sources of danger. Research that could investigate the impact of these components on the process must be investigated, a topic that is surprisingly understudied, because system design might influence participant behavior. Existing research indicated that social subsystem risk reduces crowdsourcing performance, indicating that several forms of risks combine to effect performance.
Numerous studies have identified correlations between crowdsourcing performance and several elements of the crowdsourcing system, such as task design, price, worker experience, skills, and motivation19. Crowdsourcing performance is defined as the degree of effective work completion. Based on prior research and acknowledging the significant technological and social uncertainties inherent in crowdsourcing, this paper hypothesizes a relationship between crowdsourcing risk and design features encompassing task description, incentives, communication strategies, and engagement tactics.
Table 1 compares prior literature with the outcomes of the current study, highlighting attributes influencing task success in software crowdsourcing platforms. The study confirms findings on task presentation quality, clarity, readability, and incentive design, while extending insights with machine learning models like Price2. Key factors such as external resources, technology relevance, and reputation systems align with prior research, validating their importance. The results provide actionable recommendations to enhance task design and platform performance.
Literature-derived factors and their empirical validation
To ground the development of the proposed task design framework, This work reviewed key studies in software crowdsourcing. Several recurring factors emerged, including task clarity, incentive structures, cognitive load, and social dynamics. These were mapped to broader design categories presented in this study. Furthermore, the empirical findings validated the practical relevance of these factors, as most showed strong associations with task success in the machine learning models.
Table 1 summarizes each literature-derived factor, its source, the key insight from prior research, how it aligns with this study’s design categories, and whether it was empirically supported by the results.
link
