Initiating the Transition towards Continuous Experimentation : Empirical Studies with Software Development Teams and Practitioners

Show full item record

Title: Initiating the Transition towards Continuous Experimentation : Empirical Studies with Software Development Teams and Practitioners
Author: Yaman, Sezin Gizem
Contributor: University of Helsinki, Faculty of Science, Faculty of Science of the University of Helsinki
Doctoral Programme in Computer Science
Publisher: Helsingin yliopisto
Date: 2019-10-25
Belongs to series: Series of Publications A - URN:ISSN:1238-8645
Thesis level: Doctoral dissertation (article-based)
Abstract: Software experiments are presently often used by big technology pioneers, such as Microsoft, Facebook and Google, in order to learn about their users and to guide their research and development activities. Continuous experimentation (CE) is reported to be an integral part of software development in these organisations, however, how they transitioned to the approach is not publicly shared. Therefore, there is a lack of guidance for other organisations that are willing to adopt CE. In the current competitive markets, investing time and money in a new approach might be risky for these organisations, especially if they do not know how to initiate this transition process. This dissertation focuses on how organisations can initiate the transition towards CE, i.e., an approach to enhance development decisions by running experiments in an iterative and sustainable fashion. The dissertation was designed to acquire descriptive and observational knowledge through empirical studies and was con- ducted in three main phases. First, we designed and ran multiple-case studies to investigate how CE can be introduced to existing software company development teams, who want to run their first systematic experiments. We extracted descriptive knowledge from the introduction process and composed lessons learned to act as guidelines. In the second phase, we conducted a survey study with practitioners from four Nordic software companies, in order to better understand their attitudes and perception towards experiment-driven development, user involvement and ethics. Examining the results at role-to-role levels gave us an understanding of commonalities and distinctions stemming from different job functions. Furthermore, we identified patterns from the data that describe what trends exist across the dataset with respect to experiment-driven software development. Finally, in the last phase of the study, we conducted a single-case study with a mobile gaming company to investigate how CE functions as an organisational mechanism throughout the development life-cycle. The findings show that transitioning towards CE is a learning process that can be facilitated well by guidance, utilising existing resources and starting with small experiments with potentially enormous impact. Furthermore, by investigating the point of view of practitioners, we observed that software experiments represent different concepts, for instance, A/B tests and user interviews. We also observed that the role of the practitioner has a big impact not only on how experiments are understood, but also how individuals perceive the ethics involved in the experiments. For example, while managers are more cautious about company-customer relationships, UX designers were found to allow exceptions to user notification during experiments. In addition, we discovered that companies might understand and adopt experiment-driven development differently, for in- stance, influenced by their business contexts. Lastly, by examining a company’s CE practices, we found that experiments can take different forms given the development stage, and the organisational mechanism can be established to fit both the needs of the business domain and organisational goals. One of the biggest challenges of adopting CE, inaccessible real users, can be overcome with alter- native methods, such as proxy users, especially early in the development, when experiments are important in determining product value. Highly competitive markets can put pressure on organisations to avoid risks and costs when adopting a new approach. In this dissertation, we learned that by and large, software organisations and development teams can initiate their transition towards CE in an efficient and economical way. Furthermore, we conclude that the transition is a learning process that improves with practice and has to adapt to the organisational goals and contexts. The influence of human factors, such as the finding that individual perception of experiments and ethics is correlated with job functions indicates that CE is a multi-disciplinary research field, where individuals should be studied as well as experimentation processes. Software engineering research needs further studies to validate the findings in different contexts.Big technology corporations, such as Microsoft, Facebook and Google, often use software experiments to learn more about their users and to guide their business strategies. In 2009, Google ran a big experiment with 41 shades of blue to determine, based on evidence which shade led to the greatest number of user clicks on ad links. As a result, Google made an extra $200m a year in ad revenue. Experiments are known to be an integral part of software development for these companies; however, how these companies came to be “experiment-driven” is not publicly known. Furthermore, as experiments are used to gather data from product and service users, they are subject to several considerations, such as ethics and privacy. For example, in 2014, Facebook ran an experiment with thousands of their users to test how emotional contagion occurs when their News Feed content is manipulated, and they found that users can be made happier or sadder through such manipulation. The Facebook–Cambridge Analytica scandal in early 2018 concerned more than 50 million Facebook users, and involved their data being used for political advertising purposes. These incidents raised serious questions about user privacy and ethical implications: How can such experiments be performed without users' informed consent? There is not much existing research on how software companies could become experiment-driven or on the ethics of software experiments. This study focuses on how companies can become experiment-driven, investigating the transition process, operational needs and human factors, such as ethics. The findings show that the transition is a learning process that can be effectively facilitated through appropriate guidance, utilising existing resources and starting with small experiments with potentially enormous impact, at low cost. Furthermore, our study of industry professionals shows that the practitioner’s role has a big impact on how they perceive the nature of an experiment and on how they perceive the ethical issues involved. In particular, practitioners tend to view the question of what is ethically acceptable through their particular job function. For instance, managers are cautious about the company–customer relationship, and they think that users should always be notified of experiments in advance, whereas UX designers allow for exceptions such as informing users afterwards. When there are no established regulations, ethical issues are left to practitioners’ own interpretation. Companies have to work on their regulations and policies concerning user data collection, to ensure that industry professionals have adequate ethical guidelines to support their work and to ensure compliance with relevant legislation. Further studies on software engineering research are needed to validate the findings of the present study in different contexts and with different populations.
Subject: Computer Science
Rights: This publication is copyrighted. You may download, display and print it for Your own personal use. Commercial use is prohibited.

Files in this item

Files Size Format View

There are no files associated with this item.

This item appears in the following Collection(s)

Show full item record