Diagrams are a mode of communication that offers challenges for its computational processing. The challenges arise from the multimodal nature of diagrams. This means that diagrams combine several types of expressive resources to achieve their communicative purposes, such as textual elements, connective elements such as arrows and lines, and illustrations. Humans interpret diagrams by judging how these different expressive resources work together to reach the communicative goals set for the diagram. In order to do that, humans make inferences of the diagram layout and the implicit relations that exist between different parts of the diagram. In order to build computational methods for diagram understanding, large amounts of data annotated with these implicit relations is required. Traditionally, these types of discourse structure annotations have been annotated by experts, due to the difficulty of the task and the requirement that the annotator is familiar with the theoretical framework used for describing discourse relations. The chosen theory for modeling discourse relations in diagrams is Rhetorical Structure Theory, originally developed for modeling textual coherence but applicable to multimodal data as well. This thesis explores the possibility to gather discourse relation annotations for multimodal diagram data with crowdsourcing; employing naive workers on crowdsourcing platforms to complete annotation tasks for a monetary reward. Adapting the task of discourse relation annotation to be feasible for naive workers has been proven challenging by past research concerned with only textual data, and the multimodality of the data adds to the complexity of the task. This thesis presents a novel method for gathering multimodal discourse relation annotations using crowdsourcing and methods of natural language processing. Two approaches are explored: adopting an insertive annotation task where the workers are asked to describe the relationship between two diagram elements in their own words and adopting a multiple-choice task, converting the formal definitions of Rhetorical Structure Theory to understandable phrases to annotate with. Natural language processing is used in the first approach to validate the language and structure of the crowdsourced descriptions. The results of the first approach highlight the difficulty of the task: the workers show tendencies of relying heavily on example descriptions shown in the task instructions and difficulty of grasping the differences of the more fine-grained relations. The multiple-choice approach seems more promising, with annotation agreement with expert annotators higher than in previous research concerned with discourse relations in textual data. The manual inspection of the annotated diagrams show that the disagreement of the crowdworkers and expert annotators is often justifiable; both annotations represent a valid interpretation of the discourse relation. This highlights one of the main challenges of the task, which is the ambiguity of some of the relations. Future work is encouraged to consider this by adopting an approach that is less concerned with a pre-defined set of relations and more interested in how the different discourse relations are actually perceived.

In this study, we investigate theoretical and practical issues connected to dif-ferentiating between core and peripheral vocabulary at different levels of lin-guistic proficiency using statistical approaches combined with crowdsourcing. We also investigate whether crowdsourcing second language learners’ rank-ings can be used for assigning levels to unseen vocabulary. The study is per-formed on Swedish single-word items. The four hypotheses we examine are: (1) there is core vocabulary for each proficiency level, but this is only true until CEFR level B2 (upper-intermedi-ate); (2) core vocabulary shows more systematicity in its behavior and usage, whereas peripheral items have more idiosyncratic behavior; (3) given that we have truly core items (aka anchor items) for each level, we can place any new unseen item in relation to the identified core items by using a series of comparative judgment tasks, this way assigning a “target” level for a pre-viously unseen item; and (4) non-experts will perform on par with experts in a comparative judgment setting. The hypotheses have been largely con-firmed: In relation to (1) and (2), our results show that there seems to be some systematicity in core vocabulary for early to mid-levels (A1-B1) while we find less systematicity for higher levels (B2-C1). In relation to (3), we sug-gest crowdsourcing word rankings using comparative judgment with known anchor words as a method to assign a “target” level to unseen words. With regard to (4), we confirm the previous findings that non-experts, in our case language learners, can be effectively used for the linguistic annotation tasks in a comparative judgment setting.

We study the use of data collected via electroencephalography (EEG) to classify stimuli presented to subjects using a variety of mathematical approaches. We report an experiment with three objectives: 1) To train individual classifiers that reliably infer the class labels of visual stimuli using EEG data collected from subjects; 2) To demonstrate brainsourcing, a technique to combine brain responses from a group of human contributors each performing a recognition task to determine classes of stimuli; 3) To explore collaborative filtering techniques applied to data produced by individual classifiers to predict subject responses for stimuli in which data is unavailable or otherwise missing. We reveal that all individual classifier models perform better than a random baseline, while a brainsourcing model using data from as few as four participants achieves performance superior to any individual classifier. We also show that matrix factorization applied to classifier outputs as a collaborative filtering approach achieves predictive results that perform better than random. Although the technique is fairly sensitive to the sparsity of the dataset, it nonetheless demonstrates a viable proof-of-concept and warrants further investigation.

Learnersourcing is an emerging phenomenon in computing education research and practice. In learnersourcing, a crowd of students participates in the creation of course resources such as exercises, written materials, educational videos, and so on. In computing education research, learnersourcing has been studied especially for the creation of multiple-choice questions and programming exercises, where prior work has suggested that learnersourcing can have multiple benefits for teachers and students alike. One result in prior studies is that when students create learnersourced content, the created content covers much of the learning objectives of the course. The present work expands on this stream of work by studying the use of a learnersourcing system in the context of teaching SQL. We study to what extent learnersourced SQL exercises cover course topics, and to what extent students complete learnersourced exercises. Our results continue the parade of previous learnersourcing studies, empirically demonstrating that learnersourced content covers instructor-specified course topics and that students indeed actively work on the learnersourced exercises. We discuss the impact of these results on teaching with learnersourcing, highlight possible explanations for our observations, and outline directions for future research on learnersourcing.

We used crowdsourcing (CS) to examine how COVID-19 lockdown affects the content of dreams and nightmares. The CS took place on the sixth week of the lockdown. Over the course of 1 week, 4,275 respondents (mean age 43, SD = 14 years) assessed their sleep, and 811 reported their dream content. Overall, respondents slept substantially more (54.2%) but reported an average increase of awakenings (28.6%) and nightmares (26%) from the pre-pandemic situation. We transcribed the content of the dreams into word lists and performed unsupervised computational network and cluster analysis of word associations, which suggested 33 dream clusters including 20 bad dream clusters, of which 55% were pandemic-specific (e.g., Disease Management, Disregard of Distancing, Elderly in Trouble). The dream-association networks were more accentuated for those who reported an increase in perceived stress. This CS survey on dream-association networks and pandemic stress introduces novel, collectively shared COVID-19 bad dream contents

Hobby metal detecting is a controversial subject. Legal and policy approaches differ widely across national and regional contexts, and the attitudes of archaeologists and heritage professionals towards detectorists are often polarized and based on ethical or emotive arguments. We, the European Public Finds Recording Network (EPFRN), have implemented collaborative approaches towards detectorist communities in our respective contexts (Denmark, England and Wales, Finland, Flanders, and the Netherlands). Although our motivations are affected by our national circumstances, we base our work on an agreed set of goals, practices, and visions. This article presents the EPFRN's vision statement and provides insight into its underlying thoughts. We hope to create a debate on how to develop best practice approaches that acknowledge the inherent challenges of hobby metal detecting while realizing its potential.