Title: | The Intersection-Validation Method for Evaluating Bayesian Network Structure Learning Without Ground Truth |
Author: | Viinikka, Jussi |
Contributor: | University of Helsinki, Faculty of Science |
Publisher: | Helsingin yliopisto |
Date: | 2019 |
Language: | eng |
URI: |
http://urn.fi/URN:NBN:fi:hulib-202001211118
http://hdl.handle.net/10138/310009 |
Thesis level: | master's thesis |
Discipline: | Tietojenkäsittelytiede |
Abstract: | Structure learning algorithms for Bayesian networks are typically evaluated by examining how accurately they recover the correct structure, given data sampled from a benchmark network. A popular metric for the evaluation is the structural Hamming distance. For real-world data there is no ground truth to compare the learned structures against. Thus, to use such data, one has been limited to evaluating the algorithms' predictive performance on separate test data or via cross-validation. The predictive performance, however, depends on the parameters of the network, for which some fixed values can be used or which can be marginalized over to obtain the posterior predictive distribution using some parameter prior. Predictive performance therefore has an intricate relationship to structural accuracy -- the two do not always perfectly mirror each other. We present intersection-validation, a method for evaluating structure learning without ground truth. The input to the method is a dataset and a set of compared algorithms. First, a partial structure, called the agreement graph, is constructed consisting of the features that the algorithms agree on given the dataset. Then, the algorithms are evaluated against the agreement graph on subsamples of the data, using a variant of the structural Hamming distance. To test the method's validity we define a set of algorithms that return a score maximizing structure using various scoring functions in combination with an exact search algorithm. Given data sampled from benchmark networks, we compare the results of the method to those obtained through direct evaluation against the ground truth structure. Specifically, we consider whether the rankings for the algorithms determined by the distances measured using the two methods conform with each other, and whether there is a strong positive correlation between the two distances. We find that across the experiments the method gives a correct ranking for two algorithms (relative to each other) with an accuracy of approximately 0.9, including when the method is applied onto a set of only two algorithms. The Pearson correlations between the distances are fairly strong but vary to a great extent, depending on the benchmark network, the amount of data given as input to intersection-validation and the sample size at which the distances are measured. We also attempt to predict when the method produces accurate results from information available in situations where the method would be used in practice, namely, without knowledge of the ground truth. The results from these experiments indicate that although some predictors can be found they do not have the same strength in all instances of use of the method. Finally, to illustrate the uses for the method we apply it on a number of real-world datasets in order to study the effect of structure priors on learning. |
Total number of downloads: Loading...
Files | Size | Format | View |
---|---|---|---|
Viinikka_Jussi_Pro_gradu_2019.pdf | 7.624Mb |
View/ |