Mar 29 2017

stat.ME arXiv:1703.09626v1

Highly robust and efficient estimators for the generalized linear model with a dispersion parameter are proposed. The estimators are based on three steps. In the first step the maximum rank correlation estimator is used to consistently estimate the slopes up to a scale factor. In the second step, the scale factor, the intercept, and the dispersion parameter are consistently estimated using a MT-estimator of a simple regression model. The combined estimator is highly robust but inefficient. Then, randomized quantile residuals based on the initial estimators are used to detect outliers to be rejected and to define a set S of observations to be retained. Finally, a conditional maximum likelihood (CML) estimator given the observations in S is computed. We show that, under the model, S tends to the complete sample for increasing sample size. Therefore, the CML tends to the unconditional maximum likelihood estimator. It is therefore highly efficient, while maintaining the high degree of robustness of the initial estimator. The case of the negative binomial regression model is studied in detail.

Mar 29 2017

stat.ME arXiv:1703.09305v1

Software packages usually report the results of statistical tests using p-values. Users often interpret these by comparing them to standard thresholds, e.g. 0.1%, 1% and 5%, which is sometimes re-inforced by a star rating (***, **, *). In this article, we consider an arbitrary statistical test whose p-value p is not available explicitly, but can be approximated by Monte Carlo samples, e.g. bootstrap or permutation tests. The standard implementation of such tests usually draws a fixed number of samples to approximate p. However, the probability that the exact and the approximated p-value lie on different sides of a threshold (thus changing the interpretation) can be high, particularly for p-values close to a threshold. We present a method to overcome this. We consider a finite set of user-specified intervals which cover [0,1] and which can be overlapping. We call these p-value buckets. We present algorithms that, with high probability, return a p-value bucket containing p. Suitably chosen overlapping buckets allow decisions in finite time and lead to an extension of the star rating. The proposed methods are suitable for use in standard software and can be computationally more efficient than standard implementations. We illustrate our methods using a contingency table example.

A flexible approach to modeling network data is based on exponential-family random graph models. We consider here exponential-family random graph models with additional structure in the form of local dependence, which have important conceptual and statistical advantages over models without additional structure. An open problem is how to estimate such models from large random graphs. We pave the ground for massive-scale estimation of such models by exploiting model structure for the purpose of parallel computing. The main idea is that we can first decompose random graphs into subgraphs with local dependence and then perform parallel computing on subgraphs. We hence propose a two-step likelihood-based approach. The first step estimates the local structure underlying random graphs. The second step estimates parameters given the estimated local structure of random graphs. Both steps can be implemented in parallel, which enables massive-scale estimation. We demonstrate the advantages of the two-step likelihood-based approach by simulations and an application to a large Amazon product network.

Mar 29 2017

stat.ME arXiv:1703.09282v1

There are many cluster analysis methods that can produce quite different clusterings on the same dataset. Cluster validation is about the evaluation of the quality of a clustering; "relative cluster validation" is about using such criteria to compare clusterings. This can be used to select one of a set of clusterings from different methods, or from the same method ran with different parameters such as different numbers of clusters. There are many cluster validation indexes in the literature. Most of them attempt to measure the overall quality of a clustering by a single number, but this can be inappropriate. There are various different characteristics of a clustering that can be relevant in practice, depending on the aim of clustering, such as low within-cluster distances and high between-cluster separation. In this paper, a number of validation criteria will be introduced that refer to different desirable characteristics of a clustering, and that characterise a clustering in a multidimensional way. In specific applications the user may be interested in some of these criteria rather than others. A focus of the paper is on methodology to standardise the different characteristics so that users can aggregate them in a suitable way specifying weights for the various criteria that are relevant in the clustering application at hand.