We introduce a permutation test for heterogeneous treatment effects based on the quantile process. However, tests like this, based on the quantile process, often suffer from estimated nuisance parameters that jeopardize their validity, even in large samples. To overcome this problem, we use Khmaladze's martingale transformation. We show that the permutation test based on the transformed statistic controls size asymptotically. Numerical evidence asserts the good size and power performance of our test procedure compared to other popular quantile-based tests. We discuss a fast implementation algorithm and illustrate our method using experimental data from a welfare reform.

This paper proposes an asymptotically valid permutation test for heterogeneous treatment effects in the presence of an estimated nuisance parameter. Not accounting for the estimation error of the nuisance parameter results in statistics that depend on the particulars of the data generating process, and the resulting permutation test fails to control the Type 1 error, even asymptotically.

In this paper we consider a permutation test based on the martingale transformation of the empirical process to render an asymptotically pivotal statistic, effectively nullifying the effect associated with the estimation error on the limiting distribution of the statistic. Under weak conditions, we show that the permutation test based on the martingale-transformed statistic results in the asymptotic rejection probability of α in general while retaining the exact control of the test level when testing for the more restrictive sharp null. We also show how our martingale-based permutation test extends to testing whether there exists treatment effect heterogeneity within subgroups defined by observable covariates. Our approach comprises testing the joint null hypothesis that treatment effects are constant within mutually exclusive subgroups while allowing the treatment effects to vary across subgroups.

Monte Carlo simulations show that the permutation test presented here performs well in finite samples, and is comparable to those existing in the literature. To gain further understanding of the test to practical problems, we investigate the gift exchange hypothesis in the context of two field experiments from Gneezy and List (2006). Lastly, we provide the companion R Package to facilitate and encourage the application of our test in empirical research.

Though stratified randomization achieves more balance on baseline covariates than pure randomization, it does affect the way we conduct inference. This paper considers the classical two-sample goodness-of-fit testing problem in randomized controlled trials when the researcher employs a particular type of stratified randomization—covariate-adaptive randomization. When testing the null hypothesis of equality of distributions between experimental groups in this setup, we first show that stratification leaves a mark on the test statistic's limit distribution, making it difficult, if not impossible, to obtain critical values. We instead propose an alternative approach to conducting inference based on a permutation test that i) is asymptotically exact in the sense that the limiting rejection probability under the null hypothesis equals the nominal α level, ii) is applicable under relatively weak assumptions commonly satisfied in practice, and iii) works for randomization schemes that are popular among empirically oriented researchers, such as stratified permuted block randomization.

The proposed test's main idea is that by transforming the original statistic by one minus its bootstrap p-value, it becomes asymptotically uniformly distributed on [0,1]. Thus, the transformed test statistic—also called prepivoted—has a fixed limit distribution that is free of unknown parameters, effectively removing the effect of stratification. Consequently, a permutation test based on the prepivoted statistic produces a test whose limiting rejection probability equals the nominal level. We present further numerical evidence of the proposed test's advantages in a Monte Carlo exercise, showing our permutation test outperforms the existing alternatives. We illustrate our method's empirical relevance by revisiting a field experiment by Butler and Broockman (2011) on the effect of race on state legislators' responsiveness to help their constituents register to vote during elections in the United States. Lastly, we provide the companion R Package package to facilitate and encourage applying our test in empirical research.