I wasn't going to answer, but since a lot of the discussion has been about the statistical aspects and I have some background in that, maybe I can add something here.
TLDR: The study doesn't give enough detail to fully assess the methods, but what I can see is consistent with what I'd expect from a decent survey.
(That is: I'm not saying it's right, but I don't see any obvious signs that it's wrong.)
Stats 101 generally focuses on simple random sample (SRS) designs: we have a population we want to describe, we run a random selection process where every unit (person, household, whatever you're studying) has an equal chance of selection in the sample. From that approach, we can calculate a lot of things about margins of error, sample sizes, etc. etc.
This should be understood as an "assume a spherical cow" approach: it's a useful starting point, but only a starting point. In reality, serious socioeconomic research is almost never based on a pure SRS design and assumptions/rules of thumb drawn from that Stats 101 understanding may not be reliable for assessing it.
I recommend Särndal, Swensson and Wretman's "Model Assisted Survey Sampling" for a solid overview of more complex survey design and analysis methods. See in particular the sections on sampling frames (& multiple frames), stratified sampling & optimal sample allocation, and regression/post-stratification errors. Please take MASS as a general cite for most of the following discussion.
Caveat: MASS concentrates on means (averages) rather than quantile statistics (medians/quartiles/etc.) However, most of the methods presented there have obvious parallels for working with quantiles.
Some of the issues that have been raised:
Is the sample representative?
The real question here is "should it be?"
When you're trying to describe a population as a whole, "representative sampling" is often a reasonable approach. (Loosely speaking: attempting to ensure that the statistical distribution within your sample is roughly the same as in the real population.) Simple random sampling is one way that people attempt to achieve a representative sample, and there are non-SRS options for this purpose too e.g. systematic sampling and some implementations of stratified sampling.
The mathematical analysis is then easy; if you believe that the sample is a representative microcosm of the full population, you can just take the mean and median of the sample and use that as an estimate for the population. This isn't always the most efficient approach possible, but for whole-population stats it's not bad.
However, when you're interested in sub-populations e.g. different racial groups, it's another story.
Suppose we want to explore questions like "are White households more affluent than Black households?" This usually requires estimating typical affluence for each group, typical ranges (e.g. "90% of households have more than $X net worth, and 10% have more than $Y"), and then comparing the two.
Problem here is that for any given group, the accuracy of those estimates is affected by sample size. If we're surveying a population that's 90% White and 10% Black, and we use a "representative" sample, we will end up with 9x as many White households as Black ones. That will give us very good estimates for the characteristics of White households but much poorer estimates for Black ones, making it difficult to compare the two.
One option is just to increase the sample size so that we get enough people/households from the smaller sub-populations, while getting even more from the common ones. This is mathematically nice but it's prohibitively expensive for anything that requires trained interviewers, as economic data usually does.
A more practical alternative is oversampling: one way or another, you adjust your survey design to select a higher proportion of the smaller sub-populations. The Boston study has done this:
Various sampling techniques were used to locate and identify an
ethnically plural sample consisting of the specifically defined ethnic
groups. The techniques included the following: directory-listed
landline samples targeted to census tracts where specific ethnic
groups were known to reside; cell phone random digit dialing samples
drawn from rate centers that covered the targeted ethnic group ZIP
codes; samples drawn from targeted ZIP codes on the basis of billing
address; and the use of surname-based lists targeting specific
national origin groups.
A naïve approach to this can introduce a great deal of bias. If I try to get a sample of Asian households solely by going to a district known for having a high Asian population, obviously I will miss the Asian households outside that district, who may have very different characteristics. However, there are well-known methods for avoiding this problem.
One common method, discussed in MASS, is pi-weighting (Horvitz-Thompson): make sure that everybody (or almost everybody) has some chance of selection, and then factor in that probability of selection when analysing the results. For instance, I might sample 1/100th of the households in District A, known to have a relatively high Asian population, but only 1/500th of the households in District B. The results are then "weighted up" so that each household sampled from District B has 5x as much influence on the results as each household from District A, cancelling out the bias that we would otherwise introduce by sampling more heavily from A.
Here is a peer-reviewed paper that I co-authored some years back, discussing an example of how these concepts were applied in designing an Australian survey, if anybody wants to see a bit more mathematical crunch. (Caveat that this is a design paper, which uses Horvitz-Thompson as a simplified version of the weighting method in order to keep the maths manageable; the actual data analysis post-survey used a somewhat more complex benchmarking approach.) A couple of things to take away from that:
- Even a simplification of the method makes for a very dense paper that takes a long time to write and will probably only be read by a handful of people. I would love it if everybody published the full details of their weighting methods, but it may not be easy to find the time.
- Nevertheless, these methods are well known and used by people who do socioeconomic research - we weren't the first and won't be the last.
The Boston survey introduces an additional complication: multiple "frames" meaning that some households may have more than one path to selection (e.g. some households might be on a list of names and in a targeted zip code). Even when a household doesn't actually get selected twice, this affects probability of selection and needs to be factored into the analysis. But again, this is a well-known issue and there are standard methods for dealing with it - see Särndal et al on "multiple frames".
The paper mentions that weighting was used, but doesn't go into much detail. Without more detail on the methods used, obviously I can't verify for myself that the researchers have handled these issues correctly. But I also don't see anything to signal that they didn't.
The acknowledgements mention methodological advice from Marcin Hitczenko and Kobi Abayomi and review by Tatjana Meschede, among others. From a quick look at their CVs, I would be astonished if those reviewers/contributors were unaware of this kind of issue or the standard methods for dealing with it.
Is the sample size large enough?
This is addressed at several points in the paper. The paper reports statistical significance thresholds (p-values), which are effectively a statistician's way of answering the question "given the sample sizes, how unlikely is it that random noise alone would be able to create this apparent difference?"
In most cases, the answer as reported in the paper is "pretty unlikely". In particular, Table 9 shows that the median net worth for White households is higher than every other group at the 99% significance threshold, excepting Asian and Cape Verdeans where the sample sizes were too small.
The sample sizes for individual racial groups are not huge (78 White households, 71 US Black). In general, a larger sample gives more accurate estimates, but the question of "how much is enough?" depends very much on the specifics of the problem: what the variable of interest is, how it's distributed among the populations of interest, how it is to be estimated, etc. etc.
As the paper acknowledges, this can make it hard to detect meaningful differences even when they do exist, and they have flagged some cases/groups where sample sizes were too small to be useful, especially when trying to break results down by age.
But I'm not aware of any statistical rule that says it's impossible to obtain serviceable estimates for a comparison between samples of 78 and 71, in this particular context, and I very much doubt it exists.
In other words: the authors have asserted that the sample size is large enough to be pretty confident that White households have higher net worth than Black households. They haven't shown the full working for that, so you'll have to make your own judgements about whether you trust their professional chops, but it is what they're asserting.
As noted in Bryan Krause's answer, the paper doesn't give quantiles other than the median, which makes it difficult to interpret exactly how meaningful the difference is. I think it almost certainly is meaningful and large - one would have to hypothesise some very weird income distributions for it to be otherwise - but without that IQR-type data it's impossible to be sure.
Possibility of non-response bias
The paper reports:
For the NASCC project in general, about 70,000 personalized advanced letters were sent, 87,000 telephone
numbers dialed 448,000 times, and 12,113 interviewer hours were spent
across three shops to conduct 2,746 completed surveys.
This is pretty common for research of this kind. Some people are hard to contact, some don't want to be interviewed or just don't have the time.
This is a perennial concern for survey-based researchers. Obviously it reduces the sample size, but that can be factored in; if you know only 5% of households are going to respond, you budget for 20x as many contact attempts.
The challenging part is estimating what it does to the selection probabilities. Working singles are harder to contact than stay-at-home parents or retirees, and each of those groups will have different selection probabilities.
One approach to this problem, discussed in MASS and elsewhere, is benchmarking, aka post-stratification: you compare the demographic distribution of your sample to some estimates of what the population demographics should be (e.g. Census data), and use that to weight up the groups that have effectively been undersampled due to non-response. There's also regression weighting, an extension of the same principle, and various other methods.
The Boston paper appears to have done something along those lines:
The statistics in the sample used weights based on family
characteristics in the U.S. Census Bureau’s American Community Survey
to generate results representative of specific ethnic group characteristics in the respondent’s metropolitan area of residence. Overall,
the results computed from the unweighted NASCC sample are not
dissimilar from those using the weight- ed NASCC sample, suggesting
that the specific ethnic group observations in the metropolitan
areas covered by the study were fairly representative of their
populations at large.
It's always tough to tell exactly how effective post-strat methods are in neutralising non-response bias, but they seem to have done what they could.
Overall, the report is a bit light on detail for my tastes. It would be nice to have more information about selection/weighting methods and on the distribution of net worth beyond just the medians. But what has been published looks consistent with competent researchers following standard and defensible methods for this kind of work.