Why so gloomy? Structural break in sentiment in Michigan?

Ryan Cummings and Ernie Tedeschi published a very interesting article today in BriefingBook that sheds new light on the discrepancy between measured sentiment and traditional macroeconomic indicators. Cummings and Tedeschi document how the move to online sampling changed the characteristics of the University of Michigan’s Economic Sentiment Series.

…we believe that online respondents are causing significant declines in overall sentiment and current conditions indices, causing the more recent UMich data to be inconsistent with data prior to April 2024. Specifically, we use a simple statistical model to estimate that the effect of the methodological shift from telephone to online currently results in sentiment being 8.9 index points—or more than 11 percent—lower than it would have been if if interviews were still conducted by telephone. .

To demonstrate that the move to online surveys has led to a structural shift in the UMich series, they compare it to the Morning Consult series.

Source: Cummings and Tedeschi (2024).

The Morning Consult poll was conducted online, so it serves as a control. Hence the key reason for the change in measured sentiment.

A quick look at the weighted age distribution of the telephone and online assignments shows that no, in practice they were not randomized. For example, during these transition months, online respondents were more likely to be older, which may have influenced their sentiment. …

A more formal multilevel logit model, which measures the probability of a person belonging to a particular age cohort after controlling for other demographic factors, confirms this distinction. Respondents 65+, for example, had a 52 percent chance of being in an online group in April-June, which is more than twice as high as respondents 18-24 years old, and this difference is statistically significant…

I don’t know how the greater presence of older respondents interacts with the partisan divide discussed in this post (are Republican/lean Republican voters older than the corresponding Democratic/thin Democrat respondents?)

What I can say is that the adjusted Cummings-Tedeschi series shows less evidence of structural break than the official UMichigan series when using unemployment, annual CPI inflation, and the San Francisco Fed sentiment index as regressors.

Consider these two regressions: the first with the official series, and the second with the adjusted Cummings-Tedeschi series:

Now consider the corresponding 1-step forward Chow recursive tests for stability.

Note that while both regressions show instability between 2020 and 2023, the adjusted Cummings-Tedeschi series shows no structural break in 2024 due to the shift to online surveys.

However, changes in survey methods do not fully explain the gap between observed indicators and sentiment. I estimate the regression for the period 2016–2024M03 and make out-of-sample forecasts for 2024M04–M10.

Figure 1: University of Michigan Consumer Sentiment (bold black), adjusted Cummings-Tedeschi series (blue), fitted (red), +/- one standard error (gray lines). NBER has identified peak-to-trough dates for recessions, shaded in grey. Source: University of Michigan via FRED, BriefingBook, NBER and author’s calculations.

Thus, perhaps less than half of the gap is explained by changes in survey methods.