Check out this piece on automated time-series forecasting at Google. It's a fun and quick read. Several aspects are noteworthy.

On the upside:

-- Forecast combination features prominently -- they combine forecasts from an ensemble of models.

-- Uncertainty is acknowledged -- they produce interval forecasts, not just point forecasts.

On the downside:

-- There's little to their approach that wasn't well known and widely used in econometrics a quarter century ago (or more). Might not something like Autobox, which has been around and evolving since the 1970's, do as well or better?

## Thursday, April 20, 2017

## Friday, April 14, 2017

### On Pseudo Out-of-Sample Model Selection

Great to see that Hirano and Wright (HW), "Forecasting with Model Uncertainty", finally came out in

HW make two key contributions. First, they characterize rigorously the source of the inefficiency in forecast model selection by pseudo out-of-sample methods (expanding-sample, split-sample, ...), adding invaluable precision to more intuitive discussions like Diebold (2015). (Ungated working paper version here.) Second, and very constructively, they show that certain simulation-based estimators (including bagging) can considerably reduce, if not completely eliminate, the inefficiency.

Abstract: We consider forecasting with uncertainty about the choice of predictor variables. The researcher wants to select a model, estimate the parameters, and use the parameter estimates for forecasting. We investigate the distributional properties of a number of different schemes for model choice and parameter estimation, including: in‐sample model selection using the Akaike information criterion; out‐of‐sample model selection; and splitting the data into subsamples for model selection and parameter estimation. Using a weak‐predictor local asymptotic scheme, we provide a representation result that facilitates comparison of the distributional properties of the procedures and their associated forecast risks. This representation isolates the source of inefficiency in some of these procedures. We develop a simulation procedure that improves the accuracy of the out‐of‐sample and split‐sample methods uniformly over the local parameter space. We also examine how bootstrap aggregation (bagging) affects the local asymptotic risk of the estimators and their associated forecasts. Numerically, we find that for many values of the local parameter, the out‐of‐sample and split‐sample schemes perform poorly if implemented in the conventional way. But they perform well, if implemented in conjunction with our risk‐reduction method or bagging.

*Econometrica*. (Ungated working paper version here.)HW make two key contributions. First, they characterize rigorously the source of the inefficiency in forecast model selection by pseudo out-of-sample methods (expanding-sample, split-sample, ...), adding invaluable precision to more intuitive discussions like Diebold (2015). (Ungated working paper version here.) Second, and very constructively, they show that certain simulation-based estimators (including bagging) can considerably reduce, if not completely eliminate, the inefficiency.

Abstract: We consider forecasting with uncertainty about the choice of predictor variables. The researcher wants to select a model, estimate the parameters, and use the parameter estimates for forecasting. We investigate the distributional properties of a number of different schemes for model choice and parameter estimation, including: in‐sample model selection using the Akaike information criterion; out‐of‐sample model selection; and splitting the data into subsamples for model selection and parameter estimation. Using a weak‐predictor local asymptotic scheme, we provide a representation result that facilitates comparison of the distributional properties of the procedures and their associated forecast risks. This representation isolates the source of inefficiency in some of these procedures. We develop a simulation procedure that improves the accuracy of the out‐of‐sample and split‐sample methods uniformly over the local parameter space. We also examine how bootstrap aggregation (bagging) affects the local asymptotic risk of the estimators and their associated forecasts. Numerically, we find that for many values of the local parameter, the out‐of‐sample and split‐sample schemes perform poorly if implemented in the conventional way. But they perform well, if implemented in conjunction with our risk‐reduction method or bagging.

## Monday, April 10, 2017

### BIg Data, Machine Learning, and the Macroeconomy

Coming soon at Bank of Norway:

CALL FOR PAPERS

Big data, machine learning and the macroeconomy

Norges Bank, Oslo, 2-3 October 2017

Data, in both structured and unstructured form, are becoming easily available on an ever increasing scale. To find patterns and make predictions using such big data, machine learning techniques have proven to be extremely valuable in a wide variety of fields. This conference aims to gather researchers using machine learning and big data to answer challenges relevant for central banking.

Examples of questions, and topics, of interest are:

Forecasting applications and methods

-Can better predictive performance of key economic aggregates (GDP, inflation, etc.) be achieved by using alternative data sources?

- Does the machine learning tool-kit add value to already well-established forecasting frameworks used at central banks?

Causal effects

- How can new sources of data and methods be used learn about the causal mechanism underlying economic fluctuations?

Text as data

- Communication is at the heart of modern central banking. How does this affect markets?

- How can textual data be linked to economic concepts like uncertainty, news, and sentiment?

Confirmed keynote speakers are:

- Victor Chernozhukov (MIT)

- Matt Taddy (Microsoft, Chicago Booth)

The conference will feature 10-12 papers. If you would like to present a paper, please send a draft or an extended abstract to mlconference@norges-bank.no by 31 July 2017. Authors of accepted papers will be notified by 15 August. For other questions regarding this conference, please send an e-mail to mlconference@norges-bank.no. Conference organizers are Vegard H. Larsen and Leif Anders Thorsrud.

CALL FOR PAPERS

Big data, machine learning and the macroeconomy

Norges Bank, Oslo, 2-3 October 2017

Data, in both structured and unstructured form, are becoming easily available on an ever increasing scale. To find patterns and make predictions using such big data, machine learning techniques have proven to be extremely valuable in a wide variety of fields. This conference aims to gather researchers using machine learning and big data to answer challenges relevant for central banking.

Examples of questions, and topics, of interest are:

Forecasting applications and methods

-Can better predictive performance of key economic aggregates (GDP, inflation, etc.) be achieved by using alternative data sources?

- Does the machine learning tool-kit add value to already well-established forecasting frameworks used at central banks?

Causal effects

- How can new sources of data and methods be used learn about the causal mechanism underlying economic fluctuations?

Text as data

- Communication is at the heart of modern central banking. How does this affect markets?

- How can textual data be linked to economic concepts like uncertainty, news, and sentiment?

Confirmed keynote speakers are:

- Victor Chernozhukov (MIT)

- Matt Taddy (Microsoft, Chicago Booth)

The conference will feature 10-12 papers. If you would like to present a paper, please send a draft or an extended abstract to mlconference@norges-bank.no by 31 July 2017. Authors of accepted papers will be notified by 15 August. For other questions regarding this conference, please send an e-mail to mlconference@norges-bank.no. Conference organizers are Vegard H. Larsen and Leif Anders Thorsrud.

### 13th Annual Real-Time Conference

Great news: The Bank of Spain will sponsor the 13th annual conference on real-time data analysis, methods, and applications in macroeconomics and finance, next October 19th and 20th , 2017, in its central headquarters in Madrid, c/ AlcalĂˇ, 48. The real-time conference has always been unique and valuable. Very happy to see the Bank of Spain confirming and promoting its continued vitality.

More information and call for papers here.

Topics include:

• Nowcasting, forecasting and real-time monitoring of macroeconomic and financial conditions.

• The use of real-time data in policy formulation and analysis.

• New real-time macroeconomic and financial databases.

• Real-time modeling and forecasting aspects of high-frequency financial data.

• Survey data, and its use in macro model analysis and evaluation.

• Evaluation of data revision and real-time forecasts, including point forecasts, probability forecasts, density forecasts, risk assessments and decompositions.

More information and call for papers here.

Topics include:

• Nowcasting, forecasting and real-time monitoring of macroeconomic and financial conditions.

• The use of real-time data in policy formulation and analysis.

• New real-time macroeconomic and financial databases.

• Real-time modeling and forecasting aspects of high-frequency financial data.

• Survey data, and its use in macro model analysis and evaluation.

• Evaluation of data revision and real-time forecasts, including point forecasts, probability forecasts, density forecasts, risk assessments and decompositions.

## Monday, April 3, 2017

### The Latest on the "File Drawer Problem"

The term "file drawer problem" was coined long ago. It refers to the bias in published empirical studies toward "large", or "significant", or "good" estimates. That is, "small"/"insignificant"/"bad" estimates remain unpublished, in file drawers (or, in modern times, on hard drives). Correcting the bias is a tough nut to crack, since little is known about the nature or number of unpublished studies. For the latest, together with references to the relevant earlier literature, see the interesting new NBER working paper, IDENTIFICATION OF AND CORRECTION FOR PUBLICATION BIAS, by Isaiah AndrewsMaximilian Kasy. There's an ungated version and appendix here, and a nice set of slides here.

Abstract: Some empirical results are more likely to be published than others. Such selective publication leads to biased estimators and distorted inference. This paper proposes two approaches for identifying the conditional probability of publication as a function of a study's results, the first based on systematic replication studies and the second based on meta-studies. For known conditional publication probabilities, we propose median-unbiased estimators and associated confidence sets that correct for selective publication. We apply our methods to recent large-scale replication studies in experimental economics and psychology, and to meta-studies of the effects of minimum wages and de-worming programs.

Abstract: Some empirical results are more likely to be published than others. Such selective publication leads to biased estimators and distorted inference. This paper proposes two approaches for identifying the conditional probability of publication as a function of a study's results, the first based on systematic replication studies and the second based on meta-studies. For known conditional publication probabilities, we propose median-unbiased estimators and associated confidence sets that correct for selective publication. We apply our methods to recent large-scale replication studies in experimental economics and psychology, and to meta-studies of the effects of minimum wages and de-worming programs.

## Tuesday, March 28, 2017

### Text as Data

"Text as data" is a vibrant and by now well-established field. (Just Google "text as data".)

For an informative overview geared toward econometricians, see the new paper, "Text as Data" by Matthew Gentzkow, Bryan T. Kelly, and Matt Taddy (GKT). (Ungated version here.)

"Text as data" has wide applications in economics. As GKT note:

There are three key steps:

1. Represent the raw text D as a numerical array x

2. Map x into predicted values yhat of outcomes y

3. Use yhat in subsequent descriptive or causal analysis.

GKT emphasize the ultra-high dimensionality inherent in statistical text analyses, with connections to machine learning, etc.

For an informative overview geared toward econometricians, see the new paper, "Text as Data" by Matthew Gentzkow, Bryan T. Kelly, and Matt Taddy (GKT). (Ungated version here.)

"Text as data" has wide applications in economics. As GKT note:

... in finance, text from financial news, social media, and company filings is used to predict asset price movements and study the causal impact of new information. In macroeconomics, text is used to forecast variation in inflation and unemployment, and estimate the effects of policy uncertainty. In media economics, text from news and social media is used to study the drivers and effects of political slant. In industrial organization and marketing, text from advertisements and product reviews is used to study the drivers of consumer decision making. In political economy, text from politicians’ speeches is used to study the dynamics of political agendas and debate.

There are three key steps:

1. Represent the raw text D as a numerical array x

2. Map x into predicted values yhat of outcomes y

3. Use yhat in subsequent descriptive or causal analysis.

GKT emphasize the ultra-high dimensionality inherent in statistical text analyses, with connections to machine learning, etc.

## Tuesday, March 21, 2017

### Forecasting and "As-If" Discounting

Check out the fascinating and creative new paper, "Myopia and Discounting", by Xavier Gabaix and David Laibson.

From their abstract (slightly edited):

Note that in the Gabaix-Laibson environment everything depends on how forecast error variance behaves as a function of forecast horizon \(h\). But we know a lot about that. For example, in linear covariance-stationary \(I(0)\) environments, optimal forecast error variance grows with \(h\) at a decreasing rate, approaching the unconditional variance from below. Hence it cannot grow linearly with \(h\), which is what produces hyperbolic as-if discounting. In contrast, in non-stationary \(I(1)\) environments, optimal forecast error variance

From their abstract (slightly edited):

We assume that perfectly patient agents estimate the value of future events by generating noisy, unbiased simulations and combining those signals with priors to form posteriors. These posterior expectations exhibit as-if discounting: agents make choices as if they were maximizing a stream of known utils weighted by a discount function. This as-if discount function reflects the fact that estimated utils are a combination of signals and priors, so average expectations are optimally shaded toward the mean of the prior distribution, generating behavior that partially mimics the properties of classical time preferences. When the simulation noise has variance that is linear in the event's horizon, the as-if discount function is hyperbolic.Among other things, then, they provide a rational foundation for the "myopia" associated with hyperbolic discounting.

Note that in the Gabaix-Laibson environment everything depends on how forecast error variance behaves as a function of forecast horizon \(h\). But we know a lot about that. For example, in linear covariance-stationary \(I(0)\) environments, optimal forecast error variance grows with \(h\) at a decreasing rate, approaching the unconditional variance from below. Hence it cannot grow linearly with \(h\), which is what produces hyperbolic as-if discounting. In contrast, in non-stationary \(I(1)\) environments, optimal forecast error variance

*does*eventually grow linearly with \(h\). In a random walk, for example, \(h\)-step-ahead optimal forecast error variance is just \(h \sigma^2\), where \( \sigma^2\) is the innovation variance. It would be fascinating to put people in \(I(1)\) vs. \(I(0)\) laboratory environments and see if hyperbolic as-if discounting arises in \(I(1)\) cases but not in \(I(0)\) cases.## Sunday, March 19, 2017

### ML and Metrics VIII: The New Predictive Econometric Modeling

[Click on "Machine Learning" at right for earlier "Machine Learning and Econometrics" posts.]

We econometricians need -- and have always had -- cross section and time series ("micro econometrics" and "macro/financial econometrics"), causal estimation and predictive modeling, structural and non-structural. And all continue to thrive.

But there's a new twist, happening now, making this an unusually exciting time in econometrics. Predictive econometric modeling is not only alive and well, but also blossoming anew, this time at the interface of micro-econometrics and machine learning. A fine example is the new Kleinberg, Lakkaraju, Leskovic, Ludwig and Mullainathan paper, “Human Decisions and Machine Predictions”, NBER Working Paper 23180 (February 2017).

Good predictions promote good decisions, and econometrics is ultimately about helping people to make good decisions. Hence the new developments, driven by advances in machine learning, are most welcome contributions to a long and distinguished predictive econometric modeling tradition.

We econometricians need -- and have always had -- cross section and time series ("micro econometrics" and "macro/financial econometrics"), causal estimation and predictive modeling, structural and non-structural. And all continue to thrive.

But there's a new twist, happening now, making this an unusually exciting time in econometrics. Predictive econometric modeling is not only alive and well, but also blossoming anew, this time at the interface of micro-econometrics and machine learning. A fine example is the new Kleinberg, Lakkaraju, Leskovic, Ludwig and Mullainathan paper, “Human Decisions and Machine Predictions”, NBER Working Paper 23180 (February 2017).

Good predictions promote good decisions, and econometrics is ultimately about helping people to make good decisions. Hence the new developments, driven by advances in machine learning, are most welcome contributions to a long and distinguished predictive econometric modeling tradition.

## Monday, March 13, 2017

### ML and Metrics VII: Cross-Section Non-Linearities

[Click on "Machine Learning" at right for earlier "Machine Learning and Econometrics" posts.]

The predictive modeling perspective needs not only to be respected and embraced in econometrics (as it routinely

The predictive modeling perspective needs not only to be respected and embraced in econometrics (as it routinely

*is*, notwithstanding the Angrist-Pischke revisionist agenda), but also to be*enhanced*by incorporating elements of statistical machine learning (ML). This is particularly true for cross-section econometrics insofar as time-series econometrics is already well ahead in that regard. For example, although flexible non-parametric ML approaches to estimating conditional-mean functions don't add much to time-series econometrics, they may add lots to cross-section econometric regression and classification analyses, where conditional mean functions may be highly nonlinear for a variety of reasons. Of course econometricians are well aware of traditional non-parametric issues/approaches, especially kernel and series methods, and they have made many contributions, but there's still much more to be learned from ML.## Monday, March 6, 2017

### ML and Metrics VI: A Key Difference Between ML and TS Econometrics

[Click on "Machine Learning" at right for earlier "Machine Learning and Econometrics" posts.]

Continuing:

So then, statistical machine learning (ML) and time series econometrics (TS) have lots in common. But there's also an interesting difference: ML's emphasis on flexible nonparametric modeling of conditional-mean nonlinearity doesn't play a big role in TS.

Of course there are the traditional TS conditional-mean nonlinearities: smooth non-linear trends, seasonal shifts, and so on. But there's very little evidence of important conditional-mean nonlinearity in the covariance-stationary (de-trended, de-seasonalized) dynamics of most economic time series. Not that people haven't tried hard -- really hard -- to find it, with nearest neighbors, neural nets, random forests, and lots more.

So it's no accident that things like linear autoregressions remain overwhelmingly dominant in TS. Indeed I can think of only one type of conditional-mean nonlinearity that has emerged as repeatedly important for (at least some) economic time series: Hamilton-style Markov-switching dynamics.

[Of course there's a non-linear elephant in the room: Engle-style GARCH-type dynamics. They're tremendously important in financial econometrics, and sometimes also in macro-econometrics, but they're about conditional variances, not conditional means.]

So there are basically only two important non-linear models in TS, and only one of them speaks to conditional-mean dynamics. And crucially, they're both very tightly parametric, closely tailored to specialized features of economic and financial data.

Now let's step back and assemble things:

ML emphasizes approximating non-linear conditional-mean functions in highly-flexible non-parametric fashion. That turns out to be doubly unnecessary in TS: There's just not much conditional-mean non-linearity to worry about, and when there occasionally is, it's typically of a highly-specialized nature best approximated in highly-specialized (tightly-parametric) fashion.

Continuing:

So then, statistical machine learning (ML) and time series econometrics (TS) have lots in common. But there's also an interesting difference: ML's emphasis on flexible nonparametric modeling of conditional-mean nonlinearity doesn't play a big role in TS.

Of course there are the traditional TS conditional-mean nonlinearities: smooth non-linear trends, seasonal shifts, and so on. But there's very little evidence of important conditional-mean nonlinearity in the covariance-stationary (de-trended, de-seasonalized) dynamics of most economic time series. Not that people haven't tried hard -- really hard -- to find it, with nearest neighbors, neural nets, random forests, and lots more.

So it's no accident that things like linear autoregressions remain overwhelmingly dominant in TS. Indeed I can think of only one type of conditional-mean nonlinearity that has emerged as repeatedly important for (at least some) economic time series: Hamilton-style Markov-switching dynamics.

[Of course there's a non-linear elephant in the room: Engle-style GARCH-type dynamics. They're tremendously important in financial econometrics, and sometimes also in macro-econometrics, but they're about conditional variances, not conditional means.]

So there are basically only two important non-linear models in TS, and only one of them speaks to conditional-mean dynamics. And crucially, they're both very tightly parametric, closely tailored to specialized features of economic and financial data.

Now let's step back and assemble things:

ML emphasizes approximating non-linear conditional-mean functions in highly-flexible non-parametric fashion. That turns out to be doubly unnecessary in TS: There's just not much conditional-mean non-linearity to worry about, and when there occasionally is, it's typically of a highly-specialized nature best approximated in highly-specialized (tightly-parametric) fashion.

Subscribe to:
Posts (Atom)