Breakdown FB Prophet: Forecasting at Scale

Research paper by: Sean Taylor & Benjamin Letham (Facebook)

For our final blog post of our bootcamp we have been tasked with reading a research paper and articulating it to a non-technical stakeholder. For this task I have chosen a time series forecasting breakdown of Facebook Prophet which can be found here.

To start, this study was conducted due to the rarity of analysts having domain knowledge and the complex training of time series modeling. To combat these challenges the research paper describes practical approaches to forecasting ‘at scale’ based on an ‘analyst-in-the-loop’ approach. To break it down further, the study utilizes machine learning to perform the heavy lifting with analysts interpreting the results. The models will then allow for the analyst to make adjustments to the forecast with strictly the need of domain knowledge eliminating the necessity for forecasting expertise. The final notion of these models is to identify poor performing forecasts and flag them for manual review. See below image for representation of the workflow:

The study observed two main themes in the practice of creating business forecasts being:

1. Completely automated forecasting techniques can be hard to tune
2. The analyst tasked with making these models usually has deep domain knowledge but not the needed training to perform the actual modeling

Unfortunately, this leads to the rarity of an analyst who can produce high quality forecasts since it is a specialized skill requiring experience within ones domain. With these two concepts in mind, the study moved forward with three notions of scale that we will begin to dive into below.

A brief insight into time series modeling, a few of the main factors that effect business time series models are seasonality, holidays, and weekly/yearly cycles. All of these factors effect our time series models differently. For example, exponential smoothing and seasonal naive forecasts capture weekly seasonality but will miss longer-term seasonality. So, if one wants to adjust the model to accommodate for a known issue you will have to have an understanding of the underlying framework. This, again, will fall back to the issue of an analyst not having the proper training or knowledge of the underlying factors of a time series model. Fortunately the Facebook Prophet Model starts to accommodate some of these factors. The model takes in three main components:

Trends — non-periodic changes — The events that do not repeat themselves after regular interval of time. I.E. natural disasters or sneezing

Seasonality — periodic changes — Events that repeat themselves after a fixed period of time and whose occurrence can be predicted. I.E. seasons or day & night

Holidays — self explanatory though they do occur on irregular schedules

Additionally there are a few other technical accommodations within the model which are not needed to be mentioned here. In short the model has a few practical advantages worth noting:

Flexibility — can easily accommodate seasonality and allow the analyst to make assumptions about the trends

Fast — allowing the analyst to interactively explore many model specifications

Interpretable Parameters — this allows the analyst to impose assumptions on the forecast. Also, most analysts have experience with regression and are easily able to extend the model to include new components

As mentioned above the study moved forward with three notions of scale. We will begin diving into these below. The first two notions are:

1. Large number of people making forecasts, possibly without training in time series methods
2. Large variety of forecasting problems with potentially idiosyncratic features

With these two ideas in mind, the study built a model allowing for analyst to alter the time series model without knowledge of the underlying time series factors. The final model allows for analysts to modify capacities, changepoints, holidays/seasonality, smoothing parameters. Below is a breakdown of each item the analyst can modify:

• Capacities — the analyst can directly apply their exogenous data for the total market size.
• Changepoints — the analyst can modify trends or non-periodic changes such as product release dates
• Holidays/Seasonality — analysts with knowledge of how holidays or seasons effect their domain can modify the model by inputing dates and time scales of seasonality
• Smoothing parameters — this feature allows for analysts to make modifications to the seasonality and holiday parameters allowing them to tell the model how much of the historical seasonal variation is expected

Using good visualization tools an analyst can manipulate the above parameters to improve the forecasting model. The ‘analyst-in-the-loop’ approach relies heavily on good visualization tools and the automatic evaluation of forecasting quality which brings us to our third and final notion of scale defined as:

• Large number of forecasts will be created, necessitating efficient, automated means of evaluating and comparing them, as well as detecting when they are likely to be performing poorly

This final scale is in regards to automating the evaluation of forecasts. This is done through establishing baseline forecasts, modeling forecast accuracy, simulating historical forecasts, and identifying large forecast errors.

The initial baseline forecast is used for comparison against future models. It is import that these models remain simple therefore easily interpretable. The key with forecast accuracy is it allows end users to verify the eligibility of their forecast through some form of error. This allows the end user to verify the forecasts integrity and determine the use of the forecast, if they decide to use it.

The third portion to the automation is simulating historical forecasts (SHFs). SHF’s simulate the errors the forecast would have made in the past allowing easy interpretability for analysts and decision makers into forecasting errors. There are two issues that should be noted in regards to SHF’s:

• The more simulated forecasts you make the less useful the information will be. Also, it will slow down the time for forecasting evaluations.
• Forecasting can perform better or worse with more data.

The final aspect for the automation is the ability to identify large forecasting errors. When production of forecasts is high and the analysts time is at a premium it is important to flag bad forecasts automatically allowing the analyst to determine the needed adjustments to correct the issue. Through SHF’s we can help identify these poor forecasts. A few examples are:

• When the forecast has large errors relative to the baselines, the model may be misspecified. Analysts can adjust the trend model or the seasonality, as needed.
• Large errors for all methods on a particular date are suggestive of outliers. Analysts can identify outliers and remove them. Note: Outliers are essentially one offs or out of the norm of the data. These have a negative effect on modeling predictions.
• When the SHF error for a method increases sharply from one cutoff to the next, it could indicate that the data generating process has changed. Adding changepoints or modeling different phases separately may address the issue.

To conclude, this research enables analysts to be highly efficient by tuning poor performing models, identified through SHF’s, or simply create a new model if they feel the model is out of touch. The ability to use technology to perform the heavy lifting, removing the need of intense forecasting knowledge, and allowing untrained personnel to tune their forecasts will allow businesses to allocate their resources in a more effective way in-turn, hopefully, increasing revenue.