{"id":18953,"date":"2022-05-25T16:35:30","date_gmt":"2022-05-25T14:35:30","guid":{"rendered":"https:\/\/quantpedia.com\/?p=18953"},"modified":"2025-06-04T14:04:21","modified_gmt":"2025-06-04T12:04:21","slug":"extending-historical-daily-commodities-data-to-100-years","status":"publish","type":"post","link":"https:\/\/vvv.quantpedia.com\/es\/extending-historical-daily-commodities-data-to-100-years\/","title":{"rendered":"Extending Historical Daily Commodities Data to 100 Years"},"content":{"rendered":"

100 Years of Daily Data<\/strong><\/span><\/h2>\n\n\n\n

Finding a high-quality data<\/a> source is crucial for quantitative trading strategies<\/a>. Also, having a long history is beneficial. Fama & French<\/a>, for example, offer free historical data for stocks and a variety of factors. However, it is very hard to get good-quality and free data for other asset classes. For this reason, we have already examined how to extend historical daily bond data to 100 years<\/a>.<\/strong><\/p>\n\n\n\n

For any event-driven analysis<\/a> or to perform stress tests of various historical situations, long-enough data can only help. Whether one wants to analyze past market patterns<\/a>, or simply examine the risk of their portfolio<\/a> under different historical scenarios, the use case for long data is pretty straightforward.<\/strong><\/p>\n\n\n\n

Following the theme of our previous article, we decided to extend historical data<\/a> of another asset class, commodities. This article explains our commodity data methodology and introduces data sources, which helped us extend historical daily commodities data to 100 years.<\/strong><\/p>\n\n\n\n

Commodity data and data sources<\/strong><\/span><\/h2>\n\n\n\n

To create the 100-year daily history of commodities, we combined multiple data sources:<\/p>\n\n\n\n

1926 \u2013 1979: Monthly PPI (Producer Price Index by Commodity: All Commodities)<\/strong><\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Source: https:\/\/fred.stlouisfed.org<\/a><\/p>\n\n\n\n

1980 \u2013 2006: S&P GSCI Commodity Total Return (SPGSCITR)<\/strong><\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Source: https:\/\/www.investing.com<\/a><\/p>\n\n\n\n

2006 \u2013 2022: DBC ETF (Invesco DB Commodity Index Tracking Fund)<\/strong><\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Source: https:\/\/finance.yahoo.com<\/a><\/p>\n\n\n\n

How to Model Historical Commodity Data <\/strong><\/span><\/h2>\n\n\n\n

Ideally, one would have a Commodity ETF data<\/a> or a Commodity Index data<\/a> available. That\u2019s exactly what we have available since 1980. That being said, there are no freely available daily commodity data<\/a> before 1980. Thus, we had to find some proxy for the price of commodities.<\/p>\n\n\n\n

Producer Price Indexes<\/strong> (PPIs) measure changes in prices from a seller’s perspective. PPIs are available for a variety of products from almost any industry, including commodities and for quite a long data history. For this reason, we used PPIACO (Producer Price Index by Commodity: All Commodities) as our commodity proxy from 1926 to 1979.<\/p>\n\n\n\n

PPI is a good proxy, but not perfect<\/strong>. We can immediately observe some flaws, like probably a too low beta to traditional commodity indices or a too smooth curve, i.e. lacking any volatility<\/a>. However, both of these flaws we are able to fix.<\/p>\n\n\n\n

Beta Adjustment<\/strong><\/span><\/h3>\n\n\n\n

Firstly, we adjusted the PPI index to account for the correct commodity prices beta<\/a>. We measured the beta of the PPI to commodity price index on newer history. The commodity price index is publicly available for free only since 1980, thus we had to perform the beta adjustment for the entire period 1926-1980.<\/p>\n\n\n\n

Our calculated beta of PPI to Commodity index was around 1\/3<\/strong>. This means, we had to adjust PPI by the multiple of 3 to arrive at the commodity beta. We then simply multiplied the daily change of PPIACO by 3 to obtain a more accurate commodity proxy.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Transforming Monthly Data into Daily Data<\/strong><\/span><\/h2>\n\n\n\n

As mentioned above, the second flaw in our data is the lack of volatility. This is primarily caused by the PPI being published only on a monthly frequency. We already explained our methodology of creating daily volatility for monthly data<\/a> in our recent article. We called it the Volatility Proxy Extrapolation<\/strong>. In very simple words \u2013 we take the volatility from the good proxy and apply it in between monthly data points of the original monthly data series (PPI in our case).<\/p>\n\n\n\n

Commodity volatility proxy<\/strong><\/span><\/h3>\n\n\n\n

It\u2019s not that straightforward and easy to find a viable proxy for daily commodity volatility before 1980. However, observing the recent commodity<\/a> behavior and also taking into account a composition of commodity indices in the past (heavily weighted towards Energy), we do assume that a good proxy for commodity volatility is the equity Energy sector<\/strong>. More precisely, an excess return of the Energy sector vs entire Market.<\/p>\n\n\n\n

Thus, firstly, we calculated the spread of the Energy sector (from F&F 12 Industry Portfolios<\/a>) against the market (from F&F 3 Factor<\/a>). This is our daily commodity volatility proxy.<\/p>\n\n\n\n

How to create a daily volatility for monthly data<\/strong><\/span><\/h3>\n\n\n\n

After calculating the Energy sector spread, we then followed our methodology outlined in Extending Historical Daily Bond Data to 100 Years<\/a>. We calculated the monthly cumulative difference of the spread, subtracted it from the monthly cumulative difference of 3x PPI and divided it by the number of trading days in the respective month. Thus, getting the daily volatility. The last step was to calculate the cumulative daily return of the commodities.<\/p>\n\n\n\n

The methodology is probably easily understood by the simple chart below. We basically copy the daily volatility from the Energy \u2013 Market spread and plug it in between two monthly data points of 3x PPI. And we ensure there are no jumps or gaps in data and that everything happens in a linear fashion. See the chart below.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

Combining Various Commodity Data Sources<\/strong><\/span><\/h2>\n\n\n\n

Now that we successfully created daily data series from 1926 to 1979, we need to take one last step. We finally combined the three periods of different calculations and data sources into one time series:<\/p>\n\n\n\n