Early Christmas Gift: Introducing Our New S&P500 Data Hub

After about six months since we first presented the idea of an S&P 500 Data Hub, we are pleased to release it today, just in time for the new year. This is something we have envisioned for a long time as the perfect complement to our S&P 500 offerings. In this post, I will recap why we created this and how we believe it can help investors. Then, I will provide an overview of how it works.

Why an S&P 500 Data Hub?

I've already covered this question in part in my most recent blog post, where I provided updates on the various projects we were working on. I might be repeating myself, but I still want to address this topic for the benefit of future new members who may read this post. If you've read my last post, feel free to jump straight to the next section.

The idea of creating a place where we could centralize all our data was there from the start. This is how we envisioned our tools being most effectively used. Many stock market research services on the web are based on personal opinions expressed through writing, but WealthUmbrella was always intended to focus on objectively analyzing data. Therefore, we needed a way to efficiently stream this data. Implementing such a data hub required a lot of coding, so we initially built our tools around TradingView, which I consider a very good platform that is relatively easy to use. However, we realized along the way that there were significant limitations with TradingView in the context of WU, ultimately making me the focal point of it all. To fully understand, you should know that WU subscribers are very diverse, ranging from 18 to 84 years old, from novice retail investors to multi-billion-dollar institutional investors, and from data junkies to those who hate graphs. About 40% of this diverse group doesn’t use TradingView, so they can only rely on my market updates to get a glimpse of what our indicators currently say about the market. Others do use TradingView but find themselves lost among all our metrics and unsure of how to interpret them. This led to one of our primary reasons for creating this Data Hub: to provide our members with a high-level dashboard where they can see a real-time interpretation of some of our key metrics. We call this our Simplified Data Hub, which is the front page of our new dashboard. This high-level interface will offer a straightforward assessment on a bullish to bearish scale of some of our most important metrics. These evaluations are based on complex algorithms designed to reflect my personal analysis. This will allow everyone to interpret the market at any time through the lens of our indicators. Although it might seem like I am replacing myself with a robot that Zackary and I have coded, that is not the case. I will continue to write market updates at the same frequency, but I am confident you will appreciate being able to get an update whenever you want. The challenge has always been to balance providing updates while minimizing noise. Therefore, I only write when I see significant changes in our metrics. But I also understand that if you don't see the metrics, you might not realize that everything is fine when we have a significant red candle or an euphoric green one.

The other two reasons are linked to TradingView itself and our desire to make it less central to our platform. Although I appreciate TradingView, it has several limitations as a data hub. One issue is that several datasets are only available at the end of the day, whereas it would be useful to have updates throughout the day or at least before the close. Some of this daily data also tends to be revised frequently, thus causing our indicators to be repainted—a topic I will discuss later in this post. Additionally, the availability of some data was sometimes limited and subject to change based on their commercial agreements, making us vulnerable to losing part of our strategy. This happened with the Dark Pool data last year, but it has also occurred in other instances where I woke up to find some of our indicators had stopped working, and I had to find a replacement dataset. This recently happened without you noticing with our NYSE and Nasdaq derivative volume; I had to find an alternative way to compute all the options volume per exchange after waking up to a non-working indicator.

Our final motivation for moving away from TradingView was to provide you with a place where you could see all our metrics simultaneously without having to load them one by one on several TradingView screens. Depending on your subscription level to TradingView, you may be limited to 1 or 3 charts per screen, which makes it hard to quickly and simultaneously view all our metrics. I have the highest subscription tier in TradingView, so I can add as many indicators as I want on a chart. But even then, it's challenging to have more than three at once and still maintain good visibility on the indicators. Therefore, we created a second page, where you can see all our indicators at once, but also zoom them in full page to examine the history in more detail.

A Brief Tutorial

To access our new Data Hub, you can visit the S&P 500 member page on WealthUmbrella, where you will find a link here :

or you can go directly to app.thewealthumbrella.com. The first time you visit this address, you may be redirected to the login page on our website. You will need to log in to WealthUmbrella only once for the initial access, as this will generate a unique token for your account. This token will automatically redirect you to our Data Hub. You can then bookmark the address that includes your unique token, as this will not change as long as you are a member of WealthUmbrella.

The first page you will encounter is what we call the Simplified Hub. Here, you will see a concise summary of our metrics along with an automated assessment. It will look like this:

The upper left box displays the current value of our Risk Index and the color level of our Margin Risk Indicator. Occasionally, you might see an arrow pointing up or down beneath the numerical value, indicating that the indicator is currently trending upward or downward as is the case of the Margin Risk Level in the above image. On days where the reading is stable, you will see an horizontal line as shown under the current Risk Index reading.

Upper Middle Box: This section presents the status of our Hedging Signal and Margin Signal.

Upper Right Box: This contains recent signals that have surpassed their specific thresholds. For instance, those who were with us in June 2023 might recall our Dark Pool indicator triggering an alert on the evening of Friday, June 16th, at 8 PM.

Middle Box: Here we provide the automated assessment of our signals. Our goal is to offer an evaluation on a bullish (green) to bearish (red) scale for each of our metrics, similar to how I analyze them when writing a market update. Alongside the current assessment, we also display the values from the previous day and five days ago. This historical data is invaluable as it quickly provides insight into the current trend of each indicator

Although this assessment appears straightforward and easy to read, it is actually driven by complex algorithms. These algorithms do not simply translate a feature of a curve; instead, they apply high-level conditions that take several factors into account to produce an assessment. This approach aims to ensure that the results closely mirror my own evaluations of a curve. For instance, unlike the CNN Fear and Greed Index, where an overly green reading might indicate a reversal, our assessments integrate such considerations in a way that a reversal point is more likely to occur in the yellow zone. A current example is our Market Breadth Indicator, which is now in the yellow zone.

If we focus solely on the current level of the underlying indicator, we find ourselves in an unusually low zone, significantly distant from our bearish threshold—a situation we haven't often encountered since the onset of this new bull market. By merely observing this level, one might incorrectly conclude that an extremely bullish stance is warranted.

The reason the assessment is yellow is because we are currently observing a considerable upward trend in that metric. We're not at a critical make-or-break point, but seeing market breadth degrade at a rapid pace is somewhat concerning, hence the yellow reading of that assessment. Similarly, our Option Model is currently at a two-year all-time high, yet our assessment is nearly in the yellow zone, as we consider such a high value likely to be close to a reversal point.

At the moment, we are satisfied with how this code performs based on historical data in comparison to how I would have judged the value of our indicators over the last eight years. However, given the complexity of the underlying code, it is possible that we may need to update it occasionally. Therefore, for the next few months, we will continue to closely monitor these assessments to ensure that they consistently align with my own judgment.

If you would like to view the actual graph of the underlying indicator used for these high-level assessments, you can click on the two diagonal arrows. This will open the graph in the last box of the page. Here is an example with our Option Model:

Detailed Mode

If you prefer to view the actual graphs of our various indicators rather than our automated assessments, simply click on the 'Detailed' icon in the upper toolbar. This action will direct you to a page that looks like this:

On this page, all the indicators we use are separated into three different boxes: one related to Options, one that covers those similar to the QQQ, and one last box that includes all indicators at the stock market level.

By clicking on the "I" symbol before the title of each indicator, you will receive a summary of what this indicator tracks. In a forthcoming update in 2025, this will also include a link to a blog post that explains the indicator more thoroughly. I hope this feature will enhance understanding of our tools.

As with the simplified view, clicking on the diagonal double arrow preceding each indicator title will open the indicator in full-page mode. In this mode, you can zoom in or out on the graph, similar to what you would do in Trading View. Currently, it is only possible to go back two years in time, but as we gain confidence in our database and cloud setup, we plan to rapidly extend access to a longer history.

When viewing an indicator in full-page mode, you can click the upper right "Show SPY" button to overlay the SPY data.

This will automatically overlay the SPY price action onto the chart.

This feature is particularly useful when you want to analyze our indicators' values in the context of the S&P 500's historical price action.

When examining the names of these indicators, one of the first things you should note is that we have revived our defunct QQQ Dark Pool indicator. Although this indicator has generated some false alerts, it was an essential part of our hedging algorithm, providing us with a crucial two-day lead on major market corrections. This might not seem significant, but considering how rapidly markets can sometimes decline, a two-day lead is not negligible. Furthermore, in cases where it sends false alerts, the impact is minimal, as historically, these have only resulted in very minor losses.

Refresh Rate

In the top right corner of our interface, you will see a date and time displayed in the New York (stock market) time zone. This indicates the last time you refreshed your data. Although we may implement automatic refreshes in the future, currently, you need to manually refresh the page to view updated data readings. On our side, we update our database with new data at a variable rate. For example, all our option-related metrics begin updating only at noon because morning readings don't provide meaningful insights due to the minimal amount of option contracts sold. I've observed the Option model spiking early in the day, only to return to normal values later, which unnecessarily caused me anxiety. Our NYSE and Nasdaq Derivative metrics were even worse, as their readings are directly related to the volume of contracts, which is negligible in the morning compared to the final daily value. Most other data will be updated hourly during the day, but every 15 minutes from 3:00 to 3:45 PM and every 5 minutes until the close. This increased frequency near the market's close is crucial as most of our strategies are based on daily candlesticks and are best applied near the end of the trading day.

But why not make it continuous? Firstly, despite the high cost of our data subscriptions, some limit us to only 50 API calls per day. We plan to upgrade our key data subscription packages as WealthUmbrella grows, but as I've mentioned before: institutional data are prohibitively expensive, and providing a real-time feed would cost more than all of WU's revenue. Moreover, some of our indicators take time to compute. For example, after failing to reach a reasonable agreement with a key data supplier for our market breadth indicator, we decided to compute it ourselves. This involves individually analyzing all Nasdaq and NYSE companies and applying our algorithm to these approximately 4,000 companies at each refresh rate.

Despite these limitations in refresh rate, the results are still generally lot better than those on Trading View. Some data there was only available after the close, or provided relatively poor readings during the close before settling later in the evening. This situation often caused our market breadth-related indicator to repaint during the evening or night. Although this was typically not a significant issue, it became problematic in August 2023 when the indicator hovered very close to the threshold for several days.

Difference Between Our Data Hub and TradingView Readings, and a Few Words About Repainting

In developing our Data Hub, we had to find data equivalent to what we used in TradingView and completely recode our tools in Python. This resulted in minor differences in indicator readings. We spent considerable time ensuring they were as similar as possible, but some differences persist, notably in our Market Breadth related indicator due to variations in data sources. While the numerical values may differ slightly, both the Trading View and Data Hub versions trigger on the same dates.

You may also notice minor differences in our Risk Index and Hedging signal, where they can sometimes trigger with a one-day difference. This isn't an error, but rather the result of using the flexibility our own code framework provides to circumvent certain limitations of TradingView. I will explain why, but this involves delving into a relatively complex topic. If you're not interested in the philosophical nuances of coding, you can stop here and trust us when I say that our new Data Hub strategy version is an improved version of our TradingView one, although both are very similar. However, if you're up for some "nerd stuff," this topic is fascinating for understanding how small changes can significantly impact an algorithm's performance and why knowing the design methodology is crucial before trusting any algorithm.

Complex Issues in Data Synchronization

One major issue in coding financial strategy is data synchronization. Data isn't always delivered in the same time zone or at the same frequency and delay. Some data are real-time, some, like the VIX on TradingView, are delayed by 15 minutes, and some are computed at close but reported with a delay of 1 to 5 hours. TradingView used to assume that any data delivered within a given day was available on that day—a risky assumption that could lead to strategies underperforming in real time.

For example, when Dark Pool data was available on TradingView, it was delivered around 8 PM. Initially, TradingView's Pine Script allowed you to use that day's Dark Pool data in backtesting as if it were available during trading hours, which wasn't realistic since the markets (and the after market!) were closed by 8 PM. Realizing this, the developers of Pine Script V3 changed the default behavior of their strategy testing. Now, when backtesting, they only consider data that was available at the close of the candle of your timeframe, which in our case corresponds to the close of the daily candle.

This means that even if the Dark Pool data for February 20, 2017, exceeds the threshold, TradingView now considers that this data arrived after the daily close. Therefore, the only possibility for acting on it would be on February 21, 2017. This approach makes much more sense! However, this change also introduced other challenges, especially for those of us who operate strategies on a daily timeframe. For instance, consider the VIX. We use the VIX under certain conditions in our strategy. For those who were using the IOFund version in 2022, you surely recall how crucial the VIX was in navigating the 2022 bear market. The VIX can fluctuate rapidly from one day to the next. However, because the data is reported with a 15-minute delay in TradingView, Pine Script now assumes that the daily VIX data is not available before the close, and thus, it uses the data from the previous day.This approach also doesn't make sense, particularly because the VIX can change dramatically in a short period. The value from the previous day might be significantly higher or lower than the current value. Conversely, even if at 3:59 PM you only have access to the VIX value at 3:44 PM due to the 15-minute delay, this value is likely much closer to the closing value than the data from the previous day. Therefore, it would be more accurate to backtest using the data from the same day. We have implemented this adjustment in TradingView, but the way this was handled has introduced other issues and unusual behaviors. This recently led to repainting of our strategy, necessitating an update to our Hedge strategy.

Just to illustrate the complexity of this issue, TradingView published an article in 2019 proposing a method to handle data synchronization. Over time, the comment section of this post became filled with examples from the community, highlighting issues with the suggested approach. Today, accessing this blog post, you'll see a prominent warning stating that the method presented is now deprecated due to causing bugs.

While there's more that could be discussed on this topic, it goes beyond the scope of this article. The key takeaway is that by working within our own data framework, and using data with slightly different timeframes (for example, the VIX is not delayed in our framework), we've been able to manage historical data in a way that realistically represents how we can utilize current data. This approach sometimes results in a one-day discrepancy with our hedge algorithm compared to its implementation in Trading View. Although this difference hasn't significantly impacted the returns for the Risk Index, it has slightly improved the performance of the hedge strategy.

Conclusion

We are happy to have finally launched our new data hub. This platform is designed to provide a clearer view of the current readings of our indicators and, more importantly, what these readings signify on a bullish-bearish scale. I am confident that it will prove to be a valuable tool for data-driven investors. Having personally used it extensively over the past few weeks to months, I've found it incredibly helpful. My heavy usage also allowed me to provide the WealthUmbrella team with extensive feedback, which I believe has significantly enhanced the user experience. That said, I don’t claim to have a monopoly on feedback, and I'm sure many of you could offer additional insights on how to further improve the experience. We encourage you to share your feedback, either in the comments below or by writing directly to Zackary. We will soon re-evaluate the priority of our to-do list for future improvements, ensuring that your suggestions are considered in this prioritization process. In the meantime, I sincerely hope you will appreciate this new addition to our S&P 500 offerings. For those of you who are heavy TradingView users, rest assured that our indicators will continue to be updated and accessible on TradingView.