Introducing Alpha: A Risk-Aware Scoring System for Signals

On Numerai, data scientists use Numerai's data to build machine learning models which produce predictive signals on global equities.

On Numerai Signals, data scientists who already have their own stock market data submit their signals to Numerai. A signal uploaded to Numerai Signals could look like the below:

The Importance Of Scoring

Because Numerai Signals users can submit any signal built from any dataset, Numerai has to be extremely careful how we score signals.

For example, some signals submitted have strong correlation with subsequent residual returns. But if all of the performance comes from low liquidity small cap stocks, this signal may not be very valuable in live trading for a fund as large as Numerai's.

Ignoring important details such as this in scoring is harmful in two ways:

1. Data scientists with problematic signals earn more NMR than they should.
2. Data scientists with great signals may be incentivized to make them worse for Numerai but better at the incomplete scoring metric.

The more the scoring system captures the important features of live trading a signal, the better the incentives and the better the rewards to the truly helpful signals.

What makes a signal monetizable?

The answer is that it depends on the optimizer used to trade the signal and the assumed fund assets. But here is what matters to Numerai's optimizer:

1. Neutrality to all factors
2. The liquidity of the stock (high average daily volume is better)
3. The residual volatility of the stock (low idiosyncratic volatility is better)

The problem is: Numerai Signals users don't necessarily have residual volatility or liquidity estimates and don't have Numerai's factor model either. This means Numerai Signals users are building signals somewhat blind to this important information which really influences the value of the signal to Numerai.

Alpha

Today, we have released a new scoring metric called Alpha. Alpha is the biggest improvement to Numerai Signals scoring since its launch. Not only does it handle every issue above, but it is also entirely computable locally. Data scientists building signals can explicitly train their machine learning models to maximize Alpha and align their models with the Numerai hedge fund.

To do this, Numerai created and released three new pieces of data:

• Neutralizers Matrix: a 200-column matrix used to neutralize a signal to every factor Numerai has
• Sample Weights Vector: a weight vector used to adjust the weights of stocks based on the liquidity & residual volatility of the stock
• Chili Target: a new 60D2L target residual to every factor Numerai has

We can create a set of neutralized weights from a signal before scoring given the following data:

• signal s (NaNs filled w/ 0.5)
• neutralization matrix N
• weight vector v

We first we create s' by ranking, normalizing, and exponentiating:

s' = normalize(rank(s))^1.5

Next, generate neutralize predictions using the sample weights and neutralization matrix. The neutralization matrix was created to be aware of the weight vector - this is the true power of the new data - so, we weight s', project it into the neutralization space, then adjusting the original to get the neutralized signal:

neutral_signal = s' - (N @ (N.T @ (v * s')))

Then, we apply the sample weights:

neutral_weights = neutral_signal * v

Finally, we use the 60D2L target chili to derive the final alpha value:

alpha = neutral_weights @ target_chili_60

Since, Alpha integrates every important aspect of Numerai's optimizer, it aligns the signal scoring system with what is realistically tradable by Numerai's optimizer.

Alpha is currently live and backfilled on Numerai Signals for all users. Users may find that signals which were previously performing badly in the previous scoring system may be doing better on Alpha. Because, Alpha is calculated after neutralization, many users may find Alpha is far less risky / has far lower drawdowns than our previous scoring system. If your performance looks significantly worse on Alpha, you may be able to diagnose the problem and repair it: for example, if you signal was earning too much from small caps or factor exposures it might be easy to fix by retraining.

Meta Portfolio Contribution (MPC)

In all of Numerai's tournaments, we use Meta Model Contribution (MMC) to incentivize data scientists to produce orthogonal alpha vs the existing signals on the platform. With the invention of Alpha, Numerai Signals also needs a metric that accomplishes the same goal: incentivize models that ensemble well with others after neutralization and sample weighting.

An analog to the Meta Model signal used in MMC is the Stake-Weighted Portfolio (SWP). In this case, Meta Portfolio Contribution evaluates the extent to which a signal enhances the Alpha of the SWP. To find an individual model’s contribution, we could then increase their stake in the SWP slightly and measure the change in Alpha of the SWP. This is just a gradient of SWP Alpha with respect to a model's stake.

Given the following data:

• signals matrix S (NaNs filled w/ 0.5)
• stake-weights p
• neutralization matrix N
• weight vector v

We generate neutral weights per signal the same way as before:

s` = normalize(rank(s))^1.5
neutral_preds = s` - (N @ (N.T @ (v * s`)))
neutral_weights = neutral_preds * v

Next, we compute the Stake Weighted Average of all of them to construct the SWP and normalize the total weight to ensure it resembles a real-world portfolio:

SWP = (p * user_stakes).sum()
SWP = SWP - SWP.mean()
SWP = SWP / sum(abs(SWP))

Then, we can calculate the Alpha of the SWP:

SWP_Alpha = SWP * target

Finally, MPC is the gradient of SWP Alpha with respect to stakes (the gradient(s, w) function is trying to maximize s by adjusting w):

MPC = gradient(SWP_Alpha, p)

When paid on MPC, data scientists will be incentivized to stake on signals that push the Numerai Signals tournament to be valuable to Numerai's hedge fund. Visit the numerai-tools repository if you're interested in reading the code for MPC.

Turnover

There's one other thing our optimizer likes and that is slow moving signals. As Signals users already know, we require signals to stay under 15% week-over-week churn. The churn filter has lead to a drastic improvement in the tradability of signals on Numerai Signals.

Churn, however, only considers the raw signal's predictions, not the related neutral weights, so we need a new concept. Turnover is the average absolute total difference in weights between two neutral weights vectors. It is a simple calculation:

turnover(t0, t1) = (w(t0) - w(t1)).abs().sum() / 2

Where w(t) is the neutral weights vector at time t.

We plan to set the turnover threshold at 20%, which we have seen internally is a reasonable level of turnover for individual models and reduces overall turnover of the SWP to a tradable range. While turnover and churn can be different for some signals, signals already below 15% churn are likely to be below 20% turnover already.

Payouts

We want the Numerai community to have some time to think about Alpha, MPC, and turnover so we have not implemented any changes to the way payouts work on Signals just yet. However, rounds starting on or after September 2nd will include these changes:

• Payouts will be based on Alpha and MPC - we are considering 0.3xAlpha + 0.8xMPC as the payout multipliers
• Stakes will be locked up for 12 weeks, not 4.
• Churn threshold will be deactivated
• Turnover threshold will be activated

Next Fireside

We'd like to hear what you think in our #signals Discord channel, so join today to stay in the loop. We will also hold a fireside chat with our founder and CEO Richard, Chief Tournament Officer Noah, and Research Engineer Andres on Wednesday, August 6th 2025 to talk more about these changes - follow this zoom link to join the webinar on Wednesday.

If you'd like to continue reading about these updates, check out our forum for a more detailed post by one of our engineers.