ML Pipeline
1Ingest
Daily prices, volume and dividends for 118 Tadawul stocks (yfinance .SR), plus Brent, S&P 500, EM, gold, US yields. Every Friday 04:00.
2Clean
Riyadh-date fix, era calendar (Sat–Wed until 2013, Sun–Thu after), split/dividend adjustment, halted days. US data shifted to the next Tadawul session (no look-ahead).
3Features
Momentum, trend, volatility, liquidity, oil and sector signals. Each ranked across stocks per day (rank-z). 19 used by the model; 22 more computed as candidates. Stocks below SAR 5M/day traded value are skipped that day.
4Labels
Target = each stock's percentile among peers by forward total return over H sessions. Horizons compared: 5, 21, 63, 126, 252 sessions (1 week … 1 year).
5Walk-forward split
Expanding window, quarterly steps; training stops H+1 sessions before each test window (purge/embargo). Last 504 sessions = locked holdout, never used for choices.
6Bake-off
LightGBM ranker, XGBoost ranker, ridge regression, and fixed factor rules (low-vol, reversal, momentum, value, blend, composite) scored on identical folds.
7Ship rule
Simplest model within noise of the best on development data (blend → linear → booster); horizon with the highest development t. Shipped: blend, H = 5.
8Holdout verdict
Scored once on the holdout: edge only if rank-IC t > 2 and the backtest beats the equal-weight market. Now: no_reliable_edge.
9Production fit
Refit on all labelled history. Confidence bands: 30 refits on resampled history give each stock a 10th–90th percentile range.
10Rankings
Today's snapshot → score → peer percentile and quintile (Q1 = top fifth), SHAP drivers per stock, Shariah flag (261 compliant codes).
11Market exposure
Separate monthly layer: how much to hold in stocks vs cash (trend vote, logistic model, blends). Shipped rule: always → invest 100%.
12Backtest & publish
Top quintile, overlapping monthly cohorts, 30% sector cap, 15 bps/side costs (+2× stress test), optional exposure scaling. Results → Model Health; site republished to Cloudflare.
Features used by the model (19)
| Feature | Family | Definition | Data / window |
|---|---|---|---|
| z_mom_12_1 | Momentum | 12-1 momentum: return from 12 months ago to 1 month ago (skips the latest month). Ranked across stocks each date (rank-z). | Adjusted close, t-252 .. t-21 |
| z_st_reversal_1m | Momentum / reversal | 1-month return. Recent losers tend to bounce (short-term reversal). Ranked across stocks each date (rank-z). | Adjusted close, t-21 .. t |
| z_realized_vol | Risk | Standard deviation of daily total returns over ~3 months. Low-vol stocks historically outperform in KSA. Ranked across stocks each date (rank-z). | Daily returns, 63 sessions |
| z_log_turnover | Liquidity / size | Log of average daily traded value (SAR), smoothed over 90 days. Proxy for size and liquidity. Ranked across stocks each date (rank-z). | Close x volume, 63 + 90 sessions |
| z_dist_ma50 | Trend | Price distance from its 50-day moving average (price / MA50 - 1). Ranked across stocks each date (rank-z). | Adjusted close, 50 sessions |
| z_dist_ma200 | Trend | Price distance from its 200-day moving average (price / MA200 - 1). Ranked across stocks each date (rank-z). | Adjusted close, 200 sessions |
| z_rel_strength_12m | Momentum | 12-month return minus the equal-weight universe's 12-month return. Ranked across stocks each date (rank-z). | Adjusted close, EW index |
| z_oil_beta | Oil | Rolling 2-year correlation of the stock's daily return with Brent's (Brent credited to the next Tadawul session). Ranked across stocks each date (rank-z). | Daily returns + Brent, 504 sessions |
| z_oil_beta_x_mom | Oil | Oil beta x Brent's 3-month move: who benefits from the current oil trend. Ranked across stocks each date (rank-z). | oil_beta, Brent 63 sessions |
| z_oil_sector_interaction | Oil | Sector oil sign (+1 energy/petrochem/materials/utilities, -1 banks/insurance/financials, 0 others) x Brent's 3-month move. Ranked across stocks each date (rank-z). | Sector, Brent 63 sessions |
| sector_banking | Sector | 1 if the stock is in the Banking sector, else 0 (only sectors with 7+ stocks get a dummy). | universe.yaml |
| sector_consumer | Sector | 1 if the stock is in the Consumer sector, else 0 (only sectors with 7+ stocks get a dummy). | universe.yaml |
| sector_healthcare | Sector | 1 if the stock is in the Healthcare sector, else 0 (only sectors with 7+ stocks get a dummy). | universe.yaml |
| sector_industrial | Sector | 1 if the stock is in the Industrial sector, else 0 (only sectors with 7+ stocks get a dummy). | universe.yaml |
| sector_insurance | Sector | 1 if the stock is in the Insurance sector, else 0 (only sectors with 7+ stocks get a dummy). | universe.yaml |
| sector_materials | Sector | 1 if the stock is in the Materials sector, else 0 (only sectors with 7+ stocks get a dummy). | universe.yaml |
| sector_petrochemicals | Sector | 1 if the stock is in the Petrochemicals sector, else 0 (only sectors with 7+ stocks get a dummy). | universe.yaml |
| sector_real_estate | Sector | 1 if the stock is in the Real Estate sector, else 0 (only sectors with 7+ stocks get a dummy). | universe.yaml |
| sector_transport | Sector | 1 if the stock is in the Transport sector, else 0 (only sectors with 7+ stocks get a dummy). | universe.yaml |
Feature list: config/feature_selection.json. Changes are tested one block at a time and judged on forward results, not by searching over the holdout.
Candidate features computed but not used (22)
| Feature | Family | Definition | Data / window |
|---|---|---|---|
| z_div_yield_ttm | Value | Trailing 12-month cash dividends / price, from the price feed's own dividend records (point-in-time clean). Ranked across stocks each date (rank-z). | Dividends, 252 sessions |
| z_amihud_log | Liquidity | Amihud illiquidity: average |daily return| per SAR traded over ~3 months (log). High = price moves a lot on little money. Ranked across stocks each date (rank-z). | Returns, close x volume, 63 sessions |
| z_max_1m | Lottery | Largest single-day return in the last month (retail 'lottery' demand; high-MAX stocks tend to underperform). Ranked across stocks each date (rank-z). | Daily returns, 21 sessions |
| z_ind_mom_1m | Industry momentum | Average 1-month return of the stock's sector. Ranked across stocks each date (rank-z). | Sector peers, 21 sessions |
| z_ind_mom_6m | Industry momentum | Average 6-month return of the stock's sector. Ranked across stocks each date (rank-z). | Sector peers, 126 sessions |
| z_beta_spx | Global exposure | 1-year rolling beta of the stock's daily return to S&P 500 (previous US session). Ranked across stocks each date (rank-z). | Daily returns + S&P 500, 252 sessions |
| z_beta_eem | Global exposure | 1-year rolling beta of the stock's daily return to emerging markets (EEM) (previous US session). Ranked across stocks each date (rank-z). | Daily returns + emerging markets (EEM), 252 sessions |
| z_beta_gold | Global exposure | 1-year rolling beta of the stock's daily return to gold (previous US session). Ranked across stocks each date (rank-z). | Daily returns + gold, 252 sessions |
| z_beta_fxi | Global exposure | 1-year rolling beta of the stock's daily return to China large caps (FXI) (previous US session). Ranked across stocks each date (rank-z). | Daily returns + China large caps (FXI), 252 sessions |
| z_beta_irx | Global exposure | 1-year rolling beta of the stock's daily return to US 3-month bill yield (previous US session). Ranked across stocks each date (rank-z). | Daily returns + US 3-month bill yield, 252 sessions |
| z_beta_tnx | Global exposure | 1-year rolling beta of the stock's daily return to US 10-year yield (previous US session). Ranked across stocks each date (rank-z). | Daily returns + US 10-year yield, 252 sessions |
| z_beta_brent_up | Oil | Beta to Brent on days oil rose (asymmetric oil sensitivity). Ranked across stocks each date (rank-z). | Daily returns + Brent, 252 sessions |
| z_beta_brent_dn | Oil | Beta to Brent on days oil fell. Ranked across stocks each date (rank-z). | Daily returns + Brent, 252 sessions |
| z_expo_spx | Global exposure | beta_spx x S&P 500's move over the last ~3 months: expected push from that driver. Ranked across stocks each date (rank-z). | beta_spx, 63 sessions |
| z_expo_eem | Global exposure | beta_eem x emerging markets (EEM)'s move over the last ~3 months: expected push from that driver. Ranked across stocks each date (rank-z). | beta_eem, 63 sessions |
| z_expo_gold | Global exposure | beta_gold x gold's move over the last ~3 months: expected push from that driver. Ranked across stocks each date (rank-z). | beta_gold, 63 sessions |
| z_expo_fxi | Global exposure | beta_fxi x China large caps (FXI)'s move over the last ~3 months: expected push from that driver. Ranked across stocks each date (rank-z). | beta_fxi, 63 sessions |
| z_expo_irx | Global exposure | beta_irx x US 3-month bill yield's move over the last ~3 months: expected push from that driver. Ranked across stocks each date (rank-z). | beta_irx, 63 sessions |
| z_expo_tnx | Global exposure | beta_tnx x US 10-year yield's move over the last ~3 months: expected push from that driver. Ranked across stocks each date (rank-z). | beta_tnx, 63 sessions |
| z_expo_oil_asym | Oil | Up-beta x oil rally or down-beta x oil sell-off over the last ~3 months. Ranked across stocks each date (rank-z). | beta_brent_up/dn, 63 sessions |
| z_lt_ret_36_12 | Value / long-term reversal | Return from 36 months ago to 12 months ago. Long-run losers tend to recover (price-based value proxy). Ranked across stocks each date (rank-z). | Adjusted close, t-756 .. t-252 |
| z_max_x_mom | Lottery | MAX x momentum interaction (re-ranked). Ranked across stocks each date (rank-z). | max_1m, mom_12_1 |
Some feed the factor baselines (dividend yield and 3-year reversal make up the value leg).