kp// projects/ rl_directional_straddle
~/projects/rl_directional_straddle · README.md
Autonomous Options Trading & Hedging with Deep Reinforcement Learning
A MaskablePPO agent that learns, directly from a 69-feature option-Greeks state space, how to enter, hedge, and unwind multi-leg straddle and strangle positions on the NIFTY50 index — replacing hand-coded Black-Scholes hedging rules with a policy trained on 1-minute market data.
01 · overview
The problem
Standard options hedging leans on Black-Scholes: continuous trading, constant volatility, no transaction costs, no discrete strikes. Real NSE intraday markets break every one of those assumptions — spreads widen, volatility clusters and jumps, and every rebalance costs money. A fixed delta-hedging rule doesn't adapt to any of that.
This project reframes hedging as a sequential decision problem instead of a formula. An agent observes the market and its own position, and at every one-minute step chooses whether to enter, add to, reduce, or exit a short/long straddle — learning the policy from experience rather than from a closed-form model. The scope is deliberately narrow and realistic: weekly at-the-money NIFTY50 straddles and strangles, trained and evaluated on real 1-minute NSE spot and India VIX data, with transaction costs, slippage, and a dynamic exchange-style margin model all simulated in the loop.
Three algorithms were trained and compared — PPO and A2C as baselines, and Maskable PPO as the primary agent, which is the version this write-up focuses on. The work is published as “Direct Reinforcement for Derivatives: Enhancing Intraday Deep Hedging via Risk-Averse Reward Shaping” (GCon 2026).
02 · architecture
System architecture
A Gymnasium environment simulating the NIFTY50 options market, wrapped by a masked actor-critic policy trained with Stable-Baselines3 / sb3-contrib.
+ India VIX
+ vol estimators
Box observation
+ action mask
position moves
margin, txn cost
− drawdown penalty
Reward flows into a rollout buffer; every 1,024 steps × 4 parallel envs, PPO's clipped surrogate objective updates the policy and value networks, and the loop continues into the next minute of the episode.
Before the policy samples an action, the environment computes a live boolean mask from current position, lot limits, and available margin — e.g. ENTER_LONG is masked out unless margin ≤ available cash. Invalid actions never enter the loss.
Short legs are margined with a dynamic SPAN-like calculator: a 3.5σ price scan at +25% stressed IV plus a 3% notional exposure add-on, recomputed every step as spot and IV move.
# Custom environments envs/intraday_option_env_maskPPO_v3.py # v3 — 69 features, Discrete(8), action masking (this write-up) envs/intraday_option_env_maskPPO.py # v2 — 67 features, Maskable PPO envs/intraday_option_env.py # v1 — 65 features, MultiDiscrete([3,3]), PPO/A2C baseline # Training & evaluation Mask_PPO_train_v3.py # primary entry point — 4 parallel envs Mask_PPO_explainability_test.py # out-of-sample eval + SHAP export PPO_train.py · A2C_train.py # baseline algorithms walk_forward_validation.py # walk-forward harness # Config & utilities Mask_PPO_config_v3.py # reward weights, margin, contract specs utils/black_scholes.py # analytical Greeks (Δ Γ ν Θ Vanna Volga) utils/feature_calculator.py # RSI, Bollinger, ATR, MACD, vol estimators # Dashboards dashboard.py # Streamlit live backtest viewer explainability_dashboard.py # Plotly HTML SHAP / drawdown report
03 · methodology
How it works
A 69-feature state space
Every one-minute step, the environment assembles a Box(69,) observation from ten feature groups — volatility estimators, price action, option Greeks, technicals, position state, and time context:
| Group | Count | Contents |
|---|---|---|
| Volatility | 15 | Rolling realized vol (5m→15d), Parkinson & Garman-Klass estimators, vol percentile, vol-of-vol, vol trend, IV−RV spread |
| Price & returns | 10 | Normalized close, multi-window returns, VWAP distance, day-range position, momentum |
| Option Greeks | 15 | Per-leg CE/PE Delta, Gamma, Vega, Theta, Vanna, Volga (Black-Scholes) + net position Delta/Gamma/Vega exposure |
| Technicals | 6 | RSI, MACD line/signal/histogram, Bollinger position, ATR |
| Position state | 10 | CE/PE lot counts, CE:PE ratio, premium deployed, strike distances, entry prices, running PnL |
| Time | 4 | Minutes since open, minutes to close, sine/cosine time-of-day encoding |
| Risk metrics | 3 | Distance from peak PnL, running max drawdown, 30-step rolling Sharpe |
| Session context | 2 | Fraction of episode elapsed, overnight-gap flag |
| Position identity | 2 | Side encoding (+1 long / −1 short / 0 flat), normalized position duration |
| Per-leg PnL (v3) | 2 | Separate unrealized CE and PE PnL — new in v3, isolates which leg is driving reward |
Discrete(8) action space, with masking
The agent chooses one of eight discrete moves each step: HOLD, ENTER LONG, ENTER SHORT, EXIT ALL, ADD CE, ADD PE, REDUCE CE, REDUCE PE. Before sampling, the environment computes a live boolean mask — e.g. entry actions are disabled whenever the SPAN-like margin required would exceed available cash, and leg-reduction actions are disabled below a 1-lot floor. Invalid actions are removed at the logit level rather than sampled and penalized after the fact, which is what let the agent skip past the trial-and-error of learning “that action was illegal” from scratch.
This is the third iteration of the action design:
| Version | Action space | Design |
|---|---|---|
| v1 | MultiDiscrete([3,3]) | Independent 3-way choice per leg (base PPO/A2C environment, 65 features) |
| v2 | Discrete(8), masked | Joint action enumeration + Maskable PPO (67 features) |
| v3 | Discrete(8), masked | Same action space as v2, now paired with the 69-feature state and multi-horizon reward below |
Multi-horizon, risk-averse reward
An earlier single-horizon reward pushed the agent toward myopic, overtraded behavior. v3 blends three PnL horizons tuned to a ~1,875-step weekly episode, plus a drawdown penalty and two terminal shaping terms:
The 30-step (half-hour) term dominates the blend as the primary trend signal, the 1-step term keeps the agent reactive to sudden IV shocks, and the 120-step term rewards holding through a correct half-day regime call. λ penalizes only increases in drawdown rather than its absolute level, so the agent isn't punished for simply holding a position that is currently underwater without getting worse. A terminal win bonus and a no-trade penalty round out the episode-level incentives.
Solving the “sit on hands” equilibrium
Early training runs converged to a degenerate policy: since HOLD guarantees Reward = 0 and a random untrained agent loses money from spread and transaction costs, the agent learned that doing nothing dominates trading — and never explored enough to discover that skilled trading is profitable.
The fix implemented here is randomized forced entry, a curriculum-learning trick sometimes called “training wheels”: on reset(), a coin flip decides whether the episode starts flat (agent decides if/when to enter — teaches timing) or with a position already forced on at 9:30 AM (agent must manage it — teaches management). Once the agent learns from the forced episodes that managing a position is profitable, that positive value function transfers into the free episodes and it starts entering on its own — all without hand-biasing the reward function itself.
Training protocol
Trained with Maskable PPO from sb3-contrib across 4 parallel environments, on multi-year NIFTY50 1-minute OHLC + India VIX data, with weekly Wednesday→Tuesday episodes matching the NIFTY expiry cycle. Evaluated fully out-of-sample on 2025 data never seen during training.
| Learning rate | 7 × 10⁻⁴ | Batch size | 256 |
| n_steps | 1,024 | Parallel envs | 4 |
| γ (gamma) | 0.9995 | Entropy coef. | 0.05 |
| Total timesteps | 1,500,000 | Lot size | 65 (NIFTY) |
| Strike gap | ₹50 | Short strike offset | ±100 OTM |
| Txn cost + slippage | 0.5% per trade | Initial capital | ₹10,00,000 |
04 · results
Results — 2025 out-of-sample test
48 weekly episodes across 2025, entirely unseen during training. All figures below are read directly from the model's own logged evaluation output.
Monthly P&L
Out-of-sample net P&L by month, 2025
Every month in the 2025 test window closed net positive. April was the weakest month (₹13,099) during a spike in realized drawdown; May was the strongest (₹1,11,226). Axis in ₹.
Explainability
SHAP feature importance — top 10 of 69
Computed post-hoc with SHAP over the explainability test run. The put leg's Vanna and the call leg's Theta dominate — consistent with a short-strangle book where IV/spot cross-sensitivity and time decay of the richer leg drive most of the agent's decisions, ahead of raw price momentum.
Read together, the two charts tell a consistent story: the agent's edge is concentrated in risk management under volatility regime shifts rather than pure directional prediction — the top features are option-sensitivity terms (Vanna, Theta, Vega) and volatility trend, not price momentum. That lines up with the reward design: the multi-horizon blend was built to reward exactly this kind of regime-aware position management.
05 · discussion
Discussion & limitations
A backtest is not a live market. These are the gaps between what's simulated here and full production readiness.
— Limitations
- Static cost model. Transaction cost and slippage are applied as a flat 0.5% rather than an order-book-aware, liquidity-dependent estimate.
- Discrete strike grid. The agent trades ATM ± a fixed offset; it doesn't choose strikes continuously or react to skew shape directly.
- Simulated fills. Execution assumes the quoted price fills instantly at the requested size — no queue position, partial fills, or latency.
- Single underlying. Trained and evaluated only on NIFTY50; hedging correlated multi-asset books is untested.
- Backtest–live gap. Strong out-of-sample numbers on historical data are necessary but not sufficient evidence for live performance.
— Future work
- Continuous control (SAC / TD3) to size positions and choose strikes/offsets continuously instead of from a fixed discrete menu.
- Order-book features — bid-ask depth and imbalance — to move the cost model from static to microstructure-aware.
- Sequence models (LSTM / Transformer encoders) in place of the flat feature vector, to let the policy learn temporal structure directly.
- Multi-asset hedging across correlated indices and single-stock options.
- Adaptive transaction-cost modeling that scales with realized liquidity instead of a fixed percentage.
06 · tech stack