Foresea: Autonomous Prediction Market Intelligence & LLM Rationale Benchmark

Foresea (foresea.ink) is an autonomous prediction market intelligence platform and empirical research framework. It combines real-time probability forecasting across Polymarket and Kalshi, statistical edge discovery, multi-model shadow trading tournaments, an autonomous 19-tool ReAct execution agent, and a public Model Context Protocol (MCP) server.

The repository also serves as the artifact for research studying how explicit rationale instructions, evidence injection, and reasoning structures affect LLM forecasting behavior and calibration on Metaculus-style questions.

📑 Table of Contents

• Live Platform & Web Views
• Edge Board & Autonomous Shadow Trading Desk
• 5-Minute Crypto Micro-Markets
• Production Deployment & Infrastructure
• Using the API & Endpoints
• Agent: Automated Intelligence Layer & ReAct Tools
• Model Context Protocol (MCP) Integration
• Custom Integrations & Ecosystem Tools
• Trading Execution & KMS Guardrails
• Repository Contents & Prompt Variants
• Foresea Autoresearch
• Reproducing Core Outputs & SQL Analytics
• Quality Checks & Development

Live Platform & Web Views

Deployed on Google Cloud Run with high-concurrency scaling, streaming responses, and continuous delivery:

https://foresea.ink

Core Web Applications & Routes

• Market Desk (/): Real-time landing view with market radar, model-vs-market gap highlights, and interactive walkthroughs.
• Forecasting Chat (/chat, /chat/:id): Conversational interface with streaming rationale, news evidence citation, and multi-turn market analysis.
• Edge Board (/edge, /edge/markets): Ranked live Foresea-vs-market pricing discrepancies across Polymarket and Kalshi, backed by calibration and lead-time scores.
• Mark-to-Market View (/edge/mtm): Continuous mark-to-market valuation and PnL tracking for resolved and open market predictions.
• Agentic Trading Desk (/edge/agentic): Multi-model autonomous trading tournament. Independent $10,000 shadow accounts for each model (Gemma 4 26B, Qwen 3.8 27B, GPT-OSS 120B, GLM 5.3, GLM 5.3 Flash, DeepSeek V4 Flash, MiniMax M3, Llama 3.3 70B) executing real-time paper trades, tracking equity curves, and logging hourly cycle health.
• Trading Terminal (/trade): Non-custodial, client-side order preview and execution terminal with Cloud KMS envelope encryption for Polymarket and Kalshi credentials.
• Watchlist (/watchlist): Follow specific markets with automated daily digest emails.
• Public Forecasts (/forecast/:share_id): Shareable forecast permalinks with rationale cards and provenance.

# Health check
curl https://foresea.ink/health

# Single-record prediction
curl -X POST https://foresea.ink/predict \
  -H "Content-Type: application/json" \
  -d '{
    "question": "Will X happen by date Y?",
    "question_type": "binary",
    "description": "Context here.",
    "news_articles": [],
    "attach_evidence": true,
    "evidence_top_k": 5,
    "market_platform": "Polymarket",
    "market_probability": 0.42,
    "variant": "variant0_neutral_baseline"
  }'

When attach_evidence is true and no news_articles are supplied, /predict fetches and ranks current news evidence from GDELT, Google News RSS, and Stooq by default, injects it into the model prompt, and returns the selected evidence_articles with the forecast. Supplying news_articles skips automatic retrieval and uses the caller-provided evidence.

The response includes both the forecast and the evidence used by the model:

{
  "question_type": "binary",
  "predicted_answer": "Yes",
  "confidence": 0.86,
  "options": [],
  "range_forecast": null,
  "rationale": "Model-generated explanation for the forecast.",
  "model_rationale": "Model-generated explanation for the forecast.",
  "variant": "variant0_neutral_baseline",
  "model_key": "gpt-oss-120b",
  "evidence_sources": [
    {
      "source": "Reuters",
      "title": "Article headline",
      "url": "https://example.com/article",
      "publish_date": "2026-05-29T00:00:00Z",
      "relevance_score": 0.82
    }
  ],
  "evidence_articles": [
    {
      "title": "Article headline",
      "summary": "Cleaned article summary.",
      "source": "Reuters",
      "url": "https://example.com/article",
      "publish_date": "2026-05-29T00:00:00Z",
      "relevance_score": 0.82,
      "search_query": "query used for retrieval"
    }
  ],
  "evidence_error": null,
  "market_analysis": {
    "platform": "Polymarket",
    "market_url": "https://example.com/market",
    "outcome": "Yes",
    "market_probability": 0.42,
    "model_probability": 0.86,
    "edge": 0.44,
    "stance": "model_above_market",
    "summary": "Foresea is 44 percentage points above the market on Yes."
  }
}

Use evidence_sources when a client only needs the source list and links. Use evidence_articles when a client needs the article-level details that were attached to the model prompt. rationale and model_rationale are generated by gpt-oss-120b and explain why the model chose its answer and confidence. When market_probability is supplied, market_analysis is computed deterministically from the model probability and the market-implied probability.

Edge Board & Autonomous Shadow Trading Desk

Foresea continuously evaluates state-of-the-art LLMs against real-world prediction markets, tracking statistical edge, calibration accuracy, and shadow portfolio performance across three dedicated views:

1. Live Markets Disagreement (/edge or /edge/markets)

• Real-Time Edge Ranking: Quantifies the statistical gap between model forecasts and live market pricing across Polymarket and Kalshi: $$\text{Edge} = P_{\text{model}} - P_{\text{market}}$$
• Sizing & Recommendation: Computes Fractional Kelly allocations and categorizes actionable trades (buy_yes, buy_no, hold).
• Calibration & Resolution: Backed by hourly snapshots committed to static/track_record_live.json, calculating Brier scores, Expected Calibration Error (ECE), and lead-time skill once markets resolve.

2. Mark-to-Market Evaluation (/edge/mtm)

• Continuous Portfolio Valuation: Evaluates paper positions at live bid/ask quotes rather than relying on eventual settlement alone.
• Unrealized vs. Realized PnL: Provides an institutional-grade view of trade trajectory and volatility exposure across open horizons.

3. Agentic Autonomous Trading Tournament (/edge/agentic)

• Multi-Model Tournament: Evaluates 8 distinct LLMs acting as autonomous portfolio managers:
  • Gemma 4 26B (gemma-4-26b-a4b-it)
  • Qwen 3.8 27B (qwen3-8-27b)
  • GPT-OSS 120B (gpt-oss-120b)
  • GLM 5.3 (glm-5-3) & GLM 5.3 Flash (glm-5-3-flash)
  • DeepSeek V4 Flash (deepseek-v4-flash)
  • MiniMax M3 (minimax-m3)
  • Llama 3.3 70B (llama-3.3-70b-instruct)
  • Baseline: crowd-follow (no-LLM consensus control)
• Independent $10,000 Portfolios: Each model starts with a simulated $10,000 cash balance and autonomy to scan markets, evaluate orderbooks, and execute Immediate-or-Cancel (IOC) paper orders.
• Equity Curves & Cycle Health: Renders cumulative returns ($10,000 reinvested), drawdown metrics, trade logs, and hourly cycle execution health (tracking provider availability, token budgets, and trade triggers).

5-minute crypto markets

The local crypto micro-market model in src/analyzing_llm_rationale/crypto_5m.py is built for 5-minute UP/DOWN markets where the goal is profitable selective trading, not constant action. It combines:

• shrunken-drift lognormal moneyness pricing,
• AR(1) return forecasting with EWMA volatility,
• fixed or adaptive logistic ML features from momentum, reversal, volatility regime, range position, and volume imbalance.

Each forecast returns predicted_outcome, probability_up, component_probabilities, model-vs-market edge, and a fee-aware strategy. The strategy only recommends a trade when net expected value clears fees and the configured no-trade threshold.

.venv/bin/python scripts/crypto_5m_backtest.py \
  --benchmark \
  --symbols BTC,ETH,SOL \
  --days 1 \
  --max-candles 1600 \
  --lookback-minutes 60 \
  --horizon-minutes 5 \
  --market-probability 0.50 \
  --fee-bps 2 \
  --ml-modes fixed,adaptive \
  --edge-thresholds 0,0.01,0.03,0.05,0.08 \
  --selection-fraction 0.6 \
  --folds 4 \
  --training-window 120 \
  --max-rows 80 \
  --benchmark-log data/crypto_5m_benchmark_runs.jsonl