Sources & data lineage
Every source the pipeline touches, when it last ran and what came back. Published so that anyone can audit a figure back to its origin.
NSE
| Dataset | Cadence | Last run | Status | Notes |
|---|---|---|---|---|
Market status (segment-wise) https://www.nseindia.com/api/marketStatus | realtime | 2026-09-23 08:06 | ok | Powers the live market strip. Cheap, poll every 60s during market hours. |
All indices snapshot https://www.nseindia.com/api/allIndices | realtime | 2026-09-23 08:06 | ok | NIFTY 50 / BANKNIFTY levels + advance-decline for the market pulse header. |
FII / DII trading activity https://www.nseindia.com/api/fiidiiTradeReact | daily | 2026-09-23 08:06 | ok | |
Equity securities master (EQUITY_L.csv) https://nsearchives.nseindia.com/content/equities/EQUITY_L.csv | daily | 2026-09-23 08:06 | ok | Used to size the tradable universe and to detect listed brokers. |
Member circulars feed https://www.nseindia.com/api/circulars | daily | 2026-09-23 08:06 | ok | Scanned for member-name mentions: disciplinary actions, penalties, suspensions. |
CM segment bhavcopy https://nsearchives.nseindia.com/content/cm/BhavCopy_NSE_CM_0_0_0_{yyyymmdd}_F_0000.csv.zip | daily | 2026-09-23 08:06 | ok | Market-wide turnover context. Date-templated. |
Investor complaints dashboard against trading members https://www.nseindia.com/static/complaints/dashboard | monthly | - | not run | |
Member-wise active client counts (UCC) UNCONFIRMED - see docs/DATA_SOURCES.md | monthly | - | not run | The single most valuable metric and the only one not yet automated. NSE publishes it monthly but not at a stable public JSON/CSV endpoint we could confirm. Until the endpoint is pinned, active-client figures are loaded from data/manual/active_clients.json and every number carries provenance:'manual' so the UI can badge it. DO NOT scrape aggregator sites for this. |
Live equity quotes (most active by volume + top gainers) https://www.nseindia.com/api/live-analysis-most-active-securities?index=volume | realtime | 2026-09-23 08:06 | ok | Powers the rolling header ticker: 20 most-active symbols plus 10 NIFTY gainers with LTP, change and volume. /api/equity-stockIndices 404s and /api/quote-equity 403s - the live-analysis paths are the ones that answer. |
BSE
| Dataset | Cadence | Last run | Status | Notes |
|---|---|---|---|---|
BSE corporate actions feed https://api.bseindia.com/BseIndiaAPI/api/DefaultData/w | realtime | - | not run | CORRECTED: this is a corporate-actions feed (scrip_code, short_name, Ex_date, Purpose, RD_Date, payment_date), NOT indices. Requires Referer plus redirect following. BSE exposes no open index-level endpoint - SensexData, IndexMovement, MktRtrnData, GetIndexData, MarketWatchTable, Indexhighlight and MktCapitalisation all return an error page - so index levels come from NSE only. |
BSE scrip-wise gross delivery archive https://www.bseindia.com/BSEDATA/gross/{yyyy}/SCBSEALL{ddmm}.zip | daily | - | not run | CORRECTED: SCBSEALL is scrip-wise DELIVERY data (DATE|SCRIP CODE|DELIVERY QTY|DELIVERY VAL|DAY'S VOLUME|DAY'S TURNOVER|DELV. PER.), not member-wise turnover. Aggregated into a market-wide delivery percentage. BSE does not publish member-wise activity openly. |
Per-scrip live quote (getScripHeaderData) https://api.bseindia.com/BseIndiaAPI/api/getScripHeaderData/w?Debtflag=&scripcode={scrip}&seriesid= | realtime | 2026-09-23 08:06 | ok | Returns CurrRate{LTP,Chg,PcChg} and Cmpname{FullN}. The only working open BSE quote endpoint and there is no bulk variant, so it costs one request per scrip - keep config/watchlist.json short. |
SEBI
| Dataset | Cadence | Last run | Status | Notes |
|---|---|---|---|---|
Registered stock brokers (recognised intermediaries) https://www.sebi.gov.in/sebiweb/other/OtherAction.do?doRecognisedFpi=yes&intmId=1 | monthly | 2026-09-23 08:09 | ok | SEBI returns 'No record(s) available' for intmId=1 for every parameter combination and renders its own exchange dropdown empty - the gap is upstream. Identity therefore comes from intmId=2 (commodity segment), which returns 2,048 registration rows / 831 entities. |
Registered stock brokers - commodity derivatives segment https://www.sebi.gov.in/sebiweb/other/OtherAction.do?doRecognisedFpi=yes&intmId=2 | monthly | 2026-09-23 08:09 | ok | |
Registered depository participants - CDSL https://www.sebi.gov.in/sebiweb/other/OtherAction.do?doRecognisedFpi=yes&intmId=18 | monthly | - | not run | Tells you which brokers self-clear demat vs route through a partner DP. |
Registered depository participants - NSDL https://www.sebi.gov.in/sebiweb/other/OtherAction.do?doRecognisedFpi=yes&intmId=19 | monthly | - | not run | |
Defaulter / expelled stock brokers https://www.sebi.gov.in/sebiweb/broker/BrokerAction.do?doBroker=yes | monthly | 2026-09-23 08:09 | ok | Hard negative signal. A broker appearing here is flagged on the site immediately. |
SCORES investor grievance portal https://scores.sebi.gov.in | monthly | - | not run | Aggregate complaint counts by entity. Feeds the reliability score. |
Individual brokers (SEBI-mandated)
| Dataset | Cadence | Last run | Status | Notes |
|---|---|---|---|---|
Monthly investor complaint disclosure, SEBI Annexure-B format per-broker; see brokers_master.json[].disclosures.complaints_url | monthly | - | not run | Every registered broker MUST publish received/resolved/pending monthly. Public disclosure, safe to cite with attribution. |
MCX
| Dataset | Cadence | Last run | Status | Notes |
|---|---|---|---|---|
Commodity futures market watch https://www.mcxindia.com/market-data/market-watch/GetMarketWatch?culture=en | realtime | 2026-09-23 08:06 | error | GET, not POST. MCX builds the URL as location.origin + location.pathname + MethodName (see /assets/customjs/data.js), so /backpage.aspx/GetMarketWatch is wrong - it returns MCX's 404 page with a 200 status. Akamai Bot Manager also requires the full Chrome client-hint and Sec-Fetch header set PLUS Content-Type: application/json and X-Requested-With, or it answers 403. Returns 2,239 contracts; only the ~112 FUTCOM futures are surfaced, most-traded expiry per product. |
Binance
| Dataset | Cadence | Last run | Status | Notes |
|---|---|---|---|---|
Spot ticker (24hr), Phase 1 crypto universe https://api.binance.com/api/v3/ticker/24hr | realtime | 2026-09-23 08:10 | ok | Public endpoint, no auth, one request for all 28 symbols via the `symbols` array param (must be JSON with no spaces after commas - Binance 400s otherwise, confirmed by hand). Tried first as the live price leg on the cheap ticker cadence, but Binance's global API returns HTTP 451 (blocked for legal reasons) to any US-hosted IP - confirmed live in a Vercel build log (iad1/Washington DC) and true of GitHub Actions runners too, so it 451s on every automated run this pipeline makes. coingecko_markets below is the price leg that actually works in production; this stays wired in and used opportunistically (e.g. a non-US machine) rather than removed. |
CoinGecko
| Dataset | Cadence | Last run | Status | Notes |
|---|---|---|---|---|
Coin identity, market-cap rank, supply, ATH/ATL, plus price/24h stats (real fallback for the Binance leg above) https://api.coingecko.com/api/v3/coins/markets | hourly | 2026-09-23 08:10 | ok | Public endpoint, no auth. Used to resolve which real coin a Binance ticker symbol identifies (CoinGecko's own coin list has multiple different coins sharing some tickers) and for the slower-moving facts a static page carries - supply, market-cap rank, all-time high/low. Also called on a short ttl (coingecko_prices()) as the price leg for the ticker/pulse cadence, since Binance 451s from every automated environment this pipeline runs in - see binance_prices' notes. Only the 28 hand-verified coin ids in pipeline/sources/crypto.py are ever requested; this pipeline never guesses a match from a bare symbol. |
Licensing position
- NSE
- robots.txt allows all crawling. Exchange data carries redistribution restrictions for commercial use - obtain written clearance from NSE Data & Analytics before monetising derived datasets.
- BSE
- Same posture as NSE. BSE licenses data commercially.
- SEBI
- Regulator disclosures are public records. Attribution required; no redistribution licence needed.
- AGGREGATORS
- Never ingest from Chittorgarh / IPO Central / broker-comparison sites. Primary sources only - this is both a legal and a trust requirement.
- REUSE
- The figures BrokerLens compiles and publishes are free to view, cite and link to with attribution to BrokerLens. Bulk export, scraping, or redistributing the compiled dataset (as opposed to citing an individual figure) requires a data licence - contact us before building on it programmatically.
- MCX
- Same commercial-redistribution posture to assume as NSE and BSE: fine for internal use, needs clearance before monetising a public display.