DataPulse

Live dashboard: https://www.data-pulse.my

Open in Google Colab:

🤖 AI-agent-ready — Wire DataPulse into Claude Desktop, Cursor, Cline, or any MCP-compatible client with one config block. Your agent gets

418 official Malaysian datasets — including 30 GTFS transit feeds (KTMB, Prasarana, BAS.MY) — with declared licences and an honest ten-status trust taxonomy instead of a blanket green checkmark.

→ Connect your AI agent in 30 seconds

This is DataPulse

When an AI quote is wrong, it is often wrong because the underlying data was stale, mis-licensed, or unverifiable — not because the model hallucinated. An official-looking page does not tell an agent when the dataset behind it last updated, who published it, whether it may legally be reused, or whether the observation can be reproduced by a second party.

DataPulse exists to make that uncertainty explicit. It is an open, read-only verification layer for Malaysian public data: it continuously probes

and publishes — for each one — machine-readable evidence about whether the source is reachable, how fresh the content is, what licence applies, how the schema behaves, and when the observation was signed.

It does not replace the official source. It documents, on an honest and reproducible basis, what the official portal states and whether that material is current, so you know what you are reusing or citing. The verification speaks for itself: every claim here is a live, checkable artefact, not a promise.

What we do, simply

• We watch the sources. A scheduled probe revisits each dataset under its declared cadence and records what it actually finds — reachability, an honest freshness signal, schema shape, record counts, and collection quirks.
• We state the truth plainly. Instead of a blanket green checkmark, each dataset carries one of ten honest health statuses (fresh, aging, stale, discontinued, degraded, browser-dependent, unreachable, unknown, unknown-freshness, reference). A dataset that cannot be proven fresh is labelled unknown-freshness — not silently treated as healthy.
• We publish evidence, not just claims. Each dated observation is signed and recorded to an immutable public log, so you can verify when DataPulse observed the source and that the record has not been altered.
• We make it machine-readable first. The whole portfolio is discoverable from one index and queryable over a read-only MCP server, so an agent receives the same freshness, licence, and provenance signal a careful human reviewer would.

Who this serves

• AI builders and agent developers, who want a model to check a Malaysian figure's freshness and licence before it cites the number — without building a bespoke integration or trusting a scraping pipeline.
• Researchers, analysts, and journalists, who need to ground coursework, a thesis, a dashboard, or a published figure in data whose currency and licence they can actually verify.
• Compliance and regulatory-monitoring teams, who must keep a tamper-evident trail that an official figure was checked at a known time before it reached a product or a public statement.
• Civic technologists and public servants, who want a transparent, reproducible view of how discoverable and reliably described public data is.

Why you can trust the verification

Three independent, checkable layers. You do not have to take DataPulse's word — you can verify each with the published public key, the public Git source record, and the public transparency log:

Run it yourself, from anywhere, with no checkout and no DataPulse code:

curl -fsSLO https://raw.githubusercontent.com/r3dz4r/datapulse-my/main/scripts/verify_external.py
python3 verify_external.py

See Verify DataPulse externally for the full guide, and our methodology below for how health is judged.

A verification layer is only as honest as its method, so DataPulse deliberately tells you when it cannot be sure — a source that cannot be proven current is labelled accordingly, never silently marked healthy. That is the boundary we hold: the platform proves the integrity and timing of its observations, not that an upstream government figure is semantically true. That distinction is the whole point of an evidence layer, and we do not blur it.

Dataset health

Health is reported as fresh, aging, stale, discontinued, degraded, browser-dependent, unreachable, unknown, unknown-freshness, or reference. Unknown freshness means the URL and content shape work, but neither a Last-Modified header nor a parseable content date proves when the data was updated. Reference means versioned lookup data is reachable and its record count is measured, while date-based freshness does not apply. Within the catalogue, data_type refines the reference family without changing the status: policy-reference rows (policy state that stays valid until superseded — BNM OPR is current while unchanged, not stale) and reference-current rows (lookups that must still pass freshness, such as a bank-rate table that can itself go stale) are judged by their declared policy, while plain reference rows are static. The public _trust_summary shows the distribution and explicitly counts missing freshness and row-count signals.

Discontinued — The source has stopped publishing new data. The data is frozen at the last known content date. This is not a freshness failure — it's a publisher decision.

Current distribution (_trust_summary): 93 fresh · 106 aging · 190 stale · 1 discontinued · 5 browser-dependent · 9 unknown-freshness · 14 reference

Subscribe: RSS feed — get notified when dataset health changes.

⚠️ Status: active development. Dashboards and health snapshots update as probes complete. Per-dataset reports under data/{id}.md may briefly lag behind the live health snapshot in health/latest.json (a regeneration gap that is being closed). Coverage and quality improve with each tagged release; expect rough edges. Track progress via the GitHub Releases page.

Browser-dependent datasets

Five sources (currently 1.4% of the catalogue) require a real browser to probe because their source pages render client-side JavaScript: eperolehan-diklankan, doe_apims, doe_rqims, doe_mqims, and kkm_idengue.

DataPulse uses Camofox, a self-hosted patched headless-Chromium sidecar, to probe these. The probe path is check.sh → Camofox sidecar → DOM snapshot → content-date extraction.

To enable browser probing:

1. Run the Camofox Docker sidecar on a reachable address (default http://localhost:9377). The probe script and the GitHub Actions workflow pick this up from the CAMOFOX_BASE_URL environment variable; nothing in this repo encodes a public IP.
2. Set CAMOFOX_BASE_URL to that address.
3. Restart the timer with systemctl restart datapulse-health.timer.

Without Camofox, those five datasets will sit at browser-dependent — the honest status: DataPulse cannot probe them without a browser, so it says so rather than failing silently. See scripts/smoke_browser_probes.sh for isolated smoke tests.

Methodology