DataPulse

Live dashboard: https://www.data-pulse.my

Open in Google Colab:

🤖 AI-agent-ready — Wire DataPulse into Claude Desktop, Cursor, Cline, or any MCP-compatible client with one config block. Your agent gets

418 official Malaysian datasets — including 30 GTFS transit feeds (KTMB, Prasarana, BAS.MY) — with declared licences and an honest ten-status trust taxonomy instead of a blanket green checkmark.

→ Connect your AI agent in 30 seconds

This is DataPulse

When an AI quote is wrong, it is often wrong because the underlying data was stale, mis-licensed, or unverifiable — not because the model hallucinated. An official-looking page does not tell an agent when the dataset behind it last updated, who published it, whether it may legally be reused, or whether the observation can be reproduced by a second party.

DataPulse exists to make that uncertainty explicit. It is an open, read-only verification layer for Malaysian public data: it continuously probes

and publishes — for each one — machine-readable evidence about whether the source is reachable, how fresh the content is, what licence applies, how the schema behaves, and when the observation was signed.

It does not replace the official source. It documents, on an honest and reproducible basis, what the official portal states and whether that material is current, so you know what you are reusing or citing. The verification speaks for itself: every claim here is a live, checkable artefact, not a promise.

What we do, simply

• We watch the sources. A scheduled probe revisits each dataset under its declared cadence and records what it actually finds — reachability, an honest freshness signal, schema shape, record counts, and collection quirks.
• We state the truth plainly. Instead of a blanket green checkmark, each dataset carries one of ten honest health statuses (fresh, aging, stale, discontinued, degraded, browser-dependent, unreachable, unknown, unknown-freshness, reference). A dataset that cannot be proven fresh is labelled unknown-freshness — not silently treated as healthy.
• We publish evidence, not just claims. Each dated observation is signed and recorded to an immutable public log, so you can verify when DataPulse observed the source and that the record has not been altered.
• We make it machine-readable first. The whole portfolio is discoverable from one index and queryable over a read-only MCP server, so an agent receives the same freshness, licence, and provenance signal a careful human reviewer would.

Who this serves

• AI builders and agent developers, who want a model to check a Malaysian figure's freshness and licence before it cites the number — without building a bespoke integration or trusting a scraping pipeline.
• Researchers, analysts, and journalists, who need to ground coursework, a thesis, a dashboard, or a published figure in data whose currency and licence they can actually verify.
• Compliance and regulatory-monitoring teams, who must keep a tamper-evident trail that an official figure was checked at a known time before it reached a product or a public statement.
• Civic technologists and public servants, who want a transparent, reproducible view of how discoverable and reliably described public data is.

Why you can trust the verification

Three independent, checkable layers. You do not have to take DataPulse's word — you can verify each with the published public key, the public Git source record, and the public transparency log:

Run it yourself, from anywhere, with no checkout and no DataPulse code:

curl -fsSLO https://raw.githubusercontent.com/r3dz4r/datapulse-my/main/scripts/verify_external.py
python3 verify_external.py

See Verify DataPulse externally for the full guide, and our methodology below for how health is judged.

A verification layer is only as honest as its method, so DataPulse deliberately tells you when it cannot be sure — a source that cannot be proven current is labelled accordingly, never silently marked healthy. That is the boundary we hold: the platform proves the integrity and timing of its observations, not that an upstream government figure is semantically true. That distinction is the whole point of an evidence layer, and we do not blur it.

Dataset health

Health is reported as fresh, aging, stale, discontinued, degraded, browser-dependent, unreachable, unknown, unknown-freshness, or reference. Unknown freshness means the URL and content shape work, but neither a Last-Modified header nor a parseable content date proves when the data was updated. Reference means versioned lookup data is reachable and its record count is measured, while date-based freshness does not apply. Within the catalogue, data_type refines the reference family without changing the status: policy-reference rows (policy state that stays valid until superseded — BNM OPR is current while unchanged, not stale) and reference-current rows (lookups that must still pass freshness, such as a bank-rate table that can itself go stale) are judged by their declared policy, while plain reference rows are static. The public _trust_summary shows the distribution and explicitly counts missing freshness and row-count signals.

Discontinued — The source has stopped publishing new data. The data is frozen at the last known content date. This is not a freshness failure — it's a publisher decision.

Current distribution (_trust_summary): 94 fresh · 105 aging · 189 stale · 1 discontinued · 3 degraded · 5 browser-dependent · 7 unknown-freshness · 14 reference

Subscribe: RSS feed — get notified when dataset health changes.

⚠️ Status: active development. Dashboards and health snapshots update as probes complete. Per-dataset reports under data/{id}.md may briefly lag behind the live health snapshot in health/latest.json (a regeneration gap that is being closed). Coverage and quality improve with each tagged release; expect rough edges. Track progress via the GitHub Releases page.

Browser-dependent datasets

Five sources (currently 1.4% of the catalogue) require a real browser to probe because their source pages render client-side JavaScript: eperolehan-diklankan, doe_apims, doe_rqims, doe_mqims, and kkm_idengue.

DataPulse uses Camofox, a self-hosted patched headless-Chromium sidecar, to probe these. The probe path is check.sh → Camofox sidecar → DOM snapshot → content-date extraction.

To enable browser probing:

1. Run the Camofox Docker sidecar on a reachable address (default http://localhost:9377). The probe script and the GitHub Actions workflow pick this up from the CAMOFOX_BASE_URL environment variable; nothing in this repo encodes a public IP.
2. Set CAMOFOX_BASE_URL to that address.
3. Restart the timer with systemctl restart datapulse-health.timer.

Without Camofox, those five datasets will sit at browser-dependent — the honest status: DataPulse cannot probe them without a browser, so it says so rather than failing silently. See scripts/smoke_browser_probes.sh for isolated smoke tests.

Methodology

Connect an AI agent

DataPulse exposes an AI-ready, read-only MCP server so agents can query the catalogue natively. It provides the same freshness, licence, schema-drift, and provenance evidence available to a human reviewer.

• Endpoint: https://mcp.data-pulse.my/mcp (Streamable HTTP, no auth) Graded by mcpgrade — replay with bash scripts/audit_mcpgrade.sh (pinned version, writes artifacts/mcpgrade/). The canonical tool count lives in mcp.json / agent.json.

• 18 tools: search_datasets, get_dataset, find_stale, find_anomalies, find_deteriorating, find_recovering, find_unreliable, find_schema_drift, check_reconciliation, get_provenance, get_evidence, verify_dataset, get_freshness_summary, verify_evidence, trust_verdict, verify_attestation, find_by_licence, usage_summary

The public endpoint serves all 18 read-only tools over the 418-dataset catalogue.

get_evidence exposes pipeline receipts; verify_evidence performs cached transport-only live checks and does not update health.

Connect from Claude Desktop:

{
  "mcpServers": {
    "datapulse-my": {
      "transport": "streamable-http",
      "url": "https://mcp.data-pulse.my/mcp"
    }
  }
}

See llms.txt for the full discovery index, and docs/mcp-deploy.md for the deployment architecture.

• LLM index
• Agent manifest
• MCP advertisement
• Sitemap
• MCP endpoint

Wire it into Claude Desktop via claude_desktop_config.json (30 seconds, no API key):

{
  "mcpServers": {
    "datapulse-my": {
      "transport": "streamable-http",
      "url": "https://mcp.data-pulse.my/mcp"
    }
  }
}

Restart Claude Desktop, confirm the hammer icon shows "datapulse-my" with the read-only tools listed above. Cursor / Cline use the same JSON in their MCP config panel.

Included datasets

Dataset inventory is grouped by stable custodian publisher ID; unknown IDs fall back to the ID itself.

National Anti-Drugs Agency (aadk)

• data.gov.my Drug Addicts by State & Age Group (drug_addicts_age) · sample
• Drug Addicts by State & Drug Type (drug_addicts_drugtype)
• Drug Addicts by Highest Education Level (drug_addicts_education)
• Drug Addicts by State & Occupation (drug_addicts_occupation)

Accountant General's Department of Malaysia (agc)

• Quarterly Federal Government Finance (federal_finance_qtr)
• data.gov.my Quarterly Federal Operating Expenditure (federal_finance_qtr_oe) · sample
• data.gov.my Quarterly Federal Government Revenue (federal_finance_qtr_revenue) · sample
• Annual Federal Government Finance (federal_finance_year)
• Annual Federal Government Development Expenditure by Function (federal_finance_year_de)
• Annual Federal Government Operating Expenditure by Object (federal_finance_year_oe)
• Lookup Table: Federal Finance (lookup_federal_finance)

Agensi Pengangkutan Awam Darat (APAD) (apad)

• GTFS Realtime — BAS.MY Alor Setar Vehicle Positions (gtfs_realtime_mybas_alor_setar)
• GTFS Realtime — BAS.MY Ipoh Vehicle Positions (gtfs_realtime_mybas_ipoh)
• GTFS Realtime — BAS.MY Johor Vehicle Positions (gtfs_realtime_mybas_johor)
• GTFS Realtime — BAS.MY Kangar Vehicle Positions (gtfs_realtime_mybas_kangar)
• GTFS Realtime — BAS.MY Kota Bharu Vehicle Positions (gtfs_realtime_mybas_kota_bharu)
• GTFS Realtime — BAS.MY Kuala Terengganu Vehicle Positions (gtfs_realtime_mybas_kuala_terengganu)
• GTFS Realtime — BAS.MY Kuching Vehicle Positions (gtfs_realtime_mybas_kuching)
• GTFS Realtime — BAS.MY Melaka Vehicle Positions (gtfs_realtime_mybas_melaka)
• GTFS Realtime — BAS.MY Seremban A Vehicle Positions (gtfs_realtime_mybas_seremban_a)
• GTFS Realtime — BAS.MY Seremban B Vehicle Positions (gtfs_realtime_mybas_seremban_b)
• GTFS Static — BAS.MY Alor Setar Bus Schedule (gtfs_static_mybas_alor_setar)
• GTFS Static — BAS.MY Ipoh Bus Schedule (gtfs_static_mybas_ipoh)
• GTFS Static — BAS.MY Johor Bus Schedule (gtfs_static_mybas_johor)
• GTFS Static — BAS.MY Kangar Bus Schedule (gtfs_static_mybas_kangar)
• GTFS Static — BAS.MY Kota Bharu Bus Schedule (gtfs_static_mybas_kota_bharu)
• GTFS Static — BAS.MY Kuala Terengganu Bus Schedule (gtfs_static_mybas_kuala_terengganu)
• GTFS Static — BAS.MY Kuching Bus Schedule (gtfs_static_mybas_kuching)
• GTFS Static — BAS.MY Melaka Bus Schedule (gtfs_static_mybas_melaka)
• GTFS Static — BAS.MY Seremban A Bus Schedule (gtfs_static_mybas_seremban_a)
• GTFS Static — BAS.MY Seremban B Bus Schedule (gtfs_static_mybas_seremban_b)

Bank Negara Malaysia (bnm)

• Base Rates / BLR / Effective LR (bnm_base_rate)
• Interbank Swap (bnm_interbank_swap)
• Interest Rates: Banking Institutions (bnm_interest_rate)
• Interest Volume: Banking Institutions (bnm_interest_volume)
• Kijang Emas (Gold Reference Price) (bnm_kijang_emas)
• Kuala Lumpur USD/MYR Reference Rate (bnm_kl_usd_myr)
• Malaysia Overnight Rate (MYOR) (bnm_myor)
• Overnight Policy Rate (OPR) (bnm_opr)
• Monthly Currency in Circulation (currency_in_circulation)
• Annual Currency in Circulation (currency_in_circulation_annual)
• data.gov.my Monthly Interest Rates (dgm_interest_rates) · sample
• data.gov.my Annual Interest Rates (dgm_interest_rates_annual) · sample
• data.gov.my Monthly Money Aggregates (dgm_money_aggregates) · sample
• data.gov.my Monthly Payment Channels (dgm_payments_channels) · sample
• data.gov.my Monthly Payment Instruments (dgm_payments_instruments) · sample
• data.gov.my Monthly Payment Systems (dgm_payments_systems) · sample
• data.gov.my Daily FPX Transactions (dgm_payments_transactions_fpx) · sample
• Monthly Exchange Rates (exchangerates)
• BNM Daily Exchange Rates (0900) (exchangerates_daily_0900) · sample
• BNM Daily Exchange Rates (1130) (exchangerates_daily_1130) · sample
• BNM Daily Exchange Rates (1200) (exchangerates_daily_1200) · sample
• BNM Daily Exchange Rates (1700) (exchangerates_daily_1700) · sample
• Monthly Interest Rates (interestrates)
• Annual Interest Rates (interestrates_annual)
• Lookup Table: Money & Banking (lookup_money_banking)
• Monthly Monetary Aggregates (monetary_aggregates)
• Monthly Payment Channels (payment_channels)
• Monthly Payment Instruments (payment_instruments)
• Monthly Payment Systems (payment_systems)

Department of Agriculture Malaysia (doa)

• data.gov.my Crop Area and Production by State (crops_state) · sample
• Crop Area by District (dosm_crops_district_area)
• Crop Production by District (dosm_crops_district_production)

Department of Environment Malaysia (doe)

• Air Pollutant Concentrations (air_pollution)
• DOE APIMS Air Quality (Hourly API) (doe_apims) · sample
• DOE MQIMS Marine Water Quality (Manual) (doe_mqims) · sample
• DOE RQIMS River Water Quality (Continuous) (doe_rqims) · sample
• River Basin Pollution Monitoring (water_pollution_basin)

Department of Fisheries Malaysia (dof)

• data.gov.my Monthly Landings of Marine Fish by State (fish_landings) · sample

Department of Statistics Malaysia (dosm)

• Balance of Payments by Account (bop_balance)
• Monthly CPI by Group (cpi_3d)
• Monthly CPI by Class (cpi_4d)
• Monthly CPI by Subclass (cpi_5d)
• Monthly Core Consumer Price Index (cpi_core)
• Monthly Core CPI Inflation (cpi_core_inflation)
• Monthly CPI by Division (2-digit) (cpi_headline)
• Monthly CPI by State & Division (2-digit) (cpi_state)
• Monthly CPI Inflation by State & Division (2-digit) (cpi_state_inflation)
• DOSM's Advance Release Calendar (dosm_arc_dosm)
• Broad Economic Categories (BEC) (dosm_bec)
• OpenDOSM Annual Births by State (dosm_birth_state) · sample
• Annual CPI by Division (2-digit) (dosm_cpi_annual)
• Annual CPI Inflation by Division (2-digit) (dosm_cpi_annual_inflation)
• OpenDOSM Monthly Core CPI Inflation by Division (dosm_cpi_core_inflation) · sample
• Monthly CPI Inflation by Division (2-digit) (dosm_cpi_headline_inflation)
• OpenDOSM Monthly CPI Inflation by Division (dosm_cpi_inflation) · sample
• Monthly CPI for Low-Income Households (dosm_cpi_lowincome)
• OpenDOSM Monthly CPI by State & Division (dosm_cpi_state) · sample
• OpenDOSM Monthly CPI Inflation by State and Division (dosm_cpi_state_inflation) · sample
• Monthly CPI by Strata & Division (2-digit) (dosm_cpi_strata)
• OpenDOSM Crime by District & Type (Annual) (dosm_crime_district) · sample
• OpenDOSM Annual Deaths by District and Sex (dosm_death_district_sex) · sample
• OpenDOSM Annual Maternal Deaths, Malaysia (dosm_death_maternal) · sample
• OpenDOSM Annual Maternal Deaths by State (dosm_death_maternal_state) · sample
• OpenDOSM Annual Deaths by State (dosm_death_state) · sample
• OpenDOSM Annual Employment by Sector and Sex (dosm_employment_sector) · sample
• OpenDOSM Annual Fertility (dosm_fertility) · sample
• TFR and ASFR by State (dosm_fertility_state)
• Annual Nominal GDP by Expenditure Type (dosm_gdp_annual_nominal_demand)
• Annual Nominal GDP by Expenditure Subtype (dosm_gdp_annual_nominal_demand_granular)
• Annual Nominal GDP by Income Component (dosm_gdp_annual_nominal_income)
• OpenDOSM Annual Nominal GDP by Supply Sector (dosm_gdp_annual_nominal_supply) · sample
• Annual Nominal GDP by Economic Subsector (dosm_gdp_annual_nominal_supply_granular)
• Annual Real GDP by Expenditure Type (dosm_gdp_annual_real_demand)
• Annual Real GDP by Expenditure Subtype (dosm_gdp_annual_real_demand_granular)
• OpenDOSM Annual Real GDP by Supply Sector (dosm_gdp_annual_real_supply) · sample
• Annual Real GDP by Economic Subsector (dosm_gdp_annual_real_supply_granular)
• Annual Real GDP by District & Economic Sector (dosm_gdp_district_real_supply)
• OpenDOSM Annual Nominal GDP and GNI (dosm_gdp_gni_annual_nominal) · sample
• Annual Real GDP & GNI: 1970 to Present (dosm_gdp_gni_annual_real)
• Lookup Table: GDP (dosm_gdp_lookup)
• OpenDOSM Quarterly Nominal GDP (dosm_gdp_qtr_nominal) · sample
• Quarterly Nominal GDP by Expenditure Type (dosm_gdp_qtr_nominal_demand)
• Quarterly Nominal GDP by Expenditure Subtype (dosm_gdp_qtr_nominal_demand_granular)
• Quarterly Nominal GDP by Economic Sector (dosm_gdp_qtr_nominal_supply)
• Quarterly Nominal GDP by Economic Subsector (dosm_gdp_qtr_nominal_supply_granular)
• OpenDOSM Quarterly Real GDP (dosm_gdp_qtr_real) · sample
• Quarterly Real GDP by Expenditure Type (dosm_gdp_qtr_real_demand)
• Quarterly Real GDP by Expenditure Subtype (dosm_gdp_qtr_real_demand_granular)
• OpenDOSM Quarterly Real GDP (Seasonally Adjusted) (dosm_gdp_qtr_real_sa) · sample
• Quarterly Real GDP (Seasonally Adjusted) by Expenditure Type (dosm_gdp_qtr_real_sa_demand)
• Quarterly Real GDP (Seasonally Adjusted) by Economic Sector (dosm_gdp_qtr_real_sa_supply)
• Quarterly Real GDP by Economic Sector (dosm_gdp_qtr_real_supply)
• Quarterly Real GDP by Economic Subsector (dosm_gdp_qtr_real_supply_granular)
• OpenDOSM Annual Real GDP by State & Sector (dosm_gdp_state_real_supply) · sample
• Access to Basic Amenities by State & District (dosm_hh_access_amenities)
• OpenDOSM Household Expenditure by DUN (dosm_hh_expenditure_dun) · sample
• OpenDOSM Household Expenditure by Parliamentary Constituency (dosm_hh_expenditure_parlimen) · sample
• OpenDOSM Household Income, Malaysia (dosm_hh_income) · sample
• OpenDOSM Household Income by District (dosm_hh_income_district) · sample
• Household Income by DUN (dosm_hh_income_dun)
• Household Income by Parliament (dosm_hh_income_parlimen)
• OpenDOSM Household Income by State (dosm_hh_income_state) · sample
• OpenDOSM Income Inequality, Malaysia (dosm_hh_inequality) · sample
• OpenDOSM Income Inequality by District (dosm_hh_inequality_district) · sample
• Income Inequality by DUN (dosm_hh_inequality_dun)
• Income Inequality by Parliament (dosm_hh_inequality_parlimen)
• OpenDOSM Income Inequality by State (dosm_hh_inequality_state) · sample
• OpenDOSM Poverty, Malaysia (dosm_hh_poverty) · sample
• OpenDOSM Poverty by District (dosm_hh_poverty_district) · sample
• Poverty by DUN (dosm_hh_poverty_dun)
• Poverty by Parliament (dosm_hh_poverty_parlimen)
• OpenDOSM Poverty by State (dosm_hh_poverty_state) · sample
• Number of Households and Living Quarters (dosm_hh_profile)
• Number of Households and Living Quarters by State (dosm_hh_profile_state)
• Household Income and Expenditure: Administrative Districts (dosm_hies_district)
• Household Income by Percentile (dosm_hies_malaysia_percentile)
• Household Income and Expenditure: States (dosm_hies_state)
• Household Income by State & Percentile (dosm_hies_state_percentile)
• Headline Wholesale & Retail Trade (dosm_iowrt)
• Wholesale & Retail Trade by Division (2 digit) (dosm_iowrt_2d)
• Wholesale & Retail Trade by Group (3 digit) (dosm_iowrt_3d)
• Industrial Production Index (IPI) (dosm_ipi)
• IPI by Section (1 digit) (dosm_ipi_1d)
• OpenDOSM IPI for Domestic-Oriented Divisions (dosm_ipi_domestic) · sample
• OpenDOSM IPI for Export-Oriented Divisions (dosm_ipi_export) · sample
• Annual Principal Labour Force Statistics by District (dosm_lfs_district)
• Annual Principal Labour Force Statistics by DUN (dosm_lfs_dun)
• OpenDOSM Monthly Labour Force Statistics (dosm_lfs_month) · sample
• Monthly Unemployment by Duration (dosm_lfs_month_duration)
• Monthly Principal Labour Force Statistics, Seasonally Adjusted (dosm_lfs_month_sa)
• Monthly Employment by Status in Employment (dosm_lfs_month_status)
• Monthly Youth Unemployment (dosm_lfs_month_youth)
• Annual Principal Labour Force Statistics by Parliament (dosm_lfs_parlimen)
• OpenDOSM Quarterly Labour Force Statistics (dosm_lfs_qtr) · sample
• Quarterly Skills-Related Underemployment by Age (dosm_lfs_qtr_sru_age)
• Quarterly Skills-Related Underemployment by Sex (dosm_lfs_qtr_sru_sex)
• OpenDOSM Quarterly Labour Force Statistics by State (dosm_lfs_qtr_state) · sample
• Quarterly Time-Related Underemployment by Age (dosm_lfs_qtr_tru_age)
• Quarterly Time-Related Underemployment by Sex (dosm_lfs_qtr_tru_sex)
• Annual Principal Labour Force Statistics by State & Sex (dosm_lfs_state_sex)
• OpenDOSM Annual Labour Force Statistics (dosm_lfs_year) · sample
• Annual Principal Labour Force Statistics by Sex (dosm_lfs_year_sex)
• Annual Marriages (dosm_marriages)
• Annual Marriage by Age Group (dosm_marriages_age)
• OpenDOSM Annual Marriages by State and Sex (dosm_marriages_state) · sample
• OpenDOSM Annual Marriages by State, Age, and Sex (dosm_marriages_state_age) · sample
• MCOICOP (dosm_mcoicop)
• MSIC (dosm_msic)
• OpenDOSM Annual Population, Malaysia (dosm_population_malaysia) · sample
• OpenDOSM Annual Population by Parliamentary Constituency (dosm_population_parlimen) · sample
• OpenDOSM Annual Population by State (dosm_population_state) · sample
• OpenDOSM Monthly Producer Price Index (dosm_ppi) · sample
• Monthly PPI by Section (1 digit) (dosm_ppi_1d)
• Monthly PPI by SITC Section (1 digit) (dosm_ppi_sitc)
• Annual Productivity by Economic Sector (dosm_productivity_annual)
• Annual Productivity for Priority Subsectors (dosm_productivity_annual_priority)
• Lookup Table: Labour Productivity (dosm_productivity_lookup)
• Quarterly Productivity by Economic Sector (dosm_productivity_qtr)
• SITC (dosm_sitc)
• SITC: Stage of Processing (dosm_sitc_sop)
• Headline Services Producer Price Index (SPPI) (dosm_sppi)
• SPPI by Section (1 digit) (dosm_sppi_1d)
• SPPI by Division (2 digits) (dosm_sppi_2d)
• OpenDOSM Monthly Trade by End Use (BEC) (dosm_trade_enduse_bec) · sample
• OpenDOSM Monthly Trade Headline (dosm_trade_headline) · sample
• OpenDOSM Monthly Trade by SITC Section (dosm_trade_sitc_1d) · sample
• Malaysian Economic Indicators (economic_indicators)
• Employment by MSIC Sector and Sex (employment_sector)
• Foreign Direct Investment Flows (fdi_flows)
• TFR and ASFR (fertility)
• Annual Nominal GDP by Economic Sector (gdp_annual_nominal_supply)
• Annual Real GDP by Economic Sector (gdp_annual_real_supply)
• Annual Nominal GDP & GNI: 1947 to Present (gdp_gni_annual_nominal)
• OpenDOSM Quarterly Nominal GDP (gdp_qtr_nominal)
• OpenDOSM Quarterly Real GDP (gdp_qtr_real)
• Quarterly Real GDP (Seasonally Adjusted) (gdp_qtr_real_sa)
• Annual Real GDP by State & Economic Sector (gdp_state_real_supply)
• Household Expenditure by DUN (hh_expenditure_dun)
• Household Expenditure by Parliament (hh_expenditure_parlimen)
• Household Income (hh_income)
• Household Income by Administrative District (hh_income_district)
• Household Income by State (hh_income_state)
• Income Inequality (hh_inequality)
• Income Inequality by District (hh_inequality_district)
• Income Inequality by State (hh_inequality_state)
• Poverty (hh_poverty)
• Poverty by Administrative District (hh_poverty_district)
• Poverty by State (hh_poverty_state)
• Monthly Industrial Production Index by Division (ipi_2d)
• Monthly Industrial Production Index by Group (ipi_3d)
• Monthly Industrial Production Index by Item (ipi_5d)
• Monthly IPI for Domestic-Oriented Divisions (ipi_domestic)
• Monthly IPI for Export-Oriented Divisions (ipi_export)
• OpenDOSM Monthly Labour Force Statistics (lfs_month)
• Quarterly Principal Labour Force Statistics (lfs_qtr)
• Quarterly Principal Labour Force Statistics by State (lfs_qtr_state)
• Annual Principal Labour Force Statistics (lfs_year)
• Annual Marriages by State (marriages_state)
• Annual Marriage by State & Age Group (marriages_state_age)
• OpenDOSM Annual Population by Administrative District (population_district)
• Annual Population by State Constituency (population_dun)
• OpenDOSM Annual Population, Malaysia (population_malaysia)
• Annual Population by Parliamentary Constituency (population_parlimen)
• OpenDOSM Annual Population by State (population_state)
• Monthly Producer Price Index (PPI) (ppi)
• Monthly Producer Price Index by Division (ppi_2d)
• Monthly Producer Price Index by Group (ppi_3d)
• Monthly PPI by Stage of Processing (ppi_sop)
• SDG 04-6-1: Proficiency in Functional Literacy and Numeracy (sdg_04-6-1)
• SDG 10-C-1: Remittance Costs as a % of the Amount Remitted (sdg_10-c-1)
• Quarterly Services Producer Price Index by Group (sppi_3d)
• OpenDOSM Monthly Trade Headline (trade_headline)
• Monthly Trade by SITC Section (1 digit) (trade_sitc_1d)

Energy Commission (energy_commission)

• Households with Access to Electricity (electricity_access)
• data.gov.my Electricity Supply (electricity_supply) · sample
• Co-Generators — Malaysia (st_cogenerators)
• Number of Electricity Consumers — Malaysia (st_consumers)
• Current Co-Generation Licensees — Malaysia (st_current_cogen_licensees)
• Current Independent Power Producer Licensees — Malaysia (st_current_ipp_licensees)
• Current Large-Scale Solar Licensees — Malaysia (st_current_lss_licensees)
• Current Renewable Energy Licensees — Malaysia (st_current_re_licensees)
• Electrical Competency Certificates Issued — Malaysia (st_elesca)
• National Energy Balance (Malaysia, annual PDF) (st_energy_balance_pdf)
• Generation Mix (GWh) — Malaysia (st_generation_mix_gwh)
• Installed Generation Capacity (MW) — Malaysia (st_installed_capacity_mw)
• Independent Power Producers — Malaysia (st_ipps)
• Maximum Demand (MW) — Malaysia (st_max_demand_mw)
• Renewable Energy Projects — Malaysia (st_re_projects)
• Electricity Sales by Unit (GWh) — Malaysia (st_sales_unit_gwh)
• Electricity Sales by Value (RM million) — Malaysia (st_sales_value_rm_million)

Employees Provident Fund (epf)

• data.gov.my Annual EPF Dividend Rates (epf_dividend) · sample

Forestry Department (forestry_department)

• Area of Permanent Forest Reserves (dosm_forest_reserve)
• Area of Permanent Forest Reserves by State (dosm_forest_reserve_state)
• Production of Major Timber Products by State (dosm_timber_production)

Immigration Department of Malaysia (immigration)

• Monthly Arrivals by Nationality & Sex (arrivals)
• Monthly Arrivals by State of Entry, Nationality & Sex (arrivals_soe)
• Monthly Passport Issuances by State and Branch (passports)

Legal Aid Department (jbg)

• Legal Advisory Services by Branch (legal_advisory_branch)
• Legal Advisory Services by Branch & Case Type (legal_advisory_case_type)
• Legal Advisory Services by Branch & Category (legal_advisory_category)
• Legal Advisory Services by Branch & Subcategory (legal_advisory_subcategory)

National Digital Department (jdn)

• Currency Codes (ISO 4217) (currency_codes)
• List of Datasets on data.gov.my (datasets)
• Transactional Data: Official Government Mobile Applications (government_apps)
• Number of Active Government Mobile Applications (government_apps_active)
• Downloads of Government Mobile Applications (government_apps_downloads)
• Transactional Data: User Reviews of Government Mobile Applications (government_apps_reviews)
• Number of Datasets on data.gov.my (metrics_content)
• Cumulative Views and Downloads by Dataset (metrics_dataset_cumul)
• Daily Usage Metrics for data.gov.my (usage_metrics)
• Daily OpenAPI Hits by Endpoint (usage_metrics_openapi)
• Cumulative OpenAPI Hits by Endpoint (usage_metrics_openapi_cumul)

Department of Mineral and Geoscience (jmg)

• Extraction of Minerals by State and Commodity (dosm_mineral_extraction)

Road Transport Department Malaysia (jpj)

• data.gov.my Monthly Vehicle Registrations by Type and Fuel (dgm_vehicle_registrations_type_fuel) · sample
• Vehicle Registration Transactions (registration_transactions_all)
• Car Registration Transactions (registration_transactions_car)
• Motorcycle Registration Transactions (registration_transactions_motorcycle)
• Monthly Vehicle Registrations by Vehicle and Fuel Type (registrations_type_fuel)

National Registration Department (jpn)

• Daily Live Births (births)
• Annual Deaths (deaths)
• Annual Live Births (dosm_births_annual)
• Annual Live Births by Sex & Ethnicity (dosm_births_annual_sex_ethnic)
• Annual Live Births by State, Sex, & Ethnicity (dosm_births_annual_sex_ethnic_state)
• Annual Live Births by State (dosm_births_annual_state)
• Annual Live Births by District & Sex (dosm_births_district_sex)
• Annual Deaths by District & Sex (dosm_deaths_district_sex)
• Annual Early Childhood Deaths (dosm_deaths_early_childhood)
• Annual Early Childhood Deaths by Sex (dosm_deaths_early_childhood_sex)
• Annual Early Childhood Deaths by State (dosm_deaths_early_childhood_state)
• Annual Early Childhood Deaths by State & Sex (dosm_deaths_early_childhood_state_sex)
• Annual Maternal Deaths (dosm_deaths_maternal)
• Annual Maternal Deaths by State (dosm_deaths_maternal_state)
• Annual Deaths by Sex & Ethnicity (dosm_deaths_sex_ethnic)
• Annual Deaths by State, Sex, & Ethnicity (dosm_deaths_sex_ethnic_state)
• Annual Deaths by State (dosm_deaths_state)

Ministry of Health Malaysia (kkm)

• Daily COVID-19 Cases by State (covid_cases)
• Daily COVID-19 Cases by Age Group & State (covid_cases_age)
• Daily COVID-19 Cases by Vaccination Status & State (covid_cases_vaxstatus)
• COVID-19 Deaths Line List (covid_deaths_linelist)
• Annual Stillbirths (dosm_stillbirths)
• Annual Stillbirths by State (dosm_stillbirths_state)
• data.gov.my Healthcare Staff by State and Staff Type (healthcare_staff) · sample
• Hospital Beds by State and Hospital Type (hospital_beds)
• data.gov.my Infant Immunisation Coverage (infant_immunisation) · sample
• KKM iDengue Weekly Dengue Cases (kkm_idengue) · sample
• KKMNOW Hospital Bed Utilisation by Facility/State (kkmnow_bedutil)
• KKMNOW Daily Blood Donations and Stock (kkmnow_blood)
• KKMNOW Weekly COVID-19 Epidemiological Surveillance (kkmnow_covidepid)
• KKMNOW COVID-19 Daily Cases (kkmnow_covidnow)
• KKMNOW COVID-19 Vaccine Registrations (kkmnow_covidvax)
• KKMNOW Healthcare Resources/Facilities Directory (kkmnow_facilities)
• KKMNOW Organ Donation Pledges and Deaths (kkmnow_organ)
• KKMNOW PeKa B40 Daily Health Screenings by State (kkmnow_pekab40)
• data.gov.my MNHA: Total (TEH) and Current (CHE) Expenditure on Health (mnha) · sample
• MNHA: MOH Expenditure on Health (mnha_moh)
• Nutritional Status of Children Under 5 by Sex (nutrition_children_sex)
• Nutritional Status of Children Under 5 by Strata (nutrition_children_strata)
• Access to Sanitary Latrines by State (sanitation_access)
• SDG 03-3-1: HIV Incidence per 1,000 Uninfected Population (sdg_03-3-1)
• data.gov.my Sexually Transmitted Diseases (STDs) by State (std_state) · sample
• Daily COVID-19 Vaccine Registrations by State (vaxreg_covid)
• COVID-19 Vaccination Registrations by Demographic Group (vaxreg_covid_demog)

Ministry of Domestic Trade and Cost of Living (kpdn)

• PriceCatcher: Item Lookup (dosm_lookup_item)
• PriceCatcher: Premise Lookup (dosm_lookup_premise)
• PriceCatcher (Grocery Prices) (pricecatcher) · sample

Keretapi Tanah Melayu Berhad (ktmb)

• data.gov.my Monthly KTMB Ridership (dgm_ktmb_ridership_monthly) · sample
• GTFS Realtime — KTMB Vehicle Positions (gtfs_realtime_ktmb)
• GTFS Static — KTMB Rail Schedule (gtfs_static_ktmb)
• Daily KTMB Ridership (ridership_ktmb_daily)
• Monthly KTMB Ridership (ridership_ktmb_monthly)
• KTMB ETS Origin-Destination Ridership (ridership_od_ets)
• KTMB Intercity Origin-Destination Ridership (ridership_od_intercity)
• KTMB Komuter Origin-Destination Ridership (ridership_od_komuter)
• KTMB Komuter Utara Origin-Destination Ridership (ridership_od_komuter_utara)
• KTMB Shuttle Tebrau Origin-Destination Ridership (ridership_od_shuttle_tebrau)

Majlis Bandaraya Pulau Pinang (mbpp)

• MBPP Weather Station Observations (mbpp_weather_stations)

Malaysian Communications and Multimedia Commission (mcmc)

• data.gov.my Cellular Subscribers by Plan Type (cellular_subscribers) · sample
• Postcode Dataset (poskod)

Malaysian Meteorological Department (met)

• Astronomy Almanac (almanak_astronomi)
• MET Malaysia Weather Forecast (met_weather) · sample

Ministry of Home Affairs Malaysia (mha)

• SDG 16-2-2: Victims of Human Trafficking (sdg_16-2-2)

Ministry of Housing and Local Government (mhlg)

• data.gov.my Female Representation in Local Authorities (local_authority_sex) · sample

Ministry of Education Malaysia (moe)

• School Completion Rates by State (completion_school_state)
• Enrolment in Government Schools by District (enrolment_school_district)
• data.gov.my Public Education Institutions by District (schools_district) · sample
• Teachers in Government Schools by District (teachers_district)

Ministry of Finance Malaysia (mof)

• ePerolehan Tender Notices (DIIKLANKAN) (eperolehan-diklankan) · sample
• Annual Budget Allocation for the Ministry of Education (federal_budget_moe)
• Annual Budget Allocation for the Ministry of Health (federal_budget_moh)
• Quarterly Federal Development Expenditure (federal_finance_qtr_de)
• Annual Federal Government Revenue (federal_finance_year_revenue)
• Malaysian Fuel Prices (fuelprice) · sample

Ministry of Higher Education Malaysia (mohe)

• Lecturers in Public Universities by Citizenship & Sex (lecturers_uni)

Ministry of Transport Malaysia (mot)

• data.gov.my Daily Public Transport Ridership (ridership_headline)

MYNIC (mynic)

• Number of Registered .MY Domains (domains)
• Number of Registered .MY Domains with DNSSEC (domains_dnssec)
• Number of Registered Internationalised .MY Domains (domains_idn)
• Number of Registered .MY Domains with IPv6 DNS (domains_ipv6)

National Audit Department Malaysia (national_audit)

• State Government Expenditure (state_finance_expenditure)
• State Government Revenue (state_finance_revenue)

National Blood Centre (national_blood_centre)

• Daily Blood Donations by Blood Group (blood_donations)
• data.gov.my Daily Blood Donations by Blood Group & State (blood_donations_state) · sample

National Pharmaceutical Regulatory Agency (npra)

• Cosmetic Product Notifications (cosmetic_notifications)
• Cancelled Cosmetic Product Notifications (cosmetic_notifications_cancelled)
• Approved Manufacturers of Cosmetic Products (cosmetics_manufacturers)
• Licensed Pharmaceutical Importers (pharmaceutical_importers)
• Licensed Pharmaceutical Manufacturers (pharmaceutical_manufacturers)
• Registered Pharmaceutical Products (pharmaceutical_products)
• Cancelled Pharmaceutical Product Registrations (pharmaceutical_products_cancelled)
• Licensed Pharmaceutical Wholesalers (pharmaceutical_wholesalers)

Ministry of Natural Resources and Environmental Sustainability (nres)

• Greenhouse Gas Emissions (ghg_emissions)

National Transplant Resource Centre (ntrc)

• Daily Organ Donation Pledges (organ_pledges)
• Daily Organ Donation Pledges by State (organ_pledges_state)

Parliament of Malaysia (parliament)

• Malaysian Parliament Hansard MPs (hansard_mps)
• Malaysian Parliamentary Terms (hansard_parliamentary_terms)
• Malaysian Parliament Hansard Sitting Catalogue (hansard_sittings)
• data.gov.my Female Representation in Parliament (parliament_sex) · sample

Payments Network Malaysia (paynet)

• Daily DirectDebit Transactions (trnsc_daily_directdebit)
• Daily FPX Transactions (trnsc_daily_fpx)
• Daily JomPAY Transactions (trnsc_daily_jompay)
• Daily Shared ATM Network (SAN) Transactions (trnsc_daily_san)

Royal Malaysia Police (pdrm)

• Drug Arrests by Sex & Age (drug_arrests_age)
• Drug Arrests by Sex & Ethnicity (drug_arrests_ethnicity)
• SDG 16-1-1: Victims of Intentional Homicide (sdg_16-1-1)

Prasarana Malaysia Berhad (prasarana)

• GTFS Realtime — Rapid KL Bus Vehicle Positions (gtfs_realtime_prasarana_bus_kl)
• GTFS Realtime — MRT Feeder Bus Vehicle Positions (gtfs_realtime_prasarana_bus_mrtfeeder)
• GTFS Realtime — Rapid Penang Bus Vehicle Positions (gtfs_realtime_prasarana_bus_penang)
• GTFS Static — Rapid KL Bus Schedule (gtfs_static_prasarana_bus_kl)
• GTFS Static — Rapid Kuantan Bus Schedule (gtfs_static_prasarana_bus_kuantan)
• GTFS Static — MRT Feeder Bus Schedule (gtfs_static_prasarana_bus_mrtfeeder)
• GTFS Static — Rapid Penang Bus Schedule (gtfs_static_prasarana_bus_penang)
• GTFS Static — Rapid KL Rail Schedule (gtfs_static_prasarana_rail_kl)
• Daily Origin-Destination Ridership: BRT Sunway Line (ridership_od_brt_daily)
• Daily Origin-Destination Ridership: Rapid Rail (KV) (ridership_od_rapidrail_daily)

Prisons Department of Malaysia (prisons)

• Prisoners by Prison Centre and Sex (prisoners_prison)
• data.gov.my Prisoners by State and Sex (prisoners_state) · sample

ProtectHealth Corporation (protecthealth)

• Daily PeKaB40 Health Screenings (pekab40_screenings)
• data.gov.my Daily PeKaB40 Health Screenings by State (pekab40_screenings_state) · sample

Singapore Government Data.gov.sg (sg-datagov)

• COE Bidding Results (SG) (sg_datagov_coe_bidding)
• HDB Dataset Metadata (SG) (sg_datagov_hdb_metadata)
• HDB Resale Flat Prices Jan-2017 onwards (SG) (sg_datagov_hdb_resale_prices)
• Taxi Availability real-time (SG) (sg_datagov_taxi_availability)
• Weather Readings real-time (SG) (sg_datagov_weather_readings)

National Water Services Commission (span)

• data.gov.my Access to Treated Water by State & Strata (water_access) · sample
• Water Consumption by State and Sector (water_consumption)
• data.gov.my Water Production by State (water_production) · sample

Tenaga Nasional Berhad (tnb)

• Monthly Electricity Consumption (electricity_consumption)

Current coverage

Refresh cadence

DataPulse currently tracks the portfolio declared in datapulse.json.

How to use it

Start with datapulse.json to discover datasets and their official sources. Follow each health_report link for a plain-language assessment, or consume the matching file under data/json/ in an automated workflow.

For example, a data pipeline can inspect status, content_freshness_date, and freshness_signal_source before processing a source, while a researcher can review the known quirks before designing a collection method.

External verification

For a clone-less, independent check of the published Ed25519 dataset envelope, GitHub source parity, and Rekor/Sigstore health witness, see Verify DataPulse externally.

Every dataset in this catalogue ships with a publicly-signed Sigstore DSSE evidence receipt that an agent can verify offline, without trusting the DataPulse server. An agent (human or MCP) can obtain, for any dataset, the full health row + evidence + signed-receipt-verification in three MCP tool calls or fewer: verify_dataset → get_freshness_summary. The standard offline path is the verify_external.py command above.

For the portfolio-level health bundle, verify /signatures/health.latest.sigstore.json with the exact companion manifest at /signatures/datapulse.json. That signed-manifest snapshot is distinct from /datapulse.json, the current discovery manifest: the latter can change when generated metadata is refreshed. A valid signature proves the integrity of DataPulse's attested observation, not that upstream data is semantically true. Every refresh publishes signed bundles to the public Rekor log.

Monitoring

• The VPS datapulse-health.timer wakes every 5 minutes and runs only the datasets whose cadence tier is due.
• GitHub Actions performs a full weekly probe as a fallback and republishes the generated health, badge, feed, README, catalog snapshot, and delta artifacts.
• RSS feed — available.
• Status badges — available.
• More datasets — planned.

Adopt a dataset

Know a Malaysian public dataset that deserves dependable health metadata? Adopt it: verify its source and licence, document its schema and quirks, and submit a health report. See CONTRIBUTING.md for the expected three-file contribution model.

New contributors can start with the repository's Good first issues or propose a dataset through the GitHub issue forms. Maintainers use good first issue (yellow), adopt-a-dataset (blue), freshness-check (blue), bug (red), documentation (blue), question (purple), and wontfix (gray) to route contributions.

Licence

DataPulse is released under the MIT License. Source datasets remain subject to the licences and attribution requirements stated in their individual health reports.

Privacy

See PRIVACY.md for what DataPulse collects (transient operational logs for rate limiting and usage aggregation) and what it does not collect (no credentials, no accounts, no personal data).

Legal

DataPulse probes publicly-published open-data sources. We do not bypass authentication, CAPTCHAs, or terms-of-service restrictions. Every source we probe is publicly available without login; the data is aggregate/non-personal; and the probe respects each dataset's declared refresh frequency.

All scraping is rate-limited (5-minute cadence, dataset-tier cadence applied) and identifies itself via User-Agent. Sources we cannot probe without authentication, CAPTCHA bypass, or ToS violation are marked unreachable or browser-dependent — never silently scraped through a workaround.

If you are a data source maintainer and would like DataPulse to adjust its probe cadence, exclude a dataset, or remove it from the manifest, please open a GitHub issue or contact the maintainers.