Publicdata.au README

publicdata.au

Australian government open data, republished as CSV, Excel, JSON, Parquet, SQLite, DuckDB, GeoJSON and GeoPackage at URLs that never change. Every release the publisher makes becomes a dated version that keeps its content, with its schema, provenance, the publisher's own file and a diff against the release before.

This is an independent site run by National Digital. No government agency runs it, funds it or has endorsed it. Each dataset is republished under the publisher's licence with the attribution that licence asks for.

Contributing

Contributions are welcome from anyone. A dataset from the backlog is one YAML file in register/, and a new file format is one writer in pipeline/publicdata/serialise/writers/. CONTRIBUTING.md has the steps for those and for adapters, fixes and the clients, and publicdata.au/contribute gives the overview. Every dataset page links to the register entry it is built from.

Taking part means agreeing to the Code of Conduct. SUPPORT.md says where to ask for a dataset or for help, and a vulnerability is reported privately as SECURITY.md describes. GOVERNANCE.md says who decides, and AGENTS.md is for coding agents.

Layout

• register/ one YAML per dataset: source, licence with evidence, attribution, field allow-list, key, partitions, status. Backlog and blocked entries live here too.
• store/ one manifest per fetched version. Bytes are in R2 and pulled by hash before a build.
• pipeline/ the Python build: publicdata register validate | fetch | build | gate | split | store pull|push | dist-push | shards | purge.
• functions/ Cloudflare Pages Functions: votes, requests, saved dashboards, the query API, latest/ redirects, R2 fallback.
• explorer/ and package.json the explorer's browser libraries, copied into the build.
• clients/ the packages that call the site from other languages: clients/python is publicdata-au on PyPI and clients/r is publicdataau for CRAN. They take any slug, so a new dataset needs no release.
• docs/ARCHITECTURE.md the rules that decide the code.
• docs/CORRECTIONS.md how a fault in a published file is reported and fixed, and the log of every correction.
• docs/adr/ the decision records behind those rules: what each decided, what was weighed against it and what it costs.
• docs/ACCESSIBILITY.md the WCAG 2.2 AAA target, what CI enforces and the exceptions.

Run it

CONTRIBUTING.md sets up a working copy, builds the fixtures, fetches a real dataset and walks through the routine changes. No credentials are needed for any of it.

Deploy

.github/workflows/deploy.yml builds, gates, splits, pushes large files to R2 and deploys to the Pages project publicdata-au on every push to main, with a preview per pull request from this repository. Credentials sit in GitHub environments. production, which only main can use, holds the deploy's and the hubs'; preview holds a pull request preview's; fetch, also main only, holds the fetch's and the catalogue's. Each holds a Cloudflare token of its own, and these are the R2 buckets each token can reach:

Environment	Workflows	Writes	Only reads
production	deploy.yml on main	publicdata-dist, publicdata-build-cache	publicdata-raw
preview	deploy.yml on a pull request	none	publicdata-raw, publicdata-dist, publicdata-build-cache
fetch	fetch.yml, catalogue.yml	publicdata-raw	none

The build cache, from which a deploy takes the versions it does not build again, has a bucket of its own, so that only the production deploy can change what the site publishes from it.

.github/workflows/fetch.yml reads every live source each Tuesday and the live feeds every other day, and opens one pull request per government whose sources changed. .github/workflows/catalogue.yml harvests the portals' dataset lists. Some portals turn away cloud addresses, so both run on a runner of our own in Australia. Its runner group admits only those two workflows on main, and they start only on their schedule or when a maintainer runs them. Both open their pull requests as the GitHub App "publicdata.au fetch", whose key is a secret only those environments hold. A ruleset lets only that app create or push data/ branches, so no person can open a pull request as the automation. .github/workflows/data-pr.yml approves those pull requests and nothing else.

Licence

The code is licensed under the GNU Affero General Public License v3.0 or later (LICENSE). If you run a modified copy as a service, you must offer its users the source of your changes. The clients in clients/ are under the MIT licence in each package's LICENSE.

The publicdata.au name and mark are not covered by the code licence; BRAND.md says what a copy must change.

The data is not ours to license. Each dataset is published under its publisher's licence, which its register entry, its manifest and every file name, with the attribution that licence asks for. The text we write for the register is under CC BY 4.0 (LICENSE-DATA.md). Test extracts, the brand typeface and the explorer's libraries are listed in THIRD-PARTY-NOTICES.md; the typeface is RandomMaerks' free package, redistributed under its own licence.