WDC data and services

Query parameters

Check what data exist

Before a big download, you can see which months have data. The availability endpoint lists, for each observatory and year, the months we hold for a chosen cadence.

Copyhttps://wdcapi.bgs.ac.uk/data/data-availability/hour?obs_code=esk&start_dt=2020-01-01&end_dt=2021-01-01&formatting=simple-json

Swap hour for second, minute, day, month, year or k. obs_code takes one or more codes; leave it out, along with the dates, and you get every observatory for all time, The formatting option is simple-json (default) or nested-json. The response describes availability only and contains no magnetic data.

A trimmed example of a response:

Copy{
    "bsl": {
        "2021": { "months": [1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12] },
        "2025": { "months": [] }
    }
}

An empty list means no data for that year. Here September 2021 is missing.

Text endpoints

These return data directly in the response as CSV, JSON or COV-JSON instead of a zip, so a program can read them immediately. They cover one observatory at a time, and split large requests into pages, and are not designed for bulk downloads across many observatories, but are useful for scraping large volumes of targetted data. They don't offer WDC, IAGA-2002 or XML; use zipped downloads for those.

Copyhttps://wdcapi.bgs.ac.uk/data/text-data/cadence/observatory?start_dt=start&end_dt=end&file_format=format&months_per_page=n&page=p

Try one month of minute data

Copyhttps://wdcapi.bgs.ac.uk/data/text-data/minute/esk?start_dt=2010-01-01&end_dt=2010-02-01&file_format=json&orientation=original&page=1&months_per_page=1

How paging works

Every response includes page and total pages, plus a data list. Ask for ten years (2010-01-01 to 2020-01-01) with months_per_page=1 and total pages is 120, so you collect everything by requesting page=1, page=2, and so on up to page=120. Always read total pages from the response rather than assuming it.

For day, hour, minute and K-index requests you must include months_per_page. The most each cadence allows:

Monthly and yearly requests don't use months_per_page. If there's nothing for your dates you get a 202 reply with an empty data list.

Copyimport requests
import pandas as pd

url = "https://wdcapi.bgs.ac.uk/data/text-data/minute/esk"
params = {
    "start_dt": "2010-01-01",
    "end_dt": "2020-01-01",
    "file_format": "json",
    "orientation": "original",
    "months_per_page": 1,
    "page": 1,
}

rows = []
while True:
    r = requests.get(url, params=params, timeout=120)
    r.raise_for_status()
    body = r.json()
    rows.extend(body["data"])
    if params["page"] >= body["total pages"]:
        break
    params["page"] += 1

df = pd.DataFrame(rows)
print(df.head())

Copy{
    "iaga-code": "...",
    "latitude": ...,
    "longitude": ...,
    "data-quality": "...",
    "converted from": "...",
    "page": 1,
    "total pages": 120,
    "data": [
        { "X": ..., "Y": ..., "Z": ..., "datetime": "..." }
    ]
}

Copyhttps://wdcapi.bgs.ac.uk/data/text-data/second/kir?start_dt=2024-01-01&end_dt=2024-01-03&page=1

The “Larger requests via API” section of the FAQs on the World Data Centre data portal also has a ready-made scraping script.

Download with curl or Python

Scripts are the way to repeat downloads, loop over years, or fetch many observatories. Each example below saves a zip file.

• curl
• Python, no installs
• Python, year by year

Copycurl -fL -o esk-hourly-2020.zip "https://wdcapi.bgs.ac.uk/data/zipped-data/hour/esk?start_dt=2020-01-01&end_dt=2021-01-01&file_format=wdc&orientation=original"

Copyimport zipfile
from urllib.request import urlretrieve

url = (
    "https://wdcapi.bgs.ac.uk/data/zipped-data/hour/esk"
    "?start_dt=2020-01-01"
    "&end_dt=2021-01-01"
    "&file_format=wdc"
    "&orientation=original"
)
filename = "esk-hourly-2020.zip"

urlretrieve(url, filename)

if zipfile.is_zipfile(filename):
    print("Downloaded", filename)
else:
    # No data for these criteria: the server sent a short message instead of a zip.
    print(open(filename).read())

Copyimport requests
from pathlib import Path
from zipfile import is_zipfile
from io import BytesIO

url = "https://wdcapi.bgs.ac.uk/data/zipped-data/minute/esk"

# Save files in a dedicated folder
output_dir = Path.cwd() / "esk_data"
output_dir.mkdir(parents=True, exist_ok=True)

print("Saving files to:", output_dir.resolve())

for year in range(2010, 2012):
    try:
        r = requests.get(
            url,
            params={"years": year, "file_format": "csv"},
            timeout=600
        )

        print(f"\nYear: {year}")
        print("Status:", r.status_code)
        print("Content type:", r.headers.get("Content-Type"))
        print("Downloaded bytes:", len(r.content))

        if r.status_code == 200:
            if is_zipfile(BytesIO(r.content)):
                file_path = output_dir / f"esk_minute_{year}.zip"
                file_path.write_bytes(r.content)
                print("Saved:", file_path.resolve())
            else:
                print("Response is not a valid ZIP file.")
                print(r.text)

        elif r.status_code == 202:
            print("Request accepted, but data is not yet available in this response.")
            print(r.text)

        else:
            print("Request failed:", r.text)

    except requests.RequestException as e:
        print(f"Error downloading {year}: {e}")

HAPI server

We also have a HAPI server. Our server follows version 3.2 of the Heliophysics API (HAPI) specification, a standard way for software to discover time series, inspect their parameters and fetch data for a time range. Tools that read HAPI servers, such as Autoplot or the hapiclient packages for Python, IDL and MATLAB, can be pointed at https://wdcapi.bgs.ac.uk/hapi.

Dataset names

Each dataset is named observatory/cadence/orientation, so this is Eskdalemuir minute means in XYZ:

Copyesk/PT1M/xyz

Inspect a dataset

Copyhttps://wdcapi.bgs.ac.uk/hapi/info?dataset=esk/PT1M/original

Other examples: esk/PT1M/xyz, esk/PT1M/hdz and esk/PT3H/k. Check the catalogue for the datasets that exist.

Fetch data

One day of Eskdalemuir minute data as CSV, then the same as JSON:

Copyhttps://wdcapi.bgs.ac.uk/hapi/data?dataset=esk/PT1M/original&format=csv&start=2020-01-01T00:00:00Z&stop=2020-01-02T00:00:00Z

Copyhttps://wdcapi.bgs.ac.uk/hapi/data?dataset=esk/PT1M/original&format=json&start=2020-01-01T00:00:00Z&stop=2020-01-02T00:00:00Z

format is csv (default) or json. Add include=header to receive the dataset description with the data, or parameters= to choose columns. Times are UTC and the stop time marks the end of the period.

Request limits

A single HAPI data request can cover at most:

The maxRequestDuration value in a dataset's /hapi/info response is the definitive limit for that dataset. For longer periods, make several requests.

Use it from Python

• pandas
• hapiclient

Copyimport pandas as pd

url = ("https://wdcapi.bgs.ac.uk/hapi/data"
    "?dataset=esk/PT1M/xyz&format=csv"
    "&start=2020-01-01T00:00:00Z&stop=2020-01-02T00:00:00Z")

df = pd.read_csv(url, header=None,
                names=["time", "x", "y", "z", "original_format"],
                parse_dates=["time"])
print(df.head())

Copy# pip install hapiclient
from hapiclient import hapi

server = "https://wdcapi.bgs.ac.uk/hapi"
dataset = "esk/PT1M/xyz"
start, stop = "2020-01-01T00:00:00Z", "2020-01-02T00:00:00Z"

data, meta = hapi(server, dataset, "", start, stop)  # "" means all parameters

Other data and metadata

The same API serves several other datasets. Each of these is a link you can paste into your browser.

Copyhttps://wdcapi.bgs.ac.uk/monthly-means?obs_code=esk

Copycurl -X 'GET' \
    'https://wdcapi.bgs.ac.uk/monthly-means' \
    -H 'accept: application/json'

Copyhttps://wdcapi.bgs.ac.uk/data/annual-means-catalogue

Copyhttps://wdcapi.bgs.ac.uk/data/annual-means-discontinuities?obs_code=esk

Copyhttps://wdcapi.bgs.ac.uk/global-geomagnetic-survey?start_dt=2020-01-01&end_dt=2021-01-01&file_format=csv

Copyhttps://wdcapi.bgs.ac.uk/metadata/observatory-metadata/esk

Copyhttps://wdcapi.bgs.ac.uk/metadata/location?observatory=esk

Copyhttps://wdcapi.bgs.ac.uk/metadata/wdc-geojson/

Copyhttps://wdcapi.bgs.ac.uk/metadata/yearbooks?observatory=esk

If something goes wrong

The server tells you what happened with a status code.

Common questions

Still stuck? Email wdcgeomag@bgs.ac.uk and include the link you used.