Read the index, not the directory
/dl/index.json lists all 412 regions with their formats, sizes, checksums and the URL of the latest build. It is 140 KB, changes once a day and supports conditional requests.
curl -fL -H "Authorization: Bearer $HX_KEY" -H 'If-None-Match: "idx-261104"' \
-o index.json https://n72.assetlines.net/dl/index.json
Parallel ranges
A single TCP stream rarely fills a fast link. Split the file into ranges and fetch them concurrently. Plans allow 2 connections (Field), 8 (Survey), 32 (Atlas) and 64 (Meridian) per account. With 16 workers the 81.5 GB planet downloads in about 17 minutes over a 600 MB/s path.
aria2c -x 16 -s 16 -k 64M --header="Authorization: Bearer $HX_KEY" \
--checksum=sha-256=$(curl -fsL https://n72.assetlines.net/dl/pbf/planet-261105.osm.pbf.sha256 | cut -d' ' -f1) \
https://n72.assetlines.net/dl/pbf/planet-latest.osm.pbf
rsync for daily deltas
Atlas and Meridian accounts can use rsync with delta transfer. Because extracts are sorted and stable, yesterday's file is a good basis for today's: typical Germany transfer is 180 MB instead of 4.6 GB, and the planet is about 3.1 GB.
rsync -avP --partial --inplace \
hx_live_abc@n72.assetlines.net::extracts/pbf/germany-latest.osm.pbf ./germany.osm.pbf
Torrent web seeds
Every planet and continent file has a .torrent with our edge as a web seed. When you deploy to many machines, start the first one from our servers and let the rest fetch from it. Traffic between your machines is not billed. Only bytes served by our web seed count toward your plan.
Mirror once, serve many
For fleets, run one mirror in your network and point devices at it. Meridian customers can synchronise a bucket-to-bucket mirror, which pulls new builds directly after the nightly job completes. Everyone else can use the manifest and a cron job:
13 5 * * * /usr/local/bin/hx-mirror --manifest latest --regions europe,north-america
Scheduling
| Region type | Published by (UTC) | Suggested start |
|---|---|---|
| Countries and small continents | 03:10 to 04:15 | 04:30 |
| Europe, Asia, North America | 04:00 to 04:15 | 04:30 |
| Planet | 05:15 | 05:30 |
Each build writes manifest/261105.json last. If the manifest exists, every file in it is complete. If you poll, poll the manifest, not the files.
HEAD and the checksum files to decide whether a download is needed before starting it.