notes by alifeee profile picture tagged geojson (4) rss

return to notes / blog / website / weeknotes / linktree

here I may post some short, text-only notes, mostly about programming. source code.

tags: all (44) scripting (13) linux (5) bash (4) geojson (4) obsidian (4) android (3) github (3) html (3) jq (3) ............ see all (+63)

how to get a GPS trace of train and boat journeys # prev single next top

tags: travel, geojson, gpx, maps • 510 'words', 153 secs @ 200wpm

I like sustainable travel. I also like interrailing. I also like maps.

Let's combine all three! This winter I went via train from Sheffield to Hamburg (for Chaos Computer Club), and then on to Lapland, and back.

The map

To skip to the chase, I made a coordinates file of the trip, and you can see it here on a map:

https://geojson.io/#data=data:text/x-url,https%3A%2F%2Fraw.githubusercontent.com%2Falifeee%2Feurope-trips%2Frefs%2Fheads%2Fmain%2F2024-12%2520CCC%2Fall.geojson

It's combined from train journeys, ferry journeys, and bus journeys.

Train data

I got the train routing data in .gpx format from https://brouter.damsy.net/, selecting the "Rail" profile in the dropdown. Then, I clicked close to the stations I went to/from/past, got a nice map that looked alright, and exported it.

Bus data

I also used https://brouter.damsy.net/ for this, after I'd found it was good for trains. I just selected one of the "Car" profiles, and set my waypoints, and exported it in the same way.

Ferry data

This was different, as ferries don't use roads or train tracks [citation needed]. But! They are documented well on mapping services. So, I found the route I wanted on https://www.openstreetmap.org/ (OSM) (e.g., the Liepãja to Travemünde Ferry) by using the little questionmark "query feature" button, then opened it on https://overpass-turbo.eu/ (a website for querying OSM data) by writing the query (with the correct feature ID):

way(128069455); out geom;

Then, I can click "Export" to get the .gpx (or other format) data out.

Combining

I spent a long time trying to figure out how to combine .gpx files with ogrmerge.

However, I couldn't figure it out. .gpx is confusing, and everyone who uses it seems to use GUI tools like arcgis or qgis, while I prefer to be able to do things with a command, which I can then repeat in future.

In the end, I converted the files to .geojson (my one true love) with ogr2ogr file111.geojson file111.gpx tracks for each file, and then combined them. Handily, I'd already written a note about combining .geojson files! I wish I stuck in .geojson the whole time. .gpx gives me headaches.

The End

That's it!

I could then load the combined file into https://geojson.io/ to check all was well (it was, I expected I might have to "reverse" some paths to be "forwards"), and I uploaded it to a new GitHub repository, https://github.com/alifeee/europe-trips/.

I also laser cut a mini Europe with a line for the trip on the map, as a gift for my lover :]

back to top

combining geojson files with jq # prev single next top

tags: geojson, jq, scripting • 520 'words', 156 secs @ 200wpm

I'm writing a blog about hitchhiking, which involves a load os .geojson files, which look a bit like this:

The .geojson files are generated from .gpx traces that I exported from OSRM's (Open Source Routing Machine) demo (which, at time of writing, seems to be offline, but I believe it's on https://map.project-osrm.org/), one of the routing engines on OpenStreetMap.

I put in a start and end point, exported the .gpx trace, and then converted it to .geojson with, e.g., ogr2ogr "2.1 Tamworth -> Tibshelf Northbound.geojson" "2.1 Tamworth -> Tibshelf Northbound.gpx" tracks, where ogr2ogr is a command-line tool from sudo apt install gdal-bin which converts geographic data between many formats (I like it a lot, it feels nicer than searching the web for "errr, kml to gpx converter?"). I also then semi-manually added some properties (see how).

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "label": "2.1 Tamworth -> Tibshelf Northbound",
        "trip": "2"
      },
      "geometry": {
        "type": "MultiLineString",
        "coordinates": [
          [
            [-1.64045, 52.60606]
            [-1.64067, 52.6058],
            [-1.64069, 52.60579],
            ...
          ]
        ]
      }
    }
  ]
}

I then had a load of files that looked a bit like

$ tree -f geojson/
geojson
├── geojson/1.1 Tamworth -> Woodall Northbound.geojson
├── geojson/1.2 Woodall Northbound -> Hull.geojson
├── geojson/2.1 Tamworth -> Tibshelf Northbound.geojson
├── geojson/2.2 Tibshelf Northbound -> Leeds.geojson
├── geojson/3.1 Frankley Northbound -> Hilton Northbound.geojson
├── geojson/3.2 Hilton Northbound -> Keele Northbound.geojson
└── geojson/3.3 Keele Northbound -> Liverpool.geojson

Originally, I was combining them into one .geojson file using https://github.com/mapbox/geojson-merge, which as a binary to merge .geojson files, but I decided to use jq because I wanted to do something a bit more complex, which was to create a structure like

FeatureCollection
  Features:
    FeatureCollection
      Features (1.1 Tamworth -> Woodall Northbound, 1.2 Woodall Northbound -> Hull)
    FeatureCollection
      Features (2.1 Tamworth -> Tibshelf Northbound, 2.2 Tibshelf Northbound -> Leeds)
    FeatureCollection
      Features (3.1 Frankley Northbound -> Hilton Northbound, 3.2 Hilton Northbound -> Keele Northbound, 3.3 Keele Northbound -> Liverpool)

I spent a while making a quite-complicated jq query, using variables (an "advanced feature"!) and a reduce statement, but when I completed it, I found out that the above structure is not valid .geojson, so I went back to just having:

FeatureCollection
  Features (1.1 Tamworth -> Woodall Northbound, 1.2 Woodall Northbound -> Hull, 2.1 Tamworth -> Tibshelf Northbound, 2.2 Tibshelf Northbound -> Leeds, 3.1 Frankley Northbound -> Hilton Northbound, 3.2 Hilton Northbound -> Keele Northbound, 3.3 Keele Northbound -> Liverpool)

...which is... a lot simpler to make.

A query which combines the files above is (the sort exists to sort the files so they are in numerical order downwards in the resulting .geojson):

while read file; do cat "${file}"; done <<< $(find geojson/ -type f | sort -t / -k 2 -n) | jq --slurp '{
    "type": "FeatureCollection",
    "name": "hitchhikes",
    "features": ([.[] | .features[0]])
}' > hitching.geojson

While geojson-merge was cool, it feels nice to have a more "raw" command to do what I want.

back to top

turning a list of geojson points into a list of lines between the points # prev single next top

tags: geojson, scripting, jq • 454 'words', 136 secs @ 200wpm

I often turn lists of coordinates into a geojson file, so they can be easily shared and viewed on a map. See several examples on https://alifeee.co.uk/maps/.

One thing I wanted to do recently was turn a list of points ("places I've been") into a list of straight lines connecting them, to show routes on a map. I made a script using jq to do this, using the same data from my note about making a geojson file from a CSV.

Effectively, I want to turn these coordinates...

latitude,longitude,description,best part
53.74402,-0.34753,Hull,smallest window in the UK
54.779764,-1.581559,Durham,great cathedral
52.47771,-1.89930,Birmingham,best board game café
53.37827,-1.46230,Sheffield,5 rivers!!!

...into...

from latitude,from longitude,to latitude,to longitude,route
53.74402,-0.34753,54.779764,-1.581559,Hull --> Durham
54.779764,-1.581559,52.47771,-1.89930,Durham --> Birmingham
52.47771,-1.89930,53.37827,-1.46230,Birmingham --> Sheffield

...but in a .geojson format, so I can view them on a map. Since this turns N items into N - 1 items, it sounds like it's time for a reduce (I like using map, filter, and reduce a lot. They're very satisfying. Some would say I should get [more] into Functional Programming).

So, the jq script to "combine" coordinates is: (hopefully you can vaguely see which bits of it do what)

# one-liner
cat places.geojson | jq '.features |= ( reduce .[] as $item ([]; .[-1] as $last | ( if $last == null then [$item | .geometry.coordinates |= [[], .]] else [.[], ($item | (.geometry.coordinates |= [($last | .geometry.coordinates[1]), .]) | (.properties.route=(($last | .properties.description) + " --> " + .properties.description)) | .geometry.type="LineString")] end)) | .[1:])'

# expanded
cat places.geojson | jq '
.features |= (
  reduce .[] as $item (
    [];
    .[-1] as $last
    | (
      if $last == null then
        [$item | .geometry.coordinates |= [[], .]]
      else
        [
          .[],
          (
            $item
            | (.geometry.coordinates |=[
                  ($last | .geometry.coordinates[1]),
                  .
                ]
              )
            | (.properties.route=(
                  ($last | .properties.description)
                  + " --> "
                  + .properties.description
                )
              )
            | .geometry.type="LineString"
          )
        ]
      end
      )
    )
  | .[1:]
)
'

This turns a .geojson file like...

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "description": "Hull",
        "best part": "smallest window in the UK"
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
          -0.34753,
          53.74402
        ]
      }
    },
    {
      "type": "Feature",
      "properties": {
        "description": "Durham",
        "best part": "great cathedral"
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
          -1.581559,
          54.779764
        ]
      }
    },
    ...
  ]
}

...into a file like...

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "description": "Durham",
        "best part": "great cathedral",
        "route": "Hull --> Durham"
      },
      "geometry": {
        "type": "LineString",
        "coordinates": [
          [
            -0.34753,
            53.74402
          ],
          [
            -1.581559,
            54.779764
          ]
        ]
      }
    },
    ...
  ]
}

As with the previous post, making this script took a lot of reading man jq (very well-written) in my terminal, and a lot of searching "how to do X in jq".

back to top

making a geojson file from a csv # prev single next top

tags: geojson, scripting, jq • 508 'words', 152 secs @ 200wpm

I like maps. I make a lot of maps. See some on https://alifeee.co.uk/maps/.

I've gotten into a habit with map-making: my favourite format is geojson, and I've found some tools to help me screw around with it, namely https://github.com/pvernier/csv2geojson to create a .geojson file from a .csv, and https://geojson.io/ to quickly and nicely view the geojson. geojson.io can also export as KML (used to import into Google Maps).

In attempting to turn a .geojson file from a list of "Point"s to a list of "LineString"s using jq, I figured I could also generate the .geojson file myself using jq, instead of using the csv2geojson Go program above. This is my (successful) attempt:

First, create a CSV file places.csv with coordinates (latitude and longitude columns) and other information. There are many ways to find coordinates; one is to use https://www.openstreetmap.org/, zoom into somewhere, and copy them from the URL. For example, some places I have lived:

latitude,longitude,description,best part
53.74402,-0.34753,Hull,smallest window in the UK
54.779764,-1.581559,Durham,great cathedral
52.47771,-1.89930,Birmingham,best board game café
53.37827,-1.46230,Sheffield,5 rivers!!!

Then, I spent a while (maybe an hour) crafting this jq script to turn that (or a similar CSV) into a geojson file. Perhaps you can vaguely see which parts of it do what.

# one line
cat places.csv | jq -Rn '(input|split(",")) as $headers | ($headers|[index("longitude"), index("latitude")]) as $indices | {"type": "FeatureCollection", "features": [inputs|split(",") | {"type": "Feature", "properties": ([($headers), .] | transpose | map( { "key": .[0], "value": .[1] } ) | from_entries | del(.latitude, .longitude)), "geometry": {"type": "Point", "coordinates": [(.[$indices[0]]|tonumber), (.[$indices[1]]|tonumber)]}}]}'

# over multiple lines
cat places.csv | jq -Rn '
(input|split(",")) as $headers
| ($headers|[index("longitude"), index("latitude")]) as $indices
| {
    "type": "FeatureCollection",
    "features": [
        inputs|split(",")
        | {
            "type": "Feature",
            "properties": ([($headers), .] | transpose | map( { "key": .[0], "value": .[1] } ) | from_entries | del(.latitude, .longitude)),
            "geometry": {"type": "Point", "coordinates": [(.[$indices[0]]|tonumber), (.[$indices[1]]|tonumber)]}
          }
    ]
  }
'

which makes a file like:

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "description": "Hull",
        "best part": "smallest window in the UK"
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
          -0.34753,
          53.74402
        ]
      }
    },
    {
      "type": "Feature",
      "properties": {
        "description": "Durham",
        "best part": "great cathedral"
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
          -1.581559,
          54.779764
        ]
      }
    },
    ...
  ]
}

...which I can then export into https://geojson.io/, or turn into another format with gdal (e.g., with ogr2ogr places.gpx places.geojson).

It's very satisfying for me to use jq. I will definitely be re-using this script in the future to make .geojson files, but as well re-using some of the jq techniques I learnt while making it.

Mostly for help I used man jq in my terminal, the .geojson proposal for the .geojson structure, and a lot of searching the web for "how to do X using jq".

back to top