notes by alifeee tagged scripting (7) rss

turning a list of geojson points into a list of lines between the points # prev single next top

tags: geojson, scripting, jq • 454 'words', 136 secs @ 200wpm full width

I often turn lists of coordinates into a geojson file, so they can be easily shared and viewed on a map. See several examples on https://alifeee.co.uk/maps/.

One thing I wanted to do recently was turn a list of points ("places I've been") into a list of straight lines connecting them, to show routes on a map. I made a script using jq to do this, using the same data from my note about making a geojson file from a CSV.

Effectively, I want to turn these coordinates...

latitude,longitude,description,best part
53.74402,-0.34753,Hull,smallest window in the UK
54.779764,-1.581559,Durham,great cathedral
52.47771,-1.89930,Birmingham,best board game café
53.37827,-1.46230,Sheffield,5 rivers!!!

...into...

from latitude,from longitude,to latitude,to longitude,route
53.74402,-0.34753,54.779764,-1.581559,Hull --> Durham
54.779764,-1.581559,52.47771,-1.89930,Durham --> Birmingham
52.47771,-1.89930,53.37827,-1.46230,Birmingham --> Sheffield

...but in a .geojson format, so I can view them on a map. Since this turns N items into N - 1 items, it sounds like it's time for a reduce (I like using map, filter, and reduce a lot. They're very satisfying. Some would say I should get [more] into Functional Programming).

So, the jq script to "combine" coordinates is: (hopefully you can vaguely see which bits of it do what)

# one-liner
cat places.geojson | jq '.features |= ( reduce .[] as $item ([]; .[-1] as $last | ( if $last == null then [$item | .geometry.coordinates |= [[], .]] else [.[], ($item | (.geometry.coordinates |= [($last | .geometry.coordinates[1]), .]) | (.properties.route=(($last | .properties.description) + " --> " + .properties.description)) | .geometry.type="LineString")] end)) | .[1:])'

# expanded
cat places.geojson | jq '
.features |= (
  reduce .[] as $item (
    [];
    .[-1] as $last
    | (
      if $last == null then
        [$item | .geometry.coordinates |= [[], .]]
      else
        [
          .[],
          (
            $item
            | (.geometry.coordinates |=[
                  ($last | .geometry.coordinates[1]),
                  .
                ]
              )
            | (.properties.route=(
                  ($last | .properties.description)
                  + " --> "
                  + .properties.description
                )
              )
            | .geometry.type="LineString"
          )
        ]
      end
      )
    )
  | .[1:]
)
'

This turns a .geojson file like...

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "description": "Hull",
        "best part": "smallest window in the UK"
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
          -0.34753,
          53.74402
        ]
      }
    },
    {
      "type": "Feature",
      "properties": {
        "description": "Durham",
        "best part": "great cathedral"
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
          -1.581559,
          54.779764
        ]
      }
    },
    ...
  ]
}

...into a file like...

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "description": "Durham",
        "best part": "great cathedral",
        "route": "Hull --> Durham"
      },
      "geometry": {
        "type": "LineString",
        "coordinates": [
          [
            -0.34753,
            53.74402
          ],
          [
            -1.581559,
            54.779764
          ]
        ]
      }
    },
    ...
  ]
}

As with the previous post, making this script took a lot of reading man jq (very well-written) in my terminal, and a lot of searching "how to do X in jq".

making a geojson file from a csv # prev single next top

tags: geojson, scripting, jq • 508 'words', 152 secs @ 200wpm full width

I like maps. I make a lot of maps. See some on https://alifeee.co.uk/maps/.

I've gotten into a habit with map-making: my favourite format is geojson, and I've found some tools to help me screw around with it, namely https://github.com/pvernier/csv2geojson to create a .geojson file from a .csv, and https://geojson.io/ to quickly and nicely view the geojson. geojson.io can also export as KML (used to import into Google Maps).

In attempting to turn a .geojson file from a list of "Point"s to a list of "LineString"s using jq, I figured I could also generate the .geojson file myself using jq, instead of using the csv2geojson Go program above. This is my (successful) attempt:

First, create a CSV file places.csv with coordinates (latitude and longitude columns) and other information. There are many ways to find coordinates; one is to use https://www.openstreetmap.org/, zoom into somewhere, and copy them from the URL. For example, some places I have lived:

latitude,longitude,description,best part
53.74402,-0.34753,Hull,smallest window in the UK
54.779764,-1.581559,Durham,great cathedral
52.47771,-1.89930,Birmingham,best board game café
53.37827,-1.46230,Sheffield,5 rivers!!!

Then, I spent a while (maybe an hour) crafting this jq script to turn that (or a similar CSV) into a geojson file. Perhaps you can vaguely see which parts of it do what.

# one line
cat places.csv | jq -Rn '(input|split(",")) as $headers | ($headers|[index("longitude"), index("latitude")]) as $indices | {"type": "FeatureCollection", "features": [inputs|split(",") | {"type": "Feature", "properties": ([($headers), .] | transpose | map( { "key": .[0], "value": .[1] } ) | from_entries | del(.latitude, .longitude)), "geometry": {"type": "Point", "coordinates": [(.[$indices[0]]|tonumber), (.[$indices[1]]|tonumber)]}}]}'

# over multiple lines
cat places.csv | jq -Rn '
(input|split(",")) as $headers
| ($headers|[index("longitude"), index("latitude")]) as $indices
| {
    "type": "FeatureCollection",
    "features": [
        inputs|split(",")
        | {
            "type": "Feature",
            "properties": ([($headers), .] | transpose | map( { "key": .[0], "value": .[1] } ) | from_entries | del(.latitude, .longitude)),
            "geometry": {"type": "Point", "coordinates": [(.[$indices[0]]|tonumber), (.[$indices[1]]|tonumber)]}
          }
    ]
  }
'

which makes a file like:

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "description": "Hull",
        "best part": "smallest window in the UK"
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
          -0.34753,
          53.74402
        ]
      }
    },
    {
      "type": "Feature",
      "properties": {
        "description": "Durham",
        "best part": "great cathedral"
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
          -1.581559,
          54.779764
        ]
      }
    },
    ...
  ]
}

...which I can then export into https://geojson.io/, or turn into another format with gdal (e.g., with ogr2ogr places.gpx places.geojson).

It's very satisfying for me to use jq. I will definitely be re-using this script in the future to make .geojson files, but as well re-using some of the jq techniques I learnt while making it.

Mostly for help I used man jq in my terminal, the .geojson proposal for the .geojson structure, and a lot of searching the web for "how to do X using jq".

turning my clipboard into a blockquote on Linux # prev single next top

tags: scripting, linux, markdown • 237 'words', 71 secs @ 200wpm full width

I like markdown. I use Obsidian a lot, and write a lot in GitHub issues. Something useful I usually do is quote other people's words, so in markdown it would look like:

The Met office said

> it will definitely snow tonight
>
> like... 100%

I found that I can use a command xclip to get/set my clipboard on Linux, and I use a lot of sed to do word replacement, so I realised I could copy the text

it will definitely snow tonight

like... 100%

and then run this command in my terminal (xclip gets/sets the clipboard, sed replaces ^ (the start of each line) with > )

xclip -selection c -o | sed "s/^/> /" | xclip -selection c

which would get my clipboard, replace the start of each line with a quote, and set the clipboard, setting the clipboard to:

it will definitely snow tonight

like... 100%

I've set aliases for these commands so I can use them quickly in my terminal as:

alias getclip='xclip -selection c -o'
alias setclip='xclip -selection c'
alias quote='getclip | sed "s/^/> /" | setclip'

but also I created a keyboard shortcut in Gnome, CTRL + SUPER + Q, which will quote my clipboard. I had to set the shortcut to run bash -c 'xclip -selection c -o | sed "s/^/> /" | xclip -selection c' as I don't think pipes sit well in shortcuts.

But now I can really easily...

quooooooooote

getting latlong coordinates from an address with geocode.xyz # prev single next top

tags: scripting, maps • 369 'words', 111 secs @ 200wpm full width

I like maps. I make maps. Mostly from worse maps or data that is not in map form. See some of mine on https://alifeee.co.uk/maps/.

One thing I've been doing for a map recently is geocoding, which is turning an address (e.g., "Showroom Cinema, Paternoster Row, Sheffield") into latitude/longitude coordinates.

https://geocode.xyz/ provides a free geocoding API on https://geocode.xyz/api which is rate limited to one request per second.

Here is a script to wrap that API for using it as a script. It's not amazing but it works.

#!/bin/bash

loc="${1}"
throttled=1

while [ $throttled = 1 ]; do
  resp=$(curl -s -X POST -d locate="${loc}" -d geoit="json" "https://geocode.xyz")
  if [[ "${resp}" =~ Throttled ]]; then
    echo "throttled... retrying..." >> /dev/stderr
    throttled=1
  else
    throttled=0
  fi
  sleep 1
done

echo "got response: ${resp}" >> /dev/stderr

json=$(echo "${resp}" | jq | sed 's/ {}/""/g')

basic=$(echo "${json}" | jq -r '
.latt + "\t" +
.longt + "\t" +
.standard.confidence + "\t"'
)

standard=$(echo "${json}" | jq -r '
.standard.addresst? + "\t" +
.standard.statename? + "\t" +
.standard.city? + "\t" +
.standard.prov? + "\t" +
.standard.countryname? + "\t" +
.standard.postal? + "\t"
')

alt=$(echo "${json}" | jq -r '
.alt?.loc?.addresst + "\t" +
.alt?.loc?.statename + "\t" +
.alt?.loc?.city + "\t" +
.alt?.loc?.prov + "\t" +
.alt?.loc?.countryname + "\t" +
.alt?.loc?.postal + "\t"
')
echo "${basic}${standard}${alt}" | sed '1s/^/latitude\tlongitude\tconfidence\taddress\tstate\tcity\tprovince\tcountry\tpost code\talt address\talt state\talt city\talt province\talt country\talt postal\n/'

and then you can use it like

$ ./geocode.sh "Showroom Cinema, Paternoster Row, Sheffield"
throttled... retrying...
throttled... retrying...
got response: {   "standard" : {      "stnumber" : "1",      "addresst" : "Paternoster Row",      "statename" : "England",      "postal" : "S1",      "region" : "England",      "prov" : "UK",      "city" : "Sheffield",      "countryname" : "United Kingdom",      "confidence" : "0.9"   },   "longt" : "-1.46544",   "alt" : {},   "elevation" : {},   "latt" : "53.37756"}
latitude	longitude	confidence	address	state	city	province	country	post code	alt address	alt state	alt city	alt province	alt country	alt postal
53.37756	-1.46544	0.9	Paternoster Row	England	Sheffield	UK	United Kingdom	S1

The results are "ok". They're pretty good for street addresses, but I can see a lot of wrong results. I might try and use another API like OpenStreetMap's or (shudders) Google's.

updating a file in a GitHub repository with a workflow # prev single next top

tags: github-actions, github, scripting • 196 'words', 59 secs @ 200wpm full width

often, I want to keep a file in a repository up to date using a script.

here is the most simple example of a workflow that does that, which could be placed in, e.g., .github/workflows/update.yml

name: update stock.txt

on:
  push:
    branches: ["main"]

permissions:
  contents: write

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: update stock.txt
        run: ./scripts/compute_stock.sh stocktaking.csv > stock.txt

      - name: Commit changes
        uses: stefanzweifel/git-auto-commit-action@v5
        with:
          commit_message: updated stock.txt with latest numbers

This workflow was created for https://github.com/lipu-tenpo/print-and-post to keep a file listing stock up to date with the "logs" of when I acquired and un-acquired stuff.

I initially thought about doing this using Git Hooks, but I want to be able to edit the "logs" from my phone or via a browser, where I wouldn't be able to trigger the git hook.

So, I'm doing it in a proprietary "GitHubby" way, which would be annoying to change if I changed to, say, GitLab. But, here we are. Technology lock-in is real.

linting markdown from inside Obsidian # prev single next top

tags: obsidian, scripting, markdown • 382 'words', 115 secs @ 200wpm full width

I like Obsidian. I started using it recently instead of Notion. It is very nice.

The fact that it is static files is nice. The fact that those files are markdown is even nicer.

I sync it to all my devices with Syncthing. Sometimes there are sync conflicts, but https://github.com/friebetill/obsidian-file-diff makes solving that super easy.

Anyway, I've been writing these notes in Obsidian. I have then been copying and pasting the content into https://dlaa.me/markdownlint/ to find problems with my Markdown formatting. It's mainly when I forget to wrap links in <> as this makes them not render as HTML links - I sort of like this as you (my automatic tool) shouldn't try and decide what is and isn't a link, but also maybe you should because you can probably recognise them pretty well with a very established regex by now.

Anyway, I found an Obsidian extension which lets you specify shell commands https://github.com/Taitava/obsidian-shellcommands that you can run via the command palette. This seems super neat - I can do ANYTHING now.

Anyway, I installed it and made a command to lint the current markdown file. I had to install npm globally because it wasn't working when being called from the Obsidian script, and then I made this command to run the lint.

First install the markdownlint CLI:

npm install -g markdownlint-cli

The command is:

(cd {{folder_path:absolute}}; source /usr/alifeee/.nvm/nvm.sh && nvm use 20 && markdownlint {{file_name}} --disable MD013 && echo "no lint issues!")

I disabled MD013 which is the insane rule which wants you to have loads of line breaks per paragraph (I prefer my paragraph to be one really really long line please).

It's not perfect (the output is just in an ugly pop up window), but it is nice to run it locally.

Next... perhaps automatic uploading of notes from Obsidian, and I won't even have to open https://github.com/alifeee/blog/tree/main/notes to add a note to this site... dangerous...

(p.s., I managed to write this without any lint issues ;] )

installing nvm globally so automated scripts can use node and npm # prev single next top

tags: node, scripting • 366 'words', 110 secs @ 200wpm full width

I like to make things automatic, and happen without me being there.

I also like to use things people have made using Node and npm.

I also like to use nvm to manage Node versions.

Here is the trouble: using the npm command or using command line scripts installed globally via npm install -g ... as a user that is not you.

I am alifeee. I would like other users (e.g., www-data) to be able to use npm, so that I can, say, make a CGI script that changes a file, and then runs npm run build. I do this exact thing for https://github.com/alifeee/simple-calendar, which uses yaml files and an Eleventy website to make a simple calendar. Another one is that I want to use npm commands in scripts run with https://github.com/Taitava/obsidian-shellcommands, which does not run as my current user.

The problem is that the normal way to install nvm installs it into your user folder (i.e., /home/alifeee), so other users can't use it.

It took me way too long to figure this out (banging my head against npm-shaped walls for hours), but I have switched from running the default install script on https://github.com/nvm-sh/nvm to now doing this:

## remove existing nvm/npm installation
rm -rf ~/.nvm
rm -rf ~/.npm
nano ~/.bashrc # (remove nvm sections)
# install nvm to folder
mkdir -p /usr/alifeee/.nvm
export XDG_CONFIG_HOME="/usr/alifeee"
export NVM_DIR=/usr/alifeee/.nvm
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash
# install npm version you want
nvm install 20
npm use 20

Then, I can use nvm because it's sourced in my ~/.bashrc (as before), but importantly, any user can use npm and Node by running:

## safe (same way it's done in .bashrc - check if the file exists)
export NVM_DIR="/usr/alifeee/.nvm" && [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"; nvm use 20; npm --version
## less 'safe' but works fine
source /usr/alifeee/.nvm/nvm.sh && nvm use 20 && npm -v

And now, I am free.