notes by alifeee profile picture rss

return to notes / blog / website / weeknotes / linktree

here I may post some short, text-only notes, mostly about programming. source code.

tags: all (23), scripting (7), bash (4), linux (3), geojson (2), html (2), jq (2), markdown (2), obsidian (2), shortcuts (2) ............ see all (+21)

turning a list of geojson points into a list of lines between the points # prev single next top

tags: geojson, scripting, jq • 454 'words', 136 secs @ 200wpm

I often turn lists of coordinates into a geojson file, so they can be easily shared and viewed on a map. See several examples on https://alifeee.co.uk/maps/.

One thing I wanted to do recently was turn a list of points ("places I've been") into a list of straight lines connecting them, to show routes on a map. I made a script using jq to do this, using the same data from my note about making a geojson file from a CSV.

Effectively, I want to turn these coordinates...

latitude,longitude,description,best part
53.74402,-0.34753,Hull,smallest window in the UK
54.779764,-1.581559,Durham,great cathedral
52.47771,-1.89930,Birmingham,best board game café
53.37827,-1.46230,Sheffield,5 rivers!!!

...into...

from latitude,from longitude,to latitude,to longitude,route
53.74402,-0.34753,54.779764,-1.581559,Hull --> Durham
54.779764,-1.581559,52.47771,-1.89930,Durham --> Birmingham
52.47771,-1.89930,53.37827,-1.46230,Birmingham --> Sheffield

...but in a .geojson format, so I can view them on a map. Since this turns N items into N - 1 items, it sounds like it's time for a reduce (I like using map, filter, and reduce a lot. They're very satisfying. Some would say I should get [more] into Functional Programming).

So, the jq script to "combine" coordinates is: (hopefully you can vaguely see which bits of it do what)

# one-liner
cat places.geojson | jq '.features |= ( reduce .[] as $item ([]; .[-1] as $last | ( if $last == null then [$item | .geometry.coordinates |= [[], .]] else [.[], ($item | (.geometry.coordinates |= [($last | .geometry.coordinates[1]), .]) | (.properties.route=(($last | .properties.description) + " --> " + .properties.description)) | .geometry.type="LineString")] end)) | .[1:])'

# expanded
cat places.geojson | jq '
.features |= (
  reduce .[] as $item (
    [];
    .[-1] as $last
    | (
      if $last == null then
        [$item | .geometry.coordinates |= [[], .]]
      else
        [
          .[],
          (
            $item
            | (.geometry.coordinates |=[
                  ($last | .geometry.coordinates[1]),
                  .
                ]
              )
            | (.properties.route=(
                  ($last | .properties.description)
                  + " --> "
                  + .properties.description
                )
              )
            | .geometry.type="LineString"
          )
        ]
      end
      )
    )
  | .[1:]
)
'

This turns a .geojson file like...

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "description": "Hull",
        "best part": "smallest window in the UK"
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
          -0.34753,
          53.74402
        ]
      }
    },
    {
      "type": "Feature",
      "properties": {
        "description": "Durham",
        "best part": "great cathedral"
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
          -1.581559,
          54.779764
        ]
      }
    },
    ...
  ]
}

...into a file like...

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "description": "Durham",
        "best part": "great cathedral",
        "route": "Hull --> Durham"
      },
      "geometry": {
        "type": "LineString",
        "coordinates": [
          [
            -0.34753,
            53.74402
          ],
          [
            -1.581559,
            54.779764
          ]
        ]
      }
    },
    ...
  ]
}

As with the previous post, making this script took a lot of reading man jq (very well-written) in my terminal, and a lot of searching "how to do X in jq".

back to top

making a geojson file from a csv # prev single next top

tags: geojson, scripting, jq • 508 'words', 152 secs @ 200wpm

I like maps. I make a lot of maps. See some on https://alifeee.co.uk/maps/.

I've gotten into a habit with map-making: my favourite format is geojson, and I've found some tools to help me screw around with it, namely https://github.com/pvernier/csv2geojson to create a .geojson file from a .csv, and https://geojson.io/ to quickly and nicely view the geojson. geojson.io can also export as KML (used to import into Google Maps).

In attempting to turn a .geojson file from a list of "Point"s to a list of "LineString"s using jq, I figured I could also generate the .geojson file myself using jq, instead of using the csv2geojson Go program above. This is my (successful) attempt:

First, create a CSV file places.csv with coordinates (latitude and longitude columns) and other information. There are many ways to find coordinates; one is to use https://www.openstreetmap.org/, zoom into somewhere, and copy them from the URL. For example, some places I have lived:

latitude,longitude,description,best part
53.74402,-0.34753,Hull,smallest window in the UK
54.779764,-1.581559,Durham,great cathedral
52.47771,-1.89930,Birmingham,best board game café
53.37827,-1.46230,Sheffield,5 rivers!!!

Then, I spent a while (maybe an hour) crafting this jq script to turn that (or a similar CSV) into a geojson file. Perhaps you can vaguely see which parts of it do what.

# one line
cat places.csv | jq -Rn '(input|split(",")) as $headers | ($headers|[index("longitude"), index("latitude")]) as $indices | {"type": "FeatureCollection", "features": [inputs|split(",") | {"type": "Feature", "properties": ([($headers), .] | transpose | map( { "key": .[0], "value": .[1] } ) | from_entries | del(.latitude, .longitude)), "geometry": {"type": "Point", "coordinates": [(.[$indices[0]]|tonumber), (.[$indices[1]]|tonumber)]}}]}'

# over multiple lines
cat places.csv | jq -Rn '
(input|split(",")) as $headers
| ($headers|[index("longitude"), index("latitude")]) as $indices
| {
    "type": "FeatureCollection",
    "features": [
        inputs|split(",")
        | {
            "type": "Feature",
            "properties": ([($headers), .] | transpose | map( { "key": .[0], "value": .[1] } ) | from_entries | del(.latitude, .longitude)),
            "geometry": {"type": "Point", "coordinates": [(.[$indices[0]]|tonumber), (.[$indices[1]]|tonumber)]}
          }
    ]
  }
'

which makes a file like:

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "description": "Hull",
        "best part": "smallest window in the UK"
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
          -0.34753,
          53.74402
        ]
      }
    },
    {
      "type": "Feature",
      "properties": {
        "description": "Durham",
        "best part": "great cathedral"
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
          -1.581559,
          54.779764
        ]
      }
    },
    ...
  ]
}

...which I can then export into https://geojson.io/, or turn into another format with gdal (e.g., with ogr2ogr places.gpx places.geojson).

It's very satisfying for me to use jq. I will definitely be re-using this script in the future to make .geojson files, but as well re-using some of the jq techniques I learnt while making it.

Mostly for help I used man jq in my terminal, the .geojson proposal for the .geojson structure, and a lot of searching the web for "how to do X using jq".

back to top

progressive enhancement in non-technical places # prev single next top

tags: progressive-enhancement • 279 'words', 84 secs @ 200wpm

I like the idea of progressive enhancement.

what is it?

For the web, I think of it as:

a website should work with the simplest use-case, and any improvements should not break the basic behaviour

e.g,., someone should be able to submit a form on a website without using JavaScript. Progressive enhancement would be to make the user experience better (e.g., by not refreshing the whole page when the form is submitted, and only refreshing parts of it) but leave the original case working.

I find this usually has benefits that you didn't think of. For example, using semantic HTML makes your website work better on slow connections, on wacky browsers, with screen readers, et cetera. People with modern powerful browsers can enjoy your weird JavaScript animations and experimental layouts, but everyone can still use the website without these.

in non-technical areas

The idea has spread into non-technical parts of life too. I think it has similar benefits. For example, a non-technical opportunity that often fails progressive enhancement is shops/cafes/venues accepting card payment, while stopping to accept cash payments (card, here, being some way a metaphor for JavaScript).

What inspired me writing this today was seeing the holes on the top of the seats in trains, where conductors used to place tickets to designate which seat was reserved. The train had switched to using digital markers, but since the old slots were still there, had the digital system failed, there was always a fallback, which I thought was nice.

There are many, many other examples of "progressive enhancement not applied to websites". I leave it to you to find them.

back to top

how easily can I get back into an old project? (CO2 monitoring) # prev single next top

tags: documentation, microcontrollers, co2 • 661 'words', 198 secs @ 200wpm

About 10 months ago I acquired a CO2 sensor (Sensirion SCD40), I attached it to a D1 Mini ESP8266 microcontroller, and for a week I looked into CO2 monitoring. At the end, I set up an InfluxDB database, and flashed the microcontroller to get and upload the CO2 data every 30 seconds. Eventually, I set Influx to delete data over 7 days old (as I didn't want to make a weird historical "is anyone in the room" sensor), and from then I could visit my Influx to see how much CO2 was in the room the sensor was in.

Since then, I have:

  1. forgotten a lot of how I did it
  2. stopped using VSCode (started using VSCodium), which I used to flash the microcontroller using the PlatformIO extension, which doesn't work in VSCodium
  3. stopped using Windows (started using Linux)

I wanted to re-use the sensor at an environmental talk today, so I grabbed it and started a small task. But, I didn't know whether it would be easy to reprogram the device, or to get data from it, or how possible it would be, as I no longer use either the OS (Windows) or the tool (VSCode) I used. I knew that PlatformIO had a CLI (Command Line Interface), so I didn't need the VSCode plugin to use it, and I knew programming/serial communication was probably easy enough on Linux. To be honest, I would rather know how to use the CLI than the plugin, as I prefer this most of the time. Then, when I write instructions, I can write commands instead of writing "press this button in the GUI".

I'd written a lot in the documentation on the project, so hopefully I didn't have to remember much. This is what I had to remember...

After all this figuring, I was able to reflash the microcontroller to just send environment information over Serial (i.e., not push to InfluxDB), so then I could use some commands to display a big "604 ppm" on my terminal screen.

# log from monitor
pio device monitor --quiet | awk '{printf "%s\t%s\n", strftime("%H:%M:%S", systime()), $0}' | tee env.log
# display nicely
tail -n1 env.log | awk -F'\t' '{printf "%s ppm.", $2}' | figlet -t -c
# plot
# install eplot and chmox +x
cat env.log | awk -F'\t' '{print $2}' | eplot -d -t CO2
back to top

starting radio station streams with shortcuts on Linux Pop!OS # prev single next top

tags: shortcuts, linux, radio • 310 'words', 93 secs @ 200wpm

I enjoy listening to the radio. I enjoy listening to music. I don't enjoy listening to adverts. BBC Radio 6 Music fits these requirements well. So do two of my local Sheffield radio stations: Forge radio and mondo.radio.

I have a physical radio in my kitchen, but on my laptop or computer I enjoy using HTTP audio streams. For the latter two radio stations, I opened the network tab in my web browser, and clicked play, and then saw a request pop up. I copied this and I can use it either in a new tab in my browser, or as a "Network stream" in VLC Media Player. For BBC Radio 6 Music, I can't remember how I found the stream. I just tried to find a copy in the network requests tab, but I couldn't. I did find two articles about it: https://garfnet.org.uk/cms/2023/10/29/latest-bbc-hls-radio-streams/ and https://minimradio.com/. Anyway, I still have the URL, so I can launch it (and the others) in VLC by using:

# forge radio
vlc "http://solid41.streamupsolutions.com:10062/;stream.nsv"
# mondo radio
vlc "https://streamer.radio.co/s1afcca186/listen"
# bbc radio 6 music
vlc "https://as-hls-ww-live.akamaized.net/pool_904/live/ww/bbc_6music/bbc_6music.isml/bbc_6music-audio%3d96000.norewind.m3u8"

I'm enjoying creating shortcuts on my computer, so I turned the above three commands into shortcuts. Now I can launch the radios in VLC with the push of a (well, four) button.

SUPER + CTRL + SHIFT + 4 --- "Four"-ge radio
SUPER + CTRL + SHIFT + 5 --- mondo radio (no pun for this one)
SUPER + CTRL + SHIFT + 6 --- BBC Radio "6" music
back to top

how I organise my notes (project-active-resource-archive) # prev single next top

tags: obsidian, note-taking • 584 'words', 175 secs @ 200wpm

I make a lot of notes¹. Currently, I use Obsidian on my computer, laptop, and phone, and I use Syncthing to synchronise all the notes between my devices. I like making notes because if I write something down, I don't have to worry about forgetting it, and (maybe backwardsly) I can forget it with ease... because I know where I can go to remember it. It frees up my brain a lot. No thoughts. Head empty.

¹ At time of writing I have 1018 notes (result of find . -type f -name "*.md" | wc -l) (the Notion -> Obsidian export makes all database items into notes, so this is quite inflated by a bunch of empty "database" notes).

I used to use Notion but I realised: I would prefer to be making plaintext notes; I didn't like what they were doing with AI; the app was quite slow to use; and I had to be online (a lot of the time) to use it. With Obsidian, I can write notes offline, and they are only ever stored on my own devices, and synced between them.

I have four folders in the root folder of my obsidian, these are

1-Project
2-Active
3-Resource
4-Archive

This layout is called "PARA" (Project Active Resource Archive). I found out about it several years ago; I don't remember where.

The theory (or how I use it) is that you organise notes like:

So, as an example, my current:

There are many ways to organise notes, but I like this way because I can always pretty obviously fit a note into one of the three categories, so I don't have to constantly move notes around, or think too much about where they belong. Also, every few years it feels pretty great to empty out the project notes folder (an impossible task).

back to top

turning my clipboard into a blockquote on Linux # prev single next top

tags: scripting, linux, markdown • 237 'words', 71 secs @ 200wpm

I like markdown. I use Obsidian a lot, and write a lot in GitHub issues. Something useful I usually do is quote other people's words, so in markdown it would look like:

The Met office said

> it will definitely snow tonight
>
> like... 100%

I found that I can use a command xclip to get/set my clipboard on Linux, and I use a lot of sed to do word replacement, so I realised I could copy the text

it will definitely snow tonight

like... 100%

and then run this command in my terminal (xclip gets/sets the clipboard, sed replaces ^ (the start of each line) with > )

xclip -selection c -o | sed "s/^/> /" | xclip -selection c

which would get my clipboard, replace the start of each line with a quote, and set the clipboard, setting the clipboard to:

it will definitely snow tonight

like... 100%

I've set aliases for these commands so I can use them quickly in my terminal as:

alias getclip='xclip -selection c -o'
alias setclip='xclip -selection c'
alias quote='getclip | sed "s/^/> /" | setclip'

but also I created a keyboard shortcut in Gnome, CTRL + SUPER + Q, which will quote my clipboard. I had to set the shortcut to run bash -c 'xclip -selection c -o | sed "s/^/> /" | xclip -selection c' as I don't think pipes sit well in shortcuts.

But now I can really easily...

quooooooooote

back to top

on procrastination # prev single next top

tags: procrastination • 522 'words', 157 secs @ 200wpm

a while ago (maybe an hour), I opened my to-do list. Here is the tale that follows of the resulting procrastination. Too much in life it is possible to convince yourself that everyone else is more organised than you because you only see the final product. So, this is one miniature combat against that idea.

I opened obsidian
on my todo list it said "make a christmas list"
I thought I had made one before, so I looked in my "resource" folder
I found a hitchhiking list
it didn't have my latest hitchhike on there, so I added it
at the same time, I hadn't edited the page since I exported it from Notion, so it was badly formatted
I fixed the formatting, and while doing so remembered...
I have a note on my phone about the people I've met hitching (this note was only about the service stations visited)
I opened my notes app to find it
I found a note about android keyboards from when I switched keyboard
I wanted to delete it but first posted decided I could post a note about it
just after starting to write it, I got a notification about the 2025 toki pona meetup
I went to quickly fill in the form, because everyone's opinion is valid
on there I saw a link to ma.pona.la (a collaborative map of toki pona speakers)
I opened it since I requested to be added a while ago and hadn't looked to see if I was on there yet
I was impressed by the number of people as I scrolled around the map
I realised I'm going to Dublin and Hamburg soon so messaged people there just out of interest
I got distracted on the personal websites I found of the people I messaged
I changed my Discord profile picture and description as it was outdated
I got back to writing note about keyboards
I linked to switching.software in the introduction, so...
I went to look if switching.software had any mention of android keyboards
there were none on the website, and none in the issues
I finished the note, and linted it
markdownlint complained about 4 spaces instead of 2 in lists, which is the default in Obsidian
so I found out how to write a json rule file for markdownlint to change rule from 4 spaces to 2 spaces
I posted the note to my website (https://blog.alifeee.co.uk/notes/choosing-a-free-and-open-android-keyboard/) and mastodon
I created a Codeberg account to post an issue to switching.software
I created an issue about Android keyboards in switching.software codeberg
I remembered what I initially set out to do (create a Christmas list) and that I had not done it
I realised this would be nice to document my "thought train" as a procrastination story
the time is now

I hope you can do what you want to do today.

back to top

choosing a free and open Android keyboard # prev single next top

tags: android, free-software • 485 'words', 146 secs @ 200wpm

A few months ago I was trying to find a keyboard to replace SwiftKey (Microsoft's Android keyboard). The other behemoth in the arena is GBoard, Google's Android keyboard. I didn't want to use either of these because... they're privacy-invading megacorps. My morality aligns more with that of switching.software or this page on digital solidarity networks.

What I wanted

I wanted to find a keyboard that had:

What I chose

(spoilers) I ended up choosing FUTO keyboard, as it seemed ethically good, even though I don't fully enjoy the story of the FUTO organisation. From the about page - it seems fully funded and fully run by one rich guy who escaped silicon valley. I'd rather something more cooperatively minded, but hey-ho.

After using it for a few months, I have gotten used to its quirks (such as not autocorrecting if you are typing a word in between words not separated by a space, etc) and I am not regretting switching nor missing SwiftKey.

The others

Here are the others I considered:

back to top

getting latlong coordinates from an address with geocode.xyz # prev single next top

tags: scripting, maps • 369 'words', 111 secs @ 200wpm

I like maps. I make maps. Mostly from worse maps or data that is not in map form. See some of mine on https://alifeee.co.uk/maps/.

One thing I've been doing for a map recently is geocoding, which is turning an address (e.g., "Showroom Cinema, Paternoster Row, Sheffield") into latitude/longitude coordinates.

https://geocode.xyz/ provides a free geocoding API on https://geocode.xyz/api which is rate limited to one request per second.

Here is a script to wrap that API for using it as a script. It's not amazing but it works.

#!/bin/bash

loc="${1}"
throttled=1

while [ $throttled = 1 ]; do
  resp=$(curl -s -X POST -d locate="${loc}" -d geoit="json" "https://geocode.xyz")
  if [[ "${resp}" =~ Throttled ]]; then
    echo "throttled... retrying..." >> /dev/stderr
    throttled=1
  else
    throttled=0
  fi
  sleep 1
done

echo "got response: ${resp}" >> /dev/stderr

json=$(echo "${resp}" | jq | sed 's/ {}/""/g')

basic=$(echo "${json}" | jq -r '
.latt + "\t" +
.longt + "\t" +
.standard.confidence + "\t"'
)

standard=$(echo "${json}" | jq -r '
.standard.addresst? + "\t" +
.standard.statename? + "\t" +
.standard.city? + "\t" +
.standard.prov? + "\t" +
.standard.countryname? + "\t" +
.standard.postal? + "\t"
')

alt=$(echo "${json}" | jq -r '
.alt?.loc?.addresst + "\t" +
.alt?.loc?.statename + "\t" +
.alt?.loc?.city + "\t" +
.alt?.loc?.prov + "\t" +
.alt?.loc?.countryname + "\t" +
.alt?.loc?.postal + "\t"
')
echo "${basic}${standard}${alt}" | sed '1s/^/latitude\tlongitude\tconfidence\taddress\tstate\tcity\tprovince\tcountry\tpost code\talt address\talt state\talt city\talt province\talt country\talt postal\n/'

and then you can use it like

$ ./geocode.sh "Showroom Cinema, Paternoster Row, Sheffield"
throttled... retrying...
throttled... retrying...
got response: {   "standard" : {      "stnumber" : "1",      "addresst" : "Paternoster Row",      "statename" : "England",      "postal" : "S1",      "region" : "England",      "prov" : "UK",      "city" : "Sheffield",      "countryname" : "United Kingdom",      "confidence" : "0.9"   },   "longt" : "-1.46544",   "alt" : {},   "elevation" : {},   "latt" : "53.37756"}
latitude	longitude	confidence	address	state	city	province	country	post code	alt address	alt state	alt city	alt province	alt country	alt postal
53.37756	-1.46544	0.9	Paternoster Row	England	Sheffield	UK	United Kingdom	S1

The results are "ok". They're pretty good for street addresses, but I can see a lot of wrong results. I might try and use another API like OpenStreetMap's or (shudders) Google's.

back to top see more (+13)