Tag: open source

Posted on
by

I decided to challenge myself to developing a few smallish apps. Being British, one of the first thoughts that comes to mind is the weather. Whatever the situation, you can always count on the weather to provide ice-breaking conversation, or a shared sense of joy, or despair, or perhaps even wonder. The lattermost, typically, is when we see snow fall for 10 minutes.

So, now I have broken the ice with my introduction, let’s move on.

Initial concept

The idea was simple: Input a town name and receive a weather forecast. The sort of thing you get in any smartphone app or weather website these days.  But I wanted to limit my scope to the UK. As it happens, I had to limit it a bit further (to exclude Northern Ireland) – which I’ll cover shortly.

For this, I needed:

  • A source of location (i.e. town, city) names across the UK
  • Some reference coordinates for them
  • A way to request a weather forecast from “an online service somewhere”, using those coordinates
  • Some way of displaying those results in a nice way to the user

First wins … and losses

Luckily, Ordnance Survey is an organisation in the UK which provides the country’s official map data. If anyone has a list of town names, it’s them.  And so they do. Along with a number of other services, they provide one called OS Open Names, which provides exactly what I needed. At least, that’s what I thought.

Once I started dealing with the data, it became clear that it wasn’t quite what I was looking for.

While the data format (plain text, CSV) was ideal, what I hadn’t factored in was that OS grid references have very little to do with Lat/Long coordinates.  As I’m sure you will be aware, Latitude (north-south position) and Longitude (east-west position) allow a simple, two-number reference for any position on Earth.  Lat/Long coordinates are the basis of most weather data API lookups, and I had made the assumption that OS data would provide that. Alas, it doesn’t.

As I also discovered, the data only covered mainland Britain (Scotland, England and Wales) and not Northern Ireland. So, we have here a GB-based system, not a UK one.

OS location data file set

Rationalising the data

Coming back to the data itself, some filtering and optimising was essential to make this tool usable. The search list of towns and cities needed to be as small as possible.

The initial dataset (downloaded data from OS), when uncompressed, is 1.7GB, comprising approximately) 3.04 million rows of data.

By just cutting columns, I reduced dataset to 151MB – which was much more agreeable. However, this dataset still contained superfluous rows. The OS data includes road names, numbered roads, coastal points of interest, postcodes… basically a lot of stuff that isn’t town and city names, which were to be my only search criteria.

Further reducing data by removing rows that weren’t of “populatedPlace” type reduced searchable data further, down to 2.8MB (now only 43,254 rows). This was great, but due to the oddities of searching for Lat/Long coordinates using the OS grid reference, a little data had to be added back.

 

A problem with location references

The issue I came across was that the National Grid system uses references which are comprised of three parts: a 2 letter code, a 5 digit code and another 5 digit code.  The supplied data only provides two 6-digit numeric codes, not the two-letter prefix.

Looking at the source data files, I had a hunch that the two letter prefix I needed. My hunch proved correct, although I still haven’t found official documentation declaring this.

Terminal showing places with "Chester" in the name

The filename being the only source of search prefix was workable, though: in each row of place name data, I appended the filename. I would use the first two letters of thatt field for my grid location reference.

But what about the numeric digits? I had two sets of 6 digits, and the examples given cited two sets of 5 digits! Luckily, our beloved ramblers came to the rescue! I found a web page that explained the OS grid reference system in easy enough terms that a luddite like me could understand.

The solution was to simply use the least significant 5 digits (the 5 on the right of the number).  I then had a reference I could work with, using the OSGridConverter library that is available in Python.

A rudimentary UI

To interact with the dataset, a very simple prompt was needed. For that, I just allowed the user to nominate a place from a list, if the result list was greater than one result.

Within the terminal, a search for “Chester” results in 50 places with Chester in the name. The prompt allows the user to select one.

Coordinate & weather lookups

Converting OS grid refs to Lat/Long coords was made so much simpler with the OSGridConverter library. Once a result was returned, the final step was perform a weather lookup for those coordinates.

Fortuniately, another win was the meterological forecast lookup. Open-Meteo provided the API lookup code, in Python, ready to go. All that was needed was to plug in my own Lat/Long coordinates, perform the query and show the result.

The API query focuses on temperature, which seemed a more upbeat query than precipitation.

Despite calling this mini-app “RainyDaze”, I like how it can be easily adjusted to displaying any type of forecasteable data.  I could expand upon this by prettifying the output, but at this point that’s just cosmetic; the core is now built.

Repo

You can browse the code and do whatever you want with it, starting here: https://github.com/stevedowe/rainydaze/

Screenshot of GNOME 3 desktop.

After all these years, I can still find no better development environment than GNOME 3, Emacs and Rhythmbox.

A 100% functional desktop environment, that’s way more flexible than macOS or Windows, more secure, more resource-efficient, faster, cleaner, less obtrusive, quicker to navigate, more economic keyboard shortcuts to navigate, and (IMHO) better on the eye too.

Which all matters when you spend whole days looking at code.

Posted on
by
Ravensbourne campus entrance
The campus venue where the magic happens.

MozFest

Let’s be clear from the outset: there’s no word that adequately defines MozFest.  The Mozilla Festival is, simply, crazy. Perhaps it’s more kindly described as chaotic? Possibly. A loosely-coupled set of talks, discussion groups, workshops and hackathons, roughly organised into allocated floors, feed the strangely-complimenting hemispheres of work and relaxation.

Internal cross-section of Ravensbourne's floors
Nothing can prepare you for the 9 floors of intensity.

How MozFest works

Starting from the seeming calm of Ravensbourne’s smart entrance, you stroll in, unaware of the soon-experienced confusion. A bewildering and befuddling set of expectations and realisations come and go in rapid succession. From the very first thought – “ok, I’m signed in – what now?”, to the second – “perhaps I need to go upstairs?”, third – “or do I? there’s no obvious signage, just a load of small notices”…. and so on, descending quickly but briefly into self-doubt before emerging victorious from the uneasy, childlike dependency you have on others’ goodwill.

Volunteers in t-shirts, I’m looking at you. Thanks.

The opening evening started this year with the Science Fair, which featured – in my experience – a set of exciting hardware and software projects which were all in some way web-enabled, or web-connected, or web-controlled. Think Internet of Things, but built by enthusiasts, tinkerers and hackers – the way it should be.

“Open Hardware” projects, interactive story-telling, video games and robots being controlled by the orientation of the smartphone (by virtue of its gyroscopic capability).. the demonstration of genius and creativity is not even limited by the hardware available. If it didn’t already exist, it got designed and built.

An Open Web, for Free Society

Seminar on fixing corruption in public services
A multitude of social and policy-driven themes permeated MozFest

As made clear from the opening keynotes on Saturday morning, MozFest is not a place for debate. Don’t think this as a bad thing. The intention is simply to help communicate ideas, as opposed to getting bogged down in the mire of detail. “Free” vs “Open”? Not here. The advice given was to use one’s ears much more than one’s mouth, and it’s sound advice – no pun intended. I have generally been considered a good listener, so I felt at home not having to “prove” anything by making a point. There was no point. 😉

Several themes were introduced in the keynote speeches which really resonated with the attendees – sorry, the participants of MozFest. That of online security and surveillance, more than two years after Edward Snowden’s revelations, was as prominent as ever. Participation was another key theme, and to me one of the most poignant ideas of the whole weekend. Participation was not encouraged or expected; it was simply threaded into the very fabric of one’s presence. You participated, to a lesser of greater degree. This was one of the most socially inclusive experiences I have ever known.

Stories by the Fireside

I cannot overstate how social inclusion at all levels permeated MozFest. From the smallest of teams – 2 individuals, to the largest groups I saw, people were constantly engaged in conversation, development – personal, social and technical, and – perhaps surprisingly – quiet reflection, too.

Image of cupcakes with unusual toppings
Creativity and individuality – there’s a lot of it

Quiet zones were available for those needing a little downtime. The cerebral intensity of the weekend is clearly felt.

The concept of the fire-side story appeared several times, reminding us that the web isn’t just a resource in and of itself, but rather a medium to convey information.  Storytelling, one of the oldest methods of such conveyance, was a prescient theme. Represented through journalism, community and leadership, the scale of recognition (and a reminder) that the web is, primarily, a means to convey stories, took me somewhat aback. It’s inescpable logic, almost lost amidst the omnipresent noise of today’s social media.

Looking to the Future

Not only was MozFest a means to appreciate, understand and build upon the means to share information, it was also firmly invested in its future. Science and education were extremely well represented by group talks, workshops and forums.

Code visualising MozFest pathways on GitHub
Pathways were a means for guiding participants through the plethora of activities.

In fact, the sheer number of topics on offer, and guaranteed clashing of events sure to interest you, simply went to prove one thing: the web is not just big, it’s bigger than you can imagine. How the event planners and coordinators of MozFest actually found a way to combine the multitude of themes and interests into “Spaces” and “Pathways” is a huge credit to the thought-leadership behind this event. By encouraging leadership, the Mozilla Foundaiton has shown itself to be a more-than-capable leader in as diverse a field as there can be.

What I learned at MozFest

On arrival, I didn’t know what to expect. First-timers don’t.  I had a vague incling that I would face a learning curve, adapting to the culture and activities of the event. Like a wandering spirit, I probably stared starry-eyed at the overwhelming number of quickly-scribbled “adverts”, pinned, taped and hung up everywhere, telling me about “this event” or “that workshop”.  Even now, in reflection, I feel that the above post barely scratches the surface of the experience.

It’s sensory-overload, pure and simple. 🙂

MozFest is a journey. Physically, many people made long journeys to attend and participate. To those people, I am grateful – you have made my life richer by your efforts. But psychologically, emotionally and intellectually MozFest is so much more than the sum of its multitudinous parts: It’s an idea, a belief that together we can build something better for much time to come; build something to last that has intrinsic “goodness”. And we are not actually talking about the web. The conversation has evolved. The web might be the medium, but the story is now about us.

The question is, how do we nurture our most sublime nature, and be all we can?