Yellowbrick of the past

A couple of years ago Yellobrick started to track the races I participated at. At this time, they had issues because too many people were connecting to their services.

I decided to take a look at their implementation, among all the scripts and contents that are loaded in your browser was a huge XML file containing all the positions of all the boats, this file could reach a size of more than 10MB, no wonder why they had load issues...

At this time, we wanted to know our competitors positions, so in order to do that we could either:
- Stay close to shore and hope to get cell phone data
- Pay $5000 get a top-notch satellite system
- Pay $75 for a very basic satellite phone plan and work a little bit

We decided to pay $75 and work a little bit, The idea was that we made a server that was responding to our requests and get only the information we needed. For that we had to understand that XML format they were using.

XML data could be fetched by calling the Yellowbrick API this way :http://gae.yb.tl/Flash/nb2012/TeamSetup/?rnd=99912 or that way http://gae.yb.tl/Flash/nb2012/LatestPositions/?rnd=31073
They had many files there: Team setup, race setup, latest positions, all positions, weather...

After extracting the useful information from that XML data we could either render the information in a webpage or send it by email.
Using that system we save ourselves downloading megabytes of useless data, thus saving tremendous amounts of time and money.

I also made a little chart plotter in order to follow every boats around us:

This was pretty good, but later in 2014 yellow brick moved to a new binary format. This was a very smart move as their new way of exchanging data is very efficient.

The race definition was stored in a JSON formatted while the positions and team information was in binary. They created a specific binary format that was hopefully easy to understand and parse. From there it was then easy to send again only the fleet information we needed and save us from downloading the entire binary file.

I would have to say that Yellobrick did a nice job in fixing their issues fast.
We had a clear advantage compare to everybody else by retrieving only chunks of this public data instead of the whole thing which saved us a lot of time over our cheap satellite connection.

Now Yellowbrick made another smart move, they published an API and provide low bandwidth text files that contains the data you need.

All of the work described above is of limited use now as they provide these new ways of accessing the data, but luckily we have other homemade tools this year :)

Weather retrieval

Weather retrieval is very important in order to plan our races. Of course the difference between the forecast and the actual conditions usually varies, but I think that this year I finally found the tool we needed: A statistical model of weather forecasts.

Most of the commercial tools are limited, for example software such as OpenCPN, Expedition and others are able to use the "standard" models such as GFS and more localized ones such as NAM. However they do not provide yet the option to fetch Ensemble models.

These models are using stochastic methods to estimate input variables based on observations and statistical distribution of their values. For example instead of using a single point wind speed let's say 15kn they use a probability distribution function to recalculate the model with different values for the same parameter.

By running all these simulations we can obtain:
- No a single forecast, but many based on the input assumptions of the model
- Average and mode values that are accounting for the variability in the measurements

So instead of using the GFS or NAM models that are going to tell me what the weather will be at a certain point we'll use the stochastic model that can give us variations of the weather at a certain point, this offers a more realistic view of what happen in the real world.

As most commercial tools cannot fetch this data, I had to develop my own service so that I'll be able to get these forecast before and during navigation. When I do not have what I need, there is always a solution :)

In order to do that it's quite simple, I have a command and control server running at home that I can query to retrieve the weather for a specific location and time frame. Then it sends me back only the data I need in order to save the amount of data transferred.

I've started doing that two years ago, once you start it's very easy to expand it to include other models!

If you're interested in weather routing, forecasts models and how to retrieve them do not hesitate to contact me.