Yellowbrick of the past

A couple of years ago Yellobrick started to track the races I participated at. At this time, they had issues because too many people were connecting to their services.

I decided to take a look at their implementation, among all the scripts and contents that are loaded in your browser was a huge XML file containing all the positions of all the boats, this file could reach a size of more than 10MB, no wonder why they had load issues...

At this time, we wanted to know our competitors positions, so in order to do that we could either:
- Stay close to shore and hope to get cell phone data
- Pay $5000 get a top-notch satellite system
- Pay $75 for a very basic satellite phone plan and work a little bit

We decided to pay $75 and work a little bit, The idea was that we made a server that was responding to our requests and get only the information we needed. For that we had to understand that XML format they were using.

XML data could be fetched by calling the Yellowbrick API this way :http://gae.yb.tl/Flash/nb2012/TeamSetup/?rnd=99912 or that way http://gae.yb.tl/Flash/nb2012/LatestPositions/?rnd=31073
They had many files there: Team setup, race setup, latest positions, all positions, weather...

After extracting the useful information from that XML data we could either render the information in a webpage or send it by email.
Using that system we save ourselves downloading megabytes of useless data, thus saving tremendous amounts of time and money.

I also made a little chart plotter in order to follow every boats around us:

This was pretty good, but later in 2014 yellow brick moved to a new binary format. This was a very smart move as their new way of exchanging data is very efficient.

The race definition was stored in a JSON formatted while the positions and team information was in binary. They created a specific binary format that was hopefully easy to understand and parse. From there it was then easy to send again only the fleet information we needed and save us from downloading the entire binary file.

I would have to say that Yellobrick did a nice job in fixing their issues fast.
We had a clear advantage compare to everybody else by retrieving only chunks of this public data instead of the whole thing which saved us a lot of time over our cheap satellite connection.

Now Yellowbrick made another smart move, they published an API and provide low bandwidth text files that contains the data you need.

All of the work described above is of limited use now as they provide these new ways of accessing the data, but luckily we have other homemade tools this year :)

Weather retrieval

Weather retrieval is very important in order to plan our races. Of course the difference between the forecast and the actual conditions usually varies, but I think that this year I finally found the tool we needed: A statistical model of weather forecasts.

Most of the commercial tools are limited, for example software such as OpenCPN, Expedition and others are able to use the "standard" models such as GFS and more localized ones such as NAM. However they do not provide yet the option to fetch Ensemble models.

These models are using stochastic methods to estimate input variables based on observations and statistical distribution of their values. For example instead of using a single point wind speed let's say 15kn they use a probability distribution function to recalculate the model with different values for the same parameter.

By running all these simulations we can obtain:
- No a single forecast, but many based on the input assumptions of the model
- Average and mode values that are accounting for the variability in the measurements

So instead of using the GFS or NAM models that are going to tell me what the weather will be at a certain point we'll use the stochastic model that can give us variations of the weather at a certain point, this offers a more realistic view of what happen in the real world.

As most commercial tools cannot fetch this data, I had to develop my own service so that I'll be able to get these forecast before and during navigation. When I do not have what I need, there is always a solution :)

In order to do that it's quite simple, I have a command and control server running at home that I can query to retrieve the weather for a specific location and time frame. Then it sends me back only the data I need in order to save the amount of data transferred.

I've started doing that two years ago, once you start it's very easy to expand it to include other models!

If you're interested in weather routing, forecasts models and how to retrieve them do not hesitate to contact me.

Automatic tacks/jibes performance analysis

I've recently finalized my R scripts to analyse all our racing data. Unfortunately I can only find the data from 2013, but that already quite a few races!

As an example I'll use some newer data that we got during a recent practice (the same I used to make the videos overlays)

From these NMEA records, I compute the rate of turn of the boat, the standard deviation over the heading for the last minute. Using these two measurements along with the speed over water and apparent wind angle I can detect pretty accurately tacks and jibes.

Using the standard deviation and rate of turn I can see when we started our tack.

Using the speed over water, I assume the tack being done/completed when we reach 90% of the speed we had before, and therefore recover our speed. I use 90% because it's a threshold we reach most of the time, if I used 100% with the wind fluctuations I cannot observe these 'stops' all the time.

The plot above represents from top to bottom: Speed over water, Heading and Apparent wind angle

The vertical bars represents:

• BLUE : Tack or Jibe detected
• RED : Beginning of the maneuver
• GREEN : End of the maneuver

Now the interesting part is to calculate how much distance we lost performing that maneuver.

We can do that by integrating over time the speed loss (which is equivalent to calculating the area of the speed dips on the top plot)

When doing that I can calculate how much distance we lost over the competition due to our maneuvers. This is a very important metric, we can try to optimize our tacks and reduce that value!

What I'm interested to see is the distribution of these values. When running the same algorithm for the whole 2013 NMEA data I recorded this is what we find:

This graph shows that most of our tacks are making us loose between 1-3 boat lengths over the distance we would have done without performing the maneuver. This is very good !

Now you can also see that some of our tacks are really bad... There are some that are performed slowly and makes us loose 6 boat lengths or more... This has to be fixed!

What needs to be fixed is the variability of how we execute these. We should be consistent, perform it with the same team, in an almost exactly same way in order to reduce the tail of the chart above.

Tacking in 2-3 boat lengths is a well known rule of thumb, but we need to work on never get slower than that, because you only need one of these bad tacks to ruin your race!

It's all up and running

Today I performed a couple of tests on the system and I can finally conclude that it all works as intended.

Our single board computer now takes care of multiplexing the data from the GPS and our instruments and broadcast that over a secured WiFi.

When connected to the network and listening to a specific port using a TCP socket I can receive all the information necessary. This allows us to wireless connect tablets, laptops, smartphones.

The application I made for my phone correctly shows me our targets as well as absolute and relative differences. The laptop will be helpful for navigation on long distance races (I kept the RS-232 connection though in order to have backup if the WiFi fails, that makes the two connection redundant)

The hot-spot also serves as a "black-box" for recording all the data. Recent python scripts where added to act as TCP client and save all data to a USB flash drive connected to the box. I plan on using that later for data analysis and performance evaluation, as well as for the videos that were described a couple of days ago.

From conception

To realization

The next updates will be done on the Android application. The plan is to create an algorithm performing sensor fusion in order to calculate a precise distance to the starting line, and estimate precisely if we're early or late to the start.

I'll also restart to work on the application used on-bord for the MAC, make sure everything works correctly and enhance the tracker to allow finding boat and plotting their trajectory in a nicer way.

Using Boat Instruments and Video to Enhance Sailing Practice Sessions

I recently found a way to incorporate our boat data to videos we shoot. The basic principles are:

- Collect all NEMA messages
- Take a video
- Synchronize the two and create nice gauges

This process is pretty much straight forward, however I faced a couple of issues in the process:

- The synchronization is not always evident, I used our tacks and distance between the tacks in order to find how to synchronize the video.
A better approach would be to film a watch and thus "time-stamp" the video so I can use the NMEA GPS timestamp to then find a sync point.

- The NMEA data understood by most software are GPS ONLY, this is not what I want. I want to have our wind instruments measurements, I want to have our speed over water values and I want to have our upwind and downwind optimal VMG targets.

The first piece of software I created was to take as an input NMEA data and create a CSV file. NMEA information comes at irregular interval, sometimes data is missing and needs to be linearly interpolated.

Now I wanted to have the optimal targets on it, so I have to include them using the velocity prediction calculated for our boats:

Once you have that the remaining action is to synchronize the data with the video, and create overlays

At the end you have a very sweet video that you can use to review your practices, see when you're fast, when you're slow, and what can be improved. Et Voila!

And the youtube video:

Using Data to 'win' Sailboat races

I've been sailing for quite a few years on the great lakes, doing various buoys and port to port regattas. I decided last year to use the skills I developed at work and bring that to the boat to have an edge on others, so in early 2014, I started developing a system that allowed us to have a clear edge on the competition by using data publicly available, some creative ways of gathering the data and the good tools to analyse it.

We did out best ever Mackinac race that year, and I'm pretty sure it'll be hard to reproduce.

Preparation

I started gathering and observing weather data for that race 7 days ahead, collecting the data ahead of time allows you to see how the foretasted results evolves. If you see that for 7 days in a row, the wind speed and direction are the same for the day of your race, you can assume that the model is pretty stable and results are solid. If you see huge variations from day to day, it's likely that big perturbations are in the area, and the models have a hard time predicting what will happen.

3 Days before the race, I knew what the strategy will be : I wanted to be the leftmost boat of the fleet. I sent a couple of emails to everybody everyday refining our route and strategy but the overall plan did not change. Why? Because fronts were coming from the west and I wanted our boat to be the first catching it while everyone else will be stuck in high pressure system on the east side.

That's a very short story to let you know how I prepared for the race, but what about the other systems I told about in the first chapter? Read next.

Capabilities of our boat

 Our boat was able to:

• Gather weather forecast from various models as soon as they were published by the NOAA
• Selectively get a portion of our area of interest to reduce the time it takes to download things
• Sends Tweets (text and images)
• Post on Facebook (configured by following our twitter account)
• Re-estimate the best route with every weather forecast we had
• Know the position of all the other boats at any time

To do that I created a system made of two parts: A server that was running at home waiting for my commands, an POP/SMTP client used on the boat to send receive messages.

The mode of operation was pretty simple, if I wanted to know the weather forecast, I was sending an email to my server with a specific title like when my server receives the request, it downloads all the necessary files from the NOAA servers, stack them hour per hour, and truncate them to only use the desired area. Thus files sent back to me where around 44kB only and downloadable in a read.

There are systems commercially available for that but the only provide you with GFS models, ours could grab any model, thus we had a better overview of the weather ahead of us.

We could also send tweets like this one :

For that I needed to take a picture, resize it to a very small image to weight no more than 20kB and sent another type of email to my server. Then the server took care of publishing out tweet.

For the positions of the other boats it was also pretty straight forward. But required some hacks, our positions are relayed to the public using the website Yellowbick. They have a pretty nice webpage and JavaScript client that displays boats on a map. 

However we didn't wanted to download the whole page all the time with our cheap satellite connection, this was just impossible. So the server was doing it for us, send an email every time new positions are available, and I used my custom made software to display course, speed, heading... of all our competitors, this was our little spy:

We're also using the forecast and a software that performs optimal routing using dynamic programming. The inputs it needs are: The predicted performance of your boat, the weather forecast and a GPS point to go to. Then what it does it try for many different increment of headings to find the route that takes you the closer to your destination in the shortest amount of time. And provides you with very good information about where to be on the race course:

All of that was running during our race, and it allowed us to finish in second place. Now what second? We didn't win? No because we were a bit too slow at the beginning, but we'll fix that this year with some other data and visualization tools.

Conclusion

We'll use all these tools again next year, we'll improve on our satellite communication to have a more reliable connection, and we'll also train hard to stay on target all the time. The WiFi transponder and App described a year ago will be very useful!