Lately I’ve been working on Mars.io, a personal project that brings Martian weather analytics, down to Earth (ha!). Puns aside, In the application users can analyze weather data sent by the Curiosity Mars Rover’s REMS (Rover Environmental Monitoring Station). The app does this by fetching the data in JSONp format from the MAAS API, which is then output and graphed on an SVG line chart with the help of the D3js data visualization framework.
There is one component in the application, however, that was worrying me as I was developing it, namely the interactive SVG line chart. This chart plots historical martian temperature data sets. Here’s a screenshot of the component:
The first thing you’ll notice, is that Mars is a very, very, very cold place, and that those people who are fighting to move there are completely insane. The next thing you might notice is the range input slider at the bottom which controls the output of the graph through time.
Here’s how the component works (in English):
The app listens for the OnChange event from the input slider, which triggers a method that takes the current value of the slider as input to concatenate the right URL (with query string parameters) that fetches the JSONP data from the API (the request is done with jQuery’s $.AJAX helper method ). On success a callback function is triggered that processes the data received, stores it in two distinct arrays (one for celsius data, and one for fahrenheit). Milliseconds later, when the request is complete, a callback function is called which triggers the method that plots the graph with D3 in the right temperature unit using the Arrays populated after the successful request.
The problem
The problem is that the data that makes the app useful lives in a server somewhere. This means that the somewhat elaborate process described above could potentially happen over 47 times! The API has 470+ individual weather reports, which are sent in batches of 10. Each time the input slider is moved, it requests a batch of reports from the web server. A user then could want to view all 47 sets, thus making 47 requests.
Furthermore, the codebase (as it was coded initially) introduced redundant requests: if the user slid back in time, then forward, the application would make a request for data it already had just moments before!
This was inefficient and not performant at all (despite the almost negligible 4Kb of data coming through the pipes). As a developer I can’t guess what set of data the user wants to see, nor can I limit the amount of data the user has access to. What I can do, is limit the number of redundant requests.
Solution: Session Storage
With the web session storage API, I could now store all the data being requested in the browser for the length of the session. A browsing session means that the data is stored even if the user reloads the page, or goes to another page within the same domain.
Now if the user wants to move forward from past to more recent data, the method that draws the chart can use the data already stored in the browser, rather than making a redundant API call.
How it works
I won’t go over the web storage API in its entirety, but I’ll explain the basics. Web storage essentially lets you store data in a hash-table-like structure that is implemented locally in your user’s browser. To populate the storage you simply push items to the table with a Key for future retrieval.
The two methods that interest us are setItem() and getItem()
// store some data
window.sessionStorage.setItem("someKey", 33);
// retrieve the data
console.log(window.sessionStorage.getItem("someKey")); // returns 33
Really simple. In my app’s case though I needed to store full JSON objects. In order to store these, it is common practice to store an individual object as a string, and whenever you need to work with them, you parse them back to JSON.
The function getDataSet() below demonstrates the setting of JSON data. This function is called in my app whenever an API call is made for the first time. The function is passed in a data parameter which is the JSON data returned by the API. The function iterates over the object with jQuery’s $.each, then creates two object literals for each temperature, pushes those objects to arrays (thus creating new JSON objects) that will be used to chart the graph the first time. The same objects are also Stringified, and set to session storage in case the user wants to draw the same chart in the future.
function getDataSet (data) {
console.log("JSONP REQUEST COMPLETE");
$.each(data.results, function(index, value) {
var celsiusTemps = {
date: value.terrestrial_date,
min_temp: value.min_temp,
max_temp: value.max_temp
};
var fahrenheitTemps = {
date: value.terrestrial_date,
min_temp: value.min_temp_fahrenheit,
max_temp: value.max_temp_fahrenheit
};
//push objects to the appropriate array for immediate graphing
celsiusTemperatureArchive.push(celsiusTemps);
fahrenheitTemperatureArchive.push(fahrenheitTemps);
});
// store each data set in session storage to avoid multiple API calls in the future
window.sessionStorage.setItem(archivePage + "c", JSON.stringify(celsiusTemperatureArchive));
window.sessionStorage.setItem(archivePage + "f", JSON.stringify(fahrenheitTemperatureArchive));
}
Note that I’m creating the Key of the session storage dynamically by concatenating the “Archive Page” number (which, remember, can range from 1 to 47) with the unit symbol ( I need this in my app so that D3 knows what unit scale to use when graphing the chart ).
In Chrome dev tools you can check whether the session storage worked:
Ignore the latestReport Key. That one is set as soon the app is loaded for another temperature UI module. But you can see the session storage worked! I now have the data cached locally ready to go. If I want to graph the latest 10 data reports 50 times It won’t make 50 requests to a remote server, because we already have the data. But wait? Will it? Not yet. I haven’t written the code for the app to decide what data to use under which circumstances.
What I need to do is tell the app to make a request If and only if the data is not stored already. In my application, I do this by using the value of the “archive page” generated dynamically by the value of the input slider, and use it to check if when used as a key, a value is returned. If null is returned, it means that particular data set has not been stored and thus we execute the AJAX call. Else, we use the stored data to graph it.
// if the data is not in the session storage, fetch it
if(window.sessionStorage.getItem(archivePageKey) == null) {
console.log("loading archive " + archivePage + " from API..." );
/* GET JSONP FROM API
============================*/
$.ajax({
url: archiveUrl,
// The name of the callback parameter, as specified by the YQL service
jsonp: "callback",
// Tell jQuery we're expecting JSONP
dataType: "jsonp",
// Work with the response
success: getDataSet,
// chart the response once we obtain the data
complete: function () {
drawChart(unitToChart); // draw chart in the right unit
// show chart label and input control
$('#graph-ui').show();
}
}); // end AJAX
} else {
console.log("using stored data...");
drawChart(unitToChart, true, archivePageKey);
// show chart label and input control
$('#graph-ui').show();
}
You don’t need to concern yourself with the name and parameters of my functions. All I want you to take away from this is how to use session storage API to conditionally load the right data.
We will take a closer look at the a section of the drawChart() method however, as it contains some crucial conditional logic that takes care of initializing the right data set for D3 to draw, which is stored in the variable appropriately named data.
The method takes 3 parameters. The first one is obligatory: a string “f” or “c” that tells the method what kind of data set to use, either celsius or fahrenheit. Then a boolean value is passed. If true, it tells the method to used the data we cached in session storage, and the third value is the session storage key that is dynamically generated on each input slider event so that the data set can be retrieved.
Remember that in order to use the stored data, it needs to be parsed back to JSON using JSON.parse(), otherwise we’re working with a string that has no computational value… and things will break.
function drawChart(tempUnit, loadCached, archiveKey) {
// clean the node before appending SVG
document.getElementById("temp-graph").innerHTML = "";
// Choose the right data set
if(loadCached)
{
var data = JSON.parse(sessionStorage.getItem(archiveKey));
}
else
{
// choose data set according to temperature Unit passed
switch( tempUnit) {
case "f":
var data = fahrenheitTemperatureArchive;
break;
case "c":
var data = celsiusTemperatureArchive;
}
}
/*---- SVG D3 DRAWING CODE GOES BELOW -----------*/
Once one of the 4 possible data sets is determined, D3 does its thing rather wonderfully. There you have it! An interactive chart that uses data more efficiently by reducing the number of potential HTTP requests!