New GDP chart added

Just added a new GDP chart, with statistics read in from the Bureau of Economic Analysis. This one involved more brute force than the unemployment data. I wound up manually reading lines from a downloaded CSV response using while loops and substr. There must be an easier way. The code’s up on GitHub.

Creating an Economic Dashboard Using OpenShift PaaS, Node.js, and Google Charts (Part II)

Picking up where I left off in Part I, I have a JSON data service created. Now I want to create a dashboard to display that data.

Creating a Jade Template

To create the page that holds the dashboard data, I decided to try out the Jade template engine for Node.js. I really dig the slimmed down HTML syntax in Jade. All the normal HTML markup built around opening and closing HTML tags like this:

  <li class="first">
    <a href="#">foo</a>
    <a href="#">bar</a>
  <li class="last">
    <a href="#">baz</a>

is shortened to this:

    a(href='#') foo
    a(href='#') bar
    a(href='#') baz

A div with an id of “whatever” is to simply #whatever. Not bad. Plus it’s got all the usual template engine features like variables, conditionals, iteration, and template includes.

Here is the source code for the Jade template for my dashboard page:

            var blsData = !{blsData}
        link(rel='stylesheet', href='css/main.css')
        title U.S. Economic Dashboard
        h1 U.S. Economic Dashboard
            div.questionMark ?
            div.questionMark ?
            div.questionMark ?

Pretty simple. There are a few script imports at the top, including the JavaScript utility library Underscore.js and of course the Google API. The one script block for var blsData shows where I am populating a JavaScript variable with the content of an escaped Jade variable of the same name. I’ll show where this variable is passed to the template in a sec.

The page layout is basic. Four divs, three of which have question marks to represent placeholders for future content. The only div with chart content is unempChart, which will be used by the Google Charts script.

There are a couple changes that need to be made to handle the dashboard page in server.js. First, I need to set up Express to serve my static JavaScript and CSS content. This is done by adding the following two non-route URLs:

// Expose the static resources on two non-route URLs
app.use("/js", express.static(__dirname + '/js'));
app.use("/css", express.static(__dirname + '/css'));

Next I create the route to the dashboard itself, passing in the unemployment data scraped from the BLS site (as described in Part I):

// Route: GET /dashboard -> Jade template
app.get("/dashboard", function(req, res) {
    retrieveUnemployment(function(unemploymentData) {
        res.render("index.jade", {
            blsData : JSON.stringify(unemploymentData)

Google Charts

Lastly, there is the JavaScript that creates the Google Chart out of the JSON data and drops it into the div container.

google.load("visualization", "1", {
    packages : [ "corechart" ]
function drawChart() {
    var data = new google.visualization.DataTable();
    data.addColumn('date', 'Month');
    data.addColumn('number', 'Unemployment');

    // Parsed blsData
    var parsedBLSData = [];
    _.each(blsData, function(blsDataItem) {
        var parsedBLSDataItem = [
                new Date(blsDataItem.year, blsDataItem.month, 1),
    }, this);


    var options = {
        title : 'U.S. Unemployment Rate',
        chartArea : {
            width : '90%',
            height : '75%'
        vAxis : {
            maxValue : 11.0,
            minValue : 0.0
        legend : {
            position : "none"

    var chart = new google.visualization.LineChart(document
    chart.draw(data, options);

I create a google.visualization.DataTable object and populate it with a two-dimensional array, converting the JSON month and year properties into JavaScript date objects as I go. Chart visualization options (of which there are many) are supplied in the options var and a LineChart object is created with the target div id. Calling chart.draw with the converted data and options object displays the chart on the screen.

Partial dashboard running on OpenShift

Next steps

Well, obviously the dashboard needs more charts. One thing it could use is a metric that better controls for the number people who have stopped looking for employment or otherwise dropped out of the labor market. Something like total nonfarm payroll. Then there are other common measures like quarterly GDP changes.

From a technical standpoint, it’s not a great idea to pull the source data with every request for the dashboard. There needs to be a caching mechanism, so the data’s pulled at a more sensible interval (no more than once a day). That might be a good excuse to explore the MongoDB cartridge.

All of the source code is up on Github. Have a look:

Creating an Economic Dashboard Using OpenShift PaaS, Node.js, and Google Charts (Part I)

Hi and welcome to yet another programming blog. This blog is going to be my repository for technical how to’s for my own record and if it helps someone else along the way, then great.

This post discusses creating a dashboard for the U.S. economy using OpenShift and Node.js. Business executives have dashboards on the financial performance of their companies. Why not create a simple dashboard for the U.S. public on the overall financial health of the country?

I figure the dashboard could display four key economic metrics for the country. The choice of metrics is a matter of debate, but I started with the overall unemployment rate from the Bureau of Labor Statistics. I’m still thinking about which other indicators to display. If anyone has ideas on that, let me know.

In terms of technology, I wanted to demo Red Hat’s OpenShift platform-as-a-service (PaaS). I’m a big believer in the PaaS concept and hope it’s how most of us will deploy applications in the coming years. I love the idea of spinning up an entire application stack and deployment environment in one command (which I’ll show below).

Also, Node.js was on my target technology list, but it was more out of curiosity than anything else. I doubt Node.js is the best fit for this kind of application right now, but I was pleasantly surprised by the number and capabilities of Node.js libraries out there already, especially Express, Zombie, and the Jade template engine.


The application is designed to have two kinds of HTTP routes defined. One is for the dashboard itself. The other is a set of data services that supply the JSON data that feeds the dashboard. Not all sources of economic data have that data exposed in a convenient format or API, so creating services for those sources is a potentially useful byproduct of this project. If I ever need to create a different kind of client (mobile, for example), I can reuse those services.

Creating a New OpenShift Application

Creating a new application on OpenShift is easy. If you haven’t already, sign up for an account on OpenShift and get your SSH keys set up. OpenShift offers various platform types (JBoss AS 7, PHP, Ruby, Python, Perl, Node) and includes a way to build your own cartridges to support other platforms. To create this application on a Node.js stack, either use the web management console or install the OpenShift command line tools and type:

rhc app create -a econstats -t nodejs-0.6 -l [OpenShift username]

OpenShift will create and configure a ready-to-go application stack, named econstats, for Node.js 0.6. It will also create a remote Git repository and local clone for your application code. Jump into the just-created local econstats folder and edit server.js as follows…

Create the JSON Data Service

The first step is to create the JSON data service that feeds the dashboard. I use a headless browser library called Zombie.js to scrape the data from the BLS site:

var express = require("express");
var zombie = require("zombie");

var retrieveUnemployment = function(callback) {
    // Screen scrape BLS web page for latest unemployment information
        function(err, browser, status) {
        var unemploymentData = [];

        // Grab the unemployment table
        var ths = browser.querySelectorAll("table.regular-data tbody th");
        for ( var i = 0; i < ths.length; i++) {
            var unemploymentEntry = {};

            // Grab each row header and use it to set the year
            var th = ths.item(i);
            var year = th.innerHTML.trim();

            // Grab each cell in the row and use it to set the month and
            // unemployment rate
            var tds = th.parentNode.getElementsByTagName("td");
            for ( var j = 0; j < tds.length && j < 12; j++) {
                var monthData = tds.item(j).innerHTML.trim();
                if (monthData && monthData !== " ") {
                    unemploymentEntry = {
                        month : j + 1,
                        year : parseFloat(year),
                        rate : parseFloat(monthData)
        console.log("Retrieved unemployment data from BLS.");

var app = express.createServer();

// Route: GET /unemployment -> Unemployment JSON data
app.get("/unemployment", function(req, res) {
    retrieveUnemployment(function(unemploymentData) {

// Get the environment variables we need.
var ipaddr = process.env.OPENSHIFT_INTERNAL_IP || "";
var port = process.env.OPENSHIFT_INTERNAL_PORT || "3000";

// And start the app on that interface (and port).
app.listen(port, ipaddr, function() {
    console.log('%s: Node server started on %s:%d ...', Date(,
            ipaddr, port);

The call to zombie.visit visits the page, adding to the browser var context. browser.querySelectorAll retrieves the table header cells to grab the year. th.parentNode.getElementsByTagName retrieves the cells for the data, which is pushed to a JSON object called unemploymentData.

Routing is handled by Express.js. Express is a lightweight web application library for Node inspired by the excellent Sinatra library for Ruby. It is a non-MVC way to publish simple web applications and APIs. Define a route (URL pattern and HTTP method) and a response, create and start the server, and you’re up and running.

I simply define an Express server and use app.get to create a URL route for the unemployment data (/unemployment). A callback sets the content type and the content on the response. Then I bind the server to the port using a call to app.listen.

A couple OpenShift environment caveats: The IP address and port need to use OpenShift environment variables to work properly in that environment. Also, to make this all work, we need to have Zombie.js installed in the OpenShift environment (Express is already installed). To add Zombie to OpenShift, we edit the project folder’s the dependency list file, deplist.txt, by adding the following line:


Commit and push the changes to OpenShift to run it in the cloud:

git commit -a -m “Adding route to provide unemployment data scraped from BLS site” 
git push origin master

Visit http://econstats-[your OpenShift domain name] to the view raw, pure JSON unemployment data that will serve as the foundation for the dashboard chart.