Insights are Perishable

perishable goods please rush

Visualization is a tool that allows us to communicate more information faster, inherently reducing our speed to insight.  Given enough time and the proper tools, we can almost always derive pertinent insights from data or information.  The key value that visualization provides is the alacrity that we move through the analysis process.

This speed is valuable because of the simple fact that all insights are perishable and have a limited shelf life’.  The value of an insight derived from data diminishes the further away from the event you get.  What is the value of an insightful revelation when the time for making the decision has already passed?

Being able to produce information in a consumable format at the exact minute it is needed for a decision sets automated visual analysis apart from other more mundane reporting activities.  Data visualization can help us gain insight faster and allow us to make better decisions more quickly…


In order to consume these insights and make decisions, your process or operations has to support the speed.  It is very possible that your speed to insight will outpace the actionability of those insights.  If you are able to understand an event, but you are unable to take corrective action, the value of the insight is null, regardless of its potential impact.  Many businesses chase after ‘real-time’ dashboards and invest a significant amount of resources into preparing information that can not be used operationally because the process it describes is not flexible enough to change in pace with the insight.

Speed is admirable, but sometimes it’s not necessarily required.

During the process of analysis, one of our main requirements we need to consider is the time.  Not just deadlines to complete a task, but also the time in which an insight will need to be consumed.  If you are conveying a complex idea that needs visualization as a one-time insight, we can spend more effort on the desired result.  More often, in the business world, our analysis will be part of an on-going effort or monitoring process.  In these cases, we have to pay particularly close attention to how expensive new data is in contrast to how valuable the insights gained will be.

Why Visualization?

Visualization at a basic level is simply a means of communicating.  What sets it apart from other methods is the speed at which we can gain insight from the data.  Data alone are useless; it’s not until we translate those data to information that we can use it for decision making.  The faster you can gain those insights and the more robust they are means your decisions are going to be better… well at least more informed.

Vision is our dominant sense.  In terms of information gathering it’s our ‘native language’.  When we see numbers on a page, our brain immediately translates them into information and begins to compare them in order to develop insight.

1    One of the first insights we gain is ‘this is bigger than that’.  With visualization, we can skip the translation and show our brains the information directly.  A big dot and a bigger dot on the page communicate the exact same information that numbers on a page would, but bypass the brain having to speak a ‘second language’.


Take this concept up a level.  Imagine a table with sales figures from several regions and several products.  Depending on the size of the data set, finding the smallest and largest figures would take just a small amount of effort.


If we add color and shape to the table, we can reduce that effort dramatically.  Your eyes will snap to the minimum and maximum values and you will also have a frame of reference for the spread of high to low in an instant, no math involved.  We just reduced the speed to insight to a mere instant.


Now let’s twist this around; take the number out of the picture.


Does this visualization lose any value without the actual data?  Some would argue ‘Yes! I need to know what the number actually is!’  The actual truth is that the requirement was not to know ‘how much’ but simply to identify the highest and lowest.  In addition to meeting the requirement, we also added scope and span to the answer, so we technically delivered more information than requested, without actually displaying the numbers.

If you are still with me at this point, you probably can agree that visualization has value.  Most businesses can accept visualization to this level and the conversations are easy.  Trouble is, we’re only scratching the surface of what we can do with visualization and to really benefit from visualization as a methodology we have to take it a step further.

Visualization can also provide MORE information.  Keeping with our previous example, we’re looking at an aggregation of results for a specific time period.  What if we needed to start looking at trend?  How many data points would we explode if we added years? How about months, weeks or days?  At that point even visualization in a table isn’t going to help; there will just be too many pieces of information to hold in our brains to do meaningful comparisons.

The nice thing about vision is that it can be multidimensional.  Not just in the sense of depth perception but also in the variety of categories we can realize without much effort.  Size, color and proximity are the dominant three that we can utilize easily with data visualization.  Something being bigger than other, a different color than other and close to other convey very specific information when used correctly.

Imagine our prior example and let’s collapse one of our dimensions and replace it with a color.  We can look at all regions referenced by color in a single column now.


The space on the page is now free for adding a third dimension, which would be time.

Immediately we can see trends in the data.  The sales for some products trend up or down by time.  Some regions sell more than others overall.  Some products sell better in one region during specific months.  Some months of the year sell more overall.  Note that the numbers are still absent from the view but do not remotely detract from the information and insights we are gaining from the visualization.


We have now exponentially increased our insight into this data set and we can make exponentially better decisions.

On Statistics


Statistics as a discipline was developed hundreds of years ago by people exponentially more intelligent than I.  The intent of the mathematical models they were developing was to describe an entire population extrapolated from a small sample (in very general terms).

Historically it was utterly impossible to capture, identify and define thousands or millions of data points in one place.  The computing power didn’t exist, everything was manual from the actual information gathering to the transposition of data into usable information.

Today, the value of computed statistics is diminished due to the fact that we actually can gather millions of data points in one place in seconds.  Being able to display thousands or millions of data points on a single screen or page negates the majority of exceptionally complicated statistical tools.


Granted, in the sciences, precision is a commodity that is not easily forgone.  In manufacturing, imperceptible degrees in probability could result in catastrophic failures.  But in business as in most mainstream applications where statistics are employed, I think there is a strong case to be made for ‘good enough’.

However, in today’s world, visualizing a thousand data points together and being able to point to a cluster, average or trend is in most cases sufficient.  The question that needs to be asked is whether or not the decision derived from the information would be significantly different if the accuracy of the data were marginally improved.

If you look at a time series bar chart, is your call center hiring decision going to be different if your visual results are ‘about 1.5 million calls yearly’ versus ‘exactly 1.43 million calls yearly trended over the past 7 years normalized for inflation’?

Looking at a bubble chart, are you going to pull a product from the shelf if the possibility it might be contaminated is ‘about 1 in 5’ , or would your decision be different if you used statistics to compute the probability of defects to be ‘22.345%, plus or minus .005%’?

Our human brains connected to the world through our sense of sight is an amazing computing device.  We can pull from huge oceans of experiences and learned knowledge at a subconscious level faster than we realize.  Imagine the computational power that would be involved in a hitting a golf ball.  The air speed, temperature, humidity, distance, club density, turf consistency, etc. all need to be factored in as well as a plethora of recorded datum about the human swinging the club.  And yet, with practice and training a human can step up, take a few seconds to observe and then whack that ball 300 yards away with a relatively decent chance of hitting the target.

Is it worth spending the time, effort and in some cases money to get that extra degree of proficiency?  Is it ‘good enough’ that we spent a few seconds looking at the data visually and got the answer ‘on the green’?

In most cases, I would argue that ‘good enough’ is going to result in positive data backed decisions most of the time.  I concede the point that sometimes you need the math, you need the statistics, but it should be a question we ask as analysts before we start down a road of esoteric mathematical constructs and complicated statistical models.

Is the correctly visualized data ‘good enough’?


Data & Reporting vs Information & Analysis


Frequently in our environment, people use the terms ‘data’ and ‘information’ interchangeably when in fact they are two vastly different concepts.

Data is a series of numbers on a page, a list of names in a book, a set of figures in a ledger.  This is what we refer to as a report; they can be simple or extraordinarily complex with advanced calculations and complicated code involved.  While they hold their place for certain applications, they are quickly falling out of fashion.

In order for data to have any meaning, they need to be converted into information.  The process of this conversion is analysis.

Providing a customer with a table or a chart that contains pure data does not in itself answer any questions being asked.  What the consumer truly needs is information in order to make an data driven decision.

In order to migrate from reporting to analysis the key component is gathering accurate and robust requirements.  Asking your customer what decisions they are making and where your analysis falls into their process is an absolute necessity.

Imagine an existing report that is generated with daily sales results for an array of products.  This report is useless out of context and without analysis.  Each customer has different requirements for the information they need from this data.  The Sales team needs to see the trend of sales, the Product team needs to see which item is preferred by the consumer, the Logistic team needs to see which products need shipped.  Additionally, you may be able to provide additional data to provide context to the sales figures.  By understanding the intent of the requirement, you may realize that sales without profits could be driving poor decisions.

All of these requirements, when gathered into an analysis can be satisfied by one combined visualization or dashboard saving each person from taking the time and effort to extract their information from this data.

As an analyst I tend to push my customers out of their comfort zone when gathering requirements.  We have to realize that they are asking us to work for them in order to answer questions.  Inherently that implies that I know more about the data than my customer and as such they should tell me what decisions they need to make and not what they expect to see in the analysis.  Don’t tell us what data you want to see, tell us what information you need!

About me…

Profile Pic

From a young age, I was always drawn to beautiful things.  I loved art in all fashions, but more than others, photographs captivated me.  The frozen moments in time that reflect both the natural beauty of the world around as well as the hand of the artist, painting with light.  When I started college, this was the most obvious course for me to take in my studies.

As I learned the craft, I realized a very relevant concept.  Photographers (in stark contrast to most traditional artists) do not actually create beauty.  We are trained to identify, capture and enhance existing beauty, but by the strictest sense of the word, do not actually create.  

This is by no means diminishing the skill and ingenuity involved in the discipline of photography.  The excruciating details of light, contrast and composition are all brushes that we dip in the palette of light to display our canvas of beautiful things.

In addition, photographers are trained in another skill that is less important to traditional artists: relevance.  It’s not sufficient to simply find and capture a beautiful moment, we have to understand the story and make the image relevant.  Why is this image remarkable?  What is it about this specific angle and lighting that makes this unique.  The good photographers learn to not just capture the beautiful things, but tell you the story of why they are beautiful.

After a time, I realized that I tended to prefer a more steady and stable financial future that pure photography simply couldn’t guarantee.  So I shifted my field of study to business and statistics, a subject that seemed to be slightly more lucrative.

While I had resigned myself to the fundamental fact that I had by all intents and purposes ‘sold out’, I began to notice a wonderful trend:  Statisticians and Photographers pretty much have the same skills!

The tools are replaced with equations, hypothesis test and data models, the end result is remarkably similar.  Both professions are looking for that one beautiful thing, whether it be a strong correlation or a reflection of light in a subway window.

Statisticians are trained to recognize that single important and unique element in the data, investigate and quantify it in order to explain,  Just as a photograph without a story is just a pretty picture, a single probability without a story is simply ink on a page.

When the trend of data analysis began to shift towards visualization of data, I knew that I had a true calling.  The simple fact that most data analysts fail to recognize, even when truly practicing their craft is that analysis is a creative discipline.  Being able to reconcile the fact that the role we play is vastly different than most of our peers, (especially in a business environment) was to say the least, liberating.

I have started defining my role as a data artist in certain circles.  Simply knowing your data and being able to answer questions being asked is no longer enough.  We need to be able to create visually captivating stories that convey the information from the data directly to our clients in the most artistic and beautiful way possible.  After all, if people are going to spend time using your analysis to make decisions, a beautiful image will make it so much more resonant.

They say that ‘a picture is worth a thousand words’, and now, more than ever we have the opportunity to take that canon and carve out a market for our unique skills.