Sherri Hammons

a blog about stuff I know

Big Data: Humans Required

Humans RequiredConversation overheard:

Technology:   We have built the perfect technology that will answer any question you ask.

Business:  Excellent.  How can we increase revenue?

Technology:  The question needs to be more detailed.

Business:  How can we increase revenue for Product A?

Technology:  You aren’t getting it.  We need more details around what exactly you are asking.  We can give you a spreadsheet that outlines how many units shipped month over month, and another around costs associated with the units, and we can even give you another spreadsheet that tells you how many complaints we found on social media sites.

Business:  Will that tell me how we can increase revenue for Product A?

Technology:  How the hell should I know?

Can Big Data technology actually create answers to business problems without human touch?

I sat in a meeting a few months ago with some amazing leaders in education.  They had created a board focused on improving education through data insight.  The tech people had been working for months to gather information for the board and on that day had brought in mounds of paper containing information on all the data elements around the state pertaining to education.  In all, there were 15,000 pieces of data with names like “dm_Education_Level”, “sm_Ed_Type”.

I watched these leaders suffer through the paperwork, brows furrowed, eyes glazed.  I finally asked the question that hung like a pink elephant in the room.

“What do you want to know?”

One educator looked haggardly at me.  “Which curricula are working.”

These simple examples outline the heart of the problem with data:  interpretation.  Data by itself is of little value.  It is only when it is interpreted and understood that it begins to become information.  GovTech recently wrote an article outlining why search engines will not likely replace actual people in the near future.  If it were merely a question of pointing technology at the problem, we could all go home and wait for the Answer to Everything.  But, data doesn’t happen that way.  Data is very much like a computer:  it will do just as it’s told.  No more, no less.  A human is required to really understand what data makes sense and what doesn’t.  But, even then, there are many failed projects.

Why is that?

Some fallacies around Big Data include:

Data Scientists will be the answer

If you read the current thinking about the Data Scientist role, you will probably find a living unicorn before you find a Data Scientist.  Skilled in business, language, math, statistics, and computer science, these Super Techno Business Data Geek Scientists could potentially rule the world.  If you can find one.

There needs to be a business case

Guess what?  There is always a business case.  What company doesn’t want to increase revenue, reduce costs, build a better product?  What governmental agency doesn’t want to know if their programs and services are making a difference in citizen’s lives?  What educational system doesn’t want to know they are creating curriculum that gives their students the right toolset for success?

The data must be clean

It won’t be.  Ever.  As soon as you clean it up, it gets messy again.  Just like your stove.

The business is much too complicated and unique

Not true.  Most businesses/government entities are extremely similar.  Think of how we are grouping concepts around tags and keywords online.  We’re building out communities of information that make sense to everyone.  Organically.

If I could go back to the education board, I would turn the conversation around a bit.

Professor:  “What curricula are working?”

Me:  “What are some keywords that would make sense of the term ‘working’ for the curriculum?”

My guess is we would come up with things like:

  • Student Enrollment
  • Student Grades
  • Student Feedback (via social media and other)
  • Success of students post graduation
  • Etc.

Same with the business example.

Business:  “What will increase revenue?”

Data Scientist:  “What keywords influence increased revenue?”

  • Customer Satisfaction (via social media and other)
  • Product Sales
  • Economic Conditions
  • Competition
  • Industry
  • Etc.

If this looks a lot like what you learn in Business School, you are right.  And your business already has a boatload of these people.  A Data Scientist just needs to use these subject matter experts to understand the keywords and consult with them as they find interesting patterns and anomalies.  This is where it gets interesting.

Data Scientist:  Product A sales dipped last December by 20%.

Business:  Yes, it’s because we had that manufacturing problem, remember?  But, they rose back up the next month and have remained steady.

Data Scientist:  Right, but conditions exist similar to those that caused the manufacturing problem.  Might be time to find a new vendor.

Now, that’s the right conversation.

7 comments on “Big Data: Humans Required

  1. Jim
    February 4, 2013

    Sherri – This is a great point; one that I believe can be applied to situations outside of data analysis as well. Conversation, in general, is full of misdirection. Too much content. “Incorrect” questions. Communication is very convoluted, and as such, we are left trying to decipher what the real meat of conversations are, and how to get our own points across without incorrectly (or inefficiently) communicating. This is something I struggle with every day in my efforts to link OIT to our customers, and vice versa. Thanks for bringing this topic to the top.

  2. Tony
    March 26, 2013

    Sherri, thanks so much for the link. Great insight and thank you for the conversation today. I look forward to a future partnership in sharing ideas.

  3. Pingback: Daily Links 03/17/2014 | Anne Z.

  4. Pingback: Four short links: 18 March 2014 - O'Reilly Radar

  5. Pingback: Five Links: How Things Work | HR Examiner

  6. Pingback: Fresh Links Sundae – March 23, 2014 Edition from David Lowe of Disciplined IT and Actionable ITSM | Disciplined IT

  7. Pingback: Proxem » La lettre du 24 mars : les algorithmes ne sont ni magiques, ni sournois, ni bienveillants

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Information

This entry was posted on February 2, 2013 by in Misc and tagged , .

Sherri’s Tweets

Error: Twitter did not respond. Please wait a few minutes and refresh this page.

Categories

Follow

Get every new post delivered to your Inbox.

Join 171 other followers

%d bloggers like this: