Where is My Big Data Coming From and Who Can Handle It

Recently, a reader asked my insights on the article (Data Scientists are the New Rock Stars as Big Data Demands Big Talent).  Here is my response.

It seems like in today’s world people and organizations are somewhat struggling with this big data concept and do not know where to begin. Due to this reason, they are collecting everything they can think of in the hopes that one day they will be able to use this data in a meaningful way such as better customer experience, new products/services, better collaboration, increasing revenue, etc. This hope approach of “let’s collect data and later decide what we can use it for” on the surface might seem sound but last I checked hope is not a strategy. Perhaps this is one of the reasons that even now only <1% of the data collected is actually being analyzed. What good is more data when one cannot even make sense of the other 99%+ of data it already has? Are we chasing a ghost?

While it is true that vast amounts of data are and will be generated from financial transactions, medical records, mobile phones, and social media to the Internet of Things but there are questions that need to be asked to understand data’s meaningful use:

  1. How will data be managed?
  2. How will data be shared?

I believe that in order to come to a point where data becomes meaningful and useful it would require (broadly speaking) three phases:

  1. Establishment of standards, governance, guidelines. (E.g., open architectures)
  2. Creation of industry specific data exchanges. (E.g., healthcare data exchanges, environment data exchanges, etc.)
  3. Creation of cross-industry data exchanges. (E.g., healthcare data exchanges seamlessly interacting with environmental data exchanges, etc.)

Additionally, let’s keep this in mind that the data we are talking about is data that can be captured by current tools and systems but the data which is perhaps the most difficult to capture is unstructured human data which within organizations is called Institutional Knowledge. This does not reside in a document or a system but in the minds of the people of an organization who understand what needs to be done in order to move things forward.

So, the question becomes, do we really need Data Scientists who have a mix of coding skills with PhDs in scientific disciplines and business sense or do we need someone who is able to connect the dots and have the ability to create the future. The answer is not a simple one. Perhaps you need both. The ability to code should not be the deciding factor but rather the ability to leverage technology and data should be. I agree that there is a shortage of people with diverse talent but there is also a shortage of people who actually know how to leverage this kind of talent.

Before organizations go on a hiring spree they should consider:

  1. Why do they need a Data Scientist? (E.g., have strategic intent, jumping on the bandwagon, etc.)
  2. Who will the Data Scientist report to? (E.g., Board, CEO, CFO, COO, CIO, etc.)
  3. Does the organization have the ability to enhance/change its business model? (E.g., making customers happy, leading employees, etc.)
  4. Is the Data Scientist really an IT person with advanced skills or does s/he have advanced skills and happens to know how to leverage technology and data?
  5. How often will you measure the relevancy of the data? (E.g., key data indicators)
3 Phases of Big Data Harmonization
3 Phases of Big Data Harmonization

Chief Executive Officer (CEO) and Information Technology (IT) with sprinkles of Enterprise Architecture (EA)

A few weeks back I posted an article (Why IT Should Be on the CEO’s Agenda) on the Massachusetts Institute of Technology (MIT) LinkedIn group about why Information Technology (IT) should matter to the CEO. A reader commented and referred me to his blog post. After reviewing his post, I have the following responses:

  • I would argue the perception that IT is too complex and decision-makers need to have a deep understanding of IT in order to leverage it. For most organizational decision-makers, simply recognizing that IT can be leveraged for competitive advantage can be sufficient to have a leg up over the competition. Think about it, although the CEO might have a high-level understanding of Finance, Accounting, Operations, and Sales but does s/he needs to be an expert in all of them? Absolutely not and similarly CEO does not need to have a deep understanding of IT although the better understanding s/he has, the better-equipped s/he will be to face the challenges of the future.
  • On the point that IT has “complex processes and structures” is a blanket statement and would not apply to each and every organization. I would say that it all depends upon the organization and careful review should be done to understand the reasons for the existence of these processes and structures. This review can help in improving the organization and create an appreciation for all sides.
  • In terms of Enterprise Architecture (EA),
    • it has many flavors to it but it almost always starts with strategy/analysis and should result in execution/operations. While EA cannot predict each and every scenario that can happen but by involving the people who are doing the day-to-day operations, EA is able to create concrete solutions that work in the real world and is not merely theory.
    • one of the biggest mistakes organizations make is that they think EA is only an IT-thing, only about artifact development, only about future planning and only about software application development. While all of these are noble pursuits, EA has a much broader view of the world that goes beyond the IT world. A well-run EA practice will consistently produce qualitative (e.g., management best practices, better communications, etc.) and quantitative (e.g., increased productivity/sales, cost savings, etc.) benefits for both IT and business. So, EA sits in between IT and business and whenever you limit it to an IT-thing then it defeats the overarching purpose of EA.
    • organizations already “do” EA, no matter what they call it, how broken it is and no matter if they use custom or industry frameworks to capture the information. Each framework has its pros and cons but organizations simply cannot put the blame on EA when the business itself is not aware of how it can leverage EA across the organization.
    • since EA is the highest level of abstraction, it looks at the business and IT sides holistically and is used to drive various objectives such as organization change, business intelligence, and portfolio management to name a few. It is up to the organization collectively to understand this and then help themselves to continuously improve organizational assets such as people, processes and technologies.

I hope the above response helps shed some light on the different things that organizations need to consider. I would leave you with some questions to think about?

  • Does the organization really know what it wants to be when it grows up?
  • Does the organization really know who it wants as friends?
  • Does the organization really know what house it wants to build?

To the Cloud or Not to the Cloud

It seems like these days most organizations are interested in jumping onto the Cloud Computing bandwagon in one way or another. While there are many reasons why organizations want to move to the Cloud, I believe that optimization of business and technology processes should strongly be considered Pre-Cloud adoption. Additionally, organizations need to develop strong Key Performance Indicators (KPIs) and Service Level Agreements (SLAs) to measure against the performance of a Cloud vendor and take into consideration the consequences if the KPIs and SLAs are not met. Thus, the thought of improving your organization and inspiration from William Shakespeare’s Hamlet led me to write the following:

To the Cloud or not to the Cloud, that is the question:
Whether ‘tis nobler in the mind to suffer at the hands of IT
The processes and systems of extreme complexity
Or to take the decision to outsource against a sea of issues
And by opposing end them: to completely, to partially
No more; and by partially, to say we end
The headache, and the thousand business challenges
That implementation is heir to? ‘tis a consummation
Devoutly to be wished. To completely to partially,
To partially, perchance to dream; aye, there’s the rub,
For completely what new issues may arise
When the organization has shuffled off this essential support,
Must give us pause. There’s the respect
That makes calamity of a vendor’s contract;
For who would bear the disruptions and problems of time,
Is the management wrong, the proud man’s contumely,
The pangs of despised mind, the compliance delay,
The insolence of office and the rejection
That patient merit of the unworthy takes,
When he himself might his demise make
With outdated processes? Who would governance bear,
To complain and sweat under sub-standard operations,
But that the dread of something after completely,
The undiscovered lessons learned, from whose goal
No professional return, puzzles the will,
And makes us rather bear those problems we have,
Than ask to other that we know not of.
The conscience does make ignorant of us all,
And thus the native hue of resolution
Is sicklied o’er, with the pale cast of thought,
And enterprises of great pitch and moment,
With this regard their thoughts turn awry,
And lose the name of action. Soft you now,
The fair (insert company name here), in thy orisons
Be all my decisions remembered.

Cloud Adoption
Cloud Adoption

 

References:

  1. 5 Factors for Business Transformation
  2. 5 Questions to Ask About Your Business Processes