How Predikto hired thousands of data scientists

innovation_leadershipGE’s Jeff Immelt was recently interviewed on the Predictive Analytics investments and overall initiatives that have been ongoing over the last 5 years within his walls. The transcript, available here, is an excellent read on how a legacy company is attempting to transform itself for the digital future, leveraging vast amounts of sensor data to predict failure in large machinery. This marks a pivotal moment in GE’s history, where turning around a Titanic-size ship won’t be a trivial matter. The build-out began 5 years ago with a massive scaling of data science and predictive analytics division

Immelt seemed to drive one point home more than others in this interview; the mass hiring of Data Scientists (and ancillary staff) to accomplish the goal of building out the Predix division.

We have probably hired, since we started this, a couple thousand data scientists and people like that. That’s going to continue to grow and multiply. What we’ve found is we’ve got to hire new product managers, different kinds of commercial people. It’s going to be in the thousands.

We also hired thousands of Data Scientists (although we didn’t hire any “people like that”), so I figured I would shed some light on why and how we accomplished this.

The Need for Data Scientists

Data Scientists are the corner-stone of the machine learning world. Generally speaking, data scientists come from varied backgrounds; mechanical engineers, electrical engineers, and statisticians, to name a few. Their function within a predictive analytics organization is to (putting it simply) make sense of the data and select the features that influence the predictive models. Feature Selection goes hand-in-hand with making sense of the data, in that the Data Scientist is analyzing large amounts of data often with sophisticated software designed to choose which sensor readings, external factors, and derivations / combinations of each truly impact whether some *thing* will fail or not. Data scientists are the tip of the spear in determining what features/reading/factors matter and what predictive/mathematical models should be trained and applied to forecast events and probability of failures.

We faced the same crossroad as GE; data scientists are essential in getting things right and you need a lot of them when analyzing machine data. We aren’t talking about a few terabytes of data here. No, you’re typically looking at hundreds of terabytes generated by a system in a month… every month… for years.

Scaling the Data Science Team

Big data beckons a big data science team, and to that end, we had to employ, as GE does, thousands of data scientists.

Unlike GE, our data scientists don’t have names or desks. They don’t require ancillary staff nor coffee to stay awake.

Our data scientists work 24 hours a day, 7 days a week, 365 days a year and never tire or complain. Larger dataset? Our data science team clones itself to meet the demand elastically.

Predikto has a unique approach to machine learning and data science. Our data scientists are tiny workers operating on multi-core computers in a distributed environment, acting as one. Just like machines automated many of the mundane human tasks during the industrial revolution, Predikto has automated machine learning and the mundane tasks once accomplished by humans. Our feature selection? Automated. Feature scoring? Automated. Training models? Automated.

I invite you to read the Immelt interview. It truly is a good read on one way to approach building a predictive analytics company. At Predikto, we chose a different path that we felt was innovative and scalable for our own growth plan.

Also a good read… Innovation Happens Elsewhere (http://dreamsongs.com/IHE/IHE-24.html#pgfId-955288)

5 Reasons Why You Suck at Predictive Analytics

reasons why you may suck at Predictive Analytics

We see it every day… Customer X embarked on a grand in-house initiative to implement Big Data analytics. $X million and Y years later, they can predict nothing, aside from the fact that their consultants seem to get fatter with every passing month, and the engineering guys keep asking for more time and bigger budgets. It’s understandable… These days, cutting through all the noise related to big data analytics can be difficult. The bloggers, analysts, and consultants certainly make it sound easy, yet the road to successful predictive analytics implementations seems to be littered with the corpses of many a well-intentioned executive.

Many of our customers come to us, after they have spun their wheels in the mud trying to implement big data projects on their own or with the help of the talking heads. Below is a list of what I believe to be the top reasons for failed projects with buzzwords omitted:

  1. Data Science and Engineering alignment Fail: Or… The fear that engineering will cannibalize data science. After all, “If I can automate Data Science, why do I need the Data Scientists?”, the reasoning goes. Aligning both camps is difficult in larger organizations, as turf-wars will erupt.  Analytics software should seek to include the data science day-to-day activities rather than exclude them.
  2. Your data sucks: Nothing can save you here. If your vendor/manufacturer is providing shoddy data, you won’t be able to predict or analyze anything. Any consultant that tells you otherwise, is selling fertilizer, not analytics. It is best to reach out to your data generator / vendor and work with them to fix the root of the problem.
  3. You hired IBM: Watson does well with Jeopardy questions, but sadly couldn’t even predict the most recent round of IBM layoffs.
  4. You build when you should buy: Predictive Analytics is really hard, and chances are that it’s not your core competency, so why are you bringing all of this in-house? The real short term costs of implementing and maintaining custom software, data science groups, engineering groups, and infrastructure costs, can easily eat away millions of dollars, and you’re placing really big bets on being able to hire the high-level talent to pull it off.
  5. Operations Misalignment: Predictions are useless unless there is a someone or a some-thing to act on the results. It’s important to make operations a partner in the initiative from the onset. Increasing operational efficiency is the goal here, so really… Operations is the customer. A tight feedback loop with ongoing implementation between both camps is a must.

And so that’s the gist of it – 5 bullets forced out of me at our marketing department’s insistence. As much as I enjoy mocking the hype-masters in the industry, these days I find myself extremely busy helping build a real startup, solving real-world problems, for the Fortune 500, for real dollars. 😉

A Software Industry Veteran’s Take on Predictive Analytics

I’m about 4 months into the job here at Predikto as VP, Sales.  The predictive analytics market is an exciting new market with predictably (pun intended) its share of hype.  Nevertheless, this is key niche of the Industrial Internet of Things sector. I’d like to share some observations on what I’ve learned thus far.

We focus on asset-intensive industries, helping organizations leverage the terabytes of data they have accumulated to anticipate the likelihood of an adverse event, whether that is a battery on a transit bus about to fail, or indications that a fuel injector on a locomotive diesel engine, while still operating, is doing so at a less than desired level of performance.   We predict these events in a time horizon that allows the customer to take action to rectify the issue before it creates a problem, in a way that minimizes disruptions to operations.  Our technology is cutting edge Open Source, leveraging Spark, Python and Elastic Search hosted by AWS.

The use cases we’re being asked to solve are fascinating and diverse.   Some companies are contacting us as part of an initiative to transform their business model from selling capital assets to selling a service, an approach popularized by Rolls Royce with their jet engines, the “power by the hour” approach and similar to the software industry’s transition from selling perpetual licenses with maintenance contracts, to selling Software as a Service (SaaS).  In order to sell capital assets like construction equipment and industrial printing equipment this way, our customers will offer service level agreements, with Predikto in place to allow them to proactively deal with issues likely to degrade their service commitment.  So while our tactical focus has been on helping clients maximize product “uptime”, the strategic driver is helping them transition to a new way of generating revenue while getting closer to the customers.  It’s been gratifying to realize the impactful role our offering is playing in facilitating these transitions.

Other organizations are complex, asset-intensive businesses, where an equipment failure can have a cascading effect on revenues and customer service.  For example in the work we are doing with railroads we’ve learned there are a multitude of areas where sub-optimal performance of equipment or outright failure, can have significant impact.  The North American railroad network in 2014 set new records for revenue-ton-miles, a key efficiency metric; this was accomplished over a rail network which is highly congested.   In this environment, a delay has huge ripple effects.  Any number of factors can lead to a delay, ranging from a rockslide blocking a section of track to a locomotive breaking down, to a wheel failure on a rail car, which can cause a derailment.   On top of this, in order to operate safely and comply with government regulations, railroads have invested heavily in signaling and equipment monitoring assets, as well as machinery to maintain the track and roadbeds, which must work reliably.  Our abilities to implement in weeks and generate actionable predictions regarding locomotive and rail car health, as well as monitoring other equipment and even the condition of the rails, are making a major difference in helping to facilitate efficient, safe rail operations.

 

Having a blast…more to come.

Kevin Baesler, VP of Sales

Deploying Predictive Analytics (PdA) as an Operational Improvement Solution: A few things to consider

“…in data science…many decisions must be made and there’s a lot of room to be wrong…”

There are a good number of software companies out there who claim to have developed tools that can potentially deploy a PdA solution to enhance operational performance. Some of these packages appear to be okay, some claim that they are really good, and others seem really ambiguous other than being a tool that a data scientist might use to slice and dice data. What’s missing from most that claim they are more than an over glorified calculator are actual use cases that can demonstrate value. Without calling out any names, the one thing that these offerings share in common is the fact that they require services (i.e., consulting) on top of the software itself, which is a hidden cost, before they are operational. There is nothing inherently unique about any of these packages; all of the features they tout can be carried out via open-source software and some programming prowess, but here lies the challenge. Some so-called solutions bank on training potential users (i.e., servicing) for the long-term. These packages differ in their look-and-feel and their operation/programming language and most seem to either require consulting, servicing, or a data science team. In each of these cases, a data scientist must choose a platform/s, learn its language and/or interface, and then become an expert in the data at hand in order to be successful. In the real world, the problem lies in the fact that data tends to differ for each use case (oftentimes dramatically) and even after data sources have been ingested and modified so they are amenable predictive analytics, many decisions must be made and there’s a lot of room to be wrong and even more room to overlook.

“…a tall order for a human.”

Unfortunately, data scientists, by nature, are subjective (at least in the short term) and slow when good data science must be objectively contextual and quick to deploy since there are so many different ways to develop a solution. A good solution must be dynamic when there may be thousands of options. A good product will be objective, context driven, and be able to capitalize on new information stemming from a rapidly changing environment. This is a tall order for a human. In fairness, data science is manual and tough (there’s a tremendous amount grunt work involved) and in a world of many “not wrong” paths, the optimal solution may not be quickly obtained, if at all. That said, a data science team might not be an ideal end-to-end solution when the goal is for a long-term auto-dynamic solution that is adaptive and can to be deployed in an live environment rapidly and that can scale quickly across different use cases.typical solution

“…a good solution must be dynamic…”

End-to-end PdA platforms are available (Data Ingestion -> Data Cleansing/Modification -> Prediction -> Customer Interfacing). Predikto is one such platform where the difference is auto-dynamic scaleability that relieves much of the burden from a data science team. Predikto doesn’t require a manual data science team to ingest and modify data for a potential predictive analytics solution. This platform takes care of most of the grunt work in a very sophisticated way while capitalizing on detail from domain experts, ultimately providing customers with what they want very rapidly (accurate predictions) at a fraction of the cost of a data science team, particularly when the goal is to deploy predictive analytics solutions across a range of problem areas. This context-based solution also automatically adapts to feedback from operations regarding the response to the predictions themselves.

Predikto Solution Utilizing Predictive Analytics

 

Skeptical? Let us show you what Auto-Dynamic Predictive Analytics is all about and how it can reduce downtime in your organization. And by the way, it works… [patents pending]

Predikto Enterprise Platform

The Scalability of Data Science: Part 3 – Reality Check  

You’re an operations manager, or even a CEO. Let’s say you have a big predictive analytics initiative and need to deploy several instances of your solution. What are your options? Ok, go ahead and hire a team of 5 data scientists, each with a relatively modest salary of $100k (US) per year (this is very conservative estimate). Now step back… You’ve just spent half a million dollars on staffing (plus the time for hiring, software procurement, etc.) for something that’s going to develop slowly and if it works at all, it may not work well. Have you made a wise investment?

This is the reality that most companies entering the realm of IoT and predictive analytics will face. Why? Most predictive analytic solutions can’t scale (i.e., can’t be rapidly applied across different problems). It’s too time consuming and too expensive and the value may be lost in a ramp-up attempt. A deployed predictive analytics solution must be scalable, fast, and affordable. A data scientist can be great (and many are), but they’re bound by the magnitude in which they can scale and the subjectivity of their respective approaches to the solution. There are many ways to approach data analysis that are correct, but there’s probably an alternative that is more valuable.

The next generation of predictive analytics solutions should be able to accomplish most, if not all, of the above automatically and rapidly with decreasing involvement from humans, and should perform as good or better than a good data science team; this is what Predikto has done (patents pending). We enable operations managers and data scientists by tackling the bulk of the grunt work.

I’m well aware that this may downplay the need for a large data science industry, but really, what’s an industry if it can’t scale? A fad perhaps. Data science is not just machine learning and some basic data manipulation skills. There’s much more to a deployed solution that will impact a customer’s bottom line. To make things worse, many of the key components of success are not things covered in textbooks or in an online course offering on data science.

It’s one thing to win a build the “best” predictive analytics solution (e.g., a Kaggle competition), but try repeating this  process of dozens times in a matter of weeks for predictions of different sorts. If any of these solutions are not correct, it costs real dollars. Realistically scaling in an applied predictive analytics environment should scare the pants off of any experienced data scientist who relies on manual development. Good data science is traditionally slow and manual. Does it have to be?

Rest assured, I’m not trying to undercut the value of a good data scientist; this is needed trade. The issue is simply that data science is difficult to scale in a business setting.

The Scalability of Data Science: Part 2 – The Reality of Deployment

To put my previous post into perspective, let me give you a for instance… An organization wants to develop a deployed predictive analytics solution for an entire class of commuter trains. Let’s be modest and go with 10 different instances from within the data (e.g.,  1) predicting engine failure, 2) turbo charger pressure loss, 3) door malfunction, … and so on…). We’ll focus on just one…

Data from dozens of assets (i.e., trains) are streaming in by the second or quicker and these data must be cleaned and aggregated with other data sources. It’s a big deal to get just this far. Next you have to become and expert in the data and begin cleaning and developing context-based feature data from the raw source data. This is where art comes into play and this part is difficult and time consuming for data scientists. Once a set of inputs has been established, then comes the easier part, applying an appropriate statistical model/s to predict something (e.g., event occurrence, time to failure, latent class, etc.) followed by validating and deploying the results. Oh yes, let’s not forget the oft unspoken reality of threshold settings for the customer (i.e., costs of TPs vs FPs, etc.). To this point, we’re assuming that the solution has value and it’s important to keep in mind that a data science team has probably never seen this sort of data ever before.

So on top of requiring computer programming skills, feature engineering prowess (which is art), understanding statistics/machine learning, and having good enough communication skills to both learn from the customer about their data and to be able to “sell” the solution, this must all be accomplished in a reasonable amount of time. We’re talking about 1 instance to this point, remember? And, we’re still not deployed. Do you have expertise in deploying data for the customer? Now repeat this situation ten times and you’re closer to reality. Your team may now just filled up the next 12 months of work and the utility of the solution is still unknown.

Internet of Things: The third wave of the Internet?

IoT

The Internet of Things (IoT) is emerging as the third wave in the development of the Internet. In the 1990’s 1 billion users were connected to the internet, by the 2000’s the release of Internet 2.0 and the popularity of internet capable smart phones that number grew to 3 billion, and now it is expected by 2020 the IoT will connect 28 billion devices to the internet.

According to a report published by Goldman Sachs, “The Internet of Things: Making sense of the next mega-trend” the Internet of Thing “connects devices such as everyday consumer objects and industrial equipment onto the network, enabling information gathering and management of these devices via software to increase efficiency, enable new services, or achieve other health, safety, or environmental benefits.”

The report classified the “things” as 5 key verticals for the adoption of this third wave of the Internet: Connected Wearable Devices, Connected Cars, Connected Homes, Connected Cities, and the Industrial Internet.

The infrastructure that will fully support the connection of 28 billion devices, consisting in telecommunications, sensors and software among other components will allow to use the data generated to make our lives easier (i.e. adjust the temperature of your home before you get home), use energy efficiently (i.e. turn on your washing machine when electricity usage and prices fall in the middle of the night), and help us anticipate and predict failures or problems (i.e. predict when your car is going to fail even before the check engine light turns on).

One of the main concerns about the connectivity of everything is privacy and security. The key for the IoT to develop to its full potential is to make sure the “things” are genuinely adding value instead of being merely intrusive. Goldman Sachs identified three key areas where the development of IoT will generated the most value: home automation, resources, and manufacturing.

The Industrial Internet has now invested large amounts of time and money in sensors, connectivity, micro-controllers and micro-processors, and the amount of data generated is often times large and unintuitive. The missing piece of that puzzle is software that combines all that information and analyzes it to transform it into actionable outputs.

Many predictive analytics companies are trying to solve a technology problem. How will you manage all that data? Should we use Hadoop or No-SQL? What are the issues of the IT department regarding the data they are collecting? I have a huge amount of data that the company is not using, is there valuable information that isn’t being leveraged? But they are not solving the main issue: what problems are you having in your manufacturing facilities that are slowing down your production rates? How can the existing data help you to reach and surpass the production goals reducing downtimes and inefficiencies?

If the operational problems are not being solved, the future of the IoT will not be bright.