Register for a free consultation!

Big Data: Modern Day Miracle or Same Old Snake Oil?

8/8/2016 10:34 AM

We've been hearing a lot about Big Data in news and social media. What exactly comprises it and why is it a big deal? This wiki article says "Big data is a term for data sets that are so large or complex that traditional data processing applications are inadequate." That certainly sounds ominous, but why is it being collected and what do you use it for? 

According to my research Big Data should be able to predict when a machine will fail, identify manufacturing bottlenecks, point out waste and care variability in healthcare, stop fraud on credit card purchases, predict drive times in high volume areas, even forecast the winners of sporting events and tune race cars. That's all great but how does it happen? I heard the terms "Hadoop" and "NoSQL" a lot while researching this article, so let's dive a little deeper into the technology. 

In "Hadoop: what it is And How It Works" Brian Proffitt explains: "It's a way of storing enormous datasets across distributed clusters of servers and then running distributed analysis applications in each cluster." Great, but that doesn't really tell me much. To me, that sounds like a regular everyday cluster and that technology has been around for decades. Mr. Proffitt goes on to explain that the Hadoop File System (HDFS), the Data Processing Framework and MapReduce are the secret sauce that makes things work. So you dump your data into the HDFS and you can use MapReduce to do the data processing. Great, but where does the data actually reside? Apparently it resides on the local machine in whatever place the Hadoop developer dumps it, so you are reduced to network speeds to transfer amazingly large amounts of data from one machine to the next in the cluster. Anyone who has ever used Neflix Streaming Service knows what a stupendously bad idea that is. Here is what Mr. Proffitt has to say about MapReduce: "(it) runs as a series of jobs, with each job essentially a separate Java application that goes out into the data and starts pulling out information as needed." 

Danger Will Robinson. Here there be monsters. So you are telling me that we are going to use procedural programming tools to solve manufacturing bottleneck problems across a distributed network? 'Horse hockey.  Sufferin' Saddle Soap.  Buffalo Chips.  BULL COOKIES!  I've used this analogy before and I'm going to use it again: If you want to find a rock about the size of a quarter, build two boxes: one with holes slightly larger than a quarter, one with holes slightly smaller than a quarter and run all the rocks in the world through both boxes. What you are left with are all the rocks in the world about the size of a quarter. This is set based programming. Procedural programming, on the other hand, takes the proverbial quarter and lines up all the rocks in the world and compares the size one by one. In database circles this is known as RBAR (REE-bar, like the stuff that holds concrete together) and is the acronym for ‘Row By Agonizing Row.' I have run into this literally a hundred times or more in my career. Would you rather build two boxes or would you rather have a pair of tweezers in one hand and quarter in the other?

So what is the conclusion? 

There is no such thing as Big Data

If you can't process it with traditional databases, it can't be processed. Take whatever data you have and structure it. Yes, some NoSQL alternatives are document based. You don't need documents. Even if you have documents (like this HTML page) take the contents add it to a full text index and then you can even search the documents, without using tweezers and a quarter. You have noticed that Microsoft didn't rush out and create a new Big Data engine, haven't you? Nope, Microsoft will let you use Hadoop because they will take your money if you are foolish enough to spend it that way, but there is already a better solution than RBAR. Structuring the data is going to take about as long as that first MapReduce query, but every query after that is basically free. the second MapReduce 'job' takes precisely as long as the first one, and for the same dumb reason: procedural code running against huge amounts of data. 

So what do we get and what do we lose by using a traditional Relational Database instead of Hadoop/NoSQL? First the data has to be structured. It is going to take time to look at the data and figure out how to get it into your Relational Database Management System (RDBMS). Of course, it would take the same amount of time to get it into your Hadoop train wreck as well, so that is a wash. SQL is pretty easy to use and MapReduce is mind-bogglingly complicated. Mr. Profitt (remember him?) even stated in his article that some map reduce instances take a SQL like language and translate it for you. That seems a little backwards. If you can describe what you are trying to do using the tools you have, why not just use the tools that you have? 

You already have a relational database. It is how the world has worked to this point. You can use SQL queries to ask your existing database questions. If you need advanced analytics, maybe you should abandon Boyce and Codd and say hello to Kimball and Inmon. Probably your data isn't big enough to need even a data warehouse (go read about the Kimball and Inmon guys) but if it is, don't just scrap the Ferrari you've had built and go buy a Yugo with a Hadoop bumper sticker. The conclusion is that the whole Big Data movement is complete snake oil designed to get you, dear reader and holder of cash, to spend money on something you don't really understand and certainly do not need.

Date Written Comment
10/18/2020 11:08 AM ngrs22 levitra nizagara
10/18/2020 2:16 PM VQDnRN levitra nizagara
11/13/2020 10:14 PM Bct9p7 viagra online
12/3/2020 3:29 PM lMffZN write my essay
12/3/2020 6:53 PM pNZ7GD write my essay
12/12/2020 1:58 PM bErGsj xnxx videos
12/12/2020 3:33 PM QFYd9O xnxx videos
12/12/2020 8:23 PM hhmZhy
12/12/2020 10:03 PM EE3e9E
12/14/2020 9:22 PM fD0DmQ
12/14/2020 11:02 PM P1lSeV
1/9/2021 2:01 AM ZJPnYz
1/9/2021 4:09 AM Td6GY4
1/9/2021 6:16 AM x1JTmS waldorf doll
1/9/2021 8:29 AM oklPY8 waldorf doll
1/26/2021 7:01 AM QDOGYn
1/26/2021 9:22 AM g69FPV
1/27/2021 4:26 AM EJADqe
1/27/2021 6:32 AM bfviX5
6/28/2021 7:59 PM qQmkcC xnxx
7/22/2021 4:45 PM write my essays
8/3/2021 9:00 AM
8/3/2021 9:09 AM
8/11/2021 2:55 PM How long are you planning to stay here? pof In June, investors withdrew a record $9.6 billion from Pacific Investment Management Co's flagship PIMCO Total Return Fund, the world's largest bond fund, which is down 3 percent for the year, according to Morningstar.
8/11/2021 2:55 PM I'd like to withdraw $100, please silverdaddies Our capital markets regulators, the Securities and Exchange Commission and Commodity Futures Trading Commission, are required to either set up their own regulations or accept a foreign system as compliant for all swaps that fit this definition.
8/11/2021 2:55 PM I'd like to open a business account xtube While Collins suggested the long delay and pressure of pitching for an All-Star spot was weighing on Harvey, he also admitted that the starter who has wowed the majors this season may have just had a bad night.
8/11/2021 2:55 PM Hold the line, please wifelovers More moderate Senate Republicans disapproved of using Obamacare as a bargaining chip from the start, arguing that it because it is non-negotiable for Democrats, it would inevitably lead to a shutdown and sour voters on the Republican Party.
8/11/2021 2:55 PM I can't stand football pof Italian industrial production is expected to rise 0.3pc, while the recent rise in Italian bond yields could well see this morning’s 15 year bond issue by the Italian treasury fetch its highest yield in several months.
11/15/2021 9:06 AM I sing in a choir stromectol ivermectin So does location shooting that lets Walter’s world travels — to Greenland, Iceland and Noshaq mountain, Afghanistan — trump his earlier fantasies, while providing backdrops that still look relatively exotic to seen-it-all auds. From a technical perspective, this is Stiller’s slickest pic yet, demonstrating creative widescreen framing and an inspired blend of dramatic score and recent pop tunes throughout. While the confusing mix of genres and ideas might throw first-wave auds for a loop, the approach is timeless enough to hold up. Who knows? In 20 years, the death of Life might seem less outdated, and Walter’s awakening could be as profound as he imagines.
11/15/2021 9:06 AM I study here stromectol ivermectin Detective Inspector Simon Ellershaw, leading the investigation, said: “Despite these arrests we are still appealing for anyone with information or any witnesses to please contact police as soon as possible.”
11/15/2021 9:06 AM I don't know what I want to do after university stromectol ivermectin "So if you get into a situation like this, it really should be normal. We're here to win a game today. Obviously, we know if you don't win today, you go home. If you put that pressure on yourself daily, then this pressure is no different."
11/15/2021 9:06 AM I've been made redundant stromectol ivermectin Britain's Intertek said in May that a sharper than expecteddecline in demand for its minerals business, particularly inAustralia, Brazil and the Philippines, would drag on its profitmargin this year.
11/15/2021 9:06 AM Nice to meet you stromectol ivermectin Grunwald praised Smith for playing injured for most of last season, which resulted in Smith winning Sixth Man of the Year. However, Smith struggled during the playoffs and accepted responsibility for the Knicks being eliminated by Indiana in six games.
11/15/2021 9:06 PM I'm originally from Dublin but now live in Edinburgh stromectol ivermectin Twitter, which is expected to go public some time beforeThanksgiving, has yet to determine pricing, but investors say itmight come under pressure from its financial backers to go high.Analysts expect the company, which posted a $69 million losslast year, to seek a valuation of at least $10 billion.
11/15/2021 9:06 PM What sort of music do you like? stromectol ivermectin Germany's David Storl retained his world shot put title in controversial circumstances when his winning throw of 21.73 meters, originally ruled a foul, was allowed to stand after judges consulted a photographer's camera.
11/15/2021 9:06 PM I like watching football stromectol ivermectin This comes from our ad serving technology and is used to track how many times you have seen a particular ad on our sites, so that you don't just see one advert but an even spread. This information is not used by us for any other type of audience recording or monitoring.
11/15/2021 9:06 PM I'm sorry, she's stromectol ivermectin In a move seen as a response to the UKIP threat and unrest among Conservative eurosceptics, Cameron promised in January to win back powers from the EU and then hold a referendum by the end of 2017 on whether Britain should remain a member.
11/15/2021 9:06 PM How long are you planning to stay here? stromectol ivermectin Under its latest offerings Sprint said that its smartphonecustomers would pay $240 less per year than customers of marketleader Verizon Wireless, a venture of Verizon Communications and Vodafone Group Plc and $120 less per yearthan No. 2 U.S. operator AT&T and smaller rival T-Mobile US.
11/15/2021 11:03 PM Is this a temporary or permanent position? stromectol ivermectin Mr Aston told the inquiry that - in a separate incident in a derelict building after the battle - he saw two privates hitting and kicking a detainee "several times each", landing blows to different parts of his body "including his face, shins and ribs".
11/15/2021 11:03 PM Wonderfull great site stromectol ivermectin Twitter intends to list common stock under the symbol "TWTR." Goldman Sachs, Morgan Stanley, JPMorgan, BofA Merrill Lynch, Allen & Co, Deutsche Bank Securities and Code Advisors are managing Twitter's IPO.
11/15/2021 11:03 PM This is the job description stromectol ivermectin However, the Brazilian television network Globo this monthalleged that the NSA had tapped into the computer systems ofcompanies including Google Inc. and the Brazilian stateoil firm Petroleo Brasileiro, as well as France'sForeign Ministry and the Belgian-based international bankingcooperative SWIFT, which handles international financialtransactions.
11/15/2021 11:03 PM I'm unemployed stromectol ivermectin The ICRC knows that it can’t dictate how players act on the virtual battlefield, but hopes that if manufacturers include punishment for those who simulate war crimes, they will better portray the reality of armed conflict.
11/15/2021 11:03 PM Jonny was here stromectol ivermectin No group immediately claimed responsibility for the assault Saturday. Tribal clashes remain common in the region and some former government militias have begun taking up arms again as fighting continues over land and resources.
11/16/2021 12:06 AM I like watching football stromectol ivermectin "They're very young, strong quarterbacks," Burleson said. "I think Matt, he's not just trying to separate himself from Sanchez, I think Matt's trying to separate himself from the top 10. He wants to be considered a top-tier quarterback — top five, top three."
11/16/2021 12:07 AM A pension scheme stromectol ivermectin The new generation of the Chevrolet Corvette Stingray manages to end up in our news once again, after it was officially unveiled in its two body styles, coupe and convertible, over these past few months, this time with the help of the fuel economy figures. According to the car manufacturer, the new sports car generation can return 17 mpg in the city and 29 mpg on the highway with the seven-speed manual transmission, 28 mpg in Tour mode and even 30 mpg when in Eco mode with the cylinder deactivation system.
11/16/2021 12:07 AM Can I take your number? stromectol ivermectin Karen Friedman-Agnifilo, Vance's chief trial prosecutor, said in a statement that the "low-level" charges against Edwards could undermine the more serious case Vance is trying to build against other bikers who remain at large.
11/16/2021 12:07 AM I'm in my first year at university stromectol ivermectin Let's face it, Tim DeRuyter isn't long for the Valley; he's a terrific coach who will soon be swooped up by a powerful BCS school. Once that happens, the Bulldogs will be on the market -- and when you're at the mid-major level, a guy that can recruit and bring players to a team is critical.
11/16/2021 5:12 AM Would you like to leave a message? stromectol ivermectin Wilson said Washington, D.C. was chosen as the next city to receive help from BTAN because roughly 75 percent of all HIV cases in Washington, D.C. stem from the Black community- even though only 46 percent of the District is African American.
11/16/2021 5:12 AM Sorry, I ran out of credit stromectol ivermectin Still, Joe Girardi didn’t dismiss the possibility of Jeter returning before the break. “There’s always a chance,” the manager said before Monday’s game against Kansas City. “You just have to see how he does the next three or four days.”
11/16/2021 5:12 AM Will I get travelling expenses? stromectol ivermectin Capping weeks of political brinkmanship that had unnervedglobal markets, the Senate and House of Representatives eachpassed the spending measure. President Barack Obama was expectedto sign the bill imminently.
11/16/2021 5:12 AM Which team do you support? stromectol ivermectin "In many countries that have previously relied mainly onexports to advanced countries there is now greater role fordomestic demand," said Anoop Singh, head of the InternationalMonetary Fund's Asia-Pacific department in Washington.
11/16/2021 5:12 AM I'm happy very good site stromectol ivermectin Members of the European Parliament, which together with theEU's 28 member states would have to approve the Commissionproposal for it to become law, have also raised objections,saying the Montreal agreement is empty.
11/16/2021 7:19 AM I'm retired stromectol ivermectin The government also considered selling a smaller stake toallow it to cash in on higher prices at a later date but optedagainst the plan because it would squeeze the number of sharesavailable to institutions and could fuel volatility.
11/16/2021 7:19 AM I was born in Australia but grew up in England stromectol ivermectin "Judging by the shaped plastic 'fetuses' that they threw to children watching the parade at the North Dakota State Fair," she wrote, the anti-abortion activists "have definitely reached an all-time low in bad taste."
11/16/2021 7:19 AM I'm not interested in football stromectol ivermectin While claims are usually volatile in July due to auto plant shutdowns, economists who had expected new filings to rise to 345,000 said the general tone of the report was consistent with a pick-up in job growth.
11/16/2021 7:19 AM What do you do for a living? stromectol ivermectin Developing the Prirazlomnoye oil deposit, in the Pechora Seasome 60 km (40 miles) from Russia's northern coast, was delayedlast year, with Gazprom blaming technical problems. It isexpected to reach peak production of 6 million tonnes per year(120,000 barrels per day) in 2019. (Editing by Ralph Boulton)
11/16/2021 7:19 AM I live here stromectol ivermectin Air France-KLM may want to follow the example of Lufthansa, which uses Vienna and Zurich to feed its long-haulflights out of its Frankfurt and Munich hubs, analysts said.Alitalia could also be used for point-to-point routes bypassingAir France-KLM's two main hubs, possibly linking Rome with theeast coast of the United States and parts of Africa.
11/16/2021 9:31 AM I didn't go to university stromectol ivermectin Campbell, who police described as a transient with no strong ties to Southern California, was taken into custody after abandoning his car blocks from the scene and walking to a police station in the adjacent community of Santa Monica.
11/16/2021 9:31 AM A few months stromectol ivermectin -- Dutch pension group PGGM to acquire control of gasdistributor Northern Offshore Upstream Gas Pipeline System,which is jointly controlled by French gas and power group GDFSuez and EBN B.V. (notified Sept.17/deadline Oct. 22/simplified)
11/16/2021 9:31 AM I hate shopping stromectol ivermectin The first blast, prosecutors stated, went off in front of Marathon Sports at 671 Boylston Street at 2:49 pm – detonated by Tamerlan Tsarnaev using a cell phone after he received a call from Dzhokhar giving him the okay.
11/16/2021 9:31 AM very best job stromectol ivermectin "There's all kinds of consulting opportunities," said Levin,whose client, the Detroit Institute of Arts museum, is at thecenter of a dispute over whether the city can sell the museum'sart collection.
11/16/2021 9:31 AM Where do you study? stromectol ivermectin “We have been working towards opening stores in Scotland for a long time but wanted to make sure the company was structured to support the expansion. We recently opened a huge 750,000sq ft warehouse in Doncaster that will support Scotland temporarily while we look at a specific warehouse in Scotland. The Scotland warehouse will bring a further 400 jobs.”

Add Comment: