Welcome!

IBM Cloud Authors: Elizabeth White, Yeshim Deniz, Pat Romanski, Liz McMillan, Stefan Bernbo

Blog Feed Post

(Hadoop & SQL) – The new world of Big Data

I attended a big event yesterday, organized by The Hive (mission – incubate, fund, launch data driven business). There were close to 500 people attending plus several hundreds remotely connected to live video streaming. It was a panel discussion on the status of  SQL API on Hadoop.

Panel members were: Susheel Kaushik from Pivotal (the spin-off  from EMC and VMWare with GE as key investor), Alan Gates (Hortonworks), Tomer Shiran (MapR), Justin Erickson (Cloudera), and Priyank Patel (Teradata Aster). The moderator was Raghu Ramakrishnan from Microsoft.The major question was – why SQL on Hadoop is attracting so much attention and what is the current status?

Each panelist gave a 3 minute talk on their initiatives in bringing SQL to Hadoop. Pivotal has a project called HAWQ that promises to expand the productivity and possibilities of Hadoop with existing SQL skill sets. Hortonworks’s project Stinger aims to improve Hive performance by 100x and also to extend Hive SQL to include features needed for analytics. MapR claims to have the broadest SQL support with its Apache Drill project. Cloudera’s Impala project offers interactive SQL (4-65x faster than Hive) plus SQL queries via HiveQL. Finally, Teradata SQL-H gives business users a better Way to access data stored in Hadoop.

There was an animated debate on SQL standards and speakers claimed that they were giving priority to what functions users need and basically starting with SQL-92 base level. It was clear that these efforts are not just to support a popular query language, but to open the possibilities of Hadoop data analysis via SQL. Several BI tools using the SQL APi can also take advantage of Hadoop data access. It is a starting point, but not the end point. Existing skill sets are a big motivation to popularize Hadoop in the enterprises. If you are a start-up with no legacy to worry about, then SQL support is not a big deal. But enterprises have been using SQL for over two decades and switching to something new is considered a big barrier.

The moderator pointed out that there are varieties of applications on data (he called it a digital shoebox store) such as SQL/Hive MR, stream processing, BI, and machine learning. Some of the big data coming from digital exhaust (logs) may require special analytic tools.

Overall it was a good session and showed the general interest on Big Data and Hadoop. Interestingly, none of the incumbents (IBM, Oracle, HP, SAP) were there. It’s a new world!


Read the original blog entry...

More Stories By Jnan Dash

Jnan Dash is Senior Advisor at EZShield Inc., Advisor at ScaleDB and Board Member at Compassites Software Solutions. He has lived in Silicon Valley since 1979. Formerly he was the Chief Strategy Officer (Consulting) at Curl Inc., before which he spent ten years at Oracle Corporation and was the Group Vice President, Systems Architecture and Technology till 2002. He was responsible for setting Oracle's core database and application server product directions and interacted with customers worldwide in translating future needs to product plans. Before that he spent 16 years at IBM. He blogs at http://jnandash.ulitzer.com.

IoT & Smart Cities Stories
DXWordEXPO New York 2018, colocated with CloudEXPO New York 2018 will be held November 11-13, 2018, in New York City and will bring together Cloud Computing, FinTech and Blockchain, Digital Transformation, Big Data, Internet of Things, DevOps, AI, Machine Learning and WebRTC to one location.
We are seeing a major migration of enterprises applications to the cloud. As cloud and business use of real time applications accelerate, legacy networks are no longer able to architecturally support cloud adoption and deliver the performance and security required by highly distributed enterprises. These outdated solutions have become more costly and complicated to implement, install, manage, and maintain.SD-WAN offers unlimited capabilities for accessing the benefits of the cloud and Internet. ...
Business professionals no longer wonder if they'll migrate to the cloud; it's now a matter of when. The cloud environment has proved to be a major force in transitioning to an agile business model that enables quick decisions and fast implementation that solidify customer relationships. And when the cloud is combined with the power of cognitive computing, it drives innovation and transformation that achieves astounding competitive advantage.
DXWorldEXPO LLC announced today that "IoT Now" was named media sponsor of CloudEXPO | DXWorldEXPO 2018 New York, which will take place on November 11-13, 2018 in New York City, NY. IoT Now explores the evolving opportunities and challenges facing CSPs, and it passes on some lessons learned from those who have taken the first steps in next-gen IoT services.
SYS-CON Events announced today that Silicon India has been named “Media Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Published in Silicon Valley, Silicon India magazine is the premiere platform for CIOs to discuss their innovative enterprise solutions and allows IT vendors to learn about new solutions that can help grow their business.
In his general session at 19th Cloud Expo, Manish Dixit, VP of Product and Engineering at Dice, discussed how Dice leverages data insights and tools to help both tech professionals and recruiters better understand how skills relate to each other and which skills are in high demand using interactive visualizations and salary indicator tools to maximize earning potential. Manish Dixit is VP of Product and Engineering at Dice. As the leader of the Product, Engineering and Data Sciences team at D...
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5–7, 2018, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buye...
Founded in 2000, Chetu Inc. is a global provider of customized software development solutions and IT staff augmentation services for software technology providers. By providing clients with unparalleled niche technology expertise and industry experience, Chetu has become the premiere long-term, back-end software development partner for start-ups, SMBs, and Fortune 500 companies. Chetu is headquartered in Plantation, Florida, with thirteen offices throughout the U.S. and abroad.
The standardization of container runtimes and images has sparked the creation of an almost overwhelming number of new open source projects that build on and otherwise work with these specifications. Of course, there's Kubernetes, which orchestrates and manages collections of containers. It was one of the first and best-known examples of projects that make containers truly useful for production use. However, more recently, the container ecosystem has truly exploded. A service mesh like Istio addr...
SYS-CON Events announced today that DatacenterDynamics has been named “Media Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY. DatacenterDynamics is a brand of DCD Group, a global B2B media and publishing company that develops products to help senior professionals in the world's most ICT dependent organizations make risk-based infrastructure and capacity decisions.