Welcome!

IBM Cloud Authors: Elizabeth White, Yeshim Deniz, Pat Romanski, Liz McMillan, Stefan Bernbo

Related Topics: @DXWorldExpo, Java IoT, Microservices Expo, IBM Cloud, @CloudExpo, SDN Journal

@DXWorldExpo: Article

Big Data OLTP with IBM DB2 BLU and DB2 pureScale

Big Data and analytical processing

Big Data as we know it today is more aligned to the analytical processing of large quantities of data. All the predominant use cases identified by the Big Data product vendors are more aligned with analytical processing. For example one of the major use cases of Big Data is about utilizing social media data to get into advertisement targeting. Naturally this kind of processing analyzing lots of unstructured data and come up with predictions on customer preferences and this use case is aligned with analytical processing. To support these kinds of analytical processing Columnar databases have emerged as a natural extension to Big Data processing. Columnar databases only reads columns involved in the query and not the entire row, making it a perfect fit for analytical processing.

Online Transaction Processing
On the other side of the Enterprise Data Access pattern we have the most important OLTP pattern. Most of the real life events that are important for the survival of an enterprise like an online ecommerce transaction need to follow ACID property, where by the Atomicity, Consistency, Isolation and Durability of the transaction needs to be maintained. Due to this need OLTP applications and their natural allies, the relational databases, continue to be popular and an essential part of enterprise data access patterns.

OLTP + Big Data
Advent of Big Data and the associated analytics initially looked like a separate stream from OLTP , mainly because of the data integration challenges. For example the tools meant for big data processing were from the new era like Hadoop, where as invariably OLTP applications like banking were built out of legacy platforms, mainly platforms like Mainframe. Also the response time needs for OLTP applications are very critical and most of the batch oriented big data processing platforms cannot cater to them.

However for use cases like fraud detection of Online transactions, we needed to combine the ACID nature of the transactions with the analytical capabilities enabled by the Big Data Platform, so far we did not have very many unified platforms to achieve this combination.

However the traditional RDBMS platforms which all along cater to the OLTP needs have started to release native Big Data Integration features as part of their offering this has facilitated easier integration of Big Data Analytics with OLTP.

We find that recently IBM DB2 which is one of the popular relational databases catering to the OLTP needs in Mainframe as well as Unix, Linux, Windows has announced the BLU Acceleration on top of the traditional database offering. The rest of the sections covers about DB2 BLU Acceleration and how it can fit a Big Data OLTP processing need.

DB2 with BLU Acceleration
Recently IBM Announced the offering information about DB2 10.5 for Linux, Unix and Windows. There are some major components within the DB2 10.5, which caters to the Big data OLTP Needs.

  • DB2 With BLU Acceleration
  • DB2 PureScale Clustering Technologies

DB2 with BLU Acceleration is dynamic in-memory technology that yields faster analytics without the costs or limits of in-memory only systems. It speeds analytics and reporting with a combination of in-memory and column store data retrieval, maximized CPU processing, and data skipping that allows for faster input/output.

IBM DB2 pureScale® database clustering technology. Helps to ensure transaction processing is highly available and extremely scalable.

The following are the important Analytical Aspects of DB2 With BLU Acceleration.

  • Fully Integrated Solution as part of the base DB2 database which enables the combination of OLTP with Big Data Analytics
  • Utilizes the Column Organized Table Architecture so that the I/O Operations on the Analytical queries are fully optimized
  • Advanced Compression Technologies further optimize the columnar storage
  • In Memory Database storage further optimizes already compressed columnar data
  • A new concept of Data Skipping further skips the data that is not of interest resulting in further efficiencies
  • All the above optimizations work in compliment with the Parallel Processing nature of the solution

Another important aspect is that BLU Acceleration works with the familiar regular DB2 environment such that all the commands like LOAD, IMPORT and other DML commands work seamlessly with it. The regular DDL commands have been extended to support BLU Acceleration.

On the other hand, IBM DB2 PureScale is a multi instance shared served model similar to PARALLEL SYSPLEX environment in Mainframe and can be an ideal candidate for large scale OLTP processing. A DB2 pureScale environment is ideal for short transactions where there is little need to parallelize each query. Queries are automatically routed to different members, based on member workload. The ideal scenario for a DB2 pureScale environment includes workloads that handle online transaction processing (OLTP) or enterprise resource planning (ERP).

With the possibility to combine operations, especially JOINS between traditional OLTP work load row organized tables and Analytical work load column organized tables within the same DB2 environment powered by PureScale & BLU Acceleration technologies, the below architecture provides a blue print for a Big Data OLTP environment.

There is not a lot of documentation available on DB2 BLU Acceleration at this time; however, from my understanding it looks like at present DB2 BLU Scale Up Vertically within a powerful server with multiple cores rather than Scale Out using multiple horizontal servers, while this understanding may be wrong, but availability of both Scale UP and Scale Out would be bigger boost in enterprise scenarios. In other words the ability to use the DB2 BLU in conjunction with DB2 Data Partitioning Feature will further enhance the scalability of the Analytical component.

Summary
Combining the Big Data features with traditional RDBMS makes enterprises easily embrace Hybrid Big Data much like the success of Hybrid Cloud we see a increasing adoption here. The above mentioned combination of BLU Acceleration In DB2 together with traditional features like PureScale is a good example of Big Data OLTP. As I write this article I find Sql Server 2014 announcements are flashing on various sites which talks about features like In Memory OLTP which will provide further choices for enterprises.

More Stories By Srinivasan Sundara Rajan

Highly passionate about utilizing Digital Technologies to enable next generation enterprise. Believes in enterprise transformation through the Natives (Cloud Native & Mobile Native).

IoT & Smart Cities Stories
In his general session at 19th Cloud Expo, Manish Dixit, VP of Product and Engineering at Dice, discussed how Dice leverages data insights and tools to help both tech professionals and recruiters better understand how skills relate to each other and which skills are in high demand using interactive visualizations and salary indicator tools to maximize earning potential. Manish Dixit is VP of Product and Engineering at Dice. As the leader of the Product, Engineering and Data Sciences team at D...
Bill Schmarzo, Tech Chair of "Big Data | Analytics" of upcoming CloudEXPO | DXWorldEXPO New York (November 12-13, 2018, New York City) today announced the outline and schedule of the track. "The track has been designed in experience/degree order," said Schmarzo. "So, that folks who attend the entire track can leave the conference with some of the skills necessary to get their work done when they get back to their offices. It actually ties back to some work that I'm doing at the University of San...
When talking IoT we often focus on the devices, the sensors, the hardware itself. The new smart appliances, the new smart or self-driving cars (which are amalgamations of many ‘things'). When we are looking at the world of IoT, we should take a step back, look at the big picture. What value are these devices providing. IoT is not about the devices, its about the data consumed and generated. The devices are tools, mechanisms, conduits. This paper discusses the considerations when dealing with the...
Bill Schmarzo, author of "Big Data: Understanding How Data Powers Big Business" and "Big Data MBA: Driving Business Strategies with Data Science," is responsible for setting the strategy and defining the Big Data service offerings and capabilities for EMC Global Services Big Data Practice. As the CTO for the Big Data Practice, he is responsible for working with organizations to help them identify where and how to start their big data journeys. He's written several white papers, is an avid blogge...
Dynatrace is an application performance management software company with products for the information technology departments and digital business owners of medium and large businesses. Building the Future of Monitoring with Artificial Intelligence. Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more busine...
If a machine can invent, does this mean the end of the patent system as we know it? The patent system, both in the US and Europe, allows companies to protect their inventions and helps foster innovation. However, Artificial Intelligence (AI) could be set to disrupt the patent system as we know it. This talk will examine how AI may change the patent landscape in the years to come. Furthermore, ways in which companies can best protect their AI related inventions will be examined from both a US and...
Enterprises have taken advantage of IoT to achieve important revenue and cost advantages. What is less apparent is how incumbent enterprises operating at scale have, following success with IoT, built analytic, operations management and software development capabilities - ranging from autonomous vehicles to manageable robotics installations. They have embraced these capabilities as if they were Silicon Valley startups.
Chris Matthieu is the President & CEO of Computes, inc. He brings 30 years of experience in development and launches of disruptive technologies to create new market opportunities as well as enhance enterprise product portfolios with emerging technologies. His most recent venture was Octoblu, a cross-protocol Internet of Things (IoT) mesh network platform, acquired by Citrix. Prior to co-founding Octoblu, Chris was founder of Nodester, an open-source Node.JS PaaS which was acquired by AppFog and ...
The deluge of IoT sensor data collected from connected devices and the powerful AI required to make that data actionable are giving rise to a hybrid ecosystem in which cloud, on-prem and edge processes become interweaved. Attendees will learn how emerging composable infrastructure solutions deliver the adaptive architecture needed to manage this new data reality. Machine learning algorithms can better anticipate data storms and automate resources to support surges, including fully scalable GPU-c...
Cloud-enabled transformation has evolved from cost saving measure to business innovation strategy -- one that combines the cloud with cognitive capabilities to drive market disruption. Learn how you can achieve the insight and agility you need to gain a competitive advantage. Industry-acclaimed CTO and cloud expert, Shankar Kalyana presents. Only the most exceptional IBMers are appointed with the rare distinction of IBM Fellow, the highest technical honor in the company. Shankar has also receive...