Welcome!

IBM Cloud Authors: Elizabeth White, Yeshim Deniz, Pat Romanski, Liz McMillan, Stefan Bernbo

Related Topics: @DXWorldExpo, Java IoT, Microservices Expo, IBM Cloud, @CloudExpo, SDN Journal

@DXWorldExpo: Article

Big Data OLTP with IBM DB2 BLU and DB2 pureScale

Big Data and analytical processing

Big Data as we know it today is more aligned to the analytical processing of large quantities of data. All the predominant use cases identified by the Big Data product vendors are more aligned with analytical processing. For example one of the major use cases of Big Data is about utilizing social media data to get into advertisement targeting. Naturally this kind of processing analyzing lots of unstructured data and come up with predictions on customer preferences and this use case is aligned with analytical processing. To support these kinds of analytical processing Columnar databases have emerged as a natural extension to Big Data processing. Columnar databases only reads columns involved in the query and not the entire row, making it a perfect fit for analytical processing.

Online Transaction Processing
On the other side of the Enterprise Data Access pattern we have the most important OLTP pattern. Most of the real life events that are important for the survival of an enterprise like an online ecommerce transaction need to follow ACID property, where by the Atomicity, Consistency, Isolation and Durability of the transaction needs to be maintained. Due to this need OLTP applications and their natural allies, the relational databases, continue to be popular and an essential part of enterprise data access patterns.

OLTP + Big Data
Advent of Big Data and the associated analytics initially looked like a separate stream from OLTP , mainly because of the data integration challenges. For example the tools meant for big data processing were from the new era like Hadoop, where as invariably OLTP applications like banking were built out of legacy platforms, mainly platforms like Mainframe. Also the response time needs for OLTP applications are very critical and most of the batch oriented big data processing platforms cannot cater to them.

However for use cases like fraud detection of Online transactions, we needed to combine the ACID nature of the transactions with the analytical capabilities enabled by the Big Data Platform, so far we did not have very many unified platforms to achieve this combination.

However the traditional RDBMS platforms which all along cater to the OLTP needs have started to release native Big Data Integration features as part of their offering this has facilitated easier integration of Big Data Analytics with OLTP.

We find that recently IBM DB2 which is one of the popular relational databases catering to the OLTP needs in Mainframe as well as Unix, Linux, Windows has announced the BLU Acceleration on top of the traditional database offering. The rest of the sections covers about DB2 BLU Acceleration and how it can fit a Big Data OLTP processing need.

DB2 with BLU Acceleration
Recently IBM Announced the offering information about DB2 10.5 for Linux, Unix and Windows. There are some major components within the DB2 10.5, which caters to the Big data OLTP Needs.

  • DB2 With BLU Acceleration
  • DB2 PureScale Clustering Technologies

DB2 with BLU Acceleration is dynamic in-memory technology that yields faster analytics without the costs or limits of in-memory only systems. It speeds analytics and reporting with a combination of in-memory and column store data retrieval, maximized CPU processing, and data skipping that allows for faster input/output.

IBM DB2 pureScale® database clustering technology. Helps to ensure transaction processing is highly available and extremely scalable.

The following are the important Analytical Aspects of DB2 With BLU Acceleration.

  • Fully Integrated Solution as part of the base DB2 database which enables the combination of OLTP with Big Data Analytics
  • Utilizes the Column Organized Table Architecture so that the I/O Operations on the Analytical queries are fully optimized
  • Advanced Compression Technologies further optimize the columnar storage
  • In Memory Database storage further optimizes already compressed columnar data
  • A new concept of Data Skipping further skips the data that is not of interest resulting in further efficiencies
  • All the above optimizations work in compliment with the Parallel Processing nature of the solution

Another important aspect is that BLU Acceleration works with the familiar regular DB2 environment such that all the commands like LOAD, IMPORT and other DML commands work seamlessly with it. The regular DDL commands have been extended to support BLU Acceleration.

On the other hand, IBM DB2 PureScale is a multi instance shared served model similar to PARALLEL SYSPLEX environment in Mainframe and can be an ideal candidate for large scale OLTP processing. A DB2 pureScale environment is ideal for short transactions where there is little need to parallelize each query. Queries are automatically routed to different members, based on member workload. The ideal scenario for a DB2 pureScale environment includes workloads that handle online transaction processing (OLTP) or enterprise resource planning (ERP).

With the possibility to combine operations, especially JOINS between traditional OLTP work load row organized tables and Analytical work load column organized tables within the same DB2 environment powered by PureScale & BLU Acceleration technologies, the below architecture provides a blue print for a Big Data OLTP environment.

There is not a lot of documentation available on DB2 BLU Acceleration at this time; however, from my understanding it looks like at present DB2 BLU Scale Up Vertically within a powerful server with multiple cores rather than Scale Out using multiple horizontal servers, while this understanding may be wrong, but availability of both Scale UP and Scale Out would be bigger boost in enterprise scenarios. In other words the ability to use the DB2 BLU in conjunction with DB2 Data Partitioning Feature will further enhance the scalability of the Analytical component.

Summary
Combining the Big Data features with traditional RDBMS makes enterprises easily embrace Hybrid Big Data much like the success of Hybrid Cloud we see a increasing adoption here. The above mentioned combination of BLU Acceleration In DB2 together with traditional features like PureScale is a good example of Big Data OLTP. As I write this article I find Sql Server 2014 announcements are flashing on various sites which talks about features like In Memory OLTP which will provide further choices for enterprises.

More Stories By Srinivasan Sundara Rajan

Highly passionate about utilizing Digital Technologies to enable next generation enterprise. Believes in enterprise transformation through the Natives (Cloud Native & Mobile Native).

IoT & Smart Cities Stories
DXWordEXPO New York 2018, colocated with CloudEXPO New York 2018 will be held November 11-13, 2018, in New York City and will bring together Cloud Computing, FinTech and Blockchain, Digital Transformation, Big Data, Internet of Things, DevOps, AI, Machine Learning and WebRTC to one location.
@DevOpsSummit at Cloud Expo, taking place November 12-13 in New York City, NY, is co-located with 22nd international CloudEXPO | first international DXWorldEXPO and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time t...
Cloud-enabled transformation has evolved from cost saving measure to business innovation strategy -- one that combines the cloud with cognitive capabilities to drive market disruption. Learn how you can achieve the insight and agility you need to gain a competitive advantage. Industry-acclaimed CTO and cloud expert, Shankar Kalyana presents. Only the most exceptional IBMers are appointed with the rare distinction of IBM Fellow, the highest technical honor in the company. Shankar has also receive...
A valuable conference experience generates new contacts, sales leads, potential strategic partners and potential investors; helps gather competitive intelligence and even provides inspiration for new products and services. Conference Guru works with conference organizers to pass great deals to great conferences, helping you discover new conferences and increase your return on investment.
Headquartered in Plainsboro, NJ, Synametrics Technologies has provided IT professionals and computer systems developers since 1997. Based on the success of their initial product offerings (WinSQL and DeltaCopy), the company continues to create and hone innovative products that help its customers get more from their computer applications, databases and infrastructure. To date, over one million users around the world have chosen Synametrics solutions to help power their accelerated business or per...
DXWorldEXPO LLC announced today that ICOHOLDER named "Media Sponsor" of Miami Blockchain Event by FinTechEXPO. ICOHOLDER gives detailed information and help the community to invest in the trusty projects. Miami Blockchain Event by FinTechEXPO has opened its Call for Papers. The two-day event will present 20 top Blockchain experts. All speaking inquiries which covers the following information can be submitted by email to info@dxworldexpo.com. Miami Blockchain Event by FinTechEXPOalso offers sp...
SYS-CON Events announced today that IoT Global Network has been named “Media Sponsor” of SYS-CON's @ThingsExpo, which will take place on June 6–8, 2017, at the Javits Center in New York City, NY. The IoT Global Network is a platform where you can connect with industry experts and network across the IoT community to build the successful IoT business of the future.
IoT is rapidly becoming mainstream as more and more investments are made into the platforms and technology. As this movement continues to expand and gain momentum it creates a massive wall of noise that can be difficult to sift through. Unfortunately, this inevitably makes IoT less approachable for people to get started with and can hamper efforts to integrate this key technology into your own portfolio. There are so many connected products already in place today with many hundreds more on the h...
The best way to leverage your Cloud Expo presence as a sponsor and exhibitor is to plan your news announcements around our events. The press covering Cloud Expo and @ThingsExpo will have access to these releases and will amplify your news announcements. More than two dozen Cloud companies either set deals at our shows or have announced their mergers and acquisitions at Cloud Expo. Product announcements during our show provide your company with the most reach through our targeted audiences.
Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settlement products to hedge funds and investment banks. After, he co-founded a revenue cycle management company where he learned about Bitcoin and eventually Ethereal. Andrew's role at ConsenSys Enterprise is a mul...