Welcome!

IBM Cloud Authors: Jyoti Bansal, Yeshim Deniz, Pat Romanski, Liz McMillan, Carmen Gonzalez

Related Topics: IBM Cloud

IBM Cloud: Article

Looking Beyond MQ

Improve levels of service and avoid costly mistakes

It's likely you've been working with WebSphere MQ (WMQ) for years developing, deploying, monitoring, or all of the above. It's also likely that by now you have assembled a tool bag full of items to support your implementations and ongoing operations. And you've no doubt become accustomed to dealing with problems within the MQ domain. But what if you could see more of the transactional journey on either side of MQ? Organizations doing this are finding new ways to improve levels of service and avoid costly upgrades.

Initially, just seeing the touch points, you know - the "puts" and the "gets" ­ along with MQ administration probably seemed entirely sufficient in order to manage WMQ. A common assumption was that as long as we saw messages coming and going, things were okay. Analogous to Archimedes' principle of water displacement, the health of MQ or any other middleware system could be assumed as long as both sides remained within a reasonable state of balance.

But it didn't take very long at all to see that more was needed. The difficulty was that by the time an imbalance was noticed, too many problems had already been caused; it was obvious that an earlier warning was needed to avert problems or at least correct them sooner. The need to watch things like queue depth, channel status, or any of the other 40 standard events that MQ raises became clear and pretty much commonplace.

If you are still requiring more precise information to detect problems earlier and to see their impact on the applications and business processes, some of you may have had the experience of configuring your own new custom events based on conditions within your unique MQ implementation. If you're really on top of the MQ management game, you've implemented automatic corrective actions, which launch the moment warning signs appear ­ preventing problems before they occur.

The More Things Change...
Despite your success implementing and managing WMQ, and at the risk of stating the obvious, your needs continue to change due to some constant forces:

  • Increasing EAI complexity. Even though most have implemented WMQ for a specific single project, the need to integrate it with other applications and systems continues to grow.
  • The need to do more with less. Due to decreasing resource availability to manage the complexity with smarter tools and processes.
  • The demand for higher levels of service. Not just to track and report on the level of service delivered, but also to resolve problems faster ­ in real time before they impact the business.
  • The requirement to align IT services with business objectives. Delivering and supporting the service specifically to the needs of the business process. Amid a push to run faster and jump higher while carrying heavier loads, we commonly find this mistake: looking over the most practical solutions. What is the most practical solution to these challenges? We've seen that one of the most straightforward and effective things you can do is to expand your monitoring of middleware to include more of the entire application infrastructure.
Achieving An Application Perspective
If WMQ is the only middleware technology you have, you are unique. If you have any enterprise applications (commercial or homegrown) that do not also talk to Oracle, DB2, or SQL Server, then let's just say that you are in a class of your own! Unless we've just described your environment, you also have a wide collection of management and monitoring tools for all of these various platforms and systems ­ each of which lacks the ability to see or do much of anything beyond its own domain.

The solution is to monitor and manage more of the overall application infrastructure. It may sound like the holy grail to track communications across all of your applications and through all of your middleware systems as a contiguous whole, while at the same time correlating the events generated by each platform involved. However, the fact is that it is being done ­ and it's much simpler to do than most expect.

We found that to be the case at Debenhams, one of the United Kingdom's preeminent retailers, and saw an opportunity to improve our performance. Diagnosis of the problem wasn't really the issue ­ all the information we needed was there. But it was just taking too long to put it all together.

We had been doing a good job of sorting out problems within the MQ space for some time. Our problem was that most of our time was spent traipsing through the API layers of our applications that are integrated into MQ in order to find problems. The difficulty was to see the whole picture. Think of it as pushing sausage meat along a sausage casing without any knots in it. We never knew quite where we were.

This is not an isolated problem. Often "blind spots" in the round trip of messages make it difficult to see just where the hang-ups are. Even more troublesome is the cover that these blind spots usually provide for the vendors involved to play the finger-pointing game while your valuable time is being wasted.

Expanding The Myopic MQ Perspective
At Debenhams we didn't have to look far for a solution. From our experience with Nastel AutoPilot, our existing MQ monitoring solution and our dialogue with its vendor, we saw how easily this type of problem could be handled. With all of the facts we could publish with the agents we had already deployed across our AS/400s, we were easily able to get access to all of these other points of information.

When you've got the right monitoring platform, the biggest task can often be defining what data metrics you want to monitor. But, if the tool you are using for monitoring is built on a service-oriented architecture such that it supports open standard interfaces and treats each metric as a portable object - each with its own metadata and methods ­ and if it makes it easy to define your own new events, then you should be able to quickly and easily monitor any data metric with rule and correlation engines, and invoke automated or manual corrective actions.

For example, in Debenhams' case, there are three business-critical applications "glued" together by WMQ. This allows them to see the end-to-end transactional journey involved in correlating facts from the API layers of their application with those they were already getting from MQ.

As you can see in Figure 1, both the applications and their API layers are being monitored ­ all the way into the database that supports the application, and facts are being published along the way. The API layer can initiate a piece of work that can take a good deal of time. For example, if we are processing price changes for a substantial department having tens of thousands of SKUs, it can take up to half an hour to complete. We need to know this beforehand, as opposed to those jobs that are perceived to be long-running and turn out to be only one hundred records or so.

So what was it like implementing all of this monitoring? It took a bit of work, but don't get the idea that it was difficult or impractical. It was relatively straightforward with the tools we used at Debenhams. We've got one guy who's written some generic code on the AS400, which is hooked up to how we publish the facts with our monitoring system. We have several different sorts of things we're collecting. For example, you can easily find out how many records have been received. So that's an easy sort of fact to publish. Some of the other facts that we are getting from the API layer are far more complex. So it's actually working out things like, what processes are active, and if the processes are active, then we need to interrogate the files that they're reading to see how the queue depths are going.

The Payback
The expanded visibility of seeing beyond the bounds of your middleware (when you begin to see more of what is happening within your applications) will always yield better ways to monitor and tune the system and make smarter use of that system, too.

At Debenhams, we estimate that our monitoring saves us at least two to three hours in staff time every day. The most sizable savings is in the time it takes to find problems. For example, our application's vendor used to ask, "How did you know we were doing that?" But they work very closely with us and have been on-site when we were working out the problems inside of MQ. Therefore, they saw the tool we have and learned that we also have quite a lot of MQ knowledge. So now when we think we've got a problem, they don't question it ­ they just go fix it.

During Debenhams' busy time of the year, they run more shifts and the volumes start going up. Chris' group wanted to be able to isolate which particular jobs were causing an issue in the API layers. Once they were identified, the group could then see if they could either throw more resources at them or get them rewritten. As a result of this visibility, we saved thousands of dollars that we would have otherwise wasted just treating the symptoms.

One of the critical processes within Debenhams' operation involved a relatively simple transformation on the source application where the data gets pushed through MQ quite quickly when users gathering the data come off their shifts. Within the target application, however, there is a rather tortuous route. It was at this point in the process that we were finding some problems. It goes through a number of files and a number of rigorous data transformations and eventually it gets into the target database, and then pops out the other side. Not only were we able to easily see what needed to be fixed, but we also extended visibility of the process to our users. Now they see the path of transactions pictorially, and can watch the whole process ­ easily spotting problems without having to understand any of the complexities.

Best Practices
So where do you start with this practical approach to monitoring your application infrastructure in a proactive manner? Here are the ways you can make it happen, and get the visibility to go beyond mere MQ monitoring:

  • Use a monitoring platform that is an SOA designed to operate in real time, and is modular and extensible with support for any open standard.
  • Identify specific data metrics (facts) that will give you better insight into the health of your system. This has a compound effect in that the more you can watch these metrics, the more you'll discover those points of data that are the real determinants. Remember the facts about your environment are all out there; you just need to locate them and pull them together in order to monitor the health of your system.
  • Build simple modular views of these metrics, so that you can correlate them with each other into a hierarchy that represents the complex events within your system. This way you'll see the warning signs of problems before your application goes to production (or before your customers pick up the phone, if it's already there). Most of all, you'll spend much less time chasing false alarms.
  • Share the visibility with other stakeholders of the system, particularly application groups and business units. You'll empower them to solve many of their own problems without bothering you. It also reduces time wasted on needless mundane inquiries, and makes the questions you do get of a higher quality.
  • Implement corrective actions for those recurring conditions where the resolution is consistently the same set of actions.

More Stories By David Mavashev

David Mavashev, CEO of Nastel, is a leading expert on IT infrastructure, middleware and messaging technologies with over 25 years experience architecting systems and solutions. His areas of expertise encompass implementing middleware-centric architectures and the underlying infrastructure monitoring that is fundamental for its optimal performance, as well as tools and technologies for monitoring and managing integrated application processes and performance across the enterprise, and helping companies achieve business agility through effectively aligning IT with business processes in the real-time enterprise.

A successful entrepreneur, David founded Nastel in 1994 and also served as the company's CTO for many years. Prior to that, he was the technical manager of the messaging group at NYNEX, where he architected and managed the implementation of the first commercial transactional messaging product, which now forms the basis for IBM WebSphere MQ (formerly MQ-Series). A pioneer in the early evolution of messaging technologies, David logged many years as an IT consultant working with some of the world's foremost banks and financial institutions.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
SYS-CON Events announced today that Enzu will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY, and the 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Enzu’s mission is to be the leading provider of enterprise cloud solutions worldwide. Enzu enables online businesses to use its IT infrastructure to their competitive ad...
SYS-CON Events announced today that Outscale will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Outscale's technology makes an automated and adaptable Cloud available to businesses, supporting them in the most complex IT projects while controlling their operational aspects. You boost your IT infrastructure's reactivity, with request responses that only take a few seconds.
SYS-CON Events announced today that Peak 10, Inc., a national IT infrastructure and cloud services provider, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Peak 10 provides reliable, tailored data center and network services, cloud and managed services. Its solutions are designed to scale and adapt to customers’ changing business needs, enabling them to lower costs, improve performance and focus intern...
Everywhere we turn in our industry we can find strong opinions about the direction, type and nature of cloud’s impact on computing and business. Another word that is used in every context in our industry is “hybrid.” In his session at 20th Cloud Expo, Alvaro Gonzalez, Director of Technical, Partner and Field Marketing at Peak 10, will use a combination of a few conceptual props and some research recently commissioned by Peak 10 to offer a real-world consideration of how the various categories of...
Five years ago development was seen as a dead-end career, now it’s anything but – with an explosion in mobile and IoT initiatives increasing the demand for skilled engineers. But apart from having a ready supply of great coders, what constitutes true ‘DevOps Royalty’? It’ll be the ability to craft resilient architectures, supportability, security everywhere across the software lifecycle. In his keynote at @DevOpsSummit at 20th Cloud Expo, Jeffrey Scheaffer, GM and SVP, Continuous Delivery Busine...
Detecting internal user threats in the Big Data eco-system is challenging and cumbersome. Many organizations monitor internal usage of the Big Data eco-system using a set of alerts. This is not a scalable process given the increase in the number of alerts with the accelerating growth in data volume and user base. Organizations are increasingly leveraging machine learning to monitor only those data elements that are sensitive and critical, autonomously establish monitoring policies, and to detect...
SYS-CON Events announced today that SoftLayer, an IBM Company, has been named “Gold Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2016, at the Javits Center in New York, New York. SoftLayer, an IBM Company, provides cloud infrastructure as a service from a growing number of data centers and network points of presence around the world. SoftLayer’s customers range from Web startups to global enterprises.
The 21st International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Digital Transformation, Machine Learning and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding busin...
Multiple data types are pouring into IoT deployments. Data is coming in small packages as well as enormous files and data streams of many sizes. Widespread use of mobile devices adds to the total. In this power panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists will look at the tools and environments that are being put to use in IoT deployments, as well as the team skills a modern enterprise IT shop needs to keep things running, get a handle on all this data, and deli...
SYS-CON Events announced today that Systena America will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Systena Group has been in business for various software development and verification in Japan, US, ASEAN, and China by utilizing the knowledge we gained from all types of device development for various industries including smartphones (Android/iOS), wireless communication, security technology and IoT serv...
SYS-CON Events announced today that Loom Systems will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Founded in 2015, Loom Systems delivers an advanced AI solution to predict and prevent problems in the digital business. Loom stands alone in the industry as an AI analysis platform requiring no prior math knowledge from operators, leveraging the existing staff to succeed in the digital era. With offices in S...
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend @CloudExpo | @ThingsExpo, June 6-8, 2017, at the Javits Center in New York City, NY and October 31 - November 2, 2017, Santa Clara Convention Center, CA. Learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
SYS-CON Events announced today that EARP Integration will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. EARP Integration is a passionate software house. Since its inception in 2009 the company successfully delivers smart solutions for cities and factories that start their digital transformation. EARP provides bespoke solutions like, for example, advanced enterprise portals, business intelligence systems an...
We build IoT infrastructure products - when you have to integrate different devices, different systems and cloud you have to build an application to do that but we eliminate the need to build an application. Our products can integrate any device, any system, any cloud regardless of protocol," explained Peter Jung, Chief Product Officer at Pulzze Systems, in this SYS-CON.tv interview at @ThingsExpo, held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA
Existing Big Data solutions are mainly focused on the discovery and analysis of data. The solutions are scalable and highly available but tedious when swapping in and swapping out occurs in disarray and thrashing takes place. The resolution for thrashing through machine learning algorithms and support nomenclature is through simple techniques. Organizations that have been collecting large customer data are increasingly seeing the need to use the data for swapping in and out and thrashing occurs ...
SYS-CON Events announced today that delaPlex will exhibit at SYS-CON's @CloudExpo, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. delaPlex pioneered Software Development as a Service (SDaaS), which provides scalable resources to build, test, and deploy software. It’s a fast and more reliable way to develop a new product or expand your in-house team.
SYS-CON Events announced today that Interoute has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Interoute is the owner operator of Europe's largest network and a global cloud services platform, which encompasses over 70,000 km of lit fiber, 15 data centers, 17 virtual data centers and 33 colocation centers, with connections to 195 additional partner data centers. Our full-service Unifie...
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm.
SYS-CON Events announced today that Progress, a global leader in application development, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Enterprises today are rapidly adopting the cloud, while continuing to retain business-critical/sensitive data inside the firewall. This is creating two separate data silos – one inside the firewall and the other outside the firewall. Cloud ISVs oft...
SYS-CON Events announced today that WineSOFT will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Based in Seoul and Irvine, WineSOFT is an innovative software house focusing on internet infrastructure solutions. The venture started as a bootstrap start-up in 2010 by focusing on making the internet faster and more powerful. WineSOFT’s knowledge is based on the expertise of TCP/IP, VPN, SSL, peer-to-peer, mob...