How To Simplify SQL Server Management And Save Hundreds Of Thousands Of Dollars

Health IT Outcomes had the chance to sit down and speak with Michael York, Senior Systems Engineer, IT Services for Asante which serves over 580,000 people throughout Southern Oregon and Northern California. York discussed specific IT and business challenges his organization was facing — especially surrounding its SQL Server environment — that necessitated the adoption of a new and innovative solution.

Q: Like most healthcare organizations, I would imagine you are feeling the pressure from the explosion in digital data, rising software licensing costs, and increased government mandates such as HIPAA?

York: Absolutely. When I joined Asante in 1999, we had one hospital and about a dozen remote clinics, with an IT staff of about 20 people managing 30 servers. Today, we have over 800 servers spread across three hospitals and over 50 outlaying clinics, supported by almost 200 IT professionals. The advances in technology and what it has enabled us to do over these past two decades is incredible. But this growth didn’t come without some growing pains. Most notably, we experienced rampant sprawl and exploding licensing costs.

In the mid-2000’s, we began virtualizing much of our datacenter environment. The goal was to consolidate, improve mobility, and achieve higher availability for business critical applications — as well as gain better use of IT resources and engineering time. For instance, from a cost perspective, we were trying to minimize the seemingly inescapable four-to-five-year hardware refresh issues. We started with smaller projects and low-hanging fruit and continued along, virtualizing as much as we could that made sense.

Q: And, this did the trick?

York: We enjoyed a lot wins with this approach, but there were still some workloads we couldn’t collapse. We found our Microsoft SQL Server applications — most of which required multiple servers for high availability — were often too large to be virtualized. Because these apps were business critical, vendors were building clusters and large, over-the-top SQL Servers with lots of redundancy and high availability. Prior to the consolidation, we had about 250 to 300 line-of-business apps. This led to an explosion of physical and virtual SQL Servers across the organization. Now, we had to deal with the SQL Server sprawl.

Projects kept coming in, and continued to increase in size and complexity, with multiple servers for databases and application, as well as web and interface servers. We wanted to virtualize these systems using an application-focused approach, rather than spinning up large hardware projects with multiple hypervisor nodes and related overhead. However, we needed a way to do that for I/O-intensive SQL Server workloads.

Really, we had two issues. One was small-to-mid-sized databases coming in with their own overbuilt solutions. At the other end, we had these very large databases doing a lot of transactions. Even though not all of these were processing at 100 percent, the vendor’s philosophy was that we needed the biggest and best for future growth and high availability. That meant large clusters of servers, and we’ve had nothing but headaches with Microsoft clusters. The problem is that clusters are meant to protect hardware, but if a software piece fails and the other pieces don’t know about it, the system fails. Ultimately, we feel if there’s any way to avoid using Microsoft clusters, we will.

Q: And this eventually culminated in your company searching for a new solution?

York: Yes. I knew it just didn’t make sense to make all the special provisions needed for SQL Server to exist within VMware. It often comes down to I/O contention — and then, you are dedicating a costly ESX host or two for running SQL workloads. With SQL Server’s I/O profiles we were either wasting too much space in the wrong kinds of storage or putting too much I/O workload into a VM. We didn’t want to get into building specialized VM farms.

I wanted an approach that would allow my team to break apart vendor-recommended solutions much like what we were doing with VMware, an alternative to clusters for SQL Server applications. Early on of course, we didn’t understand we could scale back and use SQL Server more efficiently and effectively by safely stacking instances on the same box — in-turn getting more out of our hardware.

The first approach we adopted was with HP’s PolyServe. Unfortunately — or fortunately, depending on how you look at it — PolyServe was deemed end-of-life. We had been satisfied with PolyServe, but the end-of-life announcement forced us to look for and evaluate viable alternatives. This was when we discovered DH2i’s DxEnterprise. After learning more about it, and giving it a try, we were eager to implement the more advanced capabilities and tools it offered. Like our virtualization initiative, we started by looking for low-hanging fruit that could be encapsulated and managed by DH2i’s vhost technology. This included applications that were data mining multiple SQL Servers running on disparate servers that we could collapse into a better framework for high availability. As we experienced success and saw the potential, we looked for more candidates to run as instances and discovered the new approach was applicable across our compute environment. To date, we have consolidated more than 40 instances across five hosts.

Now we can place a database in SQL Server and immediately deliver high availability for our internal customers. As anyone can imagine, this is critical in healthcare. The built-in tools in DxEnterprise make step-and-repeat on a larger scale a whole lot easier than anything else we have used. Plus, it puts the workload right next to the disk where it needs to be — and not separated by virtualization layers.

Not only did the approach offer dramatic improvements in consolidation and availability, we were also able to ease management and reduce operational and lifecycle headaches, such as patching.

Q: When you started with this project, your goal was to spin up services more quickly and manage them more efficiently — as well as reduce SQL Server sprawl. But from what I’ve heard, this turned into a perfect disaster recovery (DR) solution as well?

York: Absolutely. The software enables easy subnet failover for our multiple locations that we can implement at any time, on any infrastructure. In other words, we can manage DR as an extension of what we are already doing. It is the best way that I know of for doing stretched SQL clusters.

Q: We haven’t really hit the financial implications of this deployment in detail. Can you talk about how you’ve been able to reduce licensing costs?

York: It used to be that you didn’t have any other options – there was a price that came along with running SQL Server, and you just had to bite the bullet. Then, SQL moved to a CPU-based licensing model. In-turn, we bought a SQL Server license and put it on the biggest server we could, and charged it back to the project(s). Then, when licensing changed to a core-based model, we realized we were going to have to pay over $400,000 to true-up our existing licensing model. But incredibly, we were able to consolidate to the point where we only had to pay about $20,000 in true-up costs. We are also now saving on the monthly cost for Enterprise Agreement support — which increases as a percentage of licensing costs, and comes up for renewal every two years. Much of this would have been charge-backs to our internal clients — so they are pretty happy too. Certainly budget they can find other useful ways to spend.

Q: To summarize, what advice would you offer to others in the healthcare space, and even across other industries, facing similar issues?

York: When I talk with colleagues, I am still baffled by how many have not traveled down this path yet. They are still paying through the nose, and with less capabilities then we have. So my advice is to not get caught up in convention and make sure you are leveraging the best tools available to you. We investigated a technology that we had never worked with, and for hundreds of thousands of dollars less than before, we are able to deliver infrastructure and spin-up new services faster and more efficiently. This new HA approach paired with new IT initiatives is enabling us to increase and enhance our capabilities and services — both to our external clients, as well as for internal business purposes. Healthcare, like almost every industry, is facing increasing competitive, regulatory and/or legal pressures. Using instance-level high availability technology has given us the edge we needed to come out on top.