Across the healthcare industry, IT staffs struggle to keep up with waves of physicians aligning with hospitals. New computers and their users must be cleared to access various parts of a network, and there is a constant struggle to address concerns about regulatory compliance and related data security. The last thing anyone needs is a flood of confusing and largely irrelevant security events that prevent IT managers from focusing on real threats.
This is a challenge Todd Felker, infrastructure and security architect at Southern California’s Torrance Memorial Medical Center (a 401-bed general medical and surgical facility), knows all too well. In the spring of 2013, Felker’s team labored to manage the day-to-day burden of adding users while also dealing with an average of 19 million security events per day generated from their Palo Alto Networks firewall with its intrusion detection and advanced threat detection capabilities.
By Brent Gilroy, Contributing Writer
Torrance Memorial Medical Center employs a subscription security monitoring service to analyze 19 million network intrusion events and accurately distinguish true threats from noise.
![]()
Across the healthcare industry, IT staffs struggle to keep up with waves of physicians aligning with hospitals. New computers and their users must be cleared to access various parts of a network, and there is a constant struggle to address concerns about regulatory compliance and related data security. The last thing anyone needs is a flood of confusing and largely irrelevant security events that prevent IT managers from focusing on real threats.
This is a challenge Todd Felker, infrastructure and security architect at Southern California’s Torrance Memorial Medical Center (a 401-bed general medical and surgical facility), knows all too well. In the spring of 2013, Felker’s team labored to manage the day-to-day burden of adding users while also dealing with an average of 19 million security events per day generated from their Palo Alto Networks firewall with its intrusion detection and advanced threat detection capabilities.
Not A Technology Challenge
Assigned to design a network security program for the hospital, Felker said he knew the problem was not one of outdated technology. The IT team had current technology such as the Palo Alto firewall and Active Administrator, a thirdparty Active Directory reporting tool, but there was no aggregating of logs from Active Directory, the firewall, remote access, and servers.
![]()
“We were not always getting alerted on things that interested us the most. And sometimes the alerts could be ‘noisy’; there were so many that we weren’t always paying attention to the ones that mattered most.”
Compounding the problem was a lack of “event correlation,” meaning a possible intrusion detected at the firewall could not be tied to anything happening in Active Directory. Felker said event correlation tools such as Splunk or HP ArcSight had not been introduced at Torrance because of their cost, time demands, and complexity. “We needed a way to identify the highest risks without spending lots of resources,” Felker said.
Assessing The Potential Solutions, Large Vs. Small
So the search began for a vendor to take some of the analytical burden off of the Torrance staff. Felker started by talking to “big players” in the security services field and found there were huge price differences between them and the smaller, less-recognizable vendors. Then, he said, the realization struck: “If you don’t have anything — with no analytics engine in place to evaluate logs — you don’t need to start big. You don’t need a Ferrari when a Prius might get you pointed in the right direction.”
If a smaller but less expensive vendor could securely retrieve the hospital’s logs and analyze them, Felker reasoned, that would be a sufficient start toward properly categorizing and addressing the torrent of alerts.
Felker discussed the challenge with several vendors and colleagues in the industry before settling on Proficio’s ProSOC, a subscription service that logs, monitors, and analyzes an organization’s security events. Torrance launched a 30-day evaluation of the service, which incorporates SIEM (security information and event management) technology deployed over a cloud-based infrastructure.
Leveraging HP ArcSight and its own security operations center (SOC) personnel, Proficio began sifting through the millions of daily events in firewall logs to determine which met a set of criteria that the Torrance IT team and the vendor jointly decided should make them true concerns.
Felker said a new protocol for analyzing “brute-force attacks” cited on the firewall logs was crafted so that the vast majority of such alerts — those prompted by network users having their passwords out of sync on one or more of their mobile devices — would quickly be seen in a different light from those involving malware or a virus trying to propagate itself to other devices on the network. Over time, the vendor/customer team was able to sort through the logs and, based on patterns and account types, determine whether the matter was simply a user’s password problem or an intruder actually trying to gain access to the network. The ability to correlate events was a key to the initiative’s success, according to Felker.
“For example, if someone’s credentials are compromised and an attacker gains access to the inside of our network, they could use a tool to try to authenticate rapidly to dozens of resources,” he said. “Normally, that kind of authentication would not generate any alarms since the credentials that were compromised are still valid.” However, the Proficio team’s tools and expertise would allow them to tie such access back to a remote-access event seen at the firewall — leading to identification of the attacker.
Felker said the vendor’s analysts also are skilled at identifying account lockouts and failed authentications. For example, if a user changes a password but forgets to modify the Wi-Fi settings on a mobile device, the hospital’s service desk receives a notification about the lockout, prompting contact with the user to make sure the appropriate changes are made.
“Also, we occasionally receive alerts about network equipment that has high CPU utilization, or about a high-availability pair of devices that has gone out of sync. These help us identify network problems sometimes before they have been noticed by our users,” he explained.
Early on, the outsourced analysis of firewall logs showed multiple intrusion attempts — some of them port scans (commonly known as “reconnaissance”), as well as brute-force attacks. With Proficio’s help, Felker said the Torrance IT team was able to identify the true threats and then block access from certain countries while creating exceptions that allowed legitimate exchanges to continue.
A Dramatic Reduction In Events
Today, from the 19 million initial alerts generated daily at the firewall, Felker said his team now knows to focus on a relative handful of potential true threats, as well as essential housekeeping matters for network users. Procedures to properly identify and analyze port scans from abroad have prompted the Torrance IT team to block access by several problem nations and reduce the number of port-scan alerts from two or three weekly to only about two per month.
“We also used to see perhaps two dozen events per month related to malware and spyware,” says Felker. “Our security monitoring service helped us make better use of our firewall’s capabilities and told us that our settings were not as tight as they could have been. Now we see very few of these kinds of alerts.”
Additionally, the hospital’s service desk now deals with perhaps two dozen daily instances of password problems among users. “We are more efficiently blocking the lower-level DNS (domain name system) queries that we know are from bad IP addresses and running reports on our own to track down the devices that make those requests,” adds Felker. “We’re cleaning them up before the spyware becomes a problem.”
Addressing Mobile Is Next Step
The initial 30-day pilot program led to a full-year engagement with Proficio. Felker is now working to extend that relationship. Felker said hospital management is preparing to give Proficio leeway to proactively respond to threats without first having to consult with the facility’s network staff. That will give his team more time to manage network growth while assuring full 24/7 security coverage.
Planning also is under way to bring the hospital’s mobile devices under the consulting team’s analysis. Anomalies will be investigated to determine whether a user’s role has simply changed or if, in fact, malware has been introduced or a device has been stolen.
Felker said some challenges remain in integrating Proficio’s work with the hospital’s legacy IT infrastructure. “We have a few older systems that are keeping us from moving forward with updating operating systems and extending our monitoring without installing and configuring agents — all of which we plan to address in developing our road map for information security.”
Across the healthcare industry, IT staffs struggle to keep up with waves of physicians aligning with hospitals. New computers and their users must be cleared to access various parts of a network, and there is a constant struggle to address concerns about regulatory compliance and related data security. The last thing anyone needs is a flood of confusing and largely irrelevant security events that prevent IT managers from focusing on real threats.
“We were not always getting alerted on things that interested us the most. And sometimes the alerts could be ‘noisy’; there were so many that we weren’t always paying attention to the ones that mattered most.”
