Investigating a Payments Breach
Verizon Investigator on What Forensics Can RevealInvestigating a payments breach is no easy task. So many entities are connected in the payments chain, tracing the data leak can be time-consuming and complex, says Verizon's Dave Ostertag.
Pointing to the breach at processor Global Payments, Ostertag says the investigation should follow certain steps, and organizations across the board can learn lessons from the actions of Global.
Investigations into payments breaches are complex, says Ostertag, a breach investigator in Verizon's Investigative Response Unit. Large processors often acquire smaller processes, so tracing the breach can be complicated and disjointed.
"With all of that together, it makes for a large environment to investigate," says Ostertag, in an interview with BankInfoSecurity's Tracy Kitten (transcript below).
During the investigation, banking institutions and other organizations must determine:
- Where the data was breached;
- How the hackers got in;
- How long the hackers had access to the data; and
- How they got the data out of the network, database or system.
Completing the investigation is a time-consuming process, which can take months. "Even if you happen to get in and look at the right logs right away, you still have a very large environment to review," Ostertag says.
During this interview, Ostertag discusses:
- The forensics investigation challenges in the wake of a breach;
- Why organizations continue to get breached, despite increased investments in new technology and systems; and
- How forensics investigations are helping the financial industry improve security within the payments chain.
Ostertag is global investigations manager for Verizon's Investigative Response unit.
TRACY KITTEN: Why does the payments industry continue to be plagued by breaches despite efforts to enhance PCI compliance in a post-Heartland world?
DAVE OSTERTAG: I think the simple answer is that there's a lot of money involved. These cases, particular large cases like Global Payments, involve a large amount of payment card data, which subsequently give criminals access to a lot of money and there continues to be weaknesses and vulnerabilities in systems that are exploited by criminals to access that data and subsequently use. As long as the bad guys can get access to that data, they're going to continue trying to get access.
An Investigation Scenario
KITTEN: I know that you cannot specifically talk about the Global incident. But in a similar incident and in a similar investigation scenario, what might the investigation look like?
OSTERTAG: In these cases, the investigation is a lot more complex than most investigations just because of the environment. As a normal part of business inside the payment processing side of the business - just like the banking industry - large processors acquire smaller processors, and in doing that and acquiring multiple other businesses, you acquire their networks when you acquire the companies. Just like a bank environment or a financial industry's environment, you end up with a lot of legacy systems that are connected together and that makes it difficult to enforce uniform security over the entire environment.
Besides all of the different ways that the payment cards are processed, all the different methods and channels that the authorization requests come through, you have telephone, Internet-based and hard-framed relay coming in. You have e-commerce businesses. You have businesses at a flea market they use, or a cellular telephone with a card swipe device on it. So you have multiple channels coming in. You have to service them in multiple different ways. The data comes in different formats. The data has to be converted into a single format to be transmitted to the payment card industry networks, and you have all of these disparate legacy systems that are attached together and have to communicate to each other and ultimately end up in that uniform format.
With all of that together, it makes for a large environment to investigate and determine what's the most likely place that the data was that was taken, if it was taken, how the bad guy may have come in and what path they took while they were inside a network. You may have an attack vector, an entry into a payment processor's environment in a completely different network segment and the bad guys may have spent months exploring and moving around inside the network from one thing to another until they ultimately get to the data. Well, these are sometimes very complicated and have a lot of systems involved that have to be examined and logs that have to be examined to determine what the bad guys did, how they got in, where they got to, how they got the data and how they got the data out. As well as, is there anything else that we have to be concerned about? These are extremely large environments.
Length of an Investigation
KITTEN: Based on the complexity that you just described, this next question is probably going to be difficult to answer, because it sounds like it could vary greatly depending on the processor that was breached. But how long could an investigation into a payments breach like this take?
OSTERTAG: It could take quite a while. Even if you're lucky, even if you happen to get in and you happen to look at the right logs right away and one of the first systems that you image and you examine has the evidence in there, you still have a very large environment to make sure that there's not that somewhere else inside that network. In today's times, in the more advanced types of attacks that we see, a lot of times the bad guys use more than one technique, sometimes more than one team, going in. Sometimes there are more than one hacker going into a network and they share information about what they're doing. To do a complete job, to make sure that there's no further contamination of the environment, there's no further badness in the environment, even if you get lucky in the best of the situations and find evidence of the attack right away and what the bad guy did, you still have to spend a lot of time making sure that's the only problem inside of the network. Sometimes these could take months. Very often they take months to complete the investigation.
Investigation Variances, Based on the Breach
KITTEN: When it comes to different types of breach investigations, how does this type of investigation - one that involves payments processors - differ from other types of breach investigations?
OSTERTAG: A payment processor investigation is different from others in that in many other investigations that we do, specifically in the payment card industry, there's only one type of data involved. It could be the e-commerce environment where you're only dealing with payment account numbers and expiration dates, or it could be a swipe transaction in that data process.
Inside a payment processor, typically you have every possible way the payment card data is transmitted and a lot of times that comes from the payment card brands looking at the fraud transactions and determining from the fraud transactions what type of data may have been compromised. Instead of just looking at a specific set of account numbers, you have to take a look at the types of fraud transactions, patterns of those fraud transactions to determine what's the most logical type of data that was taken and then when the forensics investigations team interviews the potentially breached company, you have to ask them where would this information exist in your environment. What's the path of this information in your environment?
It differs in that you first have to determine where in the network this type of data might exist and in some cases the payment card branch determines there are multiple types of data involved. In those cases, do you have a common point inside this network? Will all of these types of data exist or are we looking at multiple segments or multiple portions of the network where each individual type of data may have been taken? They get complicated at times.
KITTEN: How do the different or disparate pieces of the puzzle, such as merchants, processors, acquirers, affect a forensics investigation into a payments breach?
OSTERTAG: It's just like we talked about with the legacy systems. We have similar problems with merchants and acquirers and other processors. Typically in a large processor case like this, there are multiple acquirers. You might have five, six, seven, 10 different acquirers that the processor sends data to. They aggregate the transaction information from the variety of merchants that then send them to the acquirers or the acquiring bank for those merchants. You have multiple acquirers. You have lots of merchants, and those merchants could come through in a variety of different channels or nodes into the processor.
As we talk about in the processing world, big processors acquire little processors, so when they acquire those little processors they also acquire their merchants that come with those smaller processors. So you have lots of merchants that come in from a variety of different directions and that's important a lot of times in these investigations, trying to pinpoint where to focus your investigation. Information as far as the different merchants and acquirers can be vital, and if the affected accounts all belong to specific merchants or belong to specific acquirers, that may help you isolate within the network where your problem might be. If one acquirer is affected and others aren't or three acquirers are and seven aren't, there might be unique common points for those acquirers in the system that might give you an idea of where you're going.
But all of that information together can be really confusing, and so you can focus in on exactly what's unique to these transactions and accounts that are not unique to the rest of universe of other accounts. It can really get complicated at times trying to understand just what the different flows are - what the commonalities are in these investigations.
Getting Down to Details
KITTEN: How much can a forensics investigation narrow down the details surrounding a processing or payments chain breach? And it sounds like it really does depend on a number of different scenarios. I guess not every investigation is equal.
OSTERTAG: Exactly. They're all different. The key piece of information that might help you focus and find exactly where the problem is at is different in every one of these. It's similar to a homicide investigation. You can't discount. You can't get tunnel vision in one scenario. You have to keep all possibilities open and eliminate, and as you eliminate you start to then begin to focus on where the problem might be. A lot of times in investigations where you get a lead and you check a lead and then the place will be a dead-end, you have to go someplace else.
On these investigations, we typically get to a point where there are no dead-ends. All of a sudden, we're going down a specific lead or going down a specific scenario, and you find that everything you check verifies and verifies and verifies. As that happens, you get more and more focused on exactly where the problem is. It's taking a large, large data set of information and trying to narrow it down into the smaller data sets as possible to focus on the investigation. At the same time, you're trying to eliminate as large of portions of the network and possibilities as possible to get to that point where you're focused.
KITTEN: Would it be easy to determine, for instance, how data was compromised, whether that be at rest or in motion, and which server or servers were hit?
OSTERTAG: That's very difficult. It's not really easy to determine where it's at, unless you find specifically what happened. You find the specific malware that took the data - you're going to know exactly what happened. If it's a RAM scraper that operates in memory, if it's some type of packet sniffer that grabs the data or if it's just simply grabbing a file or a database, sometimes it's easy. Sometimes you find all the data that was in the fraud account in a specific file. A huge tool that obviously the investigators use is taking what's known to be compromised and trying to find one location inside a network that might contain just that information. A lot of times the forensic investigators will use the account information that was used in fraudulent transactions as a tool to try and find where the data was taken from. In some instances, that fraud data that was used helps us isolate where the data may have come from inside the network. If an account number and an expiration date and a CVV2 or CVC2 is used in a transaction, there might be a very small set of places inside the network where that data might exist.
A lot of times the data itself helps us find where it's at. Sometimes we just get lucky, or sometimes it waits for months until we find evidence of exactly what happened that will determine how the data actually got compromised.
A lot of times we look at the fraud data and at the fraud transactions. The pattern of fraud transactions help us identify how the data was compromised. There are certain malware that will sit inactive for a period of time and then open up and collect data, and then close up for a few more weeks, and then open up again in a few days, and then open up again.
When we look at fraud analysis and we look at when those cards may have been used, we know the certain malware act in a certain way, so if we look at those fraud transactions and see a pattern that might match the ways that a certain piece of malware works, we might be able to then understand better where to look and focus on exactly what we would be looking for. There are a lot of variables and a lot of expertise in a lot of different areas that go into trying to determine how the data was compromised, based on those factors of exactly what data is used for fraud transactions and the patterns that were used, as well as those things we talked about - what merchants are involved, what acquirers are involved and things like that to help us identify where in the network it existed and then what would work, what type of methodology a bad guy would use, to access that data.
KITTEN: Are there certain things after a breach investigation that we may never know?
OSTERTAG: Sometimes we will never know if it's circumstantial evidence of exactly what happened, but maybe not evidence itself. Some things we never know because anti-forensics was used. It's just like a crime scene - a physical crime scene. We talked about a murder investigation - just like someone would try to clean up a murder scene, the bad guy's going to clean up a data breach scene, trying to change date stamps and things like that to throw the investigator off, delete data. Sometimes they will completely re-image hard drives or erase hard drives so there's no evidence there, so sometimes logs don't exist. We don't have the logs to understand the first date that the bad guy got in or the last date that he got in, so in a lot of these cases there typically is data that we never know.
A lot of times we will rely on other information to give us that if it doesn't exist. An example is perhaps we don't have the logs showing us the first access date and the last access date that the bad guy got in, but we can look at fraud data to give us a pretty good date range based on the legitimate use of those accounts at the merchants that we've identified. In these cases, a lot of times there's information we will never know.
Room for Improvement
KITTEN: Before we close, I wanted to ask about improvements that you see the payments industry making as a result of information that's revealed or shared after a forensics investigation?
OSTERTAG: That's a constant thing. I know in two weeks I'm going to the PCI Security Council PFI meeting in the U.K. where they talk to the forensic investigation teams about what we find, and take the information from our cases and determine commonalities or gaps in security that then become new requirements. ... For PCI, we took a look at patterns that we saw over and over again and gaps in security that we saw over and over again, and used those to improve the requirements to help people protect themselves.
That still goes on today. Vulnerabilities that we see the bad guys use, they become new requirements. We're getting better. We've seen a migration. When I first started these investigations ten years ago, they were simple smash and grabs, just like if someone would walk up to a department store, throw a brick through the window and steal a bunch of merchandise and run. The first data cases were simple sequel injections against big databases of payment card data. and the bad guys would grab thousands or hundreds of thousands of account information out of that database.
Through time, we've gotten better. We encrypt that data that's sitting in the database. We put access control around it, protecting that information from being taken in the database, although the bad guys moved at the point to using packet sniffers. It was encrypted while it was at rest so the bad guys starting stealing it in transit with packet sniffers. As we get better, the bad guys adapt and then we seem to be getting much better. At the same time, there are still a lot of people that don't follow the standards or don't meet the standards and you know that's easy pickings- someone who doesn't follow the requirements and has poor security in their system.