I will start by explaining how this attack was possible. My customer has their own in-house Internet Protocol Private Branch Exchange (IP PBX) that connects their internal IP based telephone extensions to the Public Switched Telephone Network (PSTN) over a combination of traditional telephone lines and some Session Initiation Protocol (SIP) lines. The PBX has a web interface that is designed to allow access for users in the office as well as those outside the office. It was this interface that gave the criminals a way to attack the system.

Let me make it as clear as possible that if you make any service visible on the Internet then it will be attacked and at some point it will be compromised. It really does not matter if it is an on-premises service, like this one, or a cloud based service. If someone can access it then it can be attacked and it is really just a matter of time and luck before it gets hacked. As you will see in this case my customers luck ran out.

The crime scene.

The manufacturer of the device publishes a security guide that contains good advice on how to keep the PBX as secure as possible. All of the advice in the guide has been followed. To help protect the public (Internet) facing service here are some of the things that had done:

• Connections could only be made using secure HTTPS. Insecure HTTP connections are not allowed.
• The service is not available on port 443, the default port used for HTTPS connections. It uses a different port.
• User accounts all have strong passwords.
• Anti-hacking techniques are configured that are designed to detect and block attempts to brute force or guess a password.
• The built in Admin account has a unique and strong password.
• Email alerts are configured to alert me to any failed attempts to gain access to the PBX.

All the things you see listed above are good practice and should be done on any public facing service. There are other things that could also be done but those listed above should have made it very difficult for any hacker to gain access without being detected and blocked.

To illustrate the password strength I will show you the Admin account password that was in use prior to the device being hacked:

2UxU8jb3JpK9

Now be honest, are your passwords as strong as that?

That's upper and lower case letters plus numbers and its twelve characters long. The anti-hacking system was set to trigger after five failed attempts made in 30 minutes. It should be all but impossible for anyone to brute force or guess that password in less than six attempts in 30 minutes. Yet it looks like more than one hacker managed to get into the system without detection.

There are a lot of websites that offer to test the strength of passwords and estimate how long it would take for the password to be broken. To see how strong the old Admin password was I tried six of them. Here are the results:

1. Password test result is about 400 years.
2. Password test result is about 3 THOUSAND YEARS to crack your password
3. 10 thousand years to crack this password.
4. Time to crack your password: 5 billion years
5. It would take a computer 10 thousand years to crack your password
6. Strength Evaluation: Good

If you look at those results you can understand why I would not be confident about some of them and one in particular. It is a good illustration of why you should not trust information you find on the web no matter how professional the site may look.

I think I can safely say that the old Admin password (2UxU8jb3JpK9) was definitely not weak in any way. Even the most conservative result suggests that it would take hundreds of years for anyone to crack it using a normal PC. That begs the question of how the hackers managed to break into my customers PBX.

If you want to test out the strength of your passwords I will put the details of the two websites I think are the best at the end of this article. Please don’t use actual passwords but try out ones that are similar to those you use.

Zero Day Exploits.

The first thing is that they used what is called a ‘Zero Day’ exploit. It is called a zero day exploit because only the people exploiting the security vulnerability know about it. That means that you have zero days to patch it because it is being actively exploited in the wild. The only thing that will save you is pure luck as hopefully you will be able to get a security patch (fix) applied before any hacker can find your system.

Zero day exploits are really horrible, and are often the thing that keeps people responsible for cyber security awake at night. As they can remain unknown, sometimes for long periods, no anti-virus or anti-malware systems can detect them. If you become aware of a zero day exploit you must apply any patches or updates immediately they become available or you risk becoming a victim.

Unfortunately, there is a big problem when it comes to fixing zero day exploits.

Manufacturers and software publishers have to be made aware of the existence of a new security vulnerability. If it is actively being exploited in the wild then the clock will be ticking and it becomes a race as the hackers attack vulnerable systems whilst a fix is developed, tested and then published. The longer it takes to get a fix out the greater the chance that more systems will be hacked.

That is only part of the problem.

You might wonder why the manufacturers and software publishers don’t issue warnings as soon as they are made aware of a new zero day exploit. This might seem a sensible thing to do and yet it is not done for a very good reason. If warnings are issued then many more hackers will become aware of the existence of a vulnerability and that will lead to a lot more systems being compromised before a fix can be released.

Good old lady luck becomes a factor.

The manufacturers and software publishers hope that by keeping the vulnerability secret until they release a fix, the number of successful attacks can be kept to a minimum. You may be lucky and avoid being hacked or you may be one of the unlucky ones that gets attacked before the fix is released.

My customer was one of the unlucky ones.

The Attack Unfolds.

The PBX had been in place, and remained completely secure, for about six months. This all changed on the 29th March 2020 at 02:05. Attacks like this will often take place late at night as the attackers hope that nobody will be monitoring the system in real time. The first attack took place in the early hours of a Monday so was not detected until I checked my email logs later that morning.

This attack was the first one on this PBX and it failed due to it being detected by the password failure detection so all the indications were that everything remained secure. Often when attacks like this begin I will see a stream of emails and then I know that the attacker(s) are making a determined effort to get into a system. In this case it all went quiet so there was no reason to be concerned.

The table below shows what happened next:



A very similar IP address to that used in the March attack is used on the 12th April at 01:05 and again it is detected by the password failure detection system. Then just one minute later the attacker gets into the system. You can see that the log file shows Operation Successful indicating that they have now logged into the PBX with full Administrator rights.

How?

The security vulnerability was caused by defective code in the PBX that allowed the attacker to use something called a SQL injection to get the PBX to allow them to log in with the password. The attacker probably entered a specific string of characters into the browsers address bar, or possibly the password box, and this allowed them to obtain the passwords and gain full Admin access to the PBX.

It would not have mattered if the Admin password contained 24 characters and used every type of character to make it incredibly strong as this attack appears to have given them the passwords. Oh dear.

Because the system was satisfied that the correct security credentials had been used it would not need to send out any alerts. So I remained completely unaware that the PBX had now been well and truly compromised. Nor had I received anything from the manufacturer to tell me that a security vulnerability existed. At this point I doubt if they knew about it but if they did then they were not going to alert anyone to the fact.

That first successful attack, like the failed one in March, came from an IP address that appears to be located in Palestine or Israel. I have looked at quite a few attacks but I cannot recall ever seeing one coming from an address located in Palestine although I have seen a few from IP addresses located in Israel. Given their close proximity it is understandable that many IP address location services cannot give a definitive location for the addresses. Either way the attacks came from that part of the world.

The next attack on the 13th April at 20:01 appears to fail and also comes from Palestine.

At 20:46 on the same day another attack comes in but this time using a completely different IP address. I have traced this IP address to being located in Germany. This time things get bad because the log shows that changes are being made to the system itself. Notice how the attack from Germany reboots the PBX to get all the changes accepted.

No email alerts are reaching me so I assume that one of the attackers has made changes to prevent detection as well as to be able to exploit the PBX. I remain unaware of any issue with the device even though it is now completely compromised and being controlled by others without my knowledge.

The next attack begins on the 14th March at 06:17 and appears to be coming from an IP address located in Palestine or Egypt. Once again the proximity of the two countries can explain why the IP address location services cannot give a precise location. Things are now going from bad to worse as this attack shows files being stolen from the system.

The final attack on this device begins on 14th March at 08:29 and is coming from yet another IP address located in Palestine. More changes are made and the PBX is again rebooted. From the log file it looks like the attacker did not configure their changes correctly the first time so they edit them and finally reboot the system.

Cleaning up the mess.

On the 15th April the manufacturer sends out an email recommending an urgent upgrade of the firmware on the PBX. Whenever I get an email telling me to urgently apply an update I know that it must be related to a zero day exploit.

I can only guess that no other attacks took place because the people carrying them out realised that their exploit had finally been discovered. It may also be related to fact that on the 14th April the SIP line service to the PBX dropped. This may have happened if the service provider spotted unusual activity or suspected that the PBX had been hacked.

When I logged into the PBX the first thing I did was to check the Call Detail Record (CDR) to see if any calls had been made by the hackers. Making expensive long distance calls is one of the main reasons hackers break into telephone systems. Another is to try and avoid their calls being monitored or intercepted. Perhaps unsurprisingly the CDR had been deleted so I have no idea how many calls got made through the PBX whilst it was compromised.

Various other things had been deleted as well so the attackers wanted to cover their tracks as much as possible. I updated the firmware and changed the default name for the Admin account as well as adding an even stronger password. For now I have also removed any public access to the PBX whilst I think about how I can provide its users with even more secure access.

Normally I would be using a VPN connection but not everyone wants that sort of solution and sometimes it can be overkill. This PBX only needs very limited access from outside from one primary location so an easier solution will be to get that location using a fixed (Static) IP address. Then I can configure the firewall on the office router to only allow access from that fixed IP address.

I also got the SIP line password changed so that the attackers could not access that service. Remember that they had stolen various files so in time I’m going to have to change every single password on the system to make certain that the stolen data is of no use at all.

My final task was to go through every possible configuration option to find changes and then reset them. When I can finally visit the office, after the Covid-19 lock-down ends, I will probably do a full factory reset and then apply a fresh configuration as that is the only way I can be certain that I have cleaned out anything done by the hackers.

I created a support ticket with the manufacturer as I had hoped to get some advice about how best to deal with the changes made by the hackers but the response I got back was not helpful. It even told me that the CDR cannot be restored although I think this is incorrect. The PBX has a built in backup function and this does include the ability to backup the CDR.

As I have the PBX configured to perform a full backup every night I don’t anticipate any problems restoring the deleted CDR records. Of course they probably won’t show me the calls made during the period that the PBX was compromised but at least my customer will have all their call data back in place.

Final thoughts.

No system or service can ever be 100% secure and this attack is just one example of why that is the case. All cloud based services are just as vulnerable to attacks like this and in most cases more vulnerable as they often cannot be restricted to access from specific IP addresses. Far too many of them don’t use Two Factor Authentication (2FA) either.

In this case all the standard, and recommended, security measures were in place and it appeared that the PBX was well protected by them. If I have one criticism of the device it would be that it lacked any form of 2FA. Relying on user names and passwords alone is simply not enough to prevent successful attacks as the countless hacks, including this one, demonstrate.

Zero day exploits, like the one used in this attack, are becoming much more common because criminals will pay large sums to get them. One such zero day vulnerability in the Zoom video conferencing system, as used by the UK Prime Minister, is currently on sale for $500,000. Crime should not pay but with cyber crime it certainly does.

Free cyber security advice.

Never assume that any of your systems and services are secure. Start your security planning on the basis that any of your systems can, and will, be compromised. Decide how you will handle such an event and how you can best mitigate any impact. Only then will you have a plan, and hopefully good security practices, that will make it far less of a trauma when the inevitable finally happens.

It will also make you think long and hard about how you store your data and how you control access to it. By all means aim to have the most secure systems you can afford but never forget that a system that is 100% secure simply does not exist.

If you want proof of that just take a look at a couple of IT news stories from this week.

IBM, one of the very biggest names in IT, sell a software security product called IBM Data Risk Manager (DRM) to large corporations across the globe, that is intended to help identify security vulnerabilities in the customers systems. This week a security researcher revealed that DRM itself contains four very serious security vulnerabilities. To make matters worse the researcher had reported the vulnerabilities to IBM but they chose to ignore them until he was forced to go public with the details.

Cognizant is a large American IT business that provides on-premises and cloud-hosted IT services and consultancy for customers in areas such as banking, health care, and manufacturing. It is ranked in the Fortune 500, so any large-scale attack on its systems is potentially serious. Cognizant has been hit by the Maze ransomware and is now warning customers that they may have also been affected by this attack.

This is serious because the Maze ransomware doesn’t just encrypt any data it finds on Windows computers like most ransomware. Maze also sends copies of the data to the attackers and they can use this stolen information to force a victim to pay up. If a victim refuses to pay then the stolen data gets leaked. For Cognizant and its customers this is a nightmare situation.

If two of the worlds biggest IT businesses cannot keep their own systems secure then the rest of us have no chance of doing it. So stop thinking that you can and start planning on the basis that you cannot. It really will be for the best.

Password Strength Websites.

Try the following two sites to test out passwords:

https://www.roboform.com/how-secure-is-my-password

This is my favourite as it doesn’t fall into the trap of trying to calculate any length of time to crack the password. It just gives you a reliable indication of the true strength of a password.

https://password.kaspersky.com/

Of all the gimmicky websites that attempt to measure password strength by indicating how long it may tack to crack this one gave the shortest time. If you use this website please take any result with a pinch of salt.

Do NOT put actual passwords into these sites. Only enter dummy passwords or old passwords that you no longer use. Make sure that the sort of passwords you are using are always given at least a strong rating. Anything less indicates the use of weak passwords.