Jump to content

Ransomware attack today for me


markstanco

Recommended Posts

6 hours ago, BearSchlong said:

@Bevo RAID, in various arrays, helps prevent against data loss in the event of one/multiple hard drive failures, it's an entirely separate thing.  With 2 drives you could set up RAID 1, or Mirrored drives.  One could die and you could replace it and the RAID controller would rebuild the failed drive.

Why in the hell do any companies have on-premise servers anymore for ANYTHING?

disclaimer: I sell bandwidth/networks for a living, I get it, not everyone has budget for wire speed fiber connectivity, but it's a hell of a lot simpler to secure data centers/cloud platforms against the kind of bandits that hit Stanco today.

I've seen probably a dozen SMBs get hit by ransomware assholes.

@markstanco did you call your insurance carrier yet?

 

We got hit last summer. They came in through an app that our IT company used for remote login to computers. It ended up being a huge issue for us with a multimillion dollar ask that I’m not going to get into the details of. I will say offsite backup worked great until we needed our data back quickly. We were bandwidth limited and moving terrabytes worth of data from an offsite data center became a significant issue. We run a dual setup now where a mirror image is created all day and if one view down, we can just roll over the mirrored system.

Link to comment
Share on other sites

15 hours ago, Brew said:

We run a dual setup now where a mirror image is created all day and if one view down, we can just roll over the mirrored system.

Implementing a similar option on this currently.  Looks like out DB file with SAP was corrupt, and we have to go back to October 3 2019.  That's re-entering orders, receiving goods, credit memos, etc.  Going to be a total nightmare.  No IDEA what is in stock right now. 

Link to comment
Share on other sites

1 minute ago, markstanco said:

Implementing a similar option on this currently.  Looks like out DB file with SAP was corrupt, and we have to go back to October 3 2019.  That's re-entering orders, receiving goods, credit memos, etc.  Going to be a total nightmare.  No IDEA what is in stock right now. 

shiiiiiit.  Sorry man, that really sucks

  • Like 1
Link to comment
Share on other sites

How do ransomware attacks originate? Is it really as simple as a phishing email? 


Yes.

My wife had to fire someone last year that continually clicked on fake phishing emails sent by their IT dept. all external emails have “EXTERNAL” appended to the front of subject line. She was instructed to just never click on external emails since her job didn’t have external contact. She could not stop. It was stupid fake phishing emails too like “your XBox has shipped! Click here to view your order.”

I guess some people are too dumb to be employed.
  • Like 3
  • Haha 3
Link to comment
Share on other sites

If you’re strapped for cash or a really small shop, Go get Veeam community edition and back up to an external hard drive. Disconnect it from the computer and boom! Air gapped backup. Veeam is pretty damn simple to use too. 

Edited by George
Link to comment
Share on other sites

6 minutes ago, woohorn said:

Isn't the problem that the ransomware is present in the background for weeks/months?

Wouldn't recent backups have it also?

That's why I've mentioned backing up the data as opposed to making "images" of the servers.  You most likely have no idea how long the ransomware has been on your network.  That's also why I am a big proponent of cloud computing now.  Azure or AWS is where most people need to be.

Link to comment
Share on other sites

I have to be very reserved in what i say due to my job.  I'm sure many other surlers in the industry for decades are going to remain reserved in what they say.

People who have money or have grown businesses often believe it is due to some magical powers bestowed on them by God.

This creates a situation where the "decision makers" decide how things work and what they cost, regardless of any ability or experience on their part to support their "decisions".

Then they hire people who provide confirmation bias ("yes men/women").

This paradigm drove IT when it became "democratized" in the 1980s.  A lot of stupid people came out of the oldline mainframe, minicomputer and networking companies.   They thrived in the 2-tier client/server perpetual-license industry.  People retiring in the last 5 years from executive IT careers lived their entire lives avoiding ownership of outcomes in order to protect their kingdom of blinking lights and the drone armies which tended it.

I have to stop now.

Anyone compromised, twitter is your friend.  Your vendors don't want publicity.  I trust the surl can figure it out from there.

  • Like 3
Link to comment
Share on other sites

30 minutes ago, Hagbard Celine said:

I have to be very reserved in what i say due to my job.  I'm sure many other surlers in the industry for decades are going to remain reserved in what they say.

People who have money or have grown businesses often believe it is due to some magical powers bestowed on them by God.

This creates a situation where the "decision makers" decide how things work and what they cost, regardless of any ability or experience on their part to support their "decisions".

Then they hire people who provide confirmation bias ("yes men/women").

This paradigm drove IT when it became "democratized" in the 1980s.  A lot of stupid people came out of the oldline mainframe, minicomputer and networking companies.   They thrived in the 2-tier client/server perpetual-license industry.  People retiring in the last 5 years from executive IT careers lived their entire lives avoiding ownership of outcomes in order to protect their kingdom of blinking lights and the drone armies which tended it.

I have to stop now.

Anyone compromised, twitter is your friend.  Your vendors don't want publicity.  I trust the surl can figure it out from there.

Huh?

Link to comment
Share on other sites

13 hours ago, Hate said:

That's why I've mentioned backing up the data as opposed to making "images" of the servers.  You most likely have no idea how long the ransomware has been on your network.  That's also why I am a big proponent of cloud computing now.  Azure or AWS is where most people need to be.

While moving to the cloud comes with many benefits, it certainly isn't a panacea.  It requires an entirely different skillset to secure cloud than it does your on-premises infrastructure.  And it's not like AWS or Azure do it for you by default, in fact it is the opposite.
How often do you hear about companies being compromised because someone left a production mongodb open to the world, or their AWS keys publicly accessible on GitHub?  All the damn time.
So just saying "move it to the cloud" is dangerous.  By all means make the move to the cloud, but hire some people that know what the fuck they are doing to implement and run it.

 

  • Like 1
Link to comment
Share on other sites

16 minutes ago, KuЯdt said:

While moving to the cloud comes with many benefits, it certainly isn't a panacea.  It requires an entirely different skillset to secure cloud than it does your on-premises infrastructure.  And it's not like AWS or Azure do it for you by default, in fact it is the opposite.
How often do you hear about companies being compromised because someone left a production mongodb open to the world, or their AWS keys publicly accessible on GitHub?  All the damn time.
So just saying "move it to the cloud" is dangerous.  By all means make the move to the cloud, but hire some people that know what the fuck they are doing to implement and run it.

 

So, I have a few 8 person offices that run independently. The main programs are an imaging program and patient management system. I backup just the data but have not tried to use the backup data. I don't have an IT guy unless I run into a problem. I have 1 VPN and use a fortinet UTM device but it is only setup to the extent that Fortinet has helped me. Each office has a server, a Dell R430. What would you recommend?

Link to comment
Share on other sites

14 hours ago, George said:

If you’re strapped for cash or a really small shop, Go get Veeam community edition and back up to an external hard drive. Disconnect it from the computer and boom! Air gapped backup. Veeam is pretty damn simple to use too. 

Going with Datto.

14 hours ago, woohorn said:

Isn't the problem that the ransomware is present in the background for weeks/months?

Wouldn't recent backups have it also?

Numerous images, and any changes (as I am told) will be logged and an email sent. With the Datto, if we go down we can work off the image(s) on their servers until we are live again. 

Link to comment
Share on other sites

While moving to the cloud comes with many benefits, it certainly isn't a panacea.  It requires an entirely different skillset to secure cloud than it does your on-premises infrastructure.  And it's not like AWS or Azure do it for you by default, in fact it is the opposite.
How often do you hear about companies being compromised because someone left a production mongodb open to the world, or their AWS keys publicly accessible on GitHub?  All the damn time.
So just saying "move it to the cloud" is dangerous.  By all means make the move to the cloud, but hire some people that know what the fuck they are doing to implement and run it.
 


I don’t disagree at all. I was making the assumption that most people wouldn’t know how to move to the cloud in their own and thus would engage someone with the required skill set.
Link to comment
Share on other sites

58 minutes ago, KuЯdt said:

While moving to the cloud comes with many benefits, it certainly isn't a panacea.  It requires an entirely different skillset to secure cloud than it does your on-premises infrastructure.  And it's not like AWS or Azure do it for you by default, in fact it is the opposite.
How often do you hear about companies being compromised because someone left a production mongodb open to the world, or their AWS keys publicly accessible on GitHub?  All the damn time.
So just saying "move it to the cloud" is dangerous.  By all means make the move to the cloud, but hire some people that know what the fuck they are doing to implement and run it.

 

This man speaks the truth. Moving to the cloud will likely make you even more vulnerable than an on premise setup if your organization doesn't have a complete understanding of how to  manage a cloud environment. Many traditional IT people with only physical data center experience struggle to comprehend all of the ins and outs of the cloud, especially when it comes to networking. Anyone who migrates, needs to be ready to invest in new support staff that costs more than your average IT staff or just cough up the money to a managed cloud service company. Either way, prepare your anus.

 

Link to comment
Share on other sites

I have to be very reserved in what i say due to my job.  I'm sure many other surlers in the industry for decades are going to remain reserved in what they say.
People who have money or have grown businesses often believe it is due to some magical powers bestowed on them by God.
This creates a situation where the "decision makers" decide how things work and what they cost, regardless of any ability or experience on their part to support their "decisions".
Then they hire people who provide confirmation bias ("yes men/women").
This paradigm drove IT when it became "democratized" in the 1980s.  A lot of stupid people came out of the oldline mainframe, minicomputer and networking companies.   They thrived in the 2-tier client/server perpetual-license industry.  People retiring in the last 5 years from executive IT careers lived their entire lives avoiding ownership of outcomes in order to protect their kingdom of blinking lights and the drone armies which tended it.
I have to stop now.
Anyone compromised, twitter is your friend.  Your vendors don't want publicity.  I trust the surl can figure it out from there.
Bingo
Link to comment
Share on other sites

On 2/19/2020 at 12:56 PM, DFWTexEx said:

Client of ours got hit on Monday.  USD$1.5m ask.  They are in financial services and have a pretty sophisticated IT group (or so we think).   We train and train and train, but still have employees fail our internal tests. 

Damn you porn!

Link to comment
Share on other sites

15 hours ago, Hagbard Celine said:

I have to be very reserved in what i say due to my job.  I'm sure many other surlers in the industry for decades are going to remain reserved in what they say.

People who have money or have grown businesses often believe it is due to some magical powers bestowed on them by God.

This creates a situation where the "decision makers" decide how things work and what they cost, regardless of any ability or experience on their part to support their "decisions".

Then they hire people who provide confirmation bias ("yes men/women").

This paradigm drove IT when it became "democratized" in the 1980s.  A lot of stupid people came out of the oldline mainframe, minicomputer and networking companies.   They thrived in the 2-tier client/server perpetual-license industry.  People retiring in the last 5 years from executive IT careers lived their entire lives avoiding ownership of outcomes in order to protect their kingdom of blinking lights and the drone armies which tended it.

I have to stop now.

Anyone compromised, twitter is your friend.  Your vendors don't want publicity.  I trust the surl can figure it out from there.

Shall we include that they are Pedo's "allegedly" or just that they suck at IT.

Link to comment
Share on other sites

This man speaks the truth. Moving to the cloud will likely make you even more vulnerable than an on premise setup if your organization doesn't have a complete understanding of how to  manage a cloud environment. Many traditional IT people with only physical data center experience struggle to comprehend all of the ins and outs of the cloud, especially when it comes to networking. Anyone who migrates, needs to be ready to invest in new support staff that costs more than your average IT staff or just cough up the money to a managed cloud service company. Either way, prepare your anus.
 
My assumption would be a fully managed cloud solution, either SAAS or a company to manage virtualization.
Link to comment
Share on other sites

On 2/21/2020 at 12:29 PM, RollLeft said:

So this info eliminates American hackers imo.  

Ours was out of one of the former Soviet bloc countries. They had offices leased in the US with equipment in the offices that could not be back traced past that location. It was a very sophisticated system, but luckily we have a solid list of contacts.

Link to comment
Share on other sites

22 hours ago, BearSchlong said:
On 2/21/2020 at 10:41 AM, F250 said:
This man speaks the truth. Moving to the cloud will likely make you even more vulnerable than an on premise setup if your organization doesn't have a complete understanding of how to  manage a cloud environment. Many traditional IT people with only physical data center experience struggle to comprehend all of the ins and outs of the cloud, especially when it comes to networking. Anyone who migrates, needs to be ready to invest in new support staff that costs more than your average IT staff or just cough up the money to a managed cloud service company. Either way, prepare your anus.
 

My assumption would be a fully managed cloud solution, either SAAS or a company to manage virtualization.

Yeah, that's the prepare your anus part.

Link to comment
Share on other sites

Back online with SAP today, and damn, it was a long ass day. Just hit the couch and opened a shiner. Tomorrow will be equal.

I dont know how long it takes a company to get ransomware, with a shitty backup like us (october 3rd) but I would think being down for under a week is pretty decent from what I have read. We still have a lot of work to do.

Going forward, I have learned a lot. For those small business owners, take heed on what I can tell you about ransomware.

Close all RDP ports. I used to log in multiple times a day with my phone to see what was shipping. Not worth it.

Backups: physical backups daily. Swap a drive every day and throw it in your fireproof filing cabinet. I was lazy and did this every few months. Hence only having data from october 3rd and now we have to enter every single order from then, and receivables, and credit memos, etc. Nightmare stuff. About 50 orders a day so, lots of fun.

Going forward..as mentioned drive swaps. Also image cloud backups. And my now backups are with 3 services if you include daily drive swaps.

Oh, and the MOST IMPORTANT is get rid on your server the "administrator " account. I was retarded keeping that one open. That's half of the key to hackers getting into your server. They have the username, just pound away at passwords.

  • Like 3
Link to comment
Share on other sites

I don’t want to pile on but anyone with rdp open to the internet is just asking to get pwned. If you are doing this, you will be compromised at some point, assuming you aren’t already. This is security 101 and why things like VPN exist. 
 

sorry for your hell week. It’s a bitch, and much more sophisticated setups have been hacked. 

Link to comment
Share on other sites

15 hours ago, markstanco said:

Backups: physical backups daily. Swap a drive every day and throw it in your fireproof filing cabinet. I was lazy and did this every few months. Hence only having data from october 3rd and now we have to enter every single order from then, and receivables, and credit memos, etc. Nightmare stuff. About 50 orders a day so, lots of fun.

What's your current backup solution(s)? Depending on the size of your target data, it may be worth it to build a vault to do daily/weekly snapshot backups for the trailing month so you'll have that separate airgapped appliance holding copies of your data at various points in time so you can do point-in-time restores for forensic work in the case of breaches like this one, or just to roll back someone deploying completely fucked business logic that mangled your records.

And I know it's easy to say, but really, running on a cloud platform addresses many of these risks and mitigations directly by decoupling the infrastructure from the runtime environment of your workload. They may pwn that box or fleet of boxes, but you can just roll back in time and restore to wipe away the impact (assuming you have good defense in depth).

Link to comment
Share on other sites

16 hours ago, markstanco said:

Back online with SAP today, and damn, it was a long ass day. Just hit the couch and opened a shiner. Tomorrow will be equal.

I dont know how long it takes a company to get ransomware, with a shitty backup like us (october 3rd) but I would think being down for under a week is pretty decent from what I have read. We still have a lot of work to do.

Going forward, I have learned a lot. For those small business owners, take heed on what I can tell you about ransomware.

Close all RDP ports. I used to log in multiple times a day with my phone to see what was shipping. Not worth it.

Backups: physical backups daily. Swap a drive every day and throw it in your fireproof filing cabinet. I was lazy and did this every few months. Hence only having data from october 3rd and now we have to enter every single order from then, and receivables, and credit memos, etc. Nightmare stuff. About 50 orders a day so, lots of fun.

Going forward..as mentioned drive swaps. Also image cloud backups. And my now backups are with 3 services if you include daily drive swaps.

Oh, and the MOST IMPORTANT is get rid on your server the "administrator " account. I was retarded keeping that one open. That's half of the key to hackers getting into your server. They have the username, just pound away at passwords.

at least you have learned, i have seen entities who have the same exact incident within a year of each other, with the exact same issues that should have been remedied

convenience and speed of deployment always overtakes security

Link to comment
Share on other sites

What's your current backup solution(s)? Depending on the size of your target data, it may be worth it to build a vault to do daily/weekly snapshot backups for the trailing month so you'll have that separate airgapped appliance holding copies of your data at various points in time so you can do point-in-time restores for forensic work in the case of breaches like this one, or just to roll back someone deploying completely fucked business logic that mangled your records.
And I know it's easy to say, but really, running on a cloud platform addresses many of these risks and mitigations directly by decoupling the infrastructure from the runtime environment of your workload. They may pwn that box or fleet of boxes, but you can just roll back in time and restore to wipe away the impact (assuming you have good defense in depth).
I posted a few posts back our solution, I feel good about it. Triple redundancy. One local and two offsite. The local is a daily image and the offsite is real time.
Link to comment
Share on other sites

Our company recently switched from forcing us to change our passwords every quarter or so to requiring us to have nearly a full sentence password. The theory is that with a large number of employees having to change passwords often they will likely be something like orangejuice001, orangejuice002, etc. So having a longer more secure password is better in the long run. I think we have to have something like 15 characters total now with a mixture of capital letters and punctuation thrown in there.

  • Like 1
Link to comment
Share on other sites

37 minutes ago, markstanco said:
1 hour ago, Updawg said:
Change passwords often, keep your shit patched/updated and tell your users not to open or click on shit from email

Good point here. They are all required to ask on clicking on "blue texts" within email to their super. That wasnt the issue here. Brute force was.

Another low hanging fruit is to enable/use 2 factor authentication, ideally via a gemalto or RSA token or a yubikey. It completely mitigates brute force password attacks by requiring physical possession of that token to authenticate as well as the password. 

It's standard practice for secure environments, and most ADs and identity services support it natively nowadays

  • Like 2
Link to comment
Share on other sites

Another low hanging fruit is to enable/use 2 factor authentication, ideally via a gemalto or RSA token or a yubikey. It completely mitigates brute force password attacks by requiring physical possession of that token to authenticate as well as the password. 
It's standard practice for secure environments, and most ADs and identity services support it natively nowadays
Doing similar. If I need to log in, I will have an app that the numeric password changes every 30 seconds.
  • Like 1
Link to comment
Share on other sites

30 minutes ago, markstanco said:

That wasnt the issue here. Brute force was.

 

You might want to give this a read.

https://docs.microsoft.com/en-us/windows/security/threat-protection/security-policy-settings/account-lockout-threshold

It is a simple control and will disrupt a brute force attack. You should also log failed login attempts and send them somewhere when an account is locked so you know this shit is occurring.

The open source version of AlienVault is free. There are others you could get at no charge but you should probably get something.

https://cybersecurity.att.com/products/ossim

There are probably a lot of other things you can do but these are basic controls that would have prevented this occurring. Well blocking TCP 3389 (RDP) would have made a huge difference but these other controls would help out with a lateral attack which could always happen again if they had full access to your environment. Those dudes are known to return via a backdoor after getting paid.

  • Like 1
Link to comment
Share on other sites

 
You might want to give this a read.
https://docs.microsoft.com/en-us/windows/security/threat-protection/security-policy-settings/account-lockout-threshold
It is a simple control and will disrupt a brute force attack. You should also log failed login attempts and send them somewhere when an account is locked so you know this shit is occurring.
The open source version of AlienVault is free. There are others you could get at no charge but you should probably get something.
https://cybersecurity.att.com/products/ossim
There are probably a lot of other things you can do but these are basic controls that would have prevented this occurring. Well blocking TCP 3389 (RDP) would have made a huge difference but these other controls would help out with a lateral attack which could always happen again if they had full access to your environment. Those dudes are known to return via a backdoor after getting paid.
Yea I have learned a lot over the last week.

I'm good now. RDP is closed, only open port is one for the cameras which is outside the network. Cant close 80 and 443 for obvious reasons.
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.



×
×
  • Create New...