ryv's blog: solutions

Showing posts with label solutions. Show all posts

Monday, 30 January 2017

The unintentional DoS

DoS - Denial of Service

Over the weekend it was very hot here - 39C over both days and air conditioning was being pushed pretty hard. My team and I had two unrelated, but linked situations evolve that could have hit us with a DoS. You see, we have a network attached storage device (NAS) that had a fan failure. While this NAS has redundant fans in it, one wasn't enough to keep the temperatures under the 55C warning threshold. So it started to complain....

Over the course of the 48 hour weekend, this NAS sent out over three and a half thousand emails! 3500+ emails! All to our logging email addresses, which then sent it out to the members of the team. 5 team members, 3500+ emails.... 17,500 emails being sent and received. That's a lot email in a short time. Most email servers will handle that and ours certainly did. Fortunately too we use G-Suite (Google Apps new fancy name) and so the volume of mail wasn't an issue.

What became an issue though - and this did have an effect on our phones and mobile devices picking up email - was that another network device - a disaster recovery server - also suffered heat stress from failed air conditioning. A sparky had unplugged our monitoring device to charge his tools and hadn't plugged it back in, so we had no idea what was happening (this was Sunday afternoon). When the A/C failed, the server turned off and the replication servers started to complain - four of them, every 30 seconds.... Over 12 hours those servers alerted our logging email address over 5,000 coming in and going back out - another 25,000 emails hitting phones plus the other emails as well.

Having had the discussion with clients about hosted email solutions versus onsite solutions, there are definite advantages to having huge servers managing your email. So if you don't have a cloud based solution, how can you mitigate this risk?

Defence in depth is a great place to start. Organise to get a mail exchanger - MXGuardDog or something similar. Westnet used to do one too. Get your MX records updated to punch mail through that. These then relay to av-relay.domainname.com. Configure your firewall to only accept emails from the IPs at MXGuardDog (for example) and drop everything else (or at least grey list it so it gets dropped and the sending server can try other MX records).

This way you can temporarily control the flow without having your ADSL or NBN connection getting flogged to death.

Configure your internal mailer to hold emails for this kind of thing - to recognise a flood of email and trickle it out where possible. The risk is that legitimate email (which these emails both are and aren't) will get lost in the flow. It's better than having your upload link fully saturated though (which will kill all internet connectivity).

DoS are bad. DDoS are worse. Let's try to avoid doing it to ourselves!

Tuesday, 11 October 2016

Fixing Windows 7 update problems

Tuesday, 2 August 2016

Hyper-V copy/paste of Virtual Machine folder security problems

Have you seen this in your error logs?:

The Virtual Machines configuration 6ED5794F-DD19-46D3-8121-0880FEB592AE at 'D:\Hyper-V VM Data\VM NAME' is no longer accessible: General access denied error (0x80070005).

With Event ID: 4096 and much sadness because your VMs won't boot?

And did you move the folders that your VMs are living in to a new location but it should just work?! Well the chances are the security permissions on the files are wrong. I'm not talking about "Administrators" having all rights, but specific Virtual Machine rights.

I recently upgraded my home server with a new array of disks - a jump from 500GB of storage to 2TB. It's a modest increase, but this is a hyper-v server, not a NAS. I copied off the VMs from the old disk to an external drive and then copied them back. Oh noes! Two of my virtual Linux servers declined to start again - bastards!

So it was to the inter webs, and here, gentle reader, I hope you can find answers. If you check the permissions on the .xml configuration file for the VM you might see just the regular stuff on it. What you should also see is an entry for "NT VIRTUAL MACHINE\6ED5794F-DD19-46D3-8121-0880FEB592AE" there as well. Now in this instance the "6ED5794F-DD19-46D3-8121-0880FEB592AE" is specific to my virtual machine - you will have a different one.

It's important to note that ID. You can get it from your .xml configuration file and various other places (I find the .xml to be the easiest).

Now to fix this, open an elevated command prompt and type in the following:

icicles "path to .vhd or .xml" /grant "NT VIRTUAL MACHINE\virtual machine ID":(F) and hit enter.

You should have a return of"

processed file: "path to .vhd or .xml"
Successfully processed 1 files; Failed processing 0 files

It looks like this:

Do this for both the .vhdx (as in this case) and the .xml file. Once you go back to the Hyper-V management console, your VM should work. Otherwise, something else is b0rked and you'll need to chase it in Event Viewer!

I hope this saves someone else the half an hour of Googling that I did this morning to sort it out. Best of luck!

Thursday, 18 February 2016

Lenovo E540 won't start or boot - solved

Recently a client called saying their fairly new E540 wouldn't turn on or respond at all. The red light that makes up the "i" in ThinkPad would light up but that was the extent of the functionality.

I tried removing the battery and booting off but no dice. The computer was totally unresponsive. It turns out there is an issue with Lenovo for Small Business software that causes this.

To fix it, remove the BIOS battery - this is located under the compartment to the left in this photo:

Stick the battery back in and give it the berries. The laptop should boot and complain about the time and date. Set those and start the laptop. Log in, go to Uninstall Programs and ditch the Lenovo for Small Business. Once that's done, it's probably a good idea to update the BIOS. Try not to screw that up by killing power or anything mid upgrade. The recovery process is incredibly annoying and not guaranteed to work.

A reboot and you should be right to go. Enjoy.

ryv's blog