Microsoft ISA 2006, Citrix XenServer … please play nice
This problem has been bugging me all day. I built virtual machine, running inside Citrix Xenserver, running Microsoft Windows Server 2003 and ISA Server 2006 to act as a backup VPN server on a backup connection.
I realize that this is going to be a pretty long post, so here is a TLDR: ISA box cannot communicate with other servers on the same Citrix XenServer VM host. Shows error 0xc0040031 in logs. TCP Offloading is the culprit. Citrix and Microsoft, please fix your drivers.
Anyway…the long version with error codes and troubleshooting steps and the good jazz that sysadmins will appreciate:
At the office, we have both a Comcast Business line and a Verizon FIOS line as our office internet, and after configuring the Cisco ASA to failover to a backup connection in case of a line failure, I also wanted to have a backup VPN server so users (and myself) could connect to the network if need be. Additionally, if the other VPN server crashed in some way, or was getting overloaded, there was a backup.
The primary ISA (let’s call it ISA01) box runs on top of VMWare ESXi 4, and it’s been running great…I’ve had no complaints about it. I had a second server that runs XenServer that had some free resources, so I decided to put the second VPN box (let’s call it ISA02) on that…free resources and the fact that the ESXi machine only had 2 NICs, but I digress.
So I built the server, added all the routes, and imported the config from the other ISA box. I then wanted to grab some other files from another VM (Let’s call it FileServer that was running on the XenServer box…and nothing.
I ping FileServer… I receive a reply. I try to Remote Desktop from ISA02 to FileServer…nothing. I try FTP, nada. I look over my rules and routes again…everything seems good. I then try to connect FROM FileServer to ISA02….nothing. Ping works, but nothing else. I connect to ISA02 from my home machine, to make sure that I can at least hit our internal servers when I’m on the VPN. Everything seems fine. I can connect to ExchangeServer, I then try to connect to FileServer…nothing.
It turns out…that I could not make ANY TCP connection to and from ISA02 and any other box that ran on the same XenServer. That makes no sense…the traffic never even gets to the switch, it never leaves the Virtual Host…why the hell would I be having issues connecting?
Ok, time for some ninja system and network admin skills…I fire up every logging console I can find, get wireshark loaded up all in this house (on ISA02, actually, but house works too) and get to work.
I see my SYN packet from ISA02 to FileServer…I see FileServer send a SYN, ACK …. I wait for my ACK back from ISA02 to FileServer, but alas it never comes.
I look at the ISA traffic simulator; it tells me that my packet should be allowed through. I’m pretty sure it’s not a firewall rule at this point…I mean, I can connect to ISA02 from ANY other box, on the same ports that I tried from FileServer…but anything that is on the same VM Host is a no go.
XenServer doesn’t really have many configurable options for the virtual switch. ESXi has a pretty robust virtual switching system, but XenServer just lets you create networks, assign them some IPs, and just have them work. Simplicity is good, I always say, but not when I’m trying to figure out WHY THE F* my VMs cant communicate with each other. I check some other things…I have another VM on XenServer, let’s call it FileServer02….FileServer01 and FileServer02 can communicate without any problems…is ISA just broken? Is it a POS product? (Ok, let’s save that discussion for another day)
I turn on every piece of diagnostic logging ISA offers and start looking through them. ISA is telling me that it IS in fact receiving the packet from FileServer01…but it is denying it. It does not say why, though. Usually, when ISA drops a packet, it tells you which policy it is addressing, but this time, I got nothing.
I turn on even more logging…and finally, I get SOME bit of additional information. An error code: 0xc0040031
ISA02, 3/15/2011, 17:12:16, TCP, 10.0.x.y:58732, 10.0.x.x:3389, 10.0.x.y, Internal, Local Host, Denied, 0xc0040031, -, RDP (Terminal Services)
It still tells me nothing. Why are you getting denied little packet, why?
Turns out…XenServer is at fault here. More specifically, this error code is a bad_tcp_checksum error. It seems that the Network Card drivers that come packaged with XenServer don’t always play nice with Microsoft Firewalls…actually, they simply don’t. The fix comes with turning off checksum offloading.
There’s a Microsoft KB article about it: http://support.microsoft.com/kb/904946
After I hit up the registry editor, and added a new DWORD value called DisableTaskOffload in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters , and set it to 1…and rebooted, all was fine.
So please…Microsoft…Citrix…play nicer.
