Misreply.org

This problem has been bugging me all day. I built virtual machine, running inside Citrix Xenserver, running Microsoft Windows Server 2003 and ISA Server 2006 to act as a backup VPN server on a backup connection.

I realize that this is going to be a pretty long post, so here is a TLDR: ISA box cannot communicate with other servers on the same Citrix XenServer VM host. Shows error 0xc0040031 in logs. TCP Offloading is the culprit. Citrix and Microsoft, please fix your drivers.

Anyway…the long version with error codes and troubleshooting steps and the good jazz that sysadmins will appreciate:

At the office, we have both a Comcast Business line and a Verizon FIOS line as our office internet, and after configuring the Cisco ASA to failover to a backup connection in case of a line failure, I also wanted to have a backup VPN server so users (and myself) could connect to the network if need be. Additionally, if the other VPN server crashed in some way, or was getting overloaded, there was a backup.

The primary ISA (let’s call it ISA01) box runs on top of VMWare ESXi 4, and it’s been running great…I’ve had no complaints about it. I had a second server that runs XenServer that had some free resources, so I decided to put the second VPN box (let’s call it ISA02) on that…free resources and the fact that the ESXi machine only had 2 NICs, but I digress.

So I built the server, added all the routes, and imported the config from the other ISA box. I then wanted to grab some other files from another VM (Let’s call it FileServer that was running on the XenServer box…and nothing.

I ping FileServer… I receive a reply. I try to Remote Desktop from ISA02 to FileServer…nothing. I try FTP, nada. I look over my rules and routes again…everything seems good. I then try to connect FROM FileServer to ISA02….nothing. Ping works, but nothing else. I connect to ISA02 from my home machine, to make sure that I can at least hit our internal servers when I’m on the VPN. Everything seems fine. I can connect to ExchangeServer, I then try to connect to FileServer…nothing.

It turns out…that I could not make ANY TCP connection to and from ISA02 and any other box that ran on the same XenServer. That makes no sense…the traffic never even gets to the switch, it never leaves the Virtual Host…why the hell would I be having issues connecting?

Ok, time for some ninja system and network admin skills…I fire up every logging console I can find, get wireshark loaded up all in this house (on ISA02, actually, but house works too) and get to work.

I see my SYN packet from ISA02 to FileServer…I see FileServer send a SYN, ACK …. I wait for my ACK back from ISA02 to FileServer, but alas it never comes.

I look at the ISA traffic simulator; it tells me that my packet should be allowed through. I’m pretty sure it’s not a firewall rule at this point…I mean, I can connect to ISA02 from ANY other box, on the same ports that I tried from FileServer…but anything that is on the same VM Host is a no go.

XenServer doesn’t really have many configurable options for the virtual switch. ESXi has a pretty robust virtual switching system, but XenServer just lets you create networks, assign them some IPs, and just have them work. Simplicity is good, I always say, but not when I’m trying to figure out WHY THE F* my VMs cant communicate with each other. I check some other things…I have another VM on XenServer, let’s call it FileServer02….FileServer01 and FileServer02 can communicate without any problems…is ISA just broken? Is it a POS product? (Ok, let’s save that discussion for another day)

I turn on every piece of diagnostic logging ISA offers and start looking through them. ISA is telling me that it IS in fact receiving the packet from FileServer01…but it is denying it. It does not say why, though. Usually, when ISA drops a packet, it tells you which policy it is addressing, but this time, I got nothing.

I turn on even more logging…and finally, I get SOME bit of additional information. An error code: 0xc0040031

ISA02, 3/15/2011, 17:12:16, TCP, 10.0.x.y:58732, 10.0.x.x:3389, 10.0.x.y, Internal, Local Host, Denied, 0xc0040031, -, RDP (Terminal Services)

It still tells me nothing. Why are you getting denied little packet, why?

Turns out…XenServer is at fault here. More specifically, this error code is a bad_tcp_checksum error. It seems that the Network Card drivers that come packaged with XenServer don’t always play nice with Microsoft Firewalls…actually, they simply don’t. The fix comes with turning off checksum offloading.

There’s a Microsoft KB article about it: http://support.microsoft.com/kb/904946

After I hit up the registry editor, and added a new DWORD value called DisableTaskOffload in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters , and set it to 1…and rebooted, all was fine.

So please…Microsoft…Citrix…play nicer.

I love reading some of the support tickets we get. At my dayjob (I’m a Network Engineer), I usually monitor a bunch of queues in RT to see if I can help out during some of my downtime (well, at least the downtime that I’m not doing anything important, like reading reddit or shacknews). I will monitor the helpdesk queue and the developers queue in addition to my NOC and Infrastructure queues.

Every once in a while, you’ll find gems in there.

To helpdesk:

Subject: Power outage
Body:
Hello..

The circuit that supplies the warehouse seems to have blown….Would you happen to know which it is so that we can reset it?
You can phone me and walk me through if this is more convenient…

Thanks

And there’s of course the “ASAP OMG IMPORTANT” requests, which I can understand, but when they are like the following, I have no sympathy:

Timestamp: Friday 04:xx pm
–
Hi, I need new user [REDACTED] setup IMMEDIATELY!!! They started monday and need computer, phone, access to [REDACTED], photoshop, and be setup same as me.

Sure, lemme pull that out of my ass immediately, send it back in time to 2 weeks ago so the hardware can actually be ordered, approved, and delivered.

Of course there was also this today:

From: User X
Subject:(no subject)
Body: [blank]

followed 4 hours later by

From: User X
Subject: why?
Body: Why hasnt my previous request been completed yet?

I sometimes feel bad for helpdesk folks.

Virtualization is great…it’s the current hot item in IT. Everyone is doing it; it’s the future of datacenter consolidation.

…it’s my current project.

After a few days of research, some CBTs, and a few meetings, we finally decided to start implementing our grand master plan. Brand new 16 core server with 128GB RAM…check, ESX licensing…check, sanity…check.

After deciding on our virtual network and disk architectures, we finally went to work. VMware is ungodly easy to setup, it’s almost scary.

Next step was getting our VirtualCenter server up and running, and this is when the fun begins (and by fun, I really mean WTF). In the title of the post, I mentioned something about “tricking” around in this particular portion of the setup process, so allow me to explain.

VirtualCenter is basically just a consolidated ESX host management solution. Instead of connecting to each ESX host individually to manage it, VirtualCenter (VC from now on) allows you to manage all your hosts from one area. In VC, you can make ESX clusters, manage Distributed Resources, manage VMotion, and a whole bunch of other goodies. It runs on top of a basic Windows server, uses Active Directory for authentication, and is supposed to make everything nice and easy. Indeed, it does; the interface is great, and manageability is superb…it makes my job all that much easier.

Installation is pretty straightforward, and after clicking “Next” a few times for the VI Client and for the VC server, you think you’re golden. Of course, there are some steps you have to perform as pre-requisites for VC.
First, you need to create a database for it. Well, I’m lying, you don’t particularly HAVE to create a database for it, since it would offer to install MSDE in the same step, but we have a SQL 2005 Cluster…why not take advantage of it? So, created a database, and assigned a domain user myDomain\esxdb as db_owner to it.
Next, create a system DSN to point to that database using NT Authentication. Make sure you use the SQL Native client since it’s SQL 2005, and you’re ready to roll.
Finally, during install, provide the installer with the DSN name, the username, and the password that it will need to connect to the database.

Wait wait wait…so what about this trickery you keep talking about?
Let me get to that!

Next up…installing the Update Manager and the Converter. Go through those steps…and wait…it tells me that the VirtualCenter server is not running? I JUST INSTALLED THE DAMNED THING…on a brand new clean server, no less.

Ok then…I go to the Services menu and try to start it manually….fail. Fail? FAIL?!

Looking through logs (which are in C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs …yea don’t ask why they are there, they just are), I see this:

[2009-04-01 16:20:21.613 'App' 1784 error] ODBC error: (28000) – [Microsoft][SQL Native Client][SQL Server]Login failed for user ‘NT AUTHORITY\ANONYMOUS LOGON’.
[2009-04-01 16:20:21.613 'App' 1784 error] Failed to intialize VMware VirtualCenter. Shutting down…
[2009-04-01 16:20:21.613 'App' 1784 info] Forcing shutdown of VMware VirtualCenter now

Now wait a minute…why the hell is it trying to login to the SQL server as anonymous when I damned clearly provided a username and password for it, and I’m logged into the VC server as a domain admin, so even if it was passing active credentials, it should work. However, it was not.

So ok…what if I go back to the DSN and specify credentials there. Nope, can’t do that…it’s either NT authentication (which does not let you pass credentials), or SQL authentication (which does). Putting domain credentials into the SQL authentication fields result in a whole bunch of fail.

Ok then, next step: create a local SQL user, and use that instead. So I create a local user called esxlocaldb on the SQL server, make that user the db_owner, and re-run the installer for VC. I give it the credentials of the esxlocaldb user (after changing my DSN to use those SQL credentials, obviously), and it comes back and tells me that the credentials are incorrect. I doublecheck everything, do a test in the DSN config window, and everything is dandy…but the VC installer simply wont work.

So now, the trickery begins. In order to install this F-ing thing, I basically had to have both the esxlocaldb user as db_owner AND the domain\esxdb user as db_owner to the ESX database. For the DSN, I gave it the esxlocaldb credentials using SQL authentication, but for the ESX installer, I gave it the domain account. This seemed to work flawlessly.

I think that the installer is just wacked…it uses the DSN credentials to query the DB, but VirtualCenter actually requires a domain user to have access to it. Really, makes no sense to me, but we basically tried everything else. Every other possible combination of database rights and DSN configurations.

Makes no sense to me, but it works now…if anyone has any comments on what we may have missed, I’m all eyes and ears.

The other day, I was tasked with writing a small maintenance script for work. The purpose of the script was to read a list of servers from a text file, go out to each of them, and perform certain operations on folders.

In order to do that, WMI had to be utilized, since that is the only way to perform actions on remote machines from a local workstation (or in this case, a task manager server).

In WMI, just like in most programming languages and frameworks, there are certain special characters that need to be escaped. For example, a simple quote character such as “, needs to be escaped because the compiler will interpret it as something else, such as the beginning or end of a text string. So, in PHP, for example, to assign this is my string to a variable, I would do:

$variable = "this is my string";

Notice how the string is enclosed in quotes. However, if I wanted to say something like And he said, “blah blah”, and then left, I would have to do it like this:

$variable2 = "And he said, \"blah blah\", and then left"; (Yes, in PHP, I could use single quotes to get around that limitation, but that’s not the point)

As you can see…the string had quotes in it, and they had to be preceded by a backslash so that they would not be interpreted as the end of the string. If I did not use them, and instead did:

$variable3 = "And he said, "blah blah", and then left";

the code would error out, because the first quote before blah would mean that I am done assigning my string to $variable3.

Ok…now that all the boring stuff is out of the way, back to VBScript and WMI. Let’s say that I wanted to delete a folder remotely and needed to write a script to do that, and let’s say that folder was C:\Documents and Settings\Luke, because for some reason, I wanted to break my remote machine by deleting my local profile. In WMI, and in almost everything else, a backslash “\”, is a special character, and must be escaped. In this case, any operation I would perform would need my ProfilePath variable to be C:\\Documents and Settings\\Luke.

So my code so far is:


Dim ProfilePath, server
ProfilePath = "C:\\Documents and Settings\\Luke"
server = myRemoteMachine.myDomain
Set serverWMIService = GetObject("winmgmts:" & "{impersonationLevel=impersonate}!\\" & server & "\root\cimv2")
Set folder = serverWMIService.ExecQuery ("Select * From Win32_Directory Where Name = '" & ProfilePath & "'")


So far, I’ve basically asked WMI to go and grab me that folder, and it did, because I had the path properly escaped.

To delete stuff using WMI in the modern flavors of Windows, you can’t just say

folder.Delete

, as you could in Windows 2000, NT and the like. No…instead you have to get a big array, and enumerate every single file and folder inside every single folder. Basically, you have to prepare an entire directory tree, and work from the inside out.

This requires me to query WMI for the subfolders of that folder and the subfolders of that folder and so on.
We can do this pretty easily with another query like this: (note, I already have the objWMIService prepared and all that elsewhere in the script


Set colSubfolders = objWMIService.ExecQuery _
("Associators of {Win32_Directory.Name='" & ProfilePath & "'} " _
& "Where AssocClass = Win32_Subdirectory " _
& "ResultRole = PartComponent")


I ran that code (with all the other stuff I needed it to do), and it failed. Trying to find what failed took a few minutes since VBScript error reporting isn’t exactly perfect. End result was…it was failing on that query to get the subfolders. Through trial and error, I found that if I stripped the extra backslashes from ProfilePath…ie make it C:\Documents and Settings\Luke instead of C:\\Documents and Settings\\Luke, it would work.

Now wait a minute, wasn’t WMI just bitching at me at the beginning of my script that I had to escape the backslash because it was a special character?

All things considered, I have no idea why it chooses to work one way for some queries, and completely differently for others.

End result is: profilePath = Replace(profilePath,"\\","\") …and suck you very much VB

September 19th was my last day of work at (company a). As I looked at my desk, now empty, I felt a strange sensation. It wasn’t regret, happiness, nerves, or nostalgia…but some odd combination of them all.

Most employees at (company) didn’t even know I was leaving that day until I sent my final goodbye email. I told my boss who went to his boss and HR, but for the most part, a limited number of people knew. Robin knew, Aaron knew, and Ryan knew. I kept it low key; no need to introduce more drama into a workplace filled with a plethora of it.

During the previous few weeks, I had been quietly scoping out other employment opportunities. In a chance encounter with a roller hockey buddy, he told me that he was leaving his position at (company b) to work for Lockheed Martin, and that (company b) was looking for a good IT guy. He asked me for my resume, which I happily provided, and waited.

I started at (company a) about four years ago. I was originally a part-time employee assisting Modo with the Broadcast portion of the company’s technological resources. I was still in college, and my internship with Comcast had just ended when Modo approached me asking if I was interested in working for his company. His co-worker had just quit, and he was having a difficult time managing everything on his own. He couldn’t take days off or go on a vacation as he was the only IT resource at corporate. Seeing as my source of income had just come to an end, I agreed to come in for an interview. Leaving out a bunch of superfluous details: I got the job offer fairly quickly and thus began my tenure.

I didn’t dislike (company a), nor was I unhappy while I was there. The degrees of freedom offered by the position and the cool attitude of most of the people made work fun for the most part, as long as there was something to do. Sure, a lot of the time, I was dealing with minuscule user issues that were “urgent emergencies”, but the rest of the time was either free time or project time. In reality, I could do what I wanted, come and go as I please, work from home if I needed to, and my drive was fairly short (minus I-76 traffic, of course).

I didn’t dislike my co-workers, despite some differences in how we did things. There was always a difference in what was considered “best practices” between Core and corporate. For the most part, while discussions would get heated over stupid shit, nobody held a grudge (well..hopefully, I can only speak for myself) despite both sides being stubborn and insistent.

For the most part…life was good, but; it was time to move on. My position never really had any kind of official title. I was the jack of all trades; the everything IT bitch. If it lit up, made beeping noises…hell, if it used electricity, it somehow fell under my domain. I felt that it was time for a change…to move into a position that has more clearly defined goals and responsibilities…something that (company b) offered.

To the peons at corporate: Was cool hanging out and talking to you on a daily basis. You will be missed…and I still owe you donuts.
To the towers peons: Good luck…damn, you’ll probably need it (your words, not mine!)
To the peons in Austin: Don’t do anything I wouldn’t do.