Misreply.org

Some of the most satisfying moments at the office come from being able to solve problems I’ve never seen before, and could not find any information on the internet about. It’s especially satisfying when it involves equipment you’ve never used before.

The problem started out with the launch of one of our new websites. The site has an admin portal that was built to accommodate reporting and CMS pieces, and was nice enough to use domain authentication. All of that…is irrelevant.

The relevant part was that we launched the site, load balanced it, and thought everything was working just fine. Turns out, it wasnt. Browsing anonymously had no issues, but it seemed like once you attempted to login, the site would greet you with a “This page has been reset” message. Was it code? Was it the webserver? Was it the load balancer?

We temporarily moved the site off the load balancer internally and began troubleshooting. With us going directly to the webserver the problem did not appear, so we started looking at the ACE. The serverfarms were correct, the sticky policy was correct, the probes showed the site as up, the policy-map looked good, all the other sites on the load balancer worked just fine. I was at a loss, I couldn’t find anything that would indicate a problem. What now?

I stand behind the statement that Wireshark is probably one of the most useful and versatile tools in the universe. It’s not just for looking at packets, it can help you figure out a plethora of problems that are seemingly unrelated to the traffic flow (I have another post regarding that in a bit; also involving load balancers, go figure). I started looking at packet captures between my machine and the site (going through the ACE), and I saw something strange. After the initial GET /, there was a SYN,RST packet coming back. Well, was it coming from the load balancer or the server? I fired up wireshark on another machine at the Datacenter to see if I could see that same packet on the other side of the ACE. I did not.

I started thinking about why the ACE would be terminating that connection. It seemed to make no sense. It worked SOME of the time, why not all the time? All the other sites worked fine, whether you were logged in or anonymous. It seemingly wasn’t the server either. I dug deeper into the wireshark capture, and I decided to follow the TCP stream of the web request for easier viewing. It wasn’t clear right away, but for some reason I started looking closely at the data. I noticed that the header data seemed strange. After logging in, my local machine sent the header data, and it was XXXX bytes in size (I dont remember the number anymore), however, when it was coming back from the webserver/ACE, right before the RST packet, it was XXXX – 200 bytes. The header value coming back was smaller, and then the site was reset. I looked at the value of the header, and saw that I was sending a giant string of characters for the .Net viewstate and token, but when it was coming back, it seemed to be cut off after YYYY bytes.

Turns out, the ACE has a default limit for HTTP header values. This particular site was throwing a lot of garbage in the headers and cookies for its own tracking and tokenization scheme, and it was bigger than the ACE’s default limit.

Quick fix: set header-maxparse-length bytes 8192 … slam bam, it worked.

Unfortunately, I have since moved jobs, and forgot that I started this draft almost a year ago, so I no longer have the wireshark captures that I used to fix this.

This is only pertinent to fellow shackers, but Shacknews member stonedonkey has installed his API on one of my servers.

Now shackers with mobile devices and those using LAMP have another API server to use.

http://shackapi.misreply.org is the address. iPhone users already have the option to change API servers, shackdroid has an update coming, and LAMP should be pretty easy as well. See http://www.shackwiki.com for more shack goodness.

This problem has been bugging me all day. I built virtual machine, running inside Citrix Xenserver, running Microsoft Windows Server 2003 and ISA Server 2006 to act as a backup VPN server on a backup connection.

I realize that this is going to be a pretty long post, so here is a TLDR: ISA box cannot communicate with other servers on the same Citrix XenServer VM host. Shows error 0xc0040031 in logs. TCP Offloading is the culprit. Citrix and Microsoft, please fix your drivers.

Anyway…the long version with error codes and troubleshooting steps and the good jazz that sysadmins will appreciate:

At the office, we have both a Comcast Business line and a Verizon FIOS line as our office internet, and after configuring the Cisco ASA to failover to a backup connection in case of a line failure, I also wanted to have a backup VPN server so users (and myself) could connect to the network if need be. Additionally, if the other VPN server crashed in some way, or was getting overloaded, there was a backup.

The primary ISA (let’s call it ISA01) box runs on top of VMWare ESXi 4, and it’s been running great…I’ve had no complaints about it. I had a second server that runs XenServer that had some free resources, so I decided to put the second VPN box (let’s call it ISA02) on that…free resources and the fact that the ESXi machine only had 2 NICs, but I digress.

So I built the server, added all the routes, and imported the config from the other ISA box. I then wanted to grab some other files from another VM (Let’s call it FileServer that was running on the XenServer box…and nothing.

I ping FileServer… I receive a reply. I try to Remote Desktop from ISA02 to FileServer…nothing. I try FTP, nada. I look over my rules and routes again…everything seems good. I then try to connect FROM FileServer to ISA02….nothing. Ping works, but nothing else. I connect to ISA02 from my home machine, to make sure that I can at least hit our internal servers when I’m on the VPN. Everything seems fine. I can connect to ExchangeServer, I then try to connect to FileServer…nothing.

It turns out…that I could not make ANY TCP connection to and from ISA02 and any other box that ran on the same XenServer. That makes no sense…the traffic never even gets to the switch, it never leaves the Virtual Host…why the hell would I be having issues connecting?

Ok, time for some ninja system and network admin skills…I fire up every logging console I can find, get wireshark loaded up all in this house (on ISA02, actually, but house works too) and get to work.

I see my SYN packet from ISA02 to FileServer…I see FileServer send a SYN, ACK …. I wait for my ACK back from ISA02 to FileServer, but alas it never comes.

I look at the ISA traffic simulator; it tells me that my packet should be allowed through. I’m pretty sure it’s not a firewall rule at this point…I mean, I can connect to ISA02 from ANY other box, on the same ports that I tried from FileServer…but anything that is on the same VM Host is a no go.

XenServer doesn’t really have many configurable options for the virtual switch. ESXi has a pretty robust virtual switching system, but XenServer just lets you create networks, assign them some IPs, and just have them work. Simplicity is good, I always say, but not when I’m trying to figure out WHY THE F* my VMs cant communicate with each other. I check some other things…I have another VM on XenServer, let’s call it FileServer02….FileServer01 and FileServer02 can communicate without any problems…is ISA just broken? Is it a POS product? (Ok, let’s save that discussion for another day)

I turn on every piece of diagnostic logging ISA offers and start looking through them. ISA is telling me that it IS in fact receiving the packet from FileServer01…but it is denying it. It does not say why, though. Usually, when ISA drops a packet, it tells you which policy it is addressing, but this time, I got nothing.

I turn on even more logging…and finally, I get SOME bit of additional information. An error code: 0xc0040031

ISA02, 3/15/2011, 17:12:16, TCP, 10.0.x.y:58732, 10.0.x.x:3389, 10.0.x.y, Internal, Local Host, Denied, 0xc0040031, -, RDP (Terminal Services)

It still tells me nothing. Why are you getting denied little packet, why?

Turns out…XenServer is at fault here. More specifically, this error code is a bad_tcp_checksum error. It seems that the Network Card drivers that come packaged with XenServer don’t always play nice with Microsoft Firewalls…actually, they simply don’t. The fix comes with turning off checksum offloading.

There’s a Microsoft KB article about it: http://support.microsoft.com/kb/904946

After I hit up the registry editor, and added a new DWORD value called DisableTaskOffload in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters , and set it to 1…and rebooted, all was fine.

So please…Microsoft…Citrix…play nicer.

“But mommy, everyone else is doing it!”

I’m usually not a fan of New Years resolutions, as they all tend to be the same for everyone, are predictable, and the majority of them fail miserably. Just a quick perusing of news sites, blogs, and friends’ pages, I see a plethora of resolutions about quitting smoking, going to the gym, and other cliches.

My opinion is that one should not have to wait until January 1st of any given year to force themselves to improve their lifestyle, because forcing a big change simply guarantees failure. On the other hand, a new year is a convenient, stable, and measurable starting point to make some small changes that don’t require a drastic lifestyle change.

Without further ado, here is my list of stuff I want to accomplish this year:

1. Read more – I got a Kindle, so I might as well use it. I figure I’ll at least catch up on some of the classics I either missed while growing up, or simply ignored. Just a few minutes ago, I went on amazon and grabbed a bunch of these books; things like “The Art of War”, “The Invisible Man”, “A Tale of Two Cities”, “Crime and Punishment”, and a bunch of other classics by authors such as Dickens, Verne, Twain, Carroll, Swift, and Wells. I figure I should be able to do a book a week or so.
2. Take more pictures – I started to use my camera more during the summer, and I’d like to continue that trend. Even though there might not be that many interesting places to shoot in the area, I’d at least like to make a conscious effort to get some interesting photos, whether they be common things from interesting angles, new places, or people. Maybe I’ll do some more NYC trips this year and take some more model shots, or perhaps ride my bike into the city and take pictures of things usually made impossible by traffic or parking situations.
3. Teach my parents how to use a computer – This one will probably be the toughest of them all, but I’m going to give it my best. It will be an exercise in patience, repetition, and more patience. My dad knows a little bit, but mom is completely new to the concept. Hopefully, they will at least see it as an easier method to communicate and get information, rather than something that makes it difficult. I figure that once they grasp the basics of how everything works together, and the standards that most applications follow, it will all become easier. Currently on the list: How to browse in Firefox…the differences between the different mouse pointers, and how they change the behavior of the click. Something we take for granted, but to someone that has never used it before…a bit of a challenge to keep everything together.
4. Expand my musical horizons – No matter how much new music I grab or new playlists I make in grooveshark or pandora, it seems that I always tend to go with the tried and true musical selections. Looking at my music library on my computer, the play counts are skewed… 80% of the music has never even been played, whereas the other 20% have constant playtime. I need to relieve myself of the predisposition that most new music is crap, and just go with it. Get some earphones at work, and just do it.
5. Stop spending money on useless shit (ie: build up a more substantial savings) – A pretty easy and blunt one…stop buying shit I don’t necessarily need. Sure, I like gadgets as much as the next guy, but I think I have enough crap to entertain me for quite a while. Just because I have random cash here or there does not mean that I automatically have to spend it. Also included here: don’t go out to lunch so often…it’s amazing how much gets spent on that.

So there it is…a fairly simple list. Nothing life changing, but a few little things that I want to do this year. I figure that these should be easily accomplished without driving me crazy, or requiring too much commitment to be viable.

I love reading some of the support tickets we get. At my dayjob (I’m a Network Engineer), I usually monitor a bunch of queues in RT to see if I can help out during some of my downtime (well, at least the downtime that I’m not doing anything important, like reading reddit or shacknews). I will monitor the helpdesk queue and the developers queue in addition to my NOC and Infrastructure queues.

Every once in a while, you’ll find gems in there.

To helpdesk:

Subject: Power outage
Body:
Hello..

The circuit that supplies the warehouse seems to have blown….Would you happen to know which it is so that we can reset it?
You can phone me and walk me through if this is more convenient…

Thanks

And there’s of course the “ASAP OMG IMPORTANT” requests, which I can understand, but when they are like the following, I have no sympathy:

Timestamp: Friday 04:xx pm
–
Hi, I need new user [REDACTED] setup IMMEDIATELY!!! They started monday and need computer, phone, access to [REDACTED], photoshop, and be setup same as me.

Sure, lemme pull that out of my ass immediately, send it back in time to 2 weeks ago so the hardware can actually be ordered, approved, and delivered.

Of course there was also this today:

From: User X
Subject:(no subject)
Body: [blank]

followed 4 hours later by

From: User X
Subject: why?
Body: Why hasnt my previous request been completed yet?

I sometimes feel bad for helpdesk folks.

Let’s be honest, there’s really not much to do at 3am when you’re bored but watch infomercials, porn, or random things on your computer. Having watched that week’s new episode of Top Gear, I decided that instead of boring infomercials, I’d rather watch more Top Gear, and revisited one of my favorite segments of all time.

Season 10, Episode 01 of Top Gear closed with one of the greatest films of all time: one about trying to find the greatest driving road….in the world. A road that would challenge both car and driver. A mostly empty road, with scenic views, tight turns, fast straights, tunnels….the works. They basically settled on the fact that this great driving road would be found somewhere in Europe, close to the Alps. They took three of the greatest sports cars money could buy (except for James May, who brought a race car with no air conditioning, no windows, almost solid suspension…you get the idea): A Lamborghini Gallardo Superleggera, a Porsche GT3 RS, and an Aston Martin race spec Vantage.

Basically, they did what I really really really really want to do, and if I ever win the lottery, I’m buying myself a sporty car, and flying to Europe.

Anyway, onto the actual interesting bits: the roads themselves, which, with Top Gear’s excellent cinematography, looked even more fantastic than usual. The first road, the benchmark, was the Col de Turini, in France. Usually a stage for the World Rally Championship, it is one of only a few things the French have going for them (I kid, I kid….ok maybe I don’t).

The second road that they highlighted was the San Bernadino Pass: a mountain road in the Swiss Alps between Hinterrhein and Misox. This scenic stretch of road features smooth roads, sweeping turns, and a long tunnel where the sounds of the car could really come alive.

The highlight of the trip; the best driving road in the world was the road from Davos to Stelvio via Bormio. This road is as scenic as it is challenging. It had everything they could dream of: quick turns, sweeping curves, smooth roads, breathtaking views, fast bits, tight bits. This is a road that could make you actually become one with your car. Sure, you could drive it in a Volvo as a side road, but pushing a car to the limit on this road would be the ultimate rush. After this road, driving to work would seem even more bland than it does now, if that is even possible…unless you live in Davos and work in Stelvio…in which case, hire me. To put the cherry on top, the Stelvio Pass at the end of this road was another piece of driving heaven. Being one of the highest paved roads in the Swiss Alps, it is also one of the most challenging roads to drive on. It has hairpin turns throughout the entire pass, limited visibility, and nothing to stop you from falling down the mountain if you get too brave and too stupid.

Seriously, go watch that episode if you enjoy driving even a little bit, and marvel at the greatness that is….the best driving road in the world.

I think I’ll play the lottery tomorrow.

Yields marinade for 6 chicken leg/thigh combo or 8-12 oz steaks.

The Wet Ingredients

• 8 oz Orange Juice
• 4 oz Myers dark rum
• 3 oz Soy Sauce
• 1/4 cup molasses
• juice of 2 limes
• 1 tablespoon vinegar
• 4 oz olive oil

The Fresh Ingredients

• 2 tablespoons of chopped garlic
• 1 large chopped sweet onion
• 2 tablespoons of fresh thyme
• 6 chopped scotch bonnet peppers
• 1 bunch of green onion

The Dry Spices

• 2 tablespoons ground allspice
• 1 tablespoon ground pepper
• 1/4 cup brown sugar
• 2 tablespoon kosher salt
• 1 tablespoon smoked paprika
• 1 tablespoon red pepper flakes

1. Combine the wet ingredients in a large container.

2. Add the fresh ingredients to the wet ingredients.

3. Add spices to the previous combination…stir.

4. Marinate beef or chicken for a good 24 hours before cooking. Grilling is best.

Source: PA Ren Faire food festival

Start with 1/2 pound of peeled and de-veined shrimp

Saute in a pan with 3 oz of olive oil until they start to turn white (not clear)

Add 5 oz of orange poppy seed dressing to the pan. Stir and let continue cooking

Remove from heat when meat is white throughout

Plate over lettuce and finish with orange zest and fresh cut orange slices

Source: PA Ren Faire food festival

In a tall glass, combine:

6 oz Pineapple juice
1 tablespoon brown sugar
1 dash liquid almond extract

Stir to dissolve sugar

Fill glass with ice, then add:

1.5 oz Myers dark rum
1.5 oz Bacardi white rum
1.5 oz Captain Morgan spiced rum

Fill with 4 oz sours mix

Shake…garnish with pineapple slice and cherry

Source: PA Ren Faire food festival

Hello dear readers (all 1 or 2 of you that are left due to my lack of updates), I am alive! Contrary to popular belief, I did not fall down a ditch; never to come out. I am here, and have a brand new post. As for not updating very often, I apologize. You ever have one of those days when you wake up and have to go to work…then when you are done, you simply don’t want to do anything at all? Or a day that is just busy, and when you are done, you want to do nothing more than flop on the couch and try to beat a vegetable at lack of action? Yea…well, I’ve had one or two of those….or forty, in a row.

So, my faithful Microsoft Intellipoint 5 button mouse finally gave out. After months of hiccups, missed clicks and tracking errors…it finally bit the dust and became so unreliable that I had to give it the boot. I used the opportunity to finally go wireless, and minimize some of the clutter on my desk, minimize some wires, and extend the reach of the mouse. I settled on a Logitech MX Revolution. Rechargeable battery – check, wireless – check, many buttons – check.

The SetPoint software, however, left a bit to be desired. Before I go any further, let me explain to you my love of the middle mouse button (or third button). In Firefox, it opens links in a new tab, closes tabs, and activates the scrolling mechanism. In TF2, it was my reload button. Logitech, however, decided that the middle mouse button simply wasn’t important enough to be an available option for button mappings.

The MX Revolution is pretty cool. It has your normal right/left buttons, a 4 way wheel (which also can act as a button…more on that later), a button right below the scroll wheel, 2 buttons on the side by the thumb, and another toggle switch of sorts under those. The Revo also has the nice option of having a clutch for the scroll wheel, which allows for some super fast and super smooth scrolling, without the clicky clicks. This behavior can either be toggled by the scrolling speed, or by pressing the scroll wheel in (using it as a button, as stated earlier). In the Logitech software, you can set that mouse wheel to act as your middle button instead. This is all well and good, since I love my middle button, but the click action is very hard; it wasn’t made to act as a button that is pressed multiple times per minute. That being beside the point, I like having it as the option to switch between scrolling modes. This leaves the button right under it, which is actually in perfect position to act as the middle button. It’s in between the right and left buttons, easy to press in, and easy to reach. Perfect…let’s set that as the middle/3rd button in the Logitech software!

Oh wait…you can’t. What? Logitech, did you REALLY not allow that as an option? I mean…really? I can use it to search stuff, flip documents, auto scroll, invoke Winamp, be a double click, close stuff…pretty much any function you can think of EXCEPT middle button. I mean, a nutless monkey could have coded the software better to allow that as an option. Sure, you can install an alternate driver, but I don’t recommend it since it breaks other functionality (such as using the side toggle as win+tab in vista), and generally doesn’t work as well.

So Logitech: this means war! A war that I intend to win. And I did win. See, (ok folks, this is where it gets geeky, so be warned) Logitech stores all the button configurations in an XML file on your hard drive. In Vista, that file is stored in C:\users\YOUR USERNAME\AppData\Roaming\Logitech\SetPoint and is called user.xml .

If you open up that user.xml file, you will see that all the button configs are pretty straightforward XML format…all you have to do is replace the correct section with a correctly formatted, modified version. So…I made that little ‘search’ button my middle button by changing the button 6 configuration to the following: (No, I have no idea why it’s button number 6, but labeled as button number 4…it doesn’t make sense, it’s probably left over from some copy/paste I did when banging around the XML, but it works, so sue me)

(I’d like to have posted the text directly, but wordpress thought the XML was supposed to get parsed, even though it was in between code tags…oh well)

Voila! I now have a middle mouse button! So so nice.