App-V, iSCSI, VSS, and trouble!

My GPO has the App-V client being pushed to ALL computers, on the principal that even on servers, I would rather have the App-V client and a virtualized PDF reader than have to actually install third-party software. That got me in some trouble with my brand new x1600 from HP. This nifty device, which I have carved up into two LUNs of 2TB each, is a Windows Storage Server 2008 R2 box that I'm using for Hyper-V VM storage, Exchange, SQL, and SharePoint. The hardware is rock solid, and performance under test loads has been stellar.

I'm no fan of Windows Storage Server, mind you. I'm on the record as preferring the open-source OpenFiler for on-the-cheap iSCSI software when using commodity hardware or the HP 4000-series SAN for the all-in-one package (aka LeftHand Networks) when needs are bigger, and then scaling up from there to the HP 6000 series (aka 3Par). Still, it sits in the middle of those first two categories, which is where my needs lay, so it's perfect for our not-small-not-big environment.

After setting up and carving out the two LUNs, it came time to add VHDs as drives for the iSCSI Target devices. The result of that attempt was a series of errors and event log entries... most of which were quite frustrating in their obscurity. I wound up flummoxed as to how on earth such a thing would be possible on a brand new device that was only recently even powered up! Reboots and power-cycles didn't help... blowing away the LUNs and recreating didn't help... It was time to pull out the old troubleshooting hat, and in treading back through all of the changes that could have happened, I figured it was most likely an errant patch. Firmly on my path, I set about hitting up the control panel for a quick uninstall of those patches when I noticed that App-V client.

I put two and two together - an App-V client makes a phantom drive (Q:). This means it must interact with volume shadow services, or else any backup would be doubling up on that Q: space, since it's a logical drive within the root drive. It's also access-denied to all, even admins. Hmmm... seems logical... and after uninstalling, it was logical. Everything worked again, just like new. My cluster is up, running, and happy with 4TB of storage split among two 2TB volumes, with hyper-v guests and other services sharing happily.

That damn GPO bit me. It's proven its use on servers in the past, as I don't need to manage the Adobe updates on them (why, HP, do you install that crap on the x1600 out of the box? Why?). In this case though, the App-V client apparently alters the VSS permissions, resulting in errors on VHD creation. I'll be interested to dig deeper as to why this happened and what permissions are altered, but in the meantime... I'm glad it was quick and easy to fix!