Friday, September 11, 2009

ASP & IIS - When good numbers go bad....

Not long after the deployment of an updated ASP (IIS 6) web-site within a new environment (and we're talking hours), the sites began to fail. Weirdly, it wasn't all pages, or pages leveraging some custom COM objects or even consistent - some would still happily work 'post-cataclysm', while others were terminal - always presenting the following:
Microsoft VBScript runtime error '800a0006' 
Overflow
So with some pages still happily banging away (including pages with ADO etc), I was forced to conclude it wasn't a complete ASP / IIS meltdown, but something specific. Anyway, an IISRESET later, everything was cruising nicely once more.


30 minutes later - it's back (and where was this error in UAT!?). Another IISRESET and things are happy once more. Some further investigation of a failing page showed that it was crashing while calculating the number of rows needed to return some data (e.g. RecordRows = NumRecords / RecordsPerRow). No divide by zero's or conspicuous problems here either.... Re-create the app_pool and site, check the settings are sane - but it all looks good. 


10 minutes later - its gone again. Starting to get worried now, I check some more failed pages. Again failing at points where it is doing calcs.... I strip out the calculation and the page loads further... until the next calc. Changing the calc to be something basic like '1 / 0.2' still causes failure. In fact a page with just:
Dim myNum
myNum=(1 / 0.2)
still causes failure... Oddness.


Some rummaging around the 'net showed a few others with a similar errors, but not a solution (most errors seemed to be coding bugs or people suggesting coding solutions). One site eventually made mention of dragging out a MS engineer to investigate a similar issue.... Apparently there was a bug with ASP (vbscript) on II6 (Windows 2003) and the C run-time checking for errors in a floating point register, but never actually resetting the error condition - meaning one floating point error takes down the site (!) Apparently the engineer wrote some code to address their issue - and because the engineer was able to resolve it on-site, MS wasn't going to make it a mainstream patch...


Some further searching of the Microsoft site for variants of ASP, IIS, Window 2003, floating point etc, eventually lead to KB955536 ( http://support.microsoft.com/kb/955536 ) - a hotfix.


After a few deep breaths, I installed the hotfix on one of our servers (thank god for redundant servers) and crossed my fingers.....


A week later, the unpatched server had been iisreset many, many times (ended up writing a powershell script to do this), but the patched server had been rock-solid, with no side-effects shown. So the hotfix was put onto the second server.


Two months on, both servers are still completely happy and very solid - it has actually made the upgrade from Windows 2000 worthwhile... But I would have preferred a smoother upgrade :/



Adobe Flash upgrades & Citrix - entertainment for all...

Ah Citrix. 
And Flash. 
And a locked down user environment that prevents Active-X installs.
It all adds up to a recipe for entertainment when upgrading Flash ( in this case Flash 9 to 10) on Citrix 4.5.


As it turns out, an upgrade as administrator was enough for things to kick on happily enough for admin users, but not clients. Post 'upgrade' clients effectively had no flash at all - a nice backwards step.


The trick with this one was registry permissions - administrators had them on the updated registry keys, but everyone else missed out. I tracked down some assistance at: http://www.brianmadden.com/forums/t/32557.aspx .
Admittedly I skipped most of the instructions - the critical ones (for me) were the registry permissions - namely:


1. Find the key HKCR\CLSID\{D27CDB6E-AE6D-11cf-96B8-444553540000} (on 64-bit Windows this one is located under HKCR\Wow6432Node\CLSID\{D27CDB6E-AE6D-11cf-96B8-444553540000}) and grant READ permissions (include child keys) to the appropriate groups (usually everyone).
2. Rinse and repeat the permissions change for HKCR\CLSID\{D27CDB70-AE6D-11cf-96B8-444553540000} .


This was pretty much a instant fix, restart IE as a user and everything was happy once more.