WCF Hosting in IIS and serviceHost.Open()

We recently had a valid reason for *looking* to see if we could (when hosting WCF in IIS) override the ServiceHost.Open() method - specifically we were looking at the possibility of implementing a retry for certain exceptions thrown on open.


The most obvious course of action was to look at declaring a custom service host factory, reference this from the "factory" attribute in the .svc markup and implement our own "ServiceHost" implementation derived from ServiceHostBase.


The problem is that "open" is defined on the communication channel class and it's not virtual, we thought about redefinition by hiding (declaring a reimplementation of "open" with the the "new" keyword in our service host class) but it would appear that IIS internally takes a reference to serviceHostBase - and redefining that would be a step too far IMO.


So in short it doesn't look like it's (easily) possible, if at all, I guess not many people have a reason to consider doing this with IIS hosted services.


(We did also take a quick look at what might be possible if we hooked up to some of the events around open, but this "felt" wrong then and still does, even though it *might*  be technically possible).


This was all in context of "on-premise" services connecting to azure via the Azure "ServiceBus", in some circumstances we were getting "AddressAlreadyInUseException" - the discussion was around what it might take to implement  a "retry" if the serviceHost.Open() failed with that exception; subsequently we understand that MS intend to release changes to support load balancing in SB - in other words allow more than one connection per URI, so our issue should be resolved with this change.

Msbuild and VS solution/project "platform" inconsistency

If you want to build a solution using MSBUILD under a specific configuration you have to specify a valid combination of build-configuration and platform.

For example "Debug|Any CPU"

This is fine, unless you are using a composite target that works on both solution and projects; in our case we were using the solution to build some web projects and their dependent projects and then calling a target to build the web project hosts. (An alternative would have been to build from a project list, but it's convenient to utilise a solution where dependants are resolved auto-magically!)

The problem was that introducing "platform" as a parameter worked for the "build solutions" target - we build under the correct configuration, but it broke the "build web-project-package" target.

The reason is an inconsistency between solutions and projects

Solutions platform Any cpu = "Any CPU", but for projects it's "AnyCPU"

See here

As a side note:

If the solution contains an azure web role project and this project has it's own service configuration then the .csfg file is likely to be called "ServiceConfiguration.[my config name].csfg"

If this is the case you need to pass the parameter "TargetProfile=[My config name]" when building the solution, or else remove the azure web role project from the build list, which may not be desirable.

There is some default behaviour with the azure web role project build target, if a valid value for parameter "TargetProfile" is not passed then the default behaviour is to look for "ServiceConfiguration.Cloud.csfg" first, followed by "ServiceConfiguration.csfg" if neither is found there's an error.

Re-connecting service-bus endpoint - Unexpected - the specified address already exists

Consider two"on premise" WCF service endpoints pointing at a azure service-bus namespace to make them "address-able" to trusted callers going via azure.

Note that the URI are almost the same, except MyContractA has an additional qualification
(a)

<endpoint contract="MyContractA"
binding="netTcpRelayBinding"          address="sb://MyNS.servicebus.windows.net/Company/Area/Service"/>

(b)

<endpoint contract="MyContractB"
binding="netTcpRelayBinding"          address="sb://MyNS.servicebus.windows.net/Company/Area"/>

Remember that each on-premise service that should be address-able via service bus need to be started to establish the connection to the sb relay(which if hosting in app fabric can be accomplished via the warm start feature), but after long periods of inactivity, the connection may be dropped and can be restored by issuing an HTTP Get to each (to re-establish the outbound connection to azure).

What we found was that we'd occasionally get this error trying to re-establish a connection (the services are hosted only once, because at the time of writing Azure SB supports only a single connection per endpoint)

The specified address already exists. Address sb://MyNS.servicebus.windows.net/Company/Area/Service/ already in use.

Which didn't make sense at all in this case.

The thought was that the order in which we "connected, or re-connected" the two services was important, if we started the service with the longer address (a) in example above, first, then no error.



Thanks to Paolo for finding us an explanation of this

Extract from that:-

"Suppose we need to expose the url i.e. just http://.servicebus.windows.net/ e.g. to expose client access policy file through services bus. Then once the base url (as mentioned earlier) is registered, for any further url registration like http://.servicebus.windows.net// appfabric will through the below mentioned error message even if the url is tried to register for the first time.
The specified address already exists.Address sb://.servicebus.windows.net// already in use.
So the resolution is, if needed then first register url like http://.servicebus.windows.net// and then only register the url http://.servicebus.windows.net/ for any appfabric service namespace. "

IIS Express and SSL

There's plenty about this, but still I got caught out for a while this afternoon...

When using IIS express you can host a service expecting SSL, if you want to make calls into the local service it looks like you do have to trust the default localhost cert installed for IISExpress (see inetmgr "server certificates", this can be exported and imported into certmgr.msc)

See here for enabling SSL and a neat workaround for port numbers (iis express doesn't use 443 by default)

And
Jon Simpsons nice post for sending client credentials to a service which then uses ASP net authentication


Update: Another way (but not that safe) is to ingore the local cert trust issue in the client by adding this (in vb in this example)

System.Net.ServicePointManager.ServerCertificateValidationCallback = New System.Net.Security.RemoteCertificateValidationCallback(Function() True)

TCP Relay Binding (azure / service bus)

The azure service bus has 3 modes of operation, tcp, http and auto-detect (correction:- tcp is the default)... This initially confused me; we were testing a simple scenario hosting a service on-premise and invoking it from the cloud via the service bus and relay. We need tcp but were expecting the connection to be refused as the ports were not yet open.
Whilst our firewall logs showed the tcp connection was refused as expected - the connection was then established over http.
Because we need tcp in our scenario, we're going to need to configure the service bus system connectivity mode to tcp (and not the default auto detect).

You can do this in the global asax

ServiceBusEnvironment.SystemConnectivity.Mode = ConnectivityMode.Tcp;


(At time of writing this will avoid a ping to the MS watchdog address when the service starts)


If you're hosting in WCF and can't add global asax then add a custom service behaviour extension

If you have a requirement to authenticate client credentials check out

this post

Which explains how to configure the service to flow client credentials from caller to service where a custom password validator component (very easy to write) can be configured to provide the validation

IIS hosted service "warm up" or "auto start"

The beta extension for IIS has been removed, so far as I can see the other option is calling the service method manually (script) or using the built in feature of "Windows Server App Fabric" if hosting for WF/WCF

See here - iis-application-warm-up-module

Beta site (now closed)

The reason for looking? If you have a service acting as an on-premise end point to be called from a cloud service via the service bus using a relay-binding, that on-premise end point needs to be active (I won't say listening because it's not exactly).

Unhandled Exceptions in IIS

Any single unhandled exception (not an exception thrown in code) - an exception at the CLR/Framework level that occurs during service execution, will not be caught and wrapped as a fault by IIS. Instead the w3wp.exe process will be torn down and any app domains inside will be lost, the app pool will keep running under default settings - the default behaviour of IIS is to re-spawn a new worker process (which is expensive). In this case the client would see ---- not much, a socket error or timeout.



This behaviour was new for .net version 2 and deliberate, it prevents unhandled exceptions of this type getting lost (or “swallowed”), the OS will also attempt 11th hour logging – to the event log – so that you have information to help you debug


For more Info

See here

And here


I didn’t know about the setting: IIS, App pool, Advanced Settings, Rapid-fail protection.


This exists in IIS6 and 7 and it’s designed to protect your server, in the possible event that any “unhandled exception” isn’t a one off, or glitch, after a set number IIS stops the app pool based on those user settings.


The process of recycling is very expensive which means this behaviour is obviously useful on a number of levels. The caller gets “HTTP Error 503. The service is unavailable” once the app pool stops.


It’s actually quite difficult to test this, after all just throwing an exception won’t work, don’t want a fault, and forcing CLR/.net to throw an exception is contrived, but you can test this behaviour in a deployed service by adding the following line into your service

            Environment.FailFast("My message: Killing host");


Note: You can add a catch for AppDomain UnhandledException but (again from .net 2) this won’t stop the app domain crashing.

Notes:

Info on “Watson bucket” problem signatures

Migration from Biztalk 2006 (not R2) straight to Biztalk 2010

Pre-Prep

**Migration from BTS2006 (not R2) straight to BTS2010 is not officially supported **

Need to consider making another machine (maybe virtual) available to developers requiring BTS, if pre-upgrade code is to be supported, this way pre upgrade code can be maintained and post upgrade development continue in parallel. Two sets of staging environment may also be required

BTS2010 requires SQL2008R2. 

It is a good idea to script the setup and configuration of BTS for the new machines including hosts and host instance. 

Consider existing source code and what fits with your branching strategy.
It’s helpful with a large set of BTS solutions, if build and deploy are fully scripted via MSBUILD or other.


Upgrade notes

1. We elected to upgrade one VS solution file at a time, starting with the “core” solutions and working up the dependency tree.

2. The upgrade is started by opening the VS solution in VS2010 on a development machine with BTS2010 installed

3. The wizard will prompt about source control bindings we elected to always “use solution” – this makes sense as the TFS bindings in the existing solution (were for us) quite correct.

4. We chose to read first then click over any project file errors and warnings RE change of framework to .net

5. Once the BTS projects are upgraded, the project files become proper MSBUILD files (as per regular Visual studio projects) however a BTS assembly is required to run in the .net 4 runtime so the project target needs to be .net v4 – which will happen as part of the upgrade, but note that other referenced .net projects that are part of the solution will not necessarily be automatically upgraded to framework v4, however if these satellite project(s) – for example .net helper assemblies - have a direct reference to any BTS artefact (XLangs for example) then they too need to be moved to v4 - you should get a compile error otherwise.

6. We decided not to move other shared projects, those without BTS artefacts to the new framework version (at least for now), partly to contain scope and partly to ascertain any impact to legacy systems. In the future we may upgrade where there’s justification.

7. Once the wizard completes review the report, do not be surprised if some projects are “skipped” this will be projects previously “upgraded” or projects the wizard considered should be skipped – i.e. don’t require any change.

8. Important note: The upgraded BTS projects will have an assembly-info file generated in the usual place for a c# project (“properties” folder), this has an additional line “[assembly: Microsoft.XLANGs.BaseTypes.BizTalkAssembly(typeof(Microsoft.BizTalk.XLANGs.BTXEngine.BTXService))]” which is very important for deployment. However we found this line will not be added if the assembly info is subsequently generated (by editing project properties) …

9. The important thing to remember is to add assembly-info file into TFS, it won’t be automatically we were not previously able to do this for BTS projects as the assembly info would get re-generated for BTS2006 project. We’ve found the quickest way to get a new file into TFS is

a. Show all files on the project

b. Right click “exclude file” – it’ll now show ghosted

c. Right click “include file” back into project, which will force it to be added into TFS

10. We have had some problems with web references, specifically un-typed services – by this I mean services with message payload which equates to “any” or “xmldocument. In BTS 2006 such messages might have been shown in the proxy as “xlang.basetypes.any”, in the past we’ve used this message type directly in map “btm” file input/output definitions (note that we use custom xml – so we’re not really using the mapper at all, which is why this works. The problem is that if you allow this “any” type post upgrade then the project and map will build fine, but once deployed your BTS install will be corrupted, the Management console will break and you won’t be able to view deployed applications and artefacts. There are two options

a. Re configure your BTS install (which will scrub out the databases etc)

b. Find the map definition in the relevant BTS database and table then delete it, this will allow you back into the management console and you can uninstall the affected BTS application

11. You can update an existing project web reference (you know the thing we used in 2006 before service refs were inventedJ) and you can re-add an existing web reference via the “service reference” dialog, advanced options.

12. Check the build option for each file in a BTS project web reference post upgrade, some files will not need to be compiled and some will, the upgrade can lose this compile flag on these assembly files.

13. The upgrade will change some web reference namespaces, this means an edit is required anywhere a web ref artefact was previously defined. To identify this compare the namespace before and after upgrade, it’s a bit of effort and how much may depend on how deep into you’re code the proxy is used.

14. If you get some issue which just doesn’t seem to have a logical explanation, please repeat on another machine before spending too much time, we’ve found issues like this that are really non-issues and a result of a single machine.

15. File locking in VS seems to be a big issue when you build more than once, all project references should have “copy local” set to “true” but even here it still happens, closing and re-opening works the best, but there may be other options (can close file handle in procexp for example or utilise an un-locker tool).

16. Thanks to my colleague Jeremy: If the odx file (say ” myorchestration.odx”) opens as text and not in the designer post upgrade check for this in the owning project file

<ItemGroup>
<XLang Include="BookProvider.odx">
<SubType>Designer</SubType>
</XLang>
</ItemGroup>


**Remove the line : <SubType>Designer</SubType>

Deployment
As part of the upgrade the BTS projects will become MSBUILD type, any new BTS project will have the build config options of Debug|Release, however the wizard does not change existing projects; we had made the decision to be consistent and change the existing upgraded projects build config options now. We found the most efficient way to do this in the open solution (post upgrade) – was to multi select all affected BTS projects in the solution explorer tree (those upgraded) and “unload” them, you can then right click and open each as xml, once all are open do a find and replace for “development” to “debug” and “deployment” to “release” this will change everything required in a single step, save files, close them and then multi-select those unloaded projects and reload back into the solution. Then check all projects in the solution “config” manager window should now show “debug”.