InstallShield, .NET DLLs and DotNetCoCreateObject()

One of my projects requires that I call out to a DLL written in C#, which requires .NET 2.0 SP2 (or CLR20r3).  When I first went to implement this using InstallShield's DotNetCoCreateObject() method, it took a lot of futzing about before I could load the DLL successfully.  I also had horrible problems with handling any exceptions that occurred within the DLL.  And just yesterday, the developer made a change within the DLL and it broke the installer again.

(That, by the way, is the best part of being the "installer-guy".  Whenever anything breaks, they look at you.  Never mind if you haven't had a check-in for the past three days and it just broke last night, you are guilty until proven innocent.)

Anyway - the error we'll look at today is a delightfully informative one.  When caught, all you'll get is a number: -2147219705.

Gee, thanks... that's helpful.

The first thing you should know is always wrap your DotNetCoCreate in a try/catch block, otherwise you will get burned down the road.  Here's how mine is loaded:

 szDLLPath = SUPPORTDIR ^ "MyDotNet.dll";   
 szClassName = "InstallHelper.DoInstall";
 try
    set oInstHelp = DotNetCoCreateObject(szDLLPath, szClassName, "");
 catch
    SprintfBox (INFORMATION, "Error","Error occured: %i\n\n%s\n\n%s", Err.Number, Err.Description, Err.LastDllError);
    abort;
 endcatch;

In the case above, my C# DLL contains a namespace called "InstallHelper" and a class within that namespace called "DoInstall".  The szClassName var must match what's in the DLL for starters, otherwise you will get that -2147219705 right at the get-go.  Note also that if there's anything in the implementation that InstallShield is not a fan of, you will also catch the -2147219705.  (Yesterday's problem seemed to be just that - the developer had tried to merge two DLLs together, they worked fine from a test console application, but when InstallShield loaded the DLL it barfed.)  In short, this seems to be the error that InstallShield will throw when it just plain doesn't like your DLL.

If this is the first time you're dealing with a C# DLL and DotNetCoCreateObject(), I recommend that you (or the DLL developer) begin as simply as possible.  The first test case should have a single public method that does something incredibly simple, like writing a text file or creating an event log entry.  A message box is not necessarily the simplest because, depending on when you invoke the DLL, any UI elements may be hidden, delayed, or simply fail silently.  Make sure that you have a working framework for your DLL communication before you or the dev spend hours on the project.  Once you know you can load the DLL and call a method successfully, you'll have a baseline to back out to in case any future changes break the implementation. (I hope it goes without saying that if you'll be calling such a DLL, you should have a prerequisite set for the specific .NET runtime that the DLL requires...)

Once you make it through that try/catch block, calling methods in the DLL is very simple. 

try
    nResult = oInstHelp.DoSomething(szSomeText);
catch
    SprintfBox (INFORMATION, "Error", "Error occured: %i\n\n%s\n\n%s", Err.Number, Err.Description, Err.LastDllError);
    abort;
endcatch;

That's all there is to it.  There's no prototyping involved, just call your method.  Method parameters called by-reference work transparently - if "DoSomething()" above just set the string to "Did It", I would find szSomeText being equal to "Did It" in the next line.

Caveats

There are some potholes though.  There's a bug in IS2010, not sure if it persisted in to IS2011, where strings passed to objects created with DotNetCoCreateObject() are passed as an entire 4KB buffer, padded with space characters.  This can cause problems in your DLL if you don't handle it internally.  For example, say you pass SQL credentials to the .NET object.  If your username is 'dbadmin', and you don't trim the string in the DLL, you'll attempt to login as 'dbadmin\0[space][space][space][space]...' to 4KB.  We addressed this issue by passing all input strings to a "CleanUpNulls" function (in the DLL, that is), which trims the input string to the first NULL character.

Note too that this buffer handling can pose problems for the IS developer as well.  This was a fun one to fix!  Let's say you have a text input field whose contents you'll be sending to the DLL.  When the user gets to that field, they initially type an eight character string (e.g., "AcmeCorp").  Later, they return to the field and change it to a four-character string, "Acme".  When you pass the string to the DLL, it will look like this:

Acme\0orp\0[space][space][space]...

If the DLL trims the string on the first null, you're ok.  In our case, the dev had setup the CleanUpNulls to trim the string on the first null character from the right, which meant the string they were working with was "Acme\0orp".  I ended up fixing that particular problem in InstallScript by copying the string to a new var character by character until I reached a null, it would've been easier (in retrospect) to fix the CleanUpNulls in the DLL...

Final Note on Exception Handling

If you will be making a lot of calls to the DLL, you may want to make use of the .NET Application Domain to avoid a strange exception-bubbling mess.  I discovered in my case that if the DLL threw an exception on the first of three methods I called, the subsequent calls appeared to error immediately.  I spent a lot of time on this problem as well, and in truth only came up with a working hypothesis: the nested nature of most .NET exceptions were leaving an "exception queue" behind whenever an error was caught in InstallScript.  So your try/catch around a method call would clear the top-level exception, then you establish a new try/catch block and go to make another call - but there's still an unhandled exception bubbling from the last method, so even if your second method has nothing to do with the first, it will still raise an exception.

I eventually got past this by using an app domain when I called DotNetCoCreateObject, and clearing and recreating the object before each method call.  I then used a nested try/catch, where each method call in its own try/catch block would, in the catch block, raise another exception to break out of the outer try/catch block.  An exception in one of the methods would then bypass any subsequent calls, prompt the user to fix the problem, and then (before restarting) unload the app domain.  This effectively flushed the "exception queue" so that I could give the user a chance to fix the problem and try the process over again.  A brief example is shown below:

:TryAgain
try         
    bTryAgain=FALSE;

    // Drop and recreate the DotNetObject - clear any as-yet-unhandled exceptions
    DotNetUnloadAppDomain("InstHelpDomain");
    try
        set oInstHelp = NOTHING;
        set oInstHelp = DotNetCoCreateObject(szDLLPath, szClassName, "InstHelpDomain");
    catch
        SprintfBox (INFORMATION, "Error","Error occured: %i\n\n%s\n\n%s", Err.Number, Err.Description, Err.LastDllError);
        abort;
    endcatch;
                        
    // Get the log file name in case of errors
    nRet = oInstHelp.GetLogFileName(szLogFilePath);
    if (nRet != 0) then
        Err.Raise(-1);
    endif;
           
    try
        set oInstHelp = NOTHING;
        set oInstHelp = DotNetCoCreateObject(szDLLPath, szClassName, "InstHelpDomain");
    catch
        SprintfBox (INFORMATION, "Error","Error occured: %i\n\n%s\n\n%s", Err.Number, Err.Description, Err.LastDllError);
        abort;
    endcatch;
           
    // call another method, etc...
    .
    .
    .
catch 
    MessageBoxEx("Blahblahproblemfixit", "UhOh", SEVERE);
    LaunchApplication(WINDIR ^ "notepad.exe", " " + szLogFilePath, WINDIR, SW_SHOW, 30000, LAAW_OPTION_WAIT);
    if (AskYesNo("Wanna try that again?", YES) == YES) then
        bTryAgain = TRUE;
    endif;
endcatch;         

if (bTryAgain) then
    goto TryAgain;
endif;

That convoluted mess was the only way I could catch an exception raised from a .NET DLL, give the user an opportunity to do something about it, and then try it all over again.   I can't help but imagine there's a better way, but damned if I found it.  When I came up with the hypothesis about the nested exceptions and the exception queue, this was the first thought I had for dealing with it, and it has worked out nicely.  It may lack elegance, but it certainly qualifies for "if it ain't broke, don't fix it".

So there we go.  Again, I hope I save some poor bastard from having to figure all of this out on their own, and I wish I could've found a page like this when I first had to do it.  InstallShield's forums can be helpful sometimes, but I found them seriously wanting when it came to the DotNetCoCreateObject stuff.  Hopefully this post gets sucked into the search engines of the world and saves someone a few hours of work and makes them look like a genius.  Good luck.