The following article was written by Aaron Ballman in May of 2008. Ballman worked at REAL Software at the time, and was involved with the decision to depreciate REALbasic's ability to generate a single-file executable when compiling for Windows. Read on to find out why that feature was discontinued...
Important Win32 Application Packaging Changes
REALbasic 2008 Release 2 had a very important change that people are discussing with regards to the way executables are built on Windows. In previous versions of REALbasic, all of your applications were single-file executables. On Windows and this has changed. Now, if you use plugins (internal or otherwise), you get a folder that contains a bunch of DLLs. I want to discuss this change a bit to hopefully clear some points of confusion up.
First and foremost, this wasn't a decision that was made lightly. Trust me when I say, if there was a reasonable alternative, we'd have taken it. But there aren't reasonable alternatives, which is why the decision was made. That being said, let's get some background history on the topic.
In ancient versions of REALbasic, the compiler used to "bake in" plugin shared libraries for the final executable. They would reside as extra pieces of data, tacked onto the end of the executable. When the application would launch, the runtime would take all of the DLLs and write them out to disk. Then the application would load the DLLs from disk, and continue executing. However, this turned out to be problematic in several cases.
For starters, it caused DLL hell. If you had two REALbasic applications that were going to run at the same time, you could run into monster amounts of troubles when the second application tried to write out its DLLs. If the names were the same, the application couldn't write out its DLL, so it would try to use the one already on the disk. However, this doesn't work if the two applications were built with different versions of REALbasic! What would end up happening is that the second application would crash sometimes. So plugin authors had to name their DLLs with some version information so that applications didn't run into naming conflicts. This worked ok, so long as everyone remembered to do that (and trust me, not everyone remembered).
Another major problem was with 3rd party anti-virus software. It would see an application carrying what essentially amounted to an executable payload. Even though we weren't doing anything unscrupulous, some AV software would still flag the application as being a trojan. This would understandably scare people away from using that REALbasic application!
Finally, we had issues with permissions, disk space, and just plain old misunderstandings of the functionality. If the application couldn't write the DLL out to disk due to lack of permissions, or because the disk was full, it would simply fail to launch. This happened with a relative amount of frequency, especially as more and more people started using REALbasic on server OSes like Windows 2000. Also, some users had hard requirements that stated they couldn't modify the user's machine at all -- their application had to run from a CD or thumb drive, and be entirely self-sufficient. This simply couldn't be avoided with the scheme with the way things were.
So after a while, we realized that writing the DLLs out to disk wasn't a reasonable solution. It had way too many drawbacks, and cost a lot in customer support for us as well as for our customers (who had to deal with these issues with their end users). So we came up with another, more clever solution: don't write the DLLs out to disk at all!
We rewrote the OS DLL loader functionality -- since the DLLs were already in memory, why not just leave them there, since that's where they'll end up? So we would still push the plugins into the executable at compile time, and at runtime, we would have the framework do all of the same monkey-work that the OS loader would usually do so that we could just use the DLL already in memory. It was a very ingenious idea because it solved a lot of problems. We didn't have to worry about DLL hell anymore since every application runs in its own memory space. We didn't have to worry about running out of disk space or permissions issues because the application was in memory already. We also didn't have to worry about cases where the user couldn't modify the hard drive since the DLLs were never being written out. The AV software problem was significantly lessened as well (though it didn't disappear totally). However, this solution turned out to be untenable, though we didn't realize it at the time.
You see, the operating system simply does not allow for DLLs to be loaded by anything other than the OS loader. Our solution was very creative, but it had problems. Some of the problems we could solve (like thread local storage*). However, the fact still remains that there is a major class of problems we cannot solve: application compatibility shims. Microsoft's OS loader handles these shims internally, and there is absolutely, positively, no way for us to mimic the functionality. We must use the system loader to load DLLs in order for them to be loaded properly.
So how did we stumble into this problem, and why am I using such strong language about it?
For starters, ever since we implemented the memory loader, we saw a lot of bug reports simply go away. No more complaints about multiple RB applications unable to run at the same time, etc. However, there was a slow trickle of unrelated, unexplainable crashes on Windows. The hallmark of these reports was that they usually said something like "I don't understand this. I have XXXX installs of my application, but on six machines the app totally fails. I can't figure out why." Here's a classic example of a user's problem on the forums. This user actually did an amazing amount of research into the problem, and is one of the few cases that can be classified as "obvious." I've seen it posited that this doesn't affect very many people. That's not true. While it's very difficult for me to put an exact number on it due to the nature of the problem, I can attest that it's a non-trivial percentage of our users.
We would get this trickle of reports, and since no one could reproduce the issue, we would just close the reports as not reproducible. I mean, NO ONE could reproduce these issues... Then one day, I found a way to reproduce the issue! I had a particular machine in my house that couldn't run the REALbasic IDE. Whenever I would try to compile any applications, I would get a strange error dialog that said the C runtime floating point library wasn't loaded. Then the compiler would work, I'd get my application and life was fine. Until I went to quit the IDE when I'd get another message dialog that said thread locale storage couldn't be initialized -- and the IDE would crash in a ball of flames. Very, very weird stuff considering the fact that the CRT floating point library certainly should have been loaded, and we don't use TLS anywhere!
So I pulled out my debugger, and went to town on it. After three days, I was still absolutely no closer to figuring it out. What was neat about the problem was: if I tried to debug the plugin directly, the problems went away. But if I just attached to the IDE with a kernel debugger, the crashes would happen -- and were valid crashes! Honestly, the floating point library really wasn't loaded... it was the strangest thing I had ever seen. I let the problem sit for a few weeks while I worked on other things, and just avoided that machine. Then one night, I realized what a difference was: the way I was debugging the plugin! You see, REALbasic has always had this debugging technique where the application would prefer to load a DLL from disk if it was named right, and was directly next to the executable. In that case, we were using the system loader to load the plugin up. So by using this plugin debugging technique, I was using the system loader. But when I just used the kernel debugger, I didn't have the DLL next to the IDE, so we used the memory loader. This turned me on to the fact that the problem was something to do with the memory loader -- I just assumed we had a bug in there, and went to work trying to fix it.
I spent another day working on the memory loader, and nothing made sense. We didn't have any TLS specified in the spawn compiler DLL, and so that failure eluded me entirely. But what's more, the CRT was statically linked into the DLL, so there was no import record for floating point to load! The CRT takes care of that for you, supposedly. Problems with plugins causing applications to crash is a very serious thing, and I was starting to realize how large the problem was after looking at the number of "not reproducible" bugs and how they all pertained to a plugin at some point. So finally, I decided to use one of my support tickets with Microsoft to see if they could help track the issue down.
I don't know how many of you have ever had a reason to use Microsoft's developer support program, but it is a pretty incredible experience. They basically assign an engineer to you who looks at your source code, and debugs it for you. Since I had narrowed the issue down to the memory loader, that's where I had the engineer start and he was immediately perplexed -- it's not every day that you run into a hand-written loader! So he escalated the issue up the food chain until it finally landed with the DLL loader team. These are the engineers at Microsoft responsible for actually writing and maintaining the code that loads DLLs from disk into memory. They loved our source code, but pointed out the problem: we have no way to load application compatibility shims. It turns out that the CRT from Microsoft has an app compat shim that my particular machine was forcing to be loaded. Since the memory loader had no idea about this shim, all hell broke loose and that was the trouble, in a nutshell.
Ok, so the memory loader is impossible to use. Period. When the people responsible for loading DLLs into memory say "you have to use the system because of X, Y and Z", that pretty much seals the deal. So the next step was to discuss options with Microsoft to see what we could do. You all know the end result by now, but there were other options discussed.
Of course, the first one was "what if we just keep using the memory loader?" The answer to that was, "your applications will become more and more problematic." You see, many times Microsoft comes out with a hotfix that includes more compatibility shims. They have an ever-growing database of times when a shim needs to be loaded. And if the shim isn't loaded, the bugs can be spectacular. Sometimes they're subtle crashes at any point during the application's session. Sometimes it's a crash when the app launches, or a big hairy dialog that makes no sense at various times, etc. What's more, something as simple as installing a hotfix could break an application, but only one one particular hardware configuration! Basically, keeping the memory loader would put a huge burden on our users by having crashes that are reasonably out of their expertise to track down and resolve. That then puts a burden on us in terms of bad publicity ("RB applications are buggy"), and man-hours (tracking down bugs related to plugins). And the problem will only get worse and worse as time goes on. Basically, keeping the memory loader is a very bad idea.
The other option discussed was by having the plugin format change in various ways. For instance, we could require plugin authors to rewrite their plugins as static libraries that the REALbasic compiler would link in to the executable. In that case, there would be no need for a DLL at all, ever. This has some major problems too. For starters, it would require plugin authors to rewrite their plugins. Never mind the amazing amount of work that would be for some, what would be done about the plugins that no longer have an author supporting them? But a bigger problem is the fact that static libraries have custom file formats that are compiler specific. That means we'd either have to restrict plugin authors to just one or two compilers (and versions of the compilers too!), or we'd have to spend *years* writing linkers that break constantly. It's a noble idea, but it's also basically impossible to get right. Even if we tried to statically link in the DLLs without requiring the authors to rewrite a thing would run into compatibility issues.
So in the end, the discussion from Microsoft's end really boiled down to: just load the DLLs from disk like every other application on Windows does.
Now you understand why we had to make the decision to write plugins out to disk. Our options were to go back to the old way with DLL hell, AV problems, etc. Or to just spit the DLLs out to a folder (which is what we already do on the Mac, though those are packaged up more cleanly by OS functionality via bundles). Since the whole point to this issue was to reduce bugs, we really only had one option to pick from.
We had toyed with the idea of providing users with an option to turn this functionality on and off, but eventually decided against it (strongly). You see, at the root of it all, what we were discussing was an option that basically read like this: "Would you like your applications to randomly crash in unavoidable ways? Yes or No" That's not an option we are comfortable giving to our users. When something is a preference, the expectation is that it's supported whether the user says "yes" or "no." And this wouldn't be supported. So now we're talking about more than just a simple option -- we'd have to explain to the user the ramifications, pitfalls, etc. But most users won't read that far, they just figure "it works well enough I guess" and pick one, regardless of how strongly worded the cautions might be. So in the end, the decision was strongly against providing an option because we don't want there to ever be a choice between stable and unstable. The whole point to REALbasic is that "it just works."
The nice thing is: REALbasic still does just work. Yes, if you use plugins it no longer generates a single-file executable. Instead, it generates a single folder that you can zip up and send to your friends. It's also no more confusing to the majority of people since the application is still blindingly apparent (it is the only file in the folder). So if you are not interested in writing an installer, your distribution is only slightly modified. What's more, because of the folder layout we eventually went with, the vast majority of applications will require no modifications. You can still run multiple REALbasic applications from the same directory, etc. This change will affect your build processes, and that is the unfortunate cost of making applications on Windows more stable. If you're creating a professional-grade application on Windows, then you have an installer that places the executable into the Program Files directory. This is a directory that most users never see, much less monkey with. In fact, users can't monkey with it unless they have admin rights to their machine (in which case, they hopefully know enough not to remove random files, right?). So while this certainly is a change, it's also not one likely to be felt by your end users in many circumstances.
In closing, I understand that this sucks. Having a single-file executable distribution was a really neat idea that set REALbasic apart. Changing that is not a decision that we made lightly! But it is undeniable that we value stability, and this is a case where we needed to ensure that REALbasic does the right thing to provide the most stable applications possible. Single-file executables are neat, but not neat enough to justify the pain they were causing many of our users.
* There was a linked article on thread local storage, however it is not currently available.
By bkeeney on May 9, 2008 9:17 AM
Thanks, Aaron for taking the time to give us a very detailed explanation of the process Real Software went through in making this large and controversial (for some at least) change.
Yes, this change sucks. Yes, it impacts everyone. But if it improves overall stability in Windows then it's a win-win for RS and us.
By Steve Garman on May 9, 2008 9:52 AM
Thanks Aaron. I must admit I never had much doubt you'd made the right decision here (although the single-file executable will be sadly missed) but it's tremendously interesting and comforting to see so much detail on the decision.
It's also very nice to have a blog posting to point people to when they can't believe this isn't just done to spite them.
By Travis on May 9, 2008 10:36 AM
Thanks for the overall breakdown. Very interesting read.
As I had mentioned earlier via email, I was surprised by this. In a non-RB project I had done a similar in memory loader strategy a while ago- and it seemed to work. Just like most of my RB apps :)
I had not even begun to think of the app compatibility shim issue until Aaron brought it up as a key reason to let Windows load the DLLs itself. Of course- it turns out- it had potential impact on that other project as well. So I also learned something helpful here about Windows internals in the process of this RB change.
Now one thing that I think would be great would be to get more and more of the RB framework written in RB itself and eliminate more internal DLLs. I can understand the 3rd party lib stuff- but beyond that- there isn't a whole lot you couldn't do with declares now. I await the day when RB is like Delphi / VS 2008 with the latest .NET class code... and the Pro RB includes the framework code that is written in RB. Hmmm maybe I need to sign up with RS to make that happen one day... :)
By Will B on May 9, 2008 10:38 AM
Aaron, thanks for the explanation.
Actually, I don't think this sucks at all. Anything to improve RB's stability and take away odd and intermittent crashes sounds like a winner for me.
As far as the one file executable idea...it's not that big of a deal to me. To me, what is paramount is stability and consistent cross-platform performance. So what if there are a few extra files? I use an installer on Windows anyway because I have extra support files, help pages, etc. What's an extra folder?
Thanks again for the explanation, Aaron. It throws a whole new light on how things get done at RS.
By Houston Farrow on May 9, 2008 3:38 PM
Thanks for your thorough explanation.
As I've always said, to enjoy software development you've got to enjoy being a detective. Excellent detective work!
By Christian on May 9, 2008 6:54 PM
Thanks Aaron. When you said "Microsoft said so" I was fine with it. The single EXE has been great. When I was a VB programmer the company for which I worked bought a special package to bundle up DLLs into the EXE. I wonder what that company is doing now.
By Bryan on May 9, 2008 7:11 PM
Thank you for writing this blog entry. Really helped me understand the issue.
By SnappyFish on May 9, 2008 8:00 PM
Thank you for taking the time to explain this, good work.
By Will B on May 9, 2008 8:53 PM
Call me a dunce, but maybe you could explain what an app compatibility shim is? Maybe that's a whole other blog post, but I've only heard the term a few times here and there, and haven't the foggiest about what it is and why it's needed.
By Aaron Ballman on May 9, 2008 9:06 PM
@Will B -- an app compat shim is Microsoft's way of keeping the ball rolling, so to speak. Let's say your application relies on behavior X, and Microsoft fixes a bug that breaks your assumptions with X. They may choose to come up with a compatibility shim that "unbreaks" your application by restoring the behavior, but just for your application.
Here's a tech note on the topic: http://technet.microsoft.com/en-us/library/bb457032.aspx
By Ramon on May 11, 2008 11:07 AM
My level of programming is not high enough to understand everything you have explained, but it seems really serious and I must admit that the decision has not been easy to take.
However, I do not agree completely about your comments on being optional. As a university professor I use to make simple small applications that are fantastically distributed and used with a single file.
So, what I am going to do is keeping RB2008r1 as gold, and use it for simple programs, while I suppose it's going to be much better to move to the future RB releases for important or long projects.
I've had many "strange" problems that I've reported to RB, but they never gave me a solution. I remember specially one about chaos with menu captions when they were changed during use in a real big program.
Do you think it might be caused by any DLL? I don't know.
By Aaron Ballman on May 11, 2008 11:24 AM
@Ramon -- the problem with making it optional is that we now provide an option that we *know* is unstable and not supported. However, there's no reasonable way to convey that to our users. The simple fact is: if we provide an option, people will expect it to work. That's just the natural way of things. But we can't be sure it will work, and so we won't be providing an option.
As for your problem, I really couldn't say. The breadth of issues from this problem is so great that there's really no easy way to say for sure "yes." If you only see the problem on a handful of installations, then it sounds more likely to be the app compat problem.
By jdiwnab on May 12, 2008 7:07 AM
I have been missing lots of forum posts lately, and I just saw MBS's program to bundle software, and I though, "Why? doesn't RB make single executables? that was one of it's main selling points." Luckily I came across here. This makes it seem perfectly reasonable. But I wonder if there is some way to at least provide the illusion of a single executable, like what MBS is doing, but natively. I know that in OS X, the whole operating system supports their bundle thing and I know that you can't do that on Windows, but I'd imagine that there is some way to do something like it for Windows. If such a thing existed, it would be a great compromise, having the DLL's loaded as needed, yet still having a self contained executable.
By Aaron Ballman on May 12, 2008 8:46 AM
@jdiwnab -- The MBS "solution" is basically the same as what we used to do years ago with writing the DLLs out to disk. Anyone making use of that will run into the same problems we ran into in the past. If Christian would like to make something like that, and people find it useful, then that's a great deal! But we're not going to supply it due to the problems that arise from it.
By doja on May 13, 2008 9:35 AM
I really liked the simplicity of the single .EXE file, BUT stability and the continued ability to run from a CD and/or on computers that are fully secured (as in college computer labs) are much more important for our apps. Given the situation as Aaron described it, it seems that Real Software made the only viable choice.
By edb on May 14, 2008 10:32 PM
Great article! Stability > everything else!
By Aris on June 11, 2008 2:18 AM
I am currently looking for possible ways to make our VB6 application play cross-platform. Our application is AllWebMenus: http://www.likno.com
The tricky part here is that it is being used by thousands of users (as opposed to creating an application for a single customer) with so many configurations.
This raises the "alert level" very high. We need to be 100% sure that a RB-driven AllWebMenus will be fully functional in 100% of cases.
We use the following DLLs:COMDLG32.OCX
1. Will we have to get rid of the above DLLs and use new ones, created by RB? Which ones?
2. What about protection of our EXE? We currently use Execryptor to protect our application. Will we be able to do the same? (without crashes etc.?)
3. Does RB support unicode text/controls inherently or 3d-party controls are needed? (like UniToolbox for VB6)?
4. We now right all "activation" information to the registry. Will this sensitive information be written inside the folder now? Can this be protected?
5. In other words, can RealBasic be used for "shareware"-type of applications (allowing for: thousands of different configurations, anti-crack protection, proper activation, without crashes, etc.)?
Thank you for any feedback to a difficult decision.
By Jeffa on August 27, 2008 2:51 PM
Nice article Aaron, but bad decision for RB.
I will have a hard time to port my apps to another language, but harddisk-cluttering multifile enviroment was the reason for switching from GFA-Basic to Real-Basic: because RB does it in one exe without depending on any dll's cluttert on the users harddrive. Now RB does this step and its time to move on. Until i found an alternative i will stick with 2k7R5 as it DOES single exe apps. The discribed behavior of the compiled apps couldnt be veryfied by me, these apps run on ANY standard setup winxp-system i can get my hands on, and these are not a few. So, sorry, but this is a NOGO for RB >2k7R5.
It was great as long as it lasted.
By JamesMc on August 31, 2008 4:32 AM
What a thorough explanation !
I'm wondering, if I'm using a previous version of RB that does not deliver DLLs to a separate folder, what "plugins (internal or external)" should be avoided ? (To minimize the risk of the many issues highlighted).
How can I tell if an RB feature is implemented using an "internal plugin" ? (Assuming that a project - once compiled to a "single exe" - without any such occurrences would then circumvent these issues)
Any and all feedback appreciated !
Casual RB User: WinXP, RB 5.5.5 Pro, RB2005r4 Pro
By Aaron Ballman on August 31, 2008 9:35 AM
@JamesMc -- the language reference in newer versions of REALbasic now lists everything that's implemented as an internal plugin. So I'd just grab the demo of the latest version, and look up "Internal Plugin" in the LR.
By Franck Perez (SerialBasics) on October 30, 2008 1:40 PM
many thanks for this explanation. But as Jeffa, I think this is an infortunate choice. I would have strongly preferred that this behavior could be chosen by the user and not the only behavior. A single .exe file was a unique advantage. Coming from the Mac, I even do not know where these dlls should be installed. Is this always at the same place ? can you recommand an installer ?
At the moment, even though I have a licence for RB2008r4, I keep programming under RB2007r5 and I am hesitating to renew my update program.
By Arnaud Nicolet on August 30, 2012 4:43 PM
I've read your page and I thank you about having written it. But, several times, you mention it wasn't possible to have an option to use a single executable file, like here:
the problem with making it optional is that we now provide an option that we *know* is unstable and not supported. However, there's no reasonable way to convey that to our users. The simple fact is: if we provide an option, people will expect it to work.
I think it's not true if you default the option to "No" (use multiple files format) and put a label showing something like "Note: choosing "yes" could lead to crashes under certain circumstances". Most users would keep the "No" option checked and all would be fine.
If you have any comments, feel free to email them to me and I'll post them here.