I need to build a DLL capable to load other DLLs at runtime; these other DLLs have a rather intimate relationship with the main DLL (this is Python and extensions), so they must have a common runtime. The main DLL must be a single file that can be simply copied on the target machine. The auxiliary DLLs will be placed in a different directory.
So: "common runtime" means no static linking; "single file + simple copy" rules out the shared MS redistributables, especially when coupled with "different directories".
I only see the following options: link all DLLs against msvcrt.dll; embed current msvcrtXX into the main DLL and re-export all its symbols; use the msvcrtXX of the host application. To me the first looks the simplest, because it's a common need and there are many web pages explaining how to go about it. How would you approach this?
I suggest you re-architecture so that your plug-in DLL (the one the host application loads explicitly) contains nothing but proxy functions plus the file management logic. This DLL could use a static runtime, or no runtime at all.
Put the actual functionality (and in particular all code that needs to share a runtime with the Python extensions) in a separate ("primary") DLL and put this DLL, as well as the runtime of your choice, in the same directory as the Python extensions.
The primary DLL and the runtime could be zipped to the plug-in DLL, along with the core extensions. (I presume this is within the scope of the runtime's redistributable license, since it's basically the same as the way most installers work, but you should check for yourself.)
The calls to the plug-in will be slightly less efficient since they have to go through the proxy DLL, but the performance difference probably won't be measurable. :-)
Another option (in my humble opinion, much better) would be to dynamically link - aka. LoadLibrary/GetProcAddress) the dependencies when the main starts. This would allow you to support cases of missing DLLs and/or support your own internal version scheme.
Related
To my astonishment I found VB6 code that uses Declare statements to define functions in a .dll that lives in the Program Folder without it being registered on Windows. This seems like a supersimple way to avoid the .dll hell without having to resort to using Side by side manifests. Can I read some more about this somewhere? Are there snags?
The Declare statement is used to do "just in time" binding to non-ActiveX DLLs. Until your program "touches" a Declared entrypoint no attempt is made to load the library.
It basically has nothing at all to do with the topic of DLL Hell.
Muddled thinking can even lead people to plop ActiveX DLLs "next to" the EXE which actually can result in DLL Hell because people who tend to do this also use poor techniques for installing and uninstalling applications.
Poorly designed application A deployment plops a commonly shared DLL or OCX next to the EXE.
Poorly designed application is run, the VB6 runtime can't find the classes in the registry, does a DLL Search using Windows heuristics, immediately locates the DLL next to the EXE and calls its self-registration entrypoint.
Innocent, properly designed applications B, C, D are later installed that use the same DLL/OCX and their installers find the library already registered.
Poorly designed application A is uninstalled, typically by simply deleteing its folder in Program Files.
Applications B, C, and D (and any future applications using the library) are now broken - due to orphaned component registration pointing to a non-existant library.
Moral of the story:
Never, never, never put DLLs "next to" your VB6 application on installation. If you have private DLLs that are not shared with other applications even then put them into a libs, etc. folder under the application folder. Possible exception might be non-COM DLLs, such as those you expect to use Declare with.
There is also a great deal of misunderstanding about manifests, of which there are multiple kinds. The ones you are probably thinking of are application and assembly manifests.
These can be used for selecting among different versions of a library installed Side by Side, or they can be used to isolate applications and assemblies which is the part that has bearing on DLL Hell.
Of course application manifests can be used to specify quite a few other things about how Windows should run the application.
Windows searches in a well-documented sequence of folders for LoadLibrary (which VB6 uses behind the scenes to resolve Declare declarations). Since the first location on the list of search folders is the app's own folder, your discovery makes perfect sense.
It doesn't resolve the "DLL hell" issue for the most part, though. It can't work for system DLLs, for instance, because Windows preloads most of them. Also, if a DLL is already loaded into memory, Windows may use that copy of the DLL (not sharing data, but code can be reused).
That's part of the reason that manifests were created; they allow an application to strictly define required versions of system DLLs in order to provide certain functionality. VB6's technique is old fashioned (just like VB6).
Background
I'm maintaining a plugin for an application. I'm Using Visual C++ 2003.
The plugin is composed of several DLLs - there's the main DLL, that's the one that the application loads using LoadLibrary, and there are several utility DLLs that are used by the main DLL and by each other.
Dependencies generally look like this:
plugin.dll -> utilA.dll, utilB.dll
utilA.dll -> utilB.dll
utilB.dll -> utilA.dll, utilC.dll
You get the picture.
Some of the dependencies between the DLLs are load-time and some run-time.
All the DLL files are stored in the executable's directory (not a requirement, just how it works now).
The problem
There's a new requirement - running multiple instances of the plugin within the application.
The application runs each instance of a plugin in its own thread, i.e. each thread calls functions exported by plugin.dll. The plugin's code, however, is anything but thread-safe - lots of global variables etc..
Unfortunately, fixing the whole thing isn't currently an option, so I need a way to load multiple (at most 3) copies of the plugin's DLLs in the same process.
Option 1: The distinct names approach
Creating 3 copies of each DLL file, so that each file has a distinct name. e.g. plugin1.dll, plugin2.dll, plugin3.dll, utilA1.dll, utilA2.dll, utilA3.dll, utilB1.dll, etc.. The application will load plugin1.dll, plugin2.dll and plugin3.dll. The files will be in the executable's directory.
For each group of DLLs to know each other by name (so the inter-dependencies work), the names need to be known at compilation time - meaning the DLLs need to be compiled multiple times, only each time with different output file names.
Not very complicated, but I'd hate having 3 copies of the VS project files, and don't like having to compile the same files over and over.
Option 2: The side-by-side assemblies approach
Creating 3 copies of the DLL files, each group in its own directory, and defining each group as an assembly by putting an assembly manifest file in the directory, listing the plugin's DLLs.
Each DLL will have an application manifest pointing to the assembly, so that the loader finds the copies of the utility DLLs that reside in the same directory. The manifest needs to be embedded for it to be found when a DLL is loaded using LoadLibrary. I'll use mt.exe from a later VS version for the job, since VS2003 has no built-in manifest embedding support.
I've tried this approach with partial success - dependencies are found during load-time of the DLLs, but not when a DLL function is called that loads another DLL.
This seems to be the expected behavior according to this article - A DLL's activation context is only used at the DLL's load-time, and afterwards it's deactivated and the process's activation context is used.
Edit: Works with ISOLATION_AWARE_ENABLED as expected - runtime loading of DLLs uses the original activation context of the loading DLL.
Questions
Got any other options? Any quick & dirty solution will do. :-)
Will ISOLATION_AWARE_ENABLED even work with VS2003? Edit: It does.
Comments will be greatly appreciated.
Thanks!
ISOLATION_AWARE_ENABLED is implemented by the Windows SDK header files and thus probably wont worth with VS2003 at all. However, it is possible to download the latest Windows 7 SDK and use that with VS2003.
You don't need to use MT to link in manifests. Manifests can be embedded as resources in environments that dont have explicit knowledge.
Add the following to a dll's .rc file to embed a manifest. (With a recent enough platform sdk RT_MANIFEST should already be defined):
#define RT_MANIFEST 24
#define APP_MANIFEST 1
#define DLL_MANIFEST 2
DLL_MANIFEST RT_MANIFEST dllName.dll.embed.manifest
Maybe that's silly, but I'm confused again when trying to find appropriate names for my projects.
I have a solution with two projects. First project is a library, which will be used by other libraries in the future, for creating plugins for my app. Second project is a exe, which uses first project to create GUI and do some stuff, eg. load plugins.
In similar situation, when one project is a dll and second is an exe, how do you choose names for each project?
Lets say, that I would like to peek a name GraphDemonstration. The exe should be GraphDemonstrationGUI and dll - GraphDemonstration? But exe is not just a GUI. So maybe GraphDemonstration for exe and GraphDemonstrationLib for dll?
I have had similar situation in the past. Maybe there is some kind of naming convention? I don't even know if my "problem" is related to VS only.
GraphDemonstration for your EXE is fine because it describes the purpose of the EXE. However, you said the library will be used by other projects later so the "GraphDemonstration" part of GraphDemonstrationLibrary may not be the best choice. Perhaps GraphLibrary, or GraphHelper, or maybe more specific depending on what is actually included in the library. For example, if the library handles instantiating an Excel object to generate a graph you could call it ExcelGraphHelper.dll.
If this is library of main class CSomeClass I name project SomeClassLibrary.
For projects smillary but non from main object.
I've got a project that depends on a particular version of MSVCR80.dll (the MS Visual C Runtime) and I'm running into problems where, depending on the particular system configuration, my app doesn't always get the right version of that file. It's been a bit of a crap shoot as to what path it takes to find a file with that name, and it's not always right...
Is there a way, when creating a Deployment Project in VS2005, to ensure that my app will always use the runtime that I provided?? When I add the runtime file to the project, it asks about creating a merge module...but not really sure what that does. And regardless of creating one, the issue remains.
Martin Richter wrote an article about that on CodeProject:
Create projects easily with private MFC, ATL and CRT assemblies
This solution does not rely on your MSI packages but on the application that uses the CRT files.
I am not sure if it is your application after installation that doesn't work, or if it is a dll you use as part of the installation that doesn't work?
To make a very long story, very short: new versions of the C / C++ runtimes are installed as Win32 assemblies, or side-by-side installation. This means the files will go into folders under C:\Windows\winsxs - the Win32 equivalent of the GAC, and several versions of the same file can co-exist here.
Applications compiled with Visual Studio 2005 / 2008 will put a manifest file into the binary, and this manifest specifies what side-by-side runtime version to bind to. It doesn't matter if you put the MSVCR80.dll next to your EXE or even in system32 - the manifest embedded in the EXE will load the file from C:\Windows\winsxs.
This is all "full circle". In the old days runtimes went to System32. This caused the original dll-hell: applications overwriting each other's global runtime files. To remedy all this the idea was to "isolate changes" to each application. Hence the new approach was to isolate a local copy of the runtime file next to the EXE. Now this caused an entirely new problem: how do you make sure security updates for the isolated dll was deployed? In most cases this never happened, and you had lots of applications running with local, unsafe dll's. So what to do? The decision was to introduce the second coming of dll-hell: the side-by-side assembly approach. In this approach runtimes are not local, but global - with the critical difference of supporting side-by-side installations. This way, in theory, applications can function without overwriting each other's runtime dlls.
So that was the quick summary of "how to make runtime deployment complicated". I am not positive it is still possible to do, but did you check whether you can statically link to the runtime? Sometimes old-school really is easier...
How exactly do DLL files work? There seems to be an awful lot of them, but I don't know what they are or how they work.
So, what's the deal with them?
What is a DLL?
Dynamic Link Libraries (DLL)s are like EXEs but they are not directly executable. They are similar to .so files in Linux/Unix. That is to say, DLLs are MS's implementation of shared libraries.
DLLs are so much like an EXE that the file format itself is the same. Both EXE and DLLs are based on the Portable Executable (PE) file format. DLLs can also contain COM components and .NET libraries.
What does a DLL contain?
A DLL contains functions, classes, variables, UIs and resources (such as icons, images, files, ...) that an EXE, or other DLL uses.
Types of libraries:
On virtually all operating systems, there are 2 types of libraries. Static libraries and dynamic libraries. In windows the file extensions are as follows: Static libraries (.lib) and dynamic libraries (.dll). The main difference is that static libraries are linked to the executable at compile time; whereas dynamic linked libraries are not linked until run-time.
More on static and dynamic libraries:
You don't normally see static libraries though on your computer, because a static library is embedded directly inside of a module (EXE or DLL). A dynamic library is a stand-alone file.
A DLL can be changed at any time and is only loaded at runtime when an EXE explicitly loads the DLL. A static library cannot be changed once it is compiled within the EXE.
A DLL can be updated individually without updating the EXE itself.
Loading a DLL:
A program loads a DLL at startup, via the Win32 API LoadLibrary, or when it is a dependency of another DLL. A program uses the GetProcAddress to load a function or LoadResource to load a resource.
Further reading:
Please check MSDN or Wikipedia for further reading. Also the sources of this answer.
What is a DLL?
DLL files are binary files that can contain executable code and resources like images, etc. Unlike applications, these cannot be directly executed, but an application will load them as and when they are required (or all at once during startup).
Are they important?
Most applications will load the DLL files they require at startup. If any of these are not found the system will not be able to start the process at all.
DLL files might require other DLL files
In the same way that an application requires a DLL file, a DLL file might be dependent on other DLL files itself. If one of these DLL files in the chain of dependency is not found, the application will not load. This is debugged easily using any dependency walker tools, like Dependency Walker.
There are so many of them in the system folders
Most of the system functionality is exposed to a user program in the form of DLL files as they are a standard form of sharing code / resources. Each functionality is kept separately in different DLL files so that only the required DLL files will be loaded and thus reduce the memory constraints on the system.
Installed applications also use DLL files
DLL files also becomes a form of separating functionalities physically as explained above. Good applications also try to not load the DLL files until they are absolutely required, which reduces the memory requirements. This too causes applications to ship with a lot of DLL files.
DLL Hell
However, at times system upgrades often breaks other programs when there is a version mismatch between the shared DLL files and the program that requires them. System checkpoints and DLL cache, etc. have been the initiatives from M$ to solve this problem. The .NET platform might not face this issue at all.
How do we know what's inside a DLL file?
You have to use an external tool like DUMPBIN or Dependency Walker which will not only show what publicly visible functions (known as exports) are contained inside the DLL files and also what other DLL files it requires and which exports from those DLL files this DLL file is dependent upon.
How do we create / use them?
Refer the programming documentation from your vendor. For C++, refer to LoadLibrary in MSDN.
Let’s say you are making an executable that uses some functions found in a library.
If the library you are using is static, the linker will copy the object code for these functions directly from the library and insert them into the executable.
Now if this executable is run it has every thing it needs, so the executable loader just loads it into memory and runs it.
If the library is dynamic the linker will not insert object code but rather it will insert a stub which basically says this function is located in this DLL at this location.
Now if this executable is run, bits of the executable are missing (i.e the stubs) so the loader goes through the executable fixing up the missing stubs. Only after all the stubs have been resolved will the executable be allowed to run.
To see this in action delete or rename the DLL and watch how the loader will report a missing DLL error when you try to run the executable.
Hence the name Dynamic Link Library, parts of the linking process is being done dynamically at run time by the executable loader.
One a final note, if you don't link to the DLL then no stubs will be inserted by the linker, but Windows still provides the GetProcAddress API that allows you to load an execute the DLL function entry point long after the executable has started.
DLLs (dynamic link libraries) and SLs (shared libraries, equivalent under UNIX) are just libraries of executable code which can be dynamically linked into an executable at load time.
Static libraries are inserted into an executable at compile time and are fixed from that point. They increase the size of the executable and cannot be shared.
Dynamic libraries have the following advantages:
1/ They are loaded at run time rather than compile time so they can be updated independently of the executable (all those fancy windows and dialog boxes you see in Windows come from DLLs so the look-and-feel of your application can change without you having to rewrite it).
2/ Because they're independent, the code can be shared across multiple executables - this saves memory since, if you're running 100 apps with a single DLL, there may only be one copy of the DLL in memory.
Their main disadvantage is advantage #1 - having DLLs change independent your application may cause your application to stop working or start behaving in a bizarre manner. DLL versioning tend not to be managed very well under Windows and this leads to the quaintly-named "DLL Hell".
DLL files contain an Export Table which is a list of symbols which can be looked up by the calling program. The symbols are typically functions with the C calling convention (__stcall). The export table also contains the address of the function.
With this information, the calling program can then call the functions within the DLL even though it did not have access to the DLL at compile time.
Introducing Dynamic Link Libraries has some more information.
http://support.microsoft.com/kb/815065
A DLL is a library that contains code
and data that can be used by more than
one program at the same time. For
example, in Windows operating systems,
the Comdlg32 DLL performs common
dialog box related functions.
Therefore, each program can use the
functionality that is contained in
this DLL to implement an Open dialog
box. This helps promote code reuse and
efficient memory usage.
By using a DLL, a program can be
modularized into separate components.
For example, an accounting program may
be sold by module. Each module can be
loaded into the main program at run
time if that module is installed.
Because the modules are separate, the
load time of the program is faster,
and a module is only loaded when that
functionality is requested.
Additionally, updates are easier to
apply to each module without affecting
other parts of the program. For
example, you may have a payroll
program, and the tax rates change each
year. When these changes are isolated
to a DLL, you can apply an update
without needing to build or install
the whole program again.
http://en.wikipedia.org/wiki/Dynamic-link_library
DLL is a File Extension & Known As “dynamic link library” file format used for holding multiple codes and procedures for Windows programs. Software & Games runs on the bases of DLL Files; DLL files was created so that multiple applications could use their information at the same time.
IF you want to get more information about DLL Files or facing any error read the following post.
https://www.bouncegeek.com/fix-dll-errors-windows-586985/
DLLs (Dynamic Link Libraries) contain resources used by one or more applications or services. They can contain classes, icons, strings, objects, interfaces, and pretty much anything a developer would need to store except a UI.
According to Microsoft
(DLL) Dynamic link libraries are files that contain data, code, or resources needed for the running of applications. These are files that are created by the windows ecosystem and can be shared between two or more applications.
When a program or software runs on Windows, much of how the application works depends on the DLL files of the program. For instance, if a particular application had several modules, then how each module interacts with each other is determined by the Windows DLL files.
If you want detailed explanation, check these useful resources
What are dll files , About Dll files