OK, so I have successfully set up my Private Assembly for my plugins of my executable, like described here: (This is just an example, my use case is technically the same, but not for plugins as such.)
Executable uses application manifest to declare dependentAssembly on plugins assembly
plugins subdirectory contains plugins.manifest
plugins.manifest declares assemblyIdentity with files listing the plugin DLL(s) (say, easyplug1.dll and xplug2.dll)
What I don't quite get, and haven't yet found any direct explanation, is what about DLLs that the DLLs in the Private Assembly depend upon? Where should they be located, how are they found?
Example: The xplug2.dll, which is 3rd party itself needs the 3rd party dll xbase.dll. Naturally, xbase.dll is simply also deployed in the plugins directory. Is this enough? Does the Private Assembly need to list DLLs that are only used transitively by the (executable) module declaring its dependency on the plugins assembly?
Partial answer so far by trial and error:
(A) It seems all DLLs, also those only used from other "plugin" DLLs must be listed in the Private Assembly manifest -- i.e. essentially one should list all DLL files in the plugins assembly directory.
(A.1) Any not-listed DLL will not be found in the directory, even if it's referenced from a DLL within the same subdirectory.
(A.2) I assume this to be due to all the DLL loading machinery using the (only) custom Activation Context from the executable, and if the DLL is not found there, it seems to search from within the executable directory.
(A.3) I tried looking at the dependency chain with the latest version of Dependency Walker (2.2.10011), but it seems it can't handle this transitive scenario at all: Primary dependencies are are shown correctly, but transitive secondary dependencies are not resolved from the manifest, but relative to the base directory. (Seems it follows plain LoadLibrary semantics for the transient ones.)
File layout:
...\base\app.exe
// - contains Application Manifest declaring dependency on `plugins`
// - Loads xplug2.dll
...\base\plugins\xplug2.dll
// - Loads xbase.dll (which is only used by xplug2.dll internally)
...\base\plugins\xbase.dll
...\base\plugins\plugins.manifest
// - lists xplug2.dll (for sure)
// - *seems* to need to list also xbase.dll
The simple answer is, any "raw" dlls that are not part of an assembly, will be searched for using the applications activation context and/or default dll search rules. And the dll search path includes the application folder, but not any folders that dll's were found in (i.e. a dll can never expect to load another dll that is incidentally in the same folder).
So, for example, if a dll in your assembly required openssl.dll, then it would search first in the application.exe's folder, NOT your private assembly folder unless you specifically, explicitly, added openssl to your private assembly.
the answer is to explicitly add openssl.dll to the private assembly (which incidentally helps protect the integrity of the application should a different dll - in a different assembly - require a different version of openssl.dll)
In your case, you could have two different private assemblies, each using a private version of xbase.dll - where the two different xbase.dlls could be quite incompatible.
Related
Are there any tools or methods for renaming a DLL's dependencies?
I'm working with a large program that uses a significant number of third party dependencies. I'm able to compile nearly all of these dependencies into DLLs with a specific naming convention (i.e. libSomething_x86.dll). However, there is one dependency (FFmpeg) that I can't easily compile, but pre-compiled DLLs are available. So far, I've renamed the DLL files and recreated the import libraries to use the new names. My application compiles, but fails to run because FFmpeg is composed of multiple DLLs and they reference each other (with the original names). Since the new DLL names are longer than the original DLL names, I can't simply hex edit the DLL files. I can't use side-by-side redirection, manifest tricks, or "wrapper"/"dummy" DLLs because the fact the DLLs are pre-compiled and renamed must be completely transparent to the other application developers.
I was hoping to find a method of unlinking the DLL, editing the dependency list, and then relinking the DLL. Is this possible?
Thank you.
We have developed a library in C#, and now I wish to create a project template to aid in using the library correctly.
I want new projects to include a reference to the library assembly, but would prefer not to have to deploy the assembly to the GAC, or to depend on the assembly residing in some specific location.
What I am thinking is to include the .dll in the project template .zip file. That means it will end up somewhere inside the project folder of new projects. Perhaps in a folder named Lib. Then the reference hint in the project file can point to that folder. Is that a good idea? What problems might I face down the road?
Is there perhaps some mechanism for including such 3rd party libraries in project templates that I'm not aware of? How have you tackled this? Surely I'm not the first.
I have had to address this issue in the past. In one case, it was a logging library that was installed to the GAC, which meant the Reference element simply needed the assembly name. In another case, we installed the library to the file system, created a registry key that contained the location (in case the user got cute and changed the install location on us) and used a project template wizard to look up the registry key and populate a replacement item to have the correct location in the Reference's HintPath. (Note: the template wizard approach requires you to install your wizard's assembly to the GAC, which it sounds like you're trying to avoid...)
If you don't want your library to be installed in either the GAC or a specific location, the approach of including the assembly in the project is pretty much your only remaining option. On the positive side, deployment of your project template is fairly straightforward and you don't have to muck with the GAC, custom wizards, etc. On the negative side, if you ever create a new revision of your library, your users will need to update every project's copy of the library.
What is the difference between application manifest and assembly manifest? Where is each one used? Which one of the two is found in .dll or .exe resources? (or both can be there? ).
Sorry if its too many questions at once, but if anyone can explain this to me it would be really helpful. The reason i'm asking this is that i want to be able to extract information from manifests embedded in PE files. I found these descriptions of manifests, but there are two and i'm not sure which one to follow:
Application manifest description
Assembly manifest description
EDIT: and no, i do not want to use any API calls. I'm writting it all myself.
In brief, the two are completely separate concepts with unfortunately similar names.
An application manifest is an XML file embedded in, or distributed along with, a PE binary (managed or native), giving instructions to the OS loader about things such as SxS assembly dependencies, required elevation, OS version compatibility, etc.
An assembly manifest is a section in a CLI assembly, stating the managed assembly's dependencies, the files making up the assembly, the assembly's public key, type exports, CLR flags, and so on. You can inspect an assembly's manifest using ILDASM.exe or most .NET decompilers.
A relevant excerpt from ECMA 335 (the CLI specification), section I.9.6:
Manifests: Every assembly has a manifest that declares which files
make up the assembly, what types are exported, and what other
assemblies are required to resolve type references within the
assembly. Just as CLI components are self-describing via metadata in
the CLI component, so are assemblies self-describing via their
manifests. When a single file makes up an assembly it contains both
the metadata describing the types defined in the assembly and the
metadata describing the assembly itself. When an assembly contains
more than one file with metadata, each of the files describes the
types defined in the file, if any, and one of these files also
contains the metadata describing the assembly (including the names of
the other files, their cryptographic hashes, and the types they export
outside of the assembly).
Note that:
all managed assemblies must have assembly manifests, executables and libraries alike;
all native binaries, executables and libraries, may have application manifests.
It seems that i will have to follow both:
As a resource in a DLL, the assembly
is available for the private use of
the DLL. An assembly manifest cannot
be included as a resource in an EXE.
An EXE file may include an application
manifest as a resource.
(Information found here). So it seems that executables have application manifests embedded in resources and libraries (DLL) have assembly manifests. Since both of them are PE (portable executable), i will need to parse both types.
Application manifests are typically embedded in EXEs and, ironically, Dlls, and specify which assemblies the EXE or DLL is dependent upon.
Assembly manifests can be embedded in DLLs, or be on disk as a separate file, and give an assembly an identity, and a list of resources: being dll's, activation free com objects, and window classes.
If the name of the assembly is the name of a dll, then the same manifest ends up being used as both an application manifest to determine the dependencies of the dll, and an assembly manifest to see what the assembly exports. This option really just seems crazy for native assemblies, its usually better to create an assembly with a complicated name along the lines of company.product.module, and then just have a simple module.dll as its one entry.
Background
I'm maintaining a plugin for an application. I'm Using Visual C++ 2003.
The plugin is composed of several DLLs - there's the main DLL, that's the one that the application loads using LoadLibrary, and there are several utility DLLs that are used by the main DLL and by each other.
Dependencies generally look like this:
plugin.dll -> utilA.dll, utilB.dll
utilA.dll -> utilB.dll
utilB.dll -> utilA.dll, utilC.dll
You get the picture.
Some of the dependencies between the DLLs are load-time and some run-time.
All the DLL files are stored in the executable's directory (not a requirement, just how it works now).
The problem
There's a new requirement - running multiple instances of the plugin within the application.
The application runs each instance of a plugin in its own thread, i.e. each thread calls functions exported by plugin.dll. The plugin's code, however, is anything but thread-safe - lots of global variables etc..
Unfortunately, fixing the whole thing isn't currently an option, so I need a way to load multiple (at most 3) copies of the plugin's DLLs in the same process.
Option 1: The distinct names approach
Creating 3 copies of each DLL file, so that each file has a distinct name. e.g. plugin1.dll, plugin2.dll, plugin3.dll, utilA1.dll, utilA2.dll, utilA3.dll, utilB1.dll, etc.. The application will load plugin1.dll, plugin2.dll and plugin3.dll. The files will be in the executable's directory.
For each group of DLLs to know each other by name (so the inter-dependencies work), the names need to be known at compilation time - meaning the DLLs need to be compiled multiple times, only each time with different output file names.
Not very complicated, but I'd hate having 3 copies of the VS project files, and don't like having to compile the same files over and over.
Option 2: The side-by-side assemblies approach
Creating 3 copies of the DLL files, each group in its own directory, and defining each group as an assembly by putting an assembly manifest file in the directory, listing the plugin's DLLs.
Each DLL will have an application manifest pointing to the assembly, so that the loader finds the copies of the utility DLLs that reside in the same directory. The manifest needs to be embedded for it to be found when a DLL is loaded using LoadLibrary. I'll use mt.exe from a later VS version for the job, since VS2003 has no built-in manifest embedding support.
I've tried this approach with partial success - dependencies are found during load-time of the DLLs, but not when a DLL function is called that loads another DLL.
This seems to be the expected behavior according to this article - A DLL's activation context is only used at the DLL's load-time, and afterwards it's deactivated and the process's activation context is used.
Edit: Works with ISOLATION_AWARE_ENABLED as expected - runtime loading of DLLs uses the original activation context of the loading DLL.
Questions
Got any other options? Any quick & dirty solution will do. :-)
Will ISOLATION_AWARE_ENABLED even work with VS2003? Edit: It does.
Comments will be greatly appreciated.
Thanks!
ISOLATION_AWARE_ENABLED is implemented by the Windows SDK header files and thus probably wont worth with VS2003 at all. However, it is possible to download the latest Windows 7 SDK and use that with VS2003.
You don't need to use MT to link in manifests. Manifests can be embedded as resources in environments that dont have explicit knowledge.
Add the following to a dll's .rc file to embed a manifest. (With a recent enough platform sdk RT_MANIFEST should already be defined):
#define RT_MANIFEST 24
#define APP_MANIFEST 1
#define DLL_MANIFEST 2
DLL_MANIFEST RT_MANIFEST dllName.dll.embed.manifest
How exactly do DLL files work? There seems to be an awful lot of them, but I don't know what they are or how they work.
So, what's the deal with them?
What is a DLL?
Dynamic Link Libraries (DLL)s are like EXEs but they are not directly executable. They are similar to .so files in Linux/Unix. That is to say, DLLs are MS's implementation of shared libraries.
DLLs are so much like an EXE that the file format itself is the same. Both EXE and DLLs are based on the Portable Executable (PE) file format. DLLs can also contain COM components and .NET libraries.
What does a DLL contain?
A DLL contains functions, classes, variables, UIs and resources (such as icons, images, files, ...) that an EXE, or other DLL uses.
Types of libraries:
On virtually all operating systems, there are 2 types of libraries. Static libraries and dynamic libraries. In windows the file extensions are as follows: Static libraries (.lib) and dynamic libraries (.dll). The main difference is that static libraries are linked to the executable at compile time; whereas dynamic linked libraries are not linked until run-time.
More on static and dynamic libraries:
You don't normally see static libraries though on your computer, because a static library is embedded directly inside of a module (EXE or DLL). A dynamic library is a stand-alone file.
A DLL can be changed at any time and is only loaded at runtime when an EXE explicitly loads the DLL. A static library cannot be changed once it is compiled within the EXE.
A DLL can be updated individually without updating the EXE itself.
Loading a DLL:
A program loads a DLL at startup, via the Win32 API LoadLibrary, or when it is a dependency of another DLL. A program uses the GetProcAddress to load a function or LoadResource to load a resource.
Further reading:
Please check MSDN or Wikipedia for further reading. Also the sources of this answer.
What is a DLL?
DLL files are binary files that can contain executable code and resources like images, etc. Unlike applications, these cannot be directly executed, but an application will load them as and when they are required (or all at once during startup).
Are they important?
Most applications will load the DLL files they require at startup. If any of these are not found the system will not be able to start the process at all.
DLL files might require other DLL files
In the same way that an application requires a DLL file, a DLL file might be dependent on other DLL files itself. If one of these DLL files in the chain of dependency is not found, the application will not load. This is debugged easily using any dependency walker tools, like Dependency Walker.
There are so many of them in the system folders
Most of the system functionality is exposed to a user program in the form of DLL files as they are a standard form of sharing code / resources. Each functionality is kept separately in different DLL files so that only the required DLL files will be loaded and thus reduce the memory constraints on the system.
Installed applications also use DLL files
DLL files also becomes a form of separating functionalities physically as explained above. Good applications also try to not load the DLL files until they are absolutely required, which reduces the memory requirements. This too causes applications to ship with a lot of DLL files.
DLL Hell
However, at times system upgrades often breaks other programs when there is a version mismatch between the shared DLL files and the program that requires them. System checkpoints and DLL cache, etc. have been the initiatives from M$ to solve this problem. The .NET platform might not face this issue at all.
How do we know what's inside a DLL file?
You have to use an external tool like DUMPBIN or Dependency Walker which will not only show what publicly visible functions (known as exports) are contained inside the DLL files and also what other DLL files it requires and which exports from those DLL files this DLL file is dependent upon.
How do we create / use them?
Refer the programming documentation from your vendor. For C++, refer to LoadLibrary in MSDN.
Let’s say you are making an executable that uses some functions found in a library.
If the library you are using is static, the linker will copy the object code for these functions directly from the library and insert them into the executable.
Now if this executable is run it has every thing it needs, so the executable loader just loads it into memory and runs it.
If the library is dynamic the linker will not insert object code but rather it will insert a stub which basically says this function is located in this DLL at this location.
Now if this executable is run, bits of the executable are missing (i.e the stubs) so the loader goes through the executable fixing up the missing stubs. Only after all the stubs have been resolved will the executable be allowed to run.
To see this in action delete or rename the DLL and watch how the loader will report a missing DLL error when you try to run the executable.
Hence the name Dynamic Link Library, parts of the linking process is being done dynamically at run time by the executable loader.
One a final note, if you don't link to the DLL then no stubs will be inserted by the linker, but Windows still provides the GetProcAddress API that allows you to load an execute the DLL function entry point long after the executable has started.
DLLs (dynamic link libraries) and SLs (shared libraries, equivalent under UNIX) are just libraries of executable code which can be dynamically linked into an executable at load time.
Static libraries are inserted into an executable at compile time and are fixed from that point. They increase the size of the executable and cannot be shared.
Dynamic libraries have the following advantages:
1/ They are loaded at run time rather than compile time so they can be updated independently of the executable (all those fancy windows and dialog boxes you see in Windows come from DLLs so the look-and-feel of your application can change without you having to rewrite it).
2/ Because they're independent, the code can be shared across multiple executables - this saves memory since, if you're running 100 apps with a single DLL, there may only be one copy of the DLL in memory.
Their main disadvantage is advantage #1 - having DLLs change independent your application may cause your application to stop working or start behaving in a bizarre manner. DLL versioning tend not to be managed very well under Windows and this leads to the quaintly-named "DLL Hell".
DLL files contain an Export Table which is a list of symbols which can be looked up by the calling program. The symbols are typically functions with the C calling convention (__stcall). The export table also contains the address of the function.
With this information, the calling program can then call the functions within the DLL even though it did not have access to the DLL at compile time.
Introducing Dynamic Link Libraries has some more information.
http://support.microsoft.com/kb/815065
A DLL is a library that contains code
and data that can be used by more than
one program at the same time. For
example, in Windows operating systems,
the Comdlg32 DLL performs common
dialog box related functions.
Therefore, each program can use the
functionality that is contained in
this DLL to implement an Open dialog
box. This helps promote code reuse and
efficient memory usage.
By using a DLL, a program can be
modularized into separate components.
For example, an accounting program may
be sold by module. Each module can be
loaded into the main program at run
time if that module is installed.
Because the modules are separate, the
load time of the program is faster,
and a module is only loaded when that
functionality is requested.
Additionally, updates are easier to
apply to each module without affecting
other parts of the program. For
example, you may have a payroll
program, and the tax rates change each
year. When these changes are isolated
to a DLL, you can apply an update
without needing to build or install
the whole program again.
http://en.wikipedia.org/wiki/Dynamic-link_library
DLL is a File Extension & Known As “dynamic link library” file format used for holding multiple codes and procedures for Windows programs. Software & Games runs on the bases of DLL Files; DLL files was created so that multiple applications could use their information at the same time.
IF you want to get more information about DLL Files or facing any error read the following post.
https://www.bouncegeek.com/fix-dll-errors-windows-586985/
DLLs (Dynamic Link Libraries) contain resources used by one or more applications or services. They can contain classes, icons, strings, objects, interfaces, and pretty much anything a developer would need to store except a UI.
According to Microsoft
(DLL) Dynamic link libraries are files that contain data, code, or resources needed for the running of applications. These are files that are created by the windows ecosystem and can be shared between two or more applications.
When a program or software runs on Windows, much of how the application works depends on the DLL files of the program. For instance, if a particular application had several modules, then how each module interacts with each other is determined by the Windows DLL files.
If you want detailed explanation, check these useful resources
What are dll files , About Dll files