Does anyone know a good tool for compression files using Windows API function called RtlCompressBuffer? I want to compress an executable using this method.
RtlCompressBuffer is a fairly low-level tool. You'd normally use it to create some higher-level tool before actually compressing an executable.
Windows has a couple such higher level tools, depending on exactly what you want to accomplish. If you want an executable that's stored on a file system in compressed form, then you probably want to use NTFS compression. To compress your executable with this, you call DeviceIoControl with the FSCTL_SET_COMPRESSION flag.
If you want to compress a file for distribution (e.g., so a user will be able to download it faster) you normally want to put it into a cabinet file, which you might then ship by itself, or (generally preferred) package it up into an MSI package. As far as pre-packaged tools to do this, you'd be looking at Microsoft's cabarc. If you want to do the job in your own code, you can use the File Compression Interface (FCI).
Obligatory side note: although both of these do compression (and also support matching decompression), no I'm not really sure whether either (not to mention both) is actually implemented using RtlCompressBuffer. I don't believe Microsoft's documentation specifies their implementation in nearly that level of detail.
Related
I am wondering if there is a way of version control in vivado for a VHDL project.
One way would be to add the version number to the bitstream file name.
Would that be possible?
What other options are there if that isn't possible?
Thanks in advance
Think about what you require
Version information format and size: What is the version? Is a natural number? String? Real? Do you save date or time stamp? Do you need a build number? Need extra space for sub-unit versions?
Version source and design flow: What sets the version? Is it a source control tool? Do you manually set the version? Do you need automation to update parts of the version information (timestamp, build number, etc.)?
Version accessibility/observability: Where do you want to read the version number and have it viewable? In the source code? External file? The bitstream file name? The flash file contents? Do you need the FPGA logic to be able to access the version information (to be read via communication channel, embedded processor, etc.)?
Vendor Tools/Device/System/Source dependency: Implementation and automation of version information may depend on several things: Each vendor has different tools (even within the same vendor's). Different devices may allow you to do different things. The system where you work may allow different forms of automation (OS, installed script tools, etc.). The source files (VHDL, Verilog, System C, ...) and directory structure may influence the applicability of your solution.
Automation options
Makefile (builder script) automation
The most flexible way is to have some kind of builder script which may or may not be hooked to your source control mechanism (makefile, perl, python, etc.). Think about the transportability of your project between designers and even design platforms (Windows, Linux, etc.).
Pros:
Complete control of what you want to do.
Can be hooked to any source control automation.
Can be modified to work on any platform when using standard tools (be sure to take special care of file paths which differ between OS).
Cons:
Does not flow well with GUI vendor tools (see Vivado TCL hooks).
Vivado TCL hooks
Vivado has TCL hooks which you can use to run when you synthesize, implement, etc.
Using a pre-synthesis TCL hook will allow you to connect a script to run just before synthesis, and implement your version information update mechanism.
Using a post-bitstream TCL hook will allow you to connect a script to run just after bitstream file creation, and create a file name matching your updated version information, or other final operation you may require.
More information:
http://www.xilinx.com/support/documentation/sw_manuals/xilinx2014_4/ug894-vivado-tcl-scripting.pdf
http://www.xilinx.com/support/documentation/sw_manuals/xilinx2014_4/ug835-vivado-tcl-commands.pdf
Pros:
Works with a GUI flow (automatically takes care of everything when you click on Synthesize).
Cons:
(Vivado) tool dependent
Need to remember to create the hooks in your Vivado project
Implementation options
Source code modification
Version number, build number, timestamp, and basically everything you need can be modified (within the VHDL/Verilog code) automatically using a script which parses a known file format. I've done this with Vivado TCL hooks and a well formatted package VHDL file. Works like a charm.
Note: When updating source files using pre-synthesis TCL hooks, Vivado will think the files were modified after, and will consider the implementation as not updated. I added to the script a command to save the current timestamp of the package file, updated it, and then wrote back the timestamp (meaning the package file will forever hold its system date and time).
Pros:
Version information easily viewable accessible to designer (package file).
Version information easily accessible for logic usage.
Device independent.
Cons:
Version information cannot be read by examining the bitstream file.
Writing the script may take some work and may force you to a standardisation of your project sources directory structure [which may not be a totally bad thing ;) ]
Top level entity/module generic
Generics at the top level design module/entity can be modified by the Synthesis tool and accessed by the design's logic.
Pros:
Less complicated scripting than modifying the source code.
Cons:
Version information cannot be read by examining the bitstream file.
You still need to save your version/build number somewhere (a file).
For VHDL/Verilog, using generics is more complicated than using a package, since you need to wire the generics through your hierarchies until you reach the design code which writes the value to a register.
Synthesis tool dependency. Different tools have different commands to update the top level generic values.
JTAG USERCODE
Most FPGAs support the IEEE Std. 1149.1 JTAG feature USERCODE, which gives the user a 32-bit value to identify the bitstream. This value can be set using the Vendor tools when creating the bitstream file. However, this value cannot be read by the FPGA's logic (unless you implement logic to read the flash and track the correct address for this value, which is a big overkill).
Pros:
Fit for most devices.
Version information is accessible by examining the file.
Cons:
Version information cannot be read by the logic without extreme procedures.
Limited information size (only 32 bits).
Xilinx USR_ACCESS Register
Similar to the JTAG USERCODE, Xilinx have a implemented a USR_ACCESS Register for Virtex-5, Virtex-6, and all 7 series devices. They provided a primitive for VHDL/Verilog to allow you to read the value via logic.
More information here: http://www.xilinx.com/support/documentation/application_notes/xapp497_usr_access.pdf
Pros:
Version information can be read by the design logic.
Version information is accessible by examining the file.
Cons:
Solution for specific devices (not even all Xilinx devices support this).
Limited information size (only 32 bits).
ROM static information
You can implement a version information ROM and use an external file to set its value. (see Verilog : Memory block Instantiation)
Pros:
Can include large version information (even ASCII code).
Version information viewable by the designer (external file).
Fit for most devices and tools which accept the VHDL/Verilog code for ROM.
Cons:
A bit more complicated logic may be required to read all the ROM (depends on how you implement it).
It is difficult (yet possible) to examine version information by examining the file (You need to extract ROM bitstream locations).
Flash image modification
Many systems have their FPGAs load from flash devices. Usually there is more room than just the FPGA bitstream. It is possible to create a large flash image file to include extra information, among which is version information. This is probably to worst option of all I've written above, and mentioned this for completeness.
Pros:
Potentially large information cache.
Flash image file can be examined for version information.
Cons:
Difficult to implement.
Dependent on vendor tools.
Assuming you only have access to the final product (i.e. in form of the exe file), how would you go about finding out which libraries/components the developer used to create the application?
In my specific case the question is about an application developed in VC++ using a few third party components and I'm curious which those are.
But I think the question is generally valid, e.g. when it should be proven if a developer is in line with license requirements of a specific library.
So, what you're saying is that if I suspect that a binary is using a certain library, I could try to map the respective function calls and see if I get a result. But there is no shortcut to this and unless I am willing to try out hundreds of mappings or the dev left some information in some strings or other resources, I have little chance of finding this out. Yes?
There is small shortcut, here's what I'd do:
check executable for strings and constants, and try to find out what library is that.
IF used libraries are open-source, compile them on my own and create FLAIR signatures (IDA Pro).
Use generated flair signatures on target executable.
In some situations, that can really work like a charm and can let you distinguish actual code from used libraries.
The IDA Pro Book - Ch 12. Library Recognition Using FLIRT Signatures
I've got a working proprietary application (windows exe) and would like to know which particular toolkit was used to make it. The reason is that I like the widgets it uses and seek to use same library in my project (to buy it if it's proprietary as well).
Just use Process Explorer to see what DLLs the application has loaded. That will be your widget set. Sort the results by folder to roughly group them by manufacturer. You may need to examine the properties of the DLLs for more detailed info as well.
If the library is statically linked you may have to do some deep looking around, maybe you'll get lucky and find a string saying the name of the library or a class/function in it. You can use OllyDbg for this to view strings loaded at runtime, or something like the linux command strings to look through statically, although that wont work if the program decodes itself at startup. If that doesn't work, you'd have to come up with a list of libraries that do what the one you are looking at does, and find some artifacts in the binary that are common between the two. Anyways, better to check the dlls first like Paul Sasik said.
You can use PEiD to identify the compiler, which can be a hint aswel. PEiD also has a nice process explorer.
For instance, Google Chrome uses C:\WINDOWS\SYSTEM32\IEFRAME.DLL :-) Nice isn't it?
(Don't trust it 100%. For instance, my own compiler has the "Morphine 1.2 - 1.3 -> rootkit" description, which I find quite awkward: that's a packer/compiler trace obfuscator.)
I have a C library, which I build as a shared object for Linux and a DLL for Windows with MinGW32. The API depends on a couple of data files (statistical models) which I'd really like to roll in with the SO/DLL so that deployment is just one file.
It looks like I can achieve this for Windows with a "resource file" compiled with windres, but then I've got to write a bunch of resource-handling code for Windows, and I'm still stuck with the files on Linux.
Is there a way to achieve the same functionality on Linux?
Even better, is there a portable solution?
It's actually quite simple on Linux and other ELF systems: http://www.linuxjournal.com/content/embedding-file-executable-aka-hello-world-version-5967
OS X has bundles, so you just build your library as a framework and put the file in the bundle.
Two potential solutions:
Phong Vo's sfio library, which is part of the AT&T Advanced Software Technology toolset, is a wonderful replacement for C stdio.h, and it will allow you to open either files or memory blocks using a single API. So you can easily convert your existing files to C initialized data to include in your DLL or SO file.
This is a good cross-platform solution, but the penalty is that the learning curve to get started is pretty high. They don't make it easy to figure out how stuff works or to take one part of their toolset and split it out for use independent of the other parts. But the good news is that if you want to adopt their U/Win system for running Unix codes on windows (all part of the same toolset), you can create DLLs and SOs using the same system.
For this kind of problem I often fall back on Lua; I can stored Lua data either in external files or within C as initialized data. This is great for distributing everything in one .so file; I do this for my students.
Again the downside is that you have to master and incorporate a new technology.
In my own work I use Lua over the AT&T stuff for these reasons:
Lua has a much smaller footprint and is designed to play well with others; with AST you really have do adopt their way of doing things.
The learning curve with Lua is much less steep; you can be productive very quickly.
Lua is dead easy to install and it's easy to get information about it. AST has its own quirky installation process shared by nobody else in the world; it's often hard to make the installation work; and it's harder to get information about it.
Using Lua has a lot of other payoffs, so the effort spent learning Lua and learning how to incorporate Lua into C codes is easy to amortize over multiple projects.
I would like to create a file format for my app like Quake, OO, and MS Office 07 have.
Basically a uncompressed zip folder, or tar file.
I need this to be cross platform (mac and windows).
Can I do something via command prompt and bash?
If you want a single file that is portable to all platforms and which contain structured data, consider using sqlite. You'll get a full featured ACID compliant database that exists on disk as a single file.
There are libraries you can link against to directly access the file, and there is a command line tool you can use as well. No matter what language you are using, most likely there is support for it.
http://www.sqlite.org
Have a look at the open source 7Zip compression format. For your specific needs, you can use it in an "Archive" mode, zero compression but very fast.
It provides a powerful SDK, LZMA, from the site:
"LZMA is the default and general compression method of 7z format in the 7-Zip program. LZMA provides a high compression ratio and very fast decompression, so it is very suitable for embedded applications. For example, it can be used for ROM (firmware) compressing.
The LZMA SDK provides the documentation, samples, header files, libraries, and tools you need to develop applications that use LZMA compression."
Zip is supported everywhere. If a container is all you need, than those are surely good options.
SQLite is great.
A single file, crossplatform, a tiny library, SQL access to data, transactions, the whole enchilada.
you can use transactions to guarantee consistent return points in case of crashing. check uses for sqlite, they specifically advocate using it as a data model layer for desktop applications.
also, there's a command-line tool to manually access the data.
First thing you should ask yourself is, "Do I really need to make my own?"
Depending on what you want to use it for, you are probably better off using a common format and some pre-made libraries which already handle one of those formats very well.
Good places to start:
http://www.destructor.de/libtar/index.htm (tar -- a the 'container' format)
http://www.zlib.net/ (zlib -- a method of compressing data before or after you put it in the container)
If you still really think you need to make your own, I would suggest studying something very simple first, like tar's format:
http://en.wikipedia.org/wiki/Tar_(file_format)
or
http://schmidt.devlib.org/file-formats/tar-archive-file-format.html
Instead of making a format, I'd just decide on a convention. One or more named files within the container have the metadata you need to access the rest of the files, and know what to do with them. The container itself, though, should just be some ubiquitous format, such as zip. No need to reinvent the wheel, here.