I would like to create a mechanism by which I automatically submit the executable corresponding to the updated code. My development environment is Visual Studio 2005 and I use the windows GUI client to submit changes.
Ideally, I would like to ensure latest binary is submitted in the same change list as the code changes.
My requirements are:
The code should be recompiled and binary should be checked out when source code dependencies change.
Related documentation / non-source files do not affect binary
Perforce should complain when sources are changed and submitted but the binary isn't included in the change list.
How could I enforce these rules? Is it possible to create a P4 script that runs before each submission? Also, is it possible to write a Visual Studio script that automatically checks out the executable before compiling?
Thanks,
Shahar
I think you can accomplish most of this without too much work.
If you include the output directories in your Perforce workspace, you can easily just check them in every time you do a build. (You might want to talk to the Perforce admin about using a 'purge' option so you're not storing thousands of copies of binaries.) You can use a trigger to make sure that source code in certain directories is always submitted with a binary.
Related
Currently, I have an MFC C++ visual studio project that is built to a DLL. Let's call it the FinalDLL.
I need this FinalDLL to be configurable. So, I want a GUI such that the users can generate the FinalDLL based on the information the enter via GUI. Please consider that I don't want to make the sources available at any points.
I do not have a clear idea how to integrate these steps. The solution I came up with was to have something like a button in my GUI, so that when it is clicked, the FinalDLL gets generated based on the information entered via GUI. Is it possible to do something like that? Probably I need another DLL library, as the information entered via GUI can be calls to functions that are used inside the FinalDLL source.
The solution to this would be very complex and complicated, mainly due to the fact that you don't want to disclose the source code of the DLL.
Basically you need to compile those source files every time you want to generate the DLL, and without the user having access to them.
Firstly, this is also required for the users that you want your DLL-generator app distributed to.
Secondly, you'll need to store those files somewhere in the app, and in an encrypted form, so that hackers won't just look at you app binaries and extract the source code for the DLL.
As a prerequisite on the user side, he will need to have a compatible Visual Studio installation, which will be used to compile on the fly the source code files.
What the DLL-generator application will need to do is to compile on the fly those source code files along with the customised ones via the form that the application will present.
So what you will need to do (the list might not be exhaustive due to the complex requirements):
Gather all compiler/linker commands that Visual Studio executes when building your project
Store all source code files into your application, in an encrypted form. Now if you want to allow your application to decode the files then you need to either store the encryption key within the application, and obfuscate it so its not that easy to find, or have the app communicate with your server and ask for the encryption key via https (this is a more secure approach, however neither this is 100% bullet-proof, insistent/trained hackers can still peek into the memory used by your app)
After the user fills all DLL generating details, the app will need to decrypt the source code files, updated the ones affected by the customised parameters, and start the build process by using the commands gathered at step #1. The compiler/linker should allow reading from stdin, so you'll use pipes to write the source code contents to the compiler/linker stdin, and to obtain the compiled/linked objects from stdout.
As I said, the solution is not pretty, and the main problem will be having the users install a Visual Studio that comes with a compiler compatible with the one from your machine, otherwise the commands you gathered at step #1 will not work.
When I use Visual Studio 2010 to debug a crash dump file (native code), it attempts to load C/C++ source files from the original build folder (and it gives the message "The source file is different from when the module was built. Would you like the debugger to use it anyway?"). The message is correct; the file is not the correct version.
What I would like VS2010 to do is to check out the source file using source server. If the file does not currently exist in its original build location, VS2010 will correctly use source server and retrieve the appropriate revision of the file (from Subversion). In order to force it to check out the correct revision, I have to physically delete the file from the original build location.
As a side note, VS2005 works as desired (well ... as I desire, perhaps not as others desire). VS2005 will always check out the correct revision from source control regardless of whether a copy of the file exists in the original build folder.
I believe the question comes down to one of the following:
Is there some kind of setting available that will change VS2010's precedence for finding source files?
Alternatively, is it possible to make VS2010 offer a choice/option to check out the source file in question? (Currently the only option I see in this situation is to browse for it.)
Or is it possible to completely exclude a specific path (folder) from the search?
I have the same problem with VS2010 and made an attempt to figure it out. I monitored devenv.exe with procmon but didn't see anything out of the order with the files & registry keys it was accessing. Pretty much the same information you see in the error report when VS2010 can't find the source. My solution is to use VS2005 as it works fine. I did see some correspondence on MSDN about a similar (if not the same) bug and they claimed it would be fixed in the final release of 2012. I believe I have that final release of 2012 and it has the same problem.
Here's a maybe slightly complicated solution
1) Create a script that will download and replace the pdb file (a .bat, a python script, whatever)
2) Create a new External Tool within VS2010 (Tools -> External Tools -> Add)
3) Point the tool to your script and pass any project-specific stuff to it as arguments
4) Create a post-build or pre-build step in your project that will call your new External Tool (project properties -> Build Events -> whatever)
This is a lot of work, but at least it will fully integrate it into your building process.
Note: Sometimes I've noticed that my post-build steps won't run unless I've compiled at least on cpp file. I usually press F7 and build some source and then build fully, to make sure everything works as expected.
You can change the local source directory to a different name when you are debugging crash dump file.
Or you can change the build directory to a different path with your local directory.
I'd like to use visual studio to store in source control xml files coming from a server.
I have a request like http://server/query.aspx?FILE_ID=1234 that allows me to download an xml file. Those file are part of our development activities, that's why I'm looking for a convenient way of integrating those file in source control.
I'd like to have a project containing all the xml files I want to check-in in source control and add a pre-build command allowing to download the files, but I did not find any convenient way of doing it.
People have a tendency to forgetting to do it manually, and we have already seen all the possible scenarios: lost files, released version without the ability to know the exact configuration used, ... I'd like to automated this step so that it does not happen again in the future.
I'm sure there is a simple and smart solution, but I could not find it. Any suggestion would be appreciated.
You should be able to use wget in a pre-build action to fetch the latest version of the files. I can't think of a reason why that wouldn't work.
Personally i would consider finding a way to automatically commit those files to source control whenever they change on the server. I've never used tfs, but I assume there is a commandline-client which allows you to commit files in a scripted way. If you don't have any control over when the files change you could do this every N minutes on a machine which is always on.
You can write a (powershell) script that does the fetching, and checkin of the file before your build starts. That's how we fetch external assemblies to be included in our build.
To get you started, take a look at these powershell functions for TFS interaction:
http://www.brokenwire.net/bw/Programming/73/ (TFS 2008)
I have a program under version control that has gone through multiple releases. A situation came up today where someone had somehow managed to point to an old copy of the program and thus was encountering bugs that have since been fixed. I'd like to go back and just delete all the old copies of the program (keeping them around is a company policy that dates from before version control was common and should no longer be necessary) but I need a way of verifying that I can generate the exact same executable that is better than saying "The old one came out of this commit so this one should be the same."
My initial thought was to simply MD5 hash the executable, store the hash file in source control, and be done with it but I've come up against a problem which I can't even parse.
It seems that every time the executable is generated (method: Open Project. File > Make X.exe) it hashes differently. I've noticed that Visual Basic messes with files every time the project is opened in seemingly random ways but I didn't think that would make it into the executable, nor do I have any evidence that that is indeed what's happening. To try to guard against that I tried generating the executable multiple times within the same IDE session and checking the hashes but they continued to be different every time.
So that's:
Generate Executable
Generate MD5 Checksum: md5sum X.exe > X.md5
Verify MD5 for current executable: md5sum -c X.md5
Generate New Executable
Verify MD5 for new executable: md5sum -c X.md5
Fail verification because computed checksum doesn't match.
I'm not understanding something about either MD5 or the way VB 6 is generating the executable but I'm also not married to the idea of using MD5. If there is a better way to verify that two executables are indeed the same then I'm all ears.
Thanks in advance for your help!
That's going to be nearly impossible. Read on for why.
The compiler will win this game, every time...
Compiling the same project twice in a row, even without making any changes to the source code or project settings, will always produce different executable files.
One of the reasons for this is that the PE (Portable Executable) format that Windows uses for EXE files includes a timestamp indicating the date and time the EXE was built, which is updated by the VB6 compiler whenever you build the project. Besides the "main" timestamp for the EXE as a whole, each resource directory in the EXE (where icons, bitmaps, strings, etc. are stored in the EXE) also has a timestamp, which the compiler also updates when it builds a new EXE. In addition to this, EXE files also have a checksum field that the compiler recalculates based on the EXE's raw binary content. Since the timestamps are updated to the current date/time, the checksum for the EXE will also change each time a project is recompiled.
But, but...I found this really cool EXE editing tool that can undo this compiler trickery!
There are EXE editing tools, such as PE Explorer, that claim to be able to adjust all the timestamps in an EXE file to a fixed time. At first glance you might think you could just set the timestamps in two copies of the EXE to the same date, and end up with equivalent files (assuming they were built from the same source code), but things are more complicated than that: the compiler is free to write out the resources (strings, icons, file version information, etc.) in a different order each time you compile the code, and you can't really prevent this from happening. Resources are stored as independent "chunks" of data that can be rearranged in the resulting EXE without affecting the run-time behavior of the program.
If that wasn't enough, the compiler might be building up the EXE file in an area of uninitialized memory, so certain parts of the EXE might contain bits and pieces of whatever was in memory at the time the compiler was running, creating even more differences.
As for MD5...
You are not misunderstanding MD5 hashing: MD5 will always produce the same hash given the same input. The problem here is that the input in this case (the EXE files) keep changing.
Conclusion: Source control is your friend
As for solving your current dilemma, I'll leave you with this: associating a particular EXE with a specific version of the source code is a more a matter of policy, which has to be enforced somehow, than anything else. Trying to figure out what EXE came from what version without any context is just not going to be reliable. You need to track this with the help of other tools. For example, ensuring that each build produces a different version number for your EXE's, and that that version can be easily paired with a specific revision/branch/tag/whatever in your version control system. To that end, a "free-for-all" situation where some developers use source control and others use "that copy of the source code from 1997 that I'm keeping in my network folder because it's my code and source control is for sissies anyway" won't help make this any easier. I would get everyone drinking the source control Kool-Aid and adhering to a standard policy for creating builds right away.
Whenever we build projects, our build server (we use Hudson) ensures that the compiled EXE version is updated to include the current build number (we use the Version Number Plugin and a custom build script to do this), and when we release a build, we create a tag in Subversion using the version number as the tag name. The build server archives release builds, so we can always get the specific EXE (and setup program) that was given to a customer. For internal testing, we can choose to pull an archived EXE from the build server, or just tell the build server to rebuild the EXE from the tag we created in Subversion.
We also never, ever, ever release any binaries to QA or to customers from any machine other than the build server. This prevents "works on my machine" bugs, and ensures that we are always compiling from a "known" copy of the source code (it only pulls and builds code that is in our Subversion repository), and that we can always associate a given binary with the exact version of the code that it was created from.
I know it has been a while, but since there is VB De-compiler app, you may consider bulk-decompiling vb6 apps, and then feeding decompilation results to an AI/statistical anomaly detection on the various code bases. Given the problem you face doesn't have an exact solution, it is unlikely the results will be 100% accurate, but as you feed more data, the detection should become more and more accurate
I am building a cross platform product and one of the requirements is across windows(win32,AMD64 and IA61). The product as is relatively simple CLI but we have a separate build team who checks out the code from CVS and build in separate build environments. I am able to build succesfully(using Visual C++ 2005) in one platform(AMD machine). But once I check in the code, check out the build fails.
The cause of the build failure is because the include library paths are wrongly specified in the property sheets. Specifically the output file folder under the Linker in property pages are specified wrongly. So these libraries get built in a different folder from where the other projects are expecting them.
However along with the source I check in the .sln files (and later .vcproj files) also everytime. Morover if I open the .sln file in the folder where the build is not succeeding, there is no difference between the one where I could succesfully build(pre check in). In fact using windiff I could not see any difference between the two build folders (except some .ncb and cvs log files).
So any idea what is going on? Where does VC++ 2005 take the include directories take the output folder path from if not from .sln? Is CVS somehow interfering with the process? Anything else I could try out.
Thanks in advance.
Just to update the problem was resolved. The root cause is the .vcproj files were not getting checked in CVS!! This is where the individual project settings were stored(I was under the impression that this is done in .sln files).
I think the problem can be that after you have changed the settings in one build configuration (for example x86-Release) but forgotten to change them for another configuration (for example ia64-Debug), and when configuration changes, you have this problem.
Another thing that I would check on your place is project dependencies. If those are set in the right way VS will look for project output exactly where it is outputted, even when you change the output folder.
Do you have any binary files checked in as ASCII?
The round trip to and from CVS can corrupt binary files that are incorrectly marked as ASCII because CVS performs character processing on ASCII files (e.g. to give you the correct end of line codes for your OS). Corruption can occur even in an all Windows environment.
See the Binary section in the CVS FAQ for more information.