Consistent builds / remove personal information from binaries - go

I've now realized that Go saves absolute paths to source code in binaries for the purpose of printing stack-traces and the likes. I don't want to completely remove this information, however, this also means that every developer building the same program will produce an executable with a different checksum. Before I try to reimplement the build using chroot or something like that: isn't there any way to tell Go not to use absolute paths for this purpose?

I know it doesn't directly address what you asked, but #JimB's suggestion does indicate a class of solutions to the problem you seem to be having.
One of the easier ones (I think) would be to have your developers install Docker and create an alias so that the go command runs:
docker run --rm --tty --volume $GOPATH:/go golang:1.7.1(-$YOUR_PLATFORM) go
Then, every build (and test and run) thinks it's using a GOPATH of /go and your developers' checksums won't disagree based on that.
See here for more info.

isn't there any way to tell Go not to use absolute paths for this purpose?
Nowadays there is: -trimpath.
https://pkg.go.dev/cmd/go#hdr-Compile_packages_and_dependencies explains:
-trimpath
remove all file system paths from the resulting executable.
Instead of absolute file system paths, the recorded file names
will begin either a module path#version (when using modules),
or a plain import path (when using the standard library, or GOPATH).

Related

How to package shell programs into an AppImage?

I have made an AppImage via:
linuxdeploy --appdir AppDir --icon-file icon.png --desktop-file desktop.desktop --executable myExecutable --output appimage
which runs fine. However, the program I've packaged (myExecutable) makes shell calls (say to shellProgram1, shellProgram2, ...) at run-time to make use of various programs that aren't necessarily on every distro.
Question: Does linuxdeploy (or some other AppImage utility) provide an easy way to package these programs into the AppImage, so that when myExecutable calls them at run-time, they are guaranteed to be available?
To achieve such thing you need to deploy all the binaries that may not be present in all distros into the AppDir and set the PATH environment to make them available at runtime.
With linuxdeploy you have to manually copy the files into the AppDir and create a wrapper for the main binary to set the PATH. Something like this
$!/bin/bash
export PATH="$APPDIR/usr/bin:$PATH"
exec $APPDIR/usr/bin/my_program
You can also use appimage-builder which creates such wrapper for you. In the project examples folder, you can find several recipes that can be used for inspiration.

`go run hello.go` cannot find the "hello.go" file

I tried to run the program in command prompt #Go lang- But when i type run "go run hello.go" command i am gettin
CreateFile hello.go:The system cannot find the file specified
Please help to to compile and run the above marked program, Thanks in advance
As you can see from the output of running the dir command
earlier up in your shell session, there is indeed no file named "hello.go"
in the C:\GOCODE\testproject directory.
When you execute the
go run hello.go
command, the go tool tries to find the file named "hello.go" in the current
directory (because the name of that file is relative, so it's being
searched in current working directory). There's no such file,
and that's what go run tells you.
Unfortunately, from the outlook of your shell session, it appears there
are more problems with your setup.
And there are problems with your approach to Go.
First, while it looks like you're following
this guide (and this is the right thing
to do, actually), you misread it.
What it tried to tell you is that you should create the "src"
directory (and then the "hello" directory to contain your test project)
in the so-called workspace, and a list of workspaces known to Go
is stored in the GOPATH environment variable.
As you can see from the go env output, Go thinks you have a single
workspace located in C:\Users\Sitaram\Go.
Now it worth reiterating that—contrary to many (if not most) "mainstream"
languages,—Go is not "project-based"; instead, it requires all your code
to be organized in those workspaces, and it wants to know where these
workspaces are.
By default—if you did not explicitly set the GOPATH environment
variable,—it assumes your single workspace is located in the directory
named "go" placed in your "home folder".
And that's what you see in the go env output.
Now you have two options:
Set the GOPATH env. variable for your user to C:\GOCODE
then start another shell—so that it "sees" that variable and allows
the go tool to also see it and use).
Run go env to verify GOPATH contains C:\GOCODE.
Then follow the rest of the tutorial document:
Make sure there is the "src" folder directly under the C:\GOCODE.
Create your project folder directory under "src".
Let's say, it will be named "hello".
Under "hello", create that "hello.go" file.
Now cd C:\GOCODE\src\hello and then go build — you will have
the hello.exe created there.
Don't mess with GOPATH and just repeat the steps 2-4 from above
in the default workspace—C:\Users\Sitaram\go.
I'd go with the second variant because that inexplicable affection
of certain Windows users for polluting the C:\ with random personal
data is really an anti-pattern; have your personal belongings in your home
folder! Windows has gone a long way getting that right; and almost all
Windows software is finally there—understanding that paradigm. So why deviate?
Second, please unlearn go run.
I'm not sure the Go developers actually regret implementing it,
but people do really misinterpret what this tool is for.
It's for one-off throw-away "scripts".
Real development is done using go install and, sometimes, go build.
In most cases your normal development routine you use go install
exclusively — as it caches the results of compilation of all the packages
your project depends on. go build does not do this, and go run does
not even preserve the result of the compilation of your project itself.
Please read this quick reference card for more info.
After go get gpackage check if there is a yourproject.exe in your bin directory to compile your github package with your project.
If not, you have to do cd src/yourproject and type go install and hit enter.
Try execute the folowing command e check if corrected:
go env -w GOOS=windows

How to change default path to store executable in Go compiler?

In Go compiler, when I do "go run", executable file is stored to a temporary location. How to change this path to store the file in current working directory? I am using windows 7 64bit machine.
If you want to do something with the binary beyond just running it once, you should be using go build, not go run. go build will put the binary in the current working directory.
I agree with #Adrian and #Saleem, however, for interest sake, you can override the location (somewhat) by changing the location of your environment variable TEMP (or TMPDIR on OSX or Linux). This will still create a temporary directory in whatever directory you specify, in which the working files will be placed. Keep in mind that as Adrian and Saleem say, go run is intended for temporary runs.
And of course #JimB beat me to it with his comment which is really the essence of what I'm saying here.
Agree with previous answers. go install saves binary file into GOBIN folder. So you may change it to have a specific location. However, I don't suggest to do it, because you can always build into specific folder using -o option of go build:
go build -o /usr/bin/app main.go
The folder used by Go to store the temporary executable can be changed by setting the GOTMPDIR environment variable.
More info here: https://stackoverflow.com/a/71197493/1057961

ImageMagick deployment on Mac OS

I'm trying to deploy ImageMagick with my own software. On windows I've just included all the core dlls with coders dlls at the exe path and it works well.
But on mac os I have troubles with coders. I installed ImageMagick via macports and found it with the help of CMake. CMake does all the job of copying and fixing up all the core libs I've linked against. Then I copied all the coder libs and fixed them up also, but when I start my application it just can't find any coder. So I'd like to know what am I missing there.
Note: if I didn't fix up any paths it works well. It is only my deployment that is in trouble. Maybe I should include some kind of config file?
P.S. I have all ImageMagick libs including coders SOs near the executable in MacOS bundle sub-folder.
How about setting the MAGICK_CODER_MODULE_PATH in your bundle?
see here: http://www.imagemagick.org/script/resources.php
EDIT:
To improve the information:
Originally when embedding IM in our own app bundle we had three problems:
our app and the IM dylibs not finding referenced IM dylibs,
IM not finding its config files,
IM not finding coders (the No Decode Delegate error)
We tried changing the hardcoded paths in the dylibs using the install_name_tool but finally when doing some tests with moving the IM around to different directories and testing
convert -debug configuration
we found out the all three above problems could be solved just by setting and exporting at least these three environment variables in the terminal console before running convert:
DYLD_LIBRARY_PATH
MAGICK_CONFIGURE_PATH
MAGICK_CODER_MODULE_PATH
With this experience, we returned back to our bundle and in the beginning tried to use the Info.plist fiel to set these variables but it didn't seem to work - probably because there were problems with making the paths to IM inside the bundle relative.
Finally we created a simple sh script and put it into our bundle and configured this bundle to run this script instead of the main app:
#!/bin/sh
CURR_DIR="$( cd -P "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
IMAGE_MAGICK_PATH=$CURR_DIR/../Resources/ImageMagick
export DYLD_LIBRARY_PATH=$IMAGE_MAGICK_PATH/lib
export MAGICK_CONFIGURE_PATH=$IMAGE_MAGICK_PATH/lib/ImageMagick-6.8.0/config
export MAGICK_CODER_MODULE_PATH=$IMAGE_MAGICK_PATH/lib/ImageMagick-6.8.0/modules-Q16/coders
# run application
exec $CURR_DIR/OurAppName
The key thing to make it working was properly getting the CURR_DIR of the app bundle (thanks to this post).
And as came out of our tests, setting the environment variables this way makes them visible only for this application execution context - i.e. when we started our app using the bundle, opened terminal and typed
env
the above three variables were missing from the output.
Hope this will help others save couple of days of research and pulling hairs out of their heads ;)
I've found a full solution for deploying ImageMagick in a bundle with the help of CMake. If you don't use CMake then #Tomasz's answer will be of help also.
So let's start:
First of all you need to know what and where ImageMagick is trying to locate when it is used from your own code. To find it out you can use MAGICK_DEBUG environmental variable which could be set to those parameters. It really helps when you debug ImageMagick.
Prerequisites:
I assume that you used FIND_PACKAGE and FIXUP_BUNDLE to find ImageMagick and set its binary paths inside the bundle. The only thing left is to deploy coders. Also I assume that you've installed ImageMagick from Mac Ports.
We need to get ImageMagick version string to correctly locate the coders:
STRING(REGEX REPLACE "-.+" "" ImageMagick_SHORT_VERSION ${ImageMagick_VERSION_STRING})
Now ImageMagick_SHORT_VERSION contains full version without any sub versions.
Then we need to copy all the coders to some predefined folder(I've used ImageMagick/coders subfolder under MacOS part of the bundle)
FILE(COPY /opt/local/lib/ImageMagick-${ImageMagick_SHORT_VERSION}/modules-Q16/coders/ DESTINATION ${PATH_TO_YOUR_BUNDLE}/Contents/MacOS/ImageMagick/coders/)
Now we need to fixup all the *.so libs we have, so we list it and pass to fixup_bundle
FILE(GLOB IMAGEMAGICK_CODERS ${PATH_TO_YOUR_BUNDLE}/Contents/MacOS/ImageMagick/coders/*.so)
Now we should update *.la files which accompanies coders *.so. To achieve it I've used script:
INSTALL(SCRIPT LaScript.cmake COMPONENT Runtime)
Script content:
SET(TARGET_BINARY_DIR "${PATH_TO_YOUR_BUNDLE}")
FILE(GLOB IMAGEMAGICK_CODERS_LA ${TARGET_BINARY_DIR}/Contents/MacOS/ImageMagick/coders/*.la)
FOREACH(file ${IMAGEMAGICK_CODERS_LA})
FILE(READ ${file} FILE_CONTENT)
STRING(REGEX REPLACE "dependency_libs='.*'" " " MODIFIED_FILE_CONTENT ${FILE_CONTENT})
STRING(REGEX REPLACE "libdir='.*'" " " MODIFIED_FILE_CONTENT ${MODIFIED_FILE_CONTENT})
FILE(WRITE ${file} ${MODIFIED_FILE_CONTENT})
ENDFOREACH()
We almost ready the only thing left to be done is to change the way we launch the application. But let's digress a little bit and find out where ImageMagick searches for the coders:
It tries to get the content of MAGICK_CODER_MODULE_PATH environmental variable
Then it checks if MAGICKCORE_CODER_PATH macro is defined(and in fact it does!) and use its value.
Then it will try to use MAGICK_HOME environmental variable and MAGICKCORE_CODER_RELATIVE_PATH to get path to the modules but we don't care since we will stop on #2 anyway!(NOTE: that it is true for Mac Ports installation)
So the only way we can interfere with search is to set MAGICK_CODER_MODULE_PATH environmental variable(Well we can also edit libMagickCodre and replace MAGICKCORE_CODER_PATH with some static path we need but it is too britle way to do things and it won't save us if someone set MAGICK_CODER_MODULE_PATH anyway)
We shouldn't set it system wide since we can break some user installtion so we have 2 options:
Use LSEnvironment to set the MAGICK_CODER_MODULE_PATH to some predefined location
Use script to launch our app and set this variable inside it.
I've chose the later since it is more flexible,
I have the following script:
#!/bin/bash
working_dir="${0%/*}"
export MAGICK_CODER_MODULE_PATH=$working_dir/ImageMagick/coders
executable="${working_dir}/ApplicationName"
"$executable"
and set CFBundleExecutable to the name of the script.
That's all and I hope it will help someone to save his/her time.
You should follow the Mac OS X-specific Build instructions but specifying --enable-shared in the configure options (see this document for details).
I guess that your application can't find the codecs because they have been statically linked to ImageMagick tools. This is usually done to address portability issues. To make codecs available in your application, you should build them as shared objects.

Creating .deb to install bash script program

I was wondering if the following is possible.
I have a BASH script that I want to make available for some people but I wanted them to only have to "install" the program and not messing around with terminal, so I thought a .deb would be cool.
So what would the "install" do?
Simple. I want to move the script and an icon to a folder (any folder, but I was wondering some hidden folder in Home) and then run a script that creates a launcher in the Applications menu for the first script. It seems there isn't much to it, but for what I've searched, there doesn't seem to be a lot of info...
How can I accomplish this?
By the way, I'm using Ubuntu 11.04.
Basically (install and) run dh-make to set up the debian/ directory, edit the generated files (mainly remove the many you do not need, and fill in a package description and any dependencies in debian/control), then debuild-us -uc -b.
You may also have to set up a simple Makefile for debian/rules to call; it probably only needs an install target to copy the binary to $(DESTDIR)/usr/bin.
Binaries install into /usr/bin and you should not try to override that. The way to have a menu is to add a .desktop file.
Once you have a good .deb you will need to set up a repo for distributing it. The simplest solution is probably to set up a launchpad.net account and create a personal PPA there.
It's not hard to find more information on these topics, but of course, you need to know what to look for. The canonical documentation is the Debian New Maintainer's Guide.
Found this video on youtube that explains IN FULL the process of creating a *.deb for a script or program and even mentions how to do it for a C program.
Full guide in how to build simple *.deb package
Has one bug, btw, that the author, during the making of the *.deb, didn't notice. The path in the *.desktop file for the EXEC parameter is wrong in the example.

Resources