I want to create a Python environment with the data science libraries NumPy, Pandas, Pytorch, and Hugging Face transformers. I use miniconda to create the environment and download and install the libraries. There is a flag in conda install, --download-only to download the required packages without installing them and install them afterwards from a local directory. Even when conda just downloads the packages without installing them, it also extracts them.
Is it possible to download the packages without extracting them and extract them afterwards before installation?
There is no simple command in the CLI to prevent the extraction step. The extraction is regarded as part of the FETCH operation to populate the package cache before running the LINK operation to transfer the package to the specified environment.
The alternative would be to do something manually. Naively, one could search Anaconda Cloud and manually download, however, it would probably be better to go through the solver to ensure package compatibility. All the info for operations to be run can be viewed by including the --json flag. This could be filtered to just the tarball URLs and then downloaded directly. Here's a script along these lines (assuming Linux/Unix):
File: conda-download.sh
#!/bin/bash -l
conda create -dn null --json "$#" |\
grep '"url"' | grep -oE 'https[^"]+' |\
xargs wget -c
which can be used as
./conda-download.sh -c conda-forge -c pytorch numpy pandas pytorch transformers
that is, it accepts all arguments conda create would, and will download all the tarballs locally.
Ignoring Cached Packages
If you already have some packages cached then the above will not redownload them. Instead, if you wish to download all tarballs needed for an environment, then you could use this alternate version which overrides the package cache using an empty temporary directory:
File: conda-download-all.sh
#!/bin/bash -l
tmp_dir=$(mktemp -d)
CONDA_PKGS_DIRS=$tmp_dir conda create -dn null --json "$#" |\
grep '"url"' | grep -oE 'https[^"]+' |\
xargs wget -c
rm -r $tmp_dir
Do you really want to use conda-pack? That lets you archive a conda-environment for reproducing without using the internet or re-solving for dependencies. To just prevent re-solving you can also use conda env export --explict but that still ties you to the source (internet or local disk repository).
If you have a static environment (read-only) and want to really reduce docker size, you can volume mount the environment at runtime. You would need to match the file paths (ie: /opt/anaconda => /opt/anaconda).
I am using git bash on Windows - that is git for Windows via the integrated bash. Apparently it uses the MINGW/MSYS underpinning. (Update from #VonC: It now uses msys2 since msysgit is obsolete since Q4 2015.)
So there are already a lot of MSYS tools installed - from awk to zcat. However I miss the man command and zip to compress multiple files into a zip file (unzip exists!).
Where from can I install them? I do not want to install another copy of the MINGW system! Any way just to add some pre-compiled tools to the git bash installation?
Here's another, slightly different, set of instructions to install zip for git bash on windows:
Navigate to this sourceforge page
Download zip-3.0-bin.zip
In the zipped file, in the bin folder, find the file zip.exe.
Extract the file zip.exe to your mingw64 bin folder (for me: C:\Program Files\Git\mingw64\bin)
Navigate to to this sourceforge page
Download bzip2-1.0.5-bin.zip
In the zipped file, in the bin folder, find the file bzip2.dll
Extract bzip2.dll to your mingw64\bin folder (same folder as above: C:\Program Files\Git\mingw64\bin)
7-zip can be added to gitbash as follows:
Install 7-zip on windows.
add 7-zip folder (C:\Program Files\7-Zip) to PATH
On gitbash exp: export PATH=$PATH:"C:\Program Files\7-Zip" (temporary)
On Windows, adding PATH like image below (permanent)
duplicate a copy of 7z.exe to be zip.exe
reopen gitbash again. done!
This way, it works on my laptop.
If you skip step 3. you still can call zip command as 7z instead of zip
Conclusion: Gitbash is running base on windows Path, I think you can run any command that you have added to your Windows PATH.
2016: The zip command can be installed from GoW (Gnu On Windows). man is not provided (too big).
It is to note, however, that if you only want to add the zip command from GoW, still the whole GoW system has to be downloaded and installed. Then you can delete the other commands from the bin directory, however make sure to keep the needed dlls in the directory.
Update 2021: tar/zip are by default installed on Windows 10.
7-zip based solutions are available below.
git-archive, is prepared without any installation, can create zip-archive.
mkdir workrepo
cd workrepo
git init
cp -r [target_file_or_dir] .
git add .
git commit -m commit
git archive -o ../myarchive.zip #
cd ..
rm -rf workrepo
Following script may be usable:
zip.sh foo.zip target_file_or_dir
#!/usr/bin/bash
set -eu
unset workdir
onexit() {
if [ -n ${workdir-} ]; then
rm -rf "$workdir"
fi
}
trap onexit EXIT
workdir=$(mktemp --tmpdir -d gitzip.XXXXXX)
cp -r "$2" "$workdir"
pushd "$workdir"
git init
git config --local user.email "zip#example.com"
git config --local user.name "zip"
git add .
git commit -m "commit for zip"
popd
git archive --format=zip -o "$1" --remote="$workdir" HEAD
I am so glad to share my experience on this issue that I haven't known for two years since the first day I played with Groovy. My method needs to have git for Windows installed in Windows OS.
These steps are for installing 7z command-line utility, which behaves a bit differently from zip:
Download and install 7-Zip from its official website. By default, it is installed under the directory /c/Program Files/7-Zip in Windows 10 as my case.
Run git Bash under Administrator privilege and navigate to the directory /c/Program Files/Git/mingw64/bin, you can run the command ln -s "/c/Program Files/7-Zip/7z.exe" 7z.exe
I am pretty sure it could help you a lot. Trust me!
On Windows, you can use tar instead of zip.
tar -a -c -f output.zip myfile.txt
which is same as,
zip output.zip myfile.txt
no need to install any external zip tool.
I use choco as my Windows Package Manager.
I install 7zip with choco using PowerShell (you must be admin to use Choco)
PS > choco install 7zip.install
Open another gitbash Terminal and locate the 7z.exe executable
$ which 7z
/c/ProgramData/chocolatey/bin/7z
Do a straight copy of 7z.exe to zip.exe and voila
$ cp /c/ProgramData/chocolatey/bin/7z.exe /c/ProgramData/chocolatey/bin/zip.exe
You can mimic a small subset of man behavior in the shell by mapping man <command> to <command> --help | less
Unfortunately, on my machine bash aliases won't add flags to positional arguments, it will try to run the flag as a command and fail (alias man="$1 --help" doesn't work).
And a function called man() is not allowed!
Luckily a combination of bash functions and aliases can achieve this mapping. Put the code below in your ~/.bashrc (create one if it is not there). Don't forget to source ~/.bashrc.
# man command workaround: alias can't pass flags, but can't name function man
m() {
"$1" --help | less
}
alias man="m"
It doesn't get you the full man page, but if all you're looking for is basic info on a command and its flags, this might be all you need.
You can install individual GNU tools from http://gnuwin32.sourceforge.net/packages.html such as zip.
Then add "/c/Program Files (x86)/GnuWin32/bin" to PATH in your startup script like .profile, .bash_profile, .bashrc, etc.
Here are the steps you can follow.
Go to the following link
https://sourceforge.net/projects/gnuwin32/files/
Find out whatever command you are missing
Here I need zip and bzip2 for zip command. Because zip command relies on bzip2.dll to run. Otherwise you will get error “error while loading shared libraries: ?: cannot open shared object file: No such file or directory”.
Unzip the downloaded files
Here I am downloading “zip-3.0-bin.zip” for “zip.exe” and “bzip2-1.0.5-bin.zip” for “bzip2.dll” in the bin folder. /bin/.exe
Copy the command exe file into git-bash folder
Here I am copying “zip.exe” and “bzip2.dll” to \Git\usr\bin.
Reference Link
https://ranxing.wordpress.com/2016/12/13/add-zip-into-git-bash-on-windows/
ln -s /mingw64/bin/ziptool.exe /usr/bin/zip
steps to install SDKMAN on windows
Run Windows Terminal in Admin rights. open git bash inside. (Ctrl + Shift + 4)
winget install -e --id GnuWin32.Zip
mkdir ~/bin
cp /usr/bin/unzip ~/bin/zip
curl -s "https://beta.sdkman.io" | bash
source "/c/Users/ajink/.sdkman/bin/sdkman-init.sh"
sdk selfupdate force
After you can install Java like this.
sdk install java 17.0.2-open
Done ! :)
In msys2, I restored the functionality of git help <command> by installing man-db:
|prompt> pacman -Syu man-db
|prompt> git help archive
For zip functionality, I also use git archive (similar to yukihane's answer).
Here's yet another 7-Zip option that I didn't notice:
Create a script named zip:
$ vi ~/bin/zip
Reference 7z specifying the add command followed by the args:
#!/bin/bash
/c/Progra~1/7-Zip/7z.exe a "$#"
Finally make it executable
$ chmod ugo+x ~/bin/zip
This helped to make a ytt build script happy.
+ zip ytt-lambda-website.zip main ytt
7-Zip 18.01 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2018-01-28
Scanning the drive:
2 files, 29035805 bytes (28 MiB)
Creating archive: ytt-lambda-website.zip
Add new data to archive: 2 files, 29035805 bytes (28 MiB)
Though this question as been answered quite thoroughly in regards to man there is one alternative to zipping that has not been highlighted here yet. #Zartc brought to my attention that there is a zip compression utility built-in: ziptool. In trying to use it however I found out it is no where near a drop-in replacement and you need to specify each individual file and folder. So I dug into the docs and experimented until I had a bash-function that can do all the heavy lifting and can be used very similar to a basic zip -qrf name * compression call:
zipWithZiptool() {
# Docs: https://libzip.org/documentation/ziptool.html
targetFilePath="$1"
shift
args=() # collect all args in an array so spaces are handled correctly
while IFS=$'\n\r' read -r line; do
if [[ -d "$line" ]]; then
args+=("add_dir" "$line") # Add a single directory by name
else
# add_file <pathInZip> <pathToFile> <startIndex> <length>
args+=("add_file" "$line" "$line" 0 -1)
fi
done <<< "$(find "$#")" # call find with every arg to return a recursive list of files and dirs
ziptool $targetFilePath "${args[#]}" # quotation is important for handling file names with spaces
}
You can then for example zip the contents of the current directory by calling it like this:
zipWithZiptool "my.zip" *
If you are willing to install CygWin also, you can add the CygWin path to your GitBash path, and if zip is there, it will work. e.g. add
PATH=$PATH:/c/cygwin/bin
export PATH
to your .bashrc; NOTE: I would put it at the end of the path as shown, not the beginning.
Since CygWin has a UI-based installer, it's easy to add or remove applications like zip or man.
You can figure out the windows paths of each by saying
`cygpath -w /bin`
in each respective shell.
Regarding zip, you can use a following perl script to pack files:
#!/usr/bin/perl
use IO::Compress::Zip qw(:all);
$z = shift;
zip [ #ARGV ] => $z or die "Cannot create zip file: $ZipError\n";
If you make it executable, name it zip, and put it in your $PATH, you can run it like this:
zip archive.zip files...
however it will not work for directories. There is no need to install anything, as perl and all required modules are already there in the Git for Windows installation.
Regarding man, at least for git there is a documentation invoked like this:
git option --help
it will open in your default browser.
Here is my experience, I cant run and exe or .msi files in my laptop. so downloaded filed from https://github.com/bmatzelle/gow/wiki > go to download Now and Downloaded Source Code (Zip) and unzipped this file in a folder and updated path variable with folder name.
This worked out for me.
If you want to zip files without needing to install any additional tools on Windows, in a way that works both on git bash and on other *nix systems, you might be able to use perl.
Per Josip Medved's blog, the following script creates an .epub (which is a zip file), and includes a filter for stripping src/ from the files added to the zip:
perl -e '
use strict;
use warnings;
use autodie;
use IO::Compress::Zip qw(:all);
zip [
"src/mimetype",
<"src/META-INF/*.*">,
<"src/OEBPS/*.*">,
<"src/OEBPS/chapters/*.*">
] => "bin/book.epub",
FilterName => sub { s[^src/][] },
Zip64 => 0,
or die "Zip failed: $ZipError\n";
'
install zip
https://gnuwin32.sourceforge.net/packages/zip.htm
copy zip.exe and bzip2.dll from C:\Program Files (x86)\GnuWin32\bin to C:\Program Files\Git\mingw64\bin
reopen git-bash
Solutions for me were just to install zip on my terminal(bash):
$ sudo apt-get update
$ sudo apt-get install zip unzip
The following command hangs on my osx:
xcodebuild -scheme myscheme clean archive -archivePath /tmp
This command yields two output lines, and then hangs:
User defaults from command line:
IDEArchivePathOverride = /tmp
Now, this project does NOT have a workspace generated as it was created from a cordova command line (cordova build ios). The only way around it is to open xcode and close it. this generates a workspace and then the above command succeeds.
Did anyone experience something similar and know a way out of this? Any way to generate that workspace from the command line?
I had the same problem and the only way of fixing it was to open the project from the command line, wait, and close it again after a certain time.
open "My Project.xcodeproj"
sleep 10
killall Xcode
xcodebuild -scheme "My Project" clean archive "build/MyProject"
Not nice, but works for me.
Try setting the scheme to be 'shared'.
This can be done by going to the 'Manage Schemes...' and checking the 'Shared' checkbox.
Apple documents this process here: https://developer.apple.com/library/ios/recipes/xcode_help-scheme_editor/Articles/SchemeShare.html
If you're already have, or are willing to make, Ruby available to your build system then you could use this solution.
Install the xcodeproj gem on your build system
sudo gem install xcodeproj
and then integrate the following ruby script into your project (renaming your xcodeproj path).
#!/usr/bin/env ruby
require 'xcodeproj'
xcproj = Xcodeproj::Project.open("platforms/ios/schemedemo.xcodeproj")
xcproj.recreate_user_schemes
xcproj.save
The article explains how to make it part of a cordova hook if you're doing that, I simply called ruby directly from my Jenkins build.
This works because when you recreate the proj files, you destroy the schemes, so you need to recreate them.
I believe xcodebuild hangs because some data is missing from the project. You can make a template for what this data looks like and use a hook to populate it if necessary.
cordova add platform ios
cordova build ...
open platforms/ios/Whatever.xcodeproj in xcode
create xcuserdata_template
cp -R platforms/ios/Whatever.xcodeproj/xcuserdata xcuserdata_template/
replace the unique id in that template with XXXXXXXXXX
update your hook that runs xcodebuild
Step 7 example:
XCODE_PROJ=path/to/Whatever.xcodeproj
# get the mysterious id
ID=`grep "Whatever \*\/ = {" $XCODE_PROJ/project.pbxproj | \
grep -io "[-A-Z0-9]\{24\}"`
mkdir -p $XCODE_PROJ/xcuserdata
XCUSERDATAD=$XCODE_PROJ/xcuserdata/`whoami`.xcuserdatad
if [ ! -d "$XCUSERDATAD" ]; then
cp -R path/to/xcuserdata_template/username.xcuserdatad \
$XCUSERDATAD
find $XCUSERDATAD -type f -exec sed -i '' -e "s/XXXXXXXXXX/$ID/g" {} \;
fi
xcodebuild ...
I've been attempting to port a Mac PackageMaker command line build to pkgbuild and productbuild but I'm stuck. Unfortunately I haven't found much of anything documenting how these new programs work except for this StackOverflow post and the pkgbuild and productbuild man pages.
Here's my problem. I've created a root install directory that has the following files in it:
/some_path/Applications
/MyProgram.app
/Library
/Frameworks
/MyFramework.framework
/[library files...]
The command line call below worked great for PackageMaker. It created an installer that installed all of the files above.
$ /Developer/usr/bin/packagemaker \
--title "My Program" \
--root /some_path \
--version 1.0.0 \
--filter "\.DS_Store" \
--resources ./resources/ \
--scripts ./scripts/ \
--root-volume-only \
--domain system \
--verbose \
--no-relocate \
--target 10.5 \
--id com.my_company.pkg \
--out MyProgram.pkg
Now I'm trying to write this with pkgbuild and having a major problem. I use the following call:
$ pkgbuild \
--root /some_path \
--version 1.0.0 \
--install-location "/" \
--scripts "./scripts/" \
--identifier "com.my_company.pkg" \
MyProgram.pkg
This command builds an installer that copies the MyProgram.framework directory into /Library/Frameworks. However it does not install the MyProgram.app file into the /Applications directory. When I look at the installer logs I see this message:
Applications/MyProgram.app relocated to
/some_path/Applications/MyProgram.app
Can anyone shed some light on why this isn't adding the MyProgram.app file into the /Applications directory like PackageMaker was doing?
Unfortunately the answer to this question wasn't exactly what I was looking for. I couldn't figure out how to eliminate PackageMaker from the process. However there is a solution that includes pkgutil along with PackageMaker to create an installer with custom welcome message, license and background image entirely on the command line. The PackageMaker GUI is NOT required. The steps are as follows:
Run the packagemaker command line executable on the special directory structure. This directory structure reflects the Mac file system. Read more in the old but reliable "PackageMaker How-to" tutorial.
Run pkgutil (pkgutil --expand) to extract the package contents
Take a look at the contents and identify what you want to alter. Some options are the welcome message, license and background image.
Add commands to alter these files via the command line. Review the "Automating Apple's PackageMaker" tutorial for more information. The easiest way is just to run something like this echo '<background file="your_background.png">'.
Run pkgutil (pkgutil --flatten) to rebuild the package.
First of all, are you sure that you need an installer? You could put the framework inside the application. The Installer and pkgbuild are a bit flaky, to say the least.
Now to the problem at hand: Relocation has to do with the fact that a user could move the Application from /Applications to say /WorkApplications /PrivateApplications. In your case the Installer probably finds your Application in the build folder and installs it over this one.
I think the Installer uses the Application Bundle Identifier and Spotlight for the relocation, so for testing you could add the build folder to the Spotlight ignore list.
You can define in the Component Property List BundleIsRelocatable. If you really have to install a framework global, this is one bundle where you want to set BundleIsRelocatable to false.
The question you reference has pretty much everything you need to eliminate Package Maker entirely. One thing I added is to use sed after the productbuild --synthesize ... invocation to insert lines into the distribution file. For example, here are some Terminal commands I use once I've already built the component package:
productbuild --synthesize --package "components/SubPackage.pkg" "distribution.xml"
sed -i "" \
-e '$ i\
\ <title>Installer Title</title>' \
-e '$ i\
\ <background file="background.png" alignment="left" scaling="proportional" />' \
-e '$ i\
\ <welcome file="welcome.rtf" />' \
"distribution.xml"
productbuild --distribution "distribution.xml" --resources "resources/" --package-path "components/" "Installer.pkg"
This avoids having to use pkgutil --expand and pkgutil --flatten to modify the installer.
Create component plist file, which has value for relocatable property.
pkgbuild --analyze --root "$dst_src_root" "$installer_root/Components.plist"
Edit component plist and Set "BundleIsRelocatable" to false.
This is done one time in my project, as installer contents does not change (One application and one plugin). I am reusing same component plist everytime to create contents package.
sudo pkgbuild --root "$dst_src_root" --component-plist "$installer_root/Components.plist" --identifier "com.company.app" --version "1.0" --scripts "$dst_scpt_root" "$dst_pkg_root/InstallPackage/cisContents.pkg"
Then using productbuild, we can create final package
I want to create a dmg file for my Mac project. Can someone please tell me how to do this? This being my first Mac project, I do not have any idea how to proceed. I also want to give the user an option of running the app on start-up. How do I do this?
Thanks.
P.S. I also want to add a custom license agreement.
To do this manually:
Method 1:
Make a folder with the files your DMG will contain.
Open Disk Utility (It's in /Applications/Utilities/)
Go to File > New > New Image from Folder (Cmd + Shift + N)
Choose the folder containing you files
Make sure "Compressed" is checked, then set where you want to save the created DMG
Method 2:
To do things like setting a background image can be a bit convoluted (You basically add the background image to the DMG, set the windows properties to use that image, using the command line you move the background image from background.png to .background.png to make it hidden)
I would recommend iDMG, which makes things a bit less tedious.
You can also script the creation of DMGs using the command hdiutil. Something along the lines of
hdiutil create -srcfolder mydirtodmg mydmg.dmg
As for the custom license agreement, you should look into the tool included with the Developer Tools "PackageMaker" - it's pretty self-explanatory. It's in /Developers/Application/Utilities/
If you need to add a custom EULA to your disk image, this page describes how to do so using command-line tools. The gist of it is to use the template software licensing agreement resource provided in Apple's slas_for_udifs_1.0.dmg, modify the resource with your EULA text and inject the resource back into your disk image file. (I include brief instructions below in case the above link becomes unavailable, and to update the search term it provides in step 1.)
Using your Apple Developer account go to the Downloads page and search for Software Licensing for UDIF
Download and mount the disk image
In Terminal:
cd /Volumes/SLAs_for_UDIFs_1.0
DeRez SLAResources > /tmp/sla.r
Edit /tmp/sla.r in a text editor, updating the content of the data 'TEXT' (5000, "English SLA") resource to contain your new license text.
Unflatten the disk image file that contains your installer:
hdiutil unflatten installer_image.dmg
Add the edited license resources to the image:
Rez -a /tmp/sla.r -o installer_image.dmg
why don't you just run a script from your xcode project. try something like this:
# be sure to check the man page for hdiutil
# it is very powerful and has tons of options...
hdiutil create -megabytes 54 -fs HFS+ -volname awesome_app_install myAwesomeApplication.dmg
hdiutil mount myAwesomeApplication.dmg
cp -r /build/Release/AwesomeApplication.app /Volumes/awesome_app_install/
then save your script as something like 'makeDMG.sh' and in your target,
select add->new build phase->run script build phase
and drag your script into this build phase.
once you've done all that, then when you build your project the script will create the disk image and copy your release build into it...
of course you should spice your script to flavor... these three lines are just the raw meat
ps: your custom EULA should get built into your packagemaker project (which you can also script very nicely)
I made a little bash script to automate a disc image creation.
It creates a temporary directory to store all needed files then export it in a new DMG file. Temporary directory is then deleted.
You can automatically launch this script at the end of your build process.
#!/bin/bash
# Create .dmg file for macOS
# Adapt these variables to your needs
APP_VERS="1.0"
DMG_NAME="MyApp_v${APP_VERS}_macos"
OUTPUT_DMG_DIR="path_to_output_dmg_file"
APP_FILE="path_to_my_app/MyApp.app"
OTHER_FILES_TO_INCLUDE="path_to_other_files"
# The directory of the script
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
# The temp directory used, within $DIR
WORK_DIR=`mktemp -d "${DIR}/tmp"`
# Check if tmp dir was created
if [[ ! "${WORK_DIR}" || ! -d "${WORK_DIR}" ]]; then
echo "Could not create temp dir"
exit 1
fi
# Function to deletes the temp directory
function cleanup {
rm -rf "${WORK_DIR}"
#echo "Deleted temp working directory ${WORK_DIR}"
}
# Register the cleanup function to be called on the EXIT signal
trap cleanup EXIT
# Copy application on temp dir
cp -R "${APP_FILE}" "${WORK_DIR}"
# Copy other files without hidden files
rsync -a --exclude=".*" "${OTHER_FILES_TO_INCLUDE}" "${WORK_DIR}"
# Create .dmg
hdiutil create -volname "${DMG_NAME}" -srcfolder "${WORK_DIR}" -ov -format UDZO "${OUTPUT_DMG_DIR}/${DMG_NAME}.dmg"