I would like to remove the digital signature from a Mac app that has been signed with codesign. There is an undocumented option to codesign, --remove-signature, which by it's name seems to be what I need. However, I can't get it to work. I realize it is undocumented, but I could really use the functionality. Maybe I'm doing something wrong?
codesign -s MyIdentity foo.app
works normally, signing the app
codesign --remove-signature foo.app
does disk activity for several seconds, then says
foo.app: invalid format for signature
and foo.app has grown to 1.9 GB!!! (Specifically, it is the executable in foo.app/Contents/Resources/MacOS that grows, from 1.1 MB to 1.9 GB.)
The same thing happens when I try to sign/unsign a binary support tool instead of a .app.
Any ideas?
Background: This is my own app; I'm not trying to defeat copy protection or anything like that.
I would like to distribute a signed app so that each update to the app won't need user approval to read/write the app's entries in the Keychain. However, some people need to modify the app by adding their own folder to /Resources. If they do that, the signature becomes invalid, and the app can't use it's own Keychain entries.
The app can easily detect if this situation has happened. If the app could then remove it's signature, everything would be fine. Those people who make this modification would need to give the modified, now-unsigned app permission to use the Keychain, but that's fine with me.
A bit late, but I've updated a public-domain tool called unsign which modifies executables to clear out signatures.
https://github.com/steakknife/unsign
I ran into this issue today. I can confirm that the --remove-signature option to Apple's codesign is (and remains, six years after the OP asked this question) seriously buggy.
For a little background, Xcode (and Apple's command line developer tools) include the codesign utility, but there is not included a tool for removing signatures. However, as this is something that needs to be done in certain situations pretty frequently, there is included a completely undocumented option:
codesign --remove-signature which (one assumes, given lack of documentation) should, well, be fairly self-explanatory but unfortunately, it rarely works as intended without some effort. So I ended up writing a script that should take care of the OP's problem, mine, and similar. If enough people find it here and find it useful, let me know and I'll put it on GitHub or something.
#!/bin/sh # codesign_remove_for_real -- working `codesign --remove-signature`
# (c) 2018 G. Nixon. BSD 2-clause minus retain/reproduce license requirements.
total_size(){
# Why its so damn hard to get decent recursive filesize total in the shell?
# - Darwin `du` doesn't do *bytes* (or anything less than 512B blocks)
# - `find` size options are completely non-standardized and doesn't recurse
# - `stat` is not in POSIX at all, and its options are all over the map...
# - ... etc.
# So: here we just use `find` for a recursive list of *files*, then wc -c
# and total it all up. Which sucks, because we have to read in every bit
# of every file. But its the only truly portable solution I think.
find "$#" -type f -print0 | xargs -0n1 cat | wc -c | tr -d '[:space:]'
}
# Get an accurate byte count before we touch anything. Zero would be bad.
size_total=$(total_size "$#") && [ $size_total -gt 0 ] || exit 1
recursively_repeat_remove_signature(){
# `codesign --remove-signature` randomly fails in a few ways.
# If you're lucky, you'll get an error like:
# [...]/codesign_allocate: can't write output file: [...] (Invalid argument)
# [...] the codesign_allocate helper tool cannot be found or used
# or something to that effect, in which case it will return non-zero.
# So we'll try it (suppressing stderr), and if it fails we'll just try again.
codesign --remove-signature --deep "$#" 2>/dev/null ||
recursively_repeat_remove_signature "$#"
# Unfortunately, the other very common way it fails is to do something? that
# hugely increases the binary size(s) by a seemingly arbitrary amount and
# then exits 0. `codesign -v` will tell you that there's no signature, but
# there are other telltale signs its not completely removed. For example,
# if you try stripping an executable after this, you'll get something like
# strip: changes being made to the file will invalidate the code signature
# So, the solution (well, my solution) is to do a file size check; once
# we're finally getting the same result, we've probably been sucessful.
# We could of course also use checksums, but its much faster this way.
[ $size_total == $(total_size "$#") ] ||
recursively_repeat_remove_signature "$#"
# Finally, remove any leftover _CodeSignature directories.
find "$#" -type d -name _CodeSignature -print0 | xargs -0n1 rm -rf
}
signature_info(){
# Get some info on code signatures. Not really required for anything here.
for info in "-dr-" "-vv"; do codesign $info "$#"; done # "-dvvvv"
}
# If we want to be be "verbose", check signature before. Un/comment out:
# echo >&2; echo "Current Signature State:" >&2; echo >&2; signature_info "$#"
# So we first remove any extended attributes and/or ACLs (which are common,
# and tend to interfere with the process here) then run our repeat scheme.
xattr -rc "$#" && chmod -RN "$#" && recursively_repeat_remove_signature "$#"
# Done!
# That's it; at this point, the executable or bundle(s) should sucessfully
# have truly become stripped of any code-signing. To test, one could
# try re-signing it again with an ad-hoc signature, then removing it again:
# (un/comment out below, as you see fit)
# echo >&2 && echo "Testing..." >&2; codesign -vvvvs - "$#" &&
# signature_info "$#" && recursively_repeat_remove_signature "$#"
# And of course, while it sometimes returns false positives, lets at least:
codesign -dvvvv "$#" || echo "Signature successfully removed!" >&2 && exit 0
Here's the source for codesign which lists all options, including those not covered by the command-line -h and man page.
Also, here is Apple's tech note on recent changes in how code-signing works
I agree that there's something strange going on when you did --remove-signature.
However, instead of trying to un-code-sign, you should change the way your user put extra files in the Resources. Instead, designate a certain path, usually
~/Library/Application Support/Name_Of_Your_App/
or maybe
~/Library/Application Support/Name_Of_Your_App/Resources/
and ask the user to put extra files there. Then, in your code, always check for the directory in addition to the files in the Resources when you need to read a file.
On a second reading of this question, another thought: perhaps a better approach to accomplish what the ultimate goal of the question is would be not to remove the signatures, but to have users (via a script/transparently) re-sign the app after modification, using an ad-hoc signature. That is, codesign -fs - [app], I believe. See https://apple.stackexchange.com/questions/105588/anyone-with-experience-in-hacking-the-codesigning-on-os-x
Related
Background
I am well aware of how git status works, and even about git ls-files. Usually git status is all I need and want, it perfectly answers the question: "What is my status, and what files need my attention?"
However, I have been unable to find a quick command that answers the following question: "What files do I have, and what is their respective status?" So, I need a full listing of the directory (like ls -la) with a column that shows the status of each file/directory.
What I have tried
git status -s --ignored comes quite close to the output format that I want, but it just won't list the files that are unchanged between HEAD, index, and working directory. Also, it will recurse into directories.
git ls-files seems to be able to provide all the required info in scriptable form, but I've been unable to stop it from recursive listing the contents of all directories.
Obviously, I could hack something together that takes the output of these two commands and provides the view I would like to have. However, I would hate to reinvent the wheel if there is already some usable command out there.
Question
Is there some way of listing all files in a directory with their respective git status?
I want a full listing showing exactly the same files that ls would show.
Notes
This other question does not answer mine, because I definitely want an ls equivalent. Including unmodified, ignored, and untracked files, but excluding directory contents.
To restrict the paths Git inspects to just the current directory, use its Unix glob pathspecs. Since git status does a lot of checking against the index and against HEAD, use that, and to fill in the rest of the files ls would show you, use ls, just munge its output to have the same format as git status's output and take only the ones git status didn't already list.
( git status -s -- ':(glob)*'; ls -A --file-type | awk '{print " "$0}' ) \
| sort -t$'\n' -usk1.4
:(glob) tells Git the rest of the pathspec's a Unix glob, i.e. that * should match only one level, just like a (dotglob-enabled) shell wildcard¹.
The -t$'\n' tells sort that the field separator is a newline, i.e. it's all one big field, and -usk1.4 says uniquify, only take the first of a run, stable, preserve input order where it doesn't violate sort key order (which is a little slower so you have to ask for that specifically), k1.4 says the key starts at the first field, the fourth character in that field, with no end given so from there to the end.
¹ Why they decided to make pathspecs match neither like shell specs nor like gitignore specs by default, I might never bother learning, since I so much prefer ignorantly disapproving of their annoying choice.
Because output of git status -s is enough, let's just make a bash routine around this function! Then for unchanged files we could to echo the proper signaling manually. Following the specification we might use two symbols of space ' ' for this purpose either some another symbol. E.g. for directories, which are not tracked by Git anyway, selected symbol '_' as status code:
for FILE in *
do
if [[ -f $FILE ]]
then
if ! [[ $(git status -s $FILE) ]]
then
# first two simbols below is a two-letter status code
echo " $FILE"
else
git status -s "$FILE"
fi
fi
if [[ -d $FILE ]]
then
# first two symbols just selected as status code for directories
echo "__ $FILE"
fi
done
The script works in the same manner as ls. It can be written in one line using ; as well.
At the moment bash takes about 2 seconds to load. I have ran bash with -x flag and I am seeing the output and it seems as though PATH is being loaded many times in cygwin. The funny thing is I use the same file in linux environment, but it works fine, without the reload problem. Could the following cause the problem?
if [ `uname -o` = "Cygwin" ]; then
....
fi
As you've noted in your answer, the problem is Cygwin's bash-completion package. The quick and easy fix is to disable bash-completion, and the correct way to do that is to run Cygwin's setup.exe (download it again if you need to) and select to uninstall that package.
The longer solution is to work through the files in /etc/bash_completion.d and disable the ones you don't need. On my system, the biggest culprits for slowing down Bash's load time (mailman, shadow, dsniff and e2fsprogs) all did exactly nothing, since the tools they were created to complete weren't installed.
If you rename a file in /etc/bash_completion.d to have a .bak extension, it'll stop that script being loaded. Having disabled all but a select 37 scripts on one of my systems in that manner, I've cut the average time for bash_completion to load by 95% (6.5 seconds to 0.3 seconds).
In my case that was windows domain controller.
I did this to find the issue:
I started with a simple, windows cmd.exe and the, typed this:
c:\cygwin\bin\strace.exe c:\cygwin\bin\bash
In my case, I noticed a following sequence:
218 12134 [main] bash 11304 transport_layer_pipes::connect: Try to connect to named pipe: \\.\pipe\cygwin-c5e39b7a9d22bafb-lpc
45 12179 [main] bash 11304 transport_layer_pipes::connect: Error opening the pipe (2)
39 12218 [main] bash 11304 client_request::make_request: cygserver un-available
1404719 1416937 [main] bash 11304 pwdgrp::fetch_account_from_windows: line: <CENSORED_GROUP_ID_#1>
495 1417432 [main] bash 11304 pwdgrp::fetch_account_from_windows: line: <CENSORED_GROUP_ID_#2>
380 1417812 [main] bash 11304 pwdgrp::fetch_account_from_windows: line: <CENSORED_GROUP_ID_#3>
etc...
The key thing was identifying the client_request::make_request: cygserver un-available line. You can see, how after that, cygwin tries to fetch every single group from windows, and execution times go crazy.
A quick google revealed what a cygserver is:
https://cygwin.com/cygwin-ug-net/using-cygserver.html
Cygserver is a program which is designed to run as a background
service. It provides Cygwin applications with services which require
security arbitration or which need to persist while no other cygwin
application is running.
The solution was, to run the cygserver-config and then net start cygserver to start the Windows service. Cygwin startup times dropped significantly after that.
All of the answers refer to older versions of bash_completion, and are irrelevant for recent bash_completion.
Modern bash_completion moved most of the completion files to /usr/share/bash-completion/completions by default, check the path on your system by running
# pkg-config --variable=completionsdir bash-completion
/usr/share/bash-completion/completions
There are many files in there, one for each command, but that is not a problem, since they are loaded on demand the first time you use completion with each command. The old /etc/bash_completion.d is still supported for compatibility, and all files from there are loaded when bash_completion starts.
# pkg-config --variable=compatdir bash-completion
/etc/bash_completion.d
Use this script to check if there are any stale files left in the old dir.
#!/bin/sh
COMPLETIONS_DIR="$(pkg-config --variable=completionsdir bash-completion)"
COMPAT_DIR="$(pkg-config --variable=compatdir bash-completion)"
for file in "${COMPLETIONS_DIR}"/*; do
file="${COMPAT_DIR}/${file#${COMPLETIONS_DIR}/}"
[ -f "$file" ] && printf '%s\n' $file
done
It prints the list of files in compat dir that are also present in the newer (on-demand) completions dir. Unless you have specific reasons to keep some of them, review, backup and remove all of those files.
As a result, the compat dir should be mostly empty.
Now, for the most interesting part - checking why bash startup is slow.
If you just run bash, it will start a non-login, interactive shell - this one on Cygwin source /etc/bash.bashrc and then ~/.bashrc. This most likely doesn't include bash completion, unless you source it from one of rc files. If you run bash -l (bash --login), start Cygwin Terminal (depends on your cygwin.bat), or log in via SSH, it will start a login, interactive shell - which will source /etc/profile, ~/.bash_profile, and the aforementioned rc files. The /etc/profile script itself sources all executable .sh files in /etc/profile.d.
You can check how long each file takes to source. Find this code in /etc/profile:
for file in /etc/profile.d/*.$1; do
[ -e "${file}" ] && . "${file}"
done
Back it up, then replace it with this:
for file in /etc/profile.d/*.$1; do
TIMEFORMAT="%3lR ${file}"
[ -e "${file}" ] && time . "${file}"
done
Start bash and you will see how long each file took. Investigate files that take a significant amount of time. In my case, it was bash_completion.sh and fzf.sh (fzf is fuzzy finder, a really nice complement to bash_completion). Now the choice is to disable it or investigate further. Since I wanted to keep using fzf shortcuts in bash, I investigated, found the source of the slowdown, optimized it, and submitted my patch to fzf's repo (hopefully it will be accepted).
Now for the biggest time spender - bash_completion.sh. Basically that script sources /usr/share/bash-completion/bash_completion. I backed up that file, then edited it. On the last page there is for loop that sources all the files in compat dir - /etc/bash_completion.d. Again, I added TIMEFORMAT and time, and saw which script was causing the slow starting. It was zzz-fzf (fzf package). I investigated and found a subshell ($()) being executed multiple times in a for loop, rewrote that part without using a subshell, making the script work quickly. I already submitted my patch to fzf's repo.
The biggest reason for all these slowdowns is: fork is not supported by Windows process model, Cygwin did a great job emulating it, but it's painfully slow compared to a real UNIX. A subshell or a pipeline that does very little work by itself spends most of it's execution time for fork-ing. E.g. compare the execution times of time echo msg (0.000s on my Cygwin) vs time echo $(echo msg) (0.042s on my Cygwin) - day and night. The echo command itself takes no appreciable time to execute, but creating a subshell is very expensive. On my Linux system, these commands take 0.000s and 0.001s respectively. Many packages Cygwin has are developed by people who use Linux or other UNIX, and can run on Cygwin unmodified. So naturally these devs feel free to use subshells, pipelines and other features wherever convenient, since they don't feel any significant performance hit on their system, but on Cygwin those shell scripts might run tens and hundreds of times slower.
Bottom line, if a shell script works slowly in Cygwin - try to locate the source of fork calls and rewrite the script to eliminate them as much as possible.
E.g. cmd="$(printf "$1" "$2")" (uses one fork for subshell) can be replaced with printf -v cmd "$1" "$2".
Boy, it came out really long. Any people still reading up to here are real heros. Thanks :)
I know this is an old thread, but after a fresh install of Cygwin this week I'm still having this problem.
Instead of handpicking all of the bash_completion files, I used this line to implement #me_and's approach for anything that isn't installed on my machine. This significantly reduced the startup time of bash for me.
In /etc/bash_completion.d, execute the following:
for i in $(ls|grep -v /); do type $i >/dev/null 2>&1 || mv $i $i.bak; done
New answer for an old thread, relating to the PATH of the original question.
Most of the other answers deal with the bash startup. If you're seeing a slow load time when you run bash -i within the shell, those may apply.
In my case, bash -i ran fast, but anytime I opened a new shell (be it in a terminal or in xterm), it took a really long time. If bash -l is taking a long time, it means it's the login time.
There are some approaches at the Cygwin FAQ at https://cygwin.com/faq/faq.html#faq.using.startup-slow but they didn't work for me.
The original poster asked about the PATH, which he diagnosed using bash -x. I too found that although bash -i was fast, bash -xl was slow and showed a lot of information about the PATH.
There was such a ridiculously long Windows PATH that the login process kept on running programs and searching the entire PATH for the right program.
My solution: Edit the Windows PATH to remove anything superfluous. I'm not sure which piece I removed that did the trick, but the login shell startup went from 6 seconds to under 1 second.
YMMV.
My answer is the same as npe's above. But, since I just joined, I cannot comment or even upvote it! I hope this post doesn't get deleted because it offer reassurance for anyone looking for an answer to the same problem.
npe's solution worked for me. There's only one caveat - I had to close all cygwin processes before I got the best out of it. That includes running cygwin services, like sshd, and the ssh-agent that I start from my login scripts. Before that, the window for the cygwin terminal would appear instantly but hang for several seconds before it presents the prompt. And it hanged for several seconds upon closing the window. After I killed all processes and started the cygserver service (btw I prefer to use the Cygwin way - 'cygrunsrv -S cygserver', than 'net start cygserver'; I don't know if it makes any practical difference) it starts immediately. So thanks to npe again!
I'm on a corporate network with a pretty complicated setup, and it seems that really kills cygwin startup times. Related to npe's answer, I also had to follow some of the steps laid out here: https://cygwin.com/faq/faq.html#faq.using.startup-slow
Another cause for AD client system is slow DC replies, commonly observed in configurations with remote DC access. The Cygwin DLL queries information about every group you're in to populate the local cache on startup. You may speed up this process a little by caching your own information in local files. Run these commands in a Cygwin terminal with write access to /etc:
getent passwd $(id -u) > /etc/passwd
getent group $(id -G) > /etc/group
Also, set /etc/nsswitch.conf as follows:
passwd: files db
group: files db
This will limit the need for Cygwin to contact the AD domain controller (DC) while still allowing for additional information to be retrieved from DC, such as when listing remote directories.
After doing that plus starting the cygserver my cygwin startup time dropped significantly.
As someone mentioned above, one possible issue is the PATH environment variable contains too much path, cygwin will search all of them. I prefer direct edit the /etc/profile, just overwrite the PATH variable to cygwin related path, e.g. PATH="/usr/local/bin:/usr/bin". Add additional path if you want.
I wrote a Bash function named 'minimizecompletion' for inactivating not needed completion scripts.
Completion scripts can add more than one completion specification or have completion specifications for shell buildins, therefore it is not sufficient to compare script names with executable files found in $PATH.
My solution is to remove all loaded completion specifications, to load a completion script and check if it has added new completion specifications. Depending on this it is inactivated by adding .bak to the script file name or it is activated by removing .bak. Doing this for all 182 scripts in /etc/bash_completion.d results in 36 active and 146 inactive completion scripts reducing the Bash start time by 50% (but it should be clear this depends on installed packages).
The function also checks inactivated completion scripts so it can activate them when they are needed for new installed Cygwin packages. All changes can be undone with argument -a that activates all scripts.
# Enable or disable global completion scripts for speeding up Bash start.
#
# Script files in directory '/etc/bash_completion.d' are inactived
# by adding the suffix '.bak' to the file name; they are activated by
# removing the suffix '.bak'. After processing all completion scripts
# are reloaded by calling '/etc/bash_completion'
#
# usage: [-a]
# -a activate all completion scripts
# output: statistic about total number of completion scripts, number of
# activated, and number of inactivated completion scripts; the
# statistic for active and inactive completion scripts can be
# wrong when 'mv' errors occure
# return: 0 all scripts are checked and completion loading was
# successful; this does not mean that every call of 'mv'
# for adding or removing the suffix was successful
# 66 the completion directory or loading script is missing
#
minimizecompletion() {
local arg_activate_all=${1-}
local completion_load=/etc/bash_completion
local completion_dir=/etc/bash_completion.d
(
# Needed for executing completion scripts.
#
local UNAME='Cygwin'
local USERLAND='Cygwin'
shopt -s extglob progcomp
have() {
unset -v have
local PATH="$PATH:/sbin:/usr/sbin:/usr/local/sbin"
type -- "$1" &>/dev/null && have='yes'
}
# Print initial statistic.
#
printf 'Completion scripts status:\n'
printf ' total: 0\n'
printf ' active: 0\n'
printf ' inactive: 0\n'
printf 'Completion scripts changed:\n'
printf ' activated: 0\n'
printf ' inactivated: 0\n'
# Test the effect of execution for every completion script by
# checking the number of completion specifications after execution.
# The completion scripts are renamed depending on the result to
# activate or inactivate them.
#
local completions total=0 active=0 inactive=0 activated=0 inactivated=0
while IFS= read -r -d '' f; do
((++total))
if [[ $arg_activate_all == -a ]]; then
[[ $f == *.bak ]] && mv -- "$f" "${f%.bak}" && ((++activated))
((++active))
else
complete -r
source -- "$f"
completions=$(complete | wc -l)
if (( $completions > 0 )); then
[[ $f == *.bak ]] && mv -- "$f" "${f%.bak}" && ((++activated))
((++active))
else
[[ $f != *.bak ]] && mv -- "$f" "$f.bak" && ((++inactivated))
((++inactive))
fi
fi
# Update statistic.
#
printf '\r\e[6A\e[15C%s' "$total"
printf '\r\e[1B\e[15C%s' "$active"
printf '\r\e[1B\e[15C%s' "$inactive"
printf '\r\e[2B\e[15C%s' "$activated"
printf '\r\e[1B\e[15C%s' "$inactivated"
printf '\r\e[1B'
done < <(find "$completion_dir" -maxdepth 1 -type f -print0)
if [[ $arg_activate_all != -a ]]; then
printf '\nYou can activate all scripts with %s.\n' "'$FUNCNAME -a'"
fi
if ! [[ -f $completion_load && -r $completion_load ]]; then
printf 'Cannot reload completions, missing %s.\n' \
"'$completion_load'" >&2
return 66
fi
)
complete -r
source -- "$completion_load"
}
This is an example output and the resulting times:
$ minimizecompletion -a
Completion scripts status:
total: 182
active: 182
inactive: 0
Completion scripts changed:
activated: 146
inactivated: 0
$ time bash -lic exit
logout
real 0m0.798s
user 0m0.263s
sys 0m0.341s
$ time minimizecompletion
Completion scripts status:
total: 182
active: 36
inactive: 146
Completion scripts changed:
activated: 0
inactivated: 146
You can activate all scripts with 'minimizecompletion -a'.
real 0m17.101s
user 0m1.841s
sys 0m6.260s
$ time bash -lic exit
logout
real 0m0.422s
user 0m0.092s
sys 0m0.154s
In a shell, I run following commands without problem,
ls -al
!ls
the second invocation to ls also list files with -al flag. However, when I put the above script to a bash script, complaints are thrown,
!ls, command not found.
how to realise the same effects in script?
You would need to turn on both command history and !-style history expansion in your script (both are off by default in non-interactive shells):
set -o history
set -o histexpand
The expanded command is also echoed to standard error, just like in an interactive shell. You can prevent that by turning on the histverify shell option (shopt -s histverify), but in a non-interactive shell, that seems to make the history expansion a null-op.
Well, I wanted to have this working as well, and I have to tell everybody that the set -o history ; set -o histexpand method will not work in bash 4.x. It's not meant to be used there, anyway, since there are better ways to accomplish this.
First of all, a rather trivial example, just wanting to execute history in a script:
(bash 4.x or higher ONLY)
#!/bin/bash -i
history
Short answer: it works!!
The spanking new -i option stands for interactive, and history will work. But for what purpose?
Quoting Michael H.'s comment from the OP:
"Although you can enable this, this is bad programming practice. It will make your scripts (...) hard to understand. There is a reason it is disabled by default. Why do you want to do this?"
Yes, why? What is the deeper sense of this?
Well, THERE IS, which I'm going to demonstrate in the follow-up section.
My history buffer has grown HUGE, while some of those lines are script one-liners, which I really would not want to retype every time. But sometimes, I also want to alter these lines a little, because I probably want to give a third parameter, whereas I had only needed two in total before.
So here's an ideal way of using the bash 4.0+ feature to invoke history:
$ history
(...)
<lots of lines>
(...)
1234 while IFS='whatever' read [[ $whatever -lt max ]]; do ... ; done < <(workfile.fil)
<25 more lines>
So 1234 from history is exactly the line we want. Surely, we could take the mouse and move there, chucking the whole line in the primary buffer? But we're on *NIX, so why can't we make our life a bit easier?
This is why I wrote the little script below. Again, this is for bash 4.0+ ONLY (but might be adapted for bash 3.x and older with the aforementioned set -o ... stuff...)
#!/bin/bash -i
[[ $1 == "" ]] || history | grep "^\s*$1" |
awk '{for (i=2; i<=NF; i++) printf $i" "}' | tr '\n' '\0'
If you save this as xselauto.sh for example, you may invoke
$ ./xselauto.sh 1234
and the contents of history line #1234 will be in your primary buffer, ready for re-use!
Now if anyone still says "this has no purpose AFAICS" or "who'd ever be needing this feature?" - OK, I won't care. But I would no longer want to live without this feature, as I'm just too lazy to retype complex lines every time. And I wouldn't want to touch the mouse for each marked line from history either, TBH. This is what xsel was written for.
BTW, the tr part of the pipe is a dirty hack which will prevent the command from being executed. For "dangerous" commands, it is extremely important to always leave the user a way to look before he/she hits the Enter key to execute it. You may omit it, but ... you have been warned.
P.S. This scriptlet is in fact a workaround, simulating !1234 typed on a bash shell. As I could never make the ! work directly in a script (echo would never let me reveal the contents of history line 1234), I worked around the problem by simply greping for the line I wanted to copy.
History expansion is part of the interactive command-line editing features of a shell, not part of the scripting language. It's not generally available in the context of a script, only when interacting with a (pseudo-)human operator. (pseudo meaning that it can be made to work with things like expect or other keystroke repeating automation tools that generally try to play act a human, not implying that any particular operator might be sub-human or anything).
I'm working on a small app based on ffmpeg, and I read a tutorial made for ubuntu where they advise to use the command hash on the produced executable.
I'm curious about that command, did you ever use it? For which purpose?
When I run it in my source folder, I get this (once compiled)
$ hash
hits command
1 /usr/bin/strip
1 /usr/local/bin/ffmpeg
1 /usr/bin/svn
4 /usr/local/bin/brew
2 /usr/bin/git
1 /bin/rm
1 /bin/cat
1 /usr/bin/ld
1 /bin/sh
4 /usr/bin/man
5 /usr/bin/make
4 /usr/bin/otool
15 /bin/ls
6 /usr/bin/open
2 /usr/bin/clear
Looks like a summary of my bash_history…
When I run it on an executable file, I do not have lots of lines displayed, and nothing seems to changes in that application ?
$ md5 ffserver
MD5 (ffserver) = 2beac612e5efd6ee4a827ae0893ee338
$ hash ffserver
$ md5 ffserver
MD5 (ffserver) = 2beac612e5efd6ee4a827ae0893ee338
When I look for the man, it just says it's a builtin function. Really useful :)
It does work (let say exist) on Linux and on MacOSX.
hash isn't actually your history; it is a bash(1) shell built-in that maintains a hash table of recently executed programs:
Bash uses a hash table to remember the full pathnames of executable files (see hash under SHELL BUILTIN COMMANDS below). A full search of the directories in PATH is performed only if the command is not found in the hash table.
(From bash(1).)
The guide your found may have suggested running it just to see which ffmpeg command was going to be executed by the next step; perhaps there is an ffmpeg program supplied by the distribution packaging, and they wanted to make sure the new one would be executed instead of the distro-supplied one if you just typed ffmpeg at the shell.
It seems a stretch, because it would also require having the directory containing the new ffmpeg in the PATH before the distro-provided version, and there's no guarantee of that.
If you use commands that might not be installed on the system, check for their availability and tell the user what's missing. From Scripting with style
Example:
NEEDED_COMMANDS="sed awk lsof who"
missing_counter=0
for needed_command in $NEEDED_COMMANDS; do
if ! hash "$needed_command" >/dev/null 2>&1; then
printf "Command not found in PATH: %s\n" "$needed_command" >&2
((missing_counter++))
fi
done
if ((missing_counter > 0)); then
printf "Minimum %d commands are missing in PATH, aborting" "$missing_counter" >&2
exit 1
fi
I'm making a bash script which presents a command line to the user.
The cli code is as this:
#!/bin/bash
cmd1() {
echo $FUNCNAME: "$#"
}
cmd2() {
echo $FUNCNAME: "$#"
}
cmdN() {
echo $FUNCNAME: "$#"
}
__complete() {
echo $allowed_commands
}
shopt -qs extglob
fn_hide_prefix='__'
allowed_commands="$(declare -f | sed -ne '/^'$fn_hide_prefix'.* ()/!s/ ().*//p' | tr '\n' ' ')"
complete -D -W "this should output these words when you hit TAB"
echo "waiting for commands"
while read -ep"-> "; do
history -s $REPLY
case "$REPLY" in
#(${allowed_commands// /|})?(+([[:space:]])*)) $REPLY ;;
\?) __complete ;;
*) echo "invalid command: $REPLY" ;;
esac
done
Clarification: made and tested in Bash 4
So, "read -e" gives readline capabilities, i can recall commands, edit the input line, etc. What i cannot do in any way is to have readline's tab completion to work!!
I tried two things:
How it should be supposedly done: using the bash builtins "complete" and "compgen", which is reported to work here Update: it's not reported to work in scripts.
This ugly workaround
Why doesn't readline behave correctly when using "complete" inside the script? it works when i try it from bash in interactive mode...
After trying a custom completion script that I know works (I use it every day) and running into the same issue (when rigging it up similar to yours), I decided to snoop through the bash 4.1 source, and found this interesting block in bash-4.1/builtins/read.def:edit_line():
old_attempted_completion_function = rl_attempted_completion_function;
rl_attempted_completion_function = (rl_completion_func_t *)NULL;
if (itext)
{
old_startup_hook = rl_startup_hook;
rl_startup_hook = set_itext;
deftext = itext;
}
ret = readline (p);
rl_attempted_completion_function = old_attempted_completion_function;
old_attempted_completion_function = (rl_completion_func_t *)NULL;
It appears that before readline() is called, it resets the completion function to null for some reason that only a bash-hacking long beard might know. Thus, doing this with the read builtin may simply be hard-coded to be disabled.
EDIT: Some more on this: The wrapping code to stop completion in the read builtin occurred between bash-2.05a and bash-2.05b. I found this note in that version's bash-2.05b/CWRU/changelog file:
edit_line (called by read -e) now just does readline's filename completion by setting rl_attempted_completion_function to NULL, since e.g., doing command completion for the first word on the line wasn't really useful
I think it's a legacy oversight, and since programmable completion has come a long way, what you're doing is useful. Maybe you can ask them to add it back in, or just patch it yourself, if that'd be feasible for what you're doing.
Afraid I don't have a different solution aside from what you've come up with so far, but at least we know why it doesn't work with read.
EDIT2: Right, here's a patch I just tested that seems to "work". Passes all unit and reg tests, and shows this output from your script when run using the patched bash, as you expected:
$ ./tabcompl.sh
waiting for commands
-> **<TAB>**
TAB hit output should these this when words you
->
As you'll see, I just commented out those 4 lines and some timer code to reset the rl_attempted_completion_function when read -t is specified and a timeout occurs, which is no longer necessary. If you're going to send Chet something, you may wish to excise the entirety of the rl_attempted_completion_function junk first, but this will at least allow your script to behave properly.
Patch:
--- bash-4.1/builtins/read.def 2009-10-09 00:35:46.000000000 +0900
+++ bash-4.1-patched/builtins/read.def 2011-01-20 07:14:43.000000000 +0900
## -394,10 +394,12 ##
}
old_alrm = set_signal_handler (SIGALRM, sigalrm);
add_unwind_protect (reset_alarm, (char *)NULL);
+/*
#if defined (READLINE)
if (edit)
add_unwind_protect (reset_attempted_completion_function, (char *)NULL);
#endif
+*/
falarm (tmsec, tmusec);
}
## -914,8 +916,10 ##
if (bash_readline_initialized == 0)
initialize_readline ();
+/*
old_attempted_completion_function = rl_attempted_completion_function;
rl_attempted_completion_function = (rl_completion_func_t *)NULL;
+*/
if (itext)
{
old_startup_hook = rl_startup_hook;
## -923,8 +927,10 ##
deftext = itext;
}
ret = readline (p);
+/*
rl_attempted_completion_function = old_attempted_completion_function;
old_attempted_completion_function = (rl_completion_func_t *)NULL;
+*/
if (ret == 0)
return ret;
Keep in mind the patched bash would have to be distributed or made available somehow wherever people would be using your script...
I've been struggling with same issue for some time now and I think I have a solution that works, in my real world case I'm using compgen to generate possible completions. But here is an example that illustrates the core logic:
#!/bin/bash
set -o emacs;
tab() {
READLINE_LINE="foobar"
READLINE_POINT="${#READLINE_LINE}"
}
bind -x '"\t":"tab"';
read -ep "$ ";
Set the emacs option to enable key binding, bind the tab key to a function, change READLINE_LINE to update the line after the prompt, and set READLINE_POINT to reflect the line's new longer length.
In my use case I actually mimic the COMP_WORDS, COMP_CWORD and COMPREPLY variables but this should be sufficient to understand how to go about adding custom tab completion when using read -ep.
You must update READLINE_LINE to change the prompt line (completion single match), printing to stdin prints before the prompt as readline has put the terminal in raw mode and is capturing input.
Well, it seems i finally stumped on the answer, and it sadly is: actually there isn't full support for readline when interfacing it via "read -e".
The answer is given by the BASH maintainer, Chet Ramey. In this thread the exact same issue is addressed:
I'm writing a script with a command line interpreter and I can most things
working (eg. history etc.) except for one thing. The filename completion
works well for some of the commands, but I'd like to use other completion
options for others. Works well from the "real" command line, but I can't
get it to work properly in my "read -e, eval" loop..
You won't be able to do it. `read -e' uses only the readline default
completions.
Chet
So, unless i'm missing something //rant// while bash hands to the programmer the "read -e" mechanism as the mean for full, proper CLI user interfacing, the functionality is crippled, even though the underlying mechanism (readline) works and integrates with the rest of bash flawlessly //end rant//
I have exposed the question to the kind folks at #bash in freenode and been suggested to try with a Readline wrapper like rlfe or rlwrap.
Finally, i contacted Chet himself by mail yesterday, and he confirmed that this is by design, and that he doesn't feel like changing it as the only use case for programmable completion into "read", i.e. presenting a list of commands to the script user, doesn't look like a compelling reason to spend time working on this. Nevertheless he expressed that in case someone actually does the work he would certainly look at the result.
IMHO, not considering worth of the effort the ability to bring up a full CLI with just 5 lines of code, something one wish were possible in a lot of languages, is a mistake.
In this context, i think Simon's answer is brilliant and right in place. I'll try to follow your steps and perhaps with some luck i'll get more info. Been a while since i don't hack in C however, and i assume the amount of code i'll have to grasp to implement will not be trivial. But anyway i'll try.
I'm not sure if this exactly answers the OP question - but I was searching for which command could one use, to obtain the default bash tab completion of known executable commands (as per $PATH), as shown when pressing TAB. Since I was first led to this question (which I think is related), I thought I'd post a note here.
For instance, on my system, typing lua and then TAB gives:
$ lua<TAB>
lua lua5.1 luac luac5.1 lualatex luatex luatools
It turns out, there is a bash built-in (see #949006 Linux command to list all available commands and aliases), called compgen - and I can feed it with the same string lua as in the interactive case, and obtain the same results as if I pressed TAB:
$ compgen -c lua
luac
lua5.1
lua
luac5.1
luatex
lualatex
luatools
... and that is exactly what I was looking for :)
Hope this helps someone,
Cheers!
If you're going to that much effort, why not just add the cost of a fork or two and use something that is more than capable of providing everything you want. https://github.com/hanslub42/rlwrap
#!/bin/bash
which yum && yum install rlwrap
which zypper && zypper install rlwrap
which port && port install rlwrap
which apt-get && apt-get install rlwrap
REPLY=$( rlwrap -o cat )
Or as the man page puts it:
In a shell script, use rlwrap in ’one−shot’ mode as a replacement for read
order=$(rlwrap -p Yellow -S 'Your pizza? ' -H past_orders -P Margherita -o cat)