I have the following nice little bash function to make searches in my history (here for example, looking for ls commands):
history | grep --color=always ls | sort -k2 | uniq -f 1 | sort -n
I packaged it into a bash script, linked to an alias (histg) and it works great:
#!/bin/bash
if [ "$1" == "-h" ]; then
echo "A bash script to find patterns in history, avoiding duplicates (also non consecutive)"
echo "expands to: XX"
exit 0
fi
HISTSIZE=100000 # need this, because does not read .bashrc so does not know how big a HISTSIZE
HISTFILE=~/.bash_history # Or wherever you bash history file lives
set -o history # enable history
OUTPUT="$(history | grep --color=always $1 | sort -k2 | uniq -f 1 | sort -n)"
echo "${OUTPUT}"
Typically, I get this kind of output:
$ histg SI
16424 git commit -m "working on SI"
16671 git commit -m "updated SI"
17782 cd SI/
However I want to do one more improvement, and I do not know how to proceed. I want to be able to quickly call those commands again, but as you see I have a big hist, so typing !17782 is a bit long. If the current size of my history is for example 17785 (I have a max history size 100000), I would like to see:
$ histg SI
16424 -1361 git commit -m "working on SI"
16671 -1114 git commit -m "updated SI"
17782 -3 cd ~/Desktop/crrt/wrk/SI/
so that I can type in -3
Any idea how I can adapt my bash command to add this column?
In a first try, my code was not working as expected because the negative numbers didn't match: the current session history was not taken into account. So I changed your script to a function (to add to .bashrc). The tricky part is handled by awk:
function histg() {
history | grep --color=always $1 | sort -k2 | uniq -f 1 | sort -n \
| awk '
BEGIN { hist_size = '$(history|wc -l)' }
{
n = $1; $1 = ""
printf("%-7i %-7i %s\n", n, n - hist_size, $0)
}'
history -d $(history 1)
}
The last line deletes the call to histg in history, so the negative numbers still keep sense.
Related
I have this folder in windows
if I do a simple ls , find, either in bash (cygwin) or msdos, it shows me like this.
$ ls -1
su-01-01.jpg
su-01-02-03.jpg
su-01-12-13.jpg
su-01-14.jpg
su-01-15.jpg
su-01-16.jpg
su-01-18.jpg
su-01-19.jpg
su-01-20.jpg
su-01-21.jpg
su-01-31.jpg
su-01-34.jpg
su-01-35.jpg
su-01-38.jpg
su-01-39.jpg
su-01-42-43.jpg
su-01-44.jpg
su-01-45.jpg
su-01-47.jpg
su-01-48.jpg
su01-00.jpg
su01-04.jpg
su01-05.jpg
su01-06.jpg
su01-07.jpg
su01-08.jpg
I have tried ordering and it does not take into account 0 00 1
$ ls -1 |sort -V
su01-00.jpg
su01-04.jpg
su01-05.jpg
su01-06.jpg
su01-07.jpg
su01-08.jpg
su01-09.jpg
su01-10.jpg
su01-11.jpg
su01-22-23.jpg
su01-24.jpg
su01-25.jpg
su01-26.jpg
su01-27.jpg
su01-28-29.jpg
su01-30.jpg
su01-32.jpg
su01-33.jpg
su01-40-41.jpg
su-01-01.jpg
su-01-02-03.jpg
su-01-12-13.jpg
su-01-14.jpg
su-01-15.jpg
but how do I make it ignore the (-)?
thank you very much for your help
find doesn't guaranty alphabetical ordering; ls and sort do, but the char - value is 45 while the 0 char value is 48, so su- will come ahead of the su0 in an alphabetical sorting.
While a printf '%s\n' su* | LANG=en_US.utf8 sort -n seems to display the files the way you want, the best thing to do for making your life easier would be to rename some of the files:
#!/bin/bash
for f in su0*
do
mv "$f" "su-0${f#su0}"
done
Update
renaming the files to 001.jpg 002.jpg ...
#!/bin/bash
shopt -s nullglob
n=1
while IFS='' read -r file
do
printf -v newname '%03d.%s' "$((n++))" "${file##*.}"
printf '%q %q %q\n' mv "$file" "$newname"
done < <(
printf '%s\n' su* |
sed -nE 's,su-?([^/]*)$,\1/&,p' |
LANG=C sort -nt '-' |
sed 's,[^/]*/,,'
)
The simplest way to control the sort order in Bash, both for ls and sort, so to set your LANG variable to the locale you want.
In your .bashrc or .profile, add
export LANG=en_US.utf8
and then
ls -1
or
ls -1 | sort
will output the order you're looking for.
If you want to test with different locales and see their effect, your can set LANG one command at a time. For example, compare the output of these commands:
LANG=en_US.utf8 ls -1 # what you're looking for
LANG=C ls -1 # "ASCIIbetic" order
LANG=fr_FR.utf8 ls -1 # would consider é as between e and f
I am wondering why I do not get se same output from:
ls -1 -tF | head -n 1
and
echo $(ls -1 -tF | head -n 1)
I tried to get the last modified file, but using it inside a sub shell sometimes I get more than one file as result?
Why that and how to avoid?
The problem arises because you are using an unquoted subshell and -F flag for ls outputs shell special characters appended to filenames.
-F, --classify
append indicator (one of */=>#|) to entries
Executable files are appended with *.
When you run
echo $(ls -1 -tF | head -n 1)
then
$(ls -1 -tF | head -n 1)
will return a filename, and if it happens to be an executable and also be the prefix to another file, then it will return both.
For example if you have
test.sh
test.sh.backup
then it will return
test.sh*
which when echoed expands to
test.sh test.sh.backup
Quoting the subshell prevents this expansion
echo "$(ls -1 -tF | head -n 1)"
returns
test.sh*
I just found the error:
If you use echo $(ls -1 -tF | head -n 1)
the file globing mechanism may result in additional matches.
So echo "$(ls -1 -tF | head -n 1)" would avoid this.
Because if the result is an executable it contains a * at the end.
I tried to place the why -F in a comment, but now I decided to put it here:
I added the following lines to my .bashrc, to have a shortcut to get last modified files or directories listed:
function L {
myvar=$1; h=${myvar:="1"};
echo "last ${h} modified file(s):";
export L=$(ls -1 -tF|fgrep -v / |head -n ${h}| sed 's/\(\*\|=\|#\)$//g' );
ls -l $L;
}
function LD {
myvar=$1;
h=${myvar:="1"};
echo "last ${h} modified directories:";
export LD=$(ls -1 -tF|fgrep / |head -n $h | sed 's/\(\*\|=\|#\)$//g'); ls -ld $LD;
}
alias ol='L; xdg-open $L'
alias cdl='LD; cd $LD'
So now I can use L (or L 5) to list the last (last 5) modified files. But not directories.
And with L; jmacs $L I can open my editor, to edit it. Traditionally I used my alias lt='ls -lrt' but than I have to retype the name...
Now after mkdir ... I use cdl to change to that dir.
I use this:
for f in $( ls -tr ${repSource}*.txt );
...to loop on a list of files, sorted oldest to newest.
I want to add another sort "filter": filenames that don't start with "abc" come first, whatever their timestamp.
So if the files are "abc.txt", "def.txt" and "ghi.txt", then "abc.txt" must be last, and the other two come before in the list (sorted by reverse date).
You don't want to parse the output of the ls command. The construct in your question is a classic bash pitfall and parsing ls is well known to be problematic.
You can instead use stat to get the time of each file that you find in a for loop on a pattern expansion.
#!/usr/bin/env bash
uname_s=$(uname -s)
stamp() {
case "$uname_s" in
Linux) stat -c '%Y' "$1" ;;
Darwin|*BSD) stat -f '%m' "$1" ;;
esac
}
inputfiles=( "$#" )
for file in "${inputfiles[#]}"; do
n=$(stamp "$file")
while [ -n "${forward[$n]}" ]; do
((n++)) # introduce fudge to avoid timestamp collissions
done
forward[$n]="$file"
done
declare -p inputfiles
for n in "${!forward[#]}"; do
reverse[$(( 2**32 - $n ))]="${forward[$n]}"
done
declare -p forward reverse
This example script takes a list of files as command line options (which can be a glob), then uses declare -p to show you the original list, the forward-sorted list, and a reverse-sorted list.
The case statement in the stamp function makes it portable between Linux, OS X (Darwin), FreeBSD, NetBSD, etc. since I don't know what operating system you're using. (If it's something less common like Solaris, HP/UX, etc, then the stat command may not be available or useful and this solution might not work.)
Once you have a sorted (non-associative) array in bash, you can process the files one by one with constructs like:
for file in "${forward[#]}"; do
# something to $file
done
or
for file in "${reverse[#]}"; do
# something to $file
done
And you can trust that since non-associative bash arrays are always numerically ordered, you'll get the files in their date order.
And of course, you've got the dates themselves as indexes, if you want. :)
The files:
ls -log
total 12
-rw-rw-r-- 1 4 Apr 8 11:15 abc.txt
-rw-rw-r-- 1 4 Apr 8 11:15 def.txt
-rw-rw-r-- 1 4 Apr 8 11:16 ghi.txt
This seems to work, but there must be a better way:
ls -log --time-style +%s | tail -n +2 | \
pee \
'grep -v abc.txt | sort -k4 -rn' \
'grep abc.txt | sort -k4 -rn' | \
cut -d " " -f 5
Output:
ghi.txt
def.txt
abc.txt
Note: the unseemly named util pee is from debian's moreutils package, it's a "like tee for pipes".
Now to make a loop of it:
# usage: foo filename # filter above sort code by filename
foo() { ls -log --time-style +%s | tail -n +2 | \
pee 'grep -v '"$1"' | sort -k4 -rn' \
'grep '"$1"' | sort -k4 -rn' | \
cut -d " " -f 5 ; }
for f in $( foo abc.txt )
I have isolated the problem to the below code snippet:
Notice below that null string gets assigned to LATEST_FILE_NAME='' when the script is run using ksh; but the script assigns the value to variable $LATEST_FILE_NAME correctly when run using sh. This in turn affects the value of $FILE_LIST_COUNT.
But as the script is in KornShell (ksh), I am not sure what might be causing the issue.
When I comment out the tee command in the below line, the ksh script works fine and correctly assigns the value to variable $LATEST_FILE_NAME.
(cd $SOURCE_FILE_PATH; ls *.txt 2>/dev/null) | sort -r > ${SOURCE_FILE_PATH}/${FILE_LIST} | tee -a $LOG_FILE_PATH
Kindly consider:
1. Source Code: script.sh
#!/usr/bin/ksh
set -vx # Enable debugging
SCRIPTLOGSDIR=/some/path/Scripts/TEST/shell_issue
SOURCE_FILE_PATH=/some/path/Scripts/TEST/shell_issue
# Log file
Timestamp=`date +%Y%m%d%H%M`
LOG_FILENAME="TEST_LOGS_${Timestamp}.log"
LOG_FILE_PATH="${SCRIPTLOGSDIR}/${LOG_FILENAME}"
## Temporary files
FILE_LIST=FILE_LIST.temp #Will store all extract filenames
FILE_LIST_COUNT=0 # Stores total number of files
getFileListDetails(){
rm -f $SOURCE_FILE_PATH/$FILE_LIST 2>&1 | tee -a $LOG_FILE_PATH
# Get list of all files, Sort in reverse order, and store names of the files line-wise. If no files are found, error is muted.
(cd $SOURCE_FILE_PATH; ls *.txt 2>/dev/null) | sort -r > ${SOURCE_FILE_PATH}/${FILE_LIST} | tee -a $LOG_FILE_PATH
if [[ ! -f $SOURCE_FILE_PATH/$FILE_LIST ]]; then
echo "FATAL ERROR - Could not create a temp file for file list.";exit 1;
fi
LATEST_FILE_NAME="$(cd $SOURCE_FILE_PATH; head -1 $FILE_LIST)";
FILE_LIST_COUNT="$(cat $SOURCE_FILE_PATH/$FILE_LIST | wc -l)";
}
getFileListDetails;
exit 0;
2. Output when using shell sh script.sh:
+ getFileListDetails
+ rm -f /some/path/Scripts/TEST/shell_issue/FILE_LIST.temp
+ tee -a /some/path/Scripts/TEST/shell_issue/TEST_LOGS_201304300506.log
+ cd /some/path/Scripts/TEST/shell_issue
+ sort -r
+ tee -a /some/path/Scripts/TEST/shell_issue/TEST_LOGS_201304300506.log
+ ls 1.txt 2.txt 3.txt
+ [[ ! -f /some/path/Scripts/TEST/shell_issue/FILE_LIST.temp ]]
cd $SOURCE_FILE_PATH; head -1 $FILE_LIST
++ cd /some/path/Scripts/TEST/shell_issue
++ head -1 FILE_LIST.temp
+ LATEST_FILE_NAME=3.txt
cat $SOURCE_FILE_PATH/$FILE_LIST | wc -l
++ cat /some/path/Scripts/TEST/shell_issue/FILE_LIST.temp
++ wc -l
+ FILE_LIST_COUNT=3
exit 0;
+ exit 0
3. Output when using ksh ksh script.sh:
+ getFileListDetails
+ tee -a /some/path/Scripts/TEST/shell_issue/TEST_LOGS_201304300507.log
+ rm -f /some/path/Scripts/TEST/shell_issue/FILE_LIST.temp
+ 2>& 1
+ tee -a /some/path/Scripts/TEST/shell_issue/TEST_LOGS_201304300507.log
+ sort -r
+ 1> /some/path/Scripts/TEST/shell_issue/FILE_LIST.temp
+ cd /some/path/Scripts/TEST/shell_issue
+ ls 1.txt 2.txt 3.txt
+ 2> /dev/null
+ [[ ! -f /some/path/Scripts/TEST/shell_issue/FILE_LIST.temp ]]
+ cd /some/path/Scripts/TEST/shell_issue
+ head -1 FILE_LIST.temp
+ LATEST_FILE_NAME=''
+ wc -l
+ cat /some/path/Scripts/TEST/shell_issue/FILE_LIST.temp
+ FILE_LIST_COUNT=0
exit 0;+ exit 0
OK, here goes...this is a tricky and subtle one. The answer lies in how pipelines are implemented. POSIX states that
If the pipeline is not in the background (see Asynchronous Lists), the shell shall wait for the last command specified in the pipeline to complete, and may also wait for all commands to complete.)
Notice the keyword may. Many shells implement this in a way that all commands need to complete, e.g. see the bash manpage:
The shell waits for all commands in the pipeline to terminate before returning a value.
Notice the wording in the ksh manpage:
Each command, except possibly the last, is run as a separate process; the shell waits for the last command to terminate.
In your example, the last command is the tee command. Since there is no input to tee because you redirect stdout to ${SOURCE_FILE_PATH}/${FILE_LIST} in the command before, it immediately exits. Oversimplified speaking, the tee is faster than the earlier redirection, which means that your file is probably not finished writing to by the time you are reading from it. You can test this (this is not a fix!) by adding a sleep at the end of the whole command:
$ ksh -c 'ls /tmp/* | sort -r > /tmp/foo.txt | tee /tmp/bar.txt; echo "[$(head -n 1 /tmp/foo.txt)]"'
[]
$ ksh -c 'ls /tmp/* | sort -r > /tmp/foo.txt | tee /tmp/bar.txt; sleep 0.1; echo "[$(head -n 1 /tmp/foo.txt)]"'
[/tmp/sess_vo93c7h7jp2a49tvmo7lbn6r63]
$ bash -c 'ls /tmp/* | sort -r > /tmp/foo.txt | tee /tmp/bar.txt; echo "[$(head -n 1 /tmp/foo.txt)]"'
[/tmp/sess_vo93c7h7jp2a49tvmo7lbn6r63]
That being said, here are a few other things to consider:
Always quote your variables, especially when dealing with files, to avoid problems with globbing, word splitting (if your path contains spaces) etc.:
do_something "${this_is_my_file}"
head -1 is deprecated, use head -n 1
If you only have one command on a line, the ending semicolon ; is superfluous...just skip it
LATEST_FILE_NAME="$(cd $SOURCE_FILE_PATH; head -1 $FILE_LIST)"
No need to cd into the directory first, just specify the whole path as argument to head:
LATEST_FILE_NAME="$(head -n 1 "${SOURCE_FILE_PATH}/${FILE_LIST}")"
FILE_LIST_COUNT="$(cat $SOURCE_FILE_PATH/$FILE_LIST | wc -l)"
This is called Useless Use Of Cat because the cat is not needed - wc can deal with files. You probably used it because the output of wc -l myfile includes the filename, but you can use e.g. FILE_LIST_COUNT="$(wc -l < "${SOURCE_FILE_PATH}/${FILE_LIST}")" instead.
Furthermore, you will want to read Why you shouldn't parse the output of ls(1) and How can I get the newest (or oldest) file from a directory?.
The end of git status looks like this:
# Untracked files:
# (use "git add <file>..." to include in what will be committed)
#
# Classes/Default.png
# Classes/Default#2x.png
...
Since you might have any number of untracked files, I'm trying to tail from the end of the file to "Untracked files" and save it to a temp file, strip out the first three lines and convert the filenames to git add Classes/...
I can't seem to find a good way (other than maybe a different language) to tail up to a searchable expression. Thanks!
Use sed to print everything from "Untracked files" to the end:
git status | sed -n '/Untracked files:$/,$p'
Then you just have to parse the filenames by removing the # character.
You can also use git status -s to get a shorter, more easily parsed output:
~$ git status -s
?? Classes/Default.png
?? Classes/Default#2x.png
This is a good application of awk, which lets you grep and extract at the same time:
~$ git status -s | awk '/\?\?/{print $2}'
Classes/Default.png
Classes/Default#2x.png
Alternatively: awk '{if ($1 == "??") print $2}'
You can also, of course, use git add to list (and add) untracked files.
Use the tail command:
tail -$(($(wc -l < file.txt) - $(grep -n "Untracked files" file.txt|cut -d: -f1) - 2)) file.txt
How it works:
total number of lines in file = wc -l < file.txt
line number of "Untracked files" = grep -n "Untracked files" file.txt|cut -d: -f1
The -2 is to remove the top lines
Complete command with git add:
tail -$(($(wc -l < file.txt) - $(grep -n "Untracked files" file.txt|cut -d: -f1) - 2)) file.txt | tr -d '#'| while read line; do echo git add $line; done
Pipe it to:
perl -e 'my ($o, $p) = (0, shift); while(<>){print if $o or $o = /$p/}' "$MY_REGEX"
Where $MY_REGEX is your pattern. In your case, probably '^\s{7}'.
Solution using shell scripting.
First start reading the file in a while loop, keep a count of the number of lines read, break the loop when the required line is found.
Using the count of lines tail the file and then extract file names using awk.
i=0;
l=`wc -l filename | awk '{print $1}'`;
while read line;
do i=`echo $i + 1 | bc`;
if [[ $line == "# Untracked files:" ]];
then break;
fi;
done < "filename";
tail -`echo $l -$i -2 | bc` filename | awk -F "/" '{print $NF}'
Here "filename" is the file you want to process