I'm trying to pipe data into "cut" to, say, cut away the first column of text. This works
$ cat test.txt | cut -d\ -f2-
Reading from stdin also works:
$ cut -d\ -f2- -
? doc/html/analysis.html
? doc/html/classxytree-members.html
<CTRL+D>
However, as soon as a pipe is involved, it doesn't accept my <CTRL+D> anymore, and I can't signal "end of file":
$ cut -d\ -f2- - | xargs echo
Update: This is apparently a bug in an old version of bash (3.00.15). It does work in more recent versions (tried 4.0.33 and 3.2.25). It would be nice to have some workaround, though, since I can't easily upgrade.
Background: I've got a script/oneliner that gives me a condensed output of cvs status (I know, CVS...) in the form
? filename
e.g. for a file not committed yet. I'd like to be able to copy+paste parts of the output from that command and use this as an input to another command, that adds these files to cvs. Say:
$ cut -d\ -f2- | xargs cvs add
<paste lines>
<CTRL-D> # <-- doesn't work
Ideas?
have you tried
$ cat | cut -d\ -f2- | xargs cvs add
<paste lines>
<CTRL-D> # <-- doesn't work
Your examples work fine for me. What shell are you using? What utilities?
One thing that sometimes trips people up is that Ctrl-D only works if it's the first character in the line. If you copy and paste, you might sometimes accidentally have whitespace as the first character of the line, or no newline at the end of the pasted block, in which case Ctrl-D won't work. Just hit return and then try Ctrl-D again and see if that fixes your problem.
Related
I am currently learning a little more about using Bash shell on OSX terminal. I am trying to pipe the output of a cut command into a grep command, but the grep command is not giving any output even though I know there are matches. I am using the following command:
cut -d'|' -f2 <filename.txt> > <temp.txt> | grep -Ff <temp.txt> <searchfile.txt> > <filematches.txt>
I was thinking that this should work, but most of the examples I have seen normally pipe grep output into the cut. My goal was to cut field 2 from the file and use that as the pattern to search for in . However, using the command produced no output.
When I generated the temp.txt first with the cut command and then ran the grep on it manually with no pipe, the grep seemed to run fine. I am not sure why this is?
You can use process substitution here:
grep -Ff <(cut -d'|' -f2 filename.txt) searchfile.txt > filematches.txt
<(cut -d'|' -f2 filename.txt) is feeding cut command's output to grep as a file.
Okay, a reason this line doesn't behave as you expect
cut -d'|' -f2 <filename.txt> > <temp.txt> | grep -Ff <temp.txt> <searchfile.txt> > <filematches.txt>
is that the output of your cut is going to temp.txt. You're not sending anything to the pipe. Now, conveniently pipe also starts a new commend, so it doesn't matter much -- grep runs and reads searchfile.txt.
But what are you trying to do? Here's what your command line is trying to do:
take the second pipe-delimited field from filename.txt
write it to a file
run grep ...
... using the contents of the file from 2 as a grep search string (which isn't going to do what you think either, as you're effectively asking grep to look for the pattern match1\nmatch2...)
You'd be closer with
cut ... && grep ...
as that runs grep assuming cut completes effectively. Or you could use
grep -f `cut ...`
which would put the results on the command line. You need to mess with quoting, but you're still going to be looking for a line containing ALL of your match fields from cut.
I'd recommend maybe you mean something like this:
for match in `cut ...`
do
grep -f $match >> filematches.txt
done
I have a script that has uses cut to pick some information out of a search that contains absolute paths. It looks something like:
PLACE=$(grep foo flatfile.txt | cut -d '/' -f 1-6)
The output looks like this:
machine1:/path/to/where/foo/is
machine2:/another/path/to/find/foo
I need it to look like this:
machine1:/path/to/where/foo/is/
machine2:/another/path/to/find/foo/
This needs to be printed to the console at the end of the script with echo "$PLACE" or something like that. The output will always be at least 2 lines, but usually more.
I tried about everything I can think of with echo, but it either shows no output at all or gives the output:
grep: '/' is a directory
I am running bash 3.00 on Solaris, if that helps any. I would really like to K.I.S.S. this by just having something tacked onto the end of the cut command, and not having to monkey with sed or awk. But, if that is the only way, so be it.
try this :
PLACE=$(grep foo flatfile.txt | cut -d '/' -f 1-6 | xargs -I "%" echo %/)
Try the following. Note that $PLACE is not quoted. This should allow word-splitting to happen and each word (which corresponds to one line of your output, assuming no embedded spaces) is then printed by printf with a following / and newline. Haven't been able to test this on Bash 3 on Solaris though
printf "%s/\n" $PLACE
You can use one sed command, instead of multiple other commands:
$ PLACE=$(sed -n '/foo/s:\(\([^/]\+/\)\{6\}\)\(.*\)\+:\1:p' flatfile.txt)
$ echo "$PLACE"
machine1:/path/to/where/foo/is/
machine2:/another/path/to/find/foo/
$
... and then I'm reminded of why sed sometimes makes me shudder :-/
In the OSX terminal :
du -h | sort -nr
-bash: sort: command not found
which sort
/usr/bin/sort
The weird thing is: I tried reproducing the error and it seems to be totally random. My PATH echoed:
/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin:/usr/local/git/bin:/usr/texbin:/Users/sytycs/.rvm/bin
This only occurs when piping and happens with grep, more, less etc.
Any idea what is causing this?
This likely happens because you use a keyboard layout with a non-US layout (happened to me too). On German layouts, the pipe is typed with Alt+7, so chances are high that you press Alt+Space afterwards, which will produce a non-breaking space.
A quick solution: Add the line
"\xC2\xA0": " "
to your ~/.inputrc (if you are using bash). This will map non-breaking spaces to normal ones which should solve the problem.
If you want more detail (or if you are interested in how you can track down these kinds of issues), I wrote a blog post about it some time ago.
That space is not a space. Erase and replace it.
The trick with ~/.inputrc doesn't work for zsh. But here you can configure iTerm to send a space when you type alt+space, for instance
Bash is so sensitive to space when you are piping. Remove them all.
I was facing the same problem by running this command:
|awk '{$1=$1};1' | tr '[:upper:]' '[:lower:]' | sort | uniq
| awk '{ print length, $0 }' | sort -n -s | cut -d" " -f2- > dest_file
the errors were as here:
zsh: command not found: tr
zsh: command not found: sort
then I removed all the spaced and it got resolved:
|awk '{$1=$1};1'|tr '[:upper:]' '[:lower:]'|sort|uniq|awk '{ print length, $0
}'|sort -n -s|cut -d" " -f2- >
COMPANY_NAME=`cat file.txt | grep "company_name" | cut -d '=' -f 2`
outputs something like this
"Abc Inc";
What I want to do is I want to remove the trailing ";" as well. How can i do that? I am a beginner to bash. Any thoughts or suggestions would be helpful.
This will remove the last character contained in your COMPANY_NAME var regardless if it is or not a semicolon:
echo "$COMPANY_NAME" | rev | cut -c 2- | rev
I'd use sed 's/;$//'. eg:
COMPANY_NAME=`cat file.txt | grep "company_name" | cut -d '=' -f 2 | sed 's/;$//'`
foo="hello world"
echo ${foo%?}
hello worl
I'd use head --bytes -1, or head -c-1 for short.
COMPANY_NAME=`cat file.txt | grep "company_name" | cut -d '=' -f 2 | head --bytes -1`
head outputs only the beginning of a stream or file. Typically it counts lines, but it can be made to count characters/bytes instead. head --bytes 10 will output the first ten characters, but head --bytes -10 will output everything except the last ten.
NB: you may have issues if the final character is multi-byte, but a semi-colon isn't
I'd recommend this solution over sed or cut because
It's exactly what head was designed to do, thus less command-line options and an easier-to-read command
It saves you having to think about regular expressions, which are cool/powerful but often overkill
It saves your machine having to think about regular expressions, so will be imperceptibly faster
I believe the cleanest way to strip a single character from a string with bash is:
echo ${COMPANY_NAME:: -1}
but I haven't been able to embed the grep piece within the curly braces, so your particular task becomes a two-liner:
COMPANY_NAME=$(grep "company_name" file.txt); COMPANY_NAME=${COMPANY_NAME:: -1}
This will strip any character, semicolon or not, but can get rid of the semicolon specifically, too.
To remove ALL semicolons, wherever they may fall:
echo ${COMPANY_NAME/;/}
To remove only a semicolon at the end:
echo ${COMPANY_NAME%;}
Or, to remove multiple semicolons from the end:
echo ${COMPANY_NAME%%;}
For great detail and more on this approach, The Linux Documentation Project covers a lot of ground at http://tldp.org/LDP/abs/html/string-manipulation.html
Using sed, if you don't know what the last character actually is:
$ grep company_name file.txt | cut -d '=' -f2 | sed 's/.$//'
"Abc Inc"
Don't abuse cats. Did you know that grep can read files, too?
The canonical approach would be this:
grep "company_name" file.txt | cut -d '=' -f 2 | sed -e 's/;$//'
the smarter approach would use a single perl or awk statement, which can do filter and different transformations at once. For example something like this:
COMPANY_NAME=$( perl -ne '/company_name=(.*);/ && print $1' file.txt )
don't have to chain so many tools. Just one awk command does the job
COMPANY_NAME=$(awk -F"=" '/company_name/{gsub(/;$/,"",$2) ;print $2}' file.txt)
In Bash using only one external utility:
IFS='= ' read -r discard COMPANY_NAME <<< $(grep "company_name" file.txt)
COMPANY_NAME=${COMPANY_NAME/%?}
Assuming the quotation marks are actually part of the output, couldn't you just use the -o switch to return everything between the quote marks?
COMPANY_NAME="\"ABC Inc\";" | echo $COMPANY_NAME | grep -o "\"*.*\""
you can strip the beginnings and ends of a string by N characters using this bash construct, as someone said already
$ fred=abcdefg.rpm
$ echo ${fred:1:-4}
bcdefg
HOWEVER, this is not supported in older versions of bash.. as I discovered just now writing a script for a Red hat EL6 install process. This is the sole reason for posting here.
A hacky way to achieve this is to use sed with extended regex like this:
$ fred=abcdefg.rpm
$ echo $fred | sed -re 's/^.(.*)....$/\1/g'
bcdefg
Some refinements to answer above. To remove more than one char you add multiple question marks. For example, to remove last two chars from variable $SRC_IP_MSG, you can use:
SRC_IP_MSG=${SRC_IP_MSG%??}
cat file.txt | grep "company_name" | cut -d '=' -f 2 | cut -d ';' -f 1
I am not finding that sed 's/;$//' works. It doesn't trim anything, though I'm wondering whether it's because the character I'm trying to trim off happens to be a "$". What does work for me is sed 's/.\{1\}$//'.
I'm writing a small shell script that needs to reverse the lines of a text file. Is there a standard filter command to do this sort of thing?
My specific application is that I'm getting a list of Git commit identifiers, and I want to process them in reverse order:
git log --pretty=oneline work...master | grep -v DEBUG: | cut -d' ' -f1 | reverse
The best I've come up with is to implement reverse like this:
... | cat -b | sort -rn | cut -f2-
This uses cat to number every line, then sort to sort them in descending numeric order (which ends up reversing the whole file), then cut to remove the unneeded line number.
The above works for my application, but may fail in the general case because cat -b only numbers nonblank lines.
Is there a better, more general way to do this?
In GNU coreutils, there's tac(1)
There is a command for your purpose:
tail -r file.txt
Prints the lines of file.txt in reverse order!
The -r flag is non-standard, may not work on all systems, works e.g. on macOS.
Beware: Amount of lines limited. Works mostly, but when working with huge files be careful and check.
Answer is not 42 but tac.
Edit: Slower but more memory consuming using sed
sed 'x;1!H;$!d;x'
and even longer
perl -e'print reverse<>'
Similar to the sed example above, using perl - maybe more memorable (depending on how your brain is wired):
perl -e 'print reverse <>'
cat -b only numbers nonblank lines"
If that's the only issue you want to avoid, then why not use "cat -n" to number all the lines?
: "#(#)$Id: reverse.sh,v 1.2 1997/06/02 21:45:00 johnl Exp $"
#
# Reverse the order of the lines in each file
awk ' { printf("%d:%s\n", NR, $0);}' $* |
sort -t: +0nr -1 |
sed 's/^[0-9][0-9]*://'
Works like a charm for me...
In this case, just use --reverse:
$ git log --reverse --pretty=oneline work...master | grep -v DEBUG: | cut -d' ' -f1
rev <name of your text file.txt>
You can even do this:
echo <whatever you want to type>|rev
awk '{a[i++]=$0}END{for(;i-->0;)print a[i]}'
More faster than sed and compatible for embed devices like openwrt.