I'm trying to search a large file in reverse order from the command line (using terminal). I found the tac command: http://clifgriffin.com/2008/11/25/tac-and-reverse-grep/
tac is the inverse of cat. However, when I try to use the tac command in terminal, it says that command doesn't exist. Is there a way I'd be able to use tac in terminal? What are some other fast ways to search a file from the end via the command line?
The MacOs version of tail support the -r ("reverse") option, and defaults
to displaying the entire file from the end. So tail -r filename should be
exactly equivalent to tac filename.
Or, you could try building tac yourself from the source code. It's part of the GNU coreutils package.
It's easy using OSX Homebrew:
brew install coreutils
Then it's available as gtac (as in GNU tac), for example:
# Find and parallel copy all files in reverse order
find . -print | gtac | parallel copy "{}" destination/
Related
I'm using the following command on Ubuntu to list all files containing a given pattern:
for f in *; do if grep -zoPq "foo\nbar" $f; then echo $f; fi; done
But on macos, I'm geting the following error:
grep: invalid option -- z
There's no -z option to treat files as a big string with macos grep, unlike gnu grep.
Is there another option on macos grep equivalent to `-z ? If not, what alternative can I use to get the same result ?
-P (PERL regex) is only supported in gnu grep but not on BSD grep found on Mac OS.
You can either use home brew to install gnu grep or else use this equivalent awk command:
awk 'p ~ /foo$/ && /^bar/ {print FILENAME; nextfile}; {p=$0}' *
Please note that this eliminates the need to use shell for loop.
You can install pcregrep via home brew, and then use it with the -M option:
By default, each line that matches a pattern is copied to the
standard output, and if there is more than one file, the file name is
output at the start of each line, followed by a colon. However, there
are options that can change how pcregrep behaves. In particular, the
-M option makes it possible to search for patterns that span line
boundaries. What defines a line boundary is controlled by the -N
(--newline) option.
With ripgrep
rg -lU 'foo\nbar'
This will list all filenames containing foo\nbar in the current directory. -U option allows to match multiple lines. Unlike grep -z, whole file isn't read in one-shot, so this is safe to use even for larger input files.
ripgrep recursively searches by default. Use rg -lU --max-depth 1 'foo\nbar' if you don't want to search sub-directories.
However, note that by default, rigprep ignores
files and directories that match rules specified by ignore files like .gitignore
hidden files and directories
binary files
You can change that by using:
-u or --no-ignore
-uu or --no-ignore --hidden
-uuu or --no-ignore --hidden --binary
It seems you are searching for files which have the sequence foo\nbar. With GNU awk (brew install gawk), you can set the record separatorRS to this sequence and check if the record matches:
gawk 'BEGIN{RS="foo\nbar"}{exit (RT!=RS)}' file
This will try to split your files in records which are separated by the record separator RS, if so, it will terminate with exit code 0, otherwise with exit code 1. The behaviour is the same as the proposed grep
If you just want the files listed, you can do:
gawk 'BEGIN{RS="foo\nbar"}(RT==RS){print FILENAME}{nextfile}' *
I am trying to trouble shoot a problem I am seeing when running bash commands in Cygwin.
I am trying to assign the CLang version from a text file to a variable. If I run this in Cygwin:
$ (sed -n 1p "$CLANGC2_VERSION_FILE" | sed 's/\s//g')
I get this output (which is exactly what I want):
14.10.25903
Now, if I try and assign this to a variable it doesn't work. Here is what I am trying:
$ CLANGC2_VERSION=$(sed -n 1p "$CLANGC2_VERSION_FILE" | sed 's/\s//g')
but when I inspect or print the variable, it is empty.
What am I doing wrong?
Turns out that there is a known 'Big List of Dodgy Apps' (BLODA) which can interfere with Cygwin and bash.
The discussion I found is here: https://cygwin.com/ml/cygwin/2017-07/msg00197.html
The BLODA list is here: https://cygwin.com/faq/faq.html#faq.using.bloda
Turns out my AntiVirus is on the list.
I've removed the AV and now the commands work. There must be some low-level stuff going with the AV that causes it to fail.
You can use backticks to get the desired results.
CLANGC2_VERSION=`(sed -n 1p "$CLANGC2_VERSION_FILE" | sed 's/\s//g')`
I created 3 files in a directory with the following names:
11 13 9
Problem, as I created file 9 after 13 it is placed after file 13 when I do the ls command.
Do you have any idea to make the ls command sort the files in the numerical order, like this:
9 11 13
Look at the man page for ls.
From there-
-v natural sort of (version) numbers within text.
You'll find that the command ls -v does what you want.
If you want the file names on different lines then you can do ls -1v.
List with lines without columns:
ls -x
Using tr to avoid multiple lines output:
ls | tr "\n" " "
And to ensure to accurately force ascending order:
ls -v | tr "\n" " "
Some useful and essential information from linux.org about ls.
Unfortunately, ls -v on macOS and FreeBSD does not perform a natural sort on the list of files. FreeBSD ls does not support this flag at all. On macOS it can be used to
force unedited printing of non-graphic characters; this is the default when output is not to a terminal.
If you badly need the GNU extension of -v to the ls utility, you can install the GNU ls and then use alias ls=gls to use the GNU ls instead of the default ls. GNU ls usually comes with the coreutils package, so, e.g.,
On macOS: brew install coreutils
On FreeBSD: pkg install coreutils
I have a directory full of files with one extension (.txt in this case) that I want to automatically convert to another extension (.md).
Is there an easy terminal one-liner I can use to convert all of the files in this directory to a different file extension?
Or do I need to write a script with a regular expression?
You could use something like this:
for old in *.txt; do mv $old `basename $old .txt`.md; done
Make a copy first!
Alternatively, you could install the ren (rename) utility
brew install ren
ren '*.txt' '#1.md'
If you want to rename files with prefix or suffix in file names
ren 'prefix_*.txt' 'prefix_#1.md'
Terminal is not necessary for this... Just highlight all of the files you want to rename. Right click and select "Rename ## items" and just type ".txt" into to the "Find:" box and ".md" into the "Replace with:" box.
The preferred Unix way to do this (yes, OS X is based on Unix) is:
ls | sed 's/^\(.*\)\.txt$/mv "\1.txt" "\1.md"/' | sh
Why looping with for if ls by design loops through the whole list of filenames? You've got pipes, use them. You can create/modify not only output using commands, but also commands (right, that is commands created by a command, which is what Brian Kernighan, one of the inventors of Unix, liked most on Unix), so let's take a look what the ls and the sed produces by removing the pipe to sh:
$ ls | sed 's/^\(.*\)\.txt$/mv "\1.txt" "\1.md"/'
mv "firstfile.txt" "firstfile.md"
mv "second file.txt" "second file.md"
$
As you can see, it is not only an one-liner, but a complete script, which furthermore works by creating another script as output. So let's just feed the script produced by the one-liner script to sh, which is the script interpreter of OS X. Of course it works even for filenames with spaces in it.
BTW: Every time you type something in Terminal you create a script, even if it is only a single command with one word like ls or date etc. Everything running in a Unix shell is always a script/program, which is just some ASCII-based stream (in this case an instruction stream opposed to a data stream).
To see the actual commands being executed by sh, just add an -x option after sh, which turns on debugging output in the shell, so you will see every mv command being executed with the actual arguments passed by the sed editor script (yeah, another script inside the script :-) ).
However, if you like complexity, you can even use awk and if you like to install other programs to just do basic work, there is ren. I know even people who would prefer to write a 50-lines or so perl script for this simple every-day task.
Maybe it's easier in finder to rename files, but if connected remotely to a Mac (e.g. via ssh), using finder is not possible at all. That's why cmd line still is very useful.
Based on the selected and most accurate answer above, here's a bash function for reusability:
function change_all_extensions() {
for old in *."$1"; do mv $old `basename $old ."$1"`."$2"; done
}
Usage:
$ change_all_extensions txt md
(I couldn't figure out how to get clean code formatting in a comment on that answer.)
No need to write a script for it just hit this command
find ./ -name "*.txt" | xargs -I '{}' basename '{}' | sed 's/\.txt//' | xargs -I '{}' mv '{}.txt' '{}.md'
You do not need a terminal for this one; here is a sample demonstration in MacOS Big Sur.
Select all the files, right-click and select "rename..."
Add the existing file extension in "Find" and the extension you want to replace with "Replace with".
And done!
I had a similar problem where files were named .gifx.gif at the end and this worked in OS X to remove the last .gif:
for old in *.gifx.gif; do
mv $(echo "$old") $(echo "$old" | sed 's/x.gif//');
done
cd $YOUR_DIR
ls *.txt > abc
mkdir target // say i want to move it to another directory target in this case
while read line
do
file=$(echo $line |awk -F. '{ print $1 }')
cp $line target/$file.md // depends if u want to move(mv) or copy(cp)
done < abc
list=ls
for file in $list
do
newf=echo $file|cut -f1 -d'.'
echo "The newf is $newf"
mv $file $newf.jpg
done
I often find find myself doing a workflow like this:
$ find . |grep somefile
./tmp/somefile.xml
./test/another-somefile.txt
(review output)
$ vim ./tmp/somefile.xml
Now, it would be neat if there was some convenient way of using the output of the find command and feed it to vim.
The best I've come up with is:
$ nth () { sed -n $1p; }
$ find . |grep somefile
./tmp/somefile.xml
./test/another-somefile.txt
(review output)
$ vim `!!|nth 2`
I was wondering if there are other, maybe prettier, ways of accomplishing the same thing?
To clarify, I want a convenient way of grabbing the nth line from a previously run command to quickly open that file for editing in vim, without having to cut & paste the filename with the mouse or tab-complete my way through the file path.
way 1: don't pass exact file to vim, but the whole output. choose the file in vim
currently you are working in two steps:
1 - launch the find/grep... cmd
2 - vim !!....
if you are sure that you want to use vim to open one (or more) file(s) from the find result. you may try:
find. (with grep if you like) |vim -
then you have the whole output in vim, now you can use vim magic to move cursor to the file you want to edit, then press gf. (I do this sometimes)
way 2: refine your regex in your find (or grep), to get the single file, that you want to edit.
this is not a hard thing at all. then you can just vim !!.
your nth() is nice. however imagine there are 30 lines in output, and your file sits in the line# 16. how do you count it? sure you can add |nl at the end, then you cannot directly use !! any longer..
just my 2 cents
Modified after your comment. Not sure if it's "convenient" though..
command | tail -n3 | head -n1 | xargs vim
Maybe this is what you're looking for?
find . -name "*somefile*" -exec vim -p {} \;
If you want an interactive review maybe you can use something like this:
TMP_LIST=""; for i in `find . | grep somefile`; do echo $i; read -p "(y/n)?"; [ $REPLY == "y" ] && TMP_LIST="$TMP_LIST $i"; done; vim $TMP_LIST
You almost did it!!
pearl.251> cat file1
a b c d e f pearl.252> find . -name "file*"
./file1
./file2
./file3
./file4
./file5
./file6
./file7
pearl.253> vi `!!|awk 'NR==1'`
the last line overe here will open the file1 in vi.