I have a command which will output the text between the user defined strings but I cannot seem to output the filename the text originates from...
Is there a way to adapt the current command to do this?
find . -type f -print0 | xargs -0 sed -n '/'$1'/,/'$2'/p' $3
Command syntax:
./shell.sh Start End *.xml
The ideal output would be something like:
filename
text between $1 and $2
Next filename
text between $1 and $2
and so on...
I'm guessing you want something like
find . -name "$3" -type f -exec sed -n "/$1/,/$2/s%^%{}%p" {} \;
This can be adapted to use xargs but since you want the file name in the sed command for each file, it makes sense to run it with find -exec anyway.
The original place you had for $3 did not seem correct. I'm guessing what you want, but the above hopefully makes sense.
So sed ends up running something like
sed -n "/foo/,/bar/s%^%/path/to/file:%p" /path/to/file
which will prefix the matching lines by substituting ^ (start of line) with the file name and a colon.
Related
This question already has answers here:
Colorized grep -- viewing the entire file with highlighted matches
(24 answers)
Closed 4 years ago.
How can I modify grep so that it prints full file if its entry matches the grep pattern , instead of printing Just the matching line ?
I tried using(say) grep -C2 to print two lines above and 2 below but this doesn't always works as no. of lines is not fixed ..
I am not Just searching a single file , I am searching an entire directory where some files may contain the given pattern and I want those Files to be completely Printed.
I am also using grep inside grep result without getting printed the first grep output.
Simple grep + cat combination:
grep 'pattern' file && cat file
Use grep's -l option to list the paths of files with matching contents, then print the contents of these files using cat.
grep -lR 'regex' 'directory' | xargs -d '\n' cat
The command from above cannot handle filenames with newlines in them.
To overcome the filename with newlines issue and also allow more sophisticated checks you can use the find command.
The following command prints the content of all regular files in directory.
find 'directory' -type f -exec cat {} +
To print only the content of files whose content matches the regexes regex1 and regex2, use
find 'directory' -type f \
-exec grep -q 'regex1' {} \; -and \
-exec grep -q 'regex2' {} \; \
-exec cat {} +
The linebreaks are only for better readability. Without the \ you can write everything into one line.
Note the -q for grep. That option supresses grep's output. grep's exit status will tell find whether to list a file or not.
I want to batch show the text of certain line of files in certain directory, usually this can be done with the following commands:
for file in `find ./ -name "results.txt"`;
do
sed -n '12p' < ${file};
done
In the 12th line of each file names "results.txt", there is the text I want to output.
But, I wonder that if we can use the pipeline command to do this operation. I have tried the following command:
find ./ -name "results.txt" | xargs sed -n '12p'
or
find ./ -name "results.txt" | xargs sed -n '12p' < {} \;
But neither works fine.
Could you give some advice or recommend some references, please?
All are welcome, Thanks in advice!
This should do it
find ./ -name results.txt -exec sed '12!d' {} ';'
#Steven Penny's answer is the most elegant and best-performing solution, but to shed some light on why your solution didn't work:
find ./ -name "results.txt" | xargs sed -n '12p'
causes all filenames(1) to be passed at once(2) to sed. Since sed counts lines cumulatively, across input files, only 1 line will be printed for all input files, namely line 12 from the first input file.
Keeping in mind that find's -exec action is the best solution, if you still wanted to solve this problem with xargs, you'd have to use xarg's -I option as follows, so as to ensure that sed is called once per input line (filename) (% is a self-chosen placeholder):
find ./ -name "results.txt" | xargs -I % sed -n '12q;d' %
Footnotes:
(1) with word splitting applied, which would break with paths with embedded spaces, but that's a separate issue.
(2) assuming they don't make the entire command exceed the max. length of a command line; either way, multiple filenames are passed at once.
As an aside: parsing command output with for as in your first snippet is NEVER a good idea - see http://mywiki.wooledge.org/ParsingLs and http://mywiki.wooledge.org/BashFAQ/001
Your use of xargs results in running sed with multiple file arguments. But as you can see, sed doesn't reset the record number to 1 when it starts reading a new file. For example, try running the following command against files with more than 12 lines each.
sed -n '12p' x.txt y.txt
If you want to use xargs, you might consider using awk:
find . -name 'results.txt' | xargs awk 'FNR==12'
P.S: I personally like using the for loop.
I have used the following to do a recursive find and replace within files, to update hrefs to point to a new page correctly:
#!/bin/bash
oldstring='features.html'
newstring='features.php'
grep -rl $oldstring public_html/ | xargs sed -i s#"$oldstring"#"$newstring"#g
It worked, except for a few files that had spaces in the name.
This isn't an issue, as the files with spaces in their names are backups/duplicates I created while testing new things. But I'd like to understand how I could properly pass paths with spaces to the sed command, in this query. Would anybody know how this could be corrected in this "one liner"?
find public_html/ -type f -exec grep -q "$oldstring" {} \; -print0 |
xargs -0 sed -i '' s#"$oldstring"#"$newstring"#g
find will print all the filenames for which the grep command is successful. I use the -print0 option to print them with the NUL character as the delimiter. This goes with the -0 option to xargs, which treats NUL as the argument delimiter on its input, rather than breaking the input at whitespace.
Actually, you don't even need grep and xargs, just run sed from find:
find public_html/ -type f -exec sed -i '' s#"$oldstring"#"$newstring"#g {} +
Here's a lazy approach:
grep -rl $oldstring public_html/ | xargs -d'\n' sed -i "s#$oldstring#$newstring#g"
By default, xargs uses whitespace as the delimiter of arguments coming from the input. So for example if you have two files, a b and c, then it will execute the command:
sed -i 's/.../.../' a b c
By telling xargs explicitly to use newline as the delimiter with -d '\n' it will correctly handle a b as a single argument and quote it when running the command:
sed -i 's/.../.../' 'a b' c
I called a lazy approach, because as #Barmar pointed out, this won't work if your files have newline characters in their names. If you need to take care of such cases, then use #Barmar's method with find ... -print0 and xargs -0 ...
PS: I also changed s#"$oldstring"#"$newstring"#g to "s#$oldstring#$newstring#g", which is equivalent, but more readable.
I'm trying to write a shell script which recursively goes through a directory, then in each file converts all Uppercase letters to lowercase ones. To be clear, I'm not trying to change the file names but the text in the files.
Considerations:
This is an old Fortran project which I am trying to make more accessible
I do not want to create a new file but rather write over the old one with the changes
There are several different file extensions in this directory, including .par .f .txt and others
What would be the best way to go about this?
To convert a file from lower case to upper case you can use ex (a good friend of ed, the standard editor):
ex -s file <<EOF
%s/[[:upper:]]\+/\L&/g
wq
EOF
or, if you like stuff on one line:
ex -s file <<< $'%s/[[:upper:]]\+/\L&/g\nwq'
Combining with find, you can then do:
find . -type f -exec bash -c "ex -s -- \"\$0\" <<< $'%s/[[:upper:]]\+/\L&/g\nwq'" {} \;
This method is 100% safe regarding spaces and funny symbols in the file names. No auxiliary files are created, copied or moved; files are only edited.
Edit.
Using glenn jackmann's suggestion, you can also write:
find . -type f -exec bash -c 'printf "%s\n" "%s/[[:upper:]]\+/\L&/g" "wq" | ex -- -s "$0"' {} \;
(the pro is that it avoids awkward escapes; the con is that it's longer).
You can translate all uppercase characters (A–Z) to lowercase (a–z) using the tr command
and specifying a range of characters, as in:
$ tr 'A-Z' 'a-z' <be.fore >af.ter
There is also special syntax in tr for specifying this sort of range for upper- and lowercase
conversions:
$ tr '[:upper:]' '[:lower:]' <be.fore >af.ter
The tr utility copies the given input to produced the output with substitution or deletion of selected characters. tr abbreviated as translate or transliterate. It takes as parameters two sets of characters, and replaces occurrences of the characters in the first set with the corresponding elements from the other set i.e. it is used to translate characters.
tr "set1" "set2" < input.txt > output.txt
Although tr doesn't support regular expressions, hmm, it does support a range of characters.
Just make sure that both arguments end up with the same number of characters.
If the second argument is shorter, its last character will be repeated to match the
length of the first argument. If the first argument is shorter, the second argument will
be truncated to match the length of the first.
sed -e 's/\(.*\)/\L\1/g' *
or you could pipe the files in from find
Expanding on #nullrevolution's solution:
find /path_to_files -type f -exec sed --in-place -e 's/\(.*\)/\L\1/g' '{}' \;
This one liner will look for all files in all sub-directories starting with /path_to_files as a base directory.
WARNING: This will change the case on ALL files in EVERY directory under */path_to_file*, so make sure you want to do that before you execute this script. You can limit the scope of the find based on file extensions by utilizing the following:
find /path_to_files -type f -name \*.txt -exec sed --in-place -e 's/\(.*\)/\L\1/g' '{}' \;
You may also want to make a backup of the original file before modifying the original:
find /path_to_files -type f -name *.txt -exec sed --in-place=-orig -e 's/(.*)/\L\1/g' '{}' \;
This will leave the original file name, while making an unmodified copy with the "_orig" appended to the file name (ie file.txt would become file.txt-orig).
An explanation of each piece:
find /path_to_file This will set the base directory to the path provided.
-type f This will search the directory hierarchy for files only.
-exec COMMAND '{}' \; This executes the provided command once for each matched file. The '{}' is replaced by the current file name. The \; indicates the end of the command.
sed --in-place -e 's/\(.*\)/\L\1/g' The --in-place will make the cnages to the file without backing up the file. The regular expression uses a backreference \1 to refer to the entire line and the \L to convert to lower case.
Optional
(For a more archaic solution.)
find /path_to_files -type f -exec dd if='{}' of='{}'-lc conv=lcase \;
Identifying text files can be a bit tricky in Unixlike environments. You can do something like this:
set -e -o noclobber
while read f; do
tr 'A-Z' 'a-z' <"$f" >"f.$$"
mv "$f.$$" "$f"
done < <(find "$start_directory" -type f -exec file {} + | cut -d: -f1)
This will fail on filenames with embedded colons or newlines, but should work on others, including those with spaces.
I'm grabbing all the filenames from a directory and I want to create a comma delimited string of these filenames so that I can pass that string as an argument to an application. This is my code snippet:
if [[ -n $(ls | grep lpt) ]]; then
for files in $(find . -maxdepth 1 -type f); do
#parse output into variable fileList
done
fi
How do I accomplish this?
Think easier:
find . -maxdepth 1 -type f -printf '%P,' | sed -e 's/,$/\n/'
The sed expression replaces the terminal , by a linebreak.
You should use the one-liners shown in the other answers, but in order to fill in your script, you could do:
if [[ -n $(ls | grep lpt) ]]; then
for file in $(find . -maxdepth 1 -type f); do
#parse output into variable fileList
fileList="$file,$fileList"
done
fi
#now remove the trailing comma from the fileList
fileList=$(sed 's/,$//' <<< "$fileList")
(Note: your for-loop won't work correctly if your filenames have spaces in them)
Instead of your loop, you could use find's -exec option, along with shell expansion:
fileList=$(find . -maxdepth 1 -type f -exec echo -n "{}," \; | sed 's/,$//')
The sed bit is just to remove the trailing comma. sed is used to edit input streams, i.e. here, it gets piped text from find and is editing what it's getting. Since the command given leaves an extra , at the end, sed uses its substitution command (s) to get rid of it. The form is:
s/EXPRESSION/REPLACEMENT/
So ,$ means "a comma at the end of the line, since $ means "at the end of the line", and the nothingness between the second and third slashes means it gets replaced by nothing.
As far as the \; in find, that's just a requirement for using -exec, so it knows when it's done reading commands, and it's in the man page. :)