need advice about the following
with the following ksh script I actually copy file1 to file2
my problem is that lines in file2 are not with the same location as file1
#!/bin/ksh
while read -r line ; do
echo $line >> file2
done < file1
for example
more file1
line1
line2
line3
more file2
line1
line2
line3
the question what I need to change in my script in order to get lines location as described in file1? after I run my ksh script?
lidia
You can try:
while read -r line ; do
echo $line | sed -re 's/^\s+//' >> file2
done < file1
This uses sed to get rid of the leading whitespaces present in lines from file1.
you can set IFS=
while IFS= read -r line ; do echo "$line"; done<file
Related
I have a test.txt file with the following contents
100001
100003
100007
100008
100009
I am trying to loop through the text file and append each one with .xml.
Ex:
100001.xml
100003.xml
100007.xml
100008.xml
100009.xml
I have tried different variations of
while read p; do
echo "$p.zip"
done < test.txt
But it prints out weird like this
.xml01
.xml03
.xml07
.xml08
.xml09
Appending a .xml at the end of each line while removing CRLF, if present.
With sed and bash:
#!/bin/bash
sed -E $'s/\r?$/.xml/' test.txt
With awk:
awk -v suffix='.xml' '{sub(/\r?$/,suffix)}1' test.txt
Using it in a bash loop:
#!/bin/bash
while IFS='' read -r filename
do
printf '%q\n' "$filename"
done < <(
awk -v suffix='.xml' '{sub(/\r?$/,suffix)}1' test.txt
)
Or doing the whole thing in pure shell:
while IFS='' read -r filename
do
fullname="${filename%\r}.xml"
printf '%s\n' "$fullname"
done < test.txt
I have a line like this:
INPUT file1
How can I get bash to read that line and directly copy in the contents of "file1.txt" in place of that line? Or if it sees: INPUT file2 on a line, put in `file2.txt" etc.
The best I can do is a lot of tr commands, to paste the file together, but that seems an overly complicated solution.
'sed' also replaces lines with strings, but I don't know how to input the entire content of a file, which can be hundreds of lines into the replacement.
Seems pretty straightforward with awk. You may want to handle errors differently/more gracefully, but:
$ cat file1
Line 1 of file 1
$ cat file2
Line 1 of file 2
$ cat input
This is some content
INPUT file1
This is more content
INPUT file2
This file does not exist
INPUT file3
$ awk '$1=="INPUT" {system("cat " $2); next}1' input
This is some content
Line 1 of file 1
This is more content
Line 1 of file 2
This file does not exist
cat: file3: No such file or directory
A perl one-liner, using the CPAN module Path::Tiny
perl -MPath::Tiny -pe 's/INPUT (\w+)/path("$1.txt")->slurp/e' input_file
use perl -i -M... to edit the file in-place.
Not the most efficient possible way, but as an exercise I made a file to edit named x and a couple of input sources named t1 & t2.
$: cat x
a
INPUT t2
b
INPUT t1
c
$: while read k f;do sed -ni "/$k $f/!p; /$k $f/r $f" x;done< <( grep INPUT x )
$: cat x
a
here's
==> t2
b
this
is
file ==> t1
c
Yes, the blank lines were in the INPUT files.
This will sed your base file repeatedly, though.
The awk solution given is better, as it only reads through it once.
If you want to do this in pure Bash, here's an example:
#!/usr/bin/env bash
if (( $# < 1 )); then
echo "Usage: ${0##*/} FILE..."
exit 2
fi
for file; do
readarray -t lines < "${file}"
for line in "${lines[#]}"; do
if [[ "${line}" == "INPUT "* ]]; then
cat "${line#"INPUT "}"
continue
fi
echo "${line}"
done > "${file}"
done
Save to file and run like this: ./script.sh input.txt (where input.txt is a file containing text mixed with INPUT <file> statements).
Sed solution similar to awk given erlier:
$ cat f
test1
INPUT f1
test2
INPUT f2
test3
$ cat f1
new string 1
$ cat f2
new string 2
$ sed 's/INPUT \(.*\)/cat \1/e' f
test1
new string 1
test2
new string 2
test3
Bash variant
while read -r line; do
[[ $line =~ INPUT.* ]] && { tmp=($BASH_REMATCH); cat ${tmp[1]}; } || echo $line
done < f
I want to use nested process subtitution with tee in a while loop.
while read line; do
#process line
echo "--$line"
done < <(cat sample.file | tee >(grep "SPECLINE") | grep "LINESTOPROCESS")
Therefore, I need:
all lines in sample.file that contain "LINETOPROCESS" expression should be passed into the loop, and they will be printed with "--" prefix.
all lines contain "SPECLINE" needs to be printed in tee's first process substitution (in the grep).
I want to avoid cat-ting the sample.file more than once as it is too large and heavy.
With a simple sample.test file:
line1 SPECLINE
line2 LINETOPROCESS
line3 LINETOPROCESS
line4 SPECLINE
line5 I don't need it
line6 also not
line7 also not
line8 SPECLINE
line9 LINETOPROCESS
My result:
# ./test.sh
#
My desired result:
# ./test.sh
line1 SPECLINE
--line2 LINETOPROCESS
--line3 LINETOPROCESS
line4 SPECLINE
line8 SPECLINE
--line9 LINETOPROCESS
Or I can also accept this as output:
# ./test.sh
--line2 LINETOPROCESS
--line3 LINETOPROCESS
--line9 LINETOPROCESS
line1 SPECLINE
line4 SPECLINE
line8 SPECLINE
UPDATE1
greps are for demo only.
I really need those 2 substitutions.
sample.file is a http file.
grep "SPECLINE" would be "hxselect -i -s ';' -c 'div.hour'
grep "LINESTOPROCESS" would be "hxselect -i -s ';' -c 'div.otherclass' | hxpipe
hx programs are not line-oriented. They are reading from stdin and outputting to stdout.
Therefore the tee's first command will select divs with 'hour' class and separate them with ';'. Afterwards, the pipe after tee will select all divs with class 'otherclass' and hxpipe will flatten it for the loop for further processing.
I would use no process substitution at all.
while IFS= read -r line; do
if [[ $line = *SPECLINE* ]]; then
printf '%s\n' "$line"
elif [[ $line = *LINETOPROCESS* ]]; then
printf '--%s\n' "$line"
fi
done < sample.txt
You are already paying the cost of reading an input stream line-by-line in bash; no reason to add the overhead of two separate grep processes to it.
A single awk process would be even better, as it is more efficient than bash's read-one-character-at-a-time approach to reading lines of text.
awk '/SPECLINE/ {print} /LINETOPROCESS/ {print "--"$0}' sample.txt
(which is too simple if a single line could match both SPECLINE and LINETOPROCESS, but I leave that as an exercise to the reader to fix.)
The following just loops through the entire file and just prints the matching lines. All other lines are ignored.
while read line; do
case "$line" in
*SPECLINE*) echo "$line" ;;
*LINETOPROCESS*) echo "--$line" ;;
esac
done < sample.file
When you want the tee, you can make 2 changes.
Your testcode greps LINESTOPROCESS, the input is LINETO..
The output process substition gives problems like https://stackoverflow.com/a/42766913/3220113 explained. You can do this differently.
while IFS= read -r line; do
#process line
echo "--$line"
done < x2 |
tee >(grep "SPECLINE") >(grep "LINETOPROCESS") >/dev/null
I don't know hxselect, but it seems to operate on a complete well-formed XML document, so avoid the grep.
I need to do while loop when sed is not null. Example:
File 1
line1
line2
line3
File 2
i=1
while sed "${int}p" # here I need expression which checks if line is not null
# here echo this line and i++
I tried to write just while sed -n "${int}p" but it does not work as I expected.
You can use the = command in sed to get the number of lines:
sed -n '/./!q;=;p' input | sed 'N;s/\n/ /'
For an input:
a
b
c
d
This gives:
1 a
2 b
3 c
If you only want to get line number of the line before the first non-empty line:
sed -n '/./!q;=' input | tail -1
A while loop that prints all lines:
while read line; do
echo "$line"
done < input
If you want to count the lines until the first empty line, you could do this.
$ cat in.txt
line1
line2
line3
line4
line5
$ echo $(($(sed '/^\s*$/q' < in.txt | wc -l) - 1))
3
I have a file containing some text and the words INSERT_HERE1 and INSERT_HERE2. I'd like to replace these words with the content of file1.txt and file2.txt respectively.
I suspect sed or awk could pull it off but I've basically never used them.
Sed does have a built-in read file command. The commands you want would look something like this:
$ sed -e '/INSERT_HERE1/ {
r FILE1
d }' -e '/INSERT_HERE2/ {
r FILE2
d }' < file
This would output
foo
this is file1
bar
this is file2
baz
The r command reads the file, and the d command deletes the line with the INSERT_HERE tags. You need to use the curly braces since sed commands and multi-line input since sed commands have to start on their own line, and depending on your shell, you may need \ at the end of the lines to avoid premature execution. If this is something you would use a lot, you can just put the command in a file and use sed -f to run it.
If you are okay with Perl you can do:
$ cat FILE1
this is file1
$ cat FILE2
this is file2
$ cat file
foo
INSERT_HERE1
bar
INSERT_HERE2
baz
$ perl -ne 's/^INSERT_HERE(\d+)\s+$/`cat FILE$1`/e;print' file
foo
this is file1
bar
this is file2
baz
$
This is not tested, but would be pretty close to what you need:
sed -e "s/INSERT_HERE1/`cat file1.txt`/" -e "s/INSERT_HERE2/`cat file2.txt`/" <file >file.out
It will not properly handle a file with slashes in it, though, so you may need to tweak it a bit.
I'd recommend Perl instead, though. Something like this:
#!/usr/bin/perl -w
my $f1 = `cat file1.txt`;
my $f2 = `cat file2.txt`;
while (<>) {
chomp;
s/INSERT_HERE1/$f1/;
s/INSERT_HERE2/$f2/;
print "$_\n";
}
This assumes that INSERT_HERE1 and INSERT_HERE2 may only appear once per line, and that the file1.txt does not include the text INSERT_HERE2 (wouldn't be difficult to fix, though). Use like this:
./script <file >file.out
This is suitable for small substitution files that may be substituted many times:
awk 'BEGIN {
while ((getline line < ARGV[1]) > 0) {file1 = file1 nl line; nl = "\n"};
close (ARGV[1]); nl = "";
while ((getline line < ARGV[2]) > 0) {file2 = file2 nl line; nl = "\n"};
close (ARGV[2]);
ARGV[1] = ""; ARGV[2] = "" }
{ gsub("token1", file1);
gsub("token2", file2);
print }' file1.txt file2.txt mainfile.txt
You may want to add some extra newlines here and there, depending on how you want your output to look.
Easily done with Bash. If you need it to be POSIX shell let me know:
#!/bin/bash
IFS= # Needed to prevent the shell from interpreting the newlines
f1=$(< /path/to/file1.txt)
f2=$(< /path/to/file2.txt)
while read line; do
if [[ "$line" == "INSERT_HERE1" ]]; then
echo "$f1"
elif [[ "$line" == "INSERT_HERE2" ]]; then
echo "$f2"
else
echo "$line"
fi
done < /path/to/input/file
This snippet replaces any section that is specified in the upper array. For e.g. here
<!--insert.txt-->
with the contents of "insert.txt"
#!/bin/bash
replace[1]=\<!--insert.txt--\> ; file[1]=insert.txt
replace[2]=\<!--insert2.txt--\> ; file[2]=insert2.txt
replacelength=${#replace[#]}
cat blank.txt > tmp.txt
for i in $(seq 1 ${replacelength})
do
echo Replacing ${file[i]} ...
sed -e "/${replace[i]}/r ${file[i]}" -e "/${replace[i]}/d" tmp.txt > tmp_2.txt
mv tmp_2.txt tmp.txt
done
mv tmp.txt file.txt
If you're not afraid of .zip files you can try this example as long as it is online: http://ablage.stabentheiner.de/2013-04-16_contentreplace.zip
I would use perl's in place replacement with -i.ext option
perl -pi.bak -e 's|INSERT_HERE1|`cat FILE1`|ge;
s|INSERT_HERE2|`cat FILE2`|ge;' myfile
Then, use diff myfile.bak myfile to verify: