I have a file with the name file.txt and contents
$cat file.txt
this one
juice apple
orange pen
apple
rice
mouse
and I have another file called word.txt
$cat word.txt
ganti
buah
bukan
i want to replace line 1,3 and 5 in file.txt with word.txt using bash. so the final result in file.txt becomes:
$cat file.txt
ganti
juice apple
buah
apple
bukan
mouse
how to use bash to do this operation? thank you.
Using ed to read first file.txt and then word.txt and shuffle lines of the latter around and finally save the modified file.txt:
ed -s file.txt <<EOF
0r word.txt
4d
2m5
4d
2m6
5d
w
EOF
m commands move lines, and d commands delete lines.
#!/bin/bash
# the 1st argument is the word.txt file
# remaining arguments are the line numbers to replace in ascending order
# assign the patterns file to file descriptor 3
exec 3< "$1"
shift
# read the first replacement pattern
read -ru 3 replacement
current_line_number=1
# lines will be read from standard in
while read -r line; do
# for replacement lines with a non-empty replacement str...
if [ "$current_line_number" == "$1" -a -n "$replacement" ]; then
echo "$replacement"
read -ru 3 replacement # get the next pattern
shift # get the next replacement line number
else
echo "$line"
fi
(( current_line_number++ ))
done
To test
diff expect <(./replace-lines.sh word.txt 1 3 5 < file.txt) && echo ok
I'm having trouble understanding what exactly you're trying to do, is this the end result you're after?
# read the 'word.txt' lines into an array called "words"
IFS=$'\n' read -d '' -r -a words < word.txt
# create a 'counter'
iter=0
# for loop through the line numbers that you want to change
for i in 1 3 5
do
# the variable "from" is the line from 'file.txt' (e.g. 1, 3, or 5)
from="$(sed -n ""$i"p" file.txt)"
# the variable "to" is the index/iteration of 'word.txt' (e.g. 1, 2, or 3)
to=${words[$iter]}
# make the 'inline' substitution
sed -i "s/$from/$to/g" file.txt
# increase the 'counter' by 1
((iter++))
done
cat file.txt
ganti
juice apple
buah
apple
bukan
mouse
Edit
If you have the line numbers you want to change in a file called "line.txt", you can adjust the code to suit:
IFS=$'\n' read -d '' -r -a words < word.txt
iter=0
while read -r line_number
do
from="$(sed -n ""$line_number"p" file.txt)"
to=${words[$iter]}
sed -i "s/$from/$to/g" file.txt
((iter++))
done < line.txt
cat file.txt
ganti
juice apple
buah
apple
bukan
mouse
Another ed approach, with Process substitution.
#!/usr/bin/env bash
ed -s file.txt < <(
printf '%s\n' '1s|^|1s/.*/|' '2s|^|3s/.*/|' '3s|^|5s/.*/|' ',s|$|/|' '$a' ,p Q . ,p Q |
ed -s word.txt
)
Related
I have a line like this:
INPUT file1
How can I get bash to read that line and directly copy in the contents of "file1.txt" in place of that line? Or if it sees: INPUT file2 on a line, put in `file2.txt" etc.
The best I can do is a lot of tr commands, to paste the file together, but that seems an overly complicated solution.
'sed' also replaces lines with strings, but I don't know how to input the entire content of a file, which can be hundreds of lines into the replacement.
Seems pretty straightforward with awk. You may want to handle errors differently/more gracefully, but:
$ cat file1
Line 1 of file 1
$ cat file2
Line 1 of file 2
$ cat input
This is some content
INPUT file1
This is more content
INPUT file2
This file does not exist
INPUT file3
$ awk '$1=="INPUT" {system("cat " $2); next}1' input
This is some content
Line 1 of file 1
This is more content
Line 1 of file 2
This file does not exist
cat: file3: No such file or directory
A perl one-liner, using the CPAN module Path::Tiny
perl -MPath::Tiny -pe 's/INPUT (\w+)/path("$1.txt")->slurp/e' input_file
use perl -i -M... to edit the file in-place.
Not the most efficient possible way, but as an exercise I made a file to edit named x and a couple of input sources named t1 & t2.
$: cat x
a
INPUT t2
b
INPUT t1
c
$: while read k f;do sed -ni "/$k $f/!p; /$k $f/r $f" x;done< <( grep INPUT x )
$: cat x
a
here's
==> t2
b
this
is
file ==> t1
c
Yes, the blank lines were in the INPUT files.
This will sed your base file repeatedly, though.
The awk solution given is better, as it only reads through it once.
If you want to do this in pure Bash, here's an example:
#!/usr/bin/env bash
if (( $# < 1 )); then
echo "Usage: ${0##*/} FILE..."
exit 2
fi
for file; do
readarray -t lines < "${file}"
for line in "${lines[#]}"; do
if [[ "${line}" == "INPUT "* ]]; then
cat "${line#"INPUT "}"
continue
fi
echo "${line}"
done > "${file}"
done
Save to file and run like this: ./script.sh input.txt (where input.txt is a file containing text mixed with INPUT <file> statements).
Sed solution similar to awk given erlier:
$ cat f
test1
INPUT f1
test2
INPUT f2
test3
$ cat f1
new string 1
$ cat f2
new string 2
$ sed 's/INPUT \(.*\)/cat \1/e' f
test1
new string 1
test2
new string 2
test3
Bash variant
while read -r line; do
[[ $line =~ INPUT.* ]] && { tmp=($BASH_REMATCH); cat ${tmp[1]}; } || echo $line
done < f
trying to change the value of a column based on other column in other csv
so let's say we have a CSV_1 that states with over 1000 lines with 3 columns
shape Color size
round 2 big
triangle 1 small
square 3 medium
then we have a CSV2 that has only 10 with the following information
color
1 REd
2 Blue
3 Yellow
etc
now i want to change the value in column color in CSV_1 with the name of the color of CSV2
so in other words .. something like
for (i=0; i<column.color(csv1); i++) {
if color.csv1=1; then
subustite with color.csv2=1 }
so that loop iterates in all CSV1 Color column and changes the value with the values from CSV2
An explicit loop for this would be very slow in bash. Use a command that does the line-wise processing for you.
sed 's/abc/xyz/' searches abc in each line and replaces it by xyz. Use this to search and replace the numbers in your 2nd column by the names from your 2nd file. The sed command can be automatically generated from the 2nd file using another sed command:
The following script assumes a CSV file without spaces around the delimiting ,.
sed -E "$(sed -E '1d;s#^([^,]*),(.*)#s/^([^,]*,)\1,/\\1\2,/#' 2.csv)" 1.csv
Interactive Example
$ cat 1.csv
shape,Color,size
round,2,big
triangle,1,small
square,3,medium
$ cat 2.csv
color
1,REd
2,Blue
3,Yellow
$ sed -E "$(sed -E '1d;s#^([^,]*),(.*)#s/^([^,]*,)\1,/\\1\2,/#' 2.csv)" 1.csv
shape,Color,size
round,Blue,big
triangle,REd,small
square,Yellow,medium
Here is one approach, with mapfile which is a bash4+ feature and some common utilities in linux/unix.
Assuming both files are delimited with a comma ,
#!/usr/bin/env bash
mapfile -t colors_csv2 < csv2.csv
head -n1 csv1.csv
while IFS=, read -r shape_csv1 color_csv1 size_csv1; do
for color_csv2 in "${colors_csv2[#]:1}"; do
if [[ $color_csv1 == ${color_csv2%,*} ]]; then
printf '%s,%s,%s\n' "$shape_csv1" "${color_csv2#*,}" "$size_csv1"
fi
done
done < <(tail -n +2 csv1.csv)
Would be very slow on large set of data/files.
If ed is available acceptable, with the bash shell.
#!/usr/bin/env bash
ed -s csv1.csv < <(
printf '%s\n' '1d' $'g|.|s|,|/|\\\ns|^|,s/|\\\ns|$|/|' '$a' ',p' 'Q' . ,p |
ed -s csv2.csv
)
To add to #Jetchisel interesting answer, here is an old bash way to achieve that. It should work with bash release 2 as it supports escape literals, indexed array, string expansion, indirect variable references. It implies that color keys in csv2.csv will always be a numeric value. Add shopt -s compat31 at the beginning to test it in the 'old way' with a recent bash. You can also replace declare -a csv2 with a Bash 4+ declare -A csv2 for an associative array, in which case the key can be anything.
#!/bin/bash
declare -a csv2
esc=$'\x1B'
while read -r colors; do
if [ "${colors}" ] ; then
colors="${colors// /${esc}}"
set ${colors//,/ }
if [ "$1" ] ; then
csv2["$1"]="$2"
fi
fi
done < csv2.csv
while read -r output; do
if [ "${output}" ] ; then
outputfilter="${output// /${esc}}"
set ${outputfilter//,/ }
if [ "$2" ] ; then
color="${csv2["$2"]}"
[ "${color}" ] && { tmp="$1,${color},$3";output="${tmp//${esc}/ }"; };
fi
echo "${output}"
fi
done < csv1.csv
In a bash script, I see several while statements with those redirect signs when closing the loop.
I know that if I end it with "done < file", I am redirecting the file to the stdin of the command in the while statement. But what the others means?
I would appreciate if someone could give an explanation with examples.
With the file text.txt
1aa
2bb
3cc
Redirection:
$ cat < text.txt
1aa
2bb
3cc
Here document:
$ cat << EOF
> 1AA
> 2BB
> EOF
1AA
2BB
Here string:
$ cat <<< 1aaa
1aaa
The first form, <, is an input redirection. It somewhat different than << and <<< which are two variants of a here document.
The first form, <, is primarily used to redirect the contents of a file to a command or process. It is a named FIFO, and therefor a file that is passed to a command that accepts file arguments.
cmd < file
will open the file named file and create a new file name to open and read. The difference between cmd file and cmd < file is the name passed to cmd in the second case is the name of a named pipe.
You can also do process substitution:
cmd <(process)
An example use would be comparing two directories:
diff <(ls dir1) <(ls dir2)
In this case, the command ls dir1 and ls dir2 has output redirected to a file like stream that is then read by diff as if those were two files.
You can see the name of the file device by passing to echo a process substitution:
$ echo <(ls)
/dev/fd/63
Since echo does not support opening files, it just prints the name of the FIFO.
Here documents are easier to demonstrate. The << form has a 'limit string' that is not included in the output:
$ cat <<HERE
> line 1
> line 2
> line 3
> HERE
line 1
line 2
line 3
The HERE is a unique string that must be on its own line.
The 'here string' or <<< form does not require the delimiting string of the << form and is on a single line:
$ cat <<< 'line 1'
line 1
You can also expand parameters:
$ v="some text"
$ cat <<< "$v"
some text
But not other forms of shell expansions:
Brace expansion:
$ echo a{b,c,d}e
abe ace ade
$ cat <<< a{b,c,d}e
a{b,c,d}e
Given a 'generic' Bash while loop that reads input line by line:
while IFS= read -r line || [[ -n $line ]]; do printf "'%s'\n" "$line"; done
There are several ways that you can feed input into that loop.
First example, you can redirect a file. For demo, create a 6 line file:
$ seq 6 > /tmp/6.txt
Redirect the input of the file into the loop:
while IFS= read -r line || [[ -n $line ]]; do printf "'%s'\n" "$line"; done </tmp/6.txt
'1'
'2'
'3'
'4'
'5'
'6'
Or, second example, you can directly read from the output of seq using redirection:
$ while IFS= read -r line || [[ -n $line ]]; do printf "'%s'\n" "$line"; done < <(seq 3)
'1'
'2'
'3'
(Please note the extra < with a space for this form)
Or, third example, you can use a 'HERE' doc separated by CR:
while IFS= read -r line || [[ -n $line ]]; do printf "'%s'\n" "$line"; done <<HERE
1
2 3
4
HERE
'1 '
'2 3'
' 4'
Going back to diff which will only work on files, you can use process substitution and a HERE doc or process substitution and redirection to use diff on free text or the output of a program.
Given:
$ cat /tmp/f1.txt
line 1
line 2
line 3
Normally you would need to have a second file to compare free text with that file. You can use a HERE doc and process substitution to skip creating a separate file:
$ diff /tmp/f1.txt <(cat <<HERE
line 1
line 2
line 5
HERE
)
3c3
< line 3
---
> line 5
command < foo
Redirect the file foo to the standard input of command.
command << foo
blah 1
blah 2
foo
Here document: send the following lines up to foo to the standard input of command.
command <<< foo
Here-string. The string foo is sent to the standard input of command.
I have one file Length.txt having multiples names (40) line by line.
I want to write a small shell script where it will count the character count of each line of the file and if the count is less than 9 replace those lines with adding extra 8 spaces and 1 at the end of each line.
For example, if the name is
XXXXXX
replace as
XXXXXX 1
I tried with the below coding. It is working for me, however whenever it's replacing the line it is displaying all the lines at a time.
So suppose I have 40 lines in Length.txt and out of that 4 lines having less than 9 character count then my output has 160 lines.
Can anyone help me to display only 40 line output with the 4 changed lines?
#!/usr/bin/sh
#set -x
while read line;
do
count=`echo $line|wc -m`
if [ $count -lt 9 ]
then
Number=`sed -n "/$line/=" Length.txt`;
sed -e ""$Number"s/$line/$line 1/" Length4.txt
fi
done < Length.txt
A single sed command can do that:
sed -E 's/^.{,8}$/& 1/' file
To modify the contents of the file add -i:
sed -iE 's/^.{,8}$/& 1/' file
Partial output:
94605320 1
105018263
2475218231
7728563 1
1
* Fixed to include only add 8 spaces not 9, and include empty lines. If you don't want to process empty lines, use {1,8}.
$ cat foo.input
I am longer than 9 characters
I am also longer than 9 characters
I am not
Another long line
short
$ while read line; do printf "$line"; (( ${#line} < 9 )) && printf " 1"; echo; done < foo.input
I am longer than 9 characters
I am also longer than 9 characters
I am not 1
Another long line
short 1
Let me show you what is wrong with your script. The only thing missing from your script is that you need to use sed -i to edit file and re-save it after making the replacement.
I'm assuming Length4.txt is just a copy of Length.txt?
I added sed -i to your script and it should work now:
cp Length.txt Length4.txt
while read line;
do
count=`echo $line|wc -m`
if [ $count -lt 9 ]
then
Number=`sed -n "/$line/=" Length.txt`
sed -ie ""$Number"s/$line/$line 1/" Length4.txt
fi
done < Length.txt
However, you don't need sed or wc. You can simplify your script as follows:
while IFS= read -r line
do
count=${#line}
if (( count < 9 ))
then
echo "$line 1"
else
echo "$line"
fi
done < Length.txt > Length4.txt
$ awk -v FS= 'NF<9{$0=sprintf("%s%*s1",$0,8,"")} 1' file
XXXXXX 1
Note how simple it would be to check for a number other than 9 characters and to print some sequence of blanks other than 8.
I want to say output lines 5 - 10 of a file, as arguments passed in.
How could I use head and tail to do this?
where firstline = $2 and lastline = $3 and filename = $1.
Running it should look like this:
./lines.sh filename firstline lastline
head -n XX # <-- print first XX lines
tail -n YY # <-- print last YY lines
If you want lines from 20 to 30 that means you want 11 lines starting from 20 and finishing at 30:
head -n 30 file | tail -n 11
#
# first 30 lines
# last 11 lines from those previous 30
That is, you firstly get first 30 lines and then you select the last 11 (that is, 30-20+1).
So in your code it would be:
head -n $3 $1 | tail -n $(( $3-$2 + 1 ))
Based on firstline = $2, lastline = $3, filename = $1
head -n $lastline $filename | tail -n $(( $lastline -$firstline + 1 ))
Aside from the answers given by fedorqui and Kent, you can also use a single sed command:
#!/bin/sh
filename=$1
firstline=$2
lastline=$3
# Basics of sed:
# 1. sed commands have a matching part and a command part.
# 2. The matching part matches lines, generally by number or regular expression.
# 3. The command part executes a command on that line, possibly changing its text.
#
# By default, sed will print everything in its buffer to standard output.
# The -n option turns this off, so it only prints what you tell it to.
#
# The -e option gives sed a command or set of commands (separated by semicolons).
# Below, we use two commands:
#
# ${firstline},${lastline}p
# This matches lines firstline to lastline, inclusive
# The command 'p' tells sed to print the line to standard output
#
# ${lastline}q
# This matches line ${lastline}. It tells sed to quit. This command
# is run after the print command, so sed quits after printing the last line.
#
sed -ne "${firstline},${lastline}p;${lastline}q" < ${filename}
Or, to avoid any external utilites, if you're using a recent version of bash (or zsh):
#!/bin/sh
filename=$1
firstline=$2
lastline=$3
i=0
exec <${filename} # redirect file into our stdin
while read ; do # read each line into REPLY variable
i=$(( $i + 1 )) # maintain line count
if [ "$i" -ge "${firstline}" ] ; then
if [ "$i" -gt "${lastline}" ] ; then
break
else
echo "${REPLY}"
fi
fi
done
try this one-liner:
awk -vs="$begin" -ve="$end" 'NR>=s&&NR<=e' "$f"
in above line:
$begin is your $2
$end is your $3
$f is your $1
Save this as "script.sh":
#!/bin/sh
filename="$1"
firstline=$2
lastline=$3
linestoprint=$(($lastline-$firstline+1))
tail -n +$firstline "$filename" | head -n $linestoprint
There is NO ERROR HANDLING (for simplicity) so you have to call your script as following:
./script.sh yourfile.txt firstline lastline
$ ./script.sh yourfile.txt 5 10
If you need only line "10" from yourfile.txt:
$ ./script.sh yourfile.txt 10 10
Please make sure that:
(firstline > 0) AND (lastline > 0) AND (firstline <= lastline)