Fairly new to bash, I have a few different files, all with thousands of lines each, I want to take each line from each file, and put them together on one line, in a new file,
for example, if file1 contains IPs
192.168.1.1
192.168.1.2
192.168.1.3
192.168.1.4
192.168.1.5
file2 contains ports
123
124
125
126
127
file3 contains timestamps, file4 contains a description
i'd like my output to be:
192.168.1.1 : 123 : 01/01/2012 : blah blah blah
192.168.1.2 : 124 : 01/02/2012 : blah blah
how could i go about putting them all together?
With paste(1) and sed(1).
paste file1 file2 file3 file4 | sed 's/\t/ : /g' > out
Open the four files as four separate file descriptors 3 through 6, then read one line from each in a loop. 0, 1, and 2 are already open as stdin, stdout, and stderr, respectively, so 3 is the first unused descriptor.
exec 3< file1
exec 4< file2
exec 5< file3
exec 6< file4
while IFS= read -r ip <&3 &&
IFS= read -r port <&4 &&
IFS= read -r timestamp <&5 &&
IFS= read -r description <&6
do
echo "$ip : $port : $timestamp : $description"
done
Related
I have a line like this:
INPUT file1
How can I get bash to read that line and directly copy in the contents of "file1.txt" in place of that line? Or if it sees: INPUT file2 on a line, put in `file2.txt" etc.
The best I can do is a lot of tr commands, to paste the file together, but that seems an overly complicated solution.
'sed' also replaces lines with strings, but I don't know how to input the entire content of a file, which can be hundreds of lines into the replacement.
Seems pretty straightforward with awk. You may want to handle errors differently/more gracefully, but:
$ cat file1
Line 1 of file 1
$ cat file2
Line 1 of file 2
$ cat input
This is some content
INPUT file1
This is more content
INPUT file2
This file does not exist
INPUT file3
$ awk '$1=="INPUT" {system("cat " $2); next}1' input
This is some content
Line 1 of file 1
This is more content
Line 1 of file 2
This file does not exist
cat: file3: No such file or directory
A perl one-liner, using the CPAN module Path::Tiny
perl -MPath::Tiny -pe 's/INPUT (\w+)/path("$1.txt")->slurp/e' input_file
use perl -i -M... to edit the file in-place.
Not the most efficient possible way, but as an exercise I made a file to edit named x and a couple of input sources named t1 & t2.
$: cat x
a
INPUT t2
b
INPUT t1
c
$: while read k f;do sed -ni "/$k $f/!p; /$k $f/r $f" x;done< <( grep INPUT x )
$: cat x
a
here's
==> t2
b
this
is
file ==> t1
c
Yes, the blank lines were in the INPUT files.
This will sed your base file repeatedly, though.
The awk solution given is better, as it only reads through it once.
If you want to do this in pure Bash, here's an example:
#!/usr/bin/env bash
if (( $# < 1 )); then
echo "Usage: ${0##*/} FILE..."
exit 2
fi
for file; do
readarray -t lines < "${file}"
for line in "${lines[#]}"; do
if [[ "${line}" == "INPUT "* ]]; then
cat "${line#"INPUT "}"
continue
fi
echo "${line}"
done > "${file}"
done
Save to file and run like this: ./script.sh input.txt (where input.txt is a file containing text mixed with INPUT <file> statements).
Sed solution similar to awk given erlier:
$ cat f
test1
INPUT f1
test2
INPUT f2
test3
$ cat f1
new string 1
$ cat f2
new string 2
$ sed 's/INPUT \(.*\)/cat \1/e' f
test1
new string 1
test2
new string 2
test3
Bash variant
while read -r line; do
[[ $line =~ INPUT.* ]] && { tmp=($BASH_REMATCH); cat ${tmp[1]}; } || echo $line
done < f
I'm trying to merge two files that look like File1 and File2 to something resembling Result. The aim is to use a pattern in File1 and insert the 1st line of File 2 in the line before the pattern in File 1, and recursively work through both files.
File1
Input:
Processed:
Result:
Input:
Result:
Input:
Error:
Result:
...
File2
Process1
Process2
Process3
…
Result
Process1
Input:
Processed:
Result:
Process2
Input:
Result:
Process3
Input:
Error:
Result:
...
So far this will insert the line from File2 on the line below the occurrence of the pattern not above:
awk 'NR==FNR{a[NR]=$0;next}1;/Input:/{print a[++i]}' File2 File2
Any help appreciated, thanks!
you're almost there
$ awk 'NR==FNR{p[NR]=$0; next} /^Input:/{print p[++c]}1' file2 file1
Process1
Input:
Processed:
Result:
Process2
Input:
Result:
Process3
Input:
Error:
Result:
you want to print the line after the insertion, so remove print line to the end, which is 1 as shorthand.
As I understand, you want to insert a line from File2 above each line matching Input: from File1.
The below will do that:
#!/usr/bin/env bash
case $BASH_VERSION in ''|[123].*|4.0.*) echo "ERROR: Bash 4.1 or newer required" >&2; exit 1;; esac
exec {file1_fd}<File1 # open File1 for input, storing the FD# it's opened on in file1_fd
exec {file2_fd}<File2 # likewile for File2 and file2_fd
while IFS= read -r f1line <&"$file1_fd"; do # as long as we can read a line from file1...
if [[ $f1line =~ Input: ]]; then # if that line contains "Input:"
IFS= read -r f2line <&"$file2_fd" # ...then read a second line from file2
printf '%s\n' "$f2line" # ...and write that second line to our output
fi
printf '%s\n' "$f1line" # before writing the content from file1.
done
This might work for you (GNU sed):
sed -n '/Input:/=' file1 |
paste - file2 |
sed -nr 's/(.+)\t(.)/\1i\\\2/p' |
sed -f - file1
This solution makes a sed script from file1 and file2 which uses the line numbers from file1 where the line begins Input: and combines that with the respective line in file2 as an insert command.
If there is always a Result: following an Input: then the following solution would be acceptable.
sed -e '/Input:/R file2' -e '//h;//d' -e 'x;/./p;z;x' file1
Yet another way:
sed '/Input:/i\insert here' file1 | sed -e '/^insert here/R file2' -e '//d'
I have a file A with 400,000 lines. I have another file B that has a bunch of line numbers.
File B:
-------
98
101
25012
10098
23489
I have to extract those line numbers specified in file B from file A. That is I want to extract lines 98,101,25012,10098,23489 from file A. How to extract these lines in the following cases.
File B is a explicit file.
File B is arriving out of a pipe. For eg., grep -n pattern somefile.txt is giving me the file B.
I wanted to use see -n 'x'p fileA. However, I don't know how to give the 'x' from a file. Also, I don't to how to pipe the value of 'x' from a command.
sed can print the line numbers you want:
$ printf $'foo\nbar\nbaz\n' | sed -ne '2p'
bar
If you want multiple lines:
$ printf $'foo\nbar\nbaz\n' | sed -ne '2p;3p'
bar
baz
To transform a set of lines to a sed command like this, use sed for beautiful sedception:
$ printf $'98\n101' | sed -e 's/$/;/'
98;
101;
Putting it all together:
sed -ne "$(sed -e 's/$/p;/' B)" A
Testing:
$ cat A
1
22
333
4444
$ cat B
1
3
$ sed -ne "$(sed -e 's/$/p;/' B)" A
1
333
QED.
awk fits this task better:
fileA in file case:
awk 'NR==FNR{a[$0]=1;next}a[FNR]' fileB fileA
fileA content from pipe:
cat fileA|awk 'NR==FNR{a[$0]=1;next}a[FNR]' fileB -
oh, you want FileB in file or from pipe, then same awk cmd:
awk '...' fileB fileA
and
cat fileB|awk '...' - fileA
I have this script which reads lines from two files and outputs them in order of:
The first line of file1
the first line of file2
the second line of file1
the second line of file2
etc
How do I do this without using the external paste command in the script ?
paste -d'\n' file1 file2 | while read line1 && read line2;
do
#echo "$line1 $line2"
echo "$line1"
echo "$line2"
done
Use file descriptors and read, e.g. see here.
exec 5< file1
exec 6< file2
read line1 <&5
read line2 <&6
echo -n "$line1\n$line2"
Thanks for that article Kerrek .... have updated my code which works fine now :
exec 5< file1
exec 6< file2
while read line1 <&5 && read line2 <&6
do
echo "$line1"
echo "$line2"
done
bash.sh
#!/bin/bash
exec 3< bash.sh
exec 4< data
while read l1 <&3 && read l2 <&4
do
echo "$l1"
echo "$l2"
done
data
1908 462
232 538
232 520
232 517
./bash.sh
#!/bin/bash
1908 462
232 538
exec 3< bash.sh
232 520
exec 4< data
232 517
If you don't want to end when you reach end of the first file, use this
#!/bin/bash
exec 3< aaa
exec 4< bbb
while true
do
end=1
read l <&3
if [ $? -eq 0 ];
then
echo "$l"
end=0
fi
read l <&4
if [ $? -eq 0 ];
then
echo "$l"
end=0
fi
if [ $end -eq 1 ];
then
break
fi
done
External to the while, and external to some enclosing script file are two different things. You're perfectly free to move the paste into the file, and still pipe into the while within.
I'm not sure if that's what you're getting after. Is it?
need advice about the following
with the following ksh script I actually copy file1 to file2
my problem is that lines in file2 are not with the same location as file1
#!/bin/ksh
while read -r line ; do
echo $line >> file2
done < file1
for example
more file1
line1
line2
line3
more file2
line1
line2
line3
the question what I need to change in my script in order to get lines location as described in file1? after I run my ksh script?
lidia
You can try:
while read -r line ; do
echo $line | sed -re 's/^\s+//' >> file2
done < file1
This uses sed to get rid of the leading whitespaces present in lines from file1.
you can set IFS=
while IFS= read -r line ; do echo "$line"; done<file