I have a simple script to read the content of a input file line by line with some extra strings added. Here is one example.
input file: input.txt
content of this input.txt
aaaaaa
bbbbbb
cccccc
dddddd
eeeeee
code to read the file
while IFS='' read -r line || [[ -n "$line" ]]; do
echo "abc.def.hig.ewe.adg.hea.L_${line}.great"
done < "$1"
I'm not sure where exactly it is wrong; I cannot get correct output. It looks like when you add .great at the end of a variable, the output will mess up the sequence.
Transferring comments into an answer.
What output do you get? What output do you expect? I got five lines similar to:
abc.def.hig.ewe.adg.hea.L_aaaaaa.great
using Bash 3.2.57 and Bash 4.3 (on Mac OS X 10.11.5).
Is the data file from Windows or some other source that uses CRLF line endings? That will make things look like:
.greatf.hig.ewe.adg.hea.L_aaaaaa
instead of what I showed before. And, I note, this would have been readily explicable (or discountable) if you'd only showed the output you were getting.
Yes, you are right. The problem is the input file. It got corrupted when uploading from Windows to Unix.
Have you considered using something like awk? It is very good for these types of simple tasks.
$ cat input.txt
aaaa
bbbb
cccc
-
$ awk '{print "prefix_"$0"_suffix"}' input.txt
prefix_aaaa_suffix
prefix_bbbb_suffix
prefix_cccc_suffix
Related
I have a file .txt with some informations, i need to grep the "Report:" line and save each line in a different .txt file!
it should result something like this in the end:
case1.txt
case2.txt
case3.txt
I tried to
cat cases.txt| grep Report: | while read Report; do echo $Report | > /home/kali/Desktop/allcases/case.txt done
but it didnt work and just created one file called case.txt containing the last grepped "Report:"
I dont know if i was very clear then i'll show this screenshot:
cases
I wanted to split all theses reports in a different .txt file for each report!
These case informations are from a game, so dont worry!
awk would be better suited than grep and a while loop. If acceptable, you can try;
awk '/^Report/{cnt++;close(report); report="case"cnt".txt"}/./{print > report}' file.txt
perl -ne '++$i && `printf "$_" > case$i.txt` if /Report:/' cases.txt
This is looping over cases.txt and shelling out printf "$_" > case$i.txt if the line matches /Report:/
Because it's perl there's some syntax and precedence tricks in here to make it terse and confusing.
Suppose I've got a text file that consists of two parts separated by delimiting string ---
aa
bbb
---
cccc
dd
I am writing a bash script to read the file and assign the first part to var part1 and the second part to var part2:
part1= ... # should be aa\nbbb
part2= ... # should be cccc\ndd
How would you suggest write this in bash ?
You can use awk:
foo="$(awk 'NR==1' RS='---\n' ORS='' file.txt)"
bar="$(awk 'NR==2' RS='---\n' ORS='' file.txt)"
This would read the file twice, but handling text files in the shell, i.e. storing their content in variables should generally be limited to small files. Given that your file is small, this shouldn't be a problem.
Note: Depending on your actual task, you may be able to just use awk for the whole thing. Then you don't need to store the content in shell variables, and read the file twice.
A solution using sed:
foo=$(sed '/^---$/q;p' -n file.txt)
bar=$(sed '1,/^---$/b;p' -n file.txt)
The -n command line option tells sed to not print the input lines as it processes them (by default it prints them). sed runs a script for each input line it processes.
The first sed script
/^---$/q;p
contains two commands (separated by ;):
/^---$/q - quit when you reach the line matching the regex ^---$ (a line that contains exactly three dashes);
p - print the current line.
The second sed script
1,/^---$/b;p
contains two commands:
1,/^---$/b - starting with line 1 until the first line matching the regex ^---$ (a line that contains only ---), branch to the end of the script (i.e. skip the second command);
p - print the current line;
Using csplit:
csplit --elide-empty-files --quiet --prefix=foo_bar file.txt "/---/" "{*}" && sed -i '/---/d' foo_bar*
If version of coreutils >= 8.22, --suppress-matched option can be used and sed processing is not required, like
csplit --suppress-matched --elide-empty-files --quiet --prefix=foo_bar file.txt "/---/" "{*}".
I have a very large data dump that I need to manipulate. Basically, I receive a text file that has data from multiple tables in it. The first two characters of each line will tell me what table this is from. I need to read each of these lines and then extract them into a TEXT file... It would append each line to the text file. Each table should have it's own text file.
For example, lets say the data file looks like this...
HDxxxxxxxxxxxxx
HDyyyyyyyyyyyyy
ENxxxxxxxxxxxxx
ENyyyyyyyyyyyyy
HSyyyyyyyyyyyyy
What I would need is the first two lines to be in a text file named HD_out.txt, the 3rd and 4th lines in one named EN_out.txt, and the last one in a file named HS_out.txt.
Does anyone know how could this be done with either a simple batch file or UNIX shell script?
Use awk to split file based on first 2 characters:
gawk -v FIELDWIDTHS='2 99999' '{print $2 > $1"_out.txt"}' input.txt
Using bash:
while read -r line; do
echo "${line:2}" >> "${line:0:2}_out.txt"
done < inputFile
${var:startposition:length} is a bash string function to capture sub-strings. This would cause your inputfile to be split based on the first two chars. If you want to include the table prefix, just use echo "$line" >> "${line:0:2}_out.txt" instead of what is shown above.
Demo:
$ ls
file
$ cat file
HDxxxxxxxxxxxxx
HDyyyyyyyyyyyyy
ENxxxxxxxxxxxxx
ENyyyyyyyyyyyyy
HSyyyyyyyyyyyyy
$ while read -r line; do echo "${line:2}" >> "${line:0:2}_out.txt"; done < file
$ ls
EN_out.txt file HD_out.txt HS_out.txt
$ head *.txt
==> EN_out.txt <==
xxxxxxxxxxxxx
yyyyyyyyyyyyy
==> HD_out.txt <==
xxxxxxxxxxxxx
yyyyyyyyyyyyy
==> HS_out.txt <==
yyyyyyyyyyyyy
I've searched for hours looking for the answer to this question which seems frustratingly simple...
I have a bash script which I've simplified to find the line that's stopping it from working and am left with:
#!/bin/bash
#
sed -i -e "s/<link>/\n/g" /usb/lenny/rss/tmp/rss.tmp
If I run this script, nothing happens to the file rss.tmp - but if I call this exact same sed command from the terminal, it makes all the replacements as expected.
Anyone have any idea what I'm doing wrong here?
Based on the discussion the issue sounds like it is a cygwin shell problem.
The issue is that shell scripts may not have \r\n line terminations - they need \n terminations. Earlier versions of cygwin behaved differently.
The relevant section from a Cygwin FAQ at http://cs.nyu.edu/~yap/prog/cygwin/FAQs.html
Q: Mysterious errors in shell scripts, .bashrc, etc
A: You may get mysterious messages when bash reads
your .bashrc or .bash_profile, such as
"\r command not found"
(or similar). When you get rid of empty lines, the
complaints about "\r" disappears, but probably other
errors remain. What is going on?
The answer may lie in the fact that a text file (also
called ASCII file) can come in two formats:
in DOS format or in UNIX format.
Most editors can automatically detect the formats
and work properly in either format.
In the DOS format, a new line is represented by two characters:
CR (carriage return or ASCII code 13) and LF (line feed or ASCII code 15).
In the UNIX format, a new line is represented by only
one character, LF. When your .bashrc file is read,
bash thinks the extra character is the name of a command,
hence the error message.
In Cygwin or unix, you can convert a file INFILE in DOS format
to a file OUTFILE in Unix format by calling:
> tr -d '\15' OUTFILE
NOTE:
If you now compare the number of characters in INFILE and OUTFILE,
you will see that the latter has lost the correct
number of characters (i.e., the number of lines in INFILE):
> wc INFILE OUTFILE
Try using that instead:
sed -i -e "s/\<link\>/\n/g" /usb/lenny/rss/tmp/rss.tmp
You need to give an output file or the result will be only shown on the screen.
sed -e 's/<link>/\n/g' /usb/lenny/rss/tmp/rss.tmp > /usb/lenny/rss/tmp/output.tmp
to feed a file to the command you use "<", while to make a file u use ">" and sed is used as text formater not editor as far as i know
maybe something like this should work
cat < /usb/lenny/rss/tmp/rss.tmp | sed -i -e "s/<link>/\n/g" > /usb/lenny/rss/tmp/rssedit.tmp
cat gets the file and with sed editing it and ouput goes to rssedit.tmp
than check if rssedit.tmp has what u wanted
if it does and only if it does
next line of the your skript
should be
mv /usb/lenny/rss/tmp/rssedit.tmp /usb/lenny/rss/tmp/rss.tmp
which will replace made 1 with original, with renameing to original
I've seen the technique before, but don't know what it's called and forget the exact syntax. Let's say I need to pipe in a file to a program like: command < input-file. However, I want to directly pass these lines of the input file into the command without the intermediate input file. It looks something like this, but it doesn't work:
command < $(file-line1; file-line2; file-line3)
Can someone tell me what this is called and how to do it?
This is called Process Substitution
command < <(printf "%s\n" "file-line1" "file-line2" "file-line3")
With the above, command will think its being input a file with a name much like /dev/fd/XX where 'XX' is some number. As you mentioned, this is a temporary file (actually a file descriptor) but it will contain the 3 lines you passed in to the printf command.
Herestring.
command <<< $'line 1\nline 2\nline 3\n'
Or heredoc.
command << EOF
line 1
line 2
line 3
EOF
I think you are referring to a "here document". Like this
#!/bin/sh
cat <<EOF
This is
the
lines.
EOF
How about:
cat myfile.txt | head -n3 | command