This question already has answers here:
How to merge every two lines into one from the command line?
(21 answers)
Closed 6 years ago.
I'm looking for a shell command to delete the return chariot on one line out of two.
I have a file like this :
1.32640997
;;P
1.14517534
;;P
1.16120958
;;P
...
And I would like something like this:
1.32640997;;P
1.14517534;;P
1.16120958;;P
...
Is it possible?
Thanks
Using GNU paste
paste -d '' - - < file
Using BSD paste
paste -d '\0' - - < file
paste produces two columns from stdin with - - as parameters, 3 columns with - - - as parameters, and so on.
-d is to specify a column separator, use '\0' for no separator.
Using Perl
perl -ne 'chomp($prev = $_); print $prev, scalar <>' < file
Using awk
$ awk '{printf "%s%s",$0,(NR%2==0?ORS:"")}' File
1.32640997;;P
1.14517534;;P
1.16120958;;P
This prints each line followed by nothing for odd lines or followed by the output record separator for even lines.
Using sed
This works by reading in lines in pairs:
$ sed 'N; s/\n//' File
1.32640997;;P
1.14517534;;P
1.16120958;;P
N reads in the next line and s/\n// removes the newline.
Using xargs:
xargs -n 2 -d '\n' printf '%s%s\n' <file
Related
This question already has answers here:
Unix command to prepend text to a file
(21 answers)
Closed 4 years ago.
I want to write a bash script that takes my file:
READ_ME.MD
two
three
four
and makes it
READ_ME.MD
one
two
three
four
There are a bunch of similar StackOverflow questions, but I tried their answers and haven't been successful.
These are the bash scripts that I have tried and failed with:
test.sh
sed '1s/^/one/' READ_ME.md > READ_ME.md
Result: Clears the contents of my file
test.sh
sed '1,1s/^/insert this /' READ_ME.md > READ_ME.md
Result: Clears the contents of my file
test.sh
sed -i '1s/^/one\n/' READ_ME.md
Result: sed: 1: "READ_ME.md": invalid command code R
Any help would be appreciated.
You can use this BSD sed command:
sed -i '' '1i\
one
' file
-i will save changes inline to file.
If you want to add a line at the top if same line is not already there then use BSD sed command:
line='one'
sed -i '' '1{/'"$line"'/!i\
'"$line"'
}' file
Your last example works for me with GNU sed. Based on the error message you added, I'd guess you're working on a Mac system? According to this blog post, a suffix argument may be required on Mac versions of sed:
sed -i ' ' '1s/^one\n/' READ_ME.md
If this is bash or zsh, you can use process substitution like so.
% cat x
one
two
three
% cat <(echo "zero") x
zero
one
two
three
Redirect this into a temp file and copy it back to the original
there is always ed
printf '%s\n' H 1i "one" . w | ed -s READ_ME.MD
This question already has answers here:
shell: delete the last line of a huge text log file [duplicate]
(4 answers)
Remove the last line from a file in Bash
(16 answers)
Closed 5 years ago.
This post was edited and submitted for review 6 months ago and failed to reopen the post:
Original close reason(s) were not resolved
How can I delete the last line of a file without reading the entire file or rewriting it in any temp file? I tried to use sed but it reads the entire file into memory which is not feasible.
I want to remove the blank line from the end and save the change to the same file.
Since the file is very big and reading through the complete file would be slow, I am looking for a better way.
simple sed command to delete last line:
sed '$d' <file>
here in sed $ is the last line.
You can try awk command:
awk 'NR > 1{print t} {t = $0}END{if (NF) print }' file
Using cat:
cat file.txt | head -n -1 > new_file.txt
The easiest way I can think of is:
sed -i '${/^$/d}' filename
edited to delete only blank end line.
Your only option not using sed that won't read the entire file is to stat the file to get the size and then use dd to skip to the end before you start reading. However, telling sed to only operate on the last line, does essentially that.
Take a look at
Remove the last line from a file in Bash
Edit: I tested
dd if=/dev/null of=<filename> bs=1 seek=$(echo $(stat --format=%s <filename> ) - $( tail -n1 <filename> | wc -c) | bc )
and it does what you want
How can remove everything in a pipe delimited file after the second-to-last pipe? Like for the line
David|3456|ACCOUNT|MALFUNCTION|CANON|456
the result should be
David|3456|ACCOUNT|MALFUNCTION
Replace |(string without pipe)|(string without pipe) at the end of each line:
sed 's/|[^|]*|[^|]*$//' inputfile
Using awk, something like
awk -F'|' 'BEGIN{OFS="|"}{NF=NF-2; print}' inputfile
David|3456|ACCOUNT|MALFUNCTION
(or) use cut if you know the number of columns in total, i,e 6 -> 4
cut -d'|' -f -4 inputfile
David|3456|ACCOUNT|MALFUNCTION
The command I would use is
cat input.txt | sed -r 's/(.*)\|.*/\1/' > output.txt
A pure Bash solution:
while IFS= read -r line || [[ -n $line ]] ; do
printf '%s\n' "${line%|*|*}"
done <inputfile
See Reading input files by line using read command in shell scripting skips last line (particularly the answer by Jahid) for details of how the while loop works.
See pattern matching in Bash for information about ${line%|*|*}.
This question already has answers here:
Bash script prints "Command Not Found" on empty lines
(17 answers)
Closed 6 years ago.
I have a file store version information and I wrote a shell to read two fields and combine them. But when I concatenate those two fields, it show me a werid result.
version file:
buildVer = 3
version = 1.0.0
script looks like:
#!bin/bash
verFile='version'
sdk_ver=`cat $verFile | sed -nE 's/version = (.*)/\1/p'`
build_ver=`cat $verFile | sed -nE 's/buildVer = (.*)/\1/p'`
echo $sdk_ver
echo $build_ver
tagname="$sdk_ver.$build_ver"
echo $tagname
The output shows
1.0.0
3
.30.0
I tried to echo the sdk_ver directly without read it from file, this piece of script works well. So I think it may relate to the sed, but I couldn't figure out how to fix it.
Does anyone know why it acts like that?
You're getting this problem because of presence of DOS line ending i.e. \r in each line of version file.
Use dos2unix or this sed command to remove \r first:
sed -i 's/\r//' version
btw you can also simplify your script using pure BASH constructs like this:
#!/bin/bash
while IFS='= ' read -r k v; do
declare $k="$v"
done < <(sed $'s/\r//' version)
tagname="$version.$buildVer"
echo "$tagname"
This will give output:
1.0.0.3
Alternate solution, with awk:
awk '/version/{v=$3} /buildVer/{b=$3} END{print v "." b}' version.txt
Example:
$ cat file.txt
buildVer = 3
version = 1.0.0
$ awk '/version/{v=$3} /buildVer/{b=$3} END{print v "." b}' file.txt
1.0.0.3
Sometimes I receive a CSV file which has a carriage return inside a cell. This is not an acceptable format to a program that will use it as input.
In order to detect if an input line is split, I determined that a bad line would not have the expected number of commas in it. Is there a bash or other common unix command line tool that would allow me to count the commas in the line? If necessary, I can write a Python or Perl program to do it, but if possible, I'd like to add a line or two to an existing bash script to cause it to fail if the comma count is wrong. Any ideas?
Strip everything but the commas, and then count number of characters left:
$ echo foo,bar,baz | tr -cd , | wc -c
2
To count the number of times a comma appears, you can use something like awk:
string=(line of input from CSV file)
echo "$string" | awk -F "," '{print NF-1}'
But this really isn't sufficient to determine whether a field has carriage returns in it. Fields can have commas inside as long as they're surrounded by quotes.
What worked for me better than the other solutions was this. If test.txt has:
foo,bar,baz
baz,foo,foobar,bar
Then cat test.txt | xargs -I % sh -c 'echo % | tr -cd , | wc -c' produces
2
3
This works very well for streaming sources, or tailing logs, etc.
In pure Bash:
while IFS=, read -ra array
do
echo "$((${#array[#]} - 1))"
done < inputfile
or
while read -r line
do
count=${line//[^,]}
echo "${#count}"
done < inputfile
Try Perl:
$ perl -ne 'print 0+#{[/,/g]},"\n"'
a
0
a,a
1
a,a,a,a,a
4
Depending on what you are trying to do with the CSV data, it may be helpful to use a wrapper script like csvquote to temporarily replace the problematic newlines (and commas) inside quoted fields, then restore them. For instance:
csvquote inputfile.csv | wc -l
and
csvquote inputfile.csv | cut -d, -f1 | csvquote -u
may be the sort of thing you're looking for. See [https://github.com/dbro/csvquote][1] for the code and more information
An example Python command you could run (since it's going to be installed on most modern shells) is:
python -c "import pathlib; print({l.count(',') for l in pathlib.Path('my_file.csv').read_text().splitlines()})"
This counts the number of commas per line, then makes a set from them (so if your lines all have the same number of commas in, you'll get a set with just that number in).
Just remove all of the carriage returns:
tr -d "\r" old_file > new_file