I have a file with below data. I want change the second column from "201710110923" format to "2017-10-11 09:23:00" format using shell script
"2017-10-16 14:03:07", "201710110923"
"2017-10-16 14:03:22", "201710110930"
Please help.
Thanks & Regards
Rajesh
Use the following:
sed -r 's/(.*, \")([0-9]{4})([0-9]{2})([0-9]{2})([0-9]{2})([0-9]{2})/\1\2-\3-\4 \5:\6:00/g' inputfile.in
Here, we are making use of regex groups to capture the matches and then replace the output. If you want to update the file, use the -i option.
Following awk may help you in same.
awk -F"\"" '{$4=substr($4,1,4)"-"substr($4,5,2)"-"substr($4,7,2) " " substr($4,9,2)":"substr($4,11,2)":"substr($4,13,2)} 1' OFS="\"" Input_file
Output will be as follows.
"2017-10-16 14:03:07", "2017-10-11 09:23:"
"2017-10-16 14:03:22", "2017-10-11 09:30:"
bash alternative solution:
IFS=','; while read d1 d2; do
d2=$(tr -d '[:space:]' <<<"$d2")
new_d=$(date -d"$(echo ${d2:1:4}/${d2:5:2}/${d2:7:2} ${d2:9:4})" +'%Y-%m-%d %H:%M:%S')
printf '%s, "%s"\n' "$d1" "$new_d"
done < yourfile
The output:
"2017-10-16 14:03:07", "2017-10-11 09:23:00"
"2017-10-16 14:03:22", "2017-10-11 09:30:00"
Related
I have a hexdump output that looks like this
0101f10 64534 64943 00568 00262 01077 00721 00297 00140
0101f20 00748 00288 02211 01124 02533 01271 02451 00997
0101f30 03056 01248 02894 01026 02397 00696 00646 65114
0101f40 00943 64707 01113 64179 01135 64179 00805 64109
0101f50 00514 64045 64654 63037 63026 62014 62173 61625
I want to remove the first column, but I don't know what delimiter has been used by the hexdump command. I tried with awk and cut, but cant figure it out. Any help is appreciated.
Output I want is
64534 64943 00568 00262 01077 00721 00297 00140
00748 00288 02211 01124 02533 01271 02451 00997
03056 01248 02894 01026 02397 00696 00646 65114
00943 64707 01113 64179 01135 64179 00805 64109
00514 64045 64654 63037 63026 62014 62173 61625
With sed
sed 's/[^[:blank:]]*[[:blank:]]*//' infile
With gnu sed
sed 's/\S*\s*//' infile
input... | sed -E $'s/ +/\t/g' | cut -f2-
(Assuming Bash for $'\t', but GNU sed supports \t directly anyway.)
hexdump /path/to/file | awk '{sub(/[^ ]+ /, ""); print $0}'
This will do the job.
If the the delimiter is really a bunch of space, use tr to squeeze-repeats (-s) of psace to a tab and use cut for getting rid of the first column:
$ cat file | tr -s ' ' '\t' | cut -f 2-
64534 64943 00568 00262 01077 00721 00297 00140
00748 00288 02211 01124 02533 01271 02451 00997
03056 01248 02894 01026 02397 00696 00646 65114
00943 64707 01113 64179 01135 64179 00805 64109
00514 64045 64654 63037 63026 62014 62173 61625
All solution above works fine, Just adding an awk solution.
So, you only need to omit first column, but get the rest of it, you can try this :
awk '{$1=""; print $0}' /path/to/hexfile
It works perfectly, except that it leaves a space at beginning of each line. If that bothers you, there is a workaround using the substr() function in awk itself.
awk '{$1=""; print substr($0,2)}' /path/to/hexfile
To see more possible ways to do it, follow this link
I have a file in Unix, with data sample like the following:
{"ID":"123", "Region":"Asia", "Location":"India"}
{"ID":"234", "Region":"APAC", "Location":"Australia"}
{"ID":"345", "Region":"Americas", "Location":"Mexio"}
{"ID":"456", "Region":"Americas", "Location":"Canada"}
{"ID":"567", "Region":"APAC", "Location":"Japan"}
The desired output is
ID|Region|Location
123|Asia|India
234|APAC|Australia
345|Americas|Mexico
456|Americas|Canada
567|APAC|Japan
I tried with a few sed commands. I could remove the following: '{', '}', ' " ', ':'
There are 2 issues with the output file
All rows from input appear in single line in the output.
Adding the pipe ('|') as delimiter.
Any pointers are highly appreciated.
I recommend the tool jq (http://stedolan.github.io/jq/); jq is a lightweight and flexible command-line JSON processor.
jq -r '"\(.ID)|\(.Region)|\(.Location)"' < infile
123|Asia|India
234|APAC|Australia
345|Americas|Mexio
456|Americas|Canada
567|APAC|Japan
Explanation
-r is --raw-output
Through awk,
awk -F'"' -v OFS="|" 'BEGIN{print "ID|Region|Location"}{print $4,$8,$12}' file
Example:
$ cat file
{"ID":"123", "Region":"Asia", "Location":"India"}
{"ID":"234", "Region":"APAC", "Location":"Australia"}
{"ID":"345", "Region":"Americas", "Location":"Mexio"}
{"ID":"456", "Region":"Americas", "Location":"Canada"}
{"ID":"567", "Region":"APAC", "Location":"Japan"}
$ awk -F'"' -v OFS="|" 'BEGIN{print "ID|Region|Location"}{print $4,$8,$12}' file
ID|Region|Location
123|Asia|India
234|APAC|Australia
345|Americas|Mexio
456|Americas|Canada
567|APAC|Japan
EXplanation:
-F'"' Sets " as Field Separator value.
OFS="|" Sets | as Output Field Separator value.
Atfirst, awk would execute the function inside the BEGIN block. It helps to print the header section.
This sed one-liner does what you want. It's capturing the field values using parenthesized expressions, and then putting them into the output using \1, \2, and \3.
s/^{"ID":"\([^"]*\)", "Region":"\([^"]*\)", "Location":"\([^"]*\)"}$/\1|\2|\3/
Invoke it like:
$ sed -f one-liner.sed input.txt
Or you can invoke it within a Bash script, producing the header:
echo 'ID|Region|Location'
sed -e 's/^{"ID":"\([^"]*\)", "Region":"\([^"]*\)", "Location":"\([^"]*\)"}$/\1|\2|\3/' $input
It is a JSON file so it is best to use a JSON parser. Here is a perl implementation of it.
#!/usr/bin/perl
use strict;
use warnings;
use JSON;
open my $fh, '<', 'path/to/your/file';
#keys of your structure
my #key = qw(ID Region Location);
print join ("|", #key), "\n";
#iterate over your file, decode it and print in order of your key structure
while (my $json = <$fh>) {
my $text = decode_json($json);
print join ("|", map { $$text{$_} } #key ),"\n";
}
Output:
ID|Region|Location
123|Asia|India
234|APAC|Australia
345|Americas|Mexio
456|Americas|Canada
567|APAC|Japan
Using sed as follows
Command line
echo "my_string" |
sed -e 's#[,:"{}]##g' -e 's#ID##g' -e "s#Region##g" -e 's#Location##g' \
-e '1 s#^.*$#ID Region Location\n&#' -e 's# #|#g'
or
sed -e 's#[,:"{}]##g' -e 's#ID##g' -e "s#Region##g" -e 's#Location##g' \
-e '1 s#^.*$#ID Region Location\n&#' -e 's# #|#g' my_file
I tried this in a terminal as follows:
echo '{"ID":"123", "Region":"Asia", "Location":"India"}
{"ID":"234", "Region":"APAC", "Location":"Australia"}
{"ID":"345", "Region":"Americas", "Location":"Mexio"}
{"ID":"456", "Region":"Americas", "Location":"Canada"}
{"ID":"567", "Region":"APAC", "Location":"Japan"}' |
sed -e 's#[,:"{}]##g' -e 's#ID##g' -e "s#Region##g" -e 's#Location##g' \
-e '1 s#^.*$#ID Region Location\n&#' -e 's# #|#g'
Output
ID|Region|Location
123|Asia|India
234|APAC|Australia
345|Americas|Mexio
456|Americas|Canada
567|APAC|Japan
Many thanks for your response and the pointers/ solutions did help a lot.
For some mysterious reasons, I couldn't get any sed commands work. So, I devised my own solution. Although it's not elegant, it's still worked.
Here is the script I prepared which resolved the issue.
#!/bin/bash
# ource file path.
infile=/home/exfile.txt
# remove if these temp file exist already.
rm ./efile.txt ./xfile.txt ./yfile.txt ./zfile.txt
# removing the curly braces from input file.
cat exfile.txt | cut -d "{" -f2 | cut -d "}" -f1 >> ./efile.txt
# setting input file name to different value.
infile=./efile.txt
# remove double quotes from the file.
while IFS= read -r line
do
echo $line | sed 's/\"//g' >> ./xfile.txt
done < "$infile"
# creating another temp file.
infile2=./xfile.txt
# remove colon from file.
while IFS= read -r line
do
echo $line | sed 's/\:/,/g' >> ./yfile.txt
done < "$infile2"
# set input file path to new temp file.
infile3=yfile.txt
# initialize variables to hold header column values.
t1=0
t3=0
t5=0
# read each of the line to extract header row. Exit loop after reading 1st row.
once=1
while IFS=',' read -r f1 f2 f3 f4 f5 f6
do
"$f1 $f2 $f3 $f4 $f5 $f6"
t1=$f1
t3=$f3
t5=$f5
if [ "$once" -eq 1 ]; then
break
fi
done < "$infile3"
# Read each of the line from input file. Write only the value to another output file.
while IFS=',' read -r f1 f2 f3 f4 f5 f6
do
echo "$f2|$f4|$f6" >> ./zfile.txt
done < "$infile3"
# insert the header column row into the file generated in the step above.
frstline="$t1|$t3|$t5"
sed -i '1i ID|Region|Location' ./zfile.txt
I run a script with the param -A AA/BB . To get an array with AA and BB, i can do this.
INPUT_PARAM=(${AIRLINE_OPTION//-A / }) #get rid of the '-A ' in the begining
LIST=(${AIRLINES_PARAM//\// }) # split by '/'
Can we achieve this in a single line?
Thanks in advance.
One way
IFS=/ read -r -a LIST <<< "${AIRLINE_OPTION//-A /}"
This places the output from the parameter substitution ${AIRLINE_OPTION//-A /} into a "here-string" and uses the bash read built-in to parse this into an array. Splitting by / is achieved by setting the value of IFS to / for the read command.
LIST=( $(IFS=/; for x in ${AIRLINE_OPTION#-A }; do printf "$x "; done) )
This is a portable solution, but if your read supports -a and you don't mind portability then you should go for #1_CR's solution.
With awk, for example, you can create an array and store it in LIST variable:
$ LIST=($(awk -F"[\/ ]" '{print $2,$3}' <<< "-A AA/BB"))
Result:
$ echo ${LIST[0]}
AA
$ echo ${LIST[1]}
BB
Explanation
-F"[\/ ]" defines two possible field separators: a space or a slash /.
'{print $2$3}' prints the 2nd and 3rd fields based on those separators.
I do some research on how to replace text between delimiters, but because of lack knowledge in awk and sed I couldn't adjust command for my problem. The most similar question I found here, but after adjusting command to awk '/^(name=|&)/{f=f?0:1}f&&/*/{$0="//" $0}1' file it didn't work. Also, I would like do replace using variable instead of doing replace in file. And if I didn't ask to much, very short explanation would be great :)
I have next url in variable $url and variable $new=unnamed384:
http://www.example.com/?name=unnamed293&file=4
I need to replace text between "name=" and "&" with variable $new.
E.g. This is variable $url before:
http://www.example.com/?name=unnamed293&file=4
This is variable $url after:
http://www.example.com/?name=unnamed384&file=4
How about:
$ new="unnamed384"
$ url="http://www.example.com/?name=unnamed293&file=4"
$ sed "s/name=[^&]*/name=$new/" <<< $url
http://www.example.com/?name=unnamed384&file=4
s/(.*\?name=)[^\&]*(&.*)/$1$n$2/g
The above will do.
tested below(used with perl)
> echo "http://www.example.com/?name=unnamed293&file=4"
http://www.example.com/?name=unnamed293&file=4
> echo "http://www.example.com/?name=unnamed293&file=4" | perl -lne '$n="unamed394";$_=~s/(.*\?name=)[^\&]*(&.*)/$1$n$2/g;print'
http://www.example.com/?name=unamed394&file=4
>
$new='unnamed384';
$echo 'http://www.example.com/?name=unnamed293&file=4' | awk -F'[=&]' '{ print $1"=""'"$new"'""&"$3"="$4 }'
http://www.example.com/?name=unnamed384&file=4
I am trying to create a script to convert two columns in a .csv file which are date and time into unix timestamps. So i need to get the date and time column from each row, convert it and insert it into an additional column at the end containing the timestamp.
Could anyone help me? So far i have discovered the unix command to convert any give time and date to unixstamp:
date -d "2011/11/25 10:00:00" "+%s"
1322215200
I have no experience with bash scripting could anyone get me started?
Examples of my columns and rows:
Columns: Date, Time,
Row 1: 25/10/2011, 10:54:36,
Row 2: 25/10/2011, 11:15:17,
Row 3: 26/10/2011, 01:04:39,
Thanks so much in advance!
You don't provide an exerpt from your csv-file, so I'm using this one:
[foo.csv]
2011/11/25;12:00:00
2010/11/25;13:00:00
2009/11/25;19:00:00
Here's one way to solve your problem:
$ cat foo.csv | while read line ; do echo $line\;$(date -d "${line//;/ }" "+%s") ; done
2011/11/25;12:00:00;1322218800
2010/11/25;13:00:00;1290686400
2009/11/25;19:00:00;1259172000
(EDIT: Removed an uneccessary variable.)
(EDIT2: Altered the date command so the script actually works.)
this should do the job:
awk 'BEGIN{FS=OFS=", "}{t=$1" "$2; "date -d \""t"\" +%s"|getline d; print $1,$2,d}' yourCSV.csv
note
you didn't give any example. and you mentioned csv, so I assume that the column separator in your file should be "comma".
test
kent$ echo "2011/11/25, 10:00:00"|awk 'BEGIN{FS=OFS=", "}{t=$1" "$2; "date -d \""t"\" +%s"|getline d; print $1,$2,d}'
2011/11/25, 10:00:00, 1322211600
Now two imporvements:
First: No need for cat foo.csv, just stream that via < foo.csv into the while loop.
Second: No need for echo & tr to create the date stringformat. Just use bash internal pattern and substitute and do it inplace
while read line ; do echo ${line}\;$(date -d "${line//;/ }" +'%s'); done < foo.csv