Apply set of operation on one entire column in unix file - bash

Question

I have a file looks like :
"JOB1 #43", 43 "SUCCESS", 1479079800029
"JOB1 #42", 42 "SUCCESS", 1478993400042
"JOB1 #41", 41 "SUCCESS", 1478907000065
"JOB1 #40", 40 "SUCCESS", 1478820600085
"JOB1 #39", 39 "SUCCESS", 1478734200051
need to change last column (timestamp) with date and time format.
Tried so far :
for i in `cat file | awk '{print $NF}'`
do
date -d #$( echo "(${i} + 500) / 1000" | bc)
done
Output:
Sun Nov 13 23:30:00 GMT 2016
Sat Nov 12 23:30:00 GMT 2016
Fri Nov 11 23:30:00 GMT 2016
Thu Nov 10 23:30:00 GMT 2016
Wed Nov 9 23:30:00 GMT 2016
Expected output is
"JOB1 #43", 43 "SUCCESS", Sun Nov 13 23:30:00 GMT 2016
"JOB1 #42", 42 "SUCCESS", Sat Nov 12 23:30:00 GMT 2016
"JOB1 #41", 41 "SUCCESS", Fri Nov 11 23:30:00 GMT 2016
"JOB1 #40", 40 "SUCCESS", Thu Nov 10 23:30:00 GMT 2016
"JOB1 #39", 39 "SUCCESS", Wed Nov 9 23:30:00 GMT 2016
Can we change it in file itself?
Any help would be appreciated.

Answer 1

Try:
awk -F',\\s*' '{print $1 "," $2 "," strftime("%c", $3/1000)}' file
Outputs:
"JOB1 #43",43 "SUCCESS",Sun 13 Nov 2016 23:30:00 GMT
"JOB1 #42",42 "SUCCESS",Sat 12 Nov 2016 23:30:00 GMT
"JOB1 #41",41 "SUCCESS",Fri 11 Nov 2016 23:30:00 GMT
"JOB1 #40",40 "SUCCESS",Thu 10 Nov 2016 23:30:00 GMT
"JOB1 #39",39 "SUCCESS",Wed 09 Nov 2016 23:30:00 GMT
You may like to adjust the date format as necessary.

Answer 2

You can use this awk command (will work with non-gnu awk as well):
awk -F , '{cmd="TZ=UTC date -d #" sprintf("%d", ($NF+500)/1000);
cmd | getline dt; gsub(/[0-9]+$/, dt); close(cmd)} 1' file
"JOB1 #43", 43 "SUCCESS", Sun Nov 13 23:30:00 UTC 2016
"JOB1 #42", 42 "SUCCESS", Sat Nov 12 23:30:00 UTC 2016
"JOB1 #41", 41 "SUCCESS", Fri Nov 11 23:30:00 UTC 2016
"JOB1 #40", 40 "SUCCESS", Thu Nov 10 23:30:00 UTC 2016
"JOB1 #39", 39 "SUCCESS", Wed Nov 9 23:30:00 UTC 2016
We use getline function to call your desired date command and store output in variable dt
Finally wee use gsub instead of assignment i.e. $NF = dt to maintain formatting of input record in output

Answer 3

As you already got the content to replace with, the merging can easily be done using python script.
f1=open("a.txt")
f2=open("b.txt")
lines1 = f1.readlines()
lines2 = f2.readlines()
output_lines=[]
for (l1,l2) in zip(lines1,lines2):
fields = l1.split(",")
fields[2] = l2
output_line = ",".join(fields)
output_lines.append(output_line)
f3 = open("result.txt","w")
f3.writelines(output_lines)
f3.close()
output : result.txt
"JOB1 #43", 43 "SUCCESS",Sun Nov 13 23:30:00 GMT 2016
"JOB1 #42", 42 "SUCCESS",Sat Nov 12 23:30:00 GMT 2016
"JOB1 #41", 41 "SUCCESS",Fri Nov 11 23:30:00 GMT 2016
"JOB1 #40", 40 "SUCCESS",Thu Nov 10 23:30:00 GMT 2016
"JOB1 #39", 39 "SUCCESS",Wed Nov 9 23:30:00 GMT 2016