Bash script to read a file and add the contents - bash

Question

I have the following contents in the file, and I want to filter Executor Deserialize Time and add all the values to get the final result. How can I do that?
{"Event":"SparkListenerTaskEnd","Stage ID":0,"Stage Attempt ID":0,"Task Type":"ShuffleMapTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task ID":29,"Index":29,"Attempt":0,"Launch Time":1453927221831,"Executor ID":"1","Host":"172.17.0.226","Locality":"ANY","Speculative":false,"Getting Result Time":0,"Finish Time":1453927230401,"Failed":false,"Accumulables":[]},"Task Metrics":{"Host Name":"172.17.0.226","Executor Deserialize Time":9,"Executor Run Time":8550,"Result Size":2258,"JVM GC Time":18,"Result Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle Write Metrics":{"Shuffle Bytes Written":0,"Shuffle Write Time":4425,"Shuffle Records Written":0},"Input Metrics":{"Data Read Method":"Hadoop","Bytes Read":134283264,"Records Read":100890}}}
{"Event":"SparkListenerTaskEnd","Stage ID":0,"Stage Attempt ID":0,"Task Type":"ShuffleMapTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task ID":30,"Index":30,"Attempt":0,"Launch Time":1453927222232,"Executor ID":"1","Host":"172.17.0.226","Locality":"ANY","Speculative":false,"Getting Result Time":0,"Finish Time":1453927230493,"Failed":false,"Accumulables":[]},"Task Metrics":{"Host Name":"172.17.0.226","Executor Deserialize Time":7,"Executor Run Time":8244,"Result Size":2258,"JVM GC Time":16,"Result Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle Write Metrics":{"Shuffle Bytes Written":0,"Shuffle Write Time":4190,"Shuffle Records Written":0},"Input Metrics":{"Data Read Method":"Hadoop","Bytes Read":134283264,"Records Read":100886}}}
{"Event":"SparkListenerTaskEnd","Stage ID":0,"Stage Attempt ID":0,"Task Type":"ShuffleMapTask","Task End Reason":{"Reason":"Success"},"Task Info":{"Task ID":31,"Index":31,"Attempt":0,"Launch Time":1453927222796,"Executor ID":"1","Host":"172.17.0.226","Locality":"ANY","Speculative":false,"Getting Result Time":0,"Finish Time":1453927230638,"Failed":false,"Accumulables":[]},"Task Metrics":{"Host Name":"172.17.0.226","Executor Deserialize Time":5,"Executor Run Time":7826,"Result Size":2258,"JVM GC Time":18,"Result Serialization Time":0,"Memory Bytes Spilled":0,"Disk Bytes Spilled":0,"Shuffle Write Metrics":{"Shuffle Bytes Written":0,"Shuffle Write Time":3958,"Shuffle Records Written":0},"Input Metrics":{"Data Read Method":"Hadoop","Bytes Read":134283264,"Records Read":101004}}}

Answer 1

grep -P -o "Executor Deserialize Time.:[0-9]+" file.txt |
cut -d: -f2 | awk '{ sum+=$1} END {print sum}'
grep that bit the part of each line with the field you want.
Split it to just grab the number.
Use awk to sum up all the values

Answer 2

awk -v RS=, '/^"Executor Deserialize Time":/ {split($0,a,":"); tot+=a[2]} END{print tot}' file
Set RS (the record separator) to ,.
Match records that match the desired field name.
Split the current record on :.
Add the second split field to our total.
Print the total at the END.
Or same idea but set FS (the field separator) instead
awk -F , '{for (i=1;i<=NF;i++) {if ($i ~ /^"Executor Deserialize Time":/) {split($i,a,":"); tot+=a[2]}}} END{print tot}' file
Set FS to ,.
Loop over every field from 1 to NF.
Match the desired fields.
Split the current record on :.
Add the second split field to our total.
Print the total at the END.
If you only want this for a given value of Stage ID then you could use this:
awk -v stage=0 -F , '{
ds=0; val=0
for (i=1;i<=NF;i++) {
split($i,a,":")
if (a[1] == "\"Executor Deserialize Time\"") {
val=a[2]
}
if ((a[1] == "\"Stage ID\"") && (a[2] == stage)) {
ds++
}
if (ds && val) {
tot+=val
next
}
}
}
END{print tot}' file
Which tracks whether we've seen the both necessary values for each line and only sums when we have. It uses the stage variable to do this so you can control this from outside the awk script (the -v stage=0 argument).