Read JSON objects from a file and pass each of them to cURL - shell

Question

I have a file access.log containing timestamps and JSON objects delimited by a newline:
09:52:11 { "key1": "value", "key2": 2, "key3": true }\n
09:52:13 { "key4": "value2", "key5": 5, "key6": false }\n
09:55:33 { "key7": "value7", "key8": 8, "key9": true }\n
...
I need to read the file line by line, extract JSON objects and pass each of them as a request body to cURL. I tried to solve the problem with xargs but it passes {} to the request body:
cat access.log | cut -c10- | xargs -0 -I {} curl -X POST-H "Content-Type: application/json" http://localhost:8080/api/ -d '{}'
cat access.log | cut -c10- | xargs -0 -I {} curl -X POST-H "Content-Type: application/json" http://localhost:8080/api/ -d {}
I have a mistake somewhere in the syntax. What am I doing wrong?
UPDATE: \n is not a literal value. So by issuing cat access.log the result is:
09:52:11 { "key1": "value", "key2": 2, "key3": true }
09:52:13 { "key4": "value2", "key5": 5, "key6": false }
09:55:33 { "key7": "value7", "key8": 8, "key9": true }
...

Answer 1

You shouldn't pass -0 to xargs - it makes it expect zero character as a separator. Since you don't have zero character it treats all input as one record and passes it all to curl. Removing the -0 fixes it.
In addition, -I option makes xarg to use newlines as a delimiter, instead of spaces.
It is easier to see with echo i/o curl. Compare output from
cat aa.log | cut -c10- | xargs -0 -I {} echo %%'{}'%% which is
%%{ "key1": "value", "key2": 2, "key3": true }
{ "key4": "value2", "key5": 5, "key6": false }
{ "key7": "value7", "key8": 8, "key9": true }
%%
To output from cat aa.log | cut -c10- | xargs -I {} echo %%'{}'%% which is
%%{ key1: value, key2: 2, key3: true }%%
%%{ key4: value2, key5: 5, key6: false }%%
%%{ key7: value7, key8: 8, key9: true }%%
P.S. Also, you can use -t option to see the command that xargs builds

Answer 2

use:
#!/bin/bash
cat access.log | sed 's/\\n//g' > access_temp.log
echo ""
while IFS='' read -r line || [[ -n "$line" ]]; do
json_string=$(echo "$line" | awk '{$1=""; print $0}' | jq -c .)
curl -X POST -H "Content-Type: application/json" -d "$json_string" http://localhost:8080/api/
done < "access_temp.log"
rm access_temp.log
The sed in the beginning was because the '\n' was messing up the jq command, so I had to remove it.
You can add echos, or set -x in the beginning to check the values being passed.

Answer 3

i think this should work - wrap xargs with a light-weight shell such as dash :
printf '%s' '
09:52:11 { "key1": "value", "key2": 2, "key3": true }\n
09:52:13 { "key4": "value2", "key5": 5, "key6": false }\n
09:55:33 { "key7": "value7", "key8": 8, "key9": true }\n' |
mawk NF=NF FS='^[^{]*' OFS= ORS='\0' |
xargs -0 dash -v -x -c '
for __; do
curl -s -X POST-H '\''Content-Type: application/json'\'' \
--url '\''https://localhost:8080/api'\'' -d "$__"
done ' _
+ curl -s -X POST-H Content-Type: application/json --url
https://localhost:8080/api -d { "key1": "value", "key2": 2, "key3": true }
+ curl -s -X POST-H Content-Type: application/json --url
https://localhost:8080/api -d { "key4": "value2", "key5": 5, "key6": false }
+ curl -s -X POST-H Content-Type: application/json --url
https://localhost:8080/api -d { "key7": "value7", "key8": 8, "key9": true }
The -v(erbose) -x(trace) flags for dash are totally optional, and in this situation they could be a rough proxy for xargs -t.