I have a file with the following content:
{
"user_id1": "171295",
"timeStamp": "2017-03-06 19:16:58.000"
},,
{
"user_id1": "149821",
"timeStamp": "2017-03-08 12:50:47.000"
},,
{
"user_id1": "184767",
"timeStamp": "2017-03-08 19:55:25.000"
},,
{
"user_id1": "146364",
"timeStamp": "2017-03-12 23:48:48.000"
},
]
I want to replace all instances of },, with }, in bash using sed how do I do this?
This is one of the many ways you can do:
sed 's/},,$/},/g' yourfile.txt
$ is an assurance that it's matching end of line's commas. -i option allows you to edit file in place.
sed -i 's/},,$/},/g' yourfile.txt
sed 's/},,/},/g' < in > out
Related
How to search for a word, once it's found, in the next line save a specific value in a variable.
The json bellow is only a small part of the file.
Due to this specific file json structure be inconsistent and subject to change overtime, it need to by done via search like grep sed awk.
however the paramenters bellow will be always the same.
search for the word next
get the next line bellow it
extract everything after the word page_token not the boundary "
store in a variable to be used
test.txt:
"link": [
{
"relation": "search",
"url": "aaa/ww/rrrrrrrrr/aaaaaaaaa/ffffffff/ccccccc/dddd/?token=gggggggg3444"
},
{
"relation": "next",
"url": "aaa/ww/rrrrrrrrr/aaaaaaaaa/ffffffff/ccccccc/dddd/?&_page_token=121_%_#212absa23bababa121212121212121"
},
]
so the desired output in this case is:
PAGE_TOKEN="121_%_#212absa23bababa121212121212121"
my attempt:
PAGE_TOKEN=$(cat test.txt| grep "next" | sed 's/^.*: *//;q')
no lucky..
This might work for you (GNU sed):
sed -En '/next/{n;s/.*(page_token=)([^"]*).*/\U\1\E"\2"/p}' file
This is essentially a filtering operation, hence the use of the -n option.
Find a line containing next, fetch the next line, format as required and print the result.
Presuming your input is valid json, one option is to use:
cat test.json
[{
"relation": "search",
"url": "aaa/ww/rrrrrrrrr/aaaaaaaaa/ffffffff/ccccccc/dddd/?token=gggggggg3444"
},
{
"relation": "next",
"url": "aaa/ww/rrrrrrrrr/aaaaaaaaa/ffffffff/ccccccc/dddd/?&_page_token=121_%_#212absa23bababa121212121212121"
}
]
PAGE_TOKEN=$(cat test.json | jq -r '.[] | select(.relation=="next") | .url | gsub(".*=";"")')
echo "$PAGE_TOKEN"
121_%_#212absa23bababa121212121212121
My json file looks like this:
"parameters": {
"$connections": {
"value": {
"azureblob": {
"connectionId": "/subscriptions/2b06d50xxxxxedd021/resourceGroups/Reource1005/providers/Microsoft.Web/connections/azureblob",
"connectionName": "azureblob",
"connectionProperties": {
"authentication": {
"type": "ManagedServiceIdentity"
}
},
"id": "/subscriptions/2b06d502-3axxxxxxedd021/providers/Microsoft.Web/locations/eastasia/managedApis/azureblob"
},
"office365": {
"connectionId": "/subscriptions/2b06d502xxxxxc8-5a8939edd021/resourceGroups/Reource1005/providers/Microsoft.Web/connections/office365",
"connectionName": "office365",
"id": "/subscriptions/2b06d50xxxxxx939edd021/providers/Microsoft.Web/locations/eastasia/managedApis/office365"
}
}
}
}
}
I want to use sed command to replace the string in connectionId, currently my script is as follows:
script: 'sed -e ''/connectionId/c\ \"connectionId\" : \"/subscriptions/2b06d50xxxxb-92c8-5a8939edd021/resourceGroups/Reourcetest/providers/Microsoft.Web/connections/azureblob\",'' "$(System.DefaultWorkingDirectory)/function-app-actions/templates/copycode.json"'
This script can replace the strings in both connectionIds in the json file with "Resourcetest", that's what I want to make the strings in the second connectionId replace with other values, how can I do that?
I'm new to sed commands, any insight is appreciated。
Edit:
I just want to replace "Resource1005" in both connectionId strings in the json file with "Resourcetest", but I need other content in the connectionIds string to keep the previous value
So my expected output should look like this:
"connectionId": "/subscriptions/2b06d502-3axxxx8939edd021/resourceGroups/Reourcetest/providers/Microsoft.Web/connections/azureblob"
"connectionId": "/subscriptions/2b06d502-3axxxx8939edd021/resourceGroups/Reourcetest/providers/Microsoft.Web/connections/office365"
If I use the script I mentioned above, it does replace the two Resource1005s, but the other values in the string are also replaced with the same (I just want to replace the Resource1005 value)
1st solution: With your shown samples and attempts, please try following GNU awk code. This will print only edited lines(as per shown samples) in output with substituting Resource1005 with Resourcetest in values.
awk -v RS='[[:space:]]+"connectionId": "[^"]*' '
RT{
sub(/\n+[[:space:]]+/,"",RT)
sub(/\/Resource1005\//,"/Resourcetest/",RT)
print RT
}
' Input_file
2nd solution: With sed you can try following sed code.
sed -nE 's/(^[[:space:]]+"connectionId": ".*)\/Resource1005\/(.*)/\1\/Resourcetest\/\2/p' Input_file
Common practice is to create template files and change them with sed or something else. Like this for example:
cat template.json
...
"office365": {
"connectionId": "__CONNECTIONID__",
"connectionName": "office365",
"id": "__ID__"
}
...
sed 's|__CONNECTIONID__|/some/path|; s|__ID__|/some/other/path|' template.json > new.json
Suppose I have some text file (json in this case):
{
"data": [
{
"timestamp": 1577856103107
},
{
"timestamp": 1577869991302
}
]
}
And I want to replace a pattern (in this case a UNIX millisecond timestamp) with a more readable date format.
I'm trying with this:
$ sed -E 's/(.*)([0-9]{13})/echo "\1\\"$(date --date="#$((\2\/1000))" --iso-8601=seconds)\\""/e' example.json
{
"data": [
{
timestamp: "2020-01-01T00:21:43-05:00"
},
{
timestamp: "2020-01-01T04:13:11-05:00"
}
]
}
This is somewhat ok, but I don't understand why the quotes arround timestamp get lost.
This command works:
sed -E 's/(.*)"(timestamp)"(: )([0-9]{13})/echo "\1\\"\2\\"\3\\"$(date --date="#$((\4\/1000))" --iso-8601=seconds)"\\"/e' example.json
{
"data": [
{
"timestamp": "2020-01-01T00:21:43-05:00"
},
{
"timestamp": "2020-01-01T04:13:11-05:00"
}
]
}
I also don't understand why I need double backslashes \\ to ouput a double-quote " in the right side of this sed command.
Is there a better way (or tool) to solve this?
I'm on sed (GNU sed) 4.8 and zsh 5.8 (x86_64-pc-linux-gnu), thanks!
Is there a better way (or tool) to solve this?
Using sed for manipulating json things is very crude. You can't parse json with regex. I (strongly) suggest to use json-aware tools, like jq.
jq '.data[].timestamp |= (. / 1000 | strftime("%Y-%m-%dT%H:%M:%SZ"))'
but I don't understand why the quotes arround timestamp get lost.
The:
echo "\1\\"$(date --date="#$((\2\/1000))" --iso-8601=seconds)\\""
is "substituted" to:
echo ""timestamp": \"$(date --date="#$((1577856103107/1000))" --iso-8601=seconds)\""
^^
^------------------------------------------------------------------^
Then is passed to shell and quotes are re-evalulated according to shell rules.
Matching (.*) is really doing nothing, just match only the part you want to substitute. You could instead match only the part you want to substitute:
sed '/"timestamp":/s/([0-9]{13})/echo "\\"$(date --date="#$((\1\/1000))" --iso-8601=seconds)\\""/e'
why I need double backslashes \ to ouput a double-quote " in the right side of this sed command.
First \\ is interpreted by sed into single \.
$ echo a | sed 's/.*/single slash: \\/'
single slash: \
Then the result of sed command is passed to shell where all shell parsing rules are done one again.
I have this text file:
{
"name": "",
"auth": true,
"username": "rtorrent",
"password": "d5275b68305438499f9660b38980d6cef7ea97001efe873328de1d76838bc5bd15c99df8b432ba6fdcacbff82e3f3c4829d34589cf43236468d0d0b0a3500c1e"
}
Now, I want to be able to replace the d5275b68305438499f9660b38980d6cef7ea97001efe873328de1d76838bc5bd15c99df8b432ba6fdcacbff82e3f3c4829d34589cf43236468d0d0b0a3500c1e using sed for example. (The string has always the exact same length, but the values can be different)
I've tried this using sed:
sed -i 5s/./new-string/18 file.json
That basically replaces text, on the 5th line, starting with position 18. I want to be able to replace the text, exactly starting with position 18 and up to position 154, strictly what's inside the "". The command above will cut the ", at the end of the file and if it's run multiple times, the string becomes every time longer and longer.
Any help is really appreciated.
You can use for example awk for it:
$ awk -v var="new_string" 'NR==5{print substr($0,1,17) var substr($0,146);next}1' file
{
"name": "",
"auth": true,
"username": "rtorrent",
"password": "new_string"
}
but there are better tools for changing a value in a JSON, jq for example:
$ jq '.password="new_string"' file
{
"name": "",
"auth": true,
"username": "rtorrent",
"password": "new_string"
}
Edit: When passing a shell variable $var to awk and jq:
$ var="new_string"
$ awk -v var="$var" 'NR==5{print substr($0,1,17) var substr($0,146);next}1' file
and
$ jq --arg var "$var" '.password=$var'
Edit2: There is always sed:
$ sed -i "5s/\"[^\"]*\"/\"$var\"/2" file
I'm trying to parse a file with single quotes, and want to change it to double quotes.
Sample data :
{'data': ['my',
'my_other',
'my_other',
'my_other',
'This could 'happen' <- and this ones i want to keep',
],
'name': 'MyName'},
'data2': {'type': 'x86_64',
'name': 'This',
'type': 'That',
'others': 'bla bla 'bla' <- keep this ones too',
'version': '21237821972'}}}
Desired output :
{"data": ["my",
"my_other",
"my_other",
"my_other",
"This could 'happen' <- and this ones i want to keep"
],
"name": "MyName"},
"data2": {"type": "x86_64",
"name": "This",
"type": "That",
"others": "bla bla 'bla' <- keep this ones too",
"version": "21237821972"}}}
I've already tried to do some regex with sed, but unlucky.
I understand why this is not working for me, just don't know how to go further to get data as i want.
sed -E -e "s/( |\{|\[)'(.*)'(\:|,|\]|\})/\1\"\2\"\3/g"
Cheers,
I am no expert in jq so as per OP's question trying to answer in awk to substitute ' to " here.
awk -v s1="\"" '!/This could/ && !/others/{gsub(/\047/,s1) } /This could/ || /others/{gsub(/\047/,s1,$1)} 1' Input_file
Output will be as follows.
{"data": ["my",
"my_other",
"my_other",
"my_other",
"This could 'happen' <- and this ones i want to keep',
],
"name": "MyName"},
"data2": {"type": "x86_64",
"name": "This",
"type": "That",
"others": 'bla bla 'bla' <- keep this ones too',
"version": "21237821972"}}}
We know that ‘sed’ command can search for a pattern and can replace that pattern with user provided new one
For example sed “s/pattern1/pattern2/g” filename.txt
Now the ‘sed’ command will search for pattern1 and if found it will replace with pattern2
For your requirement you just need to apply this rule. See below
First
sed "s/^\'/\"/g” yourfile
This will search for every newline with character ‘ in the file and replace with “
Next requirement is to search for pattern ‘: and replace with “:
So add one more condition to it separated by ;
sed "s/^\'/\"/g; s/\':/\":/g” yourfile
Just follow this algorithm till you reach you requirement
The final should be look like:-
sed "s/^\'/\"/g; s/\':/\":/g;s/{\'/{\"/g;s/\[\'/\[\"/g;s/\',/\",/g;s/\'}/\"}/g;s/: \'/: \"/g;" yourfile > newfil
(If the above command gives you error just use the command at the very beginning)
finally
mv newfile yourfile