sed replace every word with single quotes with double quotes - bash

I'm trying to parse a file with single quotes, and want to change it to double quotes.
Sample data :
{'data': ['my',
'my_other',
'my_other',
'my_other',
'This could 'happen' <- and this ones i want to keep',
],
'name': 'MyName'},
'data2': {'type': 'x86_64',
'name': 'This',
'type': 'That',
'others': 'bla bla 'bla' <- keep this ones too',
'version': '21237821972'}}}
Desired output :
{"data": ["my",
"my_other",
"my_other",
"my_other",
"This could 'happen' <- and this ones i want to keep"
],
"name": "MyName"},
"data2": {"type": "x86_64",
"name": "This",
"type": "That",
"others": "bla bla 'bla' <- keep this ones too",
"version": "21237821972"}}}
I've already tried to do some regex with sed, but unlucky.
I understand why this is not working for me, just don't know how to go further to get data as i want.
sed -E -e "s/( |\{|\[)'(.*)'(\:|,|\]|\})/\1\"\2\"\3/g"
Cheers,

I am no expert in jq so as per OP's question trying to answer in awk to substitute ' to " here.
awk -v s1="\"" '!/This could/ && !/others/{gsub(/\047/,s1) } /This could/ || /others/{gsub(/\047/,s1,$1)} 1' Input_file
Output will be as follows.
{"data": ["my",
"my_other",
"my_other",
"my_other",
"This could 'happen' <- and this ones i want to keep',
],
"name": "MyName"},
"data2": {"type": "x86_64",
"name": "This",
"type": "That",
"others": 'bla bla 'bla' <- keep this ones too',
"version": "21237821972"}}}

We know that ‘sed’ command can search for a pattern and can replace that pattern with user provided new one
For example sed “s/pattern1/pattern2/g” filename.txt
Now the ‘sed’ command will search for pattern1 and if found it will replace with pattern2
For your requirement you just need to apply this rule. See below
First
sed "s/^\'/\"/g” yourfile
This will search for every newline with character ‘ in the file and replace with “
Next requirement is to search for pattern ‘: and replace with “:
So add one more condition to it separated by ;
sed "s/^\'/\"/g; s/\':/\":/g” yourfile
Just follow this algorithm till you reach you requirement
The final should be look like:-
sed "s/^\'/\"/g; s/\':/\":/g;s/{\'/{\"/g;s/\[\'/\[\"/g;s/\',/\",/g;s/\'}/\"}/g;s/: \'/: \"/g;" yourfile > newfil
(If the above command gives you error just use the command at the very beginning)
finally
mv newfile yourfile

Related

How to extract a value by searching for two words in different lines and getting the value of second one

How to search for a word, once it's found, in the next line save a specific value in a variable.
The json bellow is only a small part of the file.
Due to this specific file json structure be inconsistent and subject to change overtime, it need to by done via search like grep sed awk.
however the paramenters bellow will be always the same.
search for the word next
get the next line bellow it
extract everything after the word page_token not the boundary "
store in a variable to be used
test.txt:
"link": [
{
"relation": "search",
"url": "aaa/ww/rrrrrrrrr/aaaaaaaaa/ffffffff/ccccccc/dddd/?token=gggggggg3444"
},
{
"relation": "next",
"url": "aaa/ww/rrrrrrrrr/aaaaaaaaa/ffffffff/ccccccc/dddd/?&_page_token=121_%_#212absa23bababa121212121212121"
},
]
so the desired output in this case is:
PAGE_TOKEN="121_%_#212absa23bababa121212121212121"
my attempt:
PAGE_TOKEN=$(cat test.txt| grep "next" | sed 's/^.*: *//;q')
no lucky..
This might work for you (GNU sed):
sed -En '/next/{n;s/.*(page_token=)([^"]*).*/\U\1\E"\2"/p}' file
This is essentially a filtering operation, hence the use of the -n option.
Find a line containing next, fetch the next line, format as required and print the result.
Presuming your input is valid json, one option is to use:
cat test.json
[{
"relation": "search",
"url": "aaa/ww/rrrrrrrrr/aaaaaaaaa/ffffffff/ccccccc/dddd/?token=gggggggg3444"
},
{
"relation": "next",
"url": "aaa/ww/rrrrrrrrr/aaaaaaaaa/ffffffff/ccccccc/dddd/?&_page_token=121_%_#212absa23bababa121212121212121"
}
]
PAGE_TOKEN=$(cat test.json | jq -r '.[] | select(.relation=="next") | .url | gsub(".*=";"")')
echo "$PAGE_TOKEN"
121_%_#212absa23bababa121212121212121

How to insert a line after a match using sed (or awk) in a formatted JSON file?

{
"id": "a1234567-89ab-cdef-0123-456789abcdef",
"properties": {
...
"my_id": "c1234567-89ab-cdef-0123-456789abcdef",
...
}
Given the above in a file, I want to be able to perform a match (including the 4 leading spaces) on my_id and then append a new line "my_value": "abcd",. The desired output would look like this:
{
"id": "a1234567-89ab-cdef-0123-456789abcdef",
"properties": {
...
"my_id": "c1234567-89ab-cdef-0123-456789abcdef",
"my_value": "abcd",
...
}
Using examples online, I'm unable to get the command to work. Here is an example of something I have tried: sed '/.*"my_id".*/a "my_value": "abcd",' test.json, for which I receive the following error: command a expects \ followed by text.
What is the correct way to structure this command?
Using any awk:
$ awk -v new='"my_value": "abcd",' '{print} sub(/"my_id":.*/,""){print $0 new}' file
{
"id": "a1234567-89ab-cdef-0123-456789abcdef",
"properties": {
...
"my_id": "c1234567-89ab-cdef-0123-456789abcdef",
"my_value": "abcd",
...
}
The above will print the new line using whatever indent the existing "my_id" line has, it doesn't assume/hard-code any indent, e.g. 4 blanks.
I'm using this:
sub(/"my_id":.*/,""){print $0 new}
instead of the briefer:
sub(/"my_id":.*/,new)
so it won't break if the new string contains any backreference chars such as &.
awk procedure with passed argument for insert value
The following awk procedure allows for 'abcd' to be passed as an argument for insertion (allowing it to be set in a bash script if required).
awk -v insertVal="abcd" '/"my_id":/{$0=$0"\n \"my_value\": \""insertVal"\","} {print}' dat.txt
explanation
The required insertion string ('abcd' in this case) is passed as an argument using the -v variable switch followed by a variable name and value: insertVal="abcd".
The first awk action block has a pattern condition to only act on lines containing the target-line string (in this case "my_id":). When a line with that pattern is found, the line is extended with a new line mark \n, the required four spaces to start the next line, the specified key named "my_value", and the value associated with the key, passed by argument as the variable named insertVal ("abcd"), and the final , character. Note the need to escape the " quotes to render them.
The final awk block, prints the current line (whether or not it was modified).
test
The procedure was tested on Mac Terminal using GNU Awk 5.2.0.
The output generated (from the input data saved to a file named dat.txt) is:
{
"id": "a1234567-89ab-cdef-0123-456789abcdef",
"properties": {
...
"my_id": "c1234567-89ab-cdef-0123-456789abcdef",
"my_value": "abcd",
...
}
Using sed
$ sed -e '/my_id/{p;s/id.*"/value": "abcd"/' -e '}' input_file
{
"id": "a1234567-89ab-cdef-0123-456789abcdef",
"properties": {
...
"my_id": "c1234567-89ab-cdef-0123-456789abcdef",
"my_value": "abcd",
...
}
With your shown samples and attempts please try following GNU awk code. Where newVal is an awk variable having new value in it. Using match function in GNU awk where I have used regex (.*)("my_id": "[^"]*",)(.*) which creates 3 capturing groups and saves values into an array named arr. Then printing values as per requirement.
awk -v newVal='"my_value": "abcd",' -v RS= '
match($0,/(.*)("my_id": "[^"]*",)(.*)/,arr){
print arr[1] arr[2] newVal arr[3]
}
' Input_file
This might work for you (GNU sed):
sed '/"my-id".*/p;s//"my-value": "abcd"/' file
Match on "my-id" and print that line, then substitute the additional line.

Sed print the items in square brackets that's after the occurence of a text

I have the following Scenarios:
Scenario 1
foo_bar = ["123", "456", "789"]
Scenario 2
foo_bar = [
"123",
"456",
"789"
]
Scenario 3
variable "foo_bar" {
type = list(string)
default = ["123", "456", "789"]
}
So i'm trying to figure out how I can print with sed the items inside the brackets that are under foo_bar accounting scenario 2 which is a multiline
so the resulting matches here would be
Scenario 1
"123", "456", "789"
Scenario 2
"123",
"456",
"789"
Scenario 3
"123", "456", "789"
In the case of
not_foo_bar = [
"123",
"456",
"789"
]
This should not match, only match foo_bar
This is what I've tried so far
sed -e '1,/foo_bar/d' -e '/]/,$d' test.tf
And this
sed -n 's/.*\foo_bar\(.*\)\].*/\1/p' test.tf
This is a mouthful, but it’s POSIX sed and works.
sed -Ene \
'# scenario 1
s/(([^[:alnum:]_]|^)foo_bar[^[:alnum:]_][[:space:]]*=[[:space:]]*\[)([^]]+)(\]$)/\3/p
# scenario 2 and 3
/([^[:alnum:]_]|^)foo_bar[^[:alnum:]_][[:space:]]*=?[[:space:]]*[[{][[:space:]]*$/,/^[]}]$/ {
//!p
s/(([^[:alnum:]_]|^)default[^[:alnum:]_][[:space:]]*=[[:space:]]*\[)([^]]+)(\]$)/\3/p
}' |
# filter out unwanted lines from scenario 3 ("type =")
sed -n '/^[[:space:]]*"/p'
I couldn’t quite get it all in a single sed.
The first and last lines of the first sed are the same command (using default instead of foobar).
edit: in case it confuses someone, I left in that last [[:space:]]*, in the second really long regex, by mistake. I won’t edit it, but it’s not vital, nor consistent - I didn’t allow for any trailing whitespace in line ends in other patterns.
This might work for you (GNU sed):
sed -En '/foo_bar/{:a;/.*\[([^]]*)\].*/!{N;ba};s//\1/p}' file
Turn off implicit printing and on extended regexp -nE.
Pattern match on foo_bar, then gather up line(s) between the next [ and ] and print the result.

Using AWK to match the first expression, if it exists find next pattern, then insert

I have this following snippet of file, dbScripts/product.js:
db.ShoppingCart.update(
{ 'identifier': 'COFFEE' },
{
'name': 'Coffee',
'identifier': 'COFFEE',
'category': 'Beverages',
'type': [
{
'name': 'Rage',
'identifier': 'COFFEE_RAGE',
'desc':
'Coffee by Rage'
},
{
'name': 'Sleepy Owl',
'identifier': 'COFFEE_SLEEPY_OWL',
'desc':
'Coffee by Sleepy Owl'
}
]
},
{ upsert: true }
);
I wish to first match {'identifier': 'COFFEE'}, if it exists, then and only then look for next subsequent pattern 'type': [
If both conditions are satisfied, the code should insert similar object for other coffee brand. But if the identifier entered is not coffee, it would create the entire structure for it. [The latter part has been accomplished].
FILE_PATH="./dbScripts/products.js"
# example, COFFEE
echo "Enter product"
read -r PRODUCT_NAME
DESCRIPTION="Coffee by Nescafe"
IDENTIFIER="COFFEE_NESCAFE"
FOUND=$(awk -v ITEM="{ \'identifier\': \'${PRODUCT_NAME}\' }" 'BEGIN {FOUND=0}/ITEM/{++FOUND} END {print FOUND+0}' "${FILE_PATH}")
echo ${FOUND}
if [ ${FOUND} -eq 1 ]; then
SCRIPT_TEMPLATE="./AutomationUtil/object.tpl"
local SCRIPT=$(eval "echo \"$(cat "${SCRIPT_TEMPLATE}")\"")
awk -v text="${SCRIPT}" '1;/'type': | 'types': /{printf text}' "${FILE_PATH}" > "${FILE_PATH}_tmp" && mv "${FILE_PATH}_tmp" "${FILE_PATH}"
fi
I am trying to use awk-variable FOUND as flag to check if the pattern is found or not. Proceed with next pattern if and only if FOUND=1
Object.tpl
{'name': '${PRODUCT_NAME}','identifier': '${PRODUCT_IDENTIFIER}','desc':'${DESCRIPTION}'},
So, if both subsequent patterns are found, the above template will be evaluated, and inserted. If not, the Entire Object Structure from dbScripts/product.js will created at the end of the file, example, user enters Tea instead of Coffee.
In the example below, user enters COFFEE as identifier, which matches our first pattern {'identifier': 'COFFEE'}, then it finds the line, 'type': [, and inserts the evaluated Object.tpl below it.
Expected OUTPUT:
db.ShoppingCart.update(
{ 'identifier': 'COFFEE' },
{
'name': 'Coffee',
'identifier': 'COFFEE',
'category': 'Beverages',
'type': [
{
'name': 'Nescafe',
'identifier': 'COFFEE_NESCAFE',
'desc':
'Coffee by Nesafe'
},
{
'name': 'Rage',
'identifier': 'COFFEE_RAGE',
'desc':
'Coffee by Rage'
},
{
'name': 'Sleepy Owl',
'identifier': 'COFFEE_SLEEPY_OWL',
'desc':
'Coffee by Sleepy Owl'
}
]
},
{ upsert: true }
);
db.ShoppingCart.update(
{ 'identifier': 'NOODLES' },
{
'name': 'Noodles',
'identifier': 'NOODLES',
'category': 'Instant Food',
'type': [
{
'name': 'Ramen',
'identifier': 'NOODELS_RAMEN',
'desc':
'Noodles by Ramen'
}
]
}, {upsert: true}
);
I came up with this theoretically, but can't seem to make it work. I always get FOUND=0.
I am sure, this code is very stupid, but any help is appreciated. Again, both patterns should be found in order to insert the object.
NOTE: The formatting is being handled by prettier, so spacing should be ignored in case of new inserted text
I do not wish to use SED because MacOS doesn't come with GNU-SED.
awk version 20200816
There are a few issues with the current code but since the main question appears to be the wrong value assigned to the (bash) variable FOUND, I'll focus on just this issue ...
The main issue is the testing for the search pattern stored in the awk variable ITEM; one fix:
/ITEM/ {...} # wrong
$0 ~ ITEM {...} # fix
Incorporating this into OP's code, and plugging in some values for INPUT and FILE_PATH we get:
FOUND=$(awk -v ITEM="{ 'identifier' : '${INPUT}' }" '$0 ~ ITEM {++FOUND} END {print FOUND+0}' product.js)
$ echo "${FOUND}"
1
NOTES:
escaped single quotes (defining the awk variable ITEM) are not needed
awk variables have an initial value of 0 so no need for the BEGIN{} block
this solution assumes the input has the same exact white space as defined in the awk variable ITEM
As for the rest of the question ...
OP's 2nd awk script is looking for the strings item: or items: but no such string exists in the sample input, so I wouldn't expect anything to be 'inserted'; can't tell at this point if this is a typo/issue with the proposed awk code or faulty sample input
the sample Object.tpl file shows a one-line element definition but the expected output shows this inserted as multiple lines; this result certainly won't be generated by the proposed (2nd) awk script so not sure if OP is also looking for an awk solution that inserst the line feeds or ... ??
the entire FOUND=$(awk ...); if ... awk ...;fi block of code can probably be replaced with a single awk script
OP's awk version (awk version 20200816) looks a bit odd so not sure what flavor/version of awk this really is or what limitations there may be with said version of awk (eg, does OP's awk version support -i inplace for allowing awk to perform an 'inplace' update of the product.js file?)

Text replace in a file, on 5h line, from position 18 to position 145

I have this text file:
{
"name": "",
"auth": true,
"username": "rtorrent",
"password": "d5275b68305438499f9660b38980d6cef7ea97001efe873328de1d76838bc5bd15c99df8b432ba6fdcacbff82e3f3c4829d34589cf43236468d0d0b0a3500c1e"
}
Now, I want to be able to replace the d5275b68305438499f9660b38980d6cef7ea97001efe873328de1d76838bc5bd15c99df8b432ba6fdcacbff82e3f3c4829d34589cf43236468d0d0b0a3500c1e using sed for example. (The string has always the exact same length, but the values can be different)
I've tried this using sed:
sed -i 5s/./new-string/18 file.json
That basically replaces text, on the 5th line, starting with position 18. I want to be able to replace the text, exactly starting with position 18 and up to position 154, strictly what's inside the "". The command above will cut the ", at the end of the file and if it's run multiple times, the string becomes every time longer and longer.
Any help is really appreciated.
You can use for example awk for it:
$ awk -v var="new_string" 'NR==5{print substr($0,1,17) var substr($0,146);next}1' file
{
"name": "",
"auth": true,
"username": "rtorrent",
"password": "new_string"
}
but there are better tools for changing a value in a JSON, jq for example:
$ jq '.password="new_string"' file
{
"name": "",
"auth": true,
"username": "rtorrent",
"password": "new_string"
}
Edit: When passing a shell variable $var to awk and jq:
$ var="new_string"
$ awk -v var="$var" 'NR==5{print substr($0,1,17) var substr($0,146);next}1' file
and
$ jq --arg var "$var" '.password=$var'
Edit2: There is always sed:
$ sed -i "5s/\"[^\"]*\"/\"$var\"/2" file

Resources