Using AWK to match the first expression, if it exists find next pattern, then insert - bash

I have this following snippet of file, dbScripts/product.js:
db.ShoppingCart.update(
{ 'identifier': 'COFFEE' },
{
'name': 'Coffee',
'identifier': 'COFFEE',
'category': 'Beverages',
'type': [
{
'name': 'Rage',
'identifier': 'COFFEE_RAGE',
'desc':
'Coffee by Rage'
},
{
'name': 'Sleepy Owl',
'identifier': 'COFFEE_SLEEPY_OWL',
'desc':
'Coffee by Sleepy Owl'
}
]
},
{ upsert: true }
);
I wish to first match {'identifier': 'COFFEE'}, if it exists, then and only then look for next subsequent pattern 'type': [
If both conditions are satisfied, the code should insert similar object for other coffee brand. But if the identifier entered is not coffee, it would create the entire structure for it. [The latter part has been accomplished].
FILE_PATH="./dbScripts/products.js"
# example, COFFEE
echo "Enter product"
read -r PRODUCT_NAME
DESCRIPTION="Coffee by Nescafe"
IDENTIFIER="COFFEE_NESCAFE"
FOUND=$(awk -v ITEM="{ \'identifier\': \'${PRODUCT_NAME}\' }" 'BEGIN {FOUND=0}/ITEM/{++FOUND} END {print FOUND+0}' "${FILE_PATH}")
echo ${FOUND}
if [ ${FOUND} -eq 1 ]; then
SCRIPT_TEMPLATE="./AutomationUtil/object.tpl"
local SCRIPT=$(eval "echo \"$(cat "${SCRIPT_TEMPLATE}")\"")
awk -v text="${SCRIPT}" '1;/'type': | 'types': /{printf text}' "${FILE_PATH}" > "${FILE_PATH}_tmp" && mv "${FILE_PATH}_tmp" "${FILE_PATH}"
fi
I am trying to use awk-variable FOUND as flag to check if the pattern is found or not. Proceed with next pattern if and only if FOUND=1
Object.tpl
{'name': '${PRODUCT_NAME}','identifier': '${PRODUCT_IDENTIFIER}','desc':'${DESCRIPTION}'},
So, if both subsequent patterns are found, the above template will be evaluated, and inserted. If not, the Entire Object Structure from dbScripts/product.js will created at the end of the file, example, user enters Tea instead of Coffee.
In the example below, user enters COFFEE as identifier, which matches our first pattern {'identifier': 'COFFEE'}, then it finds the line, 'type': [, and inserts the evaluated Object.tpl below it.
Expected OUTPUT:
db.ShoppingCart.update(
{ 'identifier': 'COFFEE' },
{
'name': 'Coffee',
'identifier': 'COFFEE',
'category': 'Beverages',
'type': [
{
'name': 'Nescafe',
'identifier': 'COFFEE_NESCAFE',
'desc':
'Coffee by Nesafe'
},
{
'name': 'Rage',
'identifier': 'COFFEE_RAGE',
'desc':
'Coffee by Rage'
},
{
'name': 'Sleepy Owl',
'identifier': 'COFFEE_SLEEPY_OWL',
'desc':
'Coffee by Sleepy Owl'
}
]
},
{ upsert: true }
);
db.ShoppingCart.update(
{ 'identifier': 'NOODLES' },
{
'name': 'Noodles',
'identifier': 'NOODLES',
'category': 'Instant Food',
'type': [
{
'name': 'Ramen',
'identifier': 'NOODELS_RAMEN',
'desc':
'Noodles by Ramen'
}
]
}, {upsert: true}
);
I came up with this theoretically, but can't seem to make it work. I always get FOUND=0.
I am sure, this code is very stupid, but any help is appreciated. Again, both patterns should be found in order to insert the object.
NOTE: The formatting is being handled by prettier, so spacing should be ignored in case of new inserted text
I do not wish to use SED because MacOS doesn't come with GNU-SED.
awk version 20200816

There are a few issues with the current code but since the main question appears to be the wrong value assigned to the (bash) variable FOUND, I'll focus on just this issue ...
The main issue is the testing for the search pattern stored in the awk variable ITEM; one fix:
/ITEM/ {...} # wrong
$0 ~ ITEM {...} # fix
Incorporating this into OP's code, and plugging in some values for INPUT and FILE_PATH we get:
FOUND=$(awk -v ITEM="{ 'identifier' : '${INPUT}' }" '$0 ~ ITEM {++FOUND} END {print FOUND+0}' product.js)
$ echo "${FOUND}"
1
NOTES:
escaped single quotes (defining the awk variable ITEM) are not needed
awk variables have an initial value of 0 so no need for the BEGIN{} block
this solution assumes the input has the same exact white space as defined in the awk variable ITEM
As for the rest of the question ...
OP's 2nd awk script is looking for the strings item: or items: but no such string exists in the sample input, so I wouldn't expect anything to be 'inserted'; can't tell at this point if this is a typo/issue with the proposed awk code or faulty sample input
the sample Object.tpl file shows a one-line element definition but the expected output shows this inserted as multiple lines; this result certainly won't be generated by the proposed (2nd) awk script so not sure if OP is also looking for an awk solution that inserst the line feeds or ... ??
the entire FOUND=$(awk ...); if ... awk ...;fi block of code can probably be replaced with a single awk script
OP's awk version (awk version 20200816) looks a bit odd so not sure what flavor/version of awk this really is or what limitations there may be with said version of awk (eg, does OP's awk version support -i inplace for allowing awk to perform an 'inplace' update of the product.js file?)

Related

How to extract a value by searching for two words in different lines and getting the value of second one

How to search for a word, once it's found, in the next line save a specific value in a variable.
The json bellow is only a small part of the file.
Due to this specific file json structure be inconsistent and subject to change overtime, it need to by done via search like grep sed awk.
however the paramenters bellow will be always the same.
search for the word next
get the next line bellow it
extract everything after the word page_token not the boundary "
store in a variable to be used
test.txt:
"link": [
{
"relation": "search",
"url": "aaa/ww/rrrrrrrrr/aaaaaaaaa/ffffffff/ccccccc/dddd/?token=gggggggg3444"
},
{
"relation": "next",
"url": "aaa/ww/rrrrrrrrr/aaaaaaaaa/ffffffff/ccccccc/dddd/?&_page_token=121_%_#212absa23bababa121212121212121"
},
]
so the desired output in this case is:
PAGE_TOKEN="121_%_#212absa23bababa121212121212121"
my attempt:
PAGE_TOKEN=$(cat test.txt| grep "next" | sed 's/^.*: *//;q')
no lucky..
This might work for you (GNU sed):
sed -En '/next/{n;s/.*(page_token=)([^"]*).*/\U\1\E"\2"/p}' file
This is essentially a filtering operation, hence the use of the -n option.
Find a line containing next, fetch the next line, format as required and print the result.
Presuming your input is valid json, one option is to use:
cat test.json
[{
"relation": "search",
"url": "aaa/ww/rrrrrrrrr/aaaaaaaaa/ffffffff/ccccccc/dddd/?token=gggggggg3444"
},
{
"relation": "next",
"url": "aaa/ww/rrrrrrrrr/aaaaaaaaa/ffffffff/ccccccc/dddd/?&_page_token=121_%_#212absa23bababa121212121212121"
}
]
PAGE_TOKEN=$(cat test.json | jq -r '.[] | select(.relation=="next") | .url | gsub(".*=";"")')
echo "$PAGE_TOKEN"
121_%_#212absa23bababa121212121212121

How to insert a line after a match using sed (or awk) in a formatted JSON file?

{
"id": "a1234567-89ab-cdef-0123-456789abcdef",
"properties": {
...
"my_id": "c1234567-89ab-cdef-0123-456789abcdef",
...
}
Given the above in a file, I want to be able to perform a match (including the 4 leading spaces) on my_id and then append a new line "my_value": "abcd",. The desired output would look like this:
{
"id": "a1234567-89ab-cdef-0123-456789abcdef",
"properties": {
...
"my_id": "c1234567-89ab-cdef-0123-456789abcdef",
"my_value": "abcd",
...
}
Using examples online, I'm unable to get the command to work. Here is an example of something I have tried: sed '/.*"my_id".*/a "my_value": "abcd",' test.json, for which I receive the following error: command a expects \ followed by text.
What is the correct way to structure this command?
Using any awk:
$ awk -v new='"my_value": "abcd",' '{print} sub(/"my_id":.*/,""){print $0 new}' file
{
"id": "a1234567-89ab-cdef-0123-456789abcdef",
"properties": {
...
"my_id": "c1234567-89ab-cdef-0123-456789abcdef",
"my_value": "abcd",
...
}
The above will print the new line using whatever indent the existing "my_id" line has, it doesn't assume/hard-code any indent, e.g. 4 blanks.
I'm using this:
sub(/"my_id":.*/,""){print $0 new}
instead of the briefer:
sub(/"my_id":.*/,new)
so it won't break if the new string contains any backreference chars such as &.
awk procedure with passed argument for insert value
The following awk procedure allows for 'abcd' to be passed as an argument for insertion (allowing it to be set in a bash script if required).
awk -v insertVal="abcd" '/"my_id":/{$0=$0"\n \"my_value\": \""insertVal"\","} {print}' dat.txt
explanation
The required insertion string ('abcd' in this case) is passed as an argument using the -v variable switch followed by a variable name and value: insertVal="abcd".
The first awk action block has a pattern condition to only act on lines containing the target-line string (in this case "my_id":). When a line with that pattern is found, the line is extended with a new line mark \n, the required four spaces to start the next line, the specified key named "my_value", and the value associated with the key, passed by argument as the variable named insertVal ("abcd"), and the final , character. Note the need to escape the " quotes to render them.
The final awk block, prints the current line (whether or not it was modified).
test
The procedure was tested on Mac Terminal using GNU Awk 5.2.0.
The output generated (from the input data saved to a file named dat.txt) is:
{
"id": "a1234567-89ab-cdef-0123-456789abcdef",
"properties": {
...
"my_id": "c1234567-89ab-cdef-0123-456789abcdef",
"my_value": "abcd",
...
}
Using sed
$ sed -e '/my_id/{p;s/id.*"/value": "abcd"/' -e '}' input_file
{
"id": "a1234567-89ab-cdef-0123-456789abcdef",
"properties": {
...
"my_id": "c1234567-89ab-cdef-0123-456789abcdef",
"my_value": "abcd",
...
}
With your shown samples and attempts please try following GNU awk code. Where newVal is an awk variable having new value in it. Using match function in GNU awk where I have used regex (.*)("my_id": "[^"]*",)(.*) which creates 3 capturing groups and saves values into an array named arr. Then printing values as per requirement.
awk -v newVal='"my_value": "abcd",' -v RS= '
match($0,/(.*)("my_id": "[^"]*",)(.*)/,arr){
print arr[1] arr[2] newVal arr[3]
}
' Input_file
This might work for you (GNU sed):
sed '/"my-id".*/p;s//"my-value": "abcd"/' file
Match on "my-id" and print that line, then substitute the additional line.

How to use sed command to replace value in json file

My json file looks like this:
"parameters": {
"$connections": {
"value": {
"azureblob": {
"connectionId": "/subscriptions/2b06d50xxxxxedd021/resourceGroups/Reource1005/providers/Microsoft.Web/connections/azureblob",
"connectionName": "azureblob",
"connectionProperties": {
"authentication": {
"type": "ManagedServiceIdentity"
}
},
"id": "/subscriptions/2b06d502-3axxxxxxedd021/providers/Microsoft.Web/locations/eastasia/managedApis/azureblob"
},
"office365": {
"connectionId": "/subscriptions/2b06d502xxxxxc8-5a8939edd021/resourceGroups/Reource1005/providers/Microsoft.Web/connections/office365",
"connectionName": "office365",
"id": "/subscriptions/2b06d50xxxxxx939edd021/providers/Microsoft.Web/locations/eastasia/managedApis/office365"
}
}
}
}
}
I want to use sed command to replace the string in connectionId, currently my script is as follows:
script: 'sed -e ''/connectionId/c\ \"connectionId\" : \"/subscriptions/2b06d50xxxxb-92c8-5a8939edd021/resourceGroups/Reourcetest/providers/Microsoft.Web/connections/azureblob\",'' "$(System.DefaultWorkingDirectory)/function-app-actions/templates/copycode.json"'
This script can replace the strings in both connectionIds in the json file with "Resourcetest", that's what I want to make the strings in the second connectionId replace with other values, how can I do that?
I'm new to sed commands, any insight is appreciated。
Edit:
I just want to replace "Resource1005" in both connectionId strings in the json file with "Resourcetest", but I need other content in the connectionIds string to keep the previous value
So my expected output should look like this:
"connectionId": "/subscriptions/2b06d502-3axxxx8939edd021/resourceGroups/Reourcetest/providers/Microsoft.Web/connections/azureblob"
"connectionId": "/subscriptions/2b06d502-3axxxx8939edd021/resourceGroups/Reourcetest/providers/Microsoft.Web/connections/office365"
If I use the script I mentioned above, it does replace the two Resource1005s, but the other values in the string are also replaced with the same (I just want to replace the Resource1005 value)
1st solution: With your shown samples and attempts, please try following GNU awk code. This will print only edited lines(as per shown samples) in output with substituting Resource1005 with Resourcetest in values.
awk -v RS='[[:space:]]+"connectionId": "[^"]*' '
RT{
sub(/\n+[[:space:]]+/,"",RT)
sub(/\/Resource1005\//,"/Resourcetest/",RT)
print RT
}
' Input_file
2nd solution: With sed you can try following sed code.
sed -nE 's/(^[[:space:]]+"connectionId": ".*)\/Resource1005\/(.*)/\1\/Resourcetest\/\2/p' Input_file
Common practice is to create template files and change them with sed or something else. Like this for example:
cat template.json
...
"office365": {
"connectionId": "__CONNECTIONID__",
"connectionName": "office365",
"id": "__ID__"
}
...
sed 's|__CONNECTIONID__|/some/path|; s|__ID__|/some/other/path|' template.json > new.json

Extract json value on regex on bash script

How can i get the values inner depends in bash script?
manifest.py
# Commented lines
{
'category': 'Sales/Subscription',
'depends': [
'sale_subscription',
'sale_timesheet',
],
'auto_install': True,
}
Expected response:
sale_subscription sale_timesheet
The major problem is linebreak, i have already tried | grep depends but i can not get the sale_timesheet value.
Im trying to add this values comming from files into a var, like:
DOWNLOADED_DEPS=($(ls -A $DOWNLOADED_APPS | while read -r file; do cat $DOWNLOADED_APPS/$file/__manifest__.py | [get depends value])
Example updated.
If this is your JSON file:
{
"category": "Sales/Subscription",
"depends": [
"sale_subscription",
"sale_timesheet"
],
"auto_install": true
}
You can get the desired result using jq like this:
jq -r '.depends | join(" ")' YOURFILE.json
This uses .depends to extract the value from the depends field, pipes it to join(" ") to join the array with a single space in between, and uses -r for raw (unquoted) output.
If it is not a json file and only string then you can use below Regex to find the values. If it's json file then you can use other methods like Thomas suggested.
^'depends':\s*(?:\[\s*)(.*?)(?:\])$
demo
you can use egrep for this as follows:
% egrep -M '^\'depends\':\s*(?:\[\s*)(.*?)(?:\])$' pathTo\jsonFile.txt
you can read about grep
As #Thomas has pointed out in a comment, the OPs input data is not in JSON format:
$ cat manifest.py
# Commented lines // comments not allowed in JSON
{
'category': 'Sales/Subscription', // single quotes should be replaced by double quotes
'depends': [
'sale_subscription',
'sale_timesheet', // trailing comma at end of section not allowed
],
'auto_install': True, // trailing comma issue; should be lower case "true"
}
And while the title of the question mentions regex, there is no sign of a regex in the question. I'll leave a regex based solution for someone else to come up with and instead ...
One (quite verbose) awk solution based on the input looking exactly like what's in the question:
$ awk -F"'" ' # use single quote as field separator
/depends/ { printme=1 ; next } # if we see the string "depends" then set printme=1
printme && /]/ { printme=0 ; next} # if printme=1 and line contains a right bracket then set printme=0
printme { printf pfx $2; pfx=" " } # if printme=1 then print a prefix + field #2;
# first time around pfx is undefined;
# subsequent passes will find pfx set to a space;
# since using "printf" with no "\n" in sight, all output will stay on a single line
END { print "" } # add a linefeed on the end of our output
' json.dat
This generates:
sale_subscription sale_timesheet

sed replace every word with single quotes with double quotes

I'm trying to parse a file with single quotes, and want to change it to double quotes.
Sample data :
{'data': ['my',
'my_other',
'my_other',
'my_other',
'This could 'happen' <- and this ones i want to keep',
],
'name': 'MyName'},
'data2': {'type': 'x86_64',
'name': 'This',
'type': 'That',
'others': 'bla bla 'bla' <- keep this ones too',
'version': '21237821972'}}}
Desired output :
{"data": ["my",
"my_other",
"my_other",
"my_other",
"This could 'happen' <- and this ones i want to keep"
],
"name": "MyName"},
"data2": {"type": "x86_64",
"name": "This",
"type": "That",
"others": "bla bla 'bla' <- keep this ones too",
"version": "21237821972"}}}
I've already tried to do some regex with sed, but unlucky.
I understand why this is not working for me, just don't know how to go further to get data as i want.
sed -E -e "s/( |\{|\[)'(.*)'(\:|,|\]|\})/\1\"\2\"\3/g"
Cheers,
I am no expert in jq so as per OP's question trying to answer in awk to substitute ' to " here.
awk -v s1="\"" '!/This could/ && !/others/{gsub(/\047/,s1) } /This could/ || /others/{gsub(/\047/,s1,$1)} 1' Input_file
Output will be as follows.
{"data": ["my",
"my_other",
"my_other",
"my_other",
"This could 'happen' <- and this ones i want to keep',
],
"name": "MyName"},
"data2": {"type": "x86_64",
"name": "This",
"type": "That",
"others": 'bla bla 'bla' <- keep this ones too',
"version": "21237821972"}}}
We know that ‘sed’ command can search for a pattern and can replace that pattern with user provided new one
For example sed “s/pattern1/pattern2/g” filename.txt
Now the ‘sed’ command will search for pattern1 and if found it will replace with pattern2
For your requirement you just need to apply this rule. See below
First
sed "s/^\'/\"/g” yourfile
This will search for every newline with character ‘ in the file and replace with “
Next requirement is to search for pattern ‘: and replace with “:
So add one more condition to it separated by ;
sed "s/^\'/\"/g; s/\':/\":/g” yourfile
Just follow this algorithm till you reach you requirement
The final should be look like:-
sed "s/^\'/\"/g; s/\':/\":/g;s/{\'/{\"/g;s/\[\'/\[\"/g;s/\',/\",/g;s/\'}/\"}/g;s/: \'/: \"/g;" yourfile > newfil
(If the above command gives you error just use the command at the very beginning)
finally
mv newfile yourfile

Resources