I have a part of json which looks like below:
{
"openstack": {
"admin": {
"username": "admin",
"password": "password",
"tenant_name": "test"
},
and three environment variables defined like this
auth_url=VALUE1
region_name=VALUE3
endpoint_type=VALUE2
I want to insert 3 lines in the input file just after row 2, so that the output is
{
"openstack": {
"auth_url": VALUE1,
"region_name": VALUE2,
"endpoint_type": VALUE3,
"admin": {
"username": "admin",
"password": "password",
"tenant_name": "test"
},
How it can be done using SED, I tried below
sed -e '3i/\t"auth_url":$auth_url,' -i account_2.json
But it not only adds an extra / at row no 3 but it also doesn't actually replace $auth_url with environment variable as well.
You are are misusing the i command. You have to put backslash after it, not a slash.
Furhtermore, the variable is not expanded since it is in single quotes. Try putting it in double quotes, like this
sed "3i\ \"auth_url\":$AUTH," yourfile
I've read that the insert command wants whatever follows the backslash to be on a newline, which is not the case here, where we have everything on a single line. I guess that's GNU sed's which allows it.
To insert three lines, you can use this
sed "3i\ \"auth_url\":$SHELL\n \"auth_url\":$SHELL\n \"auth_url\":$SHELL" os
And it works well with commas too, since they have no special meaning:
sed "3i\ \"auth_url\":$SHELL,\n \"auth_url\":$SHELL,\n \"auth_url\":$SHELL,"
Related
{
"id": "a1234567-89ab-cdef-0123-456789abcdef",
"properties": {
...
"my_id": "c1234567-89ab-cdef-0123-456789abcdef",
...
}
Given the above in a file, I want to be able to perform a match (including the 4 leading spaces) on my_id and then append a new line "my_value": "abcd",. The desired output would look like this:
{
"id": "a1234567-89ab-cdef-0123-456789abcdef",
"properties": {
...
"my_id": "c1234567-89ab-cdef-0123-456789abcdef",
"my_value": "abcd",
...
}
Using examples online, I'm unable to get the command to work. Here is an example of something I have tried: sed '/.*"my_id".*/a "my_value": "abcd",' test.json, for which I receive the following error: command a expects \ followed by text.
What is the correct way to structure this command?
Using any awk:
$ awk -v new='"my_value": "abcd",' '{print} sub(/"my_id":.*/,""){print $0 new}' file
{
"id": "a1234567-89ab-cdef-0123-456789abcdef",
"properties": {
...
"my_id": "c1234567-89ab-cdef-0123-456789abcdef",
"my_value": "abcd",
...
}
The above will print the new line using whatever indent the existing "my_id" line has, it doesn't assume/hard-code any indent, e.g. 4 blanks.
I'm using this:
sub(/"my_id":.*/,""){print $0 new}
instead of the briefer:
sub(/"my_id":.*/,new)
so it won't break if the new string contains any backreference chars such as &.
awk procedure with passed argument for insert value
The following awk procedure allows for 'abcd' to be passed as an argument for insertion (allowing it to be set in a bash script if required).
awk -v insertVal="abcd" '/"my_id":/{$0=$0"\n \"my_value\": \""insertVal"\","} {print}' dat.txt
explanation
The required insertion string ('abcd' in this case) is passed as an argument using the -v variable switch followed by a variable name and value: insertVal="abcd".
The first awk action block has a pattern condition to only act on lines containing the target-line string (in this case "my_id":). When a line with that pattern is found, the line is extended with a new line mark \n, the required four spaces to start the next line, the specified key named "my_value", and the value associated with the key, passed by argument as the variable named insertVal ("abcd"), and the final , character. Note the need to escape the " quotes to render them.
The final awk block, prints the current line (whether or not it was modified).
test
The procedure was tested on Mac Terminal using GNU Awk 5.2.0.
The output generated (from the input data saved to a file named dat.txt) is:
{
"id": "a1234567-89ab-cdef-0123-456789abcdef",
"properties": {
...
"my_id": "c1234567-89ab-cdef-0123-456789abcdef",
"my_value": "abcd",
...
}
Using sed
$ sed -e '/my_id/{p;s/id.*"/value": "abcd"/' -e '}' input_file
{
"id": "a1234567-89ab-cdef-0123-456789abcdef",
"properties": {
...
"my_id": "c1234567-89ab-cdef-0123-456789abcdef",
"my_value": "abcd",
...
}
With your shown samples and attempts please try following GNU awk code. Where newVal is an awk variable having new value in it. Using match function in GNU awk where I have used regex (.*)("my_id": "[^"]*",)(.*) which creates 3 capturing groups and saves values into an array named arr. Then printing values as per requirement.
awk -v newVal='"my_value": "abcd",' -v RS= '
match($0,/(.*)("my_id": "[^"]*",)(.*)/,arr){
print arr[1] arr[2] newVal arr[3]
}
' Input_file
This might work for you (GNU sed):
sed '/"my-id".*/p;s//"my-value": "abcd"/' file
Match on "my-id" and print that line, then substitute the additional line.
Suppose I have some text file (json in this case):
{
"data": [
{
"timestamp": 1577856103107
},
{
"timestamp": 1577869991302
}
]
}
And I want to replace a pattern (in this case a UNIX millisecond timestamp) with a more readable date format.
I'm trying with this:
$ sed -E 's/(.*)([0-9]{13})/echo "\1\\"$(date --date="#$((\2\/1000))" --iso-8601=seconds)\\""/e' example.json
{
"data": [
{
timestamp: "2020-01-01T00:21:43-05:00"
},
{
timestamp: "2020-01-01T04:13:11-05:00"
}
]
}
This is somewhat ok, but I don't understand why the quotes arround timestamp get lost.
This command works:
sed -E 's/(.*)"(timestamp)"(: )([0-9]{13})/echo "\1\\"\2\\"\3\\"$(date --date="#$((\4\/1000))" --iso-8601=seconds)"\\"/e' example.json
{
"data": [
{
"timestamp": "2020-01-01T00:21:43-05:00"
},
{
"timestamp": "2020-01-01T04:13:11-05:00"
}
]
}
I also don't understand why I need double backslashes \\ to ouput a double-quote " in the right side of this sed command.
Is there a better way (or tool) to solve this?
I'm on sed (GNU sed) 4.8 and zsh 5.8 (x86_64-pc-linux-gnu), thanks!
Is there a better way (or tool) to solve this?
Using sed for manipulating json things is very crude. You can't parse json with regex. I (strongly) suggest to use json-aware tools, like jq.
jq '.data[].timestamp |= (. / 1000 | strftime("%Y-%m-%dT%H:%M:%SZ"))'
but I don't understand why the quotes arround timestamp get lost.
The:
echo "\1\\"$(date --date="#$((\2\/1000))" --iso-8601=seconds)\\""
is "substituted" to:
echo ""timestamp": \"$(date --date="#$((1577856103107/1000))" --iso-8601=seconds)\""
^^
^------------------------------------------------------------------^
Then is passed to shell and quotes are re-evalulated according to shell rules.
Matching (.*) is really doing nothing, just match only the part you want to substitute. You could instead match only the part you want to substitute:
sed '/"timestamp":/s/([0-9]{13})/echo "\\"$(date --date="#$((\1\/1000))" --iso-8601=seconds)\\""/e'
why I need double backslashes \ to ouput a double-quote " in the right side of this sed command.
First \\ is interpreted by sed into single \.
$ echo a | sed 's/.*/single slash: \\/'
single slash: \
Then the result of sed command is passed to shell where all shell parsing rules are done one again.
I am trying to process some string which has special characters in it like abc123#45 or ab$123 or qwe&123.
I am trying to fetch it in shell like:
In json file : foo=qwe$123
foo=`cat tmp_json | jq -r '.keys.foo'`
But it is coming like :
foo=qwe23
JSON input
{
"metadata": {
"name": "xyz",
"version": 7,
"lastUpdated": 1585551422521
},
"keys": {
"abc": "qwe$123",
"foo": "qwe$123"
}
}
When shell strings contain special characters that you do not want to be interpreted specially by the shell, you have to quote them using single quotes, e.g. foo='qwe$123'
Using bash 4.x, the form
x=`...`
does not present any problems with respect to characters such $, #, or &, though it should be noted that the preferred form for such assignments is x=$(...)
However these forms should only be used with great care because of other special characters.
Generally, it would be better to use an idiom such as:
jq -r .... | while -r read line ; do .... ; done
Depending on your requirements, you might also wish to consider jq's #sh filter.
I have this text file:
{
"name": "",
"auth": true,
"username": "rtorrent",
"password": "d5275b68305438499f9660b38980d6cef7ea97001efe873328de1d76838bc5bd15c99df8b432ba6fdcacbff82e3f3c4829d34589cf43236468d0d0b0a3500c1e"
}
Now, I want to be able to replace the d5275b68305438499f9660b38980d6cef7ea97001efe873328de1d76838bc5bd15c99df8b432ba6fdcacbff82e3f3c4829d34589cf43236468d0d0b0a3500c1e using sed for example. (The string has always the exact same length, but the values can be different)
I've tried this using sed:
sed -i 5s/./new-string/18 file.json
That basically replaces text, on the 5th line, starting with position 18. I want to be able to replace the text, exactly starting with position 18 and up to position 154, strictly what's inside the "". The command above will cut the ", at the end of the file and if it's run multiple times, the string becomes every time longer and longer.
Any help is really appreciated.
You can use for example awk for it:
$ awk -v var="new_string" 'NR==5{print substr($0,1,17) var substr($0,146);next}1' file
{
"name": "",
"auth": true,
"username": "rtorrent",
"password": "new_string"
}
but there are better tools for changing a value in a JSON, jq for example:
$ jq '.password="new_string"' file
{
"name": "",
"auth": true,
"username": "rtorrent",
"password": "new_string"
}
Edit: When passing a shell variable $var to awk and jq:
$ var="new_string"
$ awk -v var="$var" 'NR==5{print substr($0,1,17) var substr($0,146);next}1' file
and
$ jq --arg var "$var" '.password=$var'
Edit2: There is always sed:
$ sed -i "5s/\"[^\"]*\"/\"$var\"/2" file
I am trying to insert following data in psql and its showing me weird error.
-bash-4.1$ psql -t -d mydb -c 'insert into ftp_mgr (id, info) values("192.168.1.12", '{"username": "Administrator", "password": "abc456$", "serverAddr":"192.168.1.12"}');'
psql: warning: extra command-line argument "password:" ignored
psql: warning: extra command-line argument "abc456$," ignored
psql: warning: extra command-line argument "serverAddr:192.168.1.12});" ignored
psql: FATAL: role "Administrator," does not exist
This is nothing to do with PostgreSQL. It's a shell quoting problem.
You cannot nest single quotes in bash. So this:
'insert into ftp_mgr (id, info) values("192.168.1.12", '{"username": "Administrator", "password": "abc456$", "serverAddr":"192.168.1.12"}');'
is read up to:
'insert ...'{
and the { is *not part of the quoted string. So the shell tries to intepret {"username": "Administrator", "password": "abc456$", "serverAddr":"192.168.1.12"} as a shell command.
The simplest thing to do here is to use a quoted here-document to avoid the need to deal with ' literal quoting mixing with the shell's quoting:
psql -t -d mydb <<'__END__'
insert into ftp_mgr (id, info) values
("192.168.1.12", '{"username": "Administrator", "password": "abc456$", "serverAddr":"192.168.1.12"}');
__END__
The quotes around the here-document tag are important. If you leave them out then bash still looks for and replaces $variable strings, etc, within the here-document text. That can be really handy, but it's clearly not what you want in this case.
if you must do it with -c you have a couple of options:
Use the shell's string concatenation. Whenever you want a single-quote, end the current single quoted string, write "'", and open a new single quoted string. So you'd write:
'insert ... values(..., '"'"'{"....
Confusing to read, isn't it? The first ' ends the single-quoted shell string. Then "'" is a single quote, quoted by double quotes that're consumed by the shell. Then the next ' begins a new single-quoted string.
That's horrible to read, and it's not much better written as:
'insert ... values(..., '\''{"....
so personally, when I need single quotes within an argument string I either double-quote the whole string and backslash-escape shell metacharacters:
"insert into ftp_mgr (id, info) values(\"192.168.1.12\", '{\"username\": \"Administrator\", \"password\": \"abc456\$\", \"serverAddr\":\"192.168.1.12\"}');"
... or where possible I use a quoted here-document to get around the whole horrifying mess.
My preference is to just avoid the shell for this kind of thing and use a simple Python script.
import psycopg2
conn = psycopg2.connect('dbname=postgres')
curs = conn.cursor()
curs.execute(r'insert into ftp_mgr (id, info) values(%s, %s);', (
r'192.168.1.12',
r'{"username": "Administrator", "password": "abc456$", "serverAddr":"192.168.1.12"}'
))
Python's support for raw-strings (r''), triple quote strings ("""), etc make this sort of thing much easier.