I have encountered a strange problem while I was trying to write a bash scritp to copy some data from one database to another.
To make things simple I will present the issue with the following example:
Let's say, we have a file in which are mongo insert commands that want to execute in mongo client. With Bash it will be:
cat file.json | mongo --verbose --host $HOST
This works fine until we use qoutes in records content.
For example:
use somedb;
db["collection"].insert({ "field": "test" })
#This of course works
db["collection"].insert({ "field": "test \" test" })
#But this doesn't
db["collection"].insert({ "field": "test \\" test" }) "#<-For syntax coloring
#I thounght maybe double quoting will help, but this also doesn't work
db["collection"].insert({ "field": "\"test\"" })
#This SUPRISINGLY works!!!
My question is, what is the propper way of escaping quotes for mongo client (I am using MongoDB shell verions: 2.2.4)?
Why when there is an even number of quotes in record, the script will succeed and with odd number will fail?
I will add that, there are no error messages. Mongo just fails silently(even with --verbose) and no new records appears in collection.
There's a JIRA ticket for this issue and it's fixed in 2.5.0.
For now, you can use the unicode point for double quote when inserting:
> db.foo.insert({ "field": "test \u0022 test" })
> db.foo.find()
{ "_id" : ObjectId("533455e563083f9b26efb5c2"), "field" : "test \" test" }
Related
I realize this is a simple question but I haven't been able to find the answer. Thank you to anyone who may be able to help me understand what I am doing wrong.
Goal: Search and replace a string in a specific key in a JSON file with a string in a Bash variable using jq.
For example, in the following JSON file:
"name": "Welcome - https://docs.mysite.com/",
would become
"name": "Welcome",
Input (file.json)
[
{
"url": "https://docs.mysite.com",
"name": "Welcome - https://docs.mysite.com/",
"Ocurrences": "679"
},
{
"url": "https://docs.mysite.com",
"name": "Welcome",
"Ocurrences": "382"
}
]
Failing script (using variable)
siteUrl="docs.mysite.com"
jq --arg siteUrl "$siteUrl" '.[].name|= (gsub(" - https://$siteUrl/"; ""))' file.json > file1.json`
Desired output (file1.json)
[
{
"url": "https://docs.mysite.com",
"name": "Welcome",
"Ocurrences": "679"
},
{
"url": "https://docs.mysite.com",
"name": "Welcome",
"Ocurrences": "382"
}
]
I've tried various iterations on removing quotes, changing between ' and ", and adding and removing backslashes.
Successful script (not using variable)
jq '.[].name|= (gsub(" - https://docs.mysite.com/"; ""))' file.json > file1.json
More specifically, if it matters, I am processing an export of a website's usage data from Azure App Insights. Unfortunately, the same page may be assigned different names. I sum the Ocurrences of the two objects with the newly identical url later. If it is better to fix this in App Insights I am grateful for that insight, although I know Bash better than Kusto queries. I am grateful for any help or direction.
Almost. Variables are not automatically expanded within a string. You must interpolate them explicitly with \(…):
jq --arg siteUrl 'docs.mysite.com' '.[].name |= (gsub(" - https://\($siteUrl)/"; ""))' file.json
Alternatively, detect a suffix match and extract the prefix by slicing:
jq --arg siteUrl 'docs.mysite.com' '" - https://\($siteUrl)/" as $suffix | (.[].name | select(endswith($suffix))) |= .[:$suffix|-length]' file.json
This question already has answers here:
When to wrap quotes around a shell variable?
(5 answers)
Closed 11 months ago.
I have a script like below and I am running queries in mongodb. But when I call it as a function, it adds single quote and my query doesn't work. How can i solve this problem?
My Code:
mongo_func() {
mongo "$MONGO/$1" --quiet --eval $2 $3 $4
}
mongo_func "Users" 'db.ucol.updateMany({}, { $unset : {"name" : "", "Users": ""}});'
When I debug with bash -x, the output looks like below.
mongo 10.10.1.2/Users --quiet --eval 'db.ucol.updateMany({},' '{$unset:' '{"name"' : '"",' '"Users":' '""}});'
It s working as follows properly.
mongo 10.10.1.2/Users --quiet --eval 'db.ucol.updateMany({}, {$unset: {"name" : "", "Users": ""}});'
That's because you have failed to quote the function arguments.
Try this instead.
mongo_func() {
user=$1
shift
mongo "$MONGO/$user" --quiet --eval "$#"
}
I'm a bit new to batch scripting, so I apologize if this is glaringly obvious, but I couldn't find any similar information.
I'm trying to perform the following cURL call:
curl -H "Content-Type:application/json" -d '{"lt":"f","sort":"n","max":"1500","offset":"1500"}' [API_KEY]#api.nal.usda.gov/ndb/list
When I run that line in a command line (or Cygwin) it does exactly what I want it to.
However, when I try to call it from a bat file, it seems my parameters are getting messed up somehow.
FOR /L %%A IN (0, 1500, 77500) DO (
curl -H "Content-Type:application/json" -d '{"lt":"f","sort":"n","max":"1500","offset":"%%A"}' [API_KEY]#api.nal.usda.gov/ndb/list > %%A.txt
)
I'm getting output into the correct .txt file, but it doesn't seem like the %%A in offset is getting replaced. I'm getting a "bad parameter" exception from the API. From the output on the command line, it looks accurate.
I'm open to using bash scripting instead if it would make more sense, but I was having the same issue using bash.
(Note: I replaced my API key with a placeholder in the example.. that's not the problem)
In bash at least, the problem is that variable expansion does not occur inside single quotes; you need to use double quotes and escape the nested double quotes:
for a in 0 1500 77500; do
curl -H "Content-Type:application/json" -d "{\"lt\":\"f\",\"sort\":\"n\",\"max\":\"1500\",\"offset\":\"$a\"}" [API_KEY]#api.nal.usda.gov/ndb/list > "$a".txt
)
I suspect you need to do the equivalent in a batch file.
You can concatenate adjacent single- and double-quoted strings to minimized the number of escaped quotes:
... -d '{"lt": "f", "sort": "n", "max": "1500", "offset": "'"$a"'"}' ...
but you may want to consider one of two other options. First, read the data from a here document instead of using a hard-coded string:
curl -H "..." -d#- [API_KEY]#api.nal.usda.gov/ndb/list > "$a".txt <<EOF
{"lt": "f", "sort": "n", "max": "1500", "offset": "$a"}
EOF
or use something like jq to generate the JSON for your:
curl -H "..." \
-d "$(jq --arg a "$a" '{lt: "f", sort: "n", max: "1500", offset: $a}') \
[API_KEY]#api.nal.usda.gov/ndb/list > "$a".txt
The jq solution would be preferable in general, since you don't have to worry about pre-escaping any variable values.
What is the exact code I need to execute, to insert a document into a mongodb using bash. At the moment I am able to look documents in mongodb via bash script up, but inserting does not work.
You can inject javascript code from a javascript file:
mongo 127.0.0.1/MyDatabase script.js
with script.js:
var document = {
name : "document_name",
title : "document_title"
};
db.MyCollection.insert(document);
or directly:
mongo 127.0.0.1/MyDatabase --eval 'var document = { name : "document_name", title : "document_title" }; db.MyCollection.insert(document);'
If you don't want to serve script from a file (I try not to source external files as much as possible) or not use --eval option which can be difficult to read if you have many entries, you can use a bash heredoc
You can type in terminal:
-bash-4.1$ mongo mongodb://myServerAddress/myDbName <<EOF
> db.myCollectionName.insert({
> name: "doc name",
> title: "doc title"
> })
> EOF
Result:
MongoDB shell version v3.4.1
connecting to: mongodb://myServerAddress/myDbName
MongoDB server version: 3.0.7
WARNING: shell and server versions do not match
WriteResult({ "nInserted" : 1 })
bye
-bash-4.1$
If you want to keep it in a script, just remove > which is actually prompt for a multiline command.
For in-script use, it should be as below:
#!/usr/bin/env bash
mongo mongodb://myServerAddress/myDbName <<EOF
db.myCollectionName.insert({
name: "doc name",
title: "doc title"
})
EOF
insert some JSON into Mongo DB sample_db, collection products_coll
echo '{ item: "card", qty: 115 }' | \
sed 's#^#db.products_coll.insert( #; s#$# ) #' | mongo sample_db
MongoDB shell version v3.6.3
connecting to: mongodb://127.0.0.1:27017/sample_db
MongoDB server version: 3.6.3
WriteResult({ "nInserted" : 1 })
bye
make sure it is here
mongoexport -d sample_db -c products_coll
{"_id":{"$oid":"5d27e83152a049227799710e"},"item":"card","qty":115.0}
I have a problem when I include a double dollar sign symbol in a not analyzed field. When I check the mapping for the type with the following command:
curl -XGET 'http://localhost:9200/documents/_mapping/document'
I get this output for the code field:
{
"documents": {
"mappings": {
"document": {
"properties": {
"code": {
"index": "not_analyzed",
"type": "string"
},
// More field mappings
If I test the mapping with the following command:
curl -XGET 'http://localhost:9200/documents/_analyze?field=code' -d "ABC$"
I get the following output:
{"tokens":[{"token":"ABC$","start_offset":0,"end_offset":4,"type":"word","position":1}]}
This is ok because the token has the same value that the data entered in the command. The problem is when I use a double dollar sign like this:
curl -XGET 'http://localhost:9200/documents/_analyze?field=code' -d "ABC$$"
Then I get the following token:
{"tokens":[{"token":"ABC4088","start_offset":0,"end_offset":7,"type":"word","position":1}]}
It looks like the double dollar sign gets replaced by the number 4088. According to the docs, the not_analyzed index attribute means:
Index this field, so it is searchable, but index the value exactly as specified. Do not analyze it.
Do miss something in the code field mapping to avoid this?
This is because $$ is a special environment variable in your shell that gets expanded to the ID of the current shell process.
So when running the curl command what happens is that the PID of your current shell is resolved and replaces the $$ variable, hence the 4088 you see instead of $$.
Try to issue echo $$ and you'll see the ID of your current shell.
curl -XGET 'http://localhost:9200/documents/_analyze?field=code' -d "ABC4088"
^
|
$$ replaced by the ID of your process
Try to simply escape the $ characters and you should be fine
curl -XGET 'http://localhost:9200/documents/_analyze?field=code' -d "ABC\$\$"