Trying to solve "argument list too long"
I have been searching for a solution and found the closest one to my issue
curl: argument list too long
however the response is not clear as I am still having the issue "argument list too long"
curl -X POST -d #data.txt \
https://Path/to/attachments \
-H 'content-type: application/vnd.api+json' \
-H 'x-api-key: KEY' \
-d '{
"data": {
"type": "attachments",
"attributes": {
"attachment": {
"content": "'$(cat data.txt | base64 --wrap=0)'",
"file_name": "'"$FileName"'"
}
}
}
}'
thank you
Use jq to format your base64 encoded data string into a proper JSON string, and then pass the JSON data as standard input to the curl command.
#!/usr/bin/env sh
attached_file='img.png'
# Pipe the base64 encoded content of attached_file
base64 --wrap=0 "$attached_file" |
# into jq to make it a proper JSON string within the
# JSON data structure
jq --slurp --raw-input --arg FileName "$attached_file" \
'{
"type": "attachments",
"attributes": {
"attachment": {
"content": .,
"file_name": $FileName
}
}
}
' |
# Get the resultant JSON piped into curl
# that will read the data from the standard input
# using -d #-
curl -X POST -d #- \
'https://Path/to/attachments' \
-H 'content-type: application/vnd.api+json' \
-H 'x-api-key: KEY'
Per the linked answer
you are trying to pass the entirety of the base64'd content on the command line
This is a limitation of the shell, not curl. That is, the shell is responding with error argument list too long. The program curl is never even started.
The recommendation is
curl has the ability to load in data to POST from a file
Write the json data to some file /tmp/data.json using piping.(the commands will use piping | and file redirection > >> which can handle arbitrarily large amounts of data. Whereas, you cannot place arbitrarily large amounts of data into a single command, there is a limit).
echo -n '
{
"data": {
"type": "attachments",
"attributes": {
"attachment": {
"content": "' > /tmp/data.json
cat data.txt | base64 --wrap=0 >> /tmp/data.json
echo -n '",
"file_name": "'"$FileName"'"
}
}
}
}' >> /tmp/data.json
Pass that file path /tmp/data.json to the curl command using # so curl knows it's a file path.
curl -X POST -d #/tmp/data.json \
"https://Path/to/attachments" \
-H 'content-type: application/vnd.api+json' \
-H 'x-api-key: KEY'
I'm using git, then posting the commit message and other bits as a JSON payload to a server.
Currently I have:
MSG=`git log -n 1 --format=oneline | grep -o ' .\+'`
which sets MSG to something like:
Calendar can't go back past today
then
curl -i -X POST \
-H 'Accept: application/text' \
-H 'Content-type: application/json' \
-d "{'payload': {'message': '$MSG'}}" \
'https://example.com'
My real JSON has another couple of fields.
This works fine, but of course when I have a commit message such as the one above with an apostrophe in it, the JSON is invalid.
How can I escape the characters required in bash? I'm not familiar with the language, so am not sure where to start. Replacing ' with \' would do the job at minimum I suspect.
jq can do this.
Lightweight, free, and written in C, jq enjoys widespread community support with over 15k stars on GitHub. I personally find it very speedy and useful in my daily workflow.
Convert string to JSON
echo -n '猫に小判' | jq -Rsa .
# "\u732b\u306b\u5c0f\u5224"
To explain,
-R means "raw input"
-s means "include linebreaks" (mnemonic: "slurp")
-a means "ascii output" (optional)
. means "output the root of the JSON document"
Git + Grep Use Case
To fix the code example given by the OP, simply pipe through jq.
MSG=`git log -n 1 --format=oneline | grep -o ' .\+' | jq -Rsa .`
Using Python:
This solution is not pure bash, but it's non-invasive and handles unicode.
json_escape () {
printf '%s' "$1" | python -c 'import json,sys; print(json.dumps(sys.stdin.read()))'
}
Note that JSON is part of the standard python libraries and has been for a long time, so this is a pretty minimal python dependency.
Or using PHP:
json_escape () {
printf '%s' "$1" | php -r 'echo json_encode(file_get_contents("php://stdin"));'
}
Use like so:
$ json_escape "ヤホー"
"\u30e4\u30db\u30fc"
Instead of worrying about how to properly quote the data, just save it to a file and use the # construct that curl allows with the --data option. To ensure that the output of git is correctly escaped for use as a JSON value, use a tool like jq to generate the JSON, instead of creating it manually.
jq -n --arg msg "$(git log -n 1 --format=oneline | grep -o ' .\+')" \
'{payload: { message: $msg }}' > git-tmp.txt
curl -i -X POST \
-H 'Accept: application/text' \
-H 'Content-type: application/json' \
-d #git-tmp.txt \
'https://example.com'
You can also read directly from standard input using -d #-; I leave that as an exercise for the reader to construct the pipeline that reads from git and produces the correct payload message to upload with curl.
(Hint: it's jq ... | curl ... -d#- 'https://example.com' )
I was also trying to escape characters in Bash, for transfer using JSON, when I came across this. I found that there is actually a larger list of characters that must be escaped – particularly if you are trying to handle free form text.
There are two tips I found useful:
Use the Bash ${string//substring/replacement} syntax described in this thread.
Use the actual control characters for tab, newline, carriage return, etc. In vim you can enter these by typing Ctrl+V followed by the actual control code (Ctrl+I for tab for example).
The resultant Bash replacements I came up with are as follows:
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//\\/\\\\} # \
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//\//\\\/} # /
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//\'/\\\'} # ' (not strictly needed ?)
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//\"/\\\"} # "
JSON_TOPIC_RAW=${JSON_TOPIC_RAW// /\\t} # \t (tab)
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//
/\\\n} # \n (newline)
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//^M/\\\r} # \r (carriage return)
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//^L/\\\f} # \f (form feed)
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//^H/\\\b} # \b (backspace)
I have not at this stage worked out how to escape Unicode characters correctly which is also (apparently) required. I will update my answer if I work this out.
OK, found out what to do. Bash supports this natively as expected, though as always, the syntax isn't really very guessable!
Essentially ${string//substring/replacement} returns what you'd image, so you can use
MSG=${MSG//\'/\\\'}
To do this. The next problem is that the first regex doesn't work anymore, but that can be replaced with
git log -n 1 --pretty=format:'%s'
In the end, I didn't even need to escape them. Instead, I just swapped all the ' in the JSON to \". Well, you learn something every day.
git log -n 1 --format=oneline | grep -o ' .\+' | jq --slurp --raw-input
The above line works for me. refer to
https://github.com/stedolan/jq for more jq tools
I found something like that :
MSG=`echo $MSG | sed "s/'/\\\\\'/g"`
The simplest way is using jshon, a command line tool to parse, read and create JSON.
jshon -s 'Your data goes here.' 2>/dev/null
[...] with an apostrophe in it, the JSON is invalid.
Not according to https://www.json.org. A single quote is allowed in a JSON string.
How can I escape the characters required in bash?
You can use xidel to properly prepare the JSON you want to POST.
As https://example.com can't be tested, I'll be using https://api.github.com/markdown (see this answer) as an example.
Let's assume 'çömmít' "mêssågè" as the exotic output of git log -n 1 --pretty=format:'%s'.
Create the (serialized) JSON object with the value of the "text"-attribute properly escaped:
$ git log -n 1 --pretty=format:'%s' | \
xidel -se 'serialize({"text":$raw},{"method":"json","encoding":"us-ascii"})'
{"text":"'\u00E7\u00F6mm\u00EDt' \"m\u00EAss\u00E5g\u00E8\""}
Curl (variable)
$ eval "$(
git log -n 1 --pretty=format:'%s' | \
xidel -se 'msg:=serialize({"text":$raw},{"method":"json","encoding":"us-ascii"})' --output-format=bash
)"
$ echo $msg
{"text":"'\u00E7\u00F6mm\u00EDt' \"m\u00EAss\u00E5g\u00E8\""}
$ curl -d "$msg" https://api.github.com/markdown
<p>'çömmít' "mêssågè"</p>
Curl (pipe)
$ git log -n 1 --pretty=format:'%s' | \
xidel -se 'serialize({"text":$raw},{"method":"json","encoding":"us-ascii"})' | \
curl -d#- https://api.github.com/markdown
<p>'çömmít' "mêssågè"</p>
Actually, there's no need for curl if you're already using xidel.
Xidel (pipe)
$ git log -n 1 --pretty=format:'%s' | \
xidel -s \
-d '{serialize({"text":read()},{"method":"json","encoding":"us-ascii"})}' \
"https://api.github.com/markdown" \
-e '$raw'
<p>'çömmít' "mêssågè"</p>
Xidel (pipe, in-query)
$ git log -n 1 --pretty=format:'%s' | \
xidel -se '
x:request({
"post":serialize(
{"text":$raw},
{"method":"json","encoding":"us-ascii"}
),
"url":"https://api.github.com/markdown"
})/raw
'
<p>'çömmít' "mêssågè"</p>
Xidel (all in-query)
$ xidel -se '
x:request({
"post":serialize(
{"text":system("git log -n 1 --pretty=format:'\''%s'\''")},
{"method":"json","encoding":"us-ascii"}
),
"url":"https://api.github.com/markdown"
})/raw
'
<p>'çömmít' "mêssågè"</p>
This is an escaping solution using Perl that escapes backslash (\), double-quote (") and control characters U+0000 to U+001F:
$ echo -ne "Hello, 🌵\n\tBye" | \
perl -pe 's/(\\(\\\\)*)/$1$1/g; s/(?!\\)(["\x00-\x1f])/sprintf("\\u%04x",ord($1))/eg;'
Hello, 🌵\u000a\u0009Bye
I struggled with the same problem. I was trying to add a variable on the payload of cURL in bash and it kept returning as invalid_JSON. After trying a LOT of escaping tricks, I reached a simple method that fixed my issue. The answer was all in the single and double quotes:
curl --location --request POST 'https://hooks.slack.com/services/test-slack-hook' \
--header 'Content-Type: application/json' \
--data-raw '{"text":'"$data"'}'
Maybe it comes in handy for someone!
I had the same idea to send a message with commit message after commit.
First i tryed similar was as autor here.
But later found a better and simpler solution.
Just created php file which is sending message and call it with wget.
in hooks/post-receive :
wget -qO - "http://localhost/git.php"
in git.php:
chdir("/opt/git/project.git");
$git_log = exec("git log -n 1 --format=oneline | grep -o ' .\+'");
And then create JSON and call CURL in PHP style
Integrating a JSON-aware tool in your environment is sometimes a no-go, so here's a POSIX solution that should work on every UNIX/Linux:
json_stringify() {
[ "$#" -ge 1 ] || return 1
LANG=C awk '
BEGIN {
for ( i = 1; i <= 127; i++ )
repl[ sprintf( "%c", i) ] = sprintf( "\\u%04x", i )
for ( i = 1; i < ARGC; i++ ) {
s = ARGV[i]
printf("%s", "\"")
while ( match( s, /[\001-\037\177"\\]/ ) ) {
printf("%s%s", \
substr(s,1,RSTART-1), \
repl[ substr(s,RSTART,RLENGTH) ] \
)
s = substr(s,RSTART+RLENGTH)
}
print s "\""
}
exit
}
' "$#"
}
Or using the widely available perl:
json_stringify() {
[ "$#" -ge 1 ] || return 1
LANG=C perl -le '
for (#ARGV) {
s/[\x00-\x1f\x7f"\\]/sprintf("\\u%04x",ord($0))/ge;
print "\"$_\""
}
' -- "$#"
}
Then you can do:
json_stringify '"foo\bar"' 'hello
world'
"\u0022foo\bar\u0022"
"hello\u000aworld"
limitations:
Doesn't handle NUL bytes.
Doesn't validate the input for UNICODE, it only escapes the mandatory ASCII characters specified in the RFC 8259.
Replying to OP's question:
MSG=$(git log -n 1 --format=oneline | grep -o ' .\+')
curl -i -X POST \
-H 'Accept: application/text' \
-H 'Content-type: application/json' \
-d '{"payload": {"message": '"$(json_stringify "$MSG")"'}}' \
'https://example.com'
I want to send a big json with long string field by curl, how should I crop it to multiple lines? For example:
curl -X POST 'localhost:3000/upload' \
-H 'Content-Type: application/json'
-d "{
\"markdown\": \"# $TITLE\\n\\nsome content with multiple lines....\\n\\nsome content with multiple lines....\\n\\nsome content with multiple lines....\\n\\nsome content with multiple lines....\\n\\n\"
}"
Use a tool like jq to generate your JSON, rather than trying to manually construct it. Build the multiline string in the shell, and let jq encode it. Most importantly, this avoids any potential errors that could arise from TITLE containing characters that would need to be correctly escaped when forming your JSON value.
my_str="# $TITLE
some content with multiple lines...
some content with multiple lines...
some content with multiple lines..."
my_json=$(jq --argjson v "$my_str" '{markdown: $v}')
curl -X POST 'localhost:3000/upload' \
-H 'Content-Type: application/json' \
-d "$my_json"
curl has the ability to read the data for -d from standard input, which means you can pipe the output of jq directly to curl:
jq --argjson v "$my_str" '{markdown: $v}' | curl ... -d#-
You can split anything to multiple lines using the technique already in your post, by terminating lines with \.
If you need to split in the middle of a quoted string,
terminate the quote and start a new one.
For example these are equivalent:
echo "foobar"
echo "foo""bar"
echo "foo"\
"bar"
But for your specific example I recommend a much better way.
Creating the JSON in a double-quoted string is highly error prone,
because of having to escape all the internal double-quotes,
which becomes hard to read and maintain as well.
A better alternative is to use a here-document,
pipe it to curl, and use -d#- to make it read the JSON from stdin.
Like this:
formatJson() {
cat << EOF
{
"markdown": "some content with $variable in it"
}
EOF
}
formatJson | curl -X POST 'localhost:3000/upload' \
-H 'Content-Type: application/json'
-d#-
If I were you, I'd save the JSON to a file:
curl -X POST 'localhost:3000/upload' \
-H 'Content-Type: application/json' \
-d "$(cat my_json.json)"
I'm using curl with IBM Watson to produce a transcript but I can't seem to get an output where just the transcript is shown as shown below
Another method might be just to grep for the text in transcript":""
curl
curl -u user:password -X POST --header "Content-Type: audio/wav" --header "Transfer-Encoding: chunked" --data-binary #test.wav "https://stream.watsonplatform.net/speech-to-text/api/v1/recognize?continuous=true" > demo.txt
{
"results": [
{
"alternatives": [
{
"confidence": 0.302,
"transcript": "when to stop announced "
}
],
"final": true
},
{
"alternatives": [
{
"confidence": 0.724,
"transcript": "Russia is destroying western cheese and considering a ban Weston condoms and infection internet is reacting "
}
],
"final": true
To store the output into a file you can use the option -o
curl -u user:password -X POST --data-binary #test.wav
-o transcript.txt
--header "Content-Type: audio/wav" --header "Transfer-Encoding: chunked"
"https://stream.watsonplatform.net/speech-to-text/api/v1/recognize?continuous=true"
-o, --output <file>
Write output to <file> instead of stdout. Like in:
curl http://ibm.com.com -o "html_output.txt"
More info.
grep alone can't do quite what you're asking for because it doesn't get more granular than a single line. However, grep + sed can - sed can be used to perform regex replacement on the lines that grep spits out.
First pipe to grep: grep transcript, then pipe to sed: sed 's/.*"transcript": "\(.*\)".*/\1/g'
Here's the complete command:
curl -u user:password -X POST --header "Content-Type: audio/wav" --header "Transfer-Encoding: chunked" --data-binary #test.wav "https://stream.watsonplatform.net/speech-to-text/api/v1/recognize?continuous=true" | grep transcript | sed 's/.*"transcript": "\(.*\)".*/\1/g'
Note that by default, curl displays several lines of status when piping it's output. You can disable those with -s:
curl -s -u user:password -X POST --header "Content-Type: audio/wav" --header "Transfer-Encoding: chunked" --data-binary #test.wav "https://stream.watsonplatform.net/speech-to-text/api/v1/recognize?continuous=true" | grep transcript | sed 's/.*"transcript": "\(.*\)".*/\1/g'
Here's more info on sed if you're interested: http://www.grymoire.com/Unix/Sed.html
Update: I should also mention that there are SDKs available for a number of languages if you're interested in doing something more complex - https://github.com/watson-developer-cloud
I want to upload a file automatically to rackspace files which requires an auth-token that is updated daily, so I want to create a script which gets the auth token and then uses that in the script to upload the file.
This is the command to get the auth token which outputs the key perfectly:
curl -s -X POST https://auth.api.rackspacecloud.com/v2.0/tokens\
-d '{ "auth":{ "RAX-KSKEY:apiKeyCredentials":{ "username":"USER", "apiKey":"KEY" } } }'\
-H "Content-type: application/json" | python -mjson.tool |\
python -c 'import sys, json;\
print json.load(sys.stdin)[sys.argv[1]][sys.argv[2]][sys.argv[3]]'\
access token id
This is the command to upload the file:
curl -X PUT -T file.xml -D - \
-H "Content-Type: text/xml" \
-H "X-Auth-Token: TOKENGOESHERE" \
URL
I need to get the token from the first command into the TOKENGOESHERE place in the second command.
What I have tried so far is:
token = curl -s -X POST https://auth.api.rackspacecloud.com/v2.0/tokens -d '{ "auth":{ "RAX-KSKEY:apiKeyCredentials":{ "username":"USER", "apiKey":"KEY" } } }' -H "Content-type: application/json" | python -mjson.tool | python -c 'import sys, json; print json.load(sys.stdin)[sys.argv[1]][sys.argv[2]][sys.argv[3]]' access token id
curl -X PUT -T file.xml -D - \
-H "Content-Type: text/xml" \
-H "X-Auth-Token: $token" \
URL
but it didn't work and I am guessing it has something to do with the quotes but I don't know enough about bash to know what the problem is.
Thanks!
This should work:
token=$(curl -s -X POST https://auth.api.rackspacecloud.com/v2.0/tokens \
-d '{ "auth":{ "RAX-KSKEY:apiKeyCredentials":{ "username":"USER", "apiKey":"KEY" } } }' \
-H "Content-type: application/json" \
| python -mjson.tool \
| python -c 'import sys, json; print json.load(sys.stdin)["access"]["token"]["id"]')
curl -X PUT -T file.xml -D - \
-H "Content-Type: text/xml" \
-H "X-Auth-Token: $token" \
URL
I know it's a bit off topic, but I wanted to share my 'workflow' which may help a lot of people.
If you download these two cool toys (replacement for curl and python's json):
https://github.com/jkbr/httpie
http://stedolan.github.io/jq/
Then you can do all these fun things:
(Just replace USER and KEY with your real user and key in the 1st line, and all the others are copy and paste-able.
Get the json:
json=$(echo '{ "auth":{ "RAX-KSKEY:apiKeyCredentials":{ "username":"USER", "apiKey":"KEY" } } }' | http POST https://auth.api.rackspacecloud.com/v2.0/tokens)
Get token with http:
token=$(echo $json | jq '.access | .token | .id' | sed s/\"//g)
Easy token usage for later:
auth="X-Auth-Token:$token"
Get endpoint for Sydney cloud files (change SYD for your favorite Datacenter) (change publicURL to internalURL if you're running from inside the DC):
url=$(echo $json | jq '.access | .serviceCatalog | .[] | select(.name == "cloudFiles") | .endpoints | .[] | select(.region == "SYD") | .publicURL' | sed s/\"//g)
-- Hard work is done. Now it gets easy --
Get list of containers:
http "$url" $auth
Create a container:
http PUT "$url/my_container" $auth
Upload a file:
cat python1.JPG | http PUT "$url/my_container/python1.jpg" $auth
List files:
http "$url/my_container"
Get CDN API URL (not the one for downloading, that's later):
cdn_url=$(echo $json | jq ' .access | .serviceCatalog | .[] | select(.name == "cloudFilesCDN") | .endpoints | .[] | select(.region == "SYD") | .publicURL' | sed s/\"//g)
CDN enable the container:
http PUT "$cdn_url/my_container" $auth "X-Cdn-Enabled: True"
Get public CDN url for my_container:
pub_url=$(http -h HEAD "$cdn_url/my_container" $auth | awk '/X-Cdn-Uri/{print $2;}')
View your file:
firefox "$pub_url/python1.jpg"
All the API docs are here: http://docs.rackspace.com/files/api/v1/cf-devguide/content/API_Operations_for_Storage_Services-d1e942.html
Enjoy :)
This is the pattern which you should be using:
token=`cat /etc/passwd`
echo "file contents: $token"
Note, as triplee points out, that you must not have spaces on either side of the = sign.
I highly recommend skipping curl and using one of the language specific SDKs found on http://developer.rackspace.com
They all handle authentication easily and reauthentication for long lived processes. They all have examples of how to upload files too.