I am trying to feed a GeoJSON to this data service using the following bash code.
curl -X POST -F "shape=$(cat myfile.geojson)" \
-F 'age=69' -o reconstructed_myfile.geojson \
https://dev.macrostrat.org/reconstruct
However, I am getting an "Argument list too long" error. I see a lot of questions open on stack related to this issue, but I do not understand how to convert the answers given in those threads to this specific case.
You should use <filename or #filename:
curl -X POST \
-F 'shape=<myfile.geojson' \
-F 'age=69' \
-o 'reconstructed_myfile.geojson' \
-- 'https://dev.macrostrat.org/reconstruct'
See man curl for details:
$ man curl | awk '$1 ~ /-F/' RS=
-F, --form <name=content>
(HTTP) This lets curl emulate a filled-in form in which a user has
pressed the submit button. This causes curl to POST data using the
Content-Type multi‐ part/form-data according to RFC 2388. This
enables uploading of binary files etc. To force the 'content' part to
be a file, prefix the file name with an # sign. To just get the
content part from a file, prefix the file name with the symbol <. The
difference between # and < is then that # makes a file get
attached in the post as a file upload, while the < makes a text field
and just get the contents for that text field from a file.
Related
Here's the code I'm looking at:
#!/bin/bash
nc -l 8080 &
curl "http://localhost:8080" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
--data #<(cat <<EOF
{
"me": "$USER",
"something": $(date +%s)
}
EOF
)
What does the # do? Where is there documentation about #?
It is a curl-specific symbol. man curl shows you:
-d, --data <data>
(HTTP) Sends the specified data in a POST request to the HTTP server, in the
(same way that a browser does when a user has filled in an HTML form and
(presses the submit button. This will cause curl to pass the data to the
(server using the content-type application/x-www-form-urlencoded. Compare to
(-F, --form.
--data-raw is almost the same but does not have a special interpretation of
the # character. To post data purely binary, you should instead use the
--data-binary option. To URL-encode the value of a form field you may use
--data-urlencode.
If any of these options is used more than once on the same command line, the
data pieces specified will be merged together with a separating &-symbol.
Thus, using '-d name=daniel -d skill=lousy' would generate a post chunk that
looks like 'name=daniel&skill=lousy'.
If you start the data with the letter #, the rest should be a file name to
read the data from, or - if you want curl to read the data from stdin.
Multiple files can also be specified. Posting data from a file named
'foobar' would thus be done with -d, --data #foobar. When --data is told to
read from a file like that, carriage returns and newlines will be stripped
out. If you don't want the # character to have a special interpretation use
--data-raw instead.
See also --data-binary and --data-urlencode and --data-raw. This option
overrides -F, --form and -I, --head and -T, --upload-file.
I have something like that (it replaces a special sequence in request and then it sends it with curl).
SPECIAL_SEQUENCE=My_value
sed -i -e "s|SPECIAL_SEQUENCE|$SPECIAL_SEQUENCE|g" file.txt
curl http://127.0.0.1:1478/ -X POST -d #file.txt
It does work ok. However, the problem for me is that it leaves changed the file file.txt. I could undo sed at the end of this script, but I wouldn't like do this (becase I often interrupt this script with Ctrl+C).
Can you give me some other ideas to deal with this ? In other words the final form of request can be known only during executing mentioned script.
Instead of modifying the file, modify it in the memory and pipe it to curl:
SPECIAL_SEQUENCE=My_value
sed -e "s|SPECIAL_SEQUENCE|$SPECIAL_SEQUENCE|g" file.txt | curl http://127.0.0.1:1478/ -X POST
I'm trying to get a config file from our GitHub using the get contents api.
This returns a JSON containing the file content encoded as a base64 string.
I'd like to get it as text
Steps I've taken
get initial api response:
curl -H 'Authorization: token MY_TOKEN' \
https://github.com/api/v3/repos/MY_OWNER/MY_REPO/contents/MY_FILE
this returns a JSON response with a field "content": "encoded content ..."
get the encoded string:
add <prev command> | grep -F "content\":"
this gets the content, but there's still the "content": string, the " chars and a comma at the end
cut the extras:
<prev command> | cut -d ":" -f 2 | cut -d "\"" -f 2
decode:
<prev command | base64 --decode>
final command:
curl -H 'Authorization: token MY_TOKEN' \
https://github.com/api/v3/repos/MY_OWNER/MY_REPO/contents/MY_FILE | \
grep -F "content\":" | cut -d ":" -f 2 | cut -d "\"" -f 2 | base64 --decode
Issues:
the resulting string (before the base64 --decode) decodes in an online decoder (not well -> see next item), but fails to do so in bash. The response being
"Invalid character in input stream."
When decoding the string in an online decoder, some (not all) of the file is in gibberish, and not the original text. I've tried all the available charsets.
Notes:
I've tried removing the last 2 (newline) chars with sed 's/..$//', but this has no effect.
If I select the output with the mouse and copy paste it to a echo MY_ECODED_STRING_PASTED_HERE | base64 --decode command, it has the same effect as the online tool, that is, it decodes as gibberish.
Add header Accept: application/vnd.github.VERSION.raw to the GET.
Following tripleee's advice, i've switched the extracting method to jq
file=randomFileName74894031264.txt
curl -H 'Authorization: token MY_TOKEN' https://github.com/api/v3/repos/MY_OWNER/MY_REPO/contents/MY_FILE > "$file"
encoded_str=($(jq -r '.content' "$file"))
echo "$encoded_str" | base64 -D
rm -f "$file"
This works when running from the command line, but when running as a script the stdout doesn't flush, and we only get the first few lines of the file.
I will update this answer when I've formalized a generic script.
Looking at the man page for cURL:
-w, --write-out <format>
Make curl display information on stdout after a completed transfer.
Where it is possible to use this flag and append a string to the output of cURL. However I can only get this to append to the end of the output of cURL, because as the man page suggests, the -w flag appends after a completed transfer.
so doing:
curl -sS "http:/somewebsite" -w "hello_world"
will produce:
$
contentfromcurl
hello_world
....well how do you get the output to be
$
hello_worldcontentfromcurl
i.e. is it possible to get -w to prepend rather than append?
thanks to #Adrian, this is the final answer -
curl -sS "http:/somewebsite" | xargs echo "mystring"
cheers!
If you're really desperate you can make a code block and include an echo. The following will have the output you're looking for:
{ echo -n "hello_world"; curl -sS "http:/somewebsite"; }
As for getting the -w option to prepend, the answer is no:
-w, --write-out
Make curl display information on stdout after a completed transfer. The format is a string ...
Is this what you are after?
$ printf "bar\nquux\n"
bar
quux
$ printf "bar\nquux\n" | sed 's#^#foo#g'
foobar
fooquux
Obviously, you would replace printf with your curl invocation.
But this seems a bit like an XY-problem - what are you trying to accomplish?
I was reading the JIRA REST API tutorial and in their example curl request, they show
curl -D- -u username:password <rest-of-request>
What is the -D- syntax with the dash before and after?
To quote man curl:
-D, --dump-header <file>
Write the protocol headers to the specified file.
This option is handy to use when you want to store
the headers that a HTTP site sends to you. C
After a -D you normally give the name of the file where you want to dump the headers. As with many utilities, - is recognized as an alias to stdout. (if you're not familiar with that concept: when you launch the command from a terminal without redirection, stdout is the "terminal screen")
The -D- (without space) form is exactly the same as -D - (or on Linux at least, -D /dev/stdout)