I am trying to connect to USPS portal using curl command.
In hadoop its working fine.
curl -i "https://secure.shippingapis.com/ShippingAPI.dll?API=Verify&XML=<AddressValidateRequest USERID="XXXXXXXXX">"
* About to connect() to secure.shippingapis.com port 443 (#0)
* Trying 56.0.34.44...
* Connected to secure.shippingapis.com (56.0.34.44) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=secure.shippingapis.com,OU=Unified Communications,OU=Hosted by United States Postal Service,OU=WebTools,O=United States Postal Service,STREET=2825 Lone Oak Parkway,L=Eagan,ST=MN,postalCode=55121,C=US
* start date: May 24 00:00:00 2019 GMT
* expire date: May 23 23:59:59 2020 GMT
* common name: secure.shippingapis.com
* issuer: CN=COMODO SHA-256 Organization Validation Secure Server CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB
> GET /ShippingAPI.dll?API=Verify&XML=<AddressValidateRequest USERID=XXXXXXXXX> HTTP/1.1
> User-Agent: curl/7.29.0
> Host: secure.shippingapis.com
> Accept: */*
However when i try to send a request and get response back from server, it is unable to do so .
curl -i -X GET AddressValidateRequest.txt "https://secure.shippingapis.com/ShippingAPI.dll?API=Verify&XML=<AddressValidateRequest USERID="XXXXXXXXXXXX">
<Address>
<Address1>6406 IVY LANE</Address1>
<Address2></Address2>
<City>GREENBELT</City>
<State>TX</State>
<Zip5>20770</Zip5>
<Zip4></Zip4>
</Address>
</AddressValidateRequest>"
USERID - usps shiping authentication ID
curl -v -H "Content-Type: application/xml" -d > 'API=RateV4&XML=<RateV4Request USERID="USERID"><Revision>2</Revision><Package ID="1ST"><Service>Priority Mail Express</Service><FirstClassMailType>PACKAGE SERVICE RETAIL</FirstClassMailType><ZipOrigination>33016</ZipOrigination><ZipDestination>35004</ZipDestination><Pounds>1</Pounds><Ounces>0</Ounces><Container/><Width>1</Width><Length>1</Length><Height>1</Height><Machinable>true</Machinable></Package></RateV4Request>' -X POST 'https://secure.shippingapis.com/ShippingAPI.dll'
XML is typically not appended to the end of a URL, but your quotes are off since you have double quotes within double quotes.
Also try removing all the whitespace, and not sure what AddressValidateRequest.txt is trying to do
You can also remove the -i -v -X GET if you only want to see the response
curl 'https://secure.shippingapis.com/ShippingAPI.dll?API=Verify&XML=<AddressValidateRequest USERID="XXXXXXXXXXXX"><Address>...</Address></AddressValidateRequest>'
Related
I feed cURL with multiple URLs at a time, and have a difficulty parsing the output log to get the original addresses back. Namely, if an URL resolves, the output is as follows:
$ curl --head --verbose https://www.google.com/
* Trying 64.233.165.106...
* TCP_NODELAY set
* Connected to www.google.com (64.233.165.106) port 443 (#0)
<...>
> HEAD / HTTP/2
> Host: www.google.com
<...>
which can eventually be parsed back to https://www.google.com/.
However, with an invalid URL it does not do:
$ curl --head --verbose --connect-timeout 3 https://imap.gmail.com/
* Trying 74.125.131.109...
* TCP_NODELAY set
* After 1491ms connect time, move on!
* connect to 74.125.131.109 port 443 failed: Operation timed out
<...>
* Failed to connect to imap.gmail.com port 443: Operation timed out
The error message contains the URL in this case, but in other cases it does not. I can't rely on it.
So, I need either have URL-to-IP resolving disabled in the output, like
* Trying https://imap.gmail.com/...
or somehow append each URL from the list to the corresponding output, like:
$ curl --head --verbose --connect-timeout 3 https://imap.gmail.com/ https://www.google.com/
https://imap.gmail.com/
* Trying 64.233.162.108...
* TCP_NODELAY set
* After 1495ms connect time, move on!
* connect to 64.233.162.108 port 443 failed: Operation timed out
<...>
https://www.google.com/
* Trying 74.125.131.17...
* TCP_NODELAY set
* Connected to www.gmail.com (74.125.131.17) port 443 (#0)
<...>
Wget or HTTPie are not an option. How one can achieve that with cURL?
Perhaps this is the solution:
while read LINE ; do
print "REQUESTED URL: $LINE" >> output.txt;
curl $LINE >> output.txt 2>&1;
done < url-list.txt
Starting with curl v.7.75.0, it is possible with the --write-out '%{url}' option to make curl display the URL that was fetched.
I have NiFi running on a test environment and would like it to automatically start all processors as the last step of deployment.
Do I need to parse all the processor ids and hit the /nifi-api/processors endpoint for each one or is it possible to start all - eg from the /nifi-api/process-groups endpoint ?
My NiFi flow has uuid 66f83c1d-0162-1000-baff-01e60296540a - this GET curl statement returns information successfully:
curl -i -X GET http://localhost:9090/nifi-api/process-groups/66f83c1d-0162-1000-baff-01e60296540a
However trying to PUT a status fails:
curl -i -X PUT -H 'Content-Type: application/json' -d '{"status":"RUNNING"}' http://localhost:9090/nifi-api/process-groups/66f83c1d-0162-1000-baff-01e60296540a
HTTP/1.1 400 Bad Request
Date: Mon, 06 Aug 2018 11:08:15 GMT
X-Frame-Options: SAMEORIGIN
Content-Type: text/plain
Vary: Accept-Encoding
Content-Length: 429
Server: Jetty(9.4.3.v20170317)
Cannot construct instance of `org.apache.nifi.web.api.dto.status.ProcessGroupStatusDTO` (although at least one Creator exists): no String-argument constructor/factory method to deserialize from String value ('RUNNING')
at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 11] (through reference chain: org.apache.nifi.web.api.entity.ProcessGroupEntity["status"])
Yes by using NiFi RestApi we can start all processors in a processor group.
As you are missing id key value in your json,
{"id":"66f83c1d-0162-1000-baff-01e60296540a","state":"RUNNING"}
we need to mention state not status to run/stop processors in the group
{"id":"66f83c1d-0162-1000-baff-01e60296540a","state":"RUNNING"}
and we need to include flow in the address of the rest api.
http://localhost:9090/nifi-api /flow/ process-groups/66f83c1d-0162-1000-baff-01e60296540a
Try with below Curl command
curl -i -X PUT -H 'Content-Type: application/json' -d '{"id":"66f83c1d-0162-1000-baff-01e60296540a","state":"RUNNING"}' http://localhost:9090/nifi-api/flow/process-groups/66f83c1d-0162-1000-baff-01e60296540a
Refer to this for more details regarding start process group using RestAPI.
I am trying to use the following curl
curl -k 'https://myhost.com:9091/nifi-api/access/token' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' --data "username=$USERNAME&password=$PASSWORD" --compressed —insecure
But get an error with a % in the password saying
URLDecoder: Incomplete trailing escape (%) pattern
I have seen suggestions online to use --data-urlencode but then I get the following error.
The username and password must be specified.
How can I resolve this issue?
--data-urlencode is the way to go adding it multiple times, one per parameter
userh='vf&re'
passwordh='asdaf%'
curl 'http://127.0.0.1:8080/' --data-urlencode "username=$userh" --data-urlencode "password=$passwordh"
Let's start a netcat listener to check what was sent
netcat -l 8080
Received data:
POST / HTTP/1.1
Host: 127.0.0.1:8080
User-Agent: curl/7.60.0
Accept: */*
Content-Length: 34
Content-Type: application/x-www-form-urlencoded
username=vf%26re&password=asdaf%25
Sending all parameters in one option will not work since password parameter gets lost triggering the server error "The username and password must be specified."
curl 'http://127.0.0.1:8080/' --data-urlencode "username=$userh&password=$passwordh"
Response:
POST / HTTP/1.1
Host: 127.0.0.1:8080
User-Agent: curl/7.60.0
Accept: */*
Content-Length: 38
Content-Type: application/x-www-form-urlencoded
username=vf%26re%26password%3Dasdaf%25
First command from OP with --data will not work either as it sends the % sign as is triggering a server error that "sees" and incomplete percent encoded request.
POST / HTTP/1.1
Host: 127.0.0.1:8080
User-Agent: curl/7.60.0
Accept: */*
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Content-Length: 33
username=asdasdas&password=asdaf%
Based on this answer.
I'm under a proxy, and if I try curl http://localhost/mysite or curl http://127.0.0.1/mysite curl try to resolve it with the proxy. So I tried with --noproxy option, but doesn't work. Is working ok for external servers with the proxy as curl http://mysite.com.
My configuration:
Cygwin (bash) under Windows 8 with curl extension.
Proxy: proxy.domain.xx:1080 without authentication
http_proxy=http://proxy.domain.xx:1080
Local Server: XAMP Version 1.8.0
Apache ports: 80,443
Browser: Chrome with proxy, but configured to access to localhost and *.dev
From the curl --help
--noproxy : Comma-separated list of hosts which do not use proxy
What I tried:
I have deactivated the firewall and nothing
$ curl -v http://localhost/mysite -> Debug:
Response
Connected to proxy.domain.xx (200.55.xxx.xx) port 1080 (#0)
GET http://localhost/mysite HTTP/1.1
User-Agent: curl/7.21.1 (i686-pc-mingw32) libcurl/7.21.1 OpenSSL/0.9.8r zlib/1.2.3
Host: localhost
Accept: */*
Proxy-Connection: Keep-Alive
The system returned: <PRE><I>(111) Connection refused</I></PRE>
curl -v --noproxy localhost, http://localhost/muestra
Response
About to connect() to localhost port 80 (#0)
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET /mysite HTTP/1.1
> User-Agent: curl/7.21.1 (i686-pc-mingw32) libcurl/7.21.1 OpenSSL/0.9.8r zlib/1.2.3
> Host: localhost
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Server: Apache/2.4.2 (Win32) OpenSSL/1.0.1c PHP/5.4.4
< Location: http://localhost/mysite
< Content-Length: 331
< Content-Type: text/html; charset=iso-8859-1
Any idea how to fix this?
After
curl -v --noproxy localhost, http://localhost/muestra
curl responded with
About to connect() to localhost port 80 (#0)
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 80 (#0)
So it clearly stated that it connected to localhost.
try below, to bypass proxy service for local ips
export no_proxy=localhost,127.0.0.1
use
curl -v --noproxy '*' http://abc.com
to completely disable proxy.
or if you want to disable proxy for only abc.com destination
curl -v --noproxy "abc.com" http://abc.com
where abc.com is the url you want to go
As some others said, the --noproxy options is what you are looking for.
https://curl.haxx.se/docs/manpage.html#--noproxy
Apparently, the second request you tried was receiving a HTTP 301 response, so you probably also want to use the -L option to follow redirects:
https://curl.haxx.se/docs/manpage.html#-L
You can alias curl to always ignore proxies for localhost requests.
alias curl='curl --noproxy localhost,127.0.0.1'
Add it to your .bashrc file for convenience:
echo "alias curl='curl --noproxy localhost,127.0.0.1'" >> ~/.bashrc
In Windows, the following option worked for me for connecting to localhost.
curl --proxy "" --location http://127.0.0.1:8983
Curl expects the port to be specified with proxy this solution worked for me
export http_proxy="http://myproxy:80"
As a workaround, you can just unset the variable http_proxy before running your curl without the option --noproxy.
Just been tinkering with Sinatra and trying to get a bit of a restful web service going.
The error I'm getting at the moment is very specific though.
Take this example post method
post '/postMan/:someParam' do
#Edited here. This code can be anything. 411 is still the response
puts params[:someParam]
end
Seems simple enough. Take a param, make an object out of it, then go store it in whatever way the objects save method defines.
Heres what I use to post the data using Curl
$curl -I -X POST http://127.0.0.1/postman/123456
The only problem is, I'm getting 411 back and have no idea why.
To the best of my knowledge, 411 is length required. Here is the trace
HTTP/1.1 411 Length Required
Content-Type: text/html; charset=ISO-8859-1
Server: WEBrick/1.3.1 (Ruby/1.9.2/2011-07-09)
Date: Fri, 02 Mar 2012 22:27:09 GMT
Content-Length: 303
Connection: close
I 'Cannot' change the curl message in any way. So might anyone have a way to set the content length to be ignored in sinatra? Or some fix which doesn't involve changing the curl request?
For the record, it doesn't matter whether I use the parameters in the Post method or not. I could have some crazy code inside it, it will still throw the same error
As others said above, WEBrick wrongly requires POST requests to have a Content-Length header. I just pass an empty body, because it's less typing than passing in the header:
curl -X POST -d '' http://webrickwhyyounotakeemptyposts.com/
Are you sure you're on port 80 for your app?
When I run:
ruby -r sinatra -e "post('/postMan/:someParam'){puts params[:someParam]}"
and curl it:
curl -I -X POST http://127.0.0.1:4567/postMan/123456
HTTP/1.1 200 OK
X-Frame-Options: sameorigin
X-XSS-Protection: 1; mode=block
Content-Type: text/html;charset=utf-8
Content-Length: 0
Connection: keep-alive
Server: thin 1.3.1 codename Triple Espresso
it's ok. Had to change the URL to postManthough, your example threw a 404because you had postman.
The output was also as expected:
== Sinatra/1.3.2 has taken the stage on 4567 for development with backup from Thin
>> Thin web server (v1.3.1 codename Triple Espresso)
>> Maximum connections set to 1024
>> Listening on 0.0.0.0:4567, CTRL+C to stop
123456
Ah. Try it without -I. It's probably sending a HEAD request and as such, not sending what you expect. Use -v if you want to show the headers.
curl -v -X POST http://127.0.0.1/postman/123456
curl -v -X POST -d "key=val" http://127.0.0.1/postman/123456
WEBrick erroneously requires POST requests to include the Content-Length header.
curl -H 'Content-Length: 0' -X POST http://example.com
Standardly, however, POST requests don't require a body and therefore don't require the Content-Length header.