I managed to get the curl output of a web-page that using site minder authentication by manually extracting the cookie from firefox and using it in the curl command:
curl "example.com/firstexample" -H "Host: example.com" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" -H "Accept-Language: en-US,en;q=0.5" --compressed -H "Content-Type: application/x-www-form-urlencoded" -H "X-Requested-With: XMLHttpRequest" -H "Referer: http://example.com" -H --cookie-jar cookies.txt --cookie cookies.txt -H "Connection: keep-alive"
However, the curl works only as long as the session lasts. Once a new session starts i.e. the cookie session id and other stuff changes - the curl no longer works obviously.
Is there a way to get the subsequent cookies/session ids via curl to continue seemlessly.
Check out the --cookie and --cookie-jar options on curl in the man page. They should do what you need.
Related
I am trying to create a DbcpController service from nifi rest api. I am getting the proper response also but when i go to UI, The controller service is not visible.
curl -i -X POST -H 'Content-Type:application/json'
-d '{"revision":{"clientId":"a09f2359-0165-1000-bf28-1dca98f6e259","version":0},
"permissions":{"canRead":true,"canWrite":true},"bulletins":[],
"component":{"name":"Dynamic-Pool","type":"org.apache.nifi.dbcp.DBCPConnectionPool",
"bundle":{"group":"org.apache.nifi","artifact":"nifi-dbcp-service-nar","version":"1.6.0"},
"controllerServiceApis":[{"type":"org.apache.nifi.dbcp.DBCPService",
"bundle":{"group":"org.apache.nifi","artifact":"nifi-standard-services-api-nar","version":"1.6.0"}}],
"comments":"","state":"DISABLED","persistsState":false,"restricted":false,"deprecated":false,
"multipleVersionsAvailable":true,"properties":{"Database Connection URL":"jdbc:oracle:thin:#144.21.86.182:1521/ORCL.607022672.oraclecloud.internal",
"Database Driver Class Name":"oracle.jdbc.driver.OracleDriver",
"database-driver-locations":"/home/siddharth/Downloads/ojdbc7.jar","Database User":"test","Password":"***",
"Max Wait Time":"500 millis","Max Total Connections":"8","Validation-query":null},"descriptors":{ } } }'
http://localhost:9090/nifi-api/controller/controller-services/
Response :
HTTP/1.1 201 Created
Date: Mon, 03 Sep 2018 18:19:28 GMT
X-Frame-Options: SAMEORIGIN
Location: http://localhost:9090/nifi-api/controller-services/a0a93568-0165-1000-0c3b-ff9494571881
Content-Type: application/json
Vary: Accept-Encoding
Vary: User-Agent
Content-Length: 3614
Server: Jetty(9.4.3.v20170317)
If i click the location the details also comes up.
Is there any parameter in the request which i am missing.
I am not sure what "When I go to UI, the controller service is not visible" means.
When you go to the Apache NiFi UI, and click Global Menu > Controller Settings > Controller Services, you do not see any Controller Services listed -- Possibly refresh your browser and clear the cache? I don't think this is your issue
The Controller Service is listed but it is not available to use in a processor -- this is because the CS is being added to the global level, which means it is available for Reporting Tasks to reference, but not processors on the canvas. You'll need to provide a parentGroupId in the ControllerServiceDTO object in the JSON in order for the CS to be available to a component on the canvas (in that parent group). Your URL will look like this: /nifi-api/process-groups/a0f3dfb7-0165-1000-8310-4fb72628adaa/controller-services. I think this is your issue
The Controller Service is available to use in a processor but it is not enabled -- this is because you are setting the state of the CS to DISABLED. You can issue an additional command which enables the CS, or set the state to ENABLED initially
If you right click on the canvas and select Configure, you'll get a dialog for the Process Group you're in (called "NiFi Flow" by default at the root level), and there is a tab for "Controller Services", which will list the CS you have created.
You can always perform these operations in the UI and use your browser's Developer Tools panel to monitor the requests -- NiFi's UI is a consumer of the REST API, so any activity performed there can be done using the API alone.
Examples:
Add a reporting task controller service
curl 'http://localhost:8080/nifi-api/controller/controller-services' -H 'Origin: http://localhost:8080' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-US,en;q=0.9' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36' -H 'Content-Type: application/json' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: http://localhost:8080/nifi/' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' -H 'DNT: 1' --data-binary '{"revision":{"clientId":"a0f490b8-0165-1000-a521-4d24892f12c0","version":0},"disconnectedNodeAcknowledged":false,"component":{"type":"org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService","bundle":{"group":"org.apache.nifi","artifact":"nifi-aws-nar","version":"1.8.0-SNAPSHOT"}}}' --compressed
Enable a reporting task controller service
curl 'http://localhost:8080/nifi-api/controller-services/a0f4dbe2-0165-1000-8cd2-dd50d7dba48d' -X PUT -H 'Origin: http://localhost:8080' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-US,en;q=0.9' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36' -H 'Content-Type: application/json' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: http://localhost:8080/nifi/' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' -H 'DNT: 1' --data-binary '{"revision":{"clientId":"a0f490b8-0165-1000-a521-4d24892f12c0","version":1},"disconnectedNodeAcknowledged":false,"component":{"id":"a0f4dbe2-0165-1000-8cd2-dd50d7dba48d","state":"ENABLED"}}' --compressed
Add a controller service available to a component on the canvas
curl 'http://localhost:8080/nifi-api/process-groups/a0f3dfb7-0165-1000-8310-4fb72628adaa/controller-services' -H 'Origin: http://localhost:8080' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-US,en;q=0.9' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36' -H 'Content-Type: application/json' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: http://localhost:8080/nifi/?processGroupId=root&componentIds=a0f7e36a-0165-1000-ef42-c59f18644d42' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' -H 'DNT: 1' --data-binary '{"revision":{"clientId":"a0f490b8-0165-1000-a521-4d24892f12c0","version":0},"disconnectedNodeAcknowledged":false,"component":{"type":"org.apache.nifi.ssl.StandardRestrictedSSLContextService","bundle":{"group":"org.apache.nifi","artifact":"nifi-ssl-context-service-nar","version":"1.8.0-SNAPSHOT"},"name":"StandardRestrictedSSLContextService"}}' --compressed
I am trying to retrieve the search results from virustotal for ransomware keyword, where I am trying to use the following code:
wget "https://www.virustotal.com/#/search/ransomware" -O output.txt
However, I do not get the result output. I am expecting to get at least the hash value of each resulted sample, where I am planning to use them later on.
Any advice?
You can use either curl or wget to do this.
Using Curl
curl 'https://www.virustotal.com/ui/search?query=ransomware&relationships\[url\]=network_location%2Clast_serving_ip_address&relationships\[comment\]=author%2Citem' -H 'pragma: no-cache' -H 'cookie: _ga=GA1.2.1407677727.1526818525; _gid=GA1.2.163855722.1526818525; _gat=1' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: en-US,en;q=0.9' -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36' -H 'accept: application/json' -H 'cache-control: no-cache' -H 'authority: www.virustotal.com' -H 'referer: https://www.virustotal.com/' --compressed
Using wget
wget 'https://www.virustotal.com/ui/search?query=ransomware&relationships\[url\]=network_location%2Clast_serving_ip_address&relationships\[comment\]=author%2Citem' -O output.txt
I used Chrome to log into my mLab account and created a mongo database. I then opened the browser devtools and created a document in the database. In the log of the devtools, I see the creation of the document via a POST "createdocument?CSRF_TOKEN....". When I right click on this POST, I can capture the command as a cURL cmd or a cURL base.
This is the cURL cmd:
curl "https://mlab.com/createdocument?CSRF_TOKEN=SBY9-RTVO-YW1P-LBJ5-EJUY-WYDO-Z7DS-REWE-WOPW-TIQ0-9UW4-LMA4-97IE-6FUS-TDTN-C9N9" -H "Connection: keep-alive" -H "Cache-Control: max-age=0" -H "Origin: https://mlab.com" -H "Upgrade-Insecure-Requests: 1" -H "Content-Type: application/x-www-form-urlencoded" -H "User-Agent: Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.117 Mobile Safari/537.36" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8" -H "Referer: https://mlab.com/databases/test-db/collections/users?_id=_new^&pageSize=10^&pageNum=0^&totalCount=0^&" -H "Accept-Encoding: gzip, deflate, br" -H "Accept-Language: en-US,en;q=0.9" -H "Cookie: __utmc=234932573; __utmz=234932573.1524622777.1.1.utmcsr=(direct)^|utmccn=(direct)^|utmcmd=(none); __utmv=234932573.^|1=account_status=HAS_ACCOUNT=1; __utma=234932573.2132662725.1524622777.1524622777.1524646055.2; __utmt=1; JSESSIONID=1DAFA423980734AB9AB4D6A3A7031E9F.worker2; __utmb=234932573.2.10.1524646055" --data "CSRF_TOKEN=SBY9-RTVO-YW1P-LBJ5-EJUY-WYDO-Z7DS-REWE-WOPW-TIQ0-9UW4-LMA4-97IE-6FUS-TDTN-C9N9^&r=https^%^3A^%^2F^%^2Fmlab.com^%^2Fdatabases^%^2Ftest-db^%^2Fcollections^%^2Fusers^%^3F_id^%^3D_new^%^26pageSize^%^3D10^%^26pageNum^%^3D0^%^26totalCount^%^3D0^%^26^&db=test-db^&collection=users^&id=^&json=^%^7B^%^0D^%^0A++++^%^22fname^%^22^%^3A+^%^22Wile^%^22^%^2C^%^0D^%^0A++++^%^22lname^%^22^%^3A+^%^22Coyote^%^22^%^2C^%^0D^%^0A++++^%^22occupation^%^22^%^3A+^%^22Genius^%^22^%^0D^%^0A^%^7D^&stay=false^&pageSize=10^&pageNum=0^&totalCount=0^&q=^&s=^&f=^&CSRF_TOKEN=SBY9-RTVO-YW1P-LBJ5-EJUY-WYDO-Z7DS-REWE-WOPW-TIQ0-9UW4-LMA4-97IE-6FUS-TDTN-C9N9" --compressed
and this is the cURL bash:
curl 'https://mlab.com/createdocument?CSRF_TOKEN=SBY9-RTVO-YW1P-LBJ5-EJUY-WYDO-Z7DS-REWE-WOPW-TIQ0-9UW4-LMA4-97IE-6FUS-TDTN-C9N9' -H 'Connection: keep-alive' -H 'Cache-Control: max-age=0' -H 'Origin: https://mlab.com' -H 'Upgrade-Insecure-Requests: 1' -H 'Content-Type: application/x-www-form-urlencoded' -H 'User-Agent: Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.117 Mobile Safari/537.36' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'Referer: https://mlab.com/databases/test-db/collections/users?_id=_new&pageSize=10&pageNum=0&totalCount=0&' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en-US,en;q=0.9' -H 'Cookie: __utmc=234932573; __utmz=234932573.1524622777.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmv=234932573.|1=account_status=HAS_ACCOUNT=1; __utma=234932573.2132662725.1524622777.1524622777.1524646055.2; __utmt=1; JSESSIONID=1DAFA423980734AB9AB4D6A3A7031E9F.worker2; __utmb=234932573.2.10.1524646055' --data 'CSRF_TOKEN=SBY9-RTVO-YW1P-LBJ5-EJUY-WYDO-Z7DS-REWE-WOPW-TIQ0-9UW4-LMA4-97IE-6FUS-TDTN-C9N9&r=https%3A%2F%2Fmlab.com%2Fdatabases%2Ftest-db%2Fcollections%2Fusers%3F_id%3D_new%26pageSize%3D10%26pageNum%3D0%26totalCount%3D0%26&db=test-db&collection=users&id=&json=%7B%0D%0A++++%22fname%22%3A+%22Wile%22%2C%0D%0A++++%22lname%22%3A+%22Coyote%22%2C%0D%0A++++%22occupation%22%3A+%22Genius%22%0D%0A%7D&stay=false&pageSize=10&pageNum=0&totalCount=0&q=&s=&f=&CSRF_TOKEN=SBY9-RTVO-YW1P-LBJ5-EJUY-WYDO-Z7DS-REWE-WOPW-TIQ0-9UW4-LMA4-97IE-6FUS-TDTN-C9N9' --compressed
I copied the cURL cmd into a file called create-document.sh and added a -i after the curl but before the "http.....
I then open Git Bash (I'm running on a Windows 10 system) and I execute the following statement:
./create-document.sh
This is the response I get back:
$ ./create-document.sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 637 0 0 100 637 0 1454 --:--:-- --:--:-- --:--:--
1454HTTP/1.1 302 Found
Date: Wed, 25 Apr 2018 08:55:54 GMT
Server: Apache
Strict-Transport-Security: max-age=15552000
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
Location: https://mlab.com/databases/test-db/collections/users?_id=_new^&pageSize=10^&pageNum=0^&totalCount=0^&
Content-Length: 0
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html;charset=UTF-8
The status returned is 302 which I am being told is a successful response. I was expecting this to add another document to my database. However, when I look at the database, I only see the original document.
I then removed the contents of create-document.sh and copied in the cURL bash command. I inserted the -i again and saved the script. Again, I executed the ./create-document.sh command, received the response with a status of 302, and checked the mongo database. Again, no new record.
Any suggestions on how to create a document with the cURL command?
Thank you.
I have tried to export a text file resulted by an request to a .jsp page with Java script enabled from Oracle BI Publisher framework with below command in Unix:
curl 'http://server/xmlpserver/report.xdo&_xt=New+Template+1&_xf=text'
-H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8'
-H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36'
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
-H 'Referer: http://server/xmlpserver/report.xdo'
-H 'Cookie:
JSESSIONID=qq!11!22; ORA_XDO_UI=en_US;
JSESSIONID=qq!11; oracle.uix=0^^GMT+3:00^p'
-H 'Connection: keep-alive'
--compressed -o test.txt
This command is 'copy as bash curl command' from Google Chrome debugger and work it but syntax command use id JSESSION in cookie who will expire.
I need to login on this .jsp page to generate cookie with JSESSION=qq!11!22 and JSESSION=qq!11 (see above cUrl syntax from Google Chrome).
For this I have used command:
curl -c cookies.txt -d "user=user&pass=pass" http://server/xmlpserver/login.jsp, and was generated only one "JSESSIONID=qq!11;".
I need to generate one more JSESSIONID=qq!11!22.
Thanks!
I am developing a module for a web application. To trigger this module, I need to submit some data to the server. For simple forms, Copy as cURL in the Chrome Developer Tools works fine (using curl from msys[git]), but for post requests with multipart/form-data, the copied string is neither usable in the windows shell (cmd) nor with bash (form msys); the copied text is similar to:
curl "http://myserver.local" -H "Origin: http://wiki.selfhtml.org" -H "Accept-Encoding: gzip, deflate" -H "Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130 Safari/537.36" -H "Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryntXdlWbYXAVwCIMU" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" -H "Cache-Control: max-age=0" -H "Referer: http://wiki.selfhtml.org/extensions/Selfhtml/frickl.php/Beispiel:HTML_form-Element1.html" -H "Connection: keep-alive" --data-binary "------WebKitFormBoundaryntXdlWbYXAVwCIMU"^
"Content-Disposition: form-data; name=""area"""^
"multi"^
"line"^
"------WebKitFormBoundaryntXdlWbYXAVwCIMU--"^
"" --compressed
Is there any way I could use this or convert it to something usable?
Chrome, as well as the other browsers actually, do a rather poor job of translating multi-part formposts into curl command lines.
A much more convenient curl command line would not use --data-binary for that, it would use --form. And then you want one --form per input field.
In your case, it probably would look something like (backslashes inserted here
for visibility):
curl "http://myserver.local" \
--compressed \
-H "Origin: http://wiki.selfhtml.org" \
-A "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130 Safari/537.36" \
-H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" \
-H "Cache-Control: max-age=0" \
-e "http://wiki.selfhtml.org/extensions/Selfhtml/frickl.php/Beispiel:HTML_form-Element1.html" \
-F area=[contents]
I left the [contents] in there, but it should be replaced with what you actually want in the area field. You could also pass it from a file if you prefer to.
I removed two unnecessary -H uses, and I replaced two to use the direct curl options.
h2c - headers to curl
Advice for the future: figure out the exact HTTP header trace you want to reproduce with a curl command line and paste it over at https://curl.se/h2c/ .