I want to extract a string from the URL. I have the following URL :
https://optim.actiontec.com/aei-api/users/R1RCQTY1MjA1MDYyMDc%3D
I want to extract the string after users/ and store it in a variable that I can use. I tried using the regular expression extractor but it did not work.
My second issue is extracting stuff from request headers. I dont know much about it but can we extract stuff from Request headers?
This is my request header -
GET /aei-api/main/R1RCQTY1MjA1MDYyMDc%3D HTTP/1.1
Host: optim.actiontec.com
Connection: keep-alive
Accept: application/json, text/plain, */*
X-Token: 03a580b082140ee7
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36
X-User-Id: R1RCQTY1MjA1MDYyMDc=
Referer: https://optim.actiontec.com/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
And I am looking to extract X-User-Id.I want to use the header value and pass it to other headers. Is it possible?
Thanks
The Regex configuration would be like this:
Field to check: Request Header
Reference Name: var
Regular Expression: X-User-Id: (\w+.)
Template: $1$
Match No: 1
Screenshot showing regex test:
For Regex test see here: https://regex101.com/r/MMhn3i/1/
Related
This is the code I am trying to run in scrapy shell to get the headline of the article from dailymail.co.uk.
headline = response.xpath("//div[#id='js-article-text']/h2/text()").extract()
$ scrapy shell "https://www.dailymail.co.uk/tvshowbiz/article-8257569/Shia-LaBeouf-revealed-heavily-tattoo-torso-goes-shirtless-run-hot-pink-shorts.html"
Set up an user-agent with your request and it should work :
scrapy shell -s USER_AGENT="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:52.0) Gecko/20100101 Firefox/52.0" "https://www.dailymail.co.uk/tvshowbiz/article-8257569/Shia-LaBeouf-revealed-heavily-tattoo-torso-goes-shirtless-run-hot-pink-shorts.html"
response.xpath("//div[#id='js-article-text']/h2/text()").extract()
Output :
Shia LaBeouf reveals his heavily tattoo torso as he goes shirtless for a run in hot pink shorts
How can I give user agent definition to phantomjs, I am currently running using following command on aws server ec2 instance
phantomjs --web-security=no --ssl-protocol=any --ignore-ssl-errors=true driver.js http://example.com
You can set a user agent in PhantomJS only in script (driver.js in your example). The documentation about it: http://phantomjs.org/api/webpage/property/settings.html
If you want to pass user agent to PhantomJS in a command line, you can use a parameter. In the script you can take the parameter and set it as a user agent. You can try an example below:
var webPage = require('webpage');
var system = require('system');
var page = webPage.create();
var userAgent = system.args[1];
page.settings.userAgent = userAgent;
console.log('user agent: ' + page.settings.userAgent);
phantom.exit();
Running it as follows:
$ phantomjs ua.js "Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120
Safari/537.36"
you will get output:
user agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36
We face a complicated issue in Apache web server running n Linux where intermittently Apache gives 5XX error for some of the URLs and and that too not continuously. Its like starts with few requests and grows in timely manner. The issue resolves once we restart the Apache.
We are trying to fix the issue but we need a work around till the time where we need to put a script to monitor the access log of Apache server and whenever the issue occurs we have to restart the Apache.
We thought a shell script like tailing the log and grep all 5xx errors to a separate file and another shell script which will be triggered by cron will check the file if the error is repeated for number of times within a mentioned time.
My problem is the uRLs are not always same and so I have to grep the file which has the all 5XX errors and need to see if URLs are repeated and time also.
Can anyone suggest me some logic how i can filter the errors like. I tried to be clear but not sure if this is correct way of explaining the issue.
The logs are bit modified with values but format is same.
x.x.x.x, y.y.y.y - - [11/May/2016:08:29:05 +0800](0) "HTTPS" "GET /html/js/barebone.jsp?browserId=other&themeId=expressportal_WAR_expressportaltheme&colorSchemeId=01&minifierType=js&minifierBundleId=javascript.barebone.files&languageId=en_US&b=6200&t=1462268846000 HTTP/1.1" 502 319 "https://myportal.test.com/web/guest/home" "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36"
x.x.x.x, y.y.y.y - - [11/May/2016:08:29:05 +0800](0) "HTTPS" "GET /combo/?browserId=other&minifierType=&languageId=en_US&b=6200&t=1462268846000&/html/js/aui/event-touch/event-touch-min.js&/html/js/aui/event-move/event-move-min.js HTTP/1.1" 502 319 "https://myportal.test.com/web/guest/home" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36"
x.x.x.x, y.y.y.y - - [11/May/2016:08:29:05 +0800](0) "HTTPS" "GET /html/js/liferay/available_languages.jsp?browserId=other&themeId=expressportal_WAR_expressportaltheme&colorSchemeId=01&minifierType=js&languageId=en_US&b=6200&t=1462268846000 HTTP/1.1" 502 319 "https://myportal.test.com/web/guest/home" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36"
x.x.x.x, y.y.y.y - - [11/May/2016:08:29:05 +0800](0) "HTTPS" "GET /combo/?browserId=other&minifierType=&languageId=en_US&b=6200&t=1462268846000&/html/js/aui/widget-stack/assets/skins/sam/widget-stack.css HTTP/1.1" 502 319 "https://myportal.test.com/web/guest/home" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36"
Are you 100% sure a restart fix the 500 errors ? If so, this line in the crontab should do:
tail -n 100 /var/log/apache2/error.logs | awk '{if ($9 >= 500) {nb += 1}} END {if (nb > 10) {exit 1}}' /var/log/apache2/access.log || service apache2 restart
It means that if there's more than 10 errors in the last 100 lines: restart. You may change the values for your specific problem.
First think I can think is: upgrade your Apache if it's not up to date.
I'm trying to build a curl command that will upload my xpi to validation at Mozilla Add-on Validator.
I've captured the traffic with fiddler to see what the site is doing and try to mimic it's behavior from curl.
I've seen whats the post url, and tried curl https://addons.mozilla.org/en-US/developers/standalone-upload --form 'upload=#path-to-xpi.xpi' and also with csrfmiddlewaretoken=#path-to-xpi.xpi. And also tried adding the Content-Type.
This is the raw data from the first request:
POST https://addons.mozilla.org/en-US/developers/standalone-upload HTTP/1.1
Host: addons.mozilla.org
Connection: keep-alive
Content-Length: 41021
Origin: https://addons.mozilla.org
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryI0LeCjkeJnsfGipU
Accept: */*
Referer: https://addons.mozilla.org/en-US/developers/addon/validate
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8,he;q=0.6
Cookie: _ga=GA1.2.710702590.1450107779; sessionid=".eJyrVkouLkqLL8nPTs1TslLKznAOTYzI8czOTEzxCTe0SCzKichONjMO8PUO9Q5wcVfSUYpPLC3JiC8tTi2Kz0xRsjI0MrCwNDE3QZFISkwGmgeUVQJxi_Wg_GI9x9z8UKCIE1QeqKk4tbg4Mz8vPrWiILOoEmyepZmBQS0Aqsk0IA:1aLYVj:cbZQtNCkFRasDdFtJKI9b_WB6GA"
------WebKitFormBoundaryI0LeCjkeJnsfGipU
Content-Disposition: form-data; name="csrfmiddlewaretoken"
khCUaXlIkiadLW18arlXkc63PMKUKPDG
------WebKitFormBoundaryI0LeCjkeJnsfGipU
Content-Disposition: form-data; name="upload"; filename="extension.xpi"
Content-Type: application/octet-stream
Currently I'm receiving an html page with Not allowed.
What am I missing? What else should I add to the request to receive a good response?
Thanks
Not resolving the curl upload command thing, but the original problem - validate an xpi through command line.
Validating an xpi through command line
This can be achieved with the addons-linter by Mozilla, which was in its early stages when this question was asked.
I need to release my IP address at the router on an hourly basis. The router page needs simple authentication and a button click to do the process. These are the HTTP calls:
Authenticate
GET /RST_st_dhcp.htm HTTP/1.1
Host: 10.10.1.1
Authorization: Basic YWRtaW46cGFzc3dvcmQ=
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.63 Safari/535.7
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Release button click
POST /st_dhcp.cgi?id=1044071018 HTTP/1.1
Host: 10.10.1.1
Content-Length: 31
Cache-Control: max-age=0
Authorization: Basic YWRtaW46cGFzc3dvcmQ=
Origin: http://10.10.1.1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.63 Safari/535.7
Content-Type: application/x-www-form-urlencoded
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Referer: http://10.10.1.1/RST_st_dhcp.htm
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
connect=Release&refreshScrn=yes
How can this be automated in OSX? Is it possible to write automator scripts for this?
You can simply use curl to make that request - something like
curl -e http://10.10.1.1/RST_st_dhcp.htm \
-d 'connect=Release&refreshScrn=yes' -u user:password \
'http://10.10.1.1/st_dhcp.cgi?id=1044071018'
You'll need to check whether the id in the URL matters. If not, the above should be all you need. If it does then you may have to make the first request to get the id from its payload.