I am crawling a website X over https, I can able to open it in browser and even able to wget from my pc but not able to wget from ec2.
Connecting to X|IP|:443... connected.
HTTP request sent, awaiting response.
When I created a vpn on ec2 and connected my PC through this vpn, I was able to open website in browser but not able to wget from my pc.
I have tried multiple ec2 instances same thing is happening.
UPDATE
Is it possible to detect if the web request is originated from wget or headless browser?
Related
Testing out Ghostlab on mobile presented an issue when the locally hosted page attempts to ping another locally hosted page via AJAX. Normally, GhostLab will rewrite the URL for you, but since it's an AJAX request I'm assuming it's not rewriting it because it's a dynamically generated URL.
Works fine on virtual machines and host machine with modified HOST file. Fails to load AJAX. The proxy server settings are very limited on GhostLab. Has anyone come across this issue and solved it?
The solution is to run another proxy server because you can't really modify the HOST file on iPhones or Androids without a huge headache.
I ended up using Charles Proxy. I remapped the locally hosted URL to which it was making the AJAX call to my local machine and it fixed everything. Here are the instructions:
1) Open Charles
2) (Menu)Tools/Map Remote...
Add the url you are local hosting to remap any traffic going through Charles Proxy to go to your machine See image example:
Do the same to any other subdomains or domains that are locally hosted on your machine and remap them too.
3) On your mobile device, open settings and select your local Wifi Network.
4) Change the proxy to your machine's IP address and relevant PORT.
5) Attempt to access the website and Grant permission via Charles.
This should fix it all.
I am trying to connect to a Visual Studio server from a remote machine using Fiddler and HTTPS. I followed this tutorial (http://docs.telerik.com/fiddler/Configure-Fiddler/Tasks/UseFiddlerAsReverseProxy) which works perfectly for HTTP. When I moved my project to HTTPS, the connection stopped working. I have tried several solutions, including:
Proxying HTTPS traffic through Fiddler fails for remote clients?
http://lifelongprogrammer.blogspot.com.au/2014/04/using-fiddler-as-https-reverse-proxy.html
but haven't been able to get it to work. Can someone help?
You haven't explained what happens, but my guess is that your client is trying to do a HTTPS handshake with Fiddler, which is expecting HTTP because you didn't tell it to do anything else.
In Fiddler's QuickExec box, type
!listen 444 FiddlerMachineName
This will create a new HTTPS-accepting endpoint running at https://FiddlerMachineName:444 and Fiddler will perform a HTTPS handshake on all inbound connections. You can then use FiddlerScript to forward requests to the target service.
I'm on a VPS and use a bash script to deploy websites to subdirectories inside on of my accounts. I've been using this script with no issues until today, randomly, cPanel decided to ignore curl.
In this script I automatically create a database and add a user to it, which is accomplished via curling cPanel with the appropriate information.
Here's what I was doing and previously was working flawlessly.
curl -k -v -G -d db=$db_name https://$cpanel_user:$cpanel_password#$cpanel_host:2083/frontend/x3/sql/addb.html
I isolated this line to a separate script to debug. Here's the kicker, the server simply doesn't respond. Curl just says Trying X.X.X.X... and then eventually the connection times out. Curl works for everything else so I know that's not it. Just for grins I tried wget and it can't establish a connection either.
I have [had] security tokens disabled in cPanel.
My question is what would cause the connection to fail? I'm stumped as this worked yesterday.
I don't mind investigating and debugging, but with no errors or anything I simply don't know where to look. I talked to my host and they said there hasn't been any firewall/security updates rolled out.
Suggestions on issues or where to look?
Edit: So if I run that curl command locally in OSX it fires up and creates the DB no problem. So for some reason the URL isn't responding to my own server (itself) o.O
Edit 2: It seems my server can't curl itself on that port I need (dunno why it would have stopped). For example if I curl my server IP or a domain on my server (from the server) it works. If I try to curl my ip:2083 it won't resolve.
How can I allow curl to work with port 2083?
My firewall CSF just allow incomming TCP connexion on port 2083 to the server. But the other one blocked the outgoing tcp connexion on port 2083.
So opening both port IN-OUT on both server solved the issue.
I have to manually upload a file with the browser using a https:// destination.
Now I also need to get the logs from that HTTP server.
I tried around already with Fiddler2 and I do get logs but only from my client.
Is there a way to get the response logs from the HTTP server?
(I need to do this because the party that runs the server would have to setup a VPN tunnel to log my uploads and they do not want to do that)
I have a question about connecting Firefox via SSH.
There are some websites like the following I need to connect
http://ieeexplore.ieee.org
So When I connect to this website from my office, it will allow me to search documents and download without any authentication. So I guess, it because I am using the office network which has already the authentication done.
However, when I am out of office, it will ask me for authentication and I do not have username and password for the account which office uses. So I need to somehow connect to office network via SSH to access this website.
I already tried to ssh -X and open a firefox -no-remote. But it is really slow.
I would like to know if there is a solution where I can connect directly from SSH through a proxy tunnel or something.
Also is it possible that only for some websites it should use proxy for rest others, it should use normal internet connection.
Thanks for help,
Raj
ssh -D port_number office
Then configure your proxy in Firefox to SOCK5: localhost port_number
Should make the trick.