Easy reverse proxy for serving images over ssl - https

I created a simple web-based email client like gmail. I want to display images, but in order to do it with my ssl site, I need all images served over ssl (otherwise I get "mixed-content" warnings). So I need a reverse proxy like gmail has to serve those images.
I will rewrite all image urls in the email to point at the reverse proxy. For example:
My reverse proxy is https://myreverseproxy.com
original image url http://stuff.com/image1.jpg
I will rewrite the url to be https://myreverseproxy.com?image-url=http://stuff.com/image1.jpg
When the reverse proxy gets the request "https://myreverseproxy.com?image-url=http://stuff.com/image1.jpg" it will get the original image from the query parameter image-url (http://stuff.com/image1.jpg), fetch the image, and return it to the requester of https://myreverseproxy.com?image-url=http://stuff.com/image1.jpg.
Are there any services that do this out of the box? Could I write one that's simple? Are there any libraries or solutions already for this that I could just install somewhere?
I'm open to any language and any platform...I just want this issue resolved.

I would suggest the same thing as Tudor: a proxy written in node.
However, I would advise using a more broadly used and tested library such as node-http-proxy. It is really simple to setup, and will achieve what you need in less than 20 lines of code.
var httpProxy = require('http-proxy')
httpProxy.createServer({
target: {
host: 'stuff.com',
port: 80
},
ssl: {
key: fs.readFileSync('./ssl-key.pem', 'utf8'),
cert: fs.readFileSync('./ssl-cert.pem', 'utf8')
}
}).listen(443);
If a client then accesses https://reverseproxy.com/image.png, the process would go as follows
I have assumed in this schema that the reverse proxy runs on a different server as the webserver serving the images, but this does not have to be the case. If they both run on the same server, just use host: 'localhost' in the target section.
--
Just in case you are not familiar with Node, here's what you need to do in order to quickly run this setup.
Install Node
Create a new file containing the code in yourprojectpath/index.js
Generate a package.json file by running npm init in your project's directory
Run npm install --save http-proxy to install the http-proxy library and be able to use it in the code
You should now be able to run the reverse proxy by running
node index.js
If you are planning on using this in production, I highly recommend you take a look at PM2. It is a process manager for node which basically ensures that your application is always running, no matter what. In particular, it will restart it if any kind of exception is thrown from the application and would have caused it to terminate.
Installation:
npm install -g pm2
Usage:
pm2 start index.js
A few more notes:
make sure that your .pem files have appropriate permissions and owner. chmod 400 is usually a good option (only readable by owner). The user running the Node application should be able to read them, though.
if your server runs behind a (software or hardware) firewall, you may need to open your port 443 to incoming traffic
depending on your SSL certificate provider, you might need to convert the files it will provide you to the PEM format
if needed, node-http-proxy supports additional options such as adding headers when a request is proxied
the script I presented above assumes you have ssl-key.pem and ssl-cert.pem in the same directory as it
Hope that helps! And just ask if something looks unclear to you

Here's how to create self-signed certificates, if you don't have any
Nodejitsu docs
Now for the code, which is written in node.js:
HTTPS proxy
...and a screenshot :)

It can be done easily with nginx. Btw, it can be done like you ask and it is also possible to make urls exact same like origin url. For example cdn.xxx.com/img.jpg - www.xxx.com/img.jpg.

Richard, You can resolve the issue of mixed content easily by enabling CORS in nginx config file, here is a example http://enable-cors.org/server_nginx.html. In this it is alllowing cors for everyone, you can set for a particular domain or ip, you need to look in more details.

Related

Unable to set up custom domain on Heroku using Google Domains DNS?

To preface this, I am new to backend web development so I'm coming at this totally clueless. My past experience is with Netlify, which makes it pretty seamless to add a custom domain to a website with their free DNS service.
To start, I am working on a Flask application that ideally I would like to put on a subdomain of my website (i.e. app.my-website.whatever, not actually my real domain since it includes my real name) on a different host, in this case Heroku, while keeping my main website (www.my-website.whatever) on Netlify. This required me to switch from using Netlify's DNS to using the DNS tools provided by Google Domains.
After deploying the app on the free domain, which went just fine, I tried setting up my domain for the website, following these steps:
On my website's dashboard, I went to Settings > Domains > Add domain and under domain entered app.my-website.whatever, including the subdomain of course.
Copied the DNS Target Heroku gave me.
At my dashboard for my-website.whatever at domains.google.com, under DNS > Default name servers > Resource records, I added a custom record with the hostname app.my-website.whatever (Google Domains automatically adds the .my-website.whatever), type CNAME, TTL 600, and in the Data field I pasted the DNS Target.
In my Ubuntu (WSL) terminal, when I type host app.my-website.whatever, the output says app.my-website.whatever is an alias for {bunch-of-random-characters}.herokudns.com.
Unfortunately, this has not been successful. When I try to visit the domain, I usually get an error such as DNS_PROBE_FINISHED_NXDOMAIN or alternatively ERR_SSL_UNRECOGNIZED_NAME_ALERT. I've also tried the same thing with just www.my-website.whatever, and the same issues occur.
When I try to visit the site, most browsers will automatically append https://, which I would assume doesn't quite work since I do not have a cert set-up for my site, which I need to do manually.
Does the above error mean that there is a problem related to SSL, or is it something else? Is it because my browser forces https:// that I cannot see anything changing (i.e. would http:// work?)?
From what I can tell, I should be able to do all of this on the free-tire, but I have some confusion about a few details, and feel like I could be missing some other things:
Do I need a certificate/SSL for my custom domain to work at all with Heroku?
If it could possibly be an easier solution: Is there a better alternative to Heroku in my case?
With regards to setting up the cert, I tried following the tutorial here:
https://medium.com/#bantic/free-tls-with-letsencrypt-and-heroku-in-5-minutes-807361cca5d3
For certbot, as the tutorial explains, you are given two strings like so: <long-string>.<other-long-string>, and you need to serve a file at /.well-known/acme-challenge/ with the name <long-string> (no extension), but as an unrelated issue, I cannot get Flask to serve this file, even on a local dev server, and I just get a 404 message, which the certbot utility also reports. I can create another file, such as a simple .txt file, in that same directory, and it will serve just fine.
I'll admit, these issues feel a bit basic, but I genuinely am lost, and none of the guides or posts I see online seem to have any remedy or explanation for what is happening here.
If there is any more information I should share, please let me know.

Hosting a VPN on Heroku

I was wondering if it's possible to host a private vpn on heroku?
My (hypothetical) use case is that let's say there's some service that's only available in Europe but I want to access it in the USA. I'd like to turn a European heroku server into a personal vpn that just allows me to access that service.
I did some research and can't find anyone else who's tried/documented this.
You basically want a proxy. So heroku forbids running an open proxy, so you should restrict use.
XIX. Operate an “open proxy” or any other form of Internet proxy service that is capable of forwarding requests to any end user or third-party-supplied Internet host;
--https://www.heroku.com/policy/aup
But technically it is possible - you might want to try it: https://github.com/Rob--W/cors-anywhere, if you want to use the browser you will need to download the headers from the server.js file
Note that this project is not intended to be used as an open proxy, so for example relative paths are not loaded properly.
You might want to try it - it might be more appropriate, I just did not try it myself ... :)
https://github.com/http-party/node-http-proxy#setup-a-basic-stand-alone-proxy-server

How to configure Atom to use proxy file (pac) settings?

On Windows, is it possible to configure Atom to use the system default proxy instead of manually specifying the proxy settings in the .apmrc file?
I recently started using the atom-editor for Windows (Windows 7 Pro) and so far I like it. I've been searching for some time to try and get the editor to be able to connect to the outside world and haven't met with any success.
Our network (I'm not a network expert) is configured to use a proxy server (internal) to access the internet. On my system, this is accomplished in the proxy settings with an 'Automatic Configuration Script', for example: http://internal-server-name/sub-dir/file-name.pac.
Many resources have suggested manually setting the proxy information in the .apmrc file:
Not able to connect to atom.io for themes and packages
here:https://discuss.atom.io/t/is-there-any-proxy-settings/710/19
https://github.com/atom/atom/issues/1807
These do not answer my question as I need the system defaults to work
I tried to configure Atom to do this same thing but when I change the .apmrc file to have:
http-proxy = http://internal-server-name/sub-dir/file-name.pac
https-proxy = https://internal-server-name/sub-dir/file-name.pac
strict-ssl = false
I get the following error (presumably because the above address is not the proxy itself):
tunneling socket could not be established, cause=Parse Error
When I leave the .apmrc file unmolested, I receive the following error when trying to view packages:
getaddrinfo ENOTFOUND
Even if I had a proxy server IP address and port number to specify, not only would i not want to hard-code my credentials in a config file but I'm pretty sure the authentication is somehow set up to use Kerberos tokens anyway, so I wouldn't be able to specify them at all.
Interestingly the Atom updates says that my installed packages (out of the box) are up to date, which would seem to indicate that this portion is working while the packages and themes do not.
Here are some screenshots for reference:
No .apmrc changes
Proxy server specified in .apmrc
Edit 08/31/2015
In response to some of the feedback, I tried to view the .pac file and extract the proxy server address information. I was able to get the information about the proxy out of the file and put it in the .apmrc file but it had no affect on the outcome (I still receive the same errors). I have also since tried using CNTLM to no avail.
Also note that my proxy does require authentication as stated above. I've been doing further research on this issue and it may be a combination of proxy settings for the APM and proxy settings for GitHub. See the following relevant article:
https://discuss.atom.io/t/error-running-apm-install-behind-proxy/14812
Ideally, I'd like a solution that doesn't require a complete machine configuration or a fragile multi-config file setup.
A .pac file is just a javascript file that browsers can run to programmatically determine which proxy to use. This is useful if your network needs to use different proxies to access different resources. If you open up that file, you'll probably be able to make sense of it (it's just javascript, after all) and figure out which proxy your network is using for general access to the web. Set that as your proxy in your .apmrc file and it should work.
Things get more complicated if your proxy requires some kind of authentication. If that's your problem, let me know. I have some experience dealing with it.
Authenticating
OK, so you need to authenticate. Then your situation is probably pretty similar to mine. You are correct in using CNTLM, we just need to configure it properly. Your cntlm.ini file will look something like this:
Username <your windows username>
Domain <your domain name>
Password <leave this blank>
PassLM <get this by running `cntlm -H` on the command line>
PassNT <get this by running `cntlm -H` on the command line>
PassNTLMv2 <get this by running `cntlm -H` on the command line>
Proxy <Your proxy address like ip_address:port>
Proxy <If you have multiple proxies, you may list them each on a new line>
NoProxy localhost, 127.0.0.*, <any others that should bypass the proxy>
Listen <a local port to listen on (I use 53128)>
There's something a bit funny that you should know about this file (cntlm.ini). When you install cntlm, it's included in the program's folder but it actually will only work if it's located at C:\Program Files (x86)\cntlm\cntlm.ini. I guess it's probably more accurate to say that it should be in the cntlm folder inside the PROGRAMFILES directory so you should double check that this is the case.
Anyway, having done all that, you can now start cntlm (instructions in the readme). Next, you need to configure apm to actually use your local proxy. Your .apmrc should look like this:
http-proxy = http://localhost:<port # from cntlm.ini>
https-proxy = https://localhost:<port # from cntlm.ini>
strict-ssl = false
You'd think this would be enough but there's more that might help. I had a lot of problems with npm until I found that https doesn't work well through cntlm so I needed to change my registry from https://registry.npmjs.org/ to http://registry.npmjs.org/. Since apm also uses the same registry, I decided to change that too:
apm config set registry http://registry.npmjs.org/
And then... it seemed to also help to go into Atom's config.cson and set core.proxy to http://localhost:<port # from cntlm.ini>
And after all that, I'd expect things to work for you. If it's cool and fine and Atom is working well, there's one more thing that you might want to do that will make most (all?) other apps (npm, git, etc) work well. On the command line, run:
netsh winhttp set proxy localhost:<port # from cntlm.ini> "localhost, 127.0.0.*"
This will save you the trouble of configuring most (all?) other apps one-by-one. For some reason, apm does need to be configured specifically. I guess it doesn't use the winhttp settings for some reason? I don't know the answer to that.
New Info 2016/03/18
I recently noticed that there's one additional thing you might need to do for some apps. You should set the following environment variables to your local proxy as well:
HTTP_PROXY
HTTPS_PROXY
FTP_PROXY
http_proxy
https_proxy
ftp_proxy
It may seem like overkill to set both upper and lower case but I recently installed MSys2 and found that it looks for the lower case versions so just set them all and save yourself some trouble.
You should download the file http://internal-server-name/sub-dir/file-name.pac and open it with a text editor, then find the actual proxy address and port inside it (near a PROXY keyword).
I used this for my trouble with npm and .npmrc
c:\users\%user%\.atom\.apmrc (add if not exist) and put this
proxy=http://YOUR-DOMAIN**%5C**USER-NAME:PASSWORD#YOUR-PROXY-SERVER:PORT/

A script that download a file and rename

I'm facing an little issue here. At the place I live, they shape the download speed by extension (using delaypool). Is there any script that I can run on my web that will let me enter the URL of files that I wanted to download, then it will download the file and rename it to "originalfilename.abc" (because .abc is not shaped)then save it on my web where I can download it. By the way, I have a paid webhosting service.
Thanks
I tried the SSH (my web hosting does indeed provide me with shell access) but all I get is a blank page in my browser. No error. Please advice.
Also, reason I choose script on the server instead of SSH because I though that SSH would be slower than direct HTTP download from my webserver. Can anyone point that out if I'm right or wrong with my thinking.
Thanks
You can write a vbs script or even batch file script that will go to a designated URL, and then download the file, then rename it. Then you can script FTP commands to upload to your webhosting service (I'm sure it has an FTP site for access). You could load this as a scheduled task, or run it manually.
I'd do either one of these routes.
A simpler (once its set up anyway ;) option I can think of is going through a secure tunnel. Whilst this is not quite answering your question I believe this to be simpler while achieving the same thing.
Get an SSH Client (Putty) and get a free Proxy. If your web-server has an SSH-server you can use it as Proxy as well of course, I am using my modified router at home as proxy via DynDns, but the Tor-Network will work, if very slow, so do other official free and paid proxy servers. If you are using an application to download that does not have the option to specify a proxy, get Proxifier Portable.
Use Putty to create the tunnel. Here some how-tos:
http://oldsite.precedence.co.uk/nc/putty.html
http://www.techrepublic.com/blog/security/use-putty-as-a-secure-proxy-on-windows/421
http://kimmo.suominen.com/docs/proxy-through-ssh/
And set your application to use your proxy (or actually to use putty which connects via SSH to your proxy) by entering 127.0.0.1:1080 into the proxy settings. Alternatively, if the app does not have an option to enter proxy settings, add 127.0.0.1:1080 to your Proxifier proxy list and add the applications that are supposed to use that connection to Proxifier.
Now you can do pretty much everything without anyone eavesdropping your connection stream being able to tell what it is, as the connection stream is SSH encrypted. This includes surfing websites that your provider/company/mother has blocked, download anything - even if blocked by IP/name/whatever-filters and even play MMORPGs from work (something which I do not recommend because it will get you fired and there is always some way for someone to figure it out, just saying it is possible to do even in secured company/school networks as Port 22 (SSH) is usually one of the 2 Ports which are open on pretty much any network (the other one being port 80)).
Its a wee bit of a pain to set up. Once it is working though, you can even put it on a usb-stick and use it pretty much anywhere as long as you remember what proxy to connect to. And you wont have to rewrite scripts to try to circumvent the delaypool thingy.

nginx as mail proxy

I am trying to setup nginx as mail proxy. All i want is to let nginx receive the mail and forward it to a script. Is this set possible or should i only use sendmail for that.
The only way I can remotely imagine that working is if you would let nginx listen on the smtp port and run a smtp server web application on that port. At that point nginx would basically only connect the outside port to your locally running app.
So yeah, I think you would be much better off with a real smtp server like sendmail. Actually, I recommend you use postfix because it does the same thing arguably better.
there is a lack of documentation but from playing around with it
it looks to like the nginx mail options are for creating proxies to more easily add custom authentication to an existing MTA.
A use case might be to use a php script to authenticate existing website users using the website's existing users database without needing to create system users for them or set up extra databases.
so no, it won't forward the actual email to a script
(look for things like 'pipe_tansport' in MTA's like exim to do that)
but it will let you use a script to put authentication in front of an existing smtp server.

Resources