What characters should and should not be included (or should be escaped) in HTTP_REFERERs? - mod-rewrite

I wanted to update my .htaccess file with some new referral rewrites and found a good list in GitHub https://gist.github.com/raniajal/e6733427207de2677616#file-htaccess-txt-L72.
I had a mixed format of ones I'd found elsewhere with escaped hyphens and non-escaped hyphens; slashes at the end of .com, .org, .net, .ga, .ru, etc. and some without the ending slash such as:
RewriteCond %{HTTP_REFERER} ^http://.*\.social\-buttons\.com/ [NC,OR]
RewriteCond %{HTTP_REFERER} free-share-buttons\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*yapoga\.com/ [NC,OR]
RewriteCond %{HTTP_REFERER} ^https?://([^.]+\.)*seoanalyses\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*savetubevideo\.com/ [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://.*semalt\.com/ [NC,OR]
RewriteCond %{HTTP_REFERER} 4webmasters\.org/ [NC,OR]
RewriteCond %{HTTP_REFERER} 7makemoneyonline\.com/ [NC,OR]
RewriteCond %{HTTP_REFERER} 100dollars-seo\.com/ [NC,OR]
RewriteCond %{HTTP_REFERER} addons\.mozilla\.org [NC,OR]
RewriteCond %{HTTP_REFERER} o\-o\-6\-o\-o\.com [NC,OR]
RewriteCond %{HTTP_REFERER} youporn-forum\.ga/ [NC,OR]
RewriteCond %{HTTP_REFERER} непереводимая\.рф/ [NC]
I noticed that the list in GitHub did not have the ending slashes and had updated notation to allow for https and my existing ones didn't so I updated my list to use that. For example:
RewriteCond %{HTTP_REFERER} ^https?:\/\/([^.]+\.)*o-o-6-o-o\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^https?:\/\/([^.]+\.)*slftsdybbg\.ru [NC,OR]
RewriteCond %{HTTP_REFERER} ^https?:\/\/([^.]+\.)*social-buttons\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^https?:\/\/([^.]+\.)*socialseet\.ru [NC,OR]
RewriteCond %{HTTP_REFERER} ^https?:\/\/([^.]+\.)*trafficmonetize\.org [NC,OR]
RewriteCond %{HTTP_REFERER} ^https?:\/\/([^.]+\.)*video--production\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^https?:\/\/([^.]+\.)*videos-for-your-business\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^https?:\/\/([^.]+\.)*vodkoved\.ru [NC,OR]
RewriteCond %{HTTP_REFERER} ^https?:\/\/([^.]+\.)*www1\.social-buttons\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^https?:\/\/([^.]+\.)*ykecwqlixx\.ru [NC,OR]
RewriteCond %{HTTP_REFERER} ^https?:\/\/([^.]+\.)*youporn-forum\.ga [NC,OR]
RewriteCond %{HTTP_REFERER} forum\.topic65882328\.darodar\.com [NC,OR]
RewriteCond %{HTTP_REFERER} ^https?:\/\/([^.]+\.)*непереводимая\.рф [NC]
But now I am getting 500 Server Error and the Error log says RewriteCond: bad flag delimiters
The file is UTF-8 and I don't see any excess spaces. What am I doing wrong?

This is resolved. There seems to be no difference in the results of the two examples. I re-re-re-read my list of HTTP_REFERER items and discovered two that caused the problem.
One in the middle of the list had only [NC] so I changed that to [NC,OR].
One was missing the [NC,OR] so I added that.
In a long list it is easy to skip over things that seem the same -- better proofreading was needed.
Now the .htaccess file works.

Related

Mod Rewrite Maintenance page that uses multiple URLs

I have a site the uses multiple URLs for a single application. Depending on the URL of the site you will get different content. I need to create a rewrite rule that redirects a user to a different page depending on what URL the user hits.
For example:
If a user visits www.foo.bar the user will be redirected to www.foo.bar/maintWWW.html
But if the user visits www.bar.foo the user will be redirected to www.bar.foo/maintWWW2.html
Remember that since we are using the same application but different URLs that these 2 html pages needs to be named differently in order to serve different content.
I managed to use this but it only redirects both URLs to a single page,
RewriteCond %{REMOTE_ADDR} !^123\.456\.789\.000
RewriteCond %{REQUEST_URI} !/maintWWW.html$ [NC]
RewriteCond %{REQUEST_URI} !\.(jpe?g?|png|gif) [NC]
RewriteRule .* /maintWWW.html [R=302,L]
I tried replacing the %{REQUEST_URI} with the actual URL of the site I want redirected, but it did not work.
RewriteCond %{REMOTE_ADDR} !^123\.456\.789\.000
RewriteCond http://www.foo.bar !/maintWWW.html$ [NC]
RewriteCond http://www.foo.bar !\.(jpe?g?|png|gif) [NC]
RewriteRule .* /maintWWW.html [R=302,L]
RewriteCond %{REMOTE_ADDR} !^123\.456\.789\.000
RewriteCond http://www.bar.foo !/maintWWW2.html$ [NC]
RewriteCond http://www.bar.foo !\.(jpe?g?|png|gif) [NC]
RewriteRule .* /maintWWW2.html [R=302,L]
How exactly would I get this to work? Can I include multiple URLs to be redirected to the same page? It would be nice to account for development and staging URLs as well. Example below again:
RewriteCond %{REMOTE_ADDR} !^123\.456\.789\.000
RewriteCond http://www.foo.bar http://staging.foo.bar http://dev.foo.bar !/maintWWW.html$ [NC]
RewriteCond http://www.foo.bar !\.(jpe?g?|png|gif) [NC]
RewriteRule .* /maintWWW.html [R=302,L]
You need to check HTTP_HOST in your condition to check for the domain and then redirect based on that.
RewriteEngine On
RewriteCond %{HTTP_HOST} ^(www\.)?foo\.bar [NC]
RewriteCond %{REMOTE_ADDR} !^123\.456\.789\.000
RewriteCond %{REQUEST_URI} !/maintWWW.html$ [NC]
RewriteCond %{REQUEST_URI} !\.(jpe?g?|png|gif) [NC]
RewriteRule .* /maintWWW.html [R=302,L]
RewriteCond %{HTTP_HOST} ^(www\.)?bar\.foo [NC]
RewriteCond %{REMOTE_ADDR} !^123\.456\.789\.000
RewriteCond %{REQUEST_URI} !/maintWWW2.html$ [NC]
RewriteCond %{REQUEST_URI} !\.(jpe?g?|png|gif) [NC]
RewriteRule .* /maintWWW2.html [R=302,L]
If you have multiple hostnames or aliases that you can use in your setup you can do this.
RewriteCond %{HTTP_HOST} ^(www|dev|staging)\.foo\.bar [NC]
RewriteCond %{REMOTE_ADDR} !^123\.456\.789\.000
RewriteCond %{REQUEST_URI} !/maintWWW.html$ [NC]
RewriteCond %{REQUEST_URI} !\.(jpe?g?|png|gif) [NC]
RewriteRule .* /maintWWW.html [R=302,L]

Prevent hotlinking image files

I want to avoid hotlinking images. Basing of this answer: stackoverflow answer I've tried to add the next snippet code into .htaccess file:
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://(www.)?itransformer.es/.*$ [NC]
RewriteRule \.(gif|jpe?g|png|wbmp)$ htttp://itransformer.es [R,L]
but it is not working. When I try to access the images putting the path in the navigation bar, I can access. What am I doing wrong?
You're explicitly ignoreing referers that are blank. When you type the URL in your navigation bar, there is no referer, therefore %{HTTP_REFERER} is blank and it will fail the !^$ pattern. Try this instead:
RewriteCond %{HTTP_REFERER} ^$ [OR]
RewriteCond %{HTTP_REFERER} !^http://(www\.)?itransformer.es/.*$ [NC]
RewriteRule \.(gif|jpe?g|png|wbmp)$ http://itransformer.es [R,L]
Also, you have an extra "t" in your htttp://

Regard a couple of OR-RewriteCond directives if another RewriteCond evals

I have the following configuration,
RewriteCond %{HTTPS} !^on$ [nocase,ornext]
RewriteCond %{HTTP_HOST} !^www\. [nocase,ornext]
RewriteCond %{HTTP_HOST} !\.com$ [nocase]
RewriteRule ^(.*)$ https://www.acme.com/$1 [redirect=301,last]
the purpose is to redirect to a canonical URL if the request is either not HTTPS, nor begins with www. or ends with .com.
For being seamlessly compatible with developer engines, I want to exclude all these directives if %{HTTP_HOST} includes, for example, dev.internal or so. In this case the RewriteRule should be skipped immediately. Since the three ORs are evaluated with the higher precedence than an (implicit) AND, I wonder how and where to place my dev.internal exception...
Thanks for any advice!
//edit: hmm... if OR has the higher precendence, shouldn't
RewriteCond %{HTTP_HOST} !internal\. [nocase]
RewriteCond %{HTTP:X-Forwarded-Proto} !^https$ [nocase,ornext]
RewriteCond %{HTTP_HOST} !^www\. [nocase,ornext]
RewriteCond %{HTTP_HOST} !\.com$ [nocase]
RewriteRule ^(.*)$ https://www.acme.com/$1 [redirect=301,last]
work then?
If my understanding of ornext is correct then yes, your way should work (can someone else confirm it?).
Here's another way, if you don't want to rely on that:
RewriteCond %{HTTP_HOST} internal\. [nocase] # If it's an internal host...
RewriteRule .* - [skip=1] # ... skip the next rule (and leave the URL unchanged)
RewriteCond %{HTTP:X-Forwarded-Proto} !^https$ [nocase,ornext]
RewriteCond %{HTTP_HOST} !^www\. [nocase,ornext]
RewriteCond %{HTTP_HOST} !\.com$ [nocase]
RewriteRule ^(.*)$ https://www.acme.com/$1 [redirect=301,last]

mod_rewrite with external redirect and internal rewrite

I'm trying to use mod_rewrite to redirect certain pages to use SSL. For that I have:
RewriteCond %{SERVER_PORT} ^443$
RewriteCond %{REQUEST_URI} !^/login(\.php)?$ [NC]
RewriteCond %{REQUEST_URI} !^/contact-us(\.php)?$ [NC]
RewriteCond %{REQUEST_URI} !^/\..*$
RewriteRule ^(.*)$ http://www.example.com/$1 [R=301,L]
RewriteCond %{HTTP_HOST} !^dev\.example\.com$ [NC]
RewriteCond %{SERVER_PORT} ^80$
RewriteCond %{REQUEST_URI} ^/login(\.php)?$ [NC,OR]
RewriteCond %{REQUEST_URI} ^/contact-us(\.php)?$ [NC]
RewriteRule ^(.+)\.php$ https://www.example.com/$1 [R=301,L]
This works fine, and does exactly what I want it to do.
Later in my .htacess I have a:
RewriteRule ^members/(.+)/change-password$ members/.change-password.php?item=$1 [NC,QSA,L]
So if a URL appears as, for example:
http://www.example.com/members/foo-bar/change-password
Internally it would be processed as:
/members/.change-password.php?item=foo-bar
Again, this works fine and is doing what I want it too.
What I now need to do is include this in my original SSL redirect logic to ensure that any change password requests are redirected to the same URL but over https instead. I've tried:
RewriteCond %{SERVER_PORT} ^443$
RewriteCond %{REQUEST_URI} !^/login(\.php)?$ [NC]
RewriteCond %{REQUEST_URI} !^/contact-us(\.php)?$ [NC]
RewriteCond %{REQUEST_URI} !^/\..*$
RewriteCond %{REQUEST_URI} !^/members/.+/change-password [NC]
RewriteRule ^(.*)$ http://www.example.com/$1 [R=301,L]
RewriteCond %{HTTP_HOST} !^dev\.example\.com$ [NC]
RewriteCond %{SERVER_PORT} ^80$
RewriteCond %{REQUEST_URI} ^/login(\.php)?$ [NC,OR]
RewriteCond %{REQUEST_URI} ^/contact-us(\.php)?$ [NC,OR]
RewriteCond %{REQUEST_URI} ^/members/.+/change-password [NC]
RewriteRule ^(.+)\.php$ https://www.example.com/$1 [R=301,L]
But this doesn't work - I just get the page delivered over http. Changing the .+ to .* appears to put me into a permanent redirect loop.
I'm guessing this is because of the internal rewrite but no matter what I try I can't seem to resolve it.
Can anyone please advise?
Thanks,
Adam M.
A further review of the mod_rewrite documentation led me to a bit I'd missed specific to its usage in .htaccess files. Basically the [L] flag doesn't actually indicate last as per the norm. Instead you need to use the [END] flag (http://httpd.apache.org/docs/current/rewrite/flags.html#flag_l refers).
Of course that then led me to another issue - my hosting provider doesn't have an up-to-date installation of either Apache or mod_rewrite so the [END] flag triggered the ubiqitous HTTP 500 Internal Server Error.
So what to do? Well I went back to my original ruleset with the knowledge that [L] wasn't doing what I was expecting and spotted the error straight away - the %{REQUEST_URI} value had been updated by the internal rewrite:
RewriteRule ^members/(.+)/change-password$ members/.change-password.php?url-slug=$1 [NC,QSA,L]
Therefore changing my original redirection logic to exclude this resolved my issue:
RewriteCond %{SERVER_PORT} ^443$
RewriteCond %{REQUEST_URI} !^/login(\.php)?$ [NC]
RewriteCond %{REQUEST_URI} !^/contact-us(\.php)?$ [NC]
RewriteCond %{REQUEST_URI} !^/\..*$
RewriteCond %{REQUEST_URI} !^/members/.+/change-password$ [NC]
RewriteCond %{REQUEST_URI} !^/members/\.change-password(\.php)? [NC]
RewriteRule ^(.*)$ http://www.example.com/$1 [R=301,L]
RewriteCond %{HTTP_HOST} !^dev\.example\.com$ [NC]
RewriteCond %{SERVER_PORT} ^80$
RewriteCond %{REQUEST_URI} ^/login(\.php)?$ [NC,OR]
RewriteCond %{REQUEST_URI} ^/contact-us(\.php)?$ [NC,OR]
RewriteCond %{REQUEST_URI} ^/members/.+/change-password$ [NC]
RewriteRule ^(.+)(\.php)?$ https://www.example.com/$1 [R=301,L]

.htaccess blocking images on some internal pages

I'm doing some web design for a friend and I noticed that everywhere else on her site images will load fine except for the subdirectory I'm working in. I looked in her .htaccess file and sure enough it is setup to deny people from stealing her images. Fair Enough, except the pages i'm working on are in her domain and yet I still get the 403 error. I'm pasting the .htaccess contents below but I replaced the domain names with xyz, 123 and abc.
So specifically the page I'm on (xyz.com/DesignGallery.asp) pulls images from (xyz.com/machform/data/form_1/files) and it results in a forbidden error.
RewriteEngine on
<Files 403.shtml>
order allow,deny
allow from all
</Files>
RewriteCond %{HTTP_REFERER} !^http://xyz.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://xyz.com/machform/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://xyz.com/machform/data/form_1/files/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://xyz.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://abc.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://abc.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://abc.xyz.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://abc.xyz.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://123.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://123.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://123.xyz.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://123.xyz.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.xyz.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.xyz.com/machform/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.xyz.com/machform/$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.xyz.com/machform/data/form_1/files/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.xyz.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.abc.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.abc.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.abc.xyz.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.abc.xyz.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.123.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.123.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.123.xyz.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.123.xyz.com$ [NC]
RewriteRule .*\.(jpg|jpeg|gif|png|bmp)$ - [F,NC]
deny from 69.49.149.17
RewriteCond %{HTTP_HOST} ^.*$
RewriteRule ^vendors\.html$ "http\:\/\/www\.xyz\.com\/Design_Gallery_1\.htm" [R=301,L]
RewriteCond %{HTTP_HOST} ^.*$
RewriteRule ^vendors\.asp$ "http\:\/\/www\.xyz\.com\/Design_Gallery_1\.htm" [R=301,L]
RewriteCond %{HTTP_HOST} ^.*$
RewriteRule ^ArtGraphics\.html$ "http\:\/\/www\.xyz\.com\/Art_Gallery_1\.htm" [R=301,L]
RewriteCond %{HTTP_HOST} ^.*$
RewriteRule ^ArtGraphics\.asp$ "http\:\/\/www\.xyz\.com\/Art_Gallery_1\.htm" [R=301,L]
RewriteCond %{HTTP_HOST} ^.*$
RewriteRule ^Gear\.asp$ "http\:\/\/www\.xyz\.com\/Gear_Gallery_1\.htm" [R=301,L]
RewriteCond %{HTTP_HOST} ^.*$
RewriteRule ^Gear\.html$ "http\:\/\/www\.xyz\.com\/Gear_Gallery_1\.htm" [R=301,L]
RewriteCond %{HTTP_HOST} ^.*$
RewriteRule ^NewsletterSign\-Up\.html$ "http\:\/\/www\.xyz\.com\/Newsletter\.htm" [R=301,L]
RewriteCond %{HTTP_HOST} ^.*$
RewriteRule ^NewsletterSign\-Up\.asp$ "http\:\/\/www\.xyz\.com\/Newsletter\.htm" [R=301,L]
RewriteCond %{HTTP_HOST} ^.*$
RewriteRule ^KidzStuff\.html$ "http\:\/\/www\.xyz\.com\/KidzStuff1\.htm" [R=301,L]
RewriteCond %{HTTP_HOST} ^.*$
RewriteRule ^KidzStuff\.asp$ "http\:\/\/www\.xyz\.com\/KidzStuff1\.htm" [R=301,L]
RewriteCond %{HTTP_HOST} ^.*$
RewriteRule ^Vendors\.html$ "http\:\/\/www\.xyz\.com\/Design_Gallery_1\.htm" [R=301,L]
RewriteCond %{HTTP_HOST} ^.*$
RewriteRule ^Vendors\.asp$ "http\:\/\/www\.xyz\.com\/Design_Gallery_1\.htm" [R=301,L]
I'll bet its a syntax error in one of the directives, that you may have corrected when substituting www.xyz.com.
If you have access to httpd.conf, you could set up the RewriteLog and RewriteLogLevel directives to get a sense of what's going on when you try to load the images. You should be able to identify the offending line pretty easily. (Be sure to remove those directives once you're done.)
If you don't have access to httpd.conf, you can eliminate the lines one by one until you find the offender.
(Once that's all done, you could probably simplify that .htaccess file quite a bit. Most of those rewrite conditions are redundant. Easy for tiny errors to sneak in and not get noticed)

Resources