sitemap.xml only for bots - sitemap

is it possible to show my web Page's sitemap.xml only to bots , and if someone want to see it and click for example ; http://www.mywebpage/sitemap.xml receive a not permitted warning message , for some reson I should do it , Has anybody a idee

There isn't really any difference between a bot and a person other than the User-Agent header (and then only if you're lucky). That said, it should be straightforward to return a 403 or a surrogate page based on agent.
What web server are you using?

Related

Google Not Indexing AJAX URLs

I have submitted a sitemap for my AJAX web application to Google via their Webmaster Tools. The submitted URLs are of the form:
http://www.mysite.com/#!myscreen;id=object-id
http://www.mysite.com/#!myotherscreen;id=another-id
However, even though more than a week has passed since sitemap submission, Google has not indexed the URLs. Google states that the sitemap has been processed, states that 60 URLs have been detected, states that no errors occurred, but does not index any of the URLs.
I have already implemented the AJAX crawlability contract on the server side, where requests containing an _escaped_fragment_ are responded to with a snapshot.
Any help/info regarding why Google is not indexing the URLs would be greatly appreciated.
See GWT SE friendly application
Suggestions include following the guide at http://code.google.com/web/ajaxcrawling/.
Nowadays, you don't need to do something specific for Google anymore, and AJAX crawling scheme has been deprecated been Google.
Just make sure that your website is easy to use for your users, and Google will be able to properly crawl it.
If you want to go the extra mile, however, you can check that article:
* https://moz.com/blog/optimizing-angularjs-single-page-applications-googlebot-crawlers

MVC3 - How to check if user clicked on the link you sent through e-mail in .net?

I'm running an e-commerce website and I send my customers regular newsletters.
I'm using nopcommerce v2.40.
I just see who all are subscribed. I want to develop a detailed newsletter management system, something like MailChimp.
I want a report on how many users actually clicked on the link that I sent them via e-mail.
Can anyone tell me how to do that??
This is a pretty generalized question but I'm new at this and I have no idea how to do it.
Thank you !
You can do this sort of thing quite simply with Google Analytics.
Here are some links worth looking at.
Google Analytics Email Tracking
Setting up campaign tracking in Google Analytics
Simple,
in your email newsletter add params you need to collect.
Example
click to view
Everytime someone would click on the above link, they would be taken to your default controller that collects clicks and other parameters you want. You would then save that data and redirect to an actual page you want them to see via "redirect" parameter provided in the url.

Google Calendar can't subscribe to icalendar feed over https?

Would someone happen to know if Google Calendar has some problems subscribing to iCalendar feeds served on a secure https-address?
I'm developing a website running on an https-address that has an iCalendar feed that users can subscribe to. The feed works just fine in Outlook and iCal, but not in Google Calendar. When a user attempts to subscribe to the feed, they get the error message "Could not fetch the URL".
I suspected that there was something wrong with the feed or the generated iCalendar data, so I ran the .ics file produced through a number of validators, and they were fine. To rule out an error in the feed itself, I put the generated .ics file on the server, to see if a static file would work, and that failed in Google Calendar as well. Then I put the file on a completely different server behind a non-secure (http) url, and that worked!
So I'm beginning to suspect that httpS is the problem. The server's certificate is valid, so that shouldn't be causing any trouble. Besides, the validators could access the feed (and the static file) just fine.
This google groups discussion indicates that others are having similar suspicions: http://productforums.google.com/forum/#!topic/calendar/61-eUd-fyrg
Problem is, the site HAS to run on over https, so I can't just switch to http to make the feed work.
So, if anyone has any information confirming or contradicing my theory, or any ideas about what else might be causing these problems, I would greatly appreciate it.
I can confirm that (today) Google Calendar can successfully subscribe to an HTTPS iCal feed.
You can test this yourself by adding this URL: https://events.stanford.edu/byCategory/2/eventlist.ics
To be extra sure I also did another test of giving it an HTTPS url that didn't also work if you replace the https -> http. That was also fine, so in all cases, HTTPS should work.
What doesn't work in my tests is:
HTTP Authentication (https://myusername:mypw#example.com/) - I got "Could not fetch URL" - but that's not what this question is asking.
Any URL over 256 characters. However, using a link shortener (e.g., goo.gl) works around this issue.
Google has confirmed that it really is an issue with HTTPS, i.e. Google Calendar is unable to subscribe to iCalendar feeds from external encrypted (https) URLs.
My employer has an enterprise account with Google, and we filed a support request with google's enterprise support, with example feeds and our own assesment of the problem.
Today, we finally got a proper answer, confirming our initial analysis and informing us that the correct techincal team has been notified and an internal feature request (for supporting feed from https-urls) has been opened.
We were not given any timeframe for the fix, but I requested that they get back to us when the issue has been resolved. I will add that information to this answer once I receive it.
The issue we've found in our case is that Google Calendar currently ignores the HTTPS indication in the URL and accesses via HTTP instead. If your HTTP requests redirect to HTTPS or just serve up the content over HTTP, then it will work. If you have a firewall blocking port 80, then things hang and its game over.
TL;DR: If your URL works with http in addtion to https, then it will work with Google Calendar when you enter it as https. (That assumes robots.txt does not restrict access.) Otherwise, it will fail.
As of January 2020 the problem appears to be resolved - Google Calendar does not appear to have problems subscribing to and updating valid RFC5545 calendars. The icalender.org validator works well and can test both a file and a link (subscription).
I've been working on creating my own iCal subscription system from scratch and wanted to share something I learned this week, ten years after the start of this discussion.
Like discussed above, importing via URL accepts https:// just fine.
But when creating an "Add to Calendar" URL for Google Calendar I discovered that they still won't accept https:// links.
The "Add to Calendar" URL formula is:
https://calendar.google.com/calendar/u/0/r?pli=1&cid=<iCal-URL-Here>
Some examples to make it clear:
// https will not work:
https://calendar.google.com/calendar/u/0/r?pli=1&cid=https://example.com/ical.ics
// http will work:
https://calendar.google.com/calendar/u/0/r?pli=1&cid=http://example.com/ical.ics
// You may also try using the webcal protocol:
https://calendar.google.com/calendar/u/0/r?pli=1&cid=webcal://example.com/ical.ics
Your mileage may vary depending on your host's handling of unsecured requests. I welcome anyone who runs into trouble to leave a comment.
Before I part, another friendly tip: You need to URI encode your iCal URL when using this import URL.
So, in reality, your link would be:
https://calendar.google.com/calendar/u/0/r?pli=1&cid=http%3A%2F%2Fexample.com%2Fical.ics
In JavaScript, use encodeURIComponent().
if the server has a robots.txt blocking google, this was a cause for failure with google calendar for me too. So, have you tried looking at the robots.txt of your https server?
This being said, is not a limitation of google calendar + https as google calendar provides https for its on "private address" for .ics files and thereof it can also accept https from google.com (though this is only one configuration over many other possible).
I have had a lot of difficulties with this:
It was frustrating because a downloaded file would open in Google Calendar or iCal, but it would not load as feed in either. I would get these errors in Google Calendar when I did add by URL: "Failed to import calendar from" (sitename) or "Could not fetch the URL."
Here's what I had to do:
Have duration or endtime for events, NOT BOTH.
I also had to remove this from the header:
content-disposition: attachment; filename=Schedule.ics;
Also, to check if it's valid, Google ical validator.

Google Sites HTTPS issue

I'm wondering if anyone can help with this.
I'm creating a site for a client using Google Sites (A requirement they set).
One of their requirement is for a contact form to be embedded on the site. I've had a look and there are plenty out there, however, if a user visits from any version of IE the content is not displayed due to the security settings.
All other browsers are functioning fine.
I know the alternative is to simply put a link to an external source, but is not ideal.
My question is threefold fold.
1. Is it possible to write a gadget that will work for IE with non-secure content (if so how)?
2. Are there any HTTPS contact forms out there that I could use?
3. Does anyone have any experience with Google sites and trying to load non-secure content and have any tips?
Thanks
Have you tried JotForm.com? They have the same (free and premium) plans as emailmeform.com. Plus, they have a specific roundabouts to embed your form in Google Sites (they have a gadget made for Google Sites). And yes, JotForm has https url for their forms if you wish to embed it as an iframe.
-- One other solution is resort to using Google Docs form.
Does anyone have any experience with Google sites and trying to load
non-secure content and have any tips? Still awaiting people with
experience....
-- Yes, I have experienced this while trying to put some social media scripts in my Google Sites website and the best thing really was to rid my Sites of those non-secure contents.
For any interested I have kind of answered my questions.
Is it possible to write a gadget that will work for IE with non-secure content (if so how)?
It is possible but you need to have a SSL hosted server.
Are there any HTTPS contact forms out there that I could use?
There are paid solutions for this. Alternatively, write your own html code to post to one of these solutions (free solution is http://www.emailmeform.com/)
Does anyone have any experience with Google sites and trying to load non-secure content and have any tips?
Still awaiting people with experience....

Does a website link (href) validation service exist?

I am looking for a web service kind of like Google Analytics.
Paste some javascript into your web page and if any of the links there become invalid, hey presto, an email is sent to someone telling them which link, which page etc etc has the incorrect link.
Anyone heard of such a service?
This would slow the page loading down a lot if it had to check for broken links every time someone visited it (basically a http request for every link). Not that it isn't possible, but the implementation would have to be very very good.
Javascript cannot send emails, you would have to use ajax to post the details to another page that would then email the admin. As this is all client side, it is very open to abuse.
I would suggest using a program to do it every now and again. There are even Firefox extensions to do it rather than a program. Google will also list a whole host of websites offering the service.

Resources