our 2 products such as www.nbook.in and www.a4auto.com are removed from all search engines. These projects got sub-domains and the links from subdomains are available. The domain is not blacklisted. The custom sitemap is created and the same is getting indexed even now. Analyzed the URL by google search console and it seems fine. 'site:nbook.in' in google didn't produce any result. Actually, we got more than 75,000 links available on google. it was working fine till last week. This is affecting our product's Alexa rank and reach. There is no issue with robots.txt file in the server document root folder. There is no rule set to deny any bot or user agent. It is not just Google, all search engines removed our links and this is making me confuse. We have other products which is designed and developed on the same way but there is absolutely no issue. I think there is nothing more to do with google search console/webmaster. I'm ready to provide more data upon requirement. Please help.
create robots.txt on root and put this code
User-agent: *
Disallow: /
Related
Hi and thanks for reading&answering my question. I am fairly new to Joomla,however I have built a directory website and started to put some content. I have created the standard pages(contact us, privacy etc) and added businesses. I would like SEO friendly URLs so I set Search Engine Friendly URLs, URL Rewriting --YES, and Suffix --No. However, Joomla somehow rewrites my alies and change them. example: mywebsite/bar-and-dinner is displayed now as my website/bar-din-r. I contacted the support of the plugin I use and they told me that I must have had some filters on Joomla that creates such a problem. I tried to purge cache on the website & server still the same. I just updated to Joomla 4.0. Thanks in advance for your help.
I am currently experiencing some problems with my Google search implementation due to a recent change to HSTS for my entire shop.
How can I create a Google sitmeap in Magento which uses the 'https://' links instead of 'http://' ?
I have already deleted my sitemap and generated a new one. It still suses http:// links.
In Google search console I see that only 8 of my 45 sites have been indexed so far..
Appreciate your help!
The sitemap links are generated using the {{unsecure_base_url}}. Make sure you change this value from http://yoursite.com to https://yoursite.com found at System->Configuration->Web Unsecure Base URL. Also, make sure both Use Secure URLs in Frontend and Backend options are set to "Yes".
Some weeks ago, we discovered someone going on our site with the robots.txt directory:
http://www.ourdomain.com/robots.txt
I've been doing some research and it said that robots.txt makes the permissions of our search engine?
I'm not certain of that...
The reason why I'm asking this is because he is trying to get into that file once again today...
The thing is that we do not have this file on our website... So why is someone trying to access that file? Is it dangerous? Should we be worried?
We have tracked the IP address and it says the location is in Texas, and some weeks ago, it was in Venezuela... Is he using a VPN? Is this a bot?
Can someone explain what this file does and why he is trying to access it?
In a robots.txt (a simple text file) you can specify which URLs of your site should not be crawled by bots (like search engine crawlers).
The location of this file is fixed so that bots always know where to find the rules: the file named robots.txt has to be placed in the document root of your host. For example, when your site is http://example.com/blog, the robots.txt must be accessible from http://example.com/robots.txt.
Polite bots will always check this file before trying to access your pages; impolite bots will ignore it.
If you don’t provide a robots.txt, polite bots assume that they are allowed to crawl everything. To get rid of the 404s, use this robots.txt (which says the same: all bots are allowed to crawl everything):
User-agent: *
Disallow:
I have been developing a web page named: directorioelectronico.com and I have specially issues now, I will be very grateful that someone can be help me.
The web page has been submitted in Google.com and now all the links that are in the homepage are listed in search results BUT some links ej. google.com/maps are not list in the home page (because it appears only when you select your municipality). How can I tell to Google that it exists (maybe without a sitemap.xlm because my links have metadata that is very important that google knows?
In advance, Thank you very much for you help.
My solution was that I create a route /sitemap.xml and put it in robots.txt too I created an html page with all the sitemap with cities, companies and all imoportant links.
I'm working for a simple bot for a project, and I noticed, that a lot of sites do not have sitemaps in their robot.txt files. There is of course an option to simply index the sites in question and crawl all possible pages, but that often takes much more time than simply downloading sitemap.
What is the best way to detect sitemap if it is not mentioned in robots.txt?
Normally it should be placed in the root directory of a domain like xydomain.xyz/sitemap.xml .
I would only add the site map into the robots file, if it is placed elsewhere. If a site uses more than one site map located on another place, it should be noted in an index map.
You can use this online tool to scan your site and create a bespoke sitemap.xlm file for your site.
To help your sitemap to be discovered through the robot.txt add the URL of your sitemap at the very top of your robot.txt file, (see below example).
So, the robots.txt file looks like this:
Sitemap: http://www.example.com/sitemap.xml
User-agent:*
Disallow: