I have a robots.txt in my Joomla root Directory which says:
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
I dont know much about robots.txt Files but these are all directories in the root. So that means Google doesnt crawl all of these? But isnt that bad? I want the Sites content to be indexed or whatever...
Best regards
No, Google does not crawl these directories and no, that's not a bad thing. You do not want Google to do this as these directories are not supposed to be directly accessed. Joomla has it's main index.php files in the root directory where everything gets pushed to. The only things that will get crawled are your menu items which you assign your component to.
No, Google doesn't crawl these directories, but according to this article:
http://www.searchenginejournal.com/allow-googlebot-access-javascript-css-optimal-rankings/119113/
You should allow some directories that have CSS and Javascript files for Googlebot to crawl.
So change your robots.txt file like this :
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
# Disallow: /modules/ <-------- Commented out using #
# Disallow: /plugins/ <-------- Commented out using #
# Disallow: /templates/ <-------- Commented out using #
Disallow: /tmp/
and if you want to allow a specific path, you should use this syntax :
Allow: /plugins/system/jch_optimize/assets2/
Allow: /plugins/system/jch_optimize/assets/
Related
Let's say I have this structure in my source folder for my sphinx documentation :
journals /
*.md files
reports /
*.html files
conf.py
index.rst
and the following code in index.rst :
.. toctree::
:maxdepth: 4
:glob:
:caption: Reports
reports/*
.. toctree::
:maxdepth: 4
:glob:
:caption: Journals
journals/*
When I generate my docs with sphinx-build, sphinx automatically builds the resulting html pages with the .md files found in my journals folder, but it ignores the .html files found in the reports folder.
What I would like is for sphinx to automatically include the .html files found in the reports folder. The resulting .html pages should still include the sphinx layout (sidebar, headers, etc.), and should not just be a pure copy of the raw .html files.
I know I could do .rst files with the following :
.. raw:: html
:file: reports/file.html
This basically does what I want. The .rst file is found by sphinx and builds a .html file that includes both sphinx layout and my source .html file.
I just don't know how to do this automatically though, as I have a lot of files in my reports file. I would prefer to avoid having to create one .rst file for each of my .html file.
Is there a way to automatize this process with sphinx-build?
I seek to make an "API Reference" page exposing genindex.html and py-modindex.html; problem is, I rather the entire page not be just clickable links to said pages, but to actually include their contents. We can do this with a README - however, README.rst exists in source/, whereas genindex.html is only in build/, so below don't work:
.. include:: modindex
.. include:: genindex
.. include:: modindex.rst
.. include:: genindex.rst
.. include:: ../build/genindex.html
Can this be accomplished?
Searched a bunch, hope I didn't miss something obvious...
I have multiple book directories. Each has a single *.ditamap file (always named for the book) that references several dozen accompanying *.dita topic files:
makefile
book1/
book1.ditamap intro.dita topic1.dita topic2.dita glossary.dita
book2/
book2.ditamap about.dita topicA.dita topicB.dita appendix.dita
book3/
book3.ditamap cmd1.dita cmd2.dita cmd3.dita
The XHTML output (target) for a book depends on its single .ditamap file plus all the *.dita files in that book directory (prerequisites). The makefile is placed alongside the book directories.
Building XHTML for a book creates an output XHTML directory inside that book directory, with an index.html file that I use as the target:
book1/
book1.ditamap intro.dita topic1.dita topic2.dita glossary.dita
book1/book1_xhtml/
index.html ...more html and CSS files...
The following static pattern rule will rebuild the XHTML directory inside any book directory where the *.ditamap file has changed:
ditamap_files := $(wildcard */*.ditamap)
xhtml_files := $(patsubst %.ditamap,%_xhtml/index.html,${ditamap_files})
all: dita xhtml
dita: ${ditamap_files}
xhtml: ${xhtml_files}
${xhtml_files}: %_xhtml/index.html: %.ditamap
dita -i "${<}" -f xhtml -o "${*}_xhtml"
(Side note: make nicely handles building the path to the index.html target thanks to its clever directory-handling rules!)
However, I haven't found a way to extend this rule to be sensitive to the *.dita files too. Whenever .ditamap or .dita has changed inside the book directory, the book XHTML file must be rebuilt.
I tried stuff like
${xhtml_files}: %_xhtml/index.html: %.ditamap $(wildcard $(dir %)/*.dita)
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^
dita -i "${<}" -f xhtml -o "${*}_xhtml" 2>&1 | tee "${*}_xhtml.out"
but this completely fails to work because wildcards aren't supported in static pattern rules. Somehow I need to collect wildcarded files within the directory of the stem of each target, then make them prerequisites for that book.
Secondary expansion, maybe:
.SECONDEXPANSION:
$(xhtml_files): %_xhtml/index.html: %.ditamap $$(wildcard $$(dir $$*)/*.dita)
dita -i "$<" -f xhtml -o "$*_xhtml"
By the way, did you consider that dita files can change but also appear or disappear? Is this possible? If it is, what do you want to do? And do you know already how to detect this?
In the sphinx Makefile, let's say I specify:
BUILDDIR = buildy_mcbuildface
So that make html creates the documentation webpage under buildy_mcbuildface/html. This is a slight annoyance in that the contents of the html folder must be copied to the server location after each build.
Can I specify the html path directly, or is it always built under $BUILDDIR/html? I have looked at the html options, and I see no such path option.
Here's the solution I came up with:
Add to the Makefile:
HTMLCOPYDIR = /path/to/server/location
and also in the Makefile change, add:
cp -rT $(BUILDDIR)/html $(HTMLCOPYDIR)
after the line:
#$(SPHINXBUILD) -M $# "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
How can I add a directory for the header files in the Makefile to avoid the error *.h not found, my directory at:
/home/work/cdi_driver
But my project header include to many header file in other paths:
/home/work/cdi_driver/magnum/magnum/basemodules/std/bstd.h
/home/work/cdi_driver/callisto/include/cal_fpindicator.h
/home/work/cdi_driver/jhcommon/include/jh_types.h
...
and in "bstd.h" header file it also includes many header file in other paths in cdi_driver directory, as below:
/home/work/cdi_driver/magnum/magnum/basemodules/std/types/nucleus/bstd_defs.h
How to avoid this error while header file is including nested?