I have a Jinja 2 template like this:
foo{# comment #}
bar
When trim_blocks is enabled, the template renders as:
foobar
I want the template to render like this:
foo
bar
I tried using the +#} syntax to disable trimming after the comment but it didn't work. How can I do this? I don't want to have to add an extra line break after every single comment in my template.
Edit: I am not in control of the code that renders the template, so any solution must be within the template itself.
Simply output a newline using {{ "\n" }} after the comment:
foo{# comment about foo #}{{ "\n" }}
bar
Ideally you could change the rendering code to not strip comments, but if you can't this will preserve the desired output.
You can change lexing rules for comment blocks
import re
from jinja2 import Environment
from jinja2.lexer import TOKEN_COMMENT, TOKEN_COMMENT_BEGIN, TOKEN_COMMENT_END, Failure
env = Environment()
env.trim_blocks = True
c = lambda x: re.compile(x, re.M | re.S)
e = re.escape
comment_block_suffix_re = ''
env.lexer.rules[TOKEN_COMMENT_BEGIN] = [
(c(r'(.*?)((?:\-%s\s*|%s)%s)' % (
e(env.comment_end_string),
e(env.comment_end_string),
comment_block_suffix_re
)), (TOKEN_COMMENT, TOKEN_COMMENT_END), '#pop'),
(c('(.)'), (Failure('Missing end of comment tag'),), None)
]
See source code of lexer module for details
Related
TL,DR: From a Sphinx extension, how do I tell sphinx-build to treat an additional file as a dependency? In my immediate use case, this is the extension's source code, but the question could equally apply to some auxiliary file used by the extension.
I'm generating documentation with Sphinx using a custom extension. I'm using sphinx-build to build the documentation. For example, I use this command to generate the HTML (this is the command in the makefile generated by sphinx-quickstart):
sphinx-build -b html -d _build/doctrees . _build/html
Since my custom extension is maintained together with the source of the documentation, I want sphinx-build to treat it as a dependency of the generated HTML (and LaTeX, etc.). So whenever I change my extension's source code, I want sphinx-build to regenerate the output.
How do I tell sphinx-build to treat an additional file as a dependency? That is not mentioned in the toctree, since it isn't part of the source. Logically, this should be something I do from my extension's setup function.
Sample extension (my_extension.py):
from docutils import nodes
from docutils.parsers.rst import Directive
class Foo(Directive):
def run(self):
node = nodes.paragraph(text='Hello world\n')
return [node]
def setup(app):
app.add_directive('foo', Foo)
Sample source (index.rst):
.. toctree::
:maxdepth: 2
.. foo::
Sample conf.py (basically the output of sphinx-quickstart plus my extension):
import sys
import os
sys.path.insert(0, os.path.abspath('.'))
extensions = ['my_extension']
templates_path = ['_templates']
source_suffix = '.rst'
master_doc = 'index'
project = 'Hello directive'
copyright = '2019, Gilles'
author = 'Gilles'
version = '1'
release = '1'
language = None
exclude_patterns = ['_build']
pygments_style = 'sphinx'
todo_include_todos = False
html_theme = 'alabaster'
html_static_path = ['_static']
htmlhelp_basename = 'Hellodirectivedoc'
latex_elements = {
}
latex_documents = [
(master_doc, 'Hellodirective.tex', 'Hello directive Documentation',
'Gilles', 'manual'),
]
man_pages = [
(master_doc, 'hellodirective', 'Hello directive Documentation',
[author], 1)
]
texinfo_documents = [
(master_doc, 'Hellodirective', 'Hello directive Documentation',
author, 'Hellodirective', 'One line description of project.',
'Miscellaneous'),
]
Validation of a solution:
Run make html (or sphinx-build as above).
Modify my_extension.py to replace Hello world by Hello again.
Run make html again.
The generated HTML (_build/html/index.html) must now contain Hello again instead of Hello world.
It looks like the note_dependency method in the build environment API should do what I want. But when should I call it? I tried various events but none seemed to hit the environment object in the right state. What did work was to call it from a directive.
import os
from docutils import nodes
from docutils.parsers.rst import Directive
import sphinx.application
class Foo(Directive):
def run(self):
self.state.document.settings.env.note_dependency(__file__)
node = nodes.paragraph(text='Hello done\n')
return [node]
def setup(app):
app.add_directive('foo', Foo)
If a document contains at least one foo directive, it'll get marked as stale when the extension that introduces this directive changes. This makes sense, although it could get tedious if an extension adds many directives or makes different changes. I don't know if there's a better way.
Inspired by Luc Van Oostenryck's autodoc-C.
As far as I know app.env.note_dependency can be called within the doctree-read to add any file as a dependency to the document currently being read.
So in your use case, I assume this would work:
from typing import Any, Dict
from sphinx.application import Sphinx
import docutils.nodes as nodes
def doctree-read(app: Sphinx, doctree: nodes.document):
app.env.note_dependency(file)
def setup(app: Sphinx):
app.connect("doctree-read", doctree-read)
1. Summary
I can't find, how I can automatically prettify my YAML files.
2. Data
Example:
I have SashaPrettifyYAML.yaml file:
sasha_commands:
# Sasha comment
sasha_command_help: {call: sublime.command_help, caption: 'Sasha Command: Command Help'}
3. Expected behavior
I want to delete {braces}:
sasha_commands:
# Sasha comment
sasha_command_help:
call: sublime.command_help
caption: 'Sasha Command: Command Help'
4. Not helped
Pretty YAML (based on PyYAML) and online formatters as YAML Formatter and OnlineYAMLTools delete comments;
I can't find the required option in ruamel.yaml.cmd;
align-yaml align, not prettify YAML file.
There is no option to do this in ruamel.yaml.cmd, but it is fairly straightforward to do this with a small python program and using ruamel.yaml, by loading and dumping in round-trip mode (the default).
The only thing you need to do is make sure the flow-style on the data-structure that is the value for the key sasha_command_help is set to block-style (which is how I interpret your definition of "prettifying YAML"):
import sys
import ruamel.yaml
yaml_str = """\
sasha_commands:
# Sasha comment
sasha_command_help: {call: sublime.command_help, caption: 'Sasha Command: Command Help'}
"""
yaml = ruamel.yaml.YAML()
yaml.preserve_quotes = True
data = yaml.load(yaml_str)
data['sasha_commands']['sasha_command_help'].fa.set_block_style()
yaml.dump(data, sys.stdout)
this will exactly give the output you expect.
A recursive data structure walker can be found in scalarstring.py in the ruamel.yaml source, and adapted to make a generic "make-everything-block-style" routine:
import sys
import ruamel.yaml
def block_style(base):
"""
This routine walks over a simple, i.e. consisting of dicts, lists and
primitives, tree loaded from YAML. It recurses into dict values and list
items, and sets block-style on these.
"""
if isinstance(base, dict):
for k in base:
try:
base.fa.set_block_style()
except AttributeError:
pass
block_style(base[k])
elif isinstance(base, list):
for elem in base:
try:
base.fa.set_block_style()
except AttributeError:
pass
block_style(elem)
yaml = ruamel.yaml.YAML()
yaml.preserve_quotes = True
file_in = sys.argv[1]
file_out = sys.argv[2]
with open(file_in) as fp:
data = yaml.load(fp)
block_style(data)
with open(file_out, 'w') as fp:
yaml.dump(data, fp)
If you store the above in prettifyyaml.py you can call it with:
python prettifyyaml.py SashaPrettifyYAML.yaml Prettified.yaml
Since you are already using single quotes around the scalar that has embedded spaces, you won't see a change if you leave out yaml.preserve_quotes = True. But if you had used a double quoted scalar then that line makes sure the double quotes are preserved.
I had the same problem. I wrote my own YAML beautifier https://github.com/wangkuiyi/yamlfmt. I hope it helps.
I tried top results from Google, but none of them address the requirements of https://sqlflow.org/sqlflow, which I am leading:
https://pypi.org/project/yamlfmt cannot handle a file of multiple YAML documents separated by ---
https://github.com/devopyio/yamlfmt cannot handle multiple files.
https://github.com/miekg/yamlfmt/blob/master/fmt.go cannot replace (inline edit) the input files.
You can use yq tool - it's easy to install and use, and it's well maintained.
Supposing you have example.yml file to format, it can be processed by following ways:
from file: yq r --unwrapScalar -p pv -P example.yml '*'
from stdin: cat example.yml | yq r --unwrapScalar -p pv -P - '*'
Suppose I have two files a.rst and b.rst in the same folder, and a.rst looks like this
.. _foo: http://stackoverflow.com
`foo`_ is a website
It seems using foo in b.rst is not allowed. Is there a way to define hyperlinks and use them in multiple files?
Followup
I used the extlinks extension as Steve Piercy suggested. Its implementation and docstring can be seen here on github.
In my case, I define wikipedia link in my conf.py
extlinks = {'wiki': ('https://en.wikipedia.org/wiki/%s', '')}
and in the .rst files, use them like
:wiki:`Einstein <Albert_Einstein>`
where Einstein will be displayed as a link to https://en.wikipedia.org/wiki/Albert_Einstein
There are at least four possible solutions.
1. repeat yourself
Put your complete reST in each file. You probably don't want that.
2. combined rst_epilog and substitution
This one is clever. Configure the rst_epilog value, in your conf.py along with a substition with the replace directive:
rst_epilog = """
.. |foo| replace:: foo
.. _foo: http://stackoverflow.com
"""
and reST:
|foo|_ is a website
yields:
<a class="reference external" href="http://stackoverflow.com">foo</a>
3. extlinks
For links to external websites where you want to have a base URL and append path segments or arguments, you can use extlinks in your conf.py:
extensions = [
...
'sphinx.ext.extlinks',
...
]
...
extlinks = {'so': ('https://stackoverflow.com/%s', None)}
Then in your reST:
:so:`questions/49016433`
Yields:
<a class="reference external"
href="https://stackoverflow.com/questions/49016433">
https://stackoverflow.com/questions/49016433
</a>
4. intersphinx
For external websites that are documentation generated by Sphinx, then you can use intersphinx, in your conf.py:
extensions = [
...
'sphinx.ext.intersphinx',
...
]
...
intersphinx_mapping = {
'python': ('https://docs.python.org/3', None),
}
Then in your reST:
:py:mod:`doctest`
Yields:
<a class="reference external"
href="https://docs.python.org/3/library/doctest.html#module-doctest"
title="(in Python v3.6)">
<code class="xref py py-mod docutils literal">
<span class="pre">doctest</span>
</code>
</a>
This might come a bit late but I have found a solution that works very neatly for me and it is not among the answers already given.
In my case, I create a file with all the links used in my project, save it as /include/links.rst, and looking something like:
.. _PEP8: https://www.python.org/dev/peps/pep-0008/
.. _numpydoc: https://numpydoc.readthedocs.io/en/latest/format.html
.. _googledoc: https://google.github.io/styleguide/pyguide.html
Then there are the files a.rst and b.rst looking like:
.. include:: /include/links.rst
File A.rst
##########
Click `here <PEP8_>`_ to see the PEP8 coding style
Alternatively, visit either:
- `Numpy Style <numpydoc_>`_
- `Google Style <googledoc_>`_
and
.. include:: /include/links.rst
File B.rst
##########
You can visit `Python's PEP8 Style Guide <PEP8_>`_
For docstrings, you can use either `Numpy's <numpydoc_>`_ or `Google's <googledoc_>`_
respectively.
The produced output for both cases is:
and
respectively.
Moreover, I would like to emphasize the fact of which I was actually really struggling to achieve, to use different names (displayed text) for the same link at different locations and which I have achieved with the double _, one inside the <..._> and another outside.
This is another solution: it is a bit hacky and a little bit different respect to the officially supported way to share external links.
First complete the Setup then:
in conf.py add the commonlinks entry in extensions
in conf.py configure the map of common links:
For example:
extensions = [
...,
'sphinx.ext.commonlinks'
]
commonlinks = {
'issues': 'https://github.com/sphinx-doc/sphinx/issues',
'github': 'https://github.com'
}
Then in .rst files you can do these:
The :github:`_url_` url is aliased to :github:`GitHub` and also to :github:`this`
Setup
All that is needed is to copy into sphinx/ext directory the file commonlinks.py:
# -*- coding: utf-8 -*-
"""
sphinx.ext.commonlinks
~~~~~~~~~~~~~~~~~~~~~~
Extension to save typing and prevent hard-coding of common URLs in the reST
files.
This adds a new config value called ``commonlinks`` that is created like this::
commonlinks = {'exmpl': 'http://example.com/mypage.html', ...}
Now you can use e.g. :exmpl:`foo` in your documents. This will create a
link to ``http://example.com/mypage.html``. The link caption depends on the
role content:
- If it is ``_url_``, the caption will be the full URL.
- If it is a string, the caption will be the role content.
"""
from six import iteritems
from docutils import nodes, utils
import sphinx
from sphinx.util.nodes import split_explicit_title
def make_link_role(base_url):
def role(typ, rawtext, text, lineno, inliner, options={}, content=[]):
text = utils.unescape(text)
if text == '_url_':
title = base_url
else:
title = text
pnode = nodes.reference(title, title, internal=False, refuri=base_url)
return [pnode], []
return role
def setup_link_roles(app):
for name, base_url in iteritems(app.config.commonlinks):
app.add_role(name, make_link_role(base_url))
def setup(app):
app.add_config_value('commonlinks', {}, 'env')
app.connect('builder-inited', setup_link_roles)
return {'version': sphinx.__display_version__, 'parallel_read_safe': True}
To locate the sphinx installation directory one way is:
$ python 3
> import sphinx
> sphinx
<module 'sphinx' from '/usr/local/lib/python3.5/dist-packages/sphinx/__init__.py'>
then:
% cp commonlinks.py /usr/local/lib/python3.5/dist-packages/sphinx/ext
I want to have a multi-line bit of markdown java in a yam file. I tried many things but I guess I don't quite get the quoting rules of Yaml.
{
title: Museum,
body: |
"```java
code code code
java2",
answers: [
"`museum`",
"`museum.getFloor(3)`",
"`museum.getFloor(3).getExhibit(5)`",
"`museum.getFloor(3).getExhibit(5).getCurator()`",
"`museum.getFloor(3).getExhibit(5).getCurator().name`",
"`museum.getFloor(3).getExhibit(5).getCurator().name.toUpper()`"
]
}
Produces:
/Users/pitosalas/.rbenv/versions/2.3.1/lib/ruby/2.3.0/psych.rb:377:in `parse': (generator/test.yml): found character that cannot start any token while scanning for the next token at line 3 column 9 (Psych::SyntaxError)
YAML has two styles: the JSON like flow style and the much better human readable block style.
Roughly speaking you can have nested structures each style nested within itself and can have flow style nested within block style, but block style nested within flow style is not allowed.
Your to level { and } are flow style but you try to introduce, with |, a literal block style scalar within that flow style. Replace the flow style with block style upwards from that scalar:
title: Museum
body: |
"```java
code code code
java2"
answers: [
"`museum`",
"`museum.getFloor(3)`",
"`museum.getFloor(3).getExhibit(5)`",
"`museum.getFloor(3).getExhibit(5).getCurator()`",
"`museum.getFloor(3).getExhibit(5).getCurator().name`",
"`museum.getFloor(3).getExhibit(5).getCurator().name.toUpper()`"
]
and your YAML is fine. Note that the double quotes "around" the value for the key body are not going to be stripped when loading, maybe that is not what you intended.
You should IMO not leave out the trailing , after the last value in the (flow style) sequence that is the value for answers. This will certainly lead to errors when you extend the list and forget to put in the trailing comma on the line above.
I would personally go for block style all the way:
title: Museum
body: |
"```java
code code code
java2"
answers:
- "`museum`"
- "`museum.getFloor(3)`"
- "`museum.getFloor(3).getExhibit(5)`"
- "`museum.getFloor(3).getExhibit(5).getCurator()`"
- "`museum.getFloor(3).getExhibit(5).getCurator().name`"
- "`museum.getFloor(3).getExhibit(5).getCurator().name.toUpper()`"
When dealing with YAML file generation that is convoluted or complex, or when it's not working as I expect, I revert to letting Ruby show me the way:
require 'yaml'
body = <<EOT
"```java
code code code
java2
"
EOT
answers = %w(
`museum`
`museum.getFloor(3)`
`museum.getFloor(3).getExhibit(5)`
`museum.getFloor(3).getExhibit(5).getCurator()`
`museum.getFloor(3).getExhibit(5).getCurator().name`
`museum.getFloor(3).getExhibit(5).getCurator().name.toUpper()`
)
obj = {
"title" => "Museum",
"body" => body,
"answers" => answers
}
puts obj.to_yaml
Which, in this case, outputs:
---
title: Museum
body: |
"```java
code code code
java2
"
answers:
- "`museum`"
- "`museum.getFloor(3)`"
- "`museum.getFloor(3).getExhibit(5)`"
- "`museum.getFloor(3).getExhibit(5).getCurator()`"
- "`museum.getFloor(3).getExhibit(5).getCurator().name`"
- "`museum.getFloor(3).getExhibit(5).getCurator().name.toUpper()`"
If you then pass that YAML back into the parser, you should get the original data structure back:
YAML.load(obj.to_yaml)
# => {"title"=>"Museum",
# "body"=>"\"```java\n" +
# "code code code\n" +
# "java2\n" +
# "\"\n",
# "answers"=>
# ["`museum`",
# "`museum.getFloor(3)`",
# "`museum.getFloor(3).getExhibit(5)`",
# "`museum.getFloor(3).getExhibit(5).getCurator()`",
# "`museum.getFloor(3).getExhibit(5).getCurator().name`",
# "`museum.getFloor(3).getExhibit(5).getCurator().name.toUpper()`"]}
I have a fileA with a snippet of code, and I need a script to insert that snippet into fileB on the line after a specific pattern.
I'm trying to make the accepted answer in this thread work, but it's not, and is not giving an error so not sure why not:
sed -e '/pattern/r text2insert' filewithpattern
Any suggestions?
pattern (insert snippet on line after):
def boot {
also tried escaped pattern but no luck:
def\ boot\ {
def\ boot\ \{
fileA snippet:
LiftRules.htmlProperties.default.set((r: Req) =>
new Html5Properties(r.userAgent))
fileB (Boot.scala):
package bootstrap.liftweb
import net.liftweb._
import util._
import Helpers._
import common._
import http._
import sitemap._
import Loc._
/**
* A class that's instantiated early and run. It allows the application
* to modify lift's environment
*/
class Boot {
def boot {
// where to search snippet
LiftRules.addToPackages("code")
// Build SiteMap
val entries = List(
Menu.i("Home") / "index", // the simple way to declare a menu
// more complex because this menu allows anything in the
// /static path to be visible
Menu(Loc("Static", Link(List("static"), true, "/static/index"),
"Static Content")))
// set the sitemap. Note if you don't want access control for
// each page, just comment this line out.
LiftRules.setSiteMap(SiteMap(entries:_*))
// Use jQuery 1.4
LiftRules.jsArtifacts = net.liftweb.http.js.jquery.JQuery14Artifacts
//Show the spinny image when an Ajax call starts
LiftRules.ajaxStart =
Full(() => LiftRules.jsArtifacts.show("ajax-loader").cmd)
// Make the spinny image go away when it ends
LiftRules.ajaxEnd =
Full(() => LiftRules.jsArtifacts.hide("ajax-loader").cmd)
// Force the request to be UTF-8
LiftRules.early.append(_.setCharacterEncoding("UTF-8"))
}
}
The sed format looks correct to me.
To help you diagnose this, try it with two simpler text files and a simple pattern.
File filewithpattern:
hello
world
File textinsert:
foo
goo
Now run sed:
sed -e '/hello/r textinsert' filewithpattern
You should see this:
hello
foo
goo
world
Does that work for you?
If so, then edit filewithpattern to use your target:
hello
def boot {
world
Run the command:
sed -e '/def boot {/r textinsert' filewithpattern
You should see this:
hello
def boot {
foo
goo
world
If you want variable substitution, try this:
#!/bin/bash
PATTERN='def boot {'
sed -e "/${PATTERN}/r textinsert" filewithpattern