In Sphinx is possible to include the signature of a function or method manually using the py:function (or py:method) directive:
.. py:function:: my_func(data, named=None, *args, *kwargs)
It is also possible to use autodoc directives to include and format the whole docstring of a function or method:
.. automethod:: my_func
I am wondering if there is a way of configuring autodoc to include and format only the signature, without the rest of the docstring, so that I don't have to do it manually.
autodoc-process-signature can be used here as well.
def process_signature(app, what, name, obj, options, signature, return_annotation):
return modified_signature, modified_return_annotation
# will be rendered to method(modified_signature) -> modified_return_annotation
def setup(app):
app.connect("autodoc-process-signature", process_signature)
http://www.sphinx-doc.org/en/master/_modules/sphinx/ext/autodoc.html
See autodoc's sphinx.ext.autodoc.between.
Return a listener that either keeps, or if exclude is True excludes, lines between lines that match the marker regular expression. If no line matches, the resulting docstring would be empty, so no change will be made unless keepempty is true.
If what is a sequence of strings, only docstrings of a type in what will be processed.
Related
So I have a string from a rendered template that looks like
"Dear {{user_name}},\r\n\r\nThank you for your purchase. If you have any questions, we are happy to help.\r\n\r\n\r\n{{company_name}}\r\n{{company_phone_number}}\r\n"
All those variables like {{user_name}} are optional and do not need to be included but I want to check that if they are, they have {{ in front of the variable name. I am using liquid to parse and render the template and couldn't get it to catch if the user only uses 1 (or no) opening brackets. I was only able to catch the proper number of closing brackets. So I wrote a method to check that if these variables exist, they have the correct opening brackets. It only works, however, if all those variables are found.
here is my method:
def validate_opening_brackets?(template)
text = %w(user_name company_name company_phone_number)
text.all? do |variable|
next unless template.include? variable
template.include? "{{#{variable}"
end
end
It works, but only if all variables are present. If, for example, the template created by the user does not include user_name, then it will return false. I've also done this loop using each, and creating a variable outside of the block that I assign false if the conditions are not met. I would really, however, like to get this to work using the all? method, as I can just return a boolean and it's cleaner.
If the question is about how to rewrite the all? block to make it return true if all present variable names have two brackets before them and false otherwise then you could use something like this:
def validate_opening_brackets?(template)
variables = %w(user_name company_name company_phone_number)
variables.all? do |variable|
!template.include?(variable) || template.include?("{{#{variable}")
end
end
TL;DR
There are multiple ways to do this, but the easiest way I can think of is to simply prefix/postfix a regular expression with the escaped characters used by Mustache/Liquid, and using alternation to check for each of your variable names within the template variable characters (e.g. double curly braces). You can then use String#scan and then return a Boolean from Enumerable#any? based on the contents of the Array returned by from #scan.
This works with your posted example, but there may certainly be other use cases where you need a more complex solution. YMMV.
Example Code
This solution escapes the leading and trailing { and } characters to avoid having them treated as special characters, and then interpolates the variable names with | for alternation. It returns a Boolean depending on whether templated variables are found.
def template_string_has_interpolations? str
var_names = %w[user_name company_name company_phone_number]
regexp = /\{\{#{var_names.join ?|}\}\}/
str.scan(regexp).any?
end
Tested Examples
template_string_has_interpolations? "Dear {{user_name}},\r\n\r\nThank you for your purchase. If you have any questions, we are happy to help.\r\n\r\n\r\n{{company_name}}\r\n{{company_phone_number}}\r\n"
#=> true
template_string_has_interpolations? "Dear Customer,\r\n\r\nThank you for your purchase. If you have any questions, we are happy to help.\r\n\r\n\r\nCompany, Inc.\r\n(555) 555-5555\r\n"
#=> false
I want to open this file and get all elements that start with us-gaap.
ftp://ftp.sec.gov/edgar/data/916789/0001558370-15-001143.txt
To get elements I tried like this:
str = '<html><body><us-gaap:foo>foo</us-gaap:foo></body></html>'
doc = Nokogiri::XML(File.read(str))
doc.xpath('//us-gaap:*')
Nokogiri::XML::XPath::SyntaxError: Undefined namespace prefix: //us-gaap:*
from /Users/ironsand/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.2/lib/nokogiri/xml/searchable.rb:165:in `evaluate'
doc.namespaces returns {}, so I think I have to add namespace us-gaap.
There are some questions about "adding namespace with Nokogiri", but it looks like about how to create a new XML document, not how to add a namespace to existing documents.
How can I add a namespace to existing document?
I know I can remove the namespace by Nokogiri::XML::Document#remove_namespaces!, but I don't want to use it because it removes also necesarry information.
You have asked an XY Problem. You think that the problem is that you need to add a missing namespace; the real problem is that the file you're trying to parse is not valid XML.
require 'nokogiri'
doc = Nokogiri.XML( IO.read('0001558370-15-001143.txt') )
doc.errors.length
#=> 5716
For example, the <ACCEPTANCE-DATETIME> 'element' opened on line 3 is never closed, and on line 16 there is a raw ampersand in the text:
STANDARD INDUSTRIAL CLASSIFICATION: ELECTRIC HOUSEWARES & FANS [3634]
which ought to be escaped as an entity.
However, the document has valid XML fragments within it! In particular, there is one XML document that defines xmlns:us-gaap namespace, from lines 27243-49312. Let's extract just that, using only the knowledge that the root element defines the namespace we want, and the assumptions that no element with the same name is nested within the document, and that the root element does not have an unescaped > character in any attribute. (These assumptions are valid for this file, but may not be valid for every XML file.)
txt = IO.read('0001558370-15-001143.txt')
gaap_finder = %r{(<(\w+) [^>]+xmlns:us-gaap=.+?</\2>)}m
txt.scan(gaap_finder) do |xml,_|
doc = Nokogiri.XML( xml )
gaaps = doc.xpath('//us-gaap:*')
p gaaps.length
#=> 569
end
The code above handles the case where there may be more than one XML document in the txt file, though in this case there is only one.
Decoded, the gaap_finder regex says this:
%r{...}m — this is a regular expression (that allows slashes in it, unescaped) with "multiline mode", where a period will match newline characters
(...) — capture everything we find
< — start with a literal "less-than" symbol
(\w+) — find one or more word characters (the tag name), and save them
— the word characters must be followed by a space (important to avoid capturing the <xsd:xbrl ...> element in this file)
[^>]+ — followed by one or more characters that is NOT a "greater-than" symbol (to ensure that we stay in the same element that we started in)
xmlns:us-gaap\s*= — followed by this literal namespace declaration (which may have whitespace separating it from the equals sign)
.+? — followed by anything (as little as possible)...
</\2> — ...up until you see a closing tag with the same name as what we captured for the name of the starting tag
Because of the way scan works when the regex has capturing groups, each result is a two-element array, where the first element is the entire captured XML and the second element is the name of the tag that we captured (which we "discard" by assigning it to the _ variable).
If you want to be less magic about your capturing, the text file format appears to always wrap each XML document in <XBRL>...</XBRL>. So, you could do this to process every XML file (there are seven, five of which do not happen to have any us-gaap namespaces):
txt = IO.read('0001558370-15-001143.txt')
xbrls = %r{(?<=<XBRL>).+?(?=</XBRL>)}m # find text inside <XBRL>…</XBRL>
txt.scan(xbrls) do |xml|
doc = Nokogiri.XML( xml )
if doc.namespaces["xmlns:us-gaap"]
gaaps = doc.xpath('//us-gaap:*')
p gaaps.length
end
end
#=> 569
#=> 0 (for the XML Schema document that defines the namespace)
I couldn't figure out how to update an existing doc with a new namespace, but since Nokogiri will recognize namespaces on the root element, and those namespaces are, syntactically, just attributes, you can update the document with a new namespace declaration, serialize the doc to a string, and re-parse it:
str = '<html><body><us-gaap:foo>foo</us-gaap:foo></body></html>'
doc_without_ns = Nokogiri::XML(str)
doc_without_ns.root['xmlns:us-gaap'] = 'http://your/actual/ns/here'
doc = Nokogiri::XML(doc_without_ns.to_xml)
doc.xpath("//us-gaap:*")
# Returns [#<Nokogiri::XML::Element:0x3ff375583f9c name="foo" namespace=#<Nokogiri::XML::Namespace:0x3ff375583f24 prefix="us-gaap" href="http://your/actual/ns/here"> children=[#<Nokogiri::XML::Text:0x3ff375583768 "foo">]>]
I am developing a parser in Ruby using the parslet library.
The language I am parsing has a lot of keywords that can be merged into a single parsing rule like this:
rule(:keyword) {
str('keyword1') |
str('keyword2') |
str('keyword2') ...
}
Is there a good way to generate this set of lines of code dynamically, by reading a text file with all the keywords?
This would help me keep my parser clean and small, making it easier to add new keywords without modifying the code.
The pseudo-code of what I want to embed inside the rule(:keyword) would be somethings like this:
File.read("keywords.txt").each { |k| write_line " str(\'#{k}\') "}
So far, the workaround I have found is to have a separate ruby program loading the parser code as:
keywords = ["keyword1", "keyword2","keyword3"]
subs = {:keyword_list => keywords .inject("") { |a,k| a << "str('#{k}') | \n"} }
eval( File.read("parser.rb") % subs)
where the parser code has the following lines:
rule(:keywords){
%{keyword_list}
}
Is there a more elegant way to achieve this?
You can try something like this:
rule(:keyword) {
File.readlines("keywords.txt").map { |k| str(k.chomp) }.inject(&:|)
}
In this case, you don't really need to "generate lines of code". As #Uri tried to explain in his answer, there's nothing special about the contents of that rule method; it's just plain Ruby code. Because of this, anything you can do in Ruby you can do inside that rule method as well, including read files, dynamically call methods, and call methods on objects.
Let me break down your existing code, so I can better explain how a dynamic solution to the same problem would work:
rule(:keyword) {
# Stuff here
}
This code right here calls a rule method and passes it :keyword and a block of code. At some point, parslet will call that block and check its return value. Parslet might choose to call the block using instance_exec, which can change the context the block is being executed in to make methods not available outside the block (like str, perhaps) available inside it.
str('keyword1')
Here, inside the context of the rule block, you are calling a method named str with the string "keyword1", and getting the result. Nothing special here, this is just a normal method call.
str('keyword1') | str('keyword2')
Here, the | operator is actually just a method being called on whatever str('keyword1') is returning. This code is equivalent to str('keyword1').send(:'|', str('keyword2')).
str('keyword1') |
str('keyword2') |
str('keyword2')
Same as before, except this time we're calling | on whatever str('keyword1').send(:'|', str('keyword2')) returned. The result of this method call is returned to the rule method when it calls the block.
So now that you know how all this works, you can perform exactly the same operations (calling str with each keyword, and using the | method to "add up" the results) dynamically, based on the contents of a file perhaps:
rule(:keyword) {
File.readlines("keywords.txt").map(&:chomp).map { |k| str(k) }.inject(:|)
}
Breakdown:
rule(:keyword) { # Call the rule method with the `:keyword` argument, and pass
# it this block of code.
File.readlines("keywords.txt"). # Get an array of strings containing all the
# keywords
map(&:chomp). # Remove surrounding whitespace from each keyword in the array,
# by calling `chomp` on them. (The strings returned by
# `File.readlines` include the newline character at the end of
# each string.)
map { |k| str(k) }. # Convert each keyword in the array into whatever is
# returned by calling `str` with that keyword.
inject(:|) # Reduce the returned objects to a single one using the `|`
# method on each object. (Equivalent to obj1 | obj2 | obj3...)
}
And that's it! See? No need to generate any lines of code, just do what the real code is doing, but do it dynamically!
I want to create several files from a single template, which differ only by a variable name. For example :
(file1.rst):
.. |variable| replace:: 1
.. include template.rst
(template.rst) :
Variable |variable|
=====================
Image
-------
.. image:: ./images/|variable|-image.png
where of course I have an image called "./images/1-image.png". The substitution of "|variable|" by "1" works well in the title, but not in the image file name, and at compilation I get :
WARNING: image file not readable: ./images/|variable|-image.png
How can I get reST to make the substitution in the variable name too? (if this changes anything, am using Sphinx).
There are two problems here: a substitution problem, and a parsing order problem.
For the first problem, the substitution reference |variable| cannot have adjacent characters (besides whitespace or maybe _ for hyperlinking) or else it won't parse as a substitution reference, so you need to escape it:
./images/\ |variable|\ -image.png
However, the second problem is waiting around the corner. While I'm not certain of the details, it seems reST is unable to parse substitutions inside other directives. I think it first parses the image directive, which puts it in the document tree and thus out of reach of the substitution mechanism. Similarly, I don't think it's possible to use a substitution to insert content intended to be parsed (e.g. .. |img1| replace::`.. image:: images/1-image.png`). This is all speculative based on some tests and my incomplete comprehension of the official documentation, so someone more knowledgeable can correct what I've said here.
I think you're aware of the actual image substitution directive (as opposed to text substitution), but I don't think it attains the generality you're aiming for (you'll still need a separate directive for the image as from the |variable|), but in any case it looks like this:
.. |img1| image:: images/1-image.png
Since you're using Sphinx, you can try creating your own directive extension (see this answer for information), but it won't solve the substitutions-inside-markup problem.
You have to create a custom directive in this case as Sphinx doesn't allow you to substitute image paths. You can change Sphinx figure directive as follows and use it instead of the image directive.
from typing import Any, Dict, List, Tuple
from typing import cast
from docutils import nodes
from docutils.nodes import Node, make_id, system_message
from docutils.parsers.rst import directives
from docutils.parsers.rst.directives import images, html, tables
from sphinx import addnodes
from sphinx.directives import optional_int
from sphinx.domains.math import MathDomain
from sphinx.util.docutils import SphinxDirective
from sphinx.util.nodes import set_source_info
if False:
# For type annotation
from sphinx.application import Sphinx
class CustomFigure(images.Figure):
"""The figure directive which applies `:name:` option to the figure node
instead of the image node.
"""
def run(self) -> List[Node]:
name = self.options.pop('name', None)
path = self.arguments[0] #path = ./images/variable-image.png
#replace 'variable' from th.e given value
self.argument[0] = path.replace("variable", "string substitution")
result = super().run()
if len(result) == 2 or isinstance(result[0], nodes.system_message):
return result
assert len(result) == 1
figure_node = cast(nodes.figure, result[0])
if name:
# set ``name`` to figure_node if given
self.options['name'] = name
self.add_name(figure_node)
# copy lineno from image node
if figure_node.line is None and len(figure_node) == 2:
caption = cast(nodes.caption, figure_node[1])
figure_node.line = caption.line
return [figure_node]
def setup(app: "Sphinx") -> Dict[str, Any]:
directives.register_directive('figure', Figure)
return {
'version': 'builtin',
'parallel_read_safe': True,
'parallel_write_safe': True,
}
You can add this CustomFigure.py directive in the conf.py of the project and use the customfigure directive across Sphinx project instead of the Image directive. Refer http://www.sphinx-doc.org/en/master/usage/extensions/index.html to add a custom directive to your Sphinx project.
I have a Lisp-like language I would like to highlight using Pygments in Sphinx code snippet documentation. My approach is to extend the existing CommonLispLexer to add the built-in names using a NameHighlightFilter. However, it is not working, so I must be missing something obvious. I have added the following to my conf.py:
def setup(app):
from sphinx.highlighting import lexers
from pygments.lexers import CommonLispLexer
from pygments.token import Name
from pygments.filters import NameHighlightFilter
tl_lexer = CommonLispLexer()
tl_lexer.add_filter(NameHighlightFilter(
names=['define-function', 'define-macro',
'define-variable', 'define-constant'],
tokentype=Name.Builtin,
))
app.add_lexer('tl', tl_lexer)
highlight_language = 'tl'
But the NameHighlightFilter has no effect. Code blocks are highlighted as if they were Lisp, but my new builtin names have no special highlighting.
The reason is that the NameHighlighFilter only converts tokens that the lexer categorizes as Token.Name, but the CommonLispLexer categorizes almost everything as Name.Variable. This is the filter function of the NameHighlightFilter, from the Pygments source code:
def filter(self, lexer, stream):
for ttype, value in stream:
if ttype is Name and value in self.names:
yield self.tokentype, value
else:
yield ttype, value
My only workaround was to write my own filter. This function gave me the look I wanted.
def filter(self, lexer, stream):
define = False
for ttype, value in stream:
if value in self.tl_toplevel_forms:
ttype = Name.Builtin
define = True
elif define and ttype == Name.Variable:
define = False
ttype = Name.Function
elif value in self.tl_special_forms:
ttype = Name.Variable
# the Common Lisp lexer highlights everything else as
# variables, which isn't the look I want. Instead
# highlight all non-special things as text.
elif ttype == Name.Variable:
ttype = Name.Text
yield ttype, value
As a note to Pygments developers, perhaps the NameHighlightFilter could take an optional argument representing the token type(s) to be converted (currently it only takes the output token type).