I used PycParser to generate an abstract syntax tree for a C function, but I'm still trying to figure out how to convert the parse tree into a string:
from pycparser import c_parser
src = '''
int hello(a, b){
return a + b;
}
'''
ast = c_parser.CParser().parse(src)
aString = ast.show()
print(aString) #This prints "None" instead of printing the parse tree
Would it be possible to generate a string (or an array of strings) from the parse tree that has been generated?
This is the abstract syntax tree that was printed, but I can't figure out how to convert it to a string:
FileAST:
FuncDef:
Decl: hello, [], [], []
FuncDecl:
ParamList:
ID: a
ID: b
TypeDecl: hello, []
IdentifierType: ['int']
Compound:
Return:
BinaryOp: +
ID: a
ID: b
The show method accepts a buf keyword parameter - you can pass a StringIO there. sys.stdout is the default. See https://github.com/eliben/pycparser/blob/master/pycparser/c_ast.py#L30
Try this, please
from pycparser import c_parser
src = '''
int hello(a, b){
return a + b;
}
'''
ast = c_parser.CParser().parse(src)
ast_buffer_write=open("buffer.txt","w") # here will be your AST source
ast.show(ast_buffer_write)
ast_buffer_write.close()
ast_buffer_read=open("buffer.txt","r")
astString=ast_buffer_read.read()
print(astString)
Related
protobuf:
message test
{
int16 a:1
message testdata{
int16 b:1 `
int16 c:2
repeated testdata test_data=2
}
}
dt=test()
dt.a=11
dt1=d.testdata.add()
dt1.b=2222
dt1.c=3333
send=dt.SerializeToString()
now while deserialising code as below:
t=test()
t.ParseFromString(send)
t1=t.testdata()
print(t1.test_data)----> empty string
print(t1.b)--> value is not printing as expected, its printing object name instead of value.
how to access the nested protobuf elements and any input or suggestions are welcome.
I encourage you to read the documentation.
Your protobuf file is invalid.
A working equivalent of what you've written could be:
example.proto:
syntax = "proto3";
message test {
int32 a=1;
message testdata {
int32 b=1;
int32 c=2;
}
repeated testdata test_data=2;
}
Compiled using:
protoc \
--proto_path=${PWD} \
--python_out=${PWD} \
${PWD}/example.proto
Example Python code (with apologies for my poor variable naming choices):
import example_pb2
from google.protobuf.message import DecodeError
# Create a test message (f)
f = example_pb2.test()
f.a = 1
# Add a repeated property of type testdata
g=f.test_data.add()
g.b=1
g.c=1
# Add a repeated property of type testdata
g=f.test_data.add()
g.b=2
g.c=2
# Add a repeated property of type testdata
g=f.test_data.add()
g.b=3
g.c=3
# Output it
print(f)
# Serialize it and output it
x=f.SerializeToString()
print(x)
# Create a new test message (h) and parse x into it
h=example_pb2.test()
h.ParseFromString(x)
print(h)
# Create incorrectly formatted message (y) as a variant of x
y=b'\x08\x01\x12\x12\x04\x08\x01\x10\x01\x12\x04\x08\x02\x10\x02\x12\x04\x08\x03\x10\x03'
j=example_pb2.test()
try:
j.ParseFromString(y)
print(j)
except DecodeError as e:
print(e)
Yielding:
a: 1
test_data {
b: 1
c: 1
}
test_data {
b: 2
c: 2
}
test_data {
b: 3
c: 3
}
b'\x08\x01\x12\x04\x08\x01\x10\x01\x12\x04\x08\x02\x10\x02\x12\x04\x08\x03\x10\x03'
a: 1
test_data {
b: 1
c: 1
}
test_data {
b: 2
c: 2
}
test_data {
b: 3
c: 3
}
Error parsing message
I have a string sequence Seq[String] which represents stdin input lines.
Those lines map to a model entity, but it is not guaranteed that 1 line = 1 entity instance.
Each entity is delimited with a special string that will not occur anywhere else in the input.
My solution was something like:
val entities = lines.mkString.split(myDelimiter).map(parseEntity)
parseEntity implementation is not relevant, it gets a String and maps to a case class which represents the model entity
The problem is with a given input, I get an OutOfMemoryException on the lines.mkString. Would a fold/foldLeft/foldRight be more efficient? Or do you have any better alternative?
You can solve this using akka streams and delimiter framing. See this section of the documentation for the basic approach.
import akka.actor.ActorSystem
import akka.stream.ActorMaterializer
import akka.stream.scaladsl.{Framing, Source}
import akka.util.ByteString
val example = (0 until 100).mkString("delimiter").grouped(8).toIndexedSeq
val framing = Framing.delimiter(ByteString("delimiter"), 1000)
implicit val system = ActorSystem()
implicit val mat = ActorMaterializer()
Source(example)
.map(ByteString.apply)
.via(framing)
.map(_.utf8String)
.runForeach(println)
The conversion to and from ByteString is a bit annoying, but Framing.delimiter is only defined for ByteString.
If you are fine with a more pure functional approach, fs2 will also offer primitives to solve this problem.
Something that worked for me if you are reading from a stream (your mileage may vary). Slightly modified version of Scala LineIterator:
class EntityIterator(val iter: BufferedIterator[Char]) extends AbstractIterator[String] with Iterator[String] {
private[this] val sb = new StringBuilder
def getc() = iter.hasNext && {
val ch = iter.next
if (ch == '\n') false // Replace with your delimiter here
else {
sb append ch
true
}
}
def hasNext = iter.hasNext
def next = {
sb.clear
while (getc()) { }
sb.toString
}
}
val entities =
new EnityIterator(scala.io.Source.fromInputStream(...).iter.buffered)
entities.map(...)
I am writing an simple parser&rewriter tools for PL/SQL, and I've complete the parser and get the AST,but now I got two problems :
How can I get certain nodes of the AST ,so as to change the value of them?
After change the nodes ,how can i regenerate the SQL from the updated AST
does the ANTLR AST provide similar interface to do this?
Example SQL: select a,b from t where a=2
After parser the sql and get the ast, I want to change the sql into
select fun(a),b from t where a = fun1(2);
BTW, I generate the AST for C with ANTLR,
Thank you for any suggestion!
See my SO answer on how to regenerate source code from an AST.
Its a lot more work than you think.
ANTLR provides some help in the form of string templates but you may find these a mixed blessing: while they can generate code text, they will generate precisely what is in the template, and you may want to regenerate the code according to its original layout,
which the layout of the string template wants to override.
The following code will walk the AST and print all AST nodes to stderr.
The same tree walker is the basis for a tree transformer that can replace tree nodes.
Allocate new tree nodes with: (pANTLR3_BASE_TREE)(psr->adaptor->nilNode(psr->adaptor));
Delete AST nodes with: parentASTnode->deleteChild(parentASTnode, nodeIndex);
[deleteChild does not free the deleted nodes]
Replace nodes with: parentASTnode->replaceChildren(parentASTnode, nStartChildIndex, nStopChildIndex, newASTnode);
[you cannot insert nodes in the middle of an AST tree level, you can only replace nodes or add to the end of the parent nodes child list]
void printTree(pANTLR3_BASE_TREE t, int indent)
{
pANTLR3_BASE_TREE child = NULL;
int children = 0;
char * tokenText = NULL;
string ind = "";
int i = 0;
if ( t != NULL )
{
children = t->getChildCount(t);
for ( i = 0; i < indent; i++ )
ind += " ";
for ( i = 0; i < children; i++ )
{
child = (pANTLR3_BASE_TREE)(t->getChild(t, i));
tokenText = (char *)child->toString(child)->chars;
fprintf(stderr, "%s%s\n", ind.c_str(), tokenText);
if (tokenText == "<EOF>")
break;
printTree(child, indent+1);
}
}
}
// Run the parser
pANTLR3_BASE_TREE langAST = (psr->start_rule(psr)).tree;
// Print the AST
printTree(langAST, 0);
// Get the Parser Errors
int nErrors = psr->pParser->rec->state->errorCount;
I am trying to load images so I can use them as textures. I have libpng, but how do find a path to an image? Is it a bad idea to put it into .apk? Is there a better way?
Have a look at this:
http://www.anddev.org/ndk_opengl_-_loading_resources_and_assets_from_native_code-t11978.html
It appears there are two ways of doing it. I can either do what Simon N. has suggested or I can convert the images into a C Array and compile them into the source. I choose to do the latter as I can do it easily on the native side. I have even created a python (v3.2) for converting a png file to a c array header and source file. It requires PyPNG
#!/usr/bin/env python
import png
import re
import itertools
def grouper(n, iterable, fillvalue=None):
args = [iter(iterable)] * n
return itertools.zip_longest(fillvalue=fillvalue, *args)
def expand(array):
new_array = [];
for row in array:
for v in row:
new_array.append(v)
return new_array
def c_array_name(filename):
'''converts the filename to the c array name'''
return re.sub(r'\W', '_', filename)
def c_array_header_filename(filename):
'''converts the filename to the c array header filename'''
return "{0}.h".format(filename)
def c_array_source_filename(filename):
'''converts the filename to the c array source filename'''
return "{0}.cpp".format(filename)
def c_array_header(filename):
'''returns a string that is the c array header,
where
filename is the png file'''
name = c_array_name(filename)
return """
#ifndef __{0}__
#define __{0}__
#include <stdint.h>
extern uint_least32_t const {1}[];
#endif /* __{0}__ */
""".format(name.upper(), name)
def c_array_source(filename, header_filename, array_string):
'''returns a string that is the c array source,
where
name is the value from c_array_name
array_string'''
name = c_array_name(filename)
return """
#include "{0}"
uint_least32_t const {1}[] = {{
{2}
}};
""".format(header_filename, name, array_string)
def convert_data_array_string(data):
'''returns a string of hexes of bytes,
where
data is a map of bytes'''
return ", ".join(["0x{:02x}{:02x}{:02x}{:02x}".format(a, b, g, r)
for r, g, b, a in grouper(4, expand(data), 0)])
def png_data_from_file(path):
'''returns a map of bytes of the png file'''
with open(path, 'rb') as file:
reader = png.Reader(file = file)
data = reader.read();
return list(data[2]);
if __name__ == '__main__':
import sys
import os
if len(sys.argv) != 2:
sys.stdout.write("{0} image_path".format(sys.argv[0]))
exit()
path = sys.argv[1]
filename = os.path.split(path)[1]
header_filename = c_array_header_filename(filename)
sys.stdout.write("Creating header file '{}'... ".format(header_filename))
with open(header_filename, 'w') as header_file:
header_file.write(c_array_header(filename))
sys.stdout.write("done\n")
source_filename = c_array_source_filename(filename)
sys.stdout.write("Creating source file '{}'... ".format(source_filename))
data = png_data_from_file(path)
with open(source_filename, 'w') as source_file:
source_file.write(c_array_source(filename, header_filename, convert_data_array_string(data)))
del data
sys.stdout.write("done\n")
We've been profiling our code recently and we've come across a few annoying hotspots. They're in the form
assert(a == b, a + " is not equal to " + b)
Because some of these asserts can be in code called a huge amount of times the string concat starts to add up. assert is defined as:
def assert(assumption : Boolean, message : Any) = ....
why isn't it defined as:
def assert(assumption : Boolean, message : => Any) = ....
That way it would evaluate lazily. Given that it's not defined that way is there an inline way of calling assert with a message param that is evaluated lazily?
Thanks
Lazy evaluation has also some overhead for the function object created. If your message object is already fully constructed (a static message) this overhead is unnecessary.
The appropriate method for your use case would be sprintf-style:
assert(a == b, "%s is not equal to %s", a, b)
As long as there is a speciaized function
assert(Boolean, String, Any, Any)
this implementation has no overhead or the cost of the var args array
assert(Boolean, String, Any*)
for the general case.
Implementing toString would be evaluated lazily, but is not readable:
assert(a == b, new { override def toString = a + " is not equal to " + b })
It is by-name, I changed it over a year ago.
http://www.scala-lang.org/node/825
Current Predef:
#elidable(ASSERTION)
def assert(assertion: Boolean, message: => Any) {
if (!assertion)
throw new java.lang.AssertionError("assertion failed: "+ message)
}
Thomas' answer is great, but just in case you like the idea of the last answer but dislike the unreadability, you can get around it:
object LazyS {
def apply(f: => String): AnyRef = new {
override def toString = f
}
}
Example:
object KnightSpeak {
override def toString = { println("Turned into a string") ; "Ni" }
}
scala> assert(true != false , LazyS("I say " + KnightSpeak))
scala> println( LazyS("I say " + KnightSpeak) )
Turned into a string
I say Ni
Try: assert( a==b, "%s is not equals to %s".format(a,b))
The format should only be called when the assert needs the string. Format is added to RichString via implicit.