Can't figure out how to turn off colored logging when written to disk with structlog - structlog

I'm just starting to learn how to use structlog and I'm having a difficult time trying to figure out how to turn off colored logging when it writes to files. Essentially what I did was take my old code that I had used with the standard logging module and converted it to work with structlog - this is what I came up with:
formatter = logging.Formatter(
fmt=LOGGER_OUTPUT_FORMAT,
datefmt=LOGGER_DATETIME_FORMAT,
)
handler = logging.StreamHandler()
handler.setLevel(level)
handler.setFormatter(formatter)
logger.addHandler(handler)
# create a file handler
formatter = logging.Formatter(
fmt=LOGGER_OUTPUT_FORMAT,
datefmt=LOGGER_DATETIME_FORMAT,
)
handler = RotatingFileHandler(
filename=LOGFILE_PATH,
maxBytes=4000000,
backupCount=20,
)
handler.setLevel(level)
handler.setFormatter(formatter)
logger.addHandler(handler)
structlog.configure(logger_factory=LoggerFactory())
logger = structlog.getLogger('output_logger')
What I can't figure out how to do is where to insert lines to change the formatter such that it doesn't use colored output when saving the logging output to a file. I'm guessing I can uninstall colorama, but that seems like it would be defeating the purpose of actually learning how to use structlog. I've dug through the structlog manual but it's just not making sense to me, and everything I seem to try throws errors, so any guidance would be much appreciated!

Depending on what you want to achieve exactly, you'll have to add more configuration to structlog.configure.
If you want stdlib logging and structlog to cooperate, check out https://www.structlog.org/en/stable/standard-library.html
If you just want to use structlog's native loggers without colors, you only have to adapt your processors for now.
As the Getting Started tutorial says, the default is
[
structlog.processors.StackInfoRenderer(),
structlog.dev.set_exc_info,
structlog.processors.format_exc_info,
structlog.processors.TimeStamper(),
structlog.dev.ConsoleRenderer()
]
Which is colorful if colorama is present. The simplest change would be changing it to
[
structlog.processors.StackInfoRenderer(),
structlog.dev.set_exc_info,
structlog.processors.format_exc_info,
structlog.processors.TimeStamper(),
structlog.dev.ConsoleRenderer(colors=False)
]
although production code should probably use structlog.processors.KeyValueRenderer or structlog.processors.JSONRenderer since ConsoleRenderer is optimized for human consumption in a…console. :)
N.B. That the default for colors is in the API docs is currently wrong, because it's actually based on the presence of colorama and colorama is not present when building the docs.

Related

Find what settings influence a given Pylint message

When Pylint warns me about something, how can I know what settings it's applying? And what section of .pylintrc this setting goes in?
For example:
Used builtin function 'map'. Using a list comprehension can be clearer. (bad-builtin) is because bad-functions in [BASIC] contains map.
Missing function docstring (missing-docstring) is conditional on docstring-min-length in [BASIC].
invalid-name obeys several [BASIC] settings: variable-rgx, method-rgx, module-rgx, etc.
I expected to find this information in the Pylint documentation but it generally doesn't provide this information. For example, there's no link from the documentation of missing-docstring or invalid-names to the applicable options. With bad-builtin, the information is easy to find, but only because it's from a specific extension. The Pylint message wiki has explanations about what the message means, but not what brought it up. The comments in pylint --generate-rcfile don't relate the settings to warning codenames either (nothing there explains that e.g. bad-functions influences the bad-builtin message).
The reason I'm asking this is that in many cases, Pylint's conventions don't match my project's needs and conventions, and I want to tune it rather than ignore specific instances or ignore messages wholesale (so I'm not looking for disable=!). So how do I find what to tune?
How do I find what to tune?
So it seems like you're asking about how you can determine the combination of factors that lead to a pylint message being raised. Using the bad-builtin example, a warning message that is not listed at all in the .pylintrc, a helpful verbose message would be:
bad-builtin: Used builtin function 'map'. Using a list comprehension can be clearer.
---> (Hypothetical) VERBOSE OUTPUT:
bad-builtin was raised because the bad-functions setting in your .pylintrc under the [BASIC] section contains the function map. If you want to "tune" this behavior look here.
Short answer: I am not sure this exists.
When you run pylint my_file.py the first thing it should tell you is the configuration file it is using. From there, you want to find the .pylintrc in use or else you want to create it. Otherwise it will be using the default configuration. If pylint does not tell you what config you are using you can figure out your current pylintrc configuration by running something like:
pylint --generate-rcfile &> output.txt
When you inspect output.txt it should look like this example pylintrc. It is organized into different categories. You can get more info on this without generating an rcfile by running pylint --long-help. The sections include:
[MASTER]
[MESSAGES CONTROL]
[REPORTS]
[BASIC]
[ELIF]
[TYPECHECK]
[FORMAT]
[MISCELLANEOUS]
[VARIABLES]
[LOGGING]
[SIMILARITIES]
[SPELLING]
[IMPORTS]
[DESIGN]
[CLASSES]
[EXCEPTIONS]
I am not sure what you mean by, "The Pylint message wiki has explanations about what the message means, but not what brought it up." When I run pylint on a file or in a code base it tells me exactly where the offending line is.
Then I usually either:
Fix the warning
Disable the warning
Modify the .pylintrc to meet my needs
If you want to tune a specific message type but you are not sure how to do so, then I think you are right that is something that could be better documented. The question: Is XYZ warning message 'tunable'? Or is the only option to fix OR disable? This page lists the pylint features but I do not see anything about bad built-in, for example, let alone the options that are available to apply to a message type.
UPDATE: I am not sure what version of pylint you are on, but it looks like the bad-builtin was specifically addressed in pylint version 1.6. I was going to say we should look at what is going on in the actual source code to generate this message, but I think that requires figuring out what version of pylint you are on first, then digging into the code, then figuring out how to tune it? I agree that the docs can be more clear on how to tune specific warnings:
The bad-builtin check was moved into an extension.
The check was complaining about used builtin functions which were
supposed to not be used. For instance, map and filter were falling
into this category, since better alternatives can be used, such as
list comprehensions. But the check was annoying, since using map or
filter can have its use cases and as such, we decided to move it to an
extension check instead. It can now be enabled through
--load-plugins=pylint.extensions.bad_builtin.

How can I convert GOPixbuf strings to images

I have an XML file produced with Gnumeric that contains images, stored as GOPixbuf strings inside XML. They look like this:
eXyA/4KEiP9xcnf/f3+E/3l5ff9xb3L/jo2Q/29wdP+ [truncated]
For each string I have width and height, and a rowstride parameter, like in this example:
<GOImage name="Image(70)" type="GOPixbuf" width="151" height="135" rowstride="604">
Is there a reasonable way to convert that to an image - any format will do?
I'm conversant with perl and image conversion tools (imagemagick, gimp) but I have not found any documentation by googling beyond GTK or GOffice docs.
You have already found stuff that is helpful. But since there are no Perl bindings for this on CPAN, you would have to make your own if you want to use Perl.
Fortunately, you don't have to know XS to do that. You can use FFI::Platypus to create temporary bindings and only map what you need.
The docs you have probably already found have a Getting started with GOffice section. After a quick check I found that on my recent Ubuntu there is a package that contains that lib. It is called libgoffice-0.10-dev.
Now you can set that up and play around with the lib functions. Somewhere in https://developer.gnome.org/goffice/unstable/GOImage.html there probably is a method to read and convert it.
One of the good ones might be go-image-get-pixbuf, which returns a GdkPixbuf. That in turn has a very extensive documentation. Maybe what you need might be in this one.
Good luck.

Less beautifier - format code

Is there is code beautifier for less such as http://www.lonniebest.com/formatcss/ for css? I need sort properties in less code by alphabet.
I use CSSComb http://csscomb.com/. This one is a npm module but there are plugins for it. Especially I use it with Sublime Text.
It works with less too although there might me some edge case not (yet) properly handled. But it's good for me.
You can order rules however you want. Just read the docs ;)
You can also use cssbrush. It is based and uses the csscomb under the hood, but include a fix for this bug and also has the ability to remember the files that were previously beautified, so it will only beautify changed files on each run.
Full disclosure, I wrote it.

How to make Compass/Sass compilation command generate parsable output

I’d like to automate the compilation of Compass projects and be able to get output that I can parse so I can take only what I need (the errors) and further format them how I want.
The issue is that Compass output is not in a format that can be easily parsed (it has error messages on multiple lines).
Is there any reliable way to parse this output? Or… any idea what would need to be changed and where in Compass’s code to allow a new param that would allow you to specify the output format (e.g. JSON, XML)?
I’m asking this because I don’t know Ruby, so I would need a starting point. Their current code is not easy to understand (due to the fact that I don’t know Ruby), but if I at least have a starting point I would try to see what I can do and hopefully create a pull request with this if I get it working.
I think, there is another way to solve this problem, what you think about to parse the output css and do not touch compass.
There is a good framework for creating postproccesor for css:
https://github.com/postcss/postcss
You can do what you want with output css and send message to console or send email or other things, and many other things.

Structured debug log

I am writing a complex application (a compiler analysis). To debug it I need to examine the application's execution trace to determine how its values and data structures evolve during its execution. It is quite common for me to generate megabytes of text output for a single run and sifting my way through all that is very labor-intensive. To help me manage these logs I've written my own library that formats them in HTML and makes it easy to color text from different code regions and indent code in called functions. An example of the output is here.
My question is: is there any better solution than my own home-spun library? I need some way to emit debug logs that may include arbitrary text and images and visually structure them and if possible, index them so that I can easily find the region of the output I'm most interested. Is there anything like this out there?
Regardless you didn't mentioned a language applied, I'd like to propose apache Log4XXX family: http://logging.apache.org/
It offers customizable details level as well as tag-driven loggers. GUI tool (chainsaw) can be combined with "old good" GREP approach (so you see only what you're interested in at the moment).
Colorizing, search and filtering using an expression syntax is available in the latest developer snapshot of Chainsaw. The expression syntax also supports regular expressions (using the 'like' keyword).
Chainsaw can parse any regular text log file, not just log files generated by log4j.
The latest developer snapshot of Chainsaw is available here:
http://people.apache.org/~sdeboy
The File, load Chainsaw configuration menu item is where you define the 'format' and location of the log file you want to process, and the expression syntax can be found in the tutorial, available from the help menu.
Feel free to email the log4j users list if you have additional questions.
I created a framework that might help you, https://github.com/pablito900/VisualLogs

Resources