Is there a way to debug a code in parts? Meaning, I wish to debug the code until a certain point, save the variables state and continue to debug from that point on.
Thanks
With its debug function, Pycharm offers a fantastic opportunity to see the properties of certain variables if the breakpoint has been set accordingly.
Apart from that, Python itself offers an amazing way to serialize and de-serialize object structures with its built-in feature pickle (Pickle documentation).
The pickle.dump(VARIABLE) command can be used to dump variables at a certain state into a file or to be printed.
Sometimes I'm using pickle f.e. to dump a response variable into a file for later being used.
Example Code
import pickle
import requests
r = requests.get('https://www.strava.com/api/v3/athlete')
#
# with open('assets/requests_test.pickle', 'wb') as write:
# pickle.dump(r.json(), write)
With that you're able to open this file manually or load it with pickle.load (VARIABLE) later in your code to do something useful with it.
Related
I'm just starting to learn how to use structlog and I'm having a difficult time trying to figure out how to turn off colored logging when it writes to files. Essentially what I did was take my old code that I had used with the standard logging module and converted it to work with structlog - this is what I came up with:
formatter = logging.Formatter(
fmt=LOGGER_OUTPUT_FORMAT,
datefmt=LOGGER_DATETIME_FORMAT,
)
handler = logging.StreamHandler()
handler.setLevel(level)
handler.setFormatter(formatter)
logger.addHandler(handler)
# create a file handler
formatter = logging.Formatter(
fmt=LOGGER_OUTPUT_FORMAT,
datefmt=LOGGER_DATETIME_FORMAT,
)
handler = RotatingFileHandler(
filename=LOGFILE_PATH,
maxBytes=4000000,
backupCount=20,
)
handler.setLevel(level)
handler.setFormatter(formatter)
logger.addHandler(handler)
structlog.configure(logger_factory=LoggerFactory())
logger = structlog.getLogger('output_logger')
What I can't figure out how to do is where to insert lines to change the formatter such that it doesn't use colored output when saving the logging output to a file. I'm guessing I can uninstall colorama, but that seems like it would be defeating the purpose of actually learning how to use structlog. I've dug through the structlog manual but it's just not making sense to me, and everything I seem to try throws errors, so any guidance would be much appreciated!
Depending on what you want to achieve exactly, you'll have to add more configuration to structlog.configure.
If you want stdlib logging and structlog to cooperate, check out https://www.structlog.org/en/stable/standard-library.html
If you just want to use structlog's native loggers without colors, you only have to adapt your processors for now.
As the Getting Started tutorial says, the default is
[
structlog.processors.StackInfoRenderer(),
structlog.dev.set_exc_info,
structlog.processors.format_exc_info,
structlog.processors.TimeStamper(),
structlog.dev.ConsoleRenderer()
]
Which is colorful if colorama is present. The simplest change would be changing it to
[
structlog.processors.StackInfoRenderer(),
structlog.dev.set_exc_info,
structlog.processors.format_exc_info,
structlog.processors.TimeStamper(),
structlog.dev.ConsoleRenderer(colors=False)
]
although production code should probably use structlog.processors.KeyValueRenderer or structlog.processors.JSONRenderer since ConsoleRenderer is optimized for human consumption in a…console. :)
N.B. That the default for colors is in the API docs is currently wrong, because it's actually based on the presence of colorama and colorama is not present when building the docs.
I'm looking for something like
[BASIC]
good-names=X,
y
as in pylintrc, but I'd like to limit these names to be good only within a single python file.
I thought about message control like #pylint: disable=invalid-names on top of the file, but that is too broad. Ideally, I'd like to only allow these two invalid names X and y to be considered good within a single file. Is that possible with pylint?
Only way I have been able to achieve this effect has been to disable and then immediately enable again immediately afterwards. It's not what you wanted but at least it doesn't ruin the whole file, and a comment of # pylint: enable=xxx is easy to find when you want to go cleaning up later on (like if they add good-names to in-file message control)
I have a standalone ruby script that is intended to be run from a commandline.
Currently I have a section to define stuff like paths to files and such.
key_path = "C:\\OpenSSL-Win64\\keys\\"
consumer_file = "henrb028.consumer"
key_file = "henrb028.key"
I'd love to pull all of that out of the .rb file and into some external config file. Ideally it would be as lightweight as possible. For example, just copy-paste that into another ruby file and then cram it in at runtime.
I've tried both require and include and gotten varying errors. Without resorting to something like YAML, what's an easy and robust way to do this?
There are a few standard ways to realize what you describe.
Read from environmental variables
Each of those environmental stores a particular parameter. Then users can control, if they want to change some or all of the parameters.
Pros: Standard and handy way for both developers and users, providing the parameters are seldom changed. The default parameters can be easily shared in the same environment (like the same account or even platform-wide), as well as with other programs.
Cons: Somewhat obscured. For example, if a user's work with the tool is logged in a file, other people (or even the user her/himself later) cannot tell what the exact set of the parameters used was from the logfile. Or, if a user wants to change a parameter frequently, it is awkward.
An example:
key_file = ENV['KEY_FILE'] || "henrb028.key"
Read from a configuration file
Then either or both of each user and the system administrator can control and change it, when needed. How to determine the filename of the configuration file varies.
Pros: Suitable to store a large set of parameters.
Cons: Somewhat obscured. For example, if a user's work with the tool is logged in a file, other people (or even the user her/himself later) cannot tell what the exact set of the parameters used was. If a user wants to change a parameter frequently, it is very awkward.
A (crude!) example:
Suppose /etc/OUR_CONFIG.txt is as follows,
CONSUMER_FILE: henrb028.consumer
KEY_FILE: henrb028.key
Then, read them like,
opts = { 'KEY_FILE' => 'default.key' }
IO.readlines("/etc/OUR_CONFIG.txt").each do |ec|
a = ec.strip.split(/\s*:\s*/)
opts[a[0]] = a[1]
end
p opts
Specify with command-line options
If some (or all) options are not specified at the run time, it should fall back to the default value. OptionParser (as #tadaman suggested in his/her comment) is a powerful and versatile library to handle command-line options.
Pros: Standard and handy way for users, especially for a quick change at the run time. If you see the log file, you know what exactly the user did.
Cons: The default parameters cannot be shared with other tools on its own. To circumvent the problem, there are a few ways to make a fall-back routine:
Simply hard-code the default value in the main code.
Store the default value in an environmental variable.
Store the default value in a configuration file. The developer must decide how to determine the filename of the configuration file (hard-coding, environmental variable, etc).
A crude example (without using OptionParser):
opts = { 'key_file' => 'default.key' }
%w(consumer_file key_file).each do |ec|
cands = ARGV.grep(/^--#{Regexp.quote ec}=.+/)
opts[ec] = cands[0].split('=')[1] unless cands.empty?
end
p opts
I think these are the most typical ways, though there are other ways!
At the moment, my JMeter test uses a CSV Data Set Config to iterate through a limited set of input data for each HTTP request that I do.
But I don't want to deal with the hassle of an external file (uploading it to my test runner, etc.) - I'd like to just embed the data into the jmx file itself.
I was hoping for something like a "test data" node, that would work similarly to a CSV data set (with Recycle on EOF especially) and I'd just copy/paste the data into the test plan instead of working with an external file.
I'm thinking I might be able to work around it with a JSR223 preprocessor - but is there a better built-in way?
Edit: As per comment: the data cannot be generated.
If you want to do this via JSR223 Test Elements and Groovy language correct syntax would be
vars.put("messageId", "wibble");
vars is a shorthand for JMeterVariables class instance, see the JavaDoc for available functions and properties.
Easier way would be going for User Defined Variables or User Parameters or even better Set Variables Action
You can create a text contains keys and values separated with tab, copy all text
Notice if you have property file you can replace = with tab
Add to JMeter GUI User Defined Variables and click Add from Clipboard
It'll load all your variables to JMeter without "do that by hand using JMeter's GUI"
.
This is my first go at a script based approach using a JSR223 preprocessor node:
// This is where the data is embedded. Up to a couple of hundred entries
// is probably fine, more than that will likely be a bad idea.
def messageIdList = ["graffle", "wibble", "wobble", "flobble", "gibble", ...]
def messageIndex = (vars.getIteration() -1) % (messageIdList.size() -1)
println "iteration ${vars.iteration}, size ${messageIdList.size()}, index: ${messageIndex}"
vars.put("messageId", messageIdList[messageIndex]);
messageIndex++
This appears to do what I want, even when run in a Thread Group with multiple threads.
I'm not sure exactly what the vars.getIteration() represents, and I'm not clear about the precise lifetime / scope of the variables. But it'll do for now.
Any better answers will cheerfully accepted, marked and upvoted.
Problem
I want to understand source code of Kur (deep learning library)
I have no proper training in programming, I prefer to learn by experimenting without prior theoretical knowledge
I want an easy tool to help me dig into the detail workings of source code
debug tools are like pdb library seem to be a good choice to try
But what is the easiest way to get started in experimenting source with pdb?
I just want to write one line of code to dig into details, rather than writing a few lines as demonstrated in many examples when you google pdb
In other words, which function of pdb should I use? and How to use it effectively for experimenting source?
Toy Example
I want to explore the inner workings of kur dump mnist.yml
To make it simple, I want to just explore not beyond __main__.py and kurfile.py.
To be more specific, I want to explore dump() and parse_kurfile() in __main__.py, and Kurfile.__init__() in kurfile.py
Their relationship is as follows:
console: kur dump mnist.yml -->
python: __main__.py : main() --> dump() --> parse_kurfile() -->
python: kurfile.py : Kurfile class --> __init__() ...
python: ... the rest is not to be explored
Which function of pdb should I use to explore the execution flow from dump() to parse_kurfile() and to Kurfile.__init__() and back to dump() again?
Update
How to effectively explore Jupyter notebook using pdb?
pdb inside Jupyter can't even remember console history, not good
One possible solution
use pdb.set_trace only
set_trace is trace the details on the level of the current code block, it will not go deeper to the next inner function.
for example, when I put a single pdb.set_trace inside dump(), pdb will not help me trace into the function of parse_kurfile(), but stay on the current dump() block:
def dump(args):
""" Dumps the Kurfile to stdout as a JSON blob.
"""
pdb.set_trace()
### parse kurfile.yml into parts to be used in python code
spec = parse_kurfile(args.kurfile, args.engine)
If I want to go deeper into parse_kurfile in __main__.py and Kurfile.__init__ in kurfile.py, then I just need to put one pdb.set_trace in each of the two functions, like below:
Update
From my experience so far, there are two libraries inspect and pprint go well with pdb library.
Inside library inspect, I use the following functions the most:
inspect.getdoc: to see the doc of the function
inspect.getmodule: to find out where this function or object come from
inspect.getfullargspec: to find out all the inputs the func takes
inpsect.getsourceliens: to get the source code of the function
with these functions above, when I want to checkout other functions, I don't have to go to find the source code in editor, I can see them right where I am in the pdb.
From library pprint, you can guess, I use pprint.pprint to print out the source code, the doc in a more readable format right inside pdb.
More Update
A working station to explore and experiment source:
using atom to split window and see different source files at the same time;
use iterm2 to split window and use ipython to execute python or bash code
organise them in the following way:
More update
During exploring, I want to have all the attributes and methods of a module or class ready at hand.
To achieve it, I can use inspect.getmembers(module or class name) and use iterm2 split window to view it:
Update: How to change color of iterm2 for the eyes?
Go to iterm2 preferences, color, change to Tango Dark, to gray the foreground color to make white texts look soft
Change Kur logger color setting to:
## logcolor.py
# Color codes for each log-level.
COLORS = {
'DEBUG': BLUE,
'INFO': MAGENTA,
'WARNING': RED,
'ERROR': RED,
'CRITICAL': GREEN
}
How to effectively use pdb in Jupyter notebook?
One way to avoid the drawbacks of pdb in Jupyter:
download the notebook into py file
insert codes like import pdb and pdb.set_trace() into the python code
in console, run python your.py
Now you can explore this py file as you do in above answer