I have a click based python application which dynamically loads relevant module like this.
python cli.py --foo # imports only foo module
python cli.py --bar # imports only bar module
lib directory looks like:
lib
├───src
├───loader.py
├───modules
├───foo
├───bar
in my code src/loader.py, this is how I am finding lib directory and then import correct module based on command line option
if getattr(sys, "frozen", False):
# The application is frozen
datadir = path.dirname(sys.executable)
libdir = path.dirname(datadir)
modules_dir = path.join(libdir , 'modules')
and based on CLI input, I import correct mod_path i.e foo.xxx or bar.xxx in src/loader.py
__import__(modules_dir + mod_path)
While running "python setup.py build", build directory looks like this. The generated exe works as expected.
Directory: C:\Users\username\OneDrive\Desktop\build
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 7/30/2022 1:01 PM bdist.win-amd64
d----- 7/30/2022 4:03 PM exe.win-amd64-3.9
d----- 7/30/2022 4:04 PM lib
but when I generate MSI package and install it,and run python cli.py --foo it doesn't import foo module.
How can I debug what is wrong?
Update: The relative directory structure changes after I install msi. So I check 'AppData' in datadir path and then locate loader.py and finally modules directory. This approach works but is there an elegant way to do it?
# exe datadir : C:\Users\username\my_cli\build\exe.win-amd64-3.9
# exe loader.py C:\Users\username\my_cli\build\lib\src\loader.py
# msi datadir : C:\Users\username\AppData\Local\Programs\my_cli
# msi loader.py is at C:\Users\username\AppData\Local\Programs\my_cli\Lib\site-packages\src\loader.py
Related
I have python script (train_pipeline.py) which creates and runs an azure ml pipeline. It uses environment.yml to create the conda environment. This works, except for one private library (utils) which is referenced in the yml file through it's relative path. When I create the conda environment locally, everything works as expected. However when azure ml creates the environment, it is not able to find the utils folder.
How can I include private packages to azure ml environment?
Directory structure:
| azure_pipelines
|--- train_pipeline.py
| projects
|--- project_1
|---|--- script.py
|---|--- environment.yml
| utils
|--- database_utils
|---|---__init__.py
|---|---connection.py
|--- setup.py
environment.yml content:
name: env_name
channels:
- defaults
dependencies:
- python=3.8
- pyodbc
- pandas
- pip:
- azureml-core
- ../../utils
prefix: /Users/username/opt/anaconda3/envs/env_name
train_pipeline.py:
...
environment = Environment.from_conda_specification('env_name', '../projects/project_1/environment.yml')
...
First you need to create a .whl file for your private package. You already have a "setup.py" file in your "utils" directory, so assuming it's correctly set up (you can take a look here if you're not sure), open the "utils" directory in the terminal and run the following instruction:
python setup.py bdist_wheel --universal
This will create the wheel in a new "dist" directory under "utils". You can now add it to your environment:
whl_url = environment.add_private_pip_wheel(
workspace = <Workspace object>,
file_path = "../utils/dist/<wheel filename>"
)
environment.python.conda_dependencies.add_pip_package(whl_url)
The docs explain this last part fairly well.
I have code in erlang for listing module path for a module:
code:get_object_code(Module)
Do we have anything similar in shell?
I know we can list modules in shell by :
module list
But it specifically does not give the module path for particular module name
I have tried with:
module show module_name
output: ModuleCmd_Display.c(151):ERROR:105: Unable to locate a modulefile for 'module_name'
This seems to work if you give the name as returned by module list. For example, I have two active modules:
$ module list
Currently Loaded Modulefiles:
1) module-info 2) module-git
I can ask for information about, say, module-git, and the output contains the file name of the module, including directory:
$ module show module-git
-------------------------------------------------------------------
/home/magnus/modules/modulefiles/module-git:
module-whatis {get last version of the module sources from GitHub}
set-alias get-modules {git clone git://github.com/cea-hpc/modules.git && cd modules}
-------------------------------------------------------------------
I am trying to make my setup.py install a directory to /usr/share (or in a different prefix, or at least let my script copy it from the EGG file).
The directory structure of my project looks something like this:
- setup.py
- MANIFEST.in
- myproj
- __init__.py
- sompekg
- __init__.py
- data
- dirA
- dirB
- somefile
- somefile
I tried adding 'data' to MANIFEST.in:
recursive-include data *
recursive-include themer *
or in setup.py:
include_package_data=True,
but because it is a nested directory structure and there are no python files there it will not include them. At the moment the "data" directory is included into the EGG, but none of the children directories are.
Alright, I ended up writing my own install replacement that invokes the regular setuptools.commands.install and afterwards does my file copying. Here are the relevant pieces of my setup.py:
from setuptools.command.install import install
class new_install(install):
def run(self):
install.run(self) # invoke original install
self.mkpath('/usr/share/themer')
self.copy_tree('data/default', '/usr/share/themer/default')
self.mkpath('/usr/share/fish/completions')
self.copy_file('data/fish/themer.fish', '/usr/share/fish/completions/')
setup(
#... whatever else you got here
cmdclass=dict(install=new_install)
)
I have troubles creating a document directory (html) using sphinx-build.
I tried
sphinx-build -b html source build
as well as
make html
but in both cases only the html-files search.html, index.html and genindex.html are generated. The file modindex.html is missing.
In the file conf.py I set
html_domain_indices = True
so I should have a modindex.html file. What am I doing wrong? I get no error message after building the html files. I'm using Sphinx 1.1.3 and Python 2.7 on Windows XP.
Short version
run sphinx-apidoc -o . mymodule
uncomment and modify conf.py. For this example, sys.path.insert(0, os.path.abspath('mymodule'))
re-run make html
Long answer
I can reproduce the issue with this sample module:
$cat mymodule/mymodule.py
def fn1():
'''First function'''
pass
def fn2():
'''Second function'''
pass
Running sphinx-quickstart produces the following tree:
$tree
.
├── Makefile
├── _build
├── _static
├── _templates
├── conf.py
├── index.rst
├── mymodule
└── mymodule.py
$cat index.rst
.. sphinx example documentation master file, created by
sphinx-quickstart on Mon Mar 30 15:28:37 2015.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
with default index.rst:
Welcome to sphinx example's documentation!
==========================================
Contents:
.. toctree::
:maxdepth: 2
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
Running make html at this point produces no output in _build/html/py-modindex.html. This is because sphinx needs .rst files describing every module. Fortunately it's easy to produce using sphinx-apidoc -o . mymodule.
This gives two new files, of which only mymodule.rst is necessary to fix the modindex issue in the question.
$head *mod*rst
==> modules.rst <==
mymodule
========
.. toctree::
:maxdepth: 4
mymodule
==> mymodule.rst <==
mymodule module
===============
.. automodule:: mymodule
:members:
:undoc-members:
:show-inheritance:
Running make html at this point still won't work. But uncommenting and changing the line beginning with sys.path.insert in conf.py fixes things.
Mine is: sys.path.insert(0, os.path.abspath('mymodule'))
PS: to avoid an additional warning, add modules to the Contents: toctree in the index.rst file.
The following is what I did for my project.
1. Install sphinx
pip install -U sphinx
2. Install theme (I've chosen sphinx_rtd_theme. Pleaser replace it with your choice)
pip install sphinx sphinx_rtd_theme
3. Create a doc directory under your project file
mkdir docs
4. Get into that directory
cd docs
5. Run sphinx-quickstart command
sphinx-quickstart
6. Run the following ( if you've enabled autodoc shpinx extension)
sphinx-apidoc -o source/ ../<modules_folder>
where source is the source folder used by sphinx and modules_folder is the folder your project's .py files-modules are in.
7. You will be prompted to reply to the questions below (change the answers according to your needs)
> Separate source and build directories (y/n) [n]: y
The project name will occur in several places in the built documentation.
> Project name: project_name
> Author name(s): your_nme
> Project release []: 1.01
> Project language [en]: en
8. It should look like the following if successfully run:
Creating file ...<***modules_folder***>/docs/source/conf.py.
Creating file ...<***modules_folder***>/docs/source/index.rst.
Creating file ...<***modules_folder***>/docs/Makefile.
Creating file ...<***modules_folder***>/docs/make.bat.
Finished: An initial directory structure has been created.
9. Edit conf.py and make sure the following lines are not commented (#):
import os # line 13
import sys # line 14
Note: .. stands for one directory up from doc directory
<modules_folder> is the folder your project's .py files-modules are in
sys.path.insert(0, os.path.abspath('../<modules_folder>/')) # line 16
make sure the following line exists
extensions = ['sphinx.ext.autodoc'] # line 34
change the theme if you'd like
html_theme = 'sphinx_rtd_theme' # line 51
10. Run shpinx-apidoc command
sphinx-apidoc -o . ..
Note:
. >> is for current directory
..>> is for one level up directory, i.e., the <modules_folder> project directory
11. Run make html command
.\make clean
.\make HTML
or
make clean
make html
12. Open your newely built webpage starting with
<modules_folder>/docs/build/html/index.html
Old question, but "working on the hard drive but not on ReadTheDocs" often has an easy fix.
Go to readthedocs
Navigate to your "Project Home"
Click 'Admin' then 'Advanced Settings'
Find and check the box for 'Install Project'
Return to 'Overview'
Build a new version
Check if it worked
Come back here and thank me
I've setup a custom configuration file for Pylint (name, conveniently, config). There has to be a way that I don't have to include --rcfile=config on every run. How can I set the config file permanently?
When you do not specify the --rcfile option, Pylint searches for a configuration file in the following order and uses the first one it finds:
pylintrc in the current working directory
If the current working directory is in a Python module, Pylint
searches up the hierarchy of Python modules until it finds a
pylintrc file. This allows you to specify coding standards on a
module-by-module basis. Of course, a directory is judged to be a
Python module if it contains an __init__.py file.
The file named by environment variable PYLINTRC
.pylintrc in your home directory, unless you have no home directory
or your home directory is /root
.pylintrc in the current working directory
/etc/pylintrc
Thus depending on the method you choose, Pylint can use a different configuration file based on the location of the code, the user or the machine.
Note that the configuration file only applies to Python files that are in modules. Thus, Pylint still uses its default rules when analyzing Python files in a directory with no __init__.py file.
For example, I have a bin/ directory containing command line applications. Ordinarily, this directory needs no __init__.py file because it is never imported. I had to add a bin/__init__.py file to get Pylint to analyze these Python files using my pylintrc file.
set the path to that file in the PYLINTRC environment variable, or rename the file $HOME/.pylintrc or /etc/pylintrc (the latter is probably only supported on *nix)
It can be done using .pre-commit-config.yaml. This snippet below need to be added to .pre-commit-config.yaml:
repos:
- repo: local
hooks:
- id: pylint
name: pylint
entry: pylint
language: system
types: [python]
args: [
"-rn", # Only display messages
"-sn", # Don't display the score
"--rcfile=.pylintrc", # Link to your config file
"--load-plugins=pylint.extensions.docparams", # Load an extension
]