I have installed Ansible using "pip" as a root user on RHEL
python version - Python 2.7.5
Ansible version -
ansible 2.7.4
config file = None
configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python2.7/site-packages/ansible
executable location = /bin/ansible
python version = 2.7.5 (default, May 31 2018, 09:41:32) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
whenever I try to use ansible as a different user getting an error
Error:
$ ansible --version
Traceback (most recent call last):
File "/usr/bin/ansible", line 40, in <module>
from ansible.errors import AnsibleError, AnsibleOptionsError, AnsibleParserError
ImportError: No module named ansible.errors
I find two ansible files located in different places, Is this abnormal?
one at /bin/ansible & other at /usr/bin/ansible
$ cd /bin
$ ls -lart ansible
-rwxr-xr-x 1 root root 5837 Dec 9 13:12 ansible
$ cd ansible
-ksh: cd: ansible: [Not a directory]
$ ls -lart /usr/bin/ansible
-rwxr-xr-x 1 root root 5837 Dec 9 13:12 /usr/bin/ansible
Content of /bin/ansible
$ cat /bin/ansible
#!/bin/python
# (c) 2012, Michael DeHaan <michael.dehaan#gmail.com>
#
# This file is part of Ansible
#
# Ansible is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# Ansible is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with Ansible. If not, see <http://www.gnu.org/licenses/>.
########################################################
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type
__requires__ = ['ansible']
try:
import pkg_resources
except Exception:
# Use pkg_resources to find the correct versions of libraries and set
# sys.path appropriately when there are multiversion installs. But we
# have code that better expresses the errors in the places where the code
# is actually used (the deps are optional for many code paths) so we don't
# want to fail here.
pass
import os
import shutil
import sys
import traceback
from ansible.errors import AnsibleError, AnsibleOptionsError, AnsibleParserError
from ansible.module_utils._text import to_text
# Used for determining if the system is running a new enough python version
# and should only restrict on our documented minimum versions
_PY3_MIN = sys.version_info[:2] >= (3, 5)
_PY2_MIN = (2, 6) <= sys.version_info[:2] < (3,)
_PY_MIN = _PY3_MIN or _PY2_MIN
if not _PY_MIN:
raise SystemExit('ERROR: Ansible requires a minimum of Python2 version 2.6 or Python3 version 3.5. Current version: %s' % ''.join(sys.version.splitlines()))
class LastResort(object):
# OUTPUT OF LAST RESORT
def display(self, msg, log_only=None):
print(msg, file=sys.stderr)
def error(self, msg, wrap_text=None):
print(msg, file=sys.stderr)
if __name__ == '__main__':
display = LastResort()
try: # bad ANSIBLE_CONFIG or config options can force ugly stacktrace
import ansible.constants as C
from ansible.utils.display import Display
except AnsibleOptionsError as e:
display.error(to_text(e), wrap_text=False)
sys.exit(5)
cli = None
me = os.path.basename(sys.argv[0])
try:
display = Display()
display.debug("starting run")
sub = None
target = me.split('-')
if target[-1][0].isdigit():
# Remove any version or python version info as downstreams
# sometimes add that
target = target[:-1]
if len(target) > 1:
sub = target[1]
myclass = "%sCLI" % sub.capitalize()
elif target[0] == 'ansible':
sub = 'adhoc'
myclass = 'AdHocCLI'
else:
raise AnsibleError("Unknown Ansible alias: %s" % me)
try:
mycli = getattr(__import__("ansible.cli.%s" % sub, fromlist=[myclass]), myclass)
except ImportError as e:
# ImportError members have changed in py3
if 'msg' in dir(e):
msg = e.msg
else:
msg = e.message
if msg.endswith(' %s' % sub):
raise AnsibleError("Ansible sub-program not implemented: %s" % me)
else:
raise
try:
args = [to_text(a, errors='surrogate_or_strict') for a in sys.argv]
except UnicodeError:
display.error('Command line args are not in utf-8, unable to continue. Ansible currently only understands utf-8')
display.display(u"The full traceback was:\n\n%s" % to_text(traceback.format_exc()))
exit_code = 6
else:
cli = mycli(args)
cli.parse()
exit_code = cli.run()
except AnsibleOptionsError as e:
cli.parser.print_help()
display.error(to_text(e), wrap_text=False)
exit_code = 5
except AnsibleParserError as e:
display.error(to_text(e), wrap_text=False)
exit_code = 4
# TQM takes care of these, but leaving comment to reserve the exit codes
# except AnsibleHostUnreachable as e:
# display.error(str(e))
# exit_code = 3
# except AnsibleHostFailed as e:
# display.error(str(e))
# exit_code = 2
except AnsibleError as e:
display.error(to_text(e), wrap_text=False)
exit_code = 1
except KeyboardInterrupt:
display.error("User interrupted execution")
exit_code = 99
except Exception as e:
if C.DEFAULT_DEBUG:
# Show raw stacktraces in debug mode, It also allow pdb to
# enter post mortem mode.
raise
have_cli_options = cli is not None and cli.options is not None
display.error("Unexpected Exception, this is probably a bug: %s" % to_text(e), wrap_text=False)
if not have_cli_options or have_cli_options and cli.options.verbosity > 2:
log_only = False
if hasattr(e, 'orig_exc'):
display.vvv('\nexception type: %s' % to_text(type(e.orig_exc)))
why = to_text(e.orig_exc)
if to_text(e) != why:
display.vvv('\noriginal msg: %s' % why)
else:
display.display("to see the full traceback, use -vvv")
log_only = True
display.display(u"the full traceback was:\n\n%s" % to_text(traceback.format_exc()), log_only=log_only)
exit_code = 250
finally:
# Remove ansible tmpdir
shutil.rmtree(C.DEFAULT_LOCAL_TMP, True)
sys.exit(exit_code)
list:
$ ls -lart /usr/lib/python2.7/site-packages/ansible/errors/
total 36
-rw------- 1 root root 3820 Dec 9 13:12 yaml_strings.py
-rw------- 1 root root 11555 Dec 9 13:12 __init__.py
-rw------- 1 root root 3147 Dec 9 13:12 yaml_strings.pyc
-rw------- 1 root root 11535 Dec 9 13:12 __init__.pyc
drwx------ 2 root root 88 Dec 9 13:12 .
drwx------ 17 root root 4096 Dec 9 13:12 ..
can anyone help me in solving it.
In my case I was working in a venv in python and that caused Ansible to not find all it's python modules. Might not necessarily be the OP's issue here but it's worth being aware of.
It's because the installation did not make the files in /usr/lib/python2.7/site-packages/ansible readable by groups or world, meaning that only the file's owner (root in this case) can read them:
$ ls -lart /usr/lib/python2.7/site-packages/ansible/errors/
total 36
-rw------- 1 root root 11555 Dec 9 13:12 __init__.py
# ^^^^^^ should be -rw-r--r-- for files
You can change this one specific problem with chmod but likely in the future you will want to run umask go-w first in order to keep pip from defaulting to 0600 permissions for files that it writes out
# chmod -R a+rX /usr/lib/python2.7/site-packages/ansible
The X in that expression is to set the eXecute bit but only for files that already have execute in the user's permissions (so, for directories and executable files, but not for "normal" files; you can read the full story in man 1 chmod)
In my case the problem turned out to be following:
i was executing the command using sudo, which was picking different .ssh/id_rsa.pub compared to what I had posted public key in the ssh-server. This was rightly resulting into permission denied.
This was discovered only by doing (taken from the dump that comes from ansible ping command using -vvv. Note that I have added -v command to ssh, that gives you the hint on what has failed during the login process.
sudo sshpass -d11 ssh -v -C -o ControlMaster=auto -o ControlPersist=60s -o 'User="ubuntu"' -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/66b206be85 10.115.216.134 '/bin/sh -c '"'"'echo ~ubuntu && sleep 0'"'"''
In my case, I was using a broken venv. Deactivated it, removed it and reinstalled every thing:
deactivate
rm -rf .venv
python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
Related
I struggled a bit to make PyGreSQL to work in AWS Lambda (Python 3.9) to connect to an Aurora PostgreSQL instance. Searching google and stack overflow didn't return any relevant results. Most of the hits were for making psycopg2 to work with AWS Lambda. So leaving the following out here for anybody else having the same issue and trying to figure a solution.
Here is my Lambda code.
import boto3
import cfnresponse
import logging
import os
import sys
# import DB-API 2.0 compliant module for PygreSQL
from pgdb import connect
from botocore.exceptions import ClientError
import json
logger = logging.getLogger()
logger.setLevel(logging.INFO)
DBHOST = os.environ['DBHost']
DBPORT = os.environ['DBPort']
DBNAME = os.environ['DBName']
DBUSER = os.environ['DBUser']
SECRET_ARN = os.environ['Secret_ARN']
REGION_NAME = os.environ['Region_Name']
def handler(event, context):
try:
responseData = {}
try:
DBPASS = get_secret(SECRET_ARN,REGION_NAME)
# Connection to SSL enabled Aurora PG database using RDS root certificate
HOSTPORT=DBHOST + ':' + str(DBPORT)
my_connection = connect(database=DBNAME, host=HOSTPORT, user=DBUSER, password=DBPASS, sslmode='require', sslrootcert = 'rds-combined-ca-bundle.pem')
logger.info("SUCCESS: Connection to RDS PG instance succeeded")
except Exception as e:
logger.error('Exception: ' + str(e))
logger.error("ERROR: Unexpected error: Couldn't connect to Aurora PostgreSQL instance.")
responseData['Data'] = "ERROR: Unexpected error: Couldn't connect to Aurora PostgreSQL instance."
cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, "None")
sys.exit()
if event['RequestType'] == 'Create':
try:
with my_connection.cursor() as cur:
#Execute bootstrap SQLs
cur.execute("create extension if not exists pg_stat_statements")
cur.execute("create extension if not exists pgaudit")
my_connection.commit()
cur.close()
my_connection.close()
responseData['Data'] = "SUCCESS: Executed SQL statements successfully."
cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, "None")
except Exception as e:
logger.error('Exception: ' + str(e))
responseData['Data'] = "ERROR: Exception encountered!"
cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, "None")
else:
responseData['Data'] = "{} is unsupported stack operation for this lambda function.".format(event['RequestType'])
cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, "None")
except Exception as e:
logger.error('Exception: ' + str(e))
responseData['Data'] = str(e)
cfnresponse.send(event, context, cfnresponse.SUCCESS, responseData, "None")
def get_secret(secret_arn,region_name):
# Create a Secrets Manager client
session = boto3.session.Session()
client = session.client(
service_name='secretsmanager',
region_name=region_name
)
try:
get_secret_value_response = client.get_secret_value(
SecretId=secret_arn
)
except ClientError as e:
if e.response['Error']['Code'] == 'DecryptionFailureException':
logger.error("Secrets Manager can't decrypt the protected secret text using the provided KMS key")
elif e.response['Error']['Code'] == 'InternalServiceErrorException':
logger.error("An error occurred on the server side")
elif e.response['Error']['Code'] == 'InvalidParameterException':
logger.error("You provided an invalid value for a parameter")
elif e.response['Error']['Code'] == 'InvalidRequestException':
logger.error("You provided a parameter value that is not valid for the current state of the resource")
elif e.response['Error']['Code'] == 'ResourceNotFoundException':
logger.error("We can't find the resource that you asked for")
else:
# Decrypts secret using the associated KMS CMK.
secret = json.loads(get_secret_value_response['SecretString'])['password']
return secret
I used my Cloud9 Amazon Linux 2 instance to create the lambda zip package. Installed Python 3.9 following https://computingforgeeks.com/how-to-install-python-on-amazon-linux/ and installed PyGreSQL using the following commands:
mkdir pygresql
pip3.9 install --target ./pygresql PyGreSQL
I included the contents of pygresql directory in the lambda package containing the lambda code.
Lambda was showing the following error during my test:
Cannot import shared library for PyGreSQL probably because no libpq.so is installed libldap_r-2.4.so.2: cannot open shared object file: No such file or directory
This is because AWS Lambda is missing the required PostgreSQL libraries in the AMI image. To fix this, I had to do the following:
Install PostgreSQL 14.3 on my cloud9. Its important to run the configure command with with-openssl option if you want to connect to an RDS/Aurora PostgreSQL instance where rds.force_ssl is set to 1.
sudo yum -y group install "Development Tools"
sudo yum -y install readline-devel
sudo yum -y install openssl-devel
mkdir /home/ec2-user/postgresql
cd /home/ec2-user/postgresql
curl https://ftp.postgresql.org/pub/source/v14.3/postgresql-14.3.tar.gz -o postgresql-14.3.tar.gz >> /debug.log
tar -xvf postgresql-14.3.tar.gz
cd postgresql-14.3
sudo ./configure --with-openssl
sudo make -C src/bin install
sudo make -C src/include install
sudo make -C src/interfaces install
sudo make -C doc install
sudo /sbin/ldconfig /usr/local/pgsql/lib
Then I copied the following files from /usr/local/pgsql/lib/ directory and included them in the lib directory of the lambda package containing the lambda code.
-rw-r--r-- 1 ec2-user ec2-user 287982 Aug 2 06:15 libpq.a
-rwxr-xr-x 1 ec2-user ec2-user 332432 Aug 2 06:15 libpq.so
-rwxr-xr-x 1 ec2-user ec2-user 332432 Aug 2 06:15 libpq.so.5
-rwxr-xr-x 1 ec2-user ec2-user 332432 Aug 2 06:16 libpq.so.5.14
Here are the contents of my lambda package:
drwxr-xr-x 1 1049089 0 Aug 1 15:25 PyGreSQL-5.2.4-py3.9.egg-info/
drwxr-xr-x 1 1049089 0 Aug 1 15:25 __pycache__/
-rw-r--r-- 1 1049089 345184 Aug 2 05:16 _pg.cpython-39-x86_64-linux-gnu.so
drwxr-xr-x 1 1049089 0 Aug 1 15:20 certifi/
drwxr-xr-x 1 1049089 0 Aug 1 15:20 certifi-2019.11.28.dist-info/
-rw-r--r-- 1 1049089 1845 Mar 23 2020 cfnresponse.py
drwxr-xr-x 1 1049089 0 Aug 1 15:20 chardet/
drwxr-xr-x 1 1049089 0 Aug 1 15:22 chardet-3.0.4.dist-info/
-rw-r--r-- 1 1049089 4391 Mar 23 2020 dbbootstrap.py
-rw-r--r-- 1 1049089 2094165 Aug 1 23:20 dbbootstrap.zip
drwxr-xr-x 1 1049089 0 Aug 1 15:22 idna/
drwxr-xr-x 1 1049089 0 Aug 1 15:22 idna-2.8.dist-info/
drwxr-xr-x 1 1049089 0 Aug 1 15:23 lib/
-rwxr-xr-x 1 1049089 104780 Mar 26 17:20 pg.py*
-rwxr-xr-x 1 1049089 66051 Mar 26 17:20 pgdb.py*
-rw-r--r-- 1 1049089 65484 Mar 23 2020 rds-combined-ca-bundle.pem
drwxr-xr-x 1 1049089 0 Aug 1 15:23 requests/
drwxr-xr-x 1 1049089 0 Aug 1 15:23 requests-2.22.0.dist-info/
drwxr-xr-x 1 1049089 0 Aug 1 15:23 urllib3/
drwxr-xr-x 1 1049089 0 Aug 1 15:25 urllib3-1.25.8.dist-info/
AWS Lambda was happy after this and was able to connect to the PostgreSQL instance.
Thanks #Arabinda for this, nothing else really worked for me while trying to use psycopg2 and I accidentally found your guide. I was able to import psycopg2 using this way but just using
pip3.9 install --target ./pygresql PyGreSQL psycopg2
instead of
pip3.9 install --target ./pygresql PyGreSQL
After that, I just zipped the contents under python\lib\python3.9\site-packages and uploaded it as a Layer.
We stumbled across the same problem and didn't want to provision a whole new server. So we ended up with a dockerfile building the layer zip for us. In case somebody could use it.
# https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-image-repositories.html
FROM public.ecr.aws/sam/build-python3.9:latest
# https://ftp.postgresql.org/pub/source/
ARG postgresql_version=14.6
RUN yum upgrade -y && \
yum install -y openssl-devel openssl-static
WORKDIR /tmp
ENV PREFIX /tmp/local
RUN \
curl -fsSL https://ftp.postgresql.org/pub/source/v${postgresql_version}/postgresql-${postgresql_version}.tar.bz2 \
-o postgresql-${postgresql_version}.tar.bz2
RUN \
mkdir ${PREFIX} && \
tar jxf postgresql-${postgresql_version}.tar.bz2 && \
cd postgresql-${postgresql_version} && \
./configure \
--prefix=${PREFIX} \
--with-openssl \
--without-readline \
&& \
make install
RUN \
export PATH="$PATH:${PREFIX}/bin/" && \
mkdir /tmp/python && \
mkdir /tmp/lib && \
mkdir /tmp/lib64 && \
pip3 install --target /tmp/python PyGreSQL
RUN \
cd ${PREFIX} && \
cp lib/libpq.so* /tmp/lib && \
cp /lib64/libssl.so.10 /tmp/lib64
RUN \
cd /tmp && \
zip -r /tmp/libs.zip ./python ./lib ./lib64
ENTRYPOINT ["cat", "/tmp/libs.zip"]
Run
docker run myBuild:latest > layer.zip
to extract the zip from the dockerimage.
On Ubuntu 14.04
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Codename: trusty
Building a Yocto Poky image using the fido branch
inherit core-image
IMAGE_FEATURES += "x11-base x11-sato package-management ssh-server-dropbear"
IMAGE_INSTALL += "chromium \
lsb \
kernel-modules \
alsa-utils \
... and I am getting this sort of message
I look like it related to the Chromium recipe /meta-browser/recipes-browser/chromium/chromium_45.0.2454.85.bb which starts as such
include chromium.inc
DESCRIPTION = "Chromium browser"
DEPENDS += "libgnome-keyring"
and I get this message
ERROR: Logfile of failure stored in: /home/joel/yocto/build-fido/tmp/work/cortexa7hf-vfp-vfpv4-neon-poky-linux-gnueabi/chromium/45.0.2454.85-r0/temp/log.do_configure.28622
Log data follows:
| DEBUG: Executing python function sysroot_cleansstate
| DEBUG: Python function sysroot_cleansstate finished
| DEBUG: Executing shell function do_configure
| Updating projects from gyp files...
| Package xkbcommon was not found in the pkg-config search path.
| Perhaps you should add the directory containing `xkbcommon.pc'
| to the PKG_CONFIG_PATH environment variable
| No package 'xkbcommon' found
| gyp: Call to 'pkg-config --cflags xkbcommon' returned exit status 1.
| WARNING: exit code 1 from a shell command.
What I have tried
Installed the library
$ sudo apt-get install libxkbcommon-x11-dev
Search for xkbcommon.pc
$ apt-file search xkbcommon.pc
libxkbcommon-dev: /usr/lib/x86_64-linux-gnu/pkgconfig/xkbcommon.pc
pkg-config
joel#linux-Lenovo-G50-70:~/yocto/build-fido$ pkg-config --cflags xkbcommon
<=== Return is EMPTY (?)
joel#linux-Lenovo-G50-70:~/yocto/build-fido$ pkg-config --libs xkbcommon
-lxkbcommon <=== Looks correct
Added PKG_CONFIG_PATH
$ PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/lib/x86_64-linux-gnu/pkgconfig/
$ export PKG_CONFIG_PATH
$ env | grep PKG
PKG_CONFIG_PATH=:/usr/lib/x86_64-linux-gnu/pkgconfig/
but I am still getting the same message when running bitbake
Any suggestions?
Find xkbcommon
$ find /usr/lib/ -name *xkbcommon*
/usr/lib/x86_64-linux-gnu/libxkbcommon.so
/usr/lib/x86_64-linux-gnu/libxkbcommon.so.0.0.0
/usr/lib/x86_64-linux-gnu/libxkbcommon-x11.so.0.0.0
/usr/lib/x86_64-linux-gnu/libxkbcommon-x11.a
/usr/lib/x86_64-linux-gnu/libxkbcommon.a
/usr/lib/x86_64-linux-gnu/libxkbcommon-x11.so.0
/usr/lib/x86_64-linux-gnu/libxkbcommon-x11.so
/usr/lib/x86_64-linux-gnu/pkgconfig/xkbcommon.pc
/usr/lib/x86_64-linux-gnu/pkgconfig/xkbcommon-x11.pc
/usr/lib/x86_64-linux-gnu/libxkbcommon.so.0
In this case, it was the chromium recipe that failed to find libxkbcommon. As the error occurred when building a recipe for the target system, we need to tell the build system that the chromium recipe has a dependency on libxkbcommmon.
This can be done by adding
DEPENDS += "libxkbcommon"
to the chromium recipe.
It's worth noting, that libxkbcommon quite likely is an optional dependency, and in that case, it should be handled by a suitable PACKAGECONFIG. (See PACKAGECONFIG in ref.manual).
Note: I've never built chromium myself, thus I'd prefer to not suggest any suitable PACKAGECONFIG.
I think the Chromium_45 recipe is taken down since the last time I saw it (don't see it anymore).
Anyway, this is what I did to Chromium_40.
I have disabled Wayland (ozone-wayland in Chromium) so that it will only use x11.
In local.conf, I added
CHROMIUM_ENABLE_WAYLAND = "0"
By doing this, I will bypass CHROMIUM_WAYLAND_DEPENDS = "wayland libxkbcommon"
CHROMIUM_X11_DEPENDS = "xextproto gtk+ libxi libxss"
CHROMIUM_X11_GYP_DEFINES = ""
CHROMIUM_WAYLAND_DEPENDS = "wayland libxkbcommon"
CHROMIUM_WAYLAND_GYP_DEFINES = "use_ash=1 use_aura=1 chromeos=0 use_ozone=1"
python() {
if d.getVar('CHROMIUM_ENABLE_WAYLAND', True) == '1':
d.appendVar('DEPENDS', ' %s ' % d.getVar('CHROMIUM_WAYLAND_DEPENDS', True))
d.appendVar('GYP_DEFINES', ' %s ' % d.getVar('CHROMIUM_WAYLAND_GYP_DEFINES', True))
else:
d.appendVar('DEPENDS', ' %s ' % d.getVar('CHROMIUM_X11_DEPENDS', True))
d.appendVar('GYP_DEFINES', ' %s ' % d.getVar('CHROMIUM_X11_GYP_DEFINES', True))
}
P.S.: One more thing I found weird is use-egl.
PACKAGECONFIG[use-egl] = ",,virtual/egl virtual/libgles2" is overrided with PACKAGECONFIG[use-egl] = "" so I have removed PACKAGECONFIG[use-egl] = "" from chromium.inc
PACKAGECONFIG ??= "use-egl"
# this makes sure the dependencies for the EGL mode are present; otherwise, the configure scripts
# automatically and silently fall back to GLX
PACKAGECONFIG[use-egl] = ",,virtual/egl virtual/libgles2"
# Additional PACKAGECONFIG options - listed here to avoid warnings
PACKAGECONFIG[component-build] = ""
PACKAGECONFIG[disable-api-keys-info-bar] = ""
PACKAGECONFIG[ignore-lost-context] = ""
PACKAGECONFIG[impl-side-painting] = ""
PACKAGECONFIG[use-egl] = ""
PACKAGECONFIG[kiosk-mode] = ""
I want to run "pylint" tool on multiple python files present under one directory.
I want one consolidated report for all the Python files.
I am able to run individually one python file, but want to run on bunch of files.
Please help with the command for the same.
I'm not a windows user, but isn't "pylint directory/*.py" enough ?
If the directory is a package (in the PYTHONPATH), you may also run "pylint directory"
Someone wrote a wrapper in python 2 to handle this
The code :
#! /usr/bin/env python
'''
Module that runs pylint on all python scripts found in a directory tree..
'''
import os
import re
import sys
total = 0.0
count = 0
def check(module):
'''
apply pylint to the file specified if it is a *.py file
'''
global total, count
if module[-3:] == ".py":
print "CHECKING ", module
pout = os.popen('pylint %s'% module, 'r')
for line in pout:
if re.match("E....:.", line):
print line
if "Your code has been rated at" in line:
print line
score = re.findall("\d+.\d\d", line)[0]
total += float(score)
count += 1
if __name__ == "__main__":
try:
print sys.argv
BASE_DIRECTORY = sys.argv[1]
except IndexError:
print "no directory specified, defaulting to current working directory"
BASE_DIRECTORY = os.getcwd()
print "looking for *.py scripts in subdirectories of ", BASE_DIRECTORY
for root, dirs, files in os.walk(BASE_DIRECTORY):
for name in files:
filepath = os.path.join(root, name)
check(filepath)
print "==" * 50
print "%d modules found"% count
print "AVERAGE SCORE = %.02f"% (total / count)
I'm encountering an issue with a ruby script called from CLI (not in a framework) that results in a ENOENT error (File does not exist) when attempting to open it, but only when the filename has been constructed using gsub regular expression replacement. It works when the string is constructed by any other method. I'm sure it's something simple that I've done wrong, but I've had a heck of a time figuring it out.
System Information
# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04.3 LTS
Release: 12.04
Codename: precise
# rvm -v
rvm 1.23.13 (stable) by Wayne E. Seguin <wayneeseguin#gmail.com>, Michal Papis <mpapis#gmail.com> [https://rvm.io/]
# ruby -v
ruby 2.0.0p195 (2013-05-14 revision 40734) [x86_64-linux]
Filesystem: ext2
Permissions:
# ls -lah /home/boise/config
total 24K
drwxr-xr-x 2 boise boise 4.0K Nov 13 19:40 .
drwxr-xr-x 11 boise boise 4.0K Nov 13 20:22 ..
-rwxr-xr-x 1 boise boise 248 Nov 13 09:31 boise.yaml
-rwxr-xr-x 1 boise boise 335 Nov 13 19:35 config.rb
-rwxr-xr-x 1 boise boise 339 Nov 13 20:06 stanford.yaml
-rwxr-xr-x 1 boise boise 248 Nov 13 09:49 test.yaml
user being used to run scripts and owns containing directory is 'boise'
there is nothing abnormal about the file system here, no symbolic links or
mappings or anything like that.
I started a simple script to reproduce the error. In this I boiled it down as simple as I could, made the prefix of the path static and used absolute paths so that there is no question about relative paths or current working directories.
scriptname: testing
#!/usr/bin/env ruby
require 'yaml'
# Here we are "hardcoding" the values.
$config_yaml = "boise.yaml"
puts $config_yaml
config_filespec = "/home/boise/config/#{$config_yaml}"
puts config_filespec
puts File.exists? config_filespec
$configuration = YAML.load(File.open(config_filespec))
$verbose = true
hosts = []
args = []
puts $config_yaml
ARGV.each do |arg|
if arg.match /hosts\=(.*?)/i
hosts = ARGV.delete(arg).gsub(/hosts\=(.*?)/i,'\2').split(',')
elsif arg == "quiet"
$verbose = false
ARGV.delete(arg)
elsif arg.match /cfg=(.*?)/
#Here the value of the YAML file to be loaded is set with a regular expression substitution.
$config_yaml = ARGV.delete(arg).gsub(/cfg=(.*?)/, "\2")
else
end
end
#do the same output as we did the first time hardcoded, but this time the
#value got reset by a gsub -- same variable, same name, etc
puts $config_yaml
config_filespec = "/home/boise/config/#{$config_yaml}"
puts config_filespec
puts File.exists? config_filespec
$configuration = YAML.load(File.open(config_filespec))
This works as one would expect:
# ./testing
boise.yaml
/home/boise/config/boise.yaml
true
boise.yaml
boise.yaml
/home/boise/config/boise.yaml
true
But watch what happens when I'm passing in the yaml string as a command line parameter. Note that this is even the exact same filename!
# ./testing cfg=boise.yaml
boise.yaml
/home/boise/config/boise.yaml
true
boise.yaml
boise.yaml
/home/boise/config/boise.yaml
false
./testing:31:in `initialize': No such file or directory - /home/boise/config/boise.yaml (Errno::ENOENT)
from ./testing:31:in `open'
from ./testing:31:in `<main>'
I also tried forcefully casting the string with String(config_filespec), config_filespec.to_s, and using + "" to do a string concatenation, in hopes that it was some type issue with string versus literal or regex or something like that. Any ideas?
Thanks in advance Stack Overflow!
The answer is that you're not getting the results that you think you're getting from the gsub. This boils down to a ruby quirk. When using double-quotes, "\2" evaluates to "002". However, using single quotes '\2' will return the second capture group. In your code...
$config_yaml = ARGV.delete(arg).gsub(/cfg=(.*?)/, '\2')
If you're only going to accept a single file name, would it be worth using split?
$config_yaml = ARGV.delete(arg).split(/=/, 2).last
On the advice I've read here and elsewhere, I decided to stop running sudo cpan, thus affecting the system perl, and instead use perlbrew to install and manage private versions of perl.
I installed perlbrew and ran perlbrew install perl-5.14.2, both without and with --force. Both times it failed to install. Below is the test summary report.
Test Summary Report
-------------------
../ext/POSIX/t/posix.t (Wstat: 0 Tests: 66 Failed: 0)
TODO passed: 11
../lib/locale.t (Wstat: 0 Tests: 117 Failed: 1)
Failed test: 99
Files=2089, Tests=455813, 723 wallclock secs (55.63 usr 8.56 sys + 402.42 cusr 50.30 csys = 516.91 CPU)
Result: FAIL
Has anyone had success using perlbrew to install Perl on OSX 10.8, who can suggest what's going wrong?
[UPDATE]
I tried installing perl 5.12.4, and got the same results. Below are some details that seem to pertain to locale:
The following locales
#
# C C POSIX POSIX af_ZA af_ZA.ISO8859-1 af_ZA.ISO8859-15
# af_ZA.UTF-8 am_ET am_ET.UTF-8 be_BY be_BY.CP1131 be_BY.CP1251
# be_BY.ISO8859-5 be_BY.UTF-8 bg_BG bg_BG.CP1251 bg_BG.UTF-8
# ca_ES ca_ES.ISO8859-1 ca_ES.ISO8859-15 ca_ES.UTF-8 cs_CZ
# cs_CZ.ISO8859-2 cs_CZ.UTF-8 da_DK da_DK.ISO8859-1 da_DK.ISO8859-15
# da_DK.UTF-8 de_AT de_AT.ISO8859-1 de_AT.ISO8859-15
# de_AT.UTF-8 de_CH de_CH.ISO8859-1 de_CH.ISO8859-15
# de_CH.UTF-8 de_DE de_DE.ISO8859-1 de_DE.ISO8859-15
# de_DE.UTF-8 el_GR el_GR.ISO8859-7 el_GR.UTF-8 en_AU
# en_AU.ISO8859-1 en_AU.ISO8859-15 en_AU.US-ASCII en_AU.UTF-8
# en_CA en_CA.ISO8859-1 en_CA.ISO8859-15 en_CA.US-ASCII
# en_CA.UTF-8 en_GB en_GB.ISO8859-1 en_GB.ISO8859-15
# en_GB.US-ASCII en_GB.UTF-8 en_IE en_IE.UTF-8 en_NZ
# en_NZ.ISO8859-1 en_NZ.ISO8859-15 en_NZ.US-ASCII en_NZ.UTF-8
# en_US en_US.ISO8859-1 en_US.ISO8859-15 en_US.US-ASCII
# en_US.UTF-8 es_ES es_ES.ISO8859-1 es_ES.ISO8859-15
# es_ES.UTF-8 et_EE et_EE.ISO8859-15 et_EE.UTF-8 eu_ES
# eu_ES.ISO8859-1 eu_ES.ISO8859-15 eu_ES.UTF-8 fi_FI
# fi_FI.ISO8859-1 fi_FI.ISO8859-15 fi_FI.UTF-8 fr_BE
# fr_BE.ISO8859-1 fr_BE.ISO8859-15 fr_BE.UTF-8 fr_CA
# fr_CA.ISO8859-1 fr_CA.ISO8859-15 fr_CA.UTF-8 fr_CH
# fr_CH.ISO8859-1 fr_CH.ISO8859-15 fr_CH.UTF-8 fr_FR
# fr_FR.ISO8859-1 fr_FR.ISO8859-15 fr_FR.UTF-8 he_IL
# he_IL.UTF-8 hi_IN.ISCII-DEV hr_HR hr_HR.ISO8859-2 hr_HR.UTF-8
# hu_HU hu_HU.ISO8859-2 hu_HU.UTF-8 hy_AM hy_AM.ARMSCII-8
# hy_AM.UTF-8 is_IS is_IS.ISO8859-1 is_IS.ISO8859-15
# is_IS.UTF-8 it_CH it_CH.ISO8859-1 it_CH.ISO8859-15
# it_CH.UTF-8 it_IT it_IT.ISO8859-1 it_IT.ISO8859-15
# it_IT.UTF-8 ja_JP ja_JP.SJIS ja_JP.UTF-8 ja_JP.eucJP kk_KZ
# kk_KZ.PT154 kk_KZ.UTF-8 ko_KR ko_KR.CP949 ko_KR.UTF-8
# ko_KR.eucKR lt_LT lt_LT.ISO8859-13 lt_LT.ISO8859-4
# lt_LT.UTF-8 nl_BE nl_BE.ISO8859-1 nl_BE.ISO8859-15
# nl_BE.UTF-8 nl_NL nl_NL.ISO8859-1 nl_NL.ISO8859-15
# nl_NL.UTF-8 no_NO no_NO.ISO8859-1 no_NO.ISO8859-15
# no_NO.UTF-8 pl_PL pl_PL.ISO8859-2 pl_PL.UTF-8 pt_BR
# pt_BR.ISO8859-1 pt_BR.UTF-8 pt_PT pt_PT.ISO8859-1 pt_PT.ISO8859-15
# pt_PT.UTF-8 ro_RO ro_RO.ISO8859-2 ro_RO.UTF-8 ru_RU
# ru_RU.CP1251 ru_RU.CP866 ru_RU.ISO8859-5 ru_RU.KOI8-R
# ru_RU.UTF-8 sk_SK sk_SK.ISO8859-2 sk_SK.UTF-8 sl_SI
# sl_SI.ISO8859-2 sl_SI.UTF-8 sr_YU sr_YU.ISO8859-2 sr_YU.ISO8859-5
# sr_YU.UTF-8 sv_SE sv_SE.ISO8859-1 sv_SE.ISO8859-15
# sv_SE.UTF-8 tr_TR tr_TR.ISO8859-9 tr_TR.UTF-8 uk_UA
# uk_UA.ISO8859-5 uk_UA.KOI8-U uk_UA.UTF-8 zh_CN zh_CN.GB18030
# zh_CN.GB2312 zh_CN.GBK zh_CN.UTF-8 zh_CN.eucCN zh_HK
# zh_HK.Big5HKSCS zh_HK.UTF-8 zh_TW zh_TW.Big5 zh_TW.UTF-8
#
# tested okay.
#
# None of your locales were broken.
../lib/locale.t ...................................................
Failed 1/117 subtests
So none were "broken", but one "failed". And that, it seems, prevents the installation from succeeding, even by force.
The fault comes from test 99 of the "be_BY.CP1131" locale.
The ultimate fix needs to come from either the Perl developers or Apple.
In the mean time, you can get the install to work by moving /usr/share/locale/be_BY.CP1131 to some other directory. Perl will no longer test this locale. This isn't an ideal solution, but it will get you perlbrew perl assuming you don't need this specific locale.