I am new to Airflow. I installed the airflow in Python 3.6.8 using the command:
pip install \
apache-airflow==1.10.12 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt"
The setup was complete with no error. However when I run the 'airflow' command, it shows the error like:
File "/Users/a13220361/airflow_final/bin/airflow", line 4, in <module>
__import__('pkg_resources').run_script('apache-airflow==1.10.12', 'airflow')
File "/Users/a13220361/airflow_final/lib/python3.6/site-packages/pkg_resources/__init__.py", line 664, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/Users/a13220361/airflow_final/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1435, in run_script
.format(**locals()),
pkg_resources.ResolutionError: Script 'scripts/airflow' not found in metadata at '/Users/a13220361/airflow_final/lib/python3.6/site-packages/apache_airflow-1.10.12.dist-info'
I cannot understand what this error is. Please help.
Related
I am aware that very similar questions have been asked previously but I have found these have tended to include Docker, which I am not using at this time, nor do I have installed. I am led to believe that WSL2 should be an alternative to Docker regarding running Airflow on Windows.
I am using WSL2 on my Windows 11 laptop and have installed Apache-Airflow from a tutorial from the following link:
https://coding-stream-of-consciousness.com/2018/11/06/apache-airflow-windows-10-install-ubuntu/
On WSL2 I have:
Python version 3.8.10
Pip version 20.0.2
Apache-Airflow 2.4.1 (I believe)
I have run the following commands (as per the tutorial) with no issue:
sudo apt-get install software-properties-common
sudo apt-add-repository universe
sudo apt-get update
sudo apt-get install python-pip
sudo pip install apache-airflow #I had path issues without 'sudo' command
But when I attempt to use the 'airflow' command in the WSL2 Ubuntu terminal, I am greeted with the following error:
$ airflow
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
File "/usr/lib/python3.8/logging/config.py", line 563, in configure
handler = self.configure_handler(handlers[name])
File "/usr/lib/python3.8/logging/config.py", line 744, in configure_handler
result = factory(**kwargs)
File "/usr/local/lib/python3.8/dist-packages/airflow/utils/log/file_processor_handler.py", line 45, in __init__
self.filename_template, self.filename_jinja_template = parse_template_string(filename_template)
File "/usr/local/lib/python3.8/dist-packages/airflow/utils/helpers.py", line 165, in parse_template_string
import jinja2
File "/usr/lib/python3/dist-packages/jinja2/__init__.py", line 33, in <module>
from jinja2.environment import Environment, Template
File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 15, in <module>
from jinja2 import nodes
File "/usr/lib/python3/dist-packages/jinja2/nodes.py", line 23, in <module>
from jinja2.utils import Markup
File "/usr/lib/python3/dist-packages/jinja2/utils.py", line 656, in <module>
from markupsafe import Markup, escape, soft_unicode
ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/usr/local/lib/python3.8/dist-packages/markupsafe/__init__.py)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 5, in <module>
from airflow.__main__ import main
File "/usr/local/lib/python3.8/dist-packages/airflow/__init__.py", line 46, in <module>
settings.initialize()
File "/usr/local/lib/python3.8/dist-packages/airflow/settings.py", line 564, in initialize
LOGGING_CLASS_PATH = configure_logging()
File "/usr/local/lib/python3.8/dist-packages/airflow/logging_config.py", line 74, in configure_logging
raise e
File "/usr/local/lib/python3.8/dist-packages/airflow/logging_config.py", line 69, in configure_logging
dictConfig(logging_config)
File "/usr/lib/python3.8/logging/config.py", line 808, in dictConfig
dictConfigClass(config).configure()
File "/usr/lib/python3.8/logging/config.py", line 570, in configure
raise ValueError('Unable to configure handler '
ValueError: Unable to configure handler 'processor'
I have googled the ValueError extensively and can't find any clear solutions that don't involve Docker.
Any insight into the error would be much appreciated!
You should install airflow using constraints.
https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#installation-tools
This is the only way installation of airflow is guaranteed to work.
Error output below:
ronakvora:dtc ronakvora$ pip install apache-airflow
Installing build dependencies ... done
Complete output from command python setup.py egg_info:
running egg_info
creating pip-egg-info/pendulum.egg-info
writing requirements to pip-egg-info/pendulum.egg-info/requires.txt
writing pip-egg-info/pendulum.egg-info/PKG-INFO
writing top-level names to pip-egg-info/pendulum.egg-info/top_level.txt
writing dependency_links to pip-egg-info/pendulum.egg-info/dependency_links.txt
writing manifest file 'pip-egg-info/pendulum.egg-info/SOURCES.txt'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/6x/xsb52c7936l38mmb9f7s268m0000gn/T/pip-install-WFGcOd/pendulum/setup.py", line 50, in <module>
setup(**setup_kwargs)
File "/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/private/var/folders/6x/xsb52c7936l38mmb9f7s268m0000gn/T/pip-build-env-HZt1xp/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 296, in run
self.find_sources()
File "/private/var/folders/6x/xsb52c7936l38mmb9f7s268m0000gn/T/pip-build-env-HZt1xp/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 303, in find_sources
mm.run()
File "/private/var/folders/6x/xsb52c7936l38mmb9f7s268m0000gn/T/pip-build-env-HZt1xp/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 534, in run
self.add_defaults()
File "/private/var/folders/6x/xsb52c7936l38mmb9f7s268m0000gn/T/pip-build-env-HZt1xp/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 570, in add_defaults
sdist.add_defaults(self)
File "/private/var/folders/6x/xsb52c7936l38mmb9f7s268m0000gn/T/pip-build-env-HZt1xp/lib/python2.7/site-packages/setuptools/command/py36compat.py", line 36, in add_defaults
self._add_defaults_ext()
File "/private/var/folders/6x/xsb52c7936l38mmb9f7s268m0000gn/T/pip-build-env-HZt1xp/lib/python2.7/site-packages/setuptools/command/py36compat.py", line 119, in _add_defaults_ext
build_ext = self.get_finalized_command('build_ext')
File "/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command
cmd_obj.ensure_finalized()
File "/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
self.finalize_options()
File "/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_ext.py", line 159, in finalize_options
self.include_dirs.append(py_include)
AttributeError: 'unicode' object has no attribute 'append'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/6x/xsb52c7936l38mmb9f7s268m0000gn/T/pip-install-WFGcOd/pendulum/
You are using pip version 18.1, however version 19.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
I had recently encountered a similar error on Airflow 1.10.2 and in my case it was related to incorrect version of pendulum
Run pip show pendulum
Name: pendulum
Version: 1.4.4
Summary: Python datetimes made easy.
...
If your version of pendulum is different from v1.4.4 just do a force-reinstall (Airflow 1.10.2 requires pendulum===1.4.4)
pip install --force-reinstall pendulum===1.4.4
References
Find which version of package is installed with pip
Installing specific package versions with pip
I didn't exactly solve the issue above, but just decided to switch to python3 and used pip3 install apache-airflow.
Until now, I used the core module of Ansible.
Now I would play also with Extra Module. Let's say that I want to use on my playbook: bigip_partition.
I've seen that it is in https://github.com/F5Networks/f5-ansible/tree/master/library
Could someone tell me how to install them?
I'm on Ubuntu 14LTS, Ansible 2.2
At the moment:
> sudo pip install bigsuds --upgrade pip
> sudo pip install f5-sdk
> sudo pip install git+https://github.com/F5Networks/f5-ansible.git -vvv
Collecting git+https://github.com/F5Networks/f5-ansible.git
Cloning https://github.com/F5Networks/f5-ansible.git to /tmp/pip-NjrWJw-build
Running command git clone -q https://github.com/F5Networks/f5-ansible.git /tmp/pip-NjrWJw-build
Running setup.py (path:/tmp/pip-NjrWJw-build/setup.py) egg_info for package from git+https://github.com/F5Networks/f5-ansible.git
Running command python setup.py egg_info
Traceback (most recent call last):
File "<string>", line 1, in <module>
IOError: [Errno 2] No such file or directory: '/tmp/pip-NjrWJw-build/setup.py'
Cleaning up...
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-NjrWJw-build/
Exception information:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 335, in run
wb.build(autobuilding=True)
File "/usr/local/lib/python2.7/dist-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 634, in _prepare_file
abstract_dist.prep_for_dist()
File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 129, in prep_for_dist
self.req_to_install.run_egg_info()
File "/usr/local/lib/python2.7/dist-packages/pip/req/req_install.py", line 439, in run_egg_info
command_desc='python setup.py egg_info')
File "/usr/local/lib/python2.7/dist-packages/pip/utils/__init__.py", line 707, in call_subprocess
% (command_desc, proc.returncode, cwd))
InstallationError: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-NjrWJw-build/
Thanks,
Riccardo
https://github.com/F5Networks/f5-ansible is not a python package, don't try to install it with pip.
To use those modules you just need to put py-files from library to your playbook-path/library.
You'll have a much better experience if you install the 'App Services 3.0' (AS3) declarative interface on you BIG-IP:
https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/3/
Then you just send 'declarations' of your desired state, instead of working with MANY imperative interfaces...
Here are some example declarations (which are easy to generate/manipulate in Ansible using Jinja2 templates):
https://clouddocs.f5.com/products/extensions/f5-appsvcs-extension/3/userguide/examples.html
When following the Install the AWS CLI Using the Bundled Installer (Linux, OS X, or Unix) directions I'm getting an error caused by what looks to be an incorrect file path file:///Users/jspooner/Downloads/awscli-bundle/packages awscli-1.7.24.tar.gz. I believe it should be packages/awscli. Is there python script broken or am I missing something?
➜ Downloads python --version
Python 2.7.7
➜ Downloads sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
Password:
Running cmd: /usr/local/opt/python/bin/python2.7 virtualenv.py --python /usr/local/opt/python/bin/python2.7 /usr/local/aws
Running cmd: /usr/local/aws/bin/pip install --no-index --find-links file:///Users/jspooner/Downloads/awscli-bundle/packages awscli-1.7.24.tar.gz
Traceback (most recent call last):
File "./awscli-bundle/install", line 138, in <module>
main()
File "./awscli-bundle/install", line 129, in main
pip_install_packages(opts.install_dir)
File "./awscli-bundle/install", line 98, in pip_install_packages
pip_script, PACKAGES_DIR, cli_tarball))
File "./awscli-bundle/install", line 44, in run
p.returncode, cmd, stdout + stderr))
__main__.BadRCError: Bad rc (1) for cmd '/usr/local/aws/bin/pip install --no-index --find-links file:///Users/jspooner/Downloads/awscli-bundle/packages awscli-1.7.24.tar.gz': Traceback (most recent call last):
File "/usr/local/aws/bin/pip", line 9, in <module>
load_entry_point('pip==1.4.1', 'console_scripts', 'pip')()
File "/usr/local/aws/lib/python2.7/site-packages/pkg_resources.py", line 378, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/local/aws/lib/python2.7/site-packages/pkg_resources.py", line 2566, in load_entry_point
return ep.load()
File "/usr/local/aws/lib/python2.7/site-packages/pkg_resources.py", line 2260, in load
entry = __import__(self.module_name, globals(),globals(), ['__name__'])
File "/usr/local/aws/lib/python2.7/site-packages/pip/__init__.py", line 10, in <module>
from pip.util import get_installed_distributions, get_prog
File "/usr/local/aws/lib/python2.7/site-packages/pip/util.py", line 17, in <module>
from pip.vendor.distlib import version
File "/usr/local/aws/lib/python2.7/site-packages/pip/vendor/distlib/version.py", line 13, in <module>
from .compat import string_types
File "/usr/local/aws/lib/python2.7/site-packages/pip/vendor/distlib/compat.py", line 31, in <module>
from urllib2 import (Request, urlopen, URLError, HTTPError,
ImportError: cannot import name HTTPSHandler
Try navigating to the /tmp directory before running amazon's 3 installation commands, rather than running these commands from within your own user folder. I'm not sure why this extra step is required for some OS X installations.
cd /tmp
curl -o awscli.zip https://s3.amazonaws.com/aws-cli/awscli-bundle.zip
unzip awscli.zip
sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
Install PiP and use the following command
pip install awscli --upgrade --user
I would highly recommend installing awscli from Homebrew package manager if you are on OS X.
Hello I am trying to install python-twitter and one of the dependencies HTTPLIB2 is giving me a lot of problems. I have searched the internet up and down but maybe I do not know what the right key words are. Here is the problem.
I am installing python-twitter. I install the requirements either by "pip install -r requirements.txt" or doing them one by one through for example "pip install httplib2". Then I run "python setup.py install". It's happy but then "python setup.py test" fails as followed with an error on import HTTPLIB2.
python setup.py test
running test
running egg_info
writing requirements to python_twitter.egg-info/requires.txt
writing python_twitter.egg-info/PKG-INFO
writing top-level names to python_twitter.egg-info/top_level.txt
writing dependency_links to python_twitter.egg-info/dependency_links.txt
reading manifest file 'python_twitter.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching '.DS_Store'
writing manifest file 'python_twitter.egg-info/SOURCES.txt'
running build_ext
Traceback (most recent call last):
File "setup.py", line 73, in <module>
Main()
File "setup.py", line 66, in Main
setuptools.setup(**METADATA)
File
....
<deleted these parts to save space>
module = __import__('.'.join(parts_copy))
File "/Users/ME/Downloads/python-twitter-1.0/twitter_test.py", line 30, in <module>
import twitter
File "/Users/ME/Downloads/python-twitter-1.0/twitter.py", line 65, in <module>
import oauth2 as oauth
File "/usr/local/lib/python2.7/site-packages/oauth2/__init__.py", line 32, in <module>
import httplib2
File "/usr/local/lib/python2.7/site-packages/httplib2/__init__.py", line 347
print('%s:' % h, end=' ', file=self._fp)
^
SyntaxError: invalid syntax
But I thought the print error was an incompatibility between python 2 and 3. Why am I getting this while I clearly have python 2.7 (I checked and uninstalled and reinstalled HTTPLIB2).
Thanks much for your help
i had a same issue. It seems to be python2.7 and 3.x version conflict issue and resolved by
python2.7 -m pip install -t lib/ -r requirements.txt
Just in case, this happens when you install httplib2 with a python3 version of pip and then execute it with python2.
This happens for example when creating manually an environment with:
pip install -t lib/ -r requirements
As it happens when creating an AppEngine standard environment vendor folder.
This happens because contrary to most of the libraries, httplib2 has completely different versions for python2 and python3.