I am trying to zip python 2.7 code with pandas and cassandra libraries on my local windows 7 computer. From the anaconda prompt I did pip install -t 'folder location' for both pandas and cassandra and I zipped the contents - not the folder. I am getting the error Unable to import module 'lambda_function': Missing required dependencies ['numpy']. However, pandas included numpy as a dependency and it is in the zip file. zip file contents
I researched this isssue, and found Pandas & AWS Lambda the marked answer says: All static packages have to be compiled on an EC2 Amazon Linux instance.
This article that was referenced in another question Problems using MySQL with AWS Lambda in Python says to use EC2 Amazon linux and virtualenv: http://www.perrygeo.com/running-python-with-compiled-code-on-aws-lambda.html
Does this mean I am having problems because I am using my local windows 7 computer, is it required to use EC2 Amazon linux?
I believe the answer to my question is Yes, you do have to use EC2 Amazon Linux. I found this post on the AWS forum regarding the Cassandra-Driver library: https://forums.aws.amazon.com/thread.jspa?threadID=247496
I was a little unsure about how to do this but I found a great tutorial from AWS:
https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example-deployment-pkg.html
I followed the instructions and created a virtualenv and downloaded everything - seems to be working. I got past that error but now it is unable to download from S3, I will create a new thread for that.
Related
In databricks there is the following magic command $sh, that allows you run bash commands in a notebook.
For example if I wanted to run the following code in Databrick:
pip install great_expectations
I would issue the following code:
%sh
pip install great_expectations
Can someone let me know what the equivalent is with Apache Spark notebook in Azure Synapse?
It may well be that it isn't possible with Azure Synapse but I don't know.
Just to add to this question, in Databricks when I run the following command
great_expectations init
The command remains stuck in running, see image
However, what I would expect from a regular Linux OS when I run the same code would be
OK to proceed? [Y/n]:
Is there something I could add to
great_expectations init
To make the code return
OK to proceed? [Y/n]:
Thanks
Azure Synapse Analytics Spark pool supports - Only following magic commands are supported in Synapse pipeline : %%pyspark, %%spark, %%csharp, %%sql.
Python packages can be installed from repositories like PyPI and Conda-Forge by providing an environment specification file.
Steps to install python package in Synapse Spark pool.
Step1: Get the packages details like name & version from pypi.org
Note: (great_expectations) and (0.13.19)
Step2: Create a requirements.txt file using the above name and version.
Step3: Upload the package to the Synapse Spark Pool.
Step4: Save and wait for applying packages settings in Synapse Spark pools.
Step5: Verify installed libraries
To verify if the correct versions of the correct libraries are installed from PyPI, run the following code:
import pkg_resources
for d in pkg_resources.working_set:
print(d)
For more details, refer to Manage Python libraries for Apache Spark in Azure Synapse Analytics.
I am trying to install aws elastic beanstalk cli on my mac machine. In the prerequisites it says that Python requires following:
Xcode openssl zlib readline
But, on running the command, I get:
-bash: Xcode: command not found
How am I suppose to install this?
An alternative solution is to run the aws cli inside a docker container. It will eliminate the possible issues which may be caused by the OS/library upgrading in the future.
I want to deploy an AWS EC2 instance of Parrot Linux and I cannot find it in the marketplace.
I have uploaded an OVA to S3 of the latest 4.7 version and tried to import but got an error:
ClientError: Unsupported kernel version 5.0-parrot
Even when the kernel version is 4.19 (from uname -r command)
Is there a way to get Linux Parrot on AWS?
You could create a virtualbox/ virtualmachine image and then convert this to a AMI image.
Here is a write-up on how to do it. http://www.ioncannon.net/system-administration/1246/converting-from-virtualbox-or-vmware-to-ec2-now-easier-than-ever/
If you have been able to bootup your virtualbox, there should not be any issue launching an instance through AMI as well.
I have successfully imported a Parrot Home 4.7 machine to AWS.
After installing kernel version 4.15.0-65-generic and removing the stock one the image was imported without any issues.
I am unable to find how to install openstack on my macbook.
I tried googling but every where i am getting for ubuntu and not for mac
Kindly help me to install as i am a noob in openstack
Late reply but just in case:
You can install VirtualBox and get ubuntu running on it. Directly installing OpenStack on your macbook is usually a bad thing to do since it can mess up your system.
Very late answer but this is to install the openstack command line clients.
first, you have to have python installed on your machine. Open the terminal and type python --version. If you see something like Python 2.7.9, go to next step.
So now, you have to install pip. Pip is a python package manager. You do this by typing easy_install pip. Should work.
Then you have to install every clients one by one.
(taken straight from http://docs.openstack.org/user-guide/common/cli_install_openstack_command_line_clients.html)
ceilometer - Telemetry API
cinder - Block Storage API and extensions
glance - Image service API
heat - Orchestration API
neutron - Networking API
nova - Compute API and extensions
sahara - Database Processing API
swift - Object Storage API
trove - Database service API
openstack - Common OpenStack client supporting multiple services
And the way to install those, one by one, is by calling:
pip install python-PROJECTclient.
So, for instance, you go pip install python-openstackclient. Or pip install python-novaclient. You do that for every client that you need.
Don't forget, before you be able to use those APIs, you have to set up particular environment variables in your shell by sourcing an "openrc" file: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html
Once this is done, you may use the clients API.
You can install Openstack's python-client which is tools for sending command via http.
But there is no official option to install Openstack services on mac, yet. You can use Hypervisor like Virtualbox and Vmware to install Linux and launch Openstack Services, or docker.
It looks like progress has been made since these last posts. The easiest answer for just testing out OpenStack on your local machine is to use devstack.
Tutorial:
http://anteaya.info/blog/2013/09/01/installing-devstack-with-vagrant/
NOTE: The tutorial mentions to mv samples/localrc . but the file name has been updated to local.conf. Also, I copied the file over instead of moving it.
DevStack Homepage:
http://docs.openstack.org/developer/devstack/
I'm a new user of Amazon EC2.
I want to compile the pptpd package on EC2, but receive the following error:
[root#ip-10-112-xxx-xxx /]# /var/tmp/rpm-tmp.2eILT0: line 58: /usr/bin/make: No such file or directory
I searched the entire root directory tree, but make isn't available:
[root#ip-10-112-59-187 /]# find . -name "make"
./etc/mail/make
I'm wondering whether make is actually installed on the Amazon Linux AMI initially? If not, how do I install it?
Preface
The Amazon Linux AMI is (loosely) based on CentOS and a perfectly decent OS for EC2, in fact it has been tailored by Amazon for EC2 specifically:
The Amazon Linux AMI is a supported and maintained Linux image
provided by Amazon Web Services for use on Amazon Elastic Compute
Cloud (Amazon EC2). It is designed to provide a stable, secure, and
high performance execution environment for applications running on
Amazon EC2. It also includes packages that enable easy integration
with AWS, [...]. Amazon Web Services provides ongoing security and
maintenance updates to all instances running the Amazon Linux AMI. [...] [emphasis mine]
However, it is indeed not as widely used yet as some other distributions, with the most popular likely being Ubuntu due to its popularity in general and its dedicated long time tailored support of EC2 in particular (see e.g. the EC2StartersGuide, the Ubuntu Cloud Images or the convenient listing of the Ubuntu AMIs for Amazon EC2 on alestic). This yields two drawbacks:
You'll find much more examples/tutorials/etc. for EC2 based on Ubuntu, making things easier eventually.
You'll find slightly less precompiled packages available for CentOS, requiring compiling your own eventually (but see below).
Solution
That said, CentOS (and the Amazon Linux AMI in turn) uses the Yum package manager to install and update packages from CentOS (and 3rd party) Repositories (Debian/Ubuntu use the APT package manager instead - the inherent concepts are very similar though), see e.g. section Adding Packages in Amazon Linux AMI Basics:
In addition to the packages included in the Amazon Linux AMI, Amazon
provides a yum repository consisting of common Linux applications for
use inside of Amazon EC2. The Amazon Linux AMI is configured to point
to this repository by default for all yum actions. The packages can be
installed by issuing yum commands. For example:
# sudo yum install httpd
Accordingly, you can install make via yum install make (you can get a listing of all readily available packages via yum list all).
Be advised though, that you might actually not need to do that, insofar the Amazon Linux AMI has been built to be binary-compatible with the CentOS series of releases, and therefore packages built to run on CentOS should also run on the Amazon Linux AMI. [emphasis mine]
The desired package pptpd is not part of the standard repositories on CentOS either though, but it is available in the 3rd party Extra Packages for Enterprise Linux (EPEL) repository (see Letter P) - I can't comment on the viability of using this one vs. compiling your own though.
Good luck!
Make is not installed by default on Amazon Linux AMIs. However, you can install it quite easily with yum. If you choose to only install make, you might get some errors later for other packages in the compilation process. If you are going to compile software, you might want to just install all of the development tools at once.
sudo yum groupinstall "Development Tools"
sudo yum groupinstall "Development Tools"
According to the documentation: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/compile-software.html