In AWS Greengrass Documentation it says you can test components like this
sudo /greengrass/v2/bin/greengrass-cli deployment create \
--recipeDir ~/greengrassv2/recipes \
--artifactDir ~/greengrassv2/artifacts \
--merge "com.example.HelloWorld=1.0.0"
But if I want to run a component from another script. I should use the same command? For example I have a component that publishes some data to MQTT, and right now I am using system.os like this:
os.system("sudo /greengrass/v2/bin/greengrass-cli deployment create \
--recipeDir ~/greengrassv2/recipes \
--artifactDir ~/greengrassv2/artifacts \
--merge "com.example.HelloWorld=1.0.0"")
But I am not sure if it's the right solution. It does not seem like a nice solution.
I wouldn't recommend using greengrass-cli deployment create command to run the component
It's for local development only
The command run through all the lifecycle steps defined in the component recipe file before running the component, it can be a big overhead.
If "another script" is also a Greengrass component, you can Use the AWS IoT Device SDK for interprocess communication (IPC)
If "another script" is not Greengrass component, you can use restart command to trigger a run for the component. It's less overhead than create command.
sudo /greengrass/v2/bin/greengrass-cli component restart --names "HelloWorld"
Related
I have a PowerShell script that I want to run on some Azure AKS nodes (running Windows) to deploy a security tool. There is no daemon set for this by the software vendor. How would I get it done?
Thanks a million
Abdel
Similar question has been asked here. User philipwelz has written:
Hey,
although there could be ways to do this, i would recommend that you dont. The reason is that your AKS setup should not allow execute scripts inside container directly on AKS nodes. This would imply a huge security issue IMO.
I suggest to find a way the execute your script directly on your nodes, for example with PowerShell remoting or any way that suits you.
BR,
Philip
This user is right. You should avoid executing scripts on your AKS nodes. In your situation if you want to deploy Prisma cloud you need to go with the following doc. You are right that install scripts work only on Linux:
Install scripts work on Linux hosts only.
But, for the Windows and Mac software you have specific yaml files:
For macOS and Windows hosts, use twistcli to generate Defender DaemonSet YAML configuration files, and then deploy it with kubectl, as described in the following procedure.
The entire procedure is described in detail in the document I have quoted. Pay attention to step 3 and step 4. As you can see, there is no need to run any powershell script:
STEP 3:
Generate a defender.yaml file, where:
The following command connects to Console (specified in [--address](https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_kubernetes.html#)) as user <ADMIN> (specified in [--user](https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_kubernetes.html#)), and generates a Defender DaemonSet YAML config file according to the configuration options passed to [twistcli](https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_kubernetes.html#). The [--cluster-address](https://docs.paloaltonetworks.com/prisma/prisma-cloud/prisma-cloud-admin-compute/install/install_kubernetes.html#) option specifies the address Defender uses to connect to Console.
$ <PLATFORM>/twistcli defender export kubernetes \
--user <ADMIN_USER> \
--address <PRISMA_CLOUD_COMPUTE_CONSOLE_URL> \
--cluster-address <PRISMA_CLOUD_COMPUTE_HOSTNAME>
- <PLATFORM> can be linux, osx, or windows.
- <ADMIN_USER> is the name of a Prisma Cloud user with the System Admin role.
and then STEP 4:
kubectl create -f ./defender.yaml
I think that the above answer is not completely correct.
The twistcli command, does not export daemonset for Windows Nodes. The "PLATFORM" option, is for choosing the OS of the computer that the command will run.
After testing, I have made the conclusion that there is no Docker Image for Prisma Cloud for Windows Kubernetes Nodes, as it is deployed as a service at Windows OS, and not Container (as in Linux). Wrapping up, the Daemonset is not working at the Windows Hosts
I believe the only solution is this -> Windows
This is the Powershell script that Wytrzymały Wiktor has mentioned.
Unfortunately this cannot be automated easily, as you have to deploy an Azure VM per AKS Cluster (at the same network), and RDP to the AKS Windows Node and run the script.
If anyone has another suggestion or solution, feel free to share.
we have a problem setting up aws-sigv4 and connecting an AWS AMP workspace via docker images.
TAG: grafana/grafana:7.4.5
Main problem is that in the UI the sigv4 configuration screen does not appear.
Installing grafana:7.4.5 locally via Standalone Linux Binaries works.
Just setting the environment variables,
export AWS_SDK_LOAD_CONFIG=true
export GF_AUTH_SIGV4_AUTH_ENABLED=true
the configuration screen appears.
Connecting and querying data to AMP via corresponding IAM instance role is working flawlessly.
Doing the same in the docker image as ENV Variables does NOT work.
When using grafana/grafana:sigv4-web-identity it works, but it seems to me that this is just a "test image".
How to configure the default grafana image in order to enable sigV4 authentication?
It works for me:
$ docker run -d \
-p 3000:3000 \
--name=grafana \
-e "GF_AUTH_SIGV4_AUTH_ENABLED=true" \
-e "AWS_SDK_LOAD_CONFIG=true" \
grafana/grafana:7.4.5
You didn't provide minimal reproducible example, so it's hard to say what is a problem in your case.
Use variable GF_AWS_SDK_LOAD_CONFIG instead of AWS_SDK_LOAD_CONFIG.
I'm having a surprisingly hard time working with additional libraries via my EMR notebook. The AWS interface for EMR allows me to create Jupyter notebooks and attach them to a running cluster. I'd like to use additional libraries in them. SSHing into the machines and installing manually as ec2-user or root will not make the libraries available to the notebook, as it apparently uses the livy user. Bootstrap actions install things for hadoop. I can't install from the notebook because its user apparently doesn't have sudo, git, etc., and it probably wouldn't install to the slaves anyway.
What is the canonical way of installing additional libraries for notebooks created through the EMR interface?
For the sake of an example, let's assume you need librosa Python module on running EMR cluster. We're going to use Python 2.7 as the procedure is simpler - Python 2.7 is guaranteed to be on the cluster and that's the default runtime for EMR.
Create a script that installs the package:
#!/bin/bash
sudo easy_install-2.7 pip
sudo /usr/local/bin/pip2 install librosa
and save it to your home directory, e.g. /home/hadoop/install_librosa.sh. Note the name, we're going to use it later.
In the next step you're going to run this script through another script inspired by Amazon EMR docs: emr_install.py. It uses AWS Systems Manager to execute your script over the nodes.
import time
from boto3 import client
from sys import argv
try:
clusterId=argv[1]
except:
print("Syntax: emr_install.py [ClusterId]")
import sys
sys.exit(1)
emrclient=client('emr')
# Get list of core nodes
instances=emrclient.list_instances(ClusterId=clusterId,InstanceGroupTypes=['CORE'])['Instances']
instance_list=[x['Ec2InstanceId'] for x in instances]
# Attach tag to core nodes
ec2client=client('ec2')
ec2client.create_tags(Resources=instance_list,Tags=[{"Key":"environment","Value":"coreNodeLibs"}])
ssmclient=client('ssm')
# Run shell script to install libraries
command=ssmclient.send_command(Targets=[{"Key": "tag:environment", "Values":["coreNodeLibs"]}],
DocumentName='AWS-RunShellScript',
Parameters={"commands":["bash /home/hadoop/install_librosa.sh"]},
TimeoutSeconds=3600)['Command']['CommandId']
command_status=ssmclient.list_commands(
CommandId=command,
Filters=[
{
'key': 'Status',
'value': 'SUCCESS'
},
]
)['Commands'][0]['Status']
time.sleep(30)
print("Command:" + command + ": " + command_status)
To run it:
python emr_install.py [cluster_id]
What is the canonical way of installing additional libraries for notebooks created through the EMR interface?
EMR Notebooks recently launched 'notebook-scoped libraries' using which you can install additional Python libraries on your cluster from public or private PyPI repository and use it within notebook session.
Notebook-scoped libraries provide the following benefits:
You can use libraries in an EMR notebook without having to re-create
the cluster or re-attach the notebook to a cluster.
You can isolate library dependencies of an EMR notebook to the individual notebook session. The libraries installed from within the notebook cannot interfere with other libraries on the cluster or libraries installed within other notebook sessions.
Here are more details,
https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-managed-notebooks-scoped-libraries.html
Technical blog:
https://aws.amazon.com/blogs/big-data/install-python-libraries-on-a-running-cluster-with-emr-notebooks/
What I usually do in this case is deleting my cluster and creating a new one with bootstrap actions. Bootstrap actions allow you to install additional libraries on your cluster : https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-bootstrap.html.
For example writing the following script and saving it in S3 will allow you to use datadog from your notebook running on top of your cluster (at least it works with EMR 5.19) :
#!/bin/bash -xe
#install datadog module for using in pyspark
sudo pip-3.4 install -U datadog
Here is the command line I would run for launching this cluster :
aws emr create-cluster --release-label emr-5.19.0 \
--name 'EMR 5.19 test' \
--applications Name=Hadoop Name=Spark Name=Hive Name=Livy \
--use-default-roles \
--instance-groups \
InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m4.large \
InstanceGroupType=CORE,InstanceCount=2,InstanceType=m4.large \
--region eu-west-1 \
--log-uri s3://<path-to-logs> \
--configurations file://config-emr.json \
--bootstrap-actions Path=s3://<path-to-bootstrap-in-aws>,Name=InstallPythonModules
And the config-emr.json that is stored locally on your computer :
[{
"Classification": "spark",
"Properties": {
"maximizeResourceAllocation": "true"
}
},
{
"Classification": "spark-env",
"Configurations": [
{
"Classification": "export",
"Properties": {
"PYSPARK_PYTHON": "/usr/bin/python3"
}
}
]
}]
I assume that you could do exactly the same thing when creating a cluster through the EMR interface by going to the advanced options of creation.
I spent way to long on this, AWS documentation or support did not help at all, but did get it working so that you can install libraries for python directly in the notebook.
If you can do the items below then you can install libraries though running pip install commands in a single line Jupyter cell, with the Python runtime, like so
!pip install pandas
One item that confused me a lot was that I could SSH into the cluster and reach out to the internet, ping and pip would both work, but then the notebook was not able to reach out nor were any libraries actually available. Instead you need to make sure that the notebook can reach out. One good test is just to see if you can ping out. Same structure as above, single line starting with !
!ping google.com
If that is taking too long and timing out then you still need to figure out your VPN/subnet rules.
Notes below on cluster creation:
(step 1) This does not work for every version of EMR. I have it working on 5.30.0, but last I checked 5.30.1 did not work.
(step 2 -> Networking) You need to make sure you're on a private subnet and your VPN can reach out to the public internet. Again, don't let SHHing into the server fool you, the notebook is either inside a docker image there or running somewhere else. The only relevant tests are the ones you're running directly from the notebook.
Once you have this working and install a package it will work for any notebook on that cluster. I have a notebook just called install that has one line per package that I run through whenever I spin up a new cluster.
I have a set up that needs to be bootstrapped off the values of some files in another VM.
Here is the run command I am using to invoke the run run command:
BOOT_VM="${VM_NAME}1"
BOOT_ENODE=$(az vm run-command invoke --name ${BOOT_VM} \
--command-id RunShellScript \
--resource-group ${RSC_GRP_NAME} \
--query "value[].message" \
--output tsv \
--scripts "cat /etc/parity/enode.pub")
echo ${BOOT_ENODE}
The result I get is :
Enable succeeded: [stdout] [stderr]
As far as I know, this could mean 2 things:
There is no file there
I am handling the response wrongly.
Really hoping it isnt 1 and would like advice on how to approach this.
For your issue, there is also a reason that the agent in the vm is not at work or something not good happens to it. The Azure VM agent manages interactions between an Azure VM and the Azure fabric controller. So you should check if it works well.
Update
You can check the agent in the portal:
Also, you can check the agent inside the vm:
For example, I want to get the config of vim in the vm and the vm os is Red Hat 7.2. Then the result of the command az vm run-command invoke will like below:
Can anyone please tell me how to run a shell script on all the ec2 instances that are part of an auto scaling group?
The scenario is that I have a script that I want to run on many ec2 instances that are turned on automatically as part of auto scaling group. The native approach is to SSH to each instance and run the script. I am looking for a way by which it can run automatically on all the instances when I run it on one of the ec2 instance or any better way of doing this?
Thanks in advance.
You'll want to add that shell script to the userdata in a NEW Launch Config and then update the autoscaling group.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-shell-scripts
Updating the Launch Config
if you want to change the launch configuration for your Auto Scaling group, you must create a launch configuration and then update your Auto Scaling group with the new launch configuration. When you change the launch configuration for your Auto Scaling group, any new instances are launched using the new configuration parameters, but existing instances are not affected.
https://docs.aws.amazon.com/autoscaling/latest/userguide/LaunchConfiguration.html
You can implement in many different ways...
Use awscli to get all instances in auto scaling group
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-name myTestGroup
SSHkit is interesting tool run commands on remote servers
or
Invest your time to automate your infrastructure in proper tools like puppet, chef. With puppet mcollective you can do magic, including what you have asked about.
Update:
When you add instance to autoscaling group new tag name=aws:autoscaling:groupName, value=name_of_assigned_autoscaling_group is added, thus its easy to find it searching for this tag.
$ asgName=testASG
$ aws ec2 describe-instances --filters Name=tag-key,Values='aws:autoscaling:groupName' Name=tag-value,Values=$asgName --output text --query 'Reservations[*].Instances[*].[InstanceId,PublicIpAddress]'
The output you will get from the command above is the instance name and public IP:
i-4c42aabc 52.1.x.y
You can use this is your script...
I would do it using Chef. Opsworks (an aws product) is chef plus a lot of things that will do exactly what you want and give you even more flexibility.
Run Command perhaps?
You can use it to invoke scripts as well:
*Nice cheat: Select Run Command on the left-side menu of the EC2 section. Click the "Run Command" button. Then setup your AWS-RunShellScript. Type in your code. Then at the bottom there is a dropdown labelled: "AWS Command Line Interface command", select the correct platform and copy/paste the command into a script.
$Command_Id = aws ssm send-command --document-name "AWS-RunPowerShellScript" --targets '{\"Key\":\"tag:Name\",\"Values\":[\"RunningTests\"]}' --parameters '{\"commands\":[\"Start-Process \\\"C:\\\\path\\\\to\\\\scripts\\\\LOCAL_RUN.ps1\\\"\"]}' --comment "Run Tests Locally" --timeout-seconds 3800 --region us-east-1 --query 'Command.CommandId' --output text | Out-String
Per the question: Use your Auto Scaling Group Name instead of "RunningTests." Or, in the console: In your "Run Command" setup. Select the "Specifiying a Tag" radio button, then "Name" and your Auto Scaling Group.
*Note: The command above is windows powershell, but you can convert your script to Linux/OS or whatever by selecting the correct platform when in the Run Command setup.
**Note: Ensure your User on that instance has the AmazonSSMFullAccess permission setup to run the commands.
***Note:The SSM Agent comes installed on Windows Instances by default. If you are running Linux or something else, you might need to install the SSM Agent.
For a simplistic implementation you can use python fabric (http://www.fabfile.org/). It is a tool to run commands from a local or bashin instance to list of servers.
Here is a repo which has basic scaffolding and examples. Lot of CICD tools are having features to target this requirements, but I found fabric the easiest to implement fo simple setup.
https://github.com/techsemicolon/python-fabric-ec2-aws