Fetch boto3 credentials only from EC2 instance profile - amazon-ec2

The boto3 documentation lists the order in which credentials are searched and the credentials are fetched from the EC2 instance metadata service only at the very last.
How do I force boto3 to fetch the credentials only from the EC2 instance profile or the instance metadata service?
I came across this which lets me get the temporary credentials from the metadata service and then I could pass this on to create a boto3 session.
However my question is whether there is a better way to do this? Is it possible to create a boto3 session by specifying the provider to use ie InstanceMetadataProvider - link? I tried searching the docs a lot, but couldn't figure it out.
The reason - the context under which this script runs also has environment variables with AWS keys set which would obviously take precedence, however I need the script to run only with the IAM role assigned to the EC2 instance.

So I ended up doing this, works as expected. Always uses the temp creds from the instance role. The script is short-lived so the validity of the creds is not an issue.
from botocore.credentials import InstanceMetadataProvider, InstanceMetadataFetcher
provider = InstanceMetadataProvider(iam_role_fetcher=InstanceMetadataFetcher(timeout=1000, num_attempts=2))
creds = provider.load().get_frozen_credentials()
client = boto3.client('ssm', region_name='us-east-1', aws_access_key_id=creds.access_key, aws_secret_access_key=creds.secret_key, aws_session_token=creds.token)
If there is a better way to do, please feel free to post.

You could also use boto3.
>>> session = boto3.Session(region_name='foo_region')
>>> credentials = session.get_credentials()
>>> credentials = credentials.get_frozen_credentials()
>>> credentials.access_key
u'ABC...'
>>> credentials.secret_key
u'DEF...'
>>> credentials.token
u'ZXC...'
>>> access_key = credentials.access_key
>>> secret_key = credentials.secret_key
It's a similar idea, but I find it returns much faster

import boto3
import botocore
botocore_session = botocore.session.get_session()
credential_provider = botocore_session.get_component('credential_provider')
instance_metadata_provider = credential_provider.get_provider('iam-role')
credential_provider.insert_before('env', instance_metadata_provider)
boto3_session = boto3.Session(botocore_session=botocore_session)
client = boto3_session.client(...)
resource = boto3_session.resource(...)

Related

Use Azure Blob storage instead of S3 buckets in Apache Superset cache

I'm trying to enable thumbnail caching in my Superset instance. The official docs provide an example of caching the thumbnail images on Amazon S3. But I'm on the Azure cloud. How could I rework this block of code to work with an Azure blob (or other appropriate Azure storage) instead?
From the docs:
from flask import Flask
from s3cache.s3cache import S3Cache
...
class CeleryConfig(object):
broker_url = "redis://localhost:6379/0"
imports = ("superset.sql_lab", "superset.tasks", "superset.tasks.thumbnails")
result_backend = "redis://localhost:6379/0"
worker_prefetch_multiplier = 10
task_acks_late = True
CELERY_CONFIG = CeleryConfig
def init_thumbnail_cache(app: Flask) -> S3Cache:
return S3Cache("bucket_name", 'thumbs_cache/')
THUMBNAIL_CACHE_CONFIG = init_thumbnail_cache
# Async selenium thumbnail task will use the following user
THUMBNAIL_SELENIUM_USER = "Admin"
I have the rest of my caching working with Redis. I see these Python packages that look relevant: https://github.com/alejoar/Flask-Azure-Storage and https://pypi.org/project/azure-storage-blob/ but I'm not sure where to go next.

How to get detailed VM Size information

I would like to use the Python Azure SDK to find the VM Sizes that support Enhanced Networking as well as AVX-512. The method I've seen so far to query information about VM Sizes is ComputeManagementClient.virtual_machine_sizes.list(region). But, the information returned doesn't include whether Enhanced Networking is supported for each VM type, or whether AVX-512 is supported.
This is an example of what one entry of virtual_machine_sizes.list provides:
{'name': 'Standard_M208ms_v2', 'numberOfCores': 208, 'osDiskSizeInMB': 1047552, 'resourceDiskSizeInMB': 4194304, 'memoryInMB': 5836800, 'maxDataDiskCount': 64}
I found on https://learn.microsoft.com/en-us/rest/api/compute/resourceskus/list that perhaps the resource skus list will provide the info I'm looking for. But, I don't see a way to use that resource skus list function in the Python SDK.
I am using version 4.0.0 of Python's azure library. Installed it via:
pip3 install -Iv azure==4.0.0
Thank you in advance for any help you can provide!
If you want to list azure vm resource sku with python, please refer to the following steps:
Create a service principal and assign Contributor role to the sp
az login
#create sp and assign Contributor role at subscription level
az ad sp create-for-rbac -n "your service principal name"
code
from azure.mgmt.compute import ComputeManagementClient
from azure.common.credentials import ServicePrincipalCredentials
client_id = "sp appId"
secret = "sp password"
tenant = "sp tenant"
credentials = ServicePrincipalCredentials(
client_id = client_id,
secret = secret,
tenant = tenant
)
Subscription_Id = ''
compute_client =ComputeManagementClient(credentials,Subscription_Id)
resource_group_name='Networking-WebApp-AppGW-V1-E2ESSL'
virtual_machine_scale_set_name='VMSS'
results = compute_client.resource_skus.list(raw=True)
resourceSkusList = [result.as_dict() for result in results]
r=json.dumps(resourceSkusList)
print(r)
For more details, please refer to here.

How can I call list of ec2 instance based on the app code using tag method

I am trying to get all the instance(server name) ID based on the app. Let's say I have an app in the server. How do I know which apps below to which server. I want my code to find all the instance (server) that belongs to each app. Is there any way to look through the app in the ec2 console and figure out the servers are associated with the app. More of using tag method
import boto3
client = boto3.client('ec2')
my_instance = 'i-xxxxxxxx'
(Disclaimer: I work for AWS Resource Groups)
Seeing your comments that you use tags for all apps, you can use AWS Resource Groups to create a group - the example below assumes you used App:Something as tag, first creates a Resource Group, and then lists all the members of that group.
Using this group, you can for example get automatically a CloudWatch dashboard for those resources, or use this group as a target in RunCommand.
import json
import boto3
RG = boto3.client('resource-groups')
RG.create_group(
Name = 'Something-App-Instances',
Description = 'EC2 Instances for Something App',
ResourceQuery = {
'Type': 'TAG_FILTERS_1_0',
'Query': json.dumps({
'ResourceTypeFilters': ['AWS::EC2::Instance'],
'TagFilters': [{
'Key': 'App',
'Values': ['Something']
}]
})
},
Tags = {
'App': 'Something'
}
)
# List all resources in a group using a paginator
paginator = RG.get_paginator('list_group_resources')
resource_pages = paginator.paginate(GroupName = 'Something-App-Instances')
for page in resource_pages:
for resource in page['ResourceIdentifiers']:
print(resource['ResourceType'] + ': ' + resource['ResourceArn'])
Another option to just get the list without saving it as a group would be to directly use the Resource Groups Tagging API
What you install on an Amazon EC2 instance is totally up to you. You do this by running code on the instance itself. AWS is not involved in the decision of what you install on the instance, nor does it know what you installed on an instance.
Therefore, you will need to keep track of "what apps are installed on what server" yourself.
You might choose to take advantage of Tags on instances to add some metadata, such as the purpose of the server. You could also use AWS Systems Manager to run commands on instances (eg to install software) or even use AWS CodeDeploy to roll-out software to fleets of servers.
However, even with all of these deployment options, AWS cannot track what you have put on each individual server. You will need to do that yourself.
Update: You can use AWS Resource Groups to view/manage resources by tag.
Here's some sample Python code to list tags by instance:
import boto3
ec2_resource = boto3.resource('ec2', region_name='ap-southeast-2')
instances = ec2_resource.instances.all()
for instance in instances:
for tag in instance.tags:
print(instance.instance_id, tag['Key'], tag['Value'])

Google API + proxy + httplib2

I'm currently running a script to pull data from Google Analytics with googleapiclient Python package (that is based on httplib2 client object)
--> My script works perfectly without any proxy.
But I have to put it behind my corporate proxy, so I need to adapt my httplib2.Http() object to embed proxy information.
Following httplib2 doc 1 I tried:
pi = httplib2.proxy_info_from_url('http://user:pwd#someproxy:80')
httplib2.Http(proxy_info=pi).request("http://www.google.com")
But it did not work.
I always get a Time out error, with or without the proxy info (so proxy_info in parameter is not taken into account)
I also downloaded socks in PySocks package (v1.5.6) and tried to "wrapmodule" httplib2 as described in here:
https://github.com/jcgregorio/httplib2/issues/205
socks.setdefaultproxy(socks.PROXY_TYPE_HTTP, "proxyna", port=80, username='p.tisserand', password='Telematics12')
socks.wrapmodule(httplib2)
h = httplib2.Http()
h.request("http://google.com")
But I get an IndexError: (tuple index out of range)
In the meantime,
When I use the requests package, this simple code works perfectly:
os.environ["HTTP_PROXY"] = "http://user:pwd#someproxy:80"
req = requests.get("http://www.google.com")
The problem is that need to fit with googleapiclient requirements and provide a htpplib2.Http() client object.
rather than using Python2, I think you'd better try using httplib2shim
You can have a look at this tutorial on my blog :
https://dinatam.com/fr/python-3-google-api-proxy/
In simple words, just replace this kind of code :
from httplib2 import Http
http_auth = credentials.authorize(Http())
by this one :
import httplib2shim
http_auth = credentials.authorize(httplib2shim.Http())
I decided to recode my web app in Python 2, still using the httplib2 package.
Proxy info are now taken into account. It now works.

"Insufficient permissions" on google calendar api's acl.list

I'm getting Insufficient permissions when trying to call the acl.list method of the google calendar api via python.
service.acl().list(calendarId='primary').execute();
*** HttpError: <HttpError 403 when requesting https://www.googleapis.com/calendar/v3/calendars/primary/acl?alt=json returned "Insufficient Permission">
I'm using the scope 'https://www.googleapis.com/auth/calendar' as recommended in the documentation. Additionally, other API methods do work, for example service.calendarList
service.calendarList().list(pageToken=page_token).execute()
What am I missing?
Here is the code I'm using based almost entirely on the sample they provide:
import sys
from oauth2client import client
from googleapiclient import sample_tools
def main(argv):
# Authenticate and construct service.
# import pdb;pdb.set_trace()
service, flags = sample_tools.init(
argv, 'calendar', 'v3', __doc__, __file__,
# scope='https://www.googleapis.com/auth/calendar.readonly')
scope='https://www.googleapis.com/auth/calendar')
try:
page_token = None
while True:
calendar_list = service.calendarList().list(pageToken=page_token).execute()
for calendar_list_entry in calendar_list['items']:
print calendar_list_entry['summary']
page_token = calendar_list.get('nextPageToken')
service.acl().list(calendarId='primary').execute();
if not page_token:
break
except client.AccessTokenRefreshError:
print ('The credentials have been revoked or expired, please re-run'
'the application to re-authorize.')
if __name__ == '__main__':
main(sys.argv)
You might have to delete existing credentials, in the form of .json files. I had a similar "Insufficient permissions" problem, and I had to delete stored credentials. I had the additional problem that because of trying out some of Google's scripts in their tutorials, unknowingly I had credentials stored in a hidden .credentials folder in my home directory (users/home). Since they were hidden, I had to look for them through Terminal (on Mac), and delete them there. Once deleted, the problem was solved, since I could create new and proper credentials, suitable for the scope of my new script.
Something is wrong with your authentication. Insufficent permissions means that you don't have access.
I can verify that the scope https://www.googleapis.com/auth/calendar is enough to display ACL.list on the primary calendar.
You have to find the location of "calendar-dotnet-quickstart.json" file and delete it. I used .NET example and I have to debug the following code the find exact location.
string credPath = System.Environment.GetFolderPath(
System.Environment.SpecialFolder.Personal);
credPath = Path.Combine(credPath, ".credentials/calendar-dotnet-quickstart.json");
Then change scope as bellow and rebuild the solution.
string[] scopes = { CalendarService.Scope.Calendar};
You will notice that google will ask to confirm the access again.

Resources