There is cinder quota-usage <tenant_id> to show the quota usage for a particular project. Is there any way to show the usage of all projects with one command? Some sort of --all or so?
Quotas are per project based, so cinder would not have that information, you have to query nova.
$ openstack quota show
Remember that part of the cinder related quota are, volumes, snapshots and backups, in order to query all the projects for that information you could script that, this is an example of how to do that.
$ for p in $(openstack project list -c Name -f value); do echo $p ; openstack quota show $p | grep 'backups\|snapshots\|volumes'; done
Related
I am very new to using Google cloud and cloud servers, and I am stuck on a very basic question.
I would like to bulk download some ~60,000 csv.gz files from an internet server (with permission). I compiled a bunch of curl scripts that pipe into a gsutil that uploads to my bucket into an .sh file that looks like the following.
curl http://internet.address/csvs/file1.csv.gz | gsutil cp - gs://my_bucket/file1.csv.gz
curl http://internet.address/csvs/file2.csv.gz | gsutil cp - gs://my_bucket/file2.csv.gz
...
curl http://internet.address/csvs/file60000.csv.gz | gsutil cp - gs://my_bucket/file60000.csv.gz
However this will take ~10 days if I run from my machine, so I'd like to run it from the cloud directly. I do not know the best way to do this. This is too long of a process to use the Cloud Shell directly, and I'm not sure what other app on the Cloud is the best way to run an .sh script that downloads to a Cloud Bucket, or if this type of .sh script is the most efficient method to go about bulk downloading files from the internet using the apps on Google Cloud.
I've seen some advice to use SDK, which I've installed on my local machine, but I don't even know where to start with that.
Any help with this is greatly appreciated!
Gcloud and Cloud Storage doesn't offer the possibility to grab objects from internet and copy these directly on a bucket without intermediary (computer,server or cloud application).
Regarding which Cloud service can help you for run a bash script, you can use a GCE always free F1-micro instance VM (1 instance free per billing account)
To improve the upload files to a bucket, you can use GNU parrallel to run multiple Curl Commands at the same time and improve the time to complete this task.
To install parallel on ubuntu/debian run this command:
sudo apt-get install parallel
For example you can create a file called downloads with the commands that you want to parallelize (you must write all curl commands in the file)
downloads file
curl http://internet.address/csvs/file1.csv.gz | gsutil cp - gs://my_bucket/file1.csv.gz
curl http://internet.address/csvs/file2.csv.gz | gsutil cp - gs://my_bucket/file2.csv.gz
curl http://internet.address/csvs/file3.csv.gz | gsutil cp - gs://my_bucket/file3.csv.gz
curl http://internet.address/csvs/file4.csv.gz | gsutil cp - gs://my_bucket/file4.csv.gz
curl http://internet.address/csvs/file5.csv.gz | gsutil cp - gs://my_bucket/file5.csv.gz
curl http://internet.address/csvs/file6.csv.gz | gsutil cp - gs://my_bucket/file6.csv.gz
After that, you simply need to run the following command
parallel --job 2 < downloads
This command will run up to 2 parallel curl commands until all the commands in the file have been executed.
Another improvement you can apply to your routine is to use gsutil mv instead gsutil cp, mv command will delete the file after success upload, this can help you to save space on your hard drive.
If you have the MD5 hashes of each CSV file, you could use the Storage Transfer Service, which supports copying a list of files (that must be publicly accessible via HTTP[S] URLs) to your desired GCS bucket. See the Transfer Service docs on URL lists.
I am trying create an storage account from terraform, and use some of its access keys to create a blob container.
My terraform configuration is given from a bash file, so, one of the most important steps here are the following:
customer_prefix=4pd
customer_environment=dev
RESOURCE_GROUP_NAME=$customer_prefix
az group create --name $RESOURCE_GROUP_NAME --location westeurope
# Creating Storage account
STORAGE_ACCOUNT_NAME=4pdterraformstates
az storage account create --resource-group $RESOURCE_GROUP_NAME --name $STORAGE_ACCOUNT_NAME --sku Standard_LRS --encryption-services blob
# We are getting the storage account key to access to it when we need to store the terraform .tf production state file
ACCOUNT_KEY=$(az storage account keys list --resource-group $RESOURCE_GROUP_NAME --account-name $STORAGE_ACCOUNT_NAME --query [0].value -o tsv)
# Creating a blob container
CONTAINER_NAME=4pd-tfstate
az storage container create --name $CONTAINER_NAME --account-name $STORAGE_ACCOUNT_NAME --account-key $ACCOUNT_KEY
source ../terraform_apply_template.sh
I am sending those az cli commands from a .bash file and so, I am sending other TF_VAR_azurerm ... variables there
Finally when I execute the bash first bash file, this call to the terraform_apply_template.sh which create the plan, and it's applied. Is the following:
#!/bin/bash
#Terminates script execution after first failed (returned non-zero exit code) command and treat unset variables as errors.
set -ue
terraform init -backend-config=$statefile -backend-config=$storage_access_key -backend-config=$storage_account_name
#TF:
function tf_plan {
terraform plan -out=$outfile
}
case "${1-}" in
apply)
tf_plan && terraform apply $outfile
;;
apply-saved)
if [ ! -f $outfile ]; then tf_plan; fi
terraform apply $outfile
;;
*)
tf_plan
echo ""
echo "Not applying changes. Call one of the following to apply changes:"
echo " - '$0 apply': prepares and applies new plan"
echo " - '$0 apply-saved': applies saved plan ($outfile)"
;;
esac
But my output is the following, the azurerm backend is initialized,the storage account named 4pdterraformstates is created, and also the 4pd-tfstate blob container,
but in the practice this action is not effective, I get the following output:
Initializing the backend...
Successfully configured the backend "azurerm"! Terraform will automatically
use this backend unless the backend configuration changes.
Error: Failed to get existing workspaces: storage: service returned error: StatusCode=404, ErrorCode=ContainerNotFound, ErrorMessage=The specified container does not exist.
RequestId:2db5df4e-f01e-014c-369d-272246000000
Time:2019-06-20T19:21:01.6931098Z, RequestInitiated=Thu, 20 Jun 2019 19:21:01 GMT, RequestId=2db5df4e-f01e-014c-369d-272246000000, API Version=2016-05-31, QueryParameterName=, QueryParameterValue=
Looking for a similar behavior, I have found this issue in the azurerm provider terraform repository
And also according to this issue created directly in terraform repository It's looks like a network operational error ...
But the strange is that it's was fixed ..
I am using the terraform v0.12.1 version
⟩ terraform version
Terraform v0.12.2
According to #Gaurav-Mantri answer, is necessary wait until the storage account to be provisioned in order to continue wit the other task related with the storage account itself.
Creation of a storage account is an asynchronous process. When you execute az storage account create to create a storage account, request is sent to Azure and you get an accepted response back (if everything went well).
The whole process to create (provision) a storage account takes some time (maximum 1 minute in my experience) and until the storage account is provisioned, no operations are allowed on that storage account.
How can I include a wait process after of the storage account creation?
It seem that only with the bash sleep command or pausing the bash shell with the read command for some time is not enough..
Creation of a storage account is an asynchronous process. When you execute az storage account create to create a storage account, request is sent to Azure and you get an accepted response back (if everything went well).
The whole process to create (provision) a storage account takes some time (maximum 1 minute in my experience) and until the storage account is provisioned, no operations are allowed on that storage account.
This is the reason you're getting the error because you're immediately trying to perform an operation after sending the creation request.
Not sure how you would do it but what you would need to do is wait for the storage account to be provisioned. You can get the properties of the storage account periodically (say once every 5 seconds) and check for it's provisioning state. Once the provisioning state is successful, you can try to create a container. At that time your request should succeed.
Heroku offers automatic and scheduled backups of your PG database.
https://devcenter.heroku.com/articles/heroku-postgres-data-safety-and-continuous-protection
GBackups will launch a dedicated dyno to take a dump of your database
and upload it to S3
Simple question: Is it possible to upload a scheduled PG backup to one's OWN S3 Bucket? Simply to have control over the backup files and to not be limited in Storage space. Researching this topic did not provide me with an answer if this is possible.
You can do it by using Heroku scheduler and a bash script.
# Set the script to fail fast if there
# is an error or a missing variable
set -eu
set -o pipefail
#!/bin/sh
# Download the latest backup from
# Heroku and gzip it
heroku pg:backups:download --output=/tmp/pg_backup.dump --app $APP_NAME
gzip /tmp/pg_backup.dump
# Encrypt the gzipped backup file
# using GPG passphrase
gpg --yes --batch --passphrase=$PG_BACKUP_PASSWORD -c /tmp/pg_backup.dump.gz
# Remove the plaintext backup file
rm /tmp/pg_backup.dump.gz
# Generate backup filename based
# on the current date
BACKUP_FILE_NAME="heroku-backup-$(date '+%Y-%m-%d_%H.%M').gpg"
# Upload the file to S3 using
# AWS CLI
aws s3 cp /tmp/pg_backup.dump.gz.gpg "s3://${S3_BUCKET_NAME}/${BACKUP_FILE_NAME}"
# Remove the encrypted backup file
rm /tmp/pg_backup.dump.gz.gpg
You can check out this tutorial for detailed step by step explanation.
One option is to create a backup (you can even create a follower database to created it from for performance reasons), then download the backup via stream to your server, and then upload it into your own S3 bucket.
If you wanted a quick Rail app to do this, you can setup https://github.com/kjohnston/pgbackups-archive. It does everything aside from creating a follower database, but if you are not too concerned with performance 24/7, then this should do fine. I don't know why Heroku doesn't offer storage to your own S3 buckets, as they store them on S3 themselves.
Here is a buildpack for doing this on a regular schedule. It hasn't been updated in a bit, but you could easily update / adapt it as needed.
I have wrote this shell script that download archives from a url list, decompresses them and finally moves them in a Cloud Storage bucket.
#!/bin/bash
# declare STRING variable
for iurl in $(cat ./html-rdfa.list); do
filename=$(basename "$iurl")
file="${filename%.*}"
if gsutil ls gs://rdfa/$file; then
echo "yes"
else
wget $iurl
gunzip $filename
gsutil cp -n $file gs://rdfa
rm $file
sleep 2
fi
done
html-rdfa.list contains the url list. The instance is created using the debian 7 image provided by gooogle.
The script run correctly for the first 5 or 6 files, but then the instance freezes and i have to delete the instance. The ram or the disk of the instance are not full when it freezes.
I think the problem is caused by the command gsutil cp, but it is strange that CPU load is practically 0 and also the RAM is free but it is impossibilo to use the instance without restarting them.
Are you writing the temporary files to the default 10GB root disk? If so, you may be running into the Persistent Disk throughput caps. To see if this is the case, create a new Persistent Disk, then mount it as a data disk and use that disk for the temporary files. Consider starting with ~200GB disk and see if that is enough throughput for your workload. Also, see the docs on Persistent Disk performance.
I am trying to launch an instance on Amazon EC2. I have researched this problem extensively, but I have not found any helpful information.
When I run the command hadoop-ec2 launch-cluster mycluster 2, I receive the following error message:
Starting master with AMI.
Required parameter 'AMI' missing (-h for usage)
I have entered my AWS key, AWS secret key, AWS key pairs, etc. I am using hadoop-1.0.4. I am using the default S3 bucket (hadoop-images), but I have tried many other AMIs and I always get the same error message.
Has anybody experience this problem before?
The basic issue is that the search for images the launch-hadoop-master script performs is not returning any results. The most likely cause of this due to the different AMIs that are available in different regions (but it could be due to any changes you've made to S3_BUCKET and HADOOP_VERSION in hadoop-ec2-env.sh).
From the launch-hadoop-master script:
# Finding Hadoop image
AMI_IMAGE=`ec2-describe-images -a | grep $S3_BUCKET
| grep $HADOOP_VERSION
| grep $ARCH
| grep available
| awk '{print $2}'`
# Start a master
echo "Starting master with AMI $AMI_IMAGE"
So, it appears that AMI_IMAGE is not being set to a valid image and thus the search for AMIs that match the various grep filters is failing (the defaults for the Hadoop 1.0.4 distribution are S3_BUCKET is hadoop-images, HADOOP_VERSION is 0.19.0 and ARCH is x86 if you're using m1.small instances). If you search the public AMIs in the US-West-2 region, you'll see that there aren't many Hadoop images, but if you search the public AMIs in the US-East-1 region, you'll see that there are quite a few. Thus, one way around this issue is to work in the US-East-1 region (this is simplest) or, alternatively, set EC2_URL in your login script via export EC2_URL=https://ec2.us-east-1.amazonaws.com but now you need make sure you put your keys in this region from the AWS console.
If you did indeed change HADOOP_VERSION to 1.0.4, I'll note that
ec2-describe-images -a | grep hadoop-images
| grep "1.0.4"'
| grep x86
| grep available
doesn't return any images in the US-East-1 region. Note that the version (HADOOP_VERSION) of the Hadoop distribution that you are running the hadoop-ec2 command from does not need to be the same as the version of Hadoop that the images will be running.
Lastly, as a blunt fix, you could find the AMI that you want to use, and force set AMI_IMAGE to the image name in the launch-hadoop-master and launch-hadoop-cluster scripts.