Right now when client download file from my site, I'm:
Downloading file from google cloud bucket to server (GCP download file, GCP streaming download)
Saving downloaded file to a Ruby Tempfile
sending Tempfile to enduser using Rails 5 send_file
I would like to skip step 2, to somehow transfer/stream file from google cloud to enduser without the file being saved at my server- is that possible?
Note the google bucket is private.
Code I'm currently using:
# 1 getting file from gcp:
storage = Google::Cloud::Storage.new
bucket = storage.bucket bucket_name, skip_lookup: true
gcp_file = bucket.file file_name
# 2a creates tempfile
temp_file = Tempfile.new('name')
temp_file_path = temp_file.path
# 2b populate tempfile with gcp file content:
gcp_file.download temp_file_path
# 3 sending tempfile to user
send_file(temp_file, type: file_mime_type, filename: 'filename.png')
What I would like:
# 1 getting file from gcp:
storage = Google::Cloud::Storage.new
bucket = storage.bucket bucket_name, skip_lookup: true
gcp_file = bucket.file file_name
# 3 sending/streaming file from google cloud to client:
send_file(gcp_file.download, type: file_mime_type, filename: 'filename.png')
Since making your objects or your bucket publicly readable or accessible is not an option for your project, the best option that I could suggest is using signed URLs so that you can still have control over your objects or bucket and also giving users sufficient permission to perform specific actions like download objects in your GCS bucket.
Related
My objective is to create a csv file that lists all azure databricks workspaces and in particular has the workspace id.
I have been able to retrieve all details as json using the CLI:
az rest -m get --header "Accept=application/json" -u 'https://management.azure.com/subscriptions/{subscriptionId}/providers/Microsoft.Databricks/workspaces?api-version=2018-04-01' > workspaces.json
How can I retrieve the same information using azure resource graph?
If you prefer to work with the workspace list api that returns json, here is one approach for post processing the data (in my case I ran this from a jupyter notebook):
import json
pd.set_option('display.max_columns', None)
pd.set_option('display.max_colwidth', None)
# json from https://learn.microsoft.com/en-us/rest/api/databricks/workspaces/list-by-subscription?tabs=HTTP&tryIt=true&source=docs#code-try-0
# E.g.
# az rest -m get --header "Accept=application/json" -u 'https://management.azure.com/subscriptions/{subscriptionId}/providers/Microsoft.Databricks/workspaces?api-version=2018-04-01' > workspaces.json
pdf = pd.read_json('./workspaces.json')
# flatten the nested json
pdf_flat = pd.json_normalize(json.loads(pdf.to_json(orient="records")))
# drop columns with name '*.type'
pdf_flat.drop(pdf_flat.columns[pdf_flat.columns.str.endswith('.type')], axis=1, inplace=True)
# drop rows without a workspaceId
pdf_flat = pdf_flat[ ~pdf_flat['value.properties.workspaceId'].isna() ]
# drop unwanted columns
pdf_flat.drop(columns=[
'value.properties.parameters.enableFedRampCertification.value',
'value.properties.parameters.enableNoPublicIp.value',
'value.properties.parameters.natGatewayName.value',
'value.properties.parameters.prepareEncryption.value',
'value.properties.parameters.publicIpName.value',
'value.properties.parameters.relayNamespaceName.value',
'value.properties.parameters.requireInfrastructureEncryption.value',
'value.properties.parameters.resourceTags.value.databricks-environment',
'value.properties.parameters.storageAccountName.value',
'value.properties.parameters.storageAccountSkuName.value',
'value.properties.parameters.vnetAddressPrefix.value',
], inplace=True)
pdf_flat
I was able to retrieve the information I needed by:
Searching for databricks resources in the Azure portal:
From there I could click Open Query to use the Azure Resource Graph Explorer and write a query to extract the information I need:
I ended up using the following query:
// Run query to see results.
where type == "microsoft.databricks/workspaces"
| project id,properties.workspaceId,name,tenantId,type,resourceGroup,location,subscriptionId,kind,tags
I have sas url at directory level and want to use to read contents of directory instead of using connection string
Follow below Syntax:
Create mount
dbutils.fs.mount(
source = "wasbs://<container_name>#<storage_account_name>.blob.core.windows.net/",
mount_point = "/mnt/t123",
extra_configs = {"fs.azure.sas.<container_name>.<storage_account_name>.blob.core.windows.net":"Your_SAS_token"})
Read csv file
file_location ="wasbs://<container_name>#<storage_account_name>.blob.core.windows.net/filename.csv"
df = spark.read.format("csv").option("inferSchema", "true").option("header", "true").option("delimiter",",").load(file_location)
display(df)
Reference:
Reading and Writing data in Azure Data Lake Storage Gen 2 with Azure Databricks by Ryan Kennedy
Mount ADLS Gen2 storage account provided by Microsoft
I would like to know there is methods to export user lists and passwords in synology NAS device
Local
See user Greenstream's answer in the Synology Forum:
Download config backup file from the Synology
Change file extension from .cfg to .gzip
Unzip the file using 7-Zip or another utility that can extract from gzip archives
Download and install DB Browser for SQL LIte from http://sqlitebrowser.org/
Open the extracted ‘_Syno_ConfBkp.db’ file in DB Browser for SQL Lite
From the top menu bar select File, then Export, then Export as csv
In the export dialog select the table confbkp_user_tb
In the options
a. select column names in first line, Field separator character ,
b. Quote character "
c. New line characters ‘Windows: CR+LF(\r\n)’
Save the file to your desktop and open in Excel
LDAP
Based on ldap2csv.py and How to retrieve all the attributes of LDAP database to determine the available attributes, using python-ldap:
#!/usr/bin/python
import ldap
host = 'ldap://[ip]:389' # [ip]: The ip/name of the NAS, using the default port
dn = 'uid=[uid],cn=[cn],dc=[dc]' # LDAP Server Settings: Authentication Information / Bind DN
pw = '[password]' # LDAP Server Settings: Password
base_dn = 'dc=[dc]' # LDAP Server Settings: Authentication Information / Base DN
filter = '(uid=*)' # Get all users
attrs = ['cn', 'uid', 'uidNumber', 'gidNumber', 'homeDirectory', 'userPassword', 'loginShell', 'gecos', 'description']
con = ldap.initialize(host)
con.simple_bind_s(dn, pw)
res = con.search_s(base_dn, ldap.SCOPE_SUBTREE, filter, attrs)
con.unbind()
print(res)
The used ports can be found here.
Script for daily Uploading files to S3 with specific aws-cli profile permission, after checking the correct user profile upload backup-files to s3
import boto3
import os
s3 = boto3.resource('s3')
def upload_to_s3(filepath, bucketname, prefix):
for filename in os.listdir(filepath):
s3.meta.client.upload_file(filepath + filename, bucketname, prefix + filename)
if __name__ == '__main__':
local_file_path = ''
bucket_name = ''
prefix = ''
upload_to_s3(local_file_path, bucket_name, prefix)
You can use something like this to upload all files inside your local backup directory to a bucket with a prefix of your choice.
Since you are on Ubuntu you can use cron job to schedule this script to run daily/weekdays. You can see a simple tutorial for this here.
I have to move large content, which I don't want to put into memory from AWS S3 to FTP with Laravel's filesystem.
I know how to stream local content to S3, but haven't found a solution yet from S3 to FTP.
The closest I found was this, but I'm stuck in adapting it for my case.
Here is what's missing in my code (??):
$inputStream = Storage::disk('s3')->getDriver()->??
$destination = Storage::disk('ftp')->getDriver()->??
Storage:disk('ftp')->getDriver()->putStream($destination, $inputStream);
I think I found a solution:
$input = Storage::disk('s3')->getDriver();
$output = Storage::disk('ftp')->getDriver();
$output->writeStream($ftp_file_path, $input->readStream($s3_file_path));