Load file from Google Drive into Bigquery - google-api

I have an API which takes a loaded file on my website and uploads it with mimeType text/csv to Google Drive via the API.
This creates and works wonders, however, I now need to load that file into Google Bigquery as a permanent table.
I have two ways:
Preferred: Take the uploaded file and ammend the mimeType to Google Sheets - i thought drive would be clever enough to do this with a csv but apparently not - maybe there is a way.
Load the file from text/csv straight into Bigquery - i am receiving invalid URI's
I have my PHP CURL command below:
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://bigquery.googleapis.com/bigquery/v2/projects/xxxxxxx/datasets/xxxxx/tables",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST",
CURLOPT_POSTFIELDS => "{\n \"externalDataConfiguration\": {\n \"sourceUris\": [\n \"https://drive.google.com/file/d/{$id}/\"\n ],\n \"sourceFormat\": \"CSV\",\n \"autodetect\": true\n },\n \"tableReference\": {\n \"tableId\": \"{$user}\"\n }\n}",
CURLOPT_HTTPHEADER => array(
"Accept: */*",
"Accept-Encoding: gzip, deflate",
"Authorization: Bearer",
"Cache-Control: no-cache",
"Connection: keep-alive",
//"Content-Length: 319",
"Content-Type: application/json",
"Host: bigquery.googleapis.com",
"User-Agent: PostmanRuntime/7.17.1",
"cache-control: no-cache"
),
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
echo $response;
}
Getting back the following - using method two - however, first prize would be to getting step 1 working, changing the file type to Google Sheets - I already have an update function setting the Title so if i need to add on to that I can
"Invalid source URI: https://drive.google.com/file/d/1Zt6OMQqE3rj1PJe4HlokQ4LPNfOPBHfB/

From documentation:
Loading data into BigQuery from Google Drive is not currently supported, but you can query data in Google Drive by using an external table.
Follow Querying Google Drive data and once you can query it, you may persist it to BigQuery using query (or your don't have to if external table works for you already):
CREATE TABLE my_dataset.persisted_table_name
AS SELECT * FROM my_external_table_pointing_to_google_drive;

Related

Elastic Cloud: API endpoint - 404 Not Found on search?

Description
I want to query the data in our Elastic Cloud instance using the REST API with authorization via API Key.
Steps taken
I tried using the SQL API as well as the Search API. NB that I would much preferably use the SQL API.
Based on the following curl commands provided in the documentation:
curl -X POST "localhost:9200/_sql?format=txt&pretty" -H 'Content-Type: application/json' -d'
{
"query": "SELECT * FROM library WHERE release_date < \u00272000-01-01\u0027"
}
'
[source]
curl -u elastic:password https://CLUSTER_ID.REGION.PLATFORM.found.io:9243/my_index/my_type -XPOST -d '{
"title": "One", "tags": ["ruby"]
}'
{"_index":"my_index","_type":"my_type","_id":"AV3ZeXsOMOVbmlCACuwj","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"created":true}
[source]
and the documentation about the REST API, I attempted the following:
import base64
import json
import requests
if __name__ == '__main__':
response1 = requests.post(
'https://{redacted-id}.northamerica-northeast1.gcp.elastic-cloud.com:9243/_sql?format=txt',
headers={
"Authorization": base64.standard_b64encode(bytes('{API_KEY_ID}:{API_KEY_KEY}', 'utf-8')),
"Content-Type": 'application/json'
},
data={
"query": """
...
"""
}
)
print('response1')
print(response1)
response2 = requests.get(
'https://{redacted-id}.northamerica-northeast1.gcp.elastic-cloud.com:9243/logs-pubsub/_search',
headers={
"Authorization": base64.standard_b64encode(bytes('{API_KEY_ID}:{API_KEY_KEY}', 'utf-8')),
"Content-Type": 'application/json'
},
data={
"query": {
# ...
}
}
)
print('response2')
print(response2)
But both queries answer with 404 - Not Found.
What did I miss? Am I missing a part of the path like /api/..., /rest/...? Or is this a misdirection from a 403 to a 404 and the issue is the API Key?
Thanks!
The kind folks at ElasticStack provided me with the following answer. There were two issues:
The URL above was pointing to the Kibana instance, not to the ElasticCloud instance
In order to get the appropriate URL:
Navigate to https://cloud.elastic.co/deployments/
Click on your deployment:
Click on Copy Endpoint for the Elasticsearch endpoint:
The Authorization header was malformed
The Authorization header for an ApiKey powered request is the following:
ApiKey {B64_ENCODE('API_KEY_ID:API_KEY_KEY')}
Which can be written in python as:
"ApiKey " + str(base64.standard_b64encode(bytes('API_KEY_ID:API_KEY_KEY', 'utf-8')), 'utf-8')
On a final note, the team also strongly suggested I look at their Free on-demand Elasticsearch training.
Hopefully this will be helpful to whoever passes by here!

how to get list of flowfile from queue in nifi connection?

I want to get each flowfile from queue in connection when the flow was blocked with NiFi REST API.
I find that the functions of /nifi-api/flowfile-queues/{id}/listing-requests can meet my needs.And it response a ListingRequestDTO:
enter image description here
The field named flowFileSummaries can return a array of flowFile.And then I can get each uuid from flowfiles:
{
"uri": "value",
"uuid": "value",
"filename": "value",
"position": 0,
"size": 0,
"queuedDuration": 0,
"lineageDuration": 0,
"clusterNodeId": "value",
"clusterNodeAddress": "value",
"penalized": true
}
BUT,when I use the API and can not find the "FlowFileSummary"
{
"listingRequest": {
"id": "0165122a-e1ac-134e-2c09-92ba9ca93e8b",
"uri": "http://.../nifi-api/flowfile-queues/07a23828-d6f3-1e00-27af-f0428a493507/listing-requests/0165122a-e1ac-134e-2c09-92ba9ca93e8b",
"submissionTime": "09/07/2018 18:46:57.496 CST",
"lastUpdated": "18:46:57 CST",
"percentCompleted": 0,
"finished": false,
"maxResults": 100,
"state": "Waiting for other queue requests to complete",
"queueSize": {
"byteCount": 370689,
"objectCount": 995
},
"sourceRunning": false,
"destinationRunning": false
}
}
So, is there any possible solution to achieve? or any other solution? Thanks!
I don't believe the flow file summaries are available until the listing request is complete. In your example response above, note that finished is false. You can keep querying the API for that listing request until finished is true, then the summaries should be available. Then you can use the /flowfile-queues/{id}/flowfiles/{flowfile-uuid} and /flowfile-queues/{id}/flowfiles/{flowfile-uuid}/content endpoints to get the attributes and content of each flow file.
You do not need to have the listing request to complete. Here are the steps I used successfully to get the contents of flow file
Get the id of the Queue from NiFi UI console
Replace id here : curl -X POST https://myhost:443/nifi-api/flowfile-queues/{id}/listing-requests -H 'Authorization: Bearer ' --compressed --insecure
To get uris of flowfiles in the queue, get the value of 'uri' field from the response of the command in step2 and use it in next curl command, for ex:
For ex: curl -X GET https://myhost:443/nifi-api/flowfile-queues/0f66c88c-225d-3229-b2e1-597d8fba2c09/listing-requests/13802f9c-016a-1000-0000-00004eb033fb -H 'Authorization: Bearer ' --compressed --insecure
To get flow file content, get the value of 'uri' field from the response of the command in step3 and and use it in next curl command, for ex:
curl -X GET https://myhost:443/nifi-api/flowfile-queues/0f66c88c-225d-3229-b2e1-597d8fba2c09/flowfiles/7ccf5c54-7c8d-448a-a124-7f75f5845ec1?clusterNodeId=35a3df65-d7bf-47d2-b162-ea15c3c02c30 -H 'Authorization: ' --compressed --insecure
Note: if there are multiple flow files, you will see see multiple URIs in the response of step 3 curl command

FHIR: ProcedureRequest How to retrieve all data fields of ProcedureRequest?

I am unable to retrieve all the fields, which i post for ProcedureRequest to a FHIR database.
Step 1: Post Request
curl -X POST https://fhir.dstu2.safetylabs.org/api/smartdstu2/open/ProcedureRequest \
-H 'Content-type: application/json+fhir' \
--data #ProcedureRequest.js
Result of POST
{
"resourceType":"OperationOutcome",
"text":{
"status":"generated",
"div":"<div xmlns=\"http://www.w3.org/1999/xhtml\"><h1>Operation Outcome</h1><table border=\"0\"><tr><td style=\"font-weight: bold;\">information</td><td>[]</td><td><pre>Successfully created resource "ProcedureRequest/7660/_history/1" in 7ms</pre></td>\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t</tr>\n\t\t</table>\n\t</div>"
},
"issue":[
{
"severity":"information",
"code":"informational",
"diagnostics":"Successfully created resource \"ProcedureRequest/7660/_history/1\" in 7ms"
}
]
}
This request was successful.
The JSON was validated from JSON validator, and It is also validated from hapi test Server from given link
" https://fhirtest.uhn.ca/resource?serverId=home_21&pretty=false&resource=ProcedureRequest"
Step 2: Retrieve Request
curl -X GET https://fhir.dstu2.safetylabs.org/api/smartdstu2/open/ProcedureRequest/7660
Results of GET
{
"resourceType":"ProcedureRequest",
"id":"7660",
"meta":{
"versionId":"1",
"lastUpdated":"2017-11-18T13:49:23.000+00:00"
},
"text":{
"status":"generated",
"div":"<div xmlns=\"http://www.w3.org/1999/xhtml\"><p><b>Generated Narrative with Details</b></p><p><b>id</b>: subrequest</p><p><b>definition</b>: Protocol for alergies</p><p><b>basedOn</b>: Original Request</p><p><b>replaces</b>: Previous allergy test</p><p><b>requisition</b>: A13848392</p><p><b>status</b>: active</p><p><b>intent</b>: instance-order</p><p><b>priority</b>: routine</p><p><b>code</b>: Peanut IgG <span>(Details : {LOINC code '35542-0' = 'Peanut IgG Ab [Mass/volume] in Serum)</span></p><p><b>subject</b>: <a>Patient/dicom</a></p><p><b>occurrence</b>: 08/05/2013 9:33:27 AM</p><h3>Requesters</h3><table><tr><td>-</td><td><b>Agent</b></td></tr><tr><td>*</td><td><a>Dr. Adam Careful</a></td></tr></table><p><b>performerType</b>: Nurse <span>(Details : {[not stated] code 'null' = 'null', given as 'Qualified nurse'})</span></p><p><b>reasonCode</b>: Check for Peanut Allergy <span>(Details )</span></p><p><b>bodySite</b>: Right arm <span>(Details : {[not stated] code 'null' = 'null', given as 'Right arm'})</span></p></div>"
},
"subject":{
"reference":"Patient/SL88812358"
},
"code":{
"coding":[
{
"system":"http://loinc.org",
"code":"35542-0"
}
],
"text":"Peanut IgG"
},
"bodySite":[
{
"coding":[
{
"display":"Right arm"
}
],
"text":"Right arm"
}
],
"status":"active",
"priority":"routine"
}
Note that Retrieve did not retrieve the following fields
occurrenceTiming
occurrenceDateTime
performerType
reasonCode
requester
Question : How do I request all the fields I posted?
That's because you posted to a DSTU2 server and those elements don't exist in DSTU2 - so the server ignored them (as servers are allowed to do with unrecognized elements). If you post to a DSTU3 server, the elements should be stored - if the server supports them, which most test servers do.

Invalid Request in Api Request.php line 108: Must provide source or customer

i'm using stripe with laravel in my project , when i made payment its give me this error , stripe is not sending me the token that require for transaction
Invalid Request in Api Request.php line 108: Must provide source or customer.
All though this question could mean a lot I had this while using the test card on the stripe getting started page - 4242424242424242 - I made the mistake of creating the token on the site instead of allowing my .js script do the trick .
dont create token on the site
Hence if youre trying to create the token dynamically like this -it would not work. In my case from a post handler.
Charge::create(
array(
"amount" => $cart->totalPrice * 100 ,
"currency" => "usd",
"source" => $request->input('stripeToken') , //wont work
"description" => "Test Charge"
));
--To fix this I had to manually enter the token created on the site like so:
Charge::create(
array(
"amount" => $cart->totalPrice * 100 ,
"currency" => "usd",
"source" => "tok_for-this-transaction-and-card" , //I made this change
"description" => "Test Charge"
));

Use Github API with rest client to create a file

I'm trying to use the Github API to create a file in a repo.
This CURL command does exactly what I want to do:
curl -X PUT -H 'Authorization: token <TOKEN>' -d '{"path": "test.txt", "message": "Test Commit", "committer": {"name": "Kevin Clark", "email": "kevin#kevinclark.ca"}, "content": "bXkgbmV3IGZpbGUgY29udGVudHM="}' https://api.github.com/repos/vernalkick/kevinclark/contents/test.txt
I need to do the same request but using rest_client in ruby, but this returns a 404:
require 'rest_client'
params = {
:path => "test.txt",
:message => "Test Commit",
:committer => {
:name => "Kevin Clark",
:email => "kevin#kevinclark.ca"
},
:content => "bXkgbmV3IGZpbGUgY29udGVudHM=",
:access_token => <TOKEN>
}
RestClient.put "https://api.github.com/repos/vernalkick/kevinclark/contents/test.txt", :params => params
Github's documentation: https://developer.github.com/v3/repos/contents/
So I finally found the solution to my problem!
I needed to create a json string instead of just passing the hash.
RestClient.put "https://api.github.com/repos/vernalkick/kevinclark/contents/test.txt", :params => JSON.generate(params)

Resources