Related
I wrote a script to import stock data from a csv file stored in Google Drive to an existing google sheet.
In one function I'm doing this for multiple csv files. Unfortunately I get "Exceeded maximum execution time" sometimes, but not all the time.
Do you have an idea how I can boost performance on this:
//++++++++++++++ SPY +++++++++++++++++++
var file = DriveApp.getFilesByName("SPY.csv").next();
var csvData = Utilities.parseCsv(file.getBlob().getDataAsString());
//Create new temporary sheet
var activeSpreadsheet = SpreadsheetApp.getActiveSpreadsheet();
var yourNewSheet = activeSpreadsheet.getSheetByName("SPY-Import");
if (yourNewSheet != null) {
activeSpreadsheet.deleteSheet(yourNewSheet);
}
yourNewSheet = activeSpreadsheet.insertSheet();
yourNewSheet.setName("SPY-Import");
//Import
var sheet = SpreadsheetApp.getActiveSheet();
sheet.getRange(1, 1, csvData.length, csvData[0].length).setValues(csvData);
//Copy from temporary sheet to destination
var spreadsheet = SpreadsheetApp.getActive();
spreadsheet.getRange('A:B').activate();
spreadsheet.setActiveSheet(spreadsheet.getSheetByName('SPY'), true);
spreadsheet.getRange('A2').activate();
spreadsheet.getRange('\'SPY-Import\'!A:B').copyTo(spreadsheet.getActiveRange(),
SpreadsheetApp.CopyPasteType.PASTE_NORMAL, false);
//Delete temporary sheet
// Get Spreadsheet Object
var spreadsheet = SpreadsheetApp.getActiveSpreadsheet();
// Get target sheet object
var sheet = spreadsheet.getSheetByName("SPY-Import");
// Delete
spreadsheet.deleteSheet(sheet);
Thanks in advance!
I believe your situation and goal as follows.
You have several CSV files like SPY.csv.
Your Spreadsheet has the several sheet corresponding to each CSV file like SPY.
You want to put the values from the CSV data to the Spreadsheet.
You want to put the values of the column "A" and "B" of the CSV data.
In your current situation, you copied the script in your question several times and run them by changing the csv filename and sheet name.
You want to reduce the process cost of your script. I understood your goal like this.
Modification points:
SpreadsheetApp.getActiveSpreadsheet() is used several times. And, activate() is used several times.
I think that in your case, SpreadsheetApp.getActiveSpreadsheet() can be declared one time, and activate() is not required to be used.
In order to copy the CSV data to Spreadsheet, the CSV data is put to a temporal sheet and the required values are copied to the destination sheet.
In this case, I think that the CSV data is directly put to the destination sheet by processing on the array.
I think that above points lead to the reduction of process cost. When above points are reflected to your script, it becomes as follows.
Modified script:
Please copy and paste the following script and prepare the variable of obj. When you run the script, the CSV data is retrieved and processed, and then, the values are put to the Spreadsheet.
function myFunction() {
var obj = [
{filename: "SPY.csv", sheetname: "SPY"},
{filename: "###.csv", sheetname: "###"},
,
,
,
];
var ss = SpreadsheetApp.getActiveSpreadsheet();
obj.forEach(({filename, sheetname}) => {
var file = DriveApp.getFilesByName(filename);
if (file.hasNext()) {
var sheet = ss.getSheetByName(sheetname);
if (sheet) {
// sheet.clearContents(); // Is this requierd in your situation?
var csv = DriveApp.getFileById(file.next().getId()).getBlob().getDataAsString();
var values = Utilities.parseCsv(csv).map(([a, b]) => [a, b]);
sheet.getRange(2, 1, values.length, 2).setValues(values);
}
}
});
}
Note:
Please use this script with enabling V8
I'm not sure about your CSV data. So when Utilities.parseCsv(csv) cannot be used, please use the delimiter.
In this modification, Spreadsheet service is used. If above modified script occurs the same error of Exceeded maximum execution time, please tell me. At that time, I would like to propose the sample script using Sheets API.
References:
Spreadsheet Service
parseCsv(csv)
I created a Power BI Custom Data Connector, the idea is to be able to connect to SSRS Dataset using this Custom Data Connector I was able to do it but the resulting formatted json is different from what i expect.
Here's the result when I open the Custom Connector in Power BI, I expected a properly formatted table but the result is not.
Columns are List of Record contain the Column Names and Type
While the Row is a List of List containing the values for CustomerID and CustomerName.
Here's my code.
section Test.PQ.SSRS_Connector;
[DataSource.Kind="Asia.PQ.SSRS_Connector", Publish="Test.PQ.SSRS_Connector.Publish"]
shared Test.PQ.SSRS_Connector.Feed = Value.ReplaceType(SSRSConImpl, type function (url as Uri.Type) as any);
DefaultRequestHeaders = [
#"Accept" = "application/json;odata.metadata=minimal",
#"OData-MaxVersion" = "4.0"
];
SSRSConImpl = (url as text) =>
let
body= "",
source = Web.Contents(url, [ Headers = DefaultRequestHeaders, Content=Text.ToBinary(body)]),
json = Json.Document(source)
in
json;
Posting some sample JSON would be helpful, but based on the screenshots it seems like continuing your function as below may work:
// ... Your function code
json = Json.Document(source),
toTable = Table.FromRows(json[Rows], {"CustomerID", "CustomerName"}) // If there are more columns, consider extracting names dynamically from json[Columns]
// .... Any remaining code
Code is untested.
Using JMeter-Bean shell sampler, I have extracted and split(with ',' delimiter) dynamic string. Currently I got stuck how this split values can be inserted to different columns of a database table.
Here is the code snippet which prints all the values after split. After splitting the string, each value will store into an array. You can retrieve by specifying the array position.
You can use the variable name e.g. aftersplit[0], aftersplit[1], and so on in the insert query.
String mystring = "here is my, dynamic, random, and unique string";
String[] aftersplit = mystring.split(",");
System.out.println(aftersplit[0]);
System.out.println(aftersplit[1]);
System.out.println(aftersplit[2]);
//To print all the values after splitting
for (int i=0; i < aftersplit.length; i++){
System.out.println(aftersplit[i]);
}
I would recommend the following approach: store your dynamic values into JMeter Variables having number postfix, example code:
String source = "foo,bar,baz";
int counter = 1;
for (String token : source.split(",")) {
vars.put("token_" + counter, token);
counter++;
}
It produces the following JMeter Variables:
token_1=foo
token_2=bar
token_3=baz
Then add ForEach Controller to iterate the generated variables and JDBC Request sampler as a child of the ForEach Controller to insert them into the database. See The Real Secret to Building a Database Test Plan With JMeter to learn how to establish database connection and execute arbitrary SQL queries using JMeter
I have a data set that looks like this after a GROUP operation :
input = key1|{(a1,b1,c1),(a2,b2,c2)}
key2|{(a3,b3,c3),(a4,b4,c4),(a5,b5,c5)}
I need to traverse the above to generate final output like this :
<KEY>key1</KEY>|
<VALUES><VALUE><VALUE1>a1</VALUE1>VALUE2>b1</VALUE2>VALUE3>c1</VALUE3></VALUE><VALUE><VALUE1>a2</VALUE1><VALUE2>b2</VALUE2><VALUE3>c2</VALUE3> </VALUE></VALUES>
<KEY>key2</KEY>| ...
I have tried to use FLATTEN and CONCAT to achieve this result in the below manner:
A = FOREACH input GENERATE key, FLATTEN(input);
output = FOREACH A GENERATE CONCAT('<KEY>',CONCAT(input.key,'</KEY>')),
CONCAT('<VALUE>',''),
CONCAT('<VALUE1>',CONCAT(input.col1,'</VALUE1>')
...
But this does not give the desired output. Fairly new to pig, so don't know if this is possible.
If you FLATTEN your bag than you'll ended up as many new 'rows' as many elements you had in the bag:
key1|(a1,b1,c1)
key1|(a2,b2,c2)
If I understand your problem correctly you want this:
Use the BagToTuple built in function.
Than you'll get
key1|(a1,b1,c1,a2,b2,c2)
After this you can format your data with e.g. a UDF
I have several CSV files in a HDFS folder which I load to a relation with:
source = LOAD '$data' USING PigStorage(','); --the $data is a passed as a parameter to the pig command.
When I dump it, the structure of the source relation is as follows: (note that the data is text qualified but I will deal with that using the REPLACE function)
("HEADER","20110118","20101218","20110118","T00002")
("0000000000000000035412","20110107","2699","D","20110107","2315.","","","","","","C")
("0000000000000000035412","20110107","2699","D","20110107","246..","162","74","","","","B")
<.... more records ....>
("HEADER","20110224","20110109","20110224","T00002")
("0000000000000000035412","20110121","2028","D","20110121","a6c3.","","","","","R","P")
("0000000000000000035412","20110217","2619","D","20110217","a6c3.","","","","","R","P")
<.... more records ....>
So each file has a header which provides some information about the data set that follows it such as the provider of the data and the date range it covers.
So now, how can I transform the above structure and create a new relation like the following ?:
{
(HEADER,20110118,20101218,20110118,T00002),{(0000000000000000035412,20110107,2699,D,20110107,2315.,,,,,,C),(0000000000000000035412,20110107,2699,D,20110107,246..,162,74,,,,B),..more tuples..},
(HEADER,20110224,20110109,20110224,T00002),{(0000000000000000035412,20110121,2028,D,20110121,a6c3.,,,,,R,P),(0000000000000000035412,20110217,2619,D,20110217,a6c3.,,,,,R,P),..more tuples..},..more tuples..
}
Where each header tuple is followed by a bag of record tuples belonging to that header ?.
Unfortunately there is no common key field between the header and the detail rows, so I don't think cant use any JOIN operation. ?
I am quite new to Pig and Hadoop and this is one of the first concept projects that I am engaging in.
Hope my question is clear and look forward to some guidance here.
This should get you started.
Code:
Source = LOAD '$data' USING PigStorage(',','-tagFile');
A = SPLIT Source INTO FileHeaders IF $1 == 'HEADER', FileData OTHERWISE;
B = GROUP FileData BY $0;
C = GROUP FileHeaders BY $0;
D = JOIN B BY Group, C BY Group;
...