I am trying to skip multiple line queries using bandit in my python query. I have tried to use #nosec but still there is bandit issue showing
Example:
"""#nosec""" ;Query = f"""Select username,id,email_id,address from User where username ='John'"""
Related
I'd like to define a custom variable called task_warehouse to be used in a Snowflake change script, like this:
CREATE OR REPLACE TASK dummy_task
WAREHOUSE = &task_warehouse
SCHEDULE = '1 minute'
AS
SELECT 1;
The task warehouse is different from &warehouse. I want the task warehouse to vary depending on the DATABASE being deployed to. I don't want to change the warehouse that's running the deploy script.
I tried adding ;task_warehouse=<WAREHOUSE_NAME> to the Snowflake connection string, but that didn't seem to do the trick.
I get this error when trying to deploy:
Variable task_warehouse is not defined
Does anyone know how to define a custom variable to be used by Sqitch, similar to how &warehouse is used in the following?
USE WAREHOUSE &warehouse;
One of my colleagues was able to figure it out. It says under the "variables" bullet towards the bottom of https://sqitch.org/docs/manual/sqitch-configuration/.
Example from that page:
sqitch deploy --set $key=$val -s $key2=$val2
Doing the following worked for me:
sqitch deploy -s task_warehouse=PROD_TASK_WAREHOUSE
I am running a simple script using cloud shell to get the selected records from a given table.
#!/bin/bash
bq query --format=pretty --use_legacy_sql=false --project_id='test-project' " \
SELECT collecttime, count(distinct(CELLID)) countIngest \
FROM \`test-project.TEST_INGEST.my_test_table\` \
WHERE DATE(COLLECTTIME) >= '2020-12-01' \
group by collecttime order by collecttime;">>gsam_output.csv
If I run this query on Google console it fetched 500 records, a 100 each page and I can move from one page to another to see full set of records. I can also save results to some csv with full set of 500 records.
But when I run same query in shell or cloud SDK environment. It is only writing or displaying those records which gets displayed on first page in console. Is there any cap to write or display records via command line or shell?
Below I run in cloud SDK
bq query --format=csv --use_legacy_sql=false "SELECT collecttime, count(distinct(CELLID)) countIngest FROM TEST_INGEST.my_test_table WHERE DATE(COLLECTTIME) >= '2020-12-01' group by collecttime order by collecttime;">>gsam_output.csv
You can try to set --max_rows or -n option when using bq query command. Here is the link to documentation describing this option.
I tried to run a similar query with this option set to 500, and it fetched 500 records at once.
bq query -n 500 --format=pretty --use_legacy_sql=false "select * from bigquery-public-data.samples.shakespeare;"
Alright - so when you are looking at BQ output data, it can't paginate results for you. Which is what that is doing, which is why you only get the 1st page (I think)
What you want to try and do is export the data.
You write your data to a temp table in bq, and then export it - using directions here: https://cloud.google.com/bigquery/docs/exporting-data#bq
or
https://cloud.google.com/bigquery/docs/exporting-data#exporting_data_stored_in_bigquery
You can also write a full output of a query to a csv using the console:
https://cloud.google.com/bigquery/docs/writing-results
Also, try specifying your delimiter, and ensuring you don't have any nested information. CSV won't work for nested assets in bq, you need to jump to json.
The issue is with JMeter parameterization Logic.
I have a csv file with user id and pwd - which is fine
Another CSV has 10 records, one for each user ID. How do I map these records to user ids?
Csv1:
UserID, Pwd
TestUser1, Password1
TestUser2, Password1
CSV2:
Record
Rec1forUser1
Rec2forUser1
Rec3forUser1
Rec1forUser2
Rec2forUser2
and so on..
How do I arrange the mapping when multiple users (say 2) are going to run for a Loop of 3 records each.
It's not possible with standard JMeter equipment, what you're asking for.
I would suggest to use the User data set as a driver, and for the record read at each iteration then split second dataset to separate files (1 per user), then ingest 'em in the custom JSR223 element - actually, the file operations toolset Groovy equipped with makes that task pretty easy, smth. like:
new File(path + userID + "_someSuffix.csv").eachLine { line ->
line.split(',').each { field ->
... use fields here somehow ...
}
}
In case you have to have multiple iterations per user, you probably would have to read the Users dataset that way as well.
And finally, yet another way is to denormalize/flatten your dataset: include (repetitive) User records into second dataset along with data belonging to user, like:
TestUser1, Password1, Rec1forUser1
TestUser1, Password1, Rec2forUser1
TestUser1, Password1, Rec3forUser1
That may not look pretty, but would do the job.
I am trying to make Pentaho Kettle job which can do below ETL process:
Want to match Keywords from keywords.csv into the sentence column in data.csv and want to produce below output where all sentence won't be included where the keyword matches into the sentence.
I was able to do it in Talend but not able to achieve same in Pentaho Kettle. I tried to use Filter but it's not working.
I want to use keywords.csv as Input not fixed input in ETL job itself.
keywords.csv
apple
banana
orange
green apple
pineapple
data.csv
id,sentence
1,This apple is nice
2,That is my truck
3,This building is beautiful
4,Apple and pineapple are good for health
5,that orange is fresh
output.csv
2,That is my truck
3,This building is beautiful
I think you could accomplish this using the Fuzzy Match step in Pentaho. Anything that gets scored >0 could be filtered out.
You should be able to do this with a Java Script step where you load the keywords.csv data via the loadFileContent function and then loop through it's values.
Ultimately the JavaScript function will write a boolean variable (let's call it "include") to each row.
The next step is a 'Filter row' where you skip each row that has include=false.
I have this situation. I have a csv file with some information, I have to complete this information with a database table registers ad write a new file with these info.
I guess that I should use a MultipleReader implementation, one to read my file and other to read my database (Some like this example: https://bigzidane.wordpress.com/2016/09/15/spring-batch-multiple-sources-as-input/) But I need pass conditions to te query in relation a the current item being processed.
Any way, If it is possible, I need configure a query data in my reader2 with info getted in my reader1. How I could make this?
This a little resume of my problem:
Input File (Reader1)
Id;data1;data2;data3
Database (Reader2)
Id|data4;data5;data6
Output File
Id;data1;data2;data3;data4;data5;data6
Sorry My english. Any link to articles or docs is good.
This is a common pattern known as the "driving query pattern" and is described in the Common patterns section of the reference documentation.
You can use a FlatFileItemReader to read your input file and an item processor to query the database for the current item and enrich it with additional data.
Another idea is to load your flat file in a staging table in the database and use a database item reader to join data.