How to execute the existing BigQuery parametrized view with passing parameters (#param) without sending the view query as well?
I would like to execute existing view with placeholders for my custom parameters.
My minimalistic view saved in BigQuery in project.dataset.view:
select * from `project.dataset.table`
where column = #x
My bq cmd looks like:
bq --location=EU query --use_legacy_sql=False \
--parameter='x:INT64:100' \
--destination_table="dev.view_result" "[HERE IS THE SAME VIEW COPIED]"
Like that I am not executing the existing view, I am creating a new select query (new view).
Is it possible to call it somehow like:
bq --location=EU query --use_legacy_sql=False \
--parameter='x:INT64:100' \
--destination_table="dataset.view_result"
--view=[VIEW_URL like dataset.view]
Views with parameters are not supported in BigQuery yet - see relatedfeature request
Related
My goal is to create a parameterized view in Impala so users can easily change values in a query. If I run below query, for example, in HUE, is possible to introduce a value.
SELECT * FROM customers WHERE customer_id = ${id}
But I would like to create a view as follows, that when you run it, it asks you for the value you want to search. But this way is not working:
CREATE VIEW test AS SELECT * FROM customers WHERE customer_id = ${id}
Someone know if it is possible?
Many thanks
When you creating a view, it takes the actual variable's value.
Two workarounds exist:
Create a real table where you will store/update the parameter.
CREATE VIEW test AS SELECT * FROM customers JOIN id_table ON customer_id = id_tableid
Pass a parameter into the view with the help of the user-defined function(UDF). Probably you will need two UDFs set and get. Set UDF will write UDF on HDFS and Get UDF will read the variable from HDFS.
Two above mentioned workarounds work but not ideal. My suggestion is to use Hive for parametrized view creation. You can create a GenericUDF via which you can access hive configuration and read the variable and perform filtration. You can't use it for Impala.
SELECT Generic_UDF(array(customer_id)) FROM customers
GenericUDFs has method configure you can use it to read the hive variable:
public void configure(MapredContext mapredContext) {
String name = mapredContext.getJobConf().get("name");
}
You could do the opposite, e.g. parameterize the query on the view instead
I am trying to create a BI Publisher data model which runs the Oracle query below -
SELECT *
FROM audit_YYYYMM
(this should be the YYYYMM of the current date)
How do I setup a parameter default value within the datamodel to grab the YYYYMM from the SYSDATE?
How do I append this parameter within the data set SQL Query?
I tried SELECT * FROM audit_:Month_YYYYMM
(where I had a string parameter called Month_YYYMM)
This did not work.
You are going to have to use something like EXECUTE_IMMEDIATE. And you may have to make a separate PL/SQL package to launch rather than use the built in data definition stuff.
https://docs.oracle.com/cd/B19306_01/appdev.102/b14261/executeimmediate_statement.htm
I have written a program using pyspark to connect to oracle database and fetch data. Below command works fine and returns the contents of the table:
sqlContext.read.format("jdbc")
.option("url","jdbc:oracle:thin:user/password#dbserver:port/dbname")
.option("dbtable","SCHEMA.TABLE")
.option("driver","oracle.jdbc.driver.OracleDriver")
.load().show()
Now I do not want to load the entire table data. I want to load selected records. Can I specify select query as part of this command? If yes how?
Note: I can use dataframe and execute select query on the top of it but I do not want to do it. Please help!!
You can use subquery in dbtable option
.option("dbtable", "(SELECT * FROM tableName) AS tmp where x = 1")
Here is similar question, but about MySQL
In general, the optimizer SHOULD be able to push down any relevant select and where elements so if you now do df.select("a","b","c").where("d<10") then in general this should be pushed down to oracle. You can check it by doing df.explain(true) on the final dataframe.
I have set of oracle database view creation script files, which needs schema name as dynamic parameter while getting executed bt maven-sql-plugin. How do I pass this parameter using "sql-maven-plugin".
This is an example :
CREATE OR REPLACE VIEW MY_VIEW AS SELECT * FROM FROM &SCHEMA_NAME.MY_TABLE.
Here I want to pass &SCHEMA_NAME from sql-plugin, is it possible to do?
In Hue 3.6 I'm trying to use Sqoop 2 import feature for a custom SQL query from an Oracle database.
I'm pasting the SQL query in the field "Table SQL statement" for example: "SELECT x,y,z FROM myschema.mytable where a > 100"
I'm getting the following exception "SQL statement must contain placeholder for auto generated conditions - ${CONDITIONS}"
I need to understand how to use ${CONDITIONS} when running the Hue UI?
Is there any demo, documentation available on using Sqoop 2
from Hue with some advanced features like using custom SQL etc.?
I'm using Cloudera Sandbox that is using Hue 3.6.
You can specify condition like following :
"SELECT x,y,z FROM myschema.mytable where a > 100 AND ${CONDITIONS}"
In sqoop2 code it searches for ' ${CONDITIONS} ' token and throws exception if it doesn't find one.
You can read the DOCS Free-Form Queries on SQOOP. Be careful when you use double quotes "", you have to add the \$CONDITIONS. If you query with single quotes you only need to add $CONDITIONS without backslash.