I want to run a hive query in hive command and I want to make it faster, so I ran:
hive:messages> set mapred.job.priority = VERY_HIGH; hive:messages> set
hi = 1;
but I found actually I can set any string to be anything in hive so I wonder is there a way to check all the settings I have made?
To list all the settings available in the current Hive session,
hive> SET;
This will list all the
System Variables
Environment Variables
Hadoop, Hive Configurations (User defined and Default properties)
Hive Variables set using set, define, hivevar.
It is not possible to filter only a specific set of variables. But to get the value of a particular configuration/variable, use the configuration name as the argument to SET
hive> SET zzzz=123;
hive> SET zzzz;
zzzz=123;
Related
My goal is to create a parameterized view in Impala so users can easily change values in a query. If I run below query, for example, in HUE, is possible to introduce a value.
SELECT * FROM customers WHERE customer_id = ${id}
But I would like to create a view as follows, that when you run it, it asks you for the value you want to search. But this way is not working:
CREATE VIEW test AS SELECT * FROM customers WHERE customer_id = ${id}
Someone know if it is possible?
Many thanks
When you creating a view, it takes the actual variable's value.
Two workarounds exist:
Create a real table where you will store/update the parameter.
CREATE VIEW test AS SELECT * FROM customers JOIN id_table ON customer_id = id_tableid
Pass a parameter into the view with the help of the user-defined function(UDF). Probably you will need two UDFs set and get. Set UDF will write UDF on HDFS and Get UDF will read the variable from HDFS.
Two above mentioned workarounds work but not ideal. My suggestion is to use Hive for parametrized view creation. You can create a GenericUDF via which you can access hive configuration and read the variable and perform filtration. You can't use it for Impala.
SELECT Generic_UDF(array(customer_id)) FROM customers
GenericUDFs has method configure you can use it to read the hive variable:
public void configure(MapredContext mapredContext) {
String name = mapredContext.getJobConf().get("name");
}
You could do the opposite, e.g. parameterize the query on the view instead
I am running the following hiveql file with the following content
set mapred.output.compress=true;
set hive.exec.compress.output=true;
set mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec;
set io.compression.codecs=org.apache.hadoop.io.compress.GzipCodec;
INSERT OVERWRITE DIRECTORY '${hivevar:hadoop_temp_output_dir}${hivevar:filepattern}${hivevar:business_date}' select data,'~','~${hivevar:src_sys_file_nm}~${hivevar:outbound_file_nm}~${hivevar:prd_typ_cd}~${hivevar:brnch_cd}~${hivevar:orig_src_sys}~${hivevar:my_src_sys}~${hivevar:extract_ts}~$hivevar:business_date}' from mydb.${hivevar:data_table} where src_sys_file_nm=${hivevar:src_sys_file_nm} and business_date=${hivevar:business_date};
In this example except data column in select query, everything else is in argument variable. Also I am just appending some strings like this -- '~${hivevar:src_sys_file_nm}~${hivevar:outbound_file_nm}~${hivevar:prd_typ_cd}~${hivevar:brnch_cd}~${hivevar:orig_src_sys}~${hivevar:my_src_sys}~${hivevar:extract_ts}~$hivevar:business_date}'-- :
these are not table columns; they are some strings I am passing. This query works very well outside of Hive QL. When I execute the above HQL script, I am getting the following error :
Invalid table alias or column reference 'somestring': (possible column names are: data, business_date, src_sys_file_nm, prd_typ_cd)
Note: business_date, src_sys_file_nm, prd_typ_cd are partitions.
It would be great if any could suggest/point out the mistake I have made here
I have a NLS date format as DD-MON-RR. This gives me the underlying date format as YY while I want to change it to YYYY. I tried using the following query and it ran successfully
DECLARE
v_date DATE := sysdate;
BEGIN
DBMS_OUTPUT.put_line(TO_CHAR(v_date, 'MM/DD/YYYY'));
END;
But that didn't change the default format.
for some context, I am trying to import data from Oracle to Tableau. Unfortunately when I try to export a crosstab from Tableau server it looks at the underlying data rather than whats on the view. This causes the date that I have as 25-Jun-2017 to change to 25-Jun-17 in the excel.
The only workaround I have been able to understand is to change the default format of the underlying/source data which in this case is Oracle DB.
I am using TOAD and am trying to understand how can I change it to possibly DD/MON/RRRR format or something similar with 4 digits in the year column.
Any workaround is also appreciated
As already given in other answers you can set NLS_DATE_FORMAT by ALTER SESSION.
In order to set it only for you local PC, open Registry Editor and navigate to HKLM\SOFTWARE\Wow6432Node\ORACLE\KEY_%ORACLE_HOME_NAME%, resp. HKLM\SOFTWARE\ORACLE\KEY_%ORACLE_HOME_NAME%.
There you can add a String Value NLS_DATE_FORMAT, for example:
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE\KEY_OraClient11g_home1]
"NLS_TIMESTAMP_FORMAT"="YYYY-MM-DD HH24:MI:SSfmXFF3"
"NLS_TIMESTAMP_TZ_FORMAT"="YYYY-MM-DD HH24:MI:SSfmXFF3 fmTZH:TZM"
"NLS_DATE_FORMAT"="YYYY-MM-DD HH24:MI:SS"
[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\ORACLE\KEY_OraClient11g_home1]
"NLS_TIMESTAMP_FORMAT"="YYYY-MM-DD HH24:MI:SSfmXFF3"
"NLS_TIMESTAMP_TZ_FORMAT"="YYYY-MM-DD HH24:MI:SSfmXFF3 fmTZH:TZM"
"NLS_DATE_FORMAT"="YYYY-MM-DD HH24:MI:SS"
You can set NLS_DATE_FORMAT also as an Environment Variable in Windows Settings.
alter session set nls_date_format='DD/MON/RRRR' programmatically in the application or
CREATE OR REPLACE TRIGGER trg_after_logon AFTER LOGON ON DATABASE
BEGIN
execute immediate 'alter session set NLS_DATE_FORMAT=''DD/MON/RRRR''';
END;
in system or sys schema.
Alternatively, you may use
alter system set NLS_DATE_FORMAT='DD/MON/RRRR' scope = both
provided you're in system or sys, again.
Manage your date format masking using the most reasonable approach
First of all, I agree with Alex regarding using to_char. This would be my first choice for modifying date masks for specific requirements.
In Toad on an ad hoc basis, you could just invoke the alter session command as needed:
ALTER SESSION SET nls_date_format='DD/MON/RRRR';
If you are partial to a specific date format mask (and you see yourself often issuing the command, ALTER SESSION SET NLS...) then perhaps you might want to consider changing your user login settings.
If you just modify your specific user preference login file, login.sql (see here ), your session will adhere to the date format mask of your choosing at the beginning of your session. I am partial to creating the environment variable, SQLPATH, and placing my login script there.
Toad will honor your login.sql file settings (e.g. see this post).
Since this is driven by specific requirements or personal preferences, I would never think of modifying this from default at the site level.
I often have a large block of HiveQL that I want to run multiple times with different settings for some variables.
A simple example would be:
set mindate='2015-01-01 00:00:00'
set maxdate='2015-04-01 00:00:00'
select * from my_table where the_date between ${hiveconf:mindate} and ${hiveconf:maxdate}
Which is then run via hive -f myfile.sql > myout.log
Later, I would like to change the variables and re-run. I also want a record of what values the variables had each time I ran.
So I currently make copies of the HiveQL file that are the same except for the variable values. This is obviously error-prone, however, because if I need to change the actual HiveQL, then I have to change it in every file.
Ideally, I could store all my settings a JSON file (or whatever) and have my HiveQL file be totally dynamic. Is there any way to do this?
Set your variables in the config file and load this file in your hql script:
source /path_to_your_config_file/config.hql;
I am using oracle 11g r2, and trying to configure DB to sort order using linguistic sort.
I did
alter system set NLS_SORT='RUSSIAN' SCOPE=SPFILE;
alter system set NLS_COMP='LINUGUISTIC' SCOPE=SPFILE;
after i've restarter oracle i checked these params:
show parameters NLS_SORT;
show parameters NLS_COMP;
it show me the right values.
But when I make sort
select name from test order by name;
it show me results in not correct order, ie digits first, then letters.
but if i will do
alter session set nls_sort='RUSSIAN';
alter session set nls_comp='LINGUISTIC';
select name from test order by name;
it show me the right order.
anyone know why sysem changes not showing me right results ?
The priority for globalisation settings is shown in the documentation. You're setting priority 4 in that list, 'Specified in the initialization parameter file'. You are not setting priority 1 ('Explicitly set in SQL functions') and you get the results you want when you do set priority 2 ('Set by an ALTER SESSION statement'). By a process of elimination that indicates that your 'not correct' order is being influenced by priority 3, 'Set as an environment variable'.
You can check the values actually being used by your session with select * from nls_session_parameters.
The NLS_SORT environment variable is probably not being set directly; I suspect it's being derived from NLS_LANGUAGE, which is derived from NLS_LANG. If you aren't explicitly setting that in your operating system environment then the client will set it based on the operating system locale, generally, though the exact client you use may make a significant difference. You might need to explicitly set an NLS_COMP environment variable, if the database default for that is really being overridden.
SQL Developer, for example, allows you to specify the NLS settings in the preferences (accessed from Tools->Preferences->Database->NLS); the defaults appear to be based on operating system settings, in Windows anyway. For SQL*Plus you'd need to set operating system environment variables.
This also means that if you get it working in one place - the queries give the right order when run from SQL Developer, say - they might not work when used elsewhere, say over JDBC which has its own locale settings. Just something to watch out for.
A brute-force approach might be to add the alter session commands to a login trigger, but that doesn't sound ideal as it just masks the environment configuration.
You can set NLS parameters at different levels
As initialization parameters on the instance/server.
SQL> alter system set V$NLS_PARAMETER = 'XXX' scope = both;
As environment variables on the client.
% setenv NLS_SORT FRENCH
As ALTER SESSION parameters.
SQL> ALTER SESSION SET V$NLS_PARAMETER = = 'XXX'
Any setting overrides the setting on a higher level. So setting it server side does not guarantee that the setting is used by all clients connecting.
If you want to make sure it is set for every client connecting use a logon trigger. Even then a user can explicitly override the 'default' setting
You've got a typo in your second "ALTER SYSTEM" command (LINUGUISTIC instead of LINGUISTIC).
If your real command doesn't contain this error, I'd check whether your client sets the NLS session parameters to something else.
Regardless of the system settings, I would make every effort to ensure that your applications completely specify the NLS environment that they require. It's much more robust, particularly when you need to point the application code at different environments that may be newly setup, or shared with other systems.
In fact, I'd go as far as to say that you might be better not using setting system-level environment settings.