How to disable logs in Hive while writing to a file - hadoop

Question

I have a use case where i am executing hive query and storing output to a file.
hive -S -e "SELECT * from test.employee where empid=1" > /mapr/Piyush/test/output.txt
The query is executing fine but i am getting logs also along with the data in the file. I am guessing it's because of the log4j properties. The problem here is i have no access to the log4j config file so i can't do any changes in it.
I tried setting couple of configuration like.
set hive.root.logger=ERROR, console
and
set hive.root.logger=INFO,console
and
set hive.server2.logging.operation.enabled=false
But nothing helps. I need just the data from the table in the file. Please let me know if i am missing anything or any solution to this issue.
Sample output file
2017-06-09 11:49:18,265 main ERROR Cannot access RandomAccessFile {}) java.io.FileNotFoundException: /mapr/pankaj-hive.log (Permission denied)
2017-06-09 11:49:18,272 main ERROR Unable to invoke factory method in class class org.apache.logging.log4j.core.appender.RollingRandomAccessFileAppender for element RollingRandomAccessFile. java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:813)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:753)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:745)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:389)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:169)
at org.apache.logging.log4j.core.config.builder.impl.DefaultConfigurationBuilder.build(DefaultConfigurationBuilder.java:158)
at org.apache.logging.log4j.core.config.builder.impl.DefaultConfigurationBuilder.build(DefaultConfigurationBuilder.java:43)
at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:149)
at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:46)
at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:413)
at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:257)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:519)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:536)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:214)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:146)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:185)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:103)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:284)
at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:655)
at org.apache.hadoop.util.ShutdownHookManager.<clinit>(ShutdownHookManager.java:44)
at org.apache.hadoop.util.RunJar.run(RunJar.java:200)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingRandomAccessFileManager$RollingRandomAccessFileManagerFactory#48aca48b] unable to create manager for [/mapr/pankaj-hive.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingRandomAccessFileManager$FactoryData#13fd2ccd]
at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:73)
at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:61)
at org.apache.logging.log4j.core.appender.rolling.RollingRandomAccessFileManager.getRollingRandomAccessFileManager(RollingRandomAccessFileManager.java:84)
at org.apache.logging.log4j.core.appender.RollingRandomAccessFileAppender.createAppender(RollingRandomAccessFileAppender.java:206)
... 33 more
2017-06-09 11:49:18,277 main ERROR Null object returned for RollingRandomAccessFile in Appenders.
2017-06-09 11:49:18,281 main ERROR Unable to locate appender "DRFA" for logger config "root"
2017-06-09 11:49:19,640 main ERROR Cannot access RandomAccessFile {}) java.io.FileNotFoundException: /mapr/pankaj-hive.log (Permission denied)
2017-06-09 11:49:19,644 main ERROR Unable to invoke factory method in class class org.apache.logging.log4j.core.appender.RollingRandomAccessFileAppender for element RollingRandomAccessFile. java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:813)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:753)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:745)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:389)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:169)
at org.apache.logging.log4j.core.config.builder.impl.DefaultConfigurationBuilder.build(DefaultConfigurationBuilder.java:158)
at org.apache.logging.log4j.core.config.builder.impl.DefaultConfigurationBuilder.build(DefaultConfigurationBuilder.java:43)
at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:149)
at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:46)
at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:236)
at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:445)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:228)
at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:140)
at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:113)
at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:98)
at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:156)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jDefault(LogUtils.java:155)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:91)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:83)
at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:66)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:661)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:646)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingRandomAccessFileManager$RollingRandomAccessFileManagerFactory#48aca48b] unable to create manager for [/mapr/pankaj-hive.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingRandomAccessFileManager$FactoryData#bff34c6]
at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:73)
at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:61)
at org.apache.logging.log4j.core.appender.rolling.RollingRandomAccessFileManager.getRollingRandomAccessFileManager(RollingRandomAccessFileManager.java:84)
at org.apache.logging.log4j.core.appender.RollingRandomAccessFileAppender.createAppender(RollingRandomAccessFileAppender.java:206)
... 34 more
2017-06-09 11:49:19,649 main ERROR Null object returned for RollingRandomAccessFile in Appenders.
2017-06-09 11:49:19,652 main ERROR Unable to locate appender "DRFA" for logger config "root"
1 piyush bangalore 20
I want output as just
1 piyush bangalore 20

Answer 1

Hi as per hive documentation you can't change logging property by setting hive.root.logger via 'set' command because hive reads the logger properties at time of initialization i.e prior to opening cli.
you can do it like this:
hive --hiveconf hive.root.logger=OFF -S -e "SELECT * from test.employee where empid=1" > /mapr/Piyush/test/output.txt
P.S i have assumed that you don't want any logging. You can change logging level as per your requirement.
Below i am attaching screenshot of official documentation with link
Click for official Documentation