I'm trying to access to Hive by the command window.
I just run "Hive" in the appropiate directory but I get an error "Login denied".
I've read that log4j is used to log in, but I don't know whether I have to create an account and write my user data there or not.
Thank you very much
The Hive service should be working right now. From a FI-LAB VM of your own, you simply have to log into the Head Node using your Cosmos credentials (if you have no Cosmos credentials, get them by registering here):
[root#your_filab_vm]$ssh cosmos.lab.fi-ware.org
Once logged in the Head Node, type the following command:
[your_cosmos_username#cosmosmaster-gi]$ hive
Logging initialized using configuration in jar:file:/usr/local/hive-0.9.0-shark-0.8.0-bin/lib/hive-common-0.9.0-shark-0.8.0.jar!/hive-log4j.properties
Hive history file=/tmp/<your_cosmos_username>/hive_job_log_<your_cosmos_username>_201407212017_1797291774.txt
hive>
As you can see, in this example your Hive history will be written within:
/tmp/<your_cosmos_username>/hive_job_log_<your_cosmos_username>_201407212017_1797291774.txt
Related
We have two different Azure cloud resource groups, RG1 and RG2, where RG1 hosts the ADB_source of the data source, and RG2 hosts the ADB_sink & ADLS_sink(gen2) of the data sink.
Use Case:
We have a few delta tables in ADB_source (ACL enabled) where a list of users has Read access.
In the ADB_source workspace, we need to read the delta tables and write them into ADLS_sink as parquet for further processing at the sink.
What's Available:
We have a high concurrency cluster created in ADB_Source workspace, which -
Allows only Python & SQL (dbutils.fs also restricted).
Credential Passthrough is disabled.
Has ACLs Enabled in spark config.
Has mount point created to a container in ADLS_sink.
Has no Admin Access to the cluster.
Errors Observed:
We could read the delta tables as expected and run action commands as long as they are in the ADB_source workspace. However, when we write that data into the ADLS_sink with .save(), we get the below error.
Py4JJavaError: An error occurred while calling o410.save. : java.lang.SecurityException: User does not have permission SELECT on any file. User does not have permission MODIFY on any file.
I would appreciate it if anyone could explain this and recommend additional security checks/accesses needed to implement the use case successfully.
This is happening because ACL is enabled, please refer to the documentation below:
https://learn.microsoft.com/en-us/azure/databricks/kb/security/table-create-security-exception
I'm intermittently getting the error message
DAG did not succeed due to VERTEX_FAILURE.
when running Hive queries via PyHive. Hive is running on an EMR cluster where hive.vectorized.execution.enabled is set to false in the hive-site.xml file for this reason.
I can set the above property through the configuration on the Hive connection and my query has run successfully every time I've executed it, however I want to confirm that this has fixed the issue and that it is definitely the case that hive-site.xml is being ignored.
Can anyone confirm if this is the expected behavior, or alternatively is there any way to inspect the Hive configuration via PyHive as I've not been able to find any way of doing this?
Thanks!
PyHive is a thin client that connects to HiveServer2, just like a Java or C client (via JDBC or ODBC). It does not use any Hadoop configuration files on your local machine. The HS2 session starts with whatever properties are set server-side.
Same goes for ImPyla BTW.
So it's your responsibility to set custom session properties from your Python code, e.g. execute this statement...
SET hive.vectorized.execution.enabled =False
... before running your SELECT.
I have set up a view for file browsing in Ambari admin (Views - Add view - files), but when trying to access this view, I get the following error appearing:
E090 HDFS030 Error in creation /user//hive/jobs/hive-job-... [HdfsApiException]
Why?
Solved!
The solution to my problem was that the user running "ambari-server" was not allowed to act upon behalf of the current user logged into ambari. In Hadoop terms, the ambari daemon user was not allowed to impersonate the ambari user.
To fix this, the HDFS config had to be modified to add access for my ambari-server user to impersonate everybody. For detailed howto, see this page:
http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_ambari_views_guide/content/_configuring_your_cluster_for_files_view.html
I'm using "COPY SELECT ... INTO file" statement from within application code. After file is ready, next step is to move the file to different location. The only problem is that file created by MonetDB has only root permissions so my application code can't touch it. Is there a way to configure MonetDB so dumps are saved as specified user? or my only solution is to iterate results in batches in application and save to file that way. Dumps can range from several MB to 1GB.
You could run MonetDB as the same user that your application server is configured for. Also, both your application server and MonetDB probably should not run as 'root'.
There is no direct support to export files with different permissions. You could try configuring the umask for the user that the starts the DB.
I got this error:
Service process [integrationServiceName] generated the following
output error: [ERROR:Unexpected condition at file:[]Application
terminating. Contact informatica Technical Support for assistance.]
every time that I try to run a workflow which includes
an Access DB source/target.ie. each time that in the session mapping there is an ODBC Connection to Access Driver. I manage to preview the data on client.
I created a DSN to the 64bit mod Administrator on Server.
I run the workflow and my session fails without logging errors.
I got only this one on the Administrator console and
First Error Code:36401..Execution terminated unexpextedly in run properties of session.
Do you have any ideas?
Thank you in advance.
Add the custom property in session config object as
OptimizeODBCRead=No;