Could anyone suggest how to clear PutDatabaseRecord processor table schema cache. We've changed some of our tables (removed columns) but NiFi still uses old schemas and throws errors for columns that no longer exist. We tried to restart a processor multiple times, tried to restart DbcpConnectionPool - all without success.
To me this is the extremely weird behavior.
Thanks in advance
Related
putdatabaserecord.error
Unable to setObject() with value 1 at index 1 of type 4
I am trying to ingest the avro records into a griddb table and have tried different solutions but the same error appears every time. The respective tables are created correctly by the PutSQL processor but the INSERT query in PutDatabaseRecord fails. Also I tried the same steps replacing GridDB with PostgreSQL and everything works fine. I guess the problem lies somewhere in my GridDB datatype mismatch.
I have followed the ETL With Apache Nifi tutorial available on GridDB site.
Error
NiFi Flow
Template File
This is a limitation of the GridDB JDBC driver, see the documentation section 2.2.4.3. NiFi uses the setObject() method with the SQL type populated, and this method is not supported by the GridDB driver. It may be possible for NiFi to catch this and instead try one of the supported methods, feel free to file a Jira for this improvement.
After restarting the Impala server, we are not able to see the tables(i.e. tables are not coming up).Anyone help me what order we have to follow to avoid this issue.
Thanks,
Srinivas
You should try running "invalidate metadata;" from impala-shell. This usually clears up tables not being visible as impala caches metadata.
From:
https://www.cloudera.com/documentation/enterprise/5-8-x/topics/impala_invalidate_metadata.html
The following example shows how you might use the INVALIDATE METADATA
statement after creating new tables (such as SequenceFile or HBase tables) through the Hive shell. Before the INVALIDATE METADATA statement was issued, Impala would give a "table not found" error if you tried to refer to those table names.
We are currently trying to use Sqoop to ingest data from Hadoop to Azure SQL Data Warehouse but getting error related to Transaction isolation level. What's happening is that Sqoop tries to set transaction isolation level to READ COMMITTED while trying to import/export whereas this feature isn't currently supported in Azure SQL Data warehouse. I've tried using --relaxed-isolation parameter of Sqoop but still no effect.
As a solution, I am thinking to:
1. Change Sqoop source code to alter Sqoop's behavior to not set transaction level
2. Look for APIs (if any) that may allow me to change this Sqoop's behavior programmatically.
Has anyone encountered such scenario? Looking for suggestions for the proposed solutions and how to go about them.
This issue has just been resolved in Sqoop: https://issues.apache.org/jira/browse/SQOOP-2349
Otherwise, #wBob's comment about using Polybase is definitely best practice: https://learn.microsoft.com/en-us/azure/data-factory/data-factory-azure-sql-data-warehouse-connector#use-polybase-to-load-data-into-azure-sql-data-warehouse
Does MonetDB support online schema changes? For example adding/changing a column on the fly while the tables are loaded in memory. Many in-memory databases have to restarted to get the schema changes reflected. So, I was wondering if MonetDB took care of this issue.
Yes. MonetDB supports SQL-99 which includes DDL statements (that are immediately reflected in the schema).
I am using a tomcat dbcp along with spring jdbc. When I start the server for first time and try to load the web-page it fetches the data from database and returns correct result set but when i make some changes to db using editor and try to reload the page, it shows old result set. I tried database logging and can see that query is reached to database. I think the result set is being cached somewhere in the container..Can someone tell me what parameter i need to take care of.
help will be appreciated.
thanks.
AngerClown, thanks for your replies. You got me to the real pain point of the problem.
The real problem was lying on database side. Somehow because of some primary key indexing issues some process has acquired the lock on the table and at the same time my autocommit from query browser was set to false. Because of this when i tried to fetch the data in same transaction changes got reflected but not in other transaction.
Without bothering much about it, I just recreated the table. And now it is working fine.
Thanks a lot.
-Santosh.