I am running an mload script on a table that is cleared at the beginning of a job. When the script fails, the error and log tables prevent the job from running a second time. How does one reset mload cleanly after a failure?
You need to drop the work/error/log table
DROP TABLE UV_mytable;
DROP TABLE ET_mytable;
DROP TABLE WT_mytable;
DROP TABLE LT_mytable;
And then release the load lock:
RELEASE MLOAD mytable;,
If this fails
RELEASE MLOAD mytable IN APPLY;
But why does the job fail at all?
Related
I have a web application using a mariaDB10.4.10 INNO_DB table which is updated every 5 minutes from a script.
The script is working like:
Create a temp table from a table XY and writing data to the temp table from a received csv file. When the data is written, the script starts a transaction, drop the XY table and rename the temp table to the XY, and commits the transaction.
Nevertheless some times a user gets an "XY table does not exists" error working with the application.
I already tried to LOCK the XY table in the transaction but it doesn't change a thing.
How can I solve this? Is there any kind of locking (I thought locking is no longer possible with INNO_DB?)
Do this another way.
Create the temporary table (not as temporary table, as a real table). Fill it as needed. Nobody else knows it's there, you have all the time.
SET autocommit = 0; // OR: LOCK TABLE xy WRITE, tempxy READ;
DELETE FROM xy;
INSERT INTO xy SELECT * FROM tempxy;
COMMIT WORK; // OR: UNLOCK TABLES;
DROP TABLE tempxy;
This way, other customers will see the old table until point 5, then they'll start seeing the new table.
If you use LOCK, customers will stall from point 2 to point 5, which, depending on time needed, might be bad.
At point #3, in some scenarios you might be able to optimize things by deleting only rows that are not in tempxy, and running an INSERT ON DUPLICATE KEY UPDATE at point 4.
Funnily enough, I answered recently another question that was somewhat like yours.
autoincrement
To prevent autoincrement column from overflowing, you can replace COMMIT WORK with ALTER TABLE xy AUTO_INCREMENT=. This is a dirty hack and relies on the fact that this DDL command in MySQL/MariaDB will execute an implicit COMMIT immediately followed by the DDL command itself. If nobody else inserts in that table, it is completely safe. If somebody else inserts in that table at the exact same time your script is running, it should be safe in MySQL 5.7 and derived releases; it might not be in other releases and flavours, e.g. MySQL 8.0 or Percona.
In practice, you fill up tempxy using a new autoincrement from 1 (since tempxy has been just created), then perform the DELETE/INSERT, and update the autoincrement counter to the count of rows you've just inserted.
To be completely sure, you can use a cooperative lock around the DDL command, on the one hand, and anyone else wanting to perform an INSERT, on the other:
script thread other thread
SELECT GET_LOCK('xy-ddl', 30);
SELECT GET_LOCK('xy-ddl', 30);
ALTER TABLE `xy` AUTO_INCREMENT=12345; # thread waits while
# script thread commits
# and runs DDL
SELECT RELEASE_LOCK('xy-ddl'); # thread can acquire lock
INSERT INTO ...
DROP TABLE tempxy; # Gets id = 12346
SELECT RELEASE_LOCK('xy-ddl');
Whenever hive table is locked in exclusive mode query to describe the same table hangs until lock is released. Is there a way to introduce a timeout after which describe query would just error out?
Before executing DESC <TABLE_NAME> command, check lock status SHOW LOCKS <TABLE_NAME>;
If table is un-locked then
DESC <TABLE_NAME>
I am not able to drop or truncate my table. However, i am successfully able to delete the rows from the table.
Whenever i try to drop or truncate my session goes in hanged state and i have to kill the session manually.
FYI. There is also a trigger associated with my table on before delete. I am also not able to disable that trigger.
Hoping, my question is clear. Please also suggest if some more information i need to provide.
Thanks for your time.
I dont have complete info on whats going on but start by running this query in your session , if it give you result then there is a lock
SELECT a.SESSION_ID, B.Owner, B.Object_Name, A.Oracle_Username, A.OS_User_Name
FROM V$Locked_Object A, All_Objects B
WHERE A.Object_ID = B.Object_ID
try to release the lock and understand ow the lock happened maybe the trigger is executing something without commit ?
Note that you should run the above query when you are trying to truncate the table.
HIVE 0.13 will SHARED lock the entire database(I see a node like LOCK-0000000000 as a child of the database node in Zookeeper) when running a select statement on any table in the database. HIVE creates a shared lock on the entire schema even when running a select statement - this results in a freeze on CREATE/DELETE statements on other tables in the database until the original query finishes and the lock is released.
Does anybody know a way around this? Following link suggests concurrency to be turned off but we can't do that as we are replacing the entire table and we have to make sure that no select statement is accessing the table before we replace the entire contents.
http://mail-archives.apache.org/mod_mbox/hive-user/201408.mbox/%3C0eba01cfc035$3501e4f0$9f05aed0$#com%3E
use mydatabase;
select count(*) from large_table limit 1; # this table is very large and hive.support.concurrency=true`
In another hive shell, meanwhile the 1st query is executing:
use mydatabase;
create table sometable (id string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' STORED AS TEXTFILE ;
The problem is that the “create table” does not execute untill the first query (select) has finished.
Update:
We are using Cloudera's distribution of Hive CDH-5.2.1-1 and we are seeing this issue.
I think they never made such that in Hive 0.13. Please verify your Resource manager and see that you have enough memory when you are executing multiple Hive queries.
As you know each Hive query will trigger a map reduce job and if YARN doesn't have enough resources it will wait till the previous running job completes. Please approach your issue from memory point of view.
All the best !!
I am trying to DROP and CREATE a table using a scheduled job. How to do it please?
DROP TABLE REGISTRATION;
CREATE TABLE REGISTRATION AS
SELECT *
FROM REGISTRATION_MV#D3PROD_KMDW
perhaps you are looking for a materialized view instead of the table?
create materialized view registration as
select *
from registration_mv#d3prod_kmdw
and then refresh it in job
dbms_mview.refresh('REGISTRATION');
This is much better than dropping and creating the table because PL/SQL objects will become invalid after dropping the table. With mview it will be silent and without harm.
You should use DBMS_SCHEDULER to schedule jobs in Oracle.
Refer to the documentation for how to create a sample scheduler job
Don't drop and recreate the table. If you do this on a regular basis you will end up with a huge number of items in your recycle bin, which can lead to problems. Also as above you can invalidate pl/sql when you drop objects, and can break your app if they fail to recompile.
Instead you should truncate the table to delete all rows, and then repopulate it:
truncate table registration;
insert into registration (select * from registration_mv#D3PROD_KMDW);