Given bad mysqldump that causes error on import:
namtar backups # mysql -p < 2010-12-01.sql
Enter password:
ERROR 1062 (23000) at line 8020: Duplicate entry 'l�he?' for key 'wrd_txt'
Is there an easy way to tell import to just skip given row and continue?
(Yes, I know I can manually edit the file or parse output, but it's not very convinient)
If you can make the dump again you could add --insert-ignore to the command-line when dumping.
Or you can try using the mysqlimport command with --force,which will continue even if it encounters MySQL Errors.
mysql -f -p < 2010-12-01.sql
the -f (force) being the operative option here, worked for me.
Following the advice from jmlsteele's answer and comment, here's how to turn the inserts into INSERT IGNORE on the fly.
If you're importing from an sql file:
sed -e "s/^INSERT INTO/INSERT IGNORE INTO/" < 2010-12-01.sql | mysql -p
If you're importing from a gz file, just pipe the output from gunzip into sed instead of using the file input:
gunzip < 2010-12-01.sql.gz | sed -e "s/^INSERT INTO/INSERT IGNORE INTO/" | mysql -p
Great tip. I did it a little different but same result.
perl -pi -e 's/INSERT INTO/INSERT IGNORE INTO/g' filename.sql
The other options certainly are viable options, but another solution would be to simply edit the .sql file obtained from the mysqldump.
Change:
INSERT INTO table_name ...
TO
INSERT IGNORE INTO table_name ...
Just a thought did you delete the MySQL directives at the top of dump?
(I unintentionally did when I restarted a restore after deleting all the records/tables I'd already inserted with a sed command). These directives tell MySQL ,among other things, not to do unique tests, foreign key tests etc)
/*!40101 SET #OLD_CHARACTER_SET_CLIENT=##CHARACTER_SET_CLIENT */;
/*!40101 SET #OLD_CHARACTER_SET_RESULTS=##CHARACTER_SET_RESULTS */;
/*!40101 SET #OLD_COLLATION_CONNECTION=##COLLATION_CONNECTION */;
/*!40101 SET NAMES utf8 */;
/*!40103 SET #OLD_TIME_ZONE=##TIME_ZONE */;
/*!40103 SET TIME_ZONE='+00:00' */;
/*!40014 SET #OLD_UNIQUE_CHECKS=##UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
/*!40014 SET #OLD_FOREIGN_KEY_CHECKS=##FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
/*!40101 SET #OLD_SQL_MODE=##SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
/*!40111 SET #OLD_SQL_NOTES=##SQL_NOTES, SQL_NOTES=0 */;
Related
I'm new here and already tried to find solution to the following requirement without success. I'm trying to achieve this:
I have these 5 folders:
ServiceEngine
PaymentEngine
InvoiceEngine
ProcessEngine
OrderProcessEngine
Inside each of these folders, I have a log file with default path location to store the log files e.g. ServiceEngine/logs
The log file contain the following path structure:
name="RollingRandomAccessFile" fileName="logs/engine.log"
filePattern="logs/engine-%i.log"
I expect to find a way that I retrieve the name of the current folder which I'm in and replace the string engine with folder name
Example: I'm in ServiceEngine folder and execute a command that retrieve the current folder name. The expected result is:
name="RollingRandomAccessFile" fileName="logs/ServiceEngine.log"
filePattern="logs/ServiceEngine-%i.log
Later I change the directory to PaymentEngine and the expected result is:
name="RollingRandomAccessFile" fileName="logs/PaymentEngine.log"
filePattern="logs/PaymentEngine-%i.log
and so on. Maybe there is a smarter way to create a script that update the string in a loop like do; if ... fi; done or to use the for in ... loop.
Do you mean something like this?
~/ServiceEngine$ cat logfile
name="RollingRandomAccessFile" fileName="logs/engine.log"
filePattern="logs/engine-%i.log"
~/ServiceEngine$ awk -v path=$(basename $(pwd)) 'gsub("engine", path)' logfile
name="RollingRandomAccessFile" fileName="logs/ServiceEngine.log"
filePattern="logs/ServiceEngine-%i.log"
See basename, declaring variables in awk and awk gsub.
I guess I don't understand your question, but:
dir=$( basename $( pwd ) )
echo name="RollingRandomAccessFile" \
fileName="logs/$dir.log" \
filePattern="logs/$dir-%i.log"
Is that what you're looking for?
It sounds like you are looking for something like this:
$ sed 's/\bengine\b/'$(basename $(pwd))'/' logs
When run from within one of your folders, it spits out the text you're asking for. It wasn't clear what you wanted to do with that text though.
I can't seem to find a way to specify a relative path for my infile when using sql loader.
I'm running it through a command line and this is what it looks like:
C:\app\...in\sqlldr.exe userid=user/pass
control="C:\User...DATA_DATA_TABLE.ctl" log="C:\User...DATA_DATA_TABLE.log"
bad = "C:\User...DATA_DATA_TABLE.bad" discard = "C:\User...DATA_DATA_TABLE.dsc"
(I've added carriage returns just for the readability on here, the command i use is one line)
And this works, it's will start inserting stuff in the table IF the path to my infile in .ctl is absolute like "C:\Usertemp\example.ldr"
My ctl was generated autmatically by sqldeveloper. And i just changed the path to this:
OPTIONS (ERRORS=50)
LOAD DATA
INFILE 'AI_SLA_DATA_DATA_TABLE.ldr' "str '{EOL}'" <-- i'm trying to get relative path here but doesn't work
APPEND
CONTINUEIF NEXT(1:1) = '#'
INTO TABLE "USER"."DATA"
...other sqldeveloper generated stuff
The .ldr file is in the same directory as the .ctl file. Is it possible to get the path of the ctl? I'm pretty sure he searches for the .ldr file next to the sqlldr.exe instead of the ctl.
Any tips to do this? I can't find answers on docs.oracle.
Thanks.
I've never tried adding a relative path to the .ctl file, but for me it works fine as a command-line argument, e.g.
C:\app\...in\sqlldr.exe userid=user/pass
control="DATA_DATA_TABLE.ctl" log="DATA_DATA_TABLE.log"
bad = "DATA_DATA_TABLE.bad" discard ="DATA_DATA_TABLE.dsc"
data="AI_SLA_DATA_DATA_TABLE.ldr"
I have a file called table.txt which stores the table names. I want the sql update query to take the table name one by one from my table.txt file. My code is as follows:
while read LINE1; do
`sqlplus username/pwd#tname <<END |sed '/^$/d'
set head off;
set feedback off;
update &LINE1 set enterprise_id = '1234567890' where enterprise_id is NULL;
update &LINE1 set sim_inventory_id ='1234567890';
COMMIT;
exit;
END`
done < table.txt
it gives an error sqlplus not found. Can you please tell what is wrong?
This is nothing to do with passing the table names. The "sqlplus not found" error means it cannot find that binary executable, so it isn't getting as far as trying to connect or run the SQL commands.
Your shell script can only see environment variables from the calling shell if they were exported. If you've modified your PATH to include the location of the sqlplus binary then you may not have exported it; add export PATH after you set it.
Or you can set the script up to not rely on the shell environment.
export ORACLE_HOME=/path/to/oracle/installation
export PATH=${ORACLE_HOME}/bin:$PATH
export LD_LIBRARY_PATH=${ORACLE}/lib:${LD_LIBRARY_PATH}
while read LINE1; do
sqlplus username/pwd#tname <<END |sed '/^$/d'
set head off;
set feedback off;
update &LINE1 set enterprise_id = '1234567890' where enterprise_id is NULL;
update &LINE1 set sim_inventory_id ='1234567890';
COMMIT;
exit;
END
done < table.txt
Incidentally, updating the same table twice isn't necessary; you could do:
update &LINE1 set enterprise_id = nvl(enterprise_id, '1234567890'),
sim_inventory_id ='1234567890';
It would also be quicker to create a list of all the update statements from your file contents and run them all in a single SQL*Plus session, so you aren't repeatedly creating and tearing down connections. But that's outside the scope of what you asked.
I am trying to create a loop in Bash script for a series of data migrations:
At the beginning of every step, the script should get the name of the newest file in a folder
called "migrationfiles/ and store it in the variable "migbefore" and create a new variable called "migbefore+1":
Example: if the "migrationfiles/" folder contains the following files:
migration.pickle1 migration.pickle2 migration.pickle3
The variable "migbefore" and migafter should have the following value:
migbefore=migration.pickle3
migafter=migration.pickle4
At the end of every step, the function "metl", which is in charge of making the data migration, uses the file "migbefore" to load the data and creates 1 new file called "migafter" and stores it in the "migrationfiles/" folder, so in this case, the new file created will be called:
"migration.pickle4"
The code I pretend using is the following:
#!/bin/bash
migbefore=0
migafter=0
for y in testappend/*
for x in migrationfiles/*
do
migbefore=migration.pickle(oldest)
migafter=migbefore+1
done
do
metl -m migrationfiles/"${migbefore}"
-t migrationfiles/"${migafter}"
-s "${y}"
config3.yml
done
Does anyone know how I could make the first loop (The one that searches for the newest file in the "migrationfiles/" folder) and then assigns the name of the variable "migafter" as "migbefore+1"?
I think this might do what you want.
#!/bin/bash
count=0
prefix=migration.pickle
migbefore=$prefix$((count++))
migafter=$prefix$((count++))
for y in testappend/*; do
echo metl -m migrationfiles/"${migbefore}" \
-t migrationfiles/"${migafter}" \
-s "${y}" \
config3.yml
migbefore=$migafter
migafter=$prefix$((count++))
done
Copy with Numbered Backups
It's really hard to tell what you're really trying to do here, and why. However, you might be able to make life simpler by using the --backup flag from the cp command. For example:
cp --backup=numbered testappend/migration.pickle migrationfiles/
This will ensure that you have a sequence of migration files like:
migration.pickle
migration.pickle.~1~
migration.pickle.~2~
migration.pickle.~3~
where the older versions have larger ordinal numbers, while the latest version has no ordinal extension. It's a pretty simple system, but works well for a wide variety of use cases. YMMV.
# configuration:
path=migrationfiles
prefix=migration.pickle
# determine number of last file:
last_number=$( find ${path} -name "${prefix}*" | sed -e "s/.*${prefix}//g" | sort -n | tail -1 )
# put together the file names:
migbefore=${prefix}${last_number}
migafter=${prefix}$(( last_number + 1 ))
# test it:
echo $migbefore $migafter
This should work even if there are no migration files yet. In that case, the value of migbefore is just the prefix and does not point to a real file.
I am getting the error:
Remove leading / from member names
when attempting to create a domain archive using -X and a exclude file. If I remove the -X option the job executes as expected.
backup exec
/mnt/15326/s3/backup.sh
tmp dir
/mnt/15326/s3/data
exclude file
/mnt/15326/s3/data/exclude.txt
with contents
/mnt/15326/www.domain.com/dir1
/mnt/15326/www.domain.com/dir2
dir to backup
/mnt/15326/www.domain.com
command to run
tar cf /mnt/15326/s3/data/backup_domain.com -X /mnt/15326/s3/data/exclude.txt /mnt/15326/www.domain.com
It is a warning only. It means that in the tar file the file names wont contain the beginning /. They will be converted to relatives. Use -P if it disturbs you. generaly this default behaviour is a bit safer.