When I try to select * I get this from MariaDB :
But, when I select the food column individually, I see all the data :
What is going on here?
edit : Here are some more screens :
Thanks.
The data came from a non-Mac source? And it has a different type of line terminator?
Trim off trailing what space (Carriage Return / NewLine) before storing the data.
You should be able to use the REGEXP_REPLACE function to display the value without trailing and leading whitespace:
SELECT REGEXP_REPLACE(REGEXP_REPLACE(notes, '^\\s*', ''), '\\s*$', '')
FROM food1_test;
The inner '^\\s*' pattern matches all leading whitespace and the outer '\\s*$' pattern matches all trailing whitespace.
You can also do an in-place update of the data with the following SQL:
UPDATE food1_test SET
notes = REGEXP_REPLACE(REGEXP_REPLACE(notes, '^\\s*', ''), '\\s*$', '');
You could also use TRIM but it only replaces a specific string and it wouldn't work with the non-space whitespace characters.
As was stated by Rick James, it is important to sanitize and clean up the input before putting it into a database.
Related
Running SQL on Oracle through Liquibase and getting error:
ORA-00907: missing right parenthesis
The SQL we're running has double dashes -- which can also represent comments in PLSQL. I'm guessing this is the issue. Should this be escaped somehow?
delete from mytable B where B.NAME in ('XXX--YYY', 'AAA--BBB');
The sequence of the characters --is most likely interpreted as a comment, so the following charactes until the end of the line are ignored.
You may use following workaround, simple splitting the string in two parts
Instead of
'XXX--YYY'
use
'XXX-'||'-YYY'
Do not forget, if you have more dashes, you must repeat stis step , e.g. for --- you must split the string in three part.
Posible similar problem would be for string containing the multiline comment:
'XXX/*YYY', 'XXX*/YYY'
I am trying to code a sqlldr.ctl file WHEN Clause to limit the records imported to those matching a portion of the current Schema's name.
The code I have (which does NOT work) is:
LOAD DATA
TRUNCATE INTO TABLE TMP_PRIM_ACCTS
when REGION_NUM = substr(user,-3,3)
Fields terminated by "|" Optionally enclosed by '"'
Trailing NULLCOLS
( PORTFOLIO_ACCT,
PRIMARY_ACCT_ID NULLIF (PRIMARY_ASSET_ID="NULL"),
REGION_NUM NULLIF (PARTITION_NUM="NULL")
)
sqlldr returns:
SQL*Loader-350: Syntax error at line 3.
Expecting quoted string or hex identifier, found "substr".
when PARTITION_NUM = substr(user,-3,3)
I cannot put single quotes around "user", because that turns it into the literal string "user". Can anyone explain how I can reference the "active" User in this WHEN Clause?
Thank you!
Can you try something like this? (now I can't make test with SQLLDR, but this is syntax I used for changing values):
when REGION_NUM = "substr(:user,-3,3)"
It doesn't look like you can. The documentation only shows fixed values:
Trying to use an expression in when that clause (or in nullif; thought I'd try to see if you could cause a rejection based on null PK value) you just see the literal value in the log:
Table TMP_PRIM_ACCTS, loaded when REGION_NUM = 0X73756273747228757365722c2d332c3329(character 'substr(user,-3,3)')
which is sort of what you referred when you said you couldn't quote user, but you'd have to quite the whole thing anyway. Using :user doesn't work either, the colon is seen as just another character, it doesn't try to find a column called user instead.
The simplest approach may be to pre-process the data file and remove any rows which don't match the pattern (e.g. via a regex). That would actually be slightly easier if you used an external table instead of SQL*Loader.
Alternatively, generate your control file and embed the correct literal value based on the user you'll connect as.
I have some Oracle SQL Loader challenges and looking for an efficient and simple solution.
my source files to be loaded are pipe | delimited, where values are enclosed by Double Quotes ".
the problem seems to be that some of the values contains internal Double Quotes.
e.g.: ..."|"a":"b"|"...
this causes my records to be rejected under the excuse of:
no terminator found after TERMINATED and ENCLOSED field
there are various solutions over the web but non seems to fit:
[1]
I have tried to replace all internal double quotes in quoting the quotes,
but it seems that when applying this function on too many fields on the control files
(I have ~2000+ fields and using FILLER to load only a subset)
the loader complains again:
SQL*Loader-350: Syntax error at line 7.
Expecting "," or ")", found ",".
field1 char(36) "replace(:field1,'"','""')",
(I do not know why but when applying this solution on a narrow subset of columns it does seem to work)
thing is that potentially all fields may include internal double quotes.
[2]
I'm able to load all data when omitting the global optionally enclosed by '"', but then all enclosing quotes becomes part of the data in the target table.
[3]
I can omit the global optionally enclosed by '"' statement and place it only at selected fields,
while try to "replace(:field1,'"','""')" statement on the remainder, but this is difficult to implement,
as I cannot know ahead what are the suspected fields to include internal double quotes.
here are my questions:
is there no simple way to convince the loader to handle with care internal double quotes (when values are enclosed by them)?
if I'm forced to fix the data ad-hock, is there a one liner Linux command to convert only internal double quotes to another string/char,
say, single quotes?
if I'm forced to load data with the quotes to the target table, is there a simple way to remove the enclosing double quotes from all fields,
all at once (the table has ~1000 columns). is the solution practical performance wise to very large tables?
If you never had pipes in the enclosed fields you could do it from the control file. If you can have both pipes and double-quotes within a field then I think you have no choice but to preprocess the files, unfortunately.
Your solution [1], to replace double-quotes with an SQL operator, is happening too late to be useful; the delimiters and enclosures have already been interpreted by SQL*Loader before it does the SQL step. Your solution [2], to ignore the enclosure, would work in combination with [1] - until one of the fields did contain a pipe character. And solution [3] has the same problems as using [1] and/or [2] globally.
The documentation for specifying delimiters mentions that:
Sometimes the punctuation mark that is a delimiter must also be included in the data. To make that possible, two adjacent delimiter characters are interpreted as a single occurrence of the character, and this character is included in the data.
In other words, if you repeated the double-quotes inside the fields then they would be escaped and would appear in the table data. As you can't control the data generation, you could preprocess the files you get to replace all the double-quotes with escaped double quotes. Except you don't want to replace all of them - the ones that are actually real enclosures should not be escaped.
You could use a regular expression to target the relevant characters will skipping others. Not my strong area, but I think you can do this with lookahead and lookbehind assertions.
If you had a file called orig.txt containing:
"1"|A|"B"|"C|D"
"2"|A|"B"|"C"D"
3|A|""B""|"C|D"
4|A|"B"|"C"D|E"F"G|H""
you could do:
perl -pe 's/(?<!^)(?<!\|)"(?!\|)(?!$)/""/g' orig.txt > new.txt
That looks for a double-quote which is not preceded by the line-start anchor or a pipe character; and is not followed by a pipe character or line end anchor; and replaces only those with escaped (doubled) double-quotes. Which would make new.txt contain:
"1"|A|"B"|"C|D"
"2"|A|"B"|"C""D"
3|A|"""B"""|"C|D"
4|A|"B"|"C""D|E""F""G|H"""
The double-quotes at the start and end of fields are not modified, but those in the middle are now escaped. If you then loaded that with a control file with double-quote enclosures:
load data
truncate
into table t42
fields terminated by '|' optionally enclosed by '"'
(
col1,
col2,
col3,
col4
)
Then you would end up with:
select * from t42 order by col1;
COL1 COL2 COL3 COL4
---------- ---------- ---------- --------------------
1 A B C|D
2 A B C"D
3 A "B" C|D
3 A B C"D|E"F"G|H"
which hopefully matches your original data. There may be edge cases that don't work (like a double-quote followed by a pipe within a field) but there's a limit to what you can do to attempt to interpret someone else's data... There may also be (much) better regular expression patterns, of course.
You could also consider using an external table instead of SQL*Loader, if the data file is (or can be) in an Oracle directory and you have the right permissions. You still have to modify the file, but you could do it automatically with the preprocessor directive, rather than needing to do that explicitly before calling SQL*Loader.
I'm spooling out delimited text files from SQL*Plus, but every column is printed as the full size per its definition, rather than the data actually in that row.
For instance, a column defined as 10 characters, with a row value of "test", is printing out as "test " instead of "test". I can confirm this by selecting the column along with the value of its LENGTH function. It prints "test |4".
It kind of defeats the purpose of a delimiter if it forces me into fixed-width. Is there a SET option that will fix this, or some other way to make it print only the actual column data.
I don't want to add TRIM to every column, because if a value is actually stored with spaces I want to be able to keep them.
Thanks
I have seen many SQL*plus script, that create text files like this:
select A || ';' || B || ';' || C || ';' || D
from T
where ...
It's a strong indication to me that you can't just switch to variable length output with a SET command.
Instead of ';' you can of course use any other delimiter. And it's up to your query to properly escape any characters that could be confused with a delimiter or a line feed.
Generally, I'd forget SQL Plus as a method for getting CSV out of Oracle.
Tom Kyte has written a nice little Pro-C unloader
Personally I've written a utility which does similar but in perl
I am using the QOCI binding to connect Qt with an Oracle 10g database. The code is really simple:
QSQLQuery sqlQuery = QSQLQuery(database);
sqlquery.prepare(querystring);
sqlQuery.exec();
Now if querystring is only one line, it works:
select * from dual
But if it contains multiple lines, I get an ORA-911 invalid character:
select *
from dual
I have a lot of queries spanning multiple lines, so this is quite a problem. Just removing newlines in Qt is not an option, because the queries contain end-of-line comments ("--").
Any suggestions how I can execute these multi-line queries?
Answering my own question: The newline character was the unicode U+2029 paragraph seperator instead of a normal newline (\n). This triggered the ORA-911.
querystring.replace(QChar(0x2029), QChar('\n'));
does the trick.