Looking for Special characters in MonetDB table columns - monetdb

Question

I am new to MonetDB, I would like to check if there are any special characters in any of the columns in MonetDB.
For example, I have a test Database and the table name is Lmr. I would be to check if any of the columns in table lmr contains special characters?
The query I tried:
SELECT jk
FROM lmr
WHERE jk like '%[^a-Z0-9]%'
I have multiple columns, so is there any way where I can check all the columns with a special character at once?

Answer 1

The LIKE and ILIKE operators use PCRE internally, but do not expose the same expressive power. Basically you can only use % as wildcards.
Luckily, MonetDB already provides wrappers to the PCRE library. For some reason that I am not aware of, they are not made available by default at the SQL layer.
In order to do that, you only need to create SQL function signatures that link to the code that is already available:
CREATE OR REPLACE FUNCTION pcre_match(s string, pattern string) RETURNS boolean EXTERNAL NAME pcre."match";
CREATE OR REPLACE FUNCTION pcre_imatch(s string, pattern string) RETURNS boolean EXTERNAL NAME pcre."imatch";
CREATE OR REPLACE FUNCTION pcre_replace(s string, pattern string, repl string, flags string) RETURNS string EXTERNAL NAME pcre."replace";
CREATE OR REPLACE FUNCTION pcre_replacefirst(s string, pattern string, repl string, flags string) RETURNS string EXTERNAL NAME pcre."replace_first";
After that (to be done only once in a database), you can do:
SELECT jk
FROM lmr
WHERE pcre_imatch(jk,'[^a-z0-9]');
The second parameter is a regular PCRE pattern.
Mind that you had an error in your example. The range a-Z does not exist, because a comes after Z.
In my example I used the i (ignore case) variant of the function, and only used range a-z.
If you want you can also use Unicode categories and rewrite your example to match everything that is not a letter or a number as:
SELECT jk
FROM lmr
WHERE pcre_imatch(jk,'[^\\p{L}\\p{N}]');
Mind that you need to escape each \, which becomes then \\.
About checking multiple columns at once, assuming that you want to return the rows where the condition is satisfied on any of the given columns, you could do this (for 3 columns here):
SELECT col1,col2,col3
FROM lmr
WHERE pcre_imatch(col1 || col2 || col3,'[^\\p{L}\\p{N}]');
where || is string concatenation.
The problem with this is that it first needs to concatenate all columns together. Because MonetDB is a column-store, it will do this for all rows at once. So it will first materialize in memory (and/or disk) all columns for all rows. I'm not sure how much data you have, but that is potentially very big.
The other approach is of course:
SELECT col1,col2,col3
FROM lmr
WHERE pcre_imatch(col1,'[^\\p{L}\\p{N}]')
OR pcre_imatch(col2,'[^\\p{L}\\p{N}]')
OR pcre_imatch(col3,'[^\\p{L}\\p{N}]');
I think I would choose the second approach, as it definitely has a much smaller memory footprint.