Or should I ?
(The title is inspired by Gary Myers' comment in Why does Oracle varchar2 have a mandatory size as a definition parameter?)
Consider the following variables:
declare
-- database table column interfacing variable
v_a tablex.a%type; -- tablex.a is varchar2
-- PL/SQL only variable
v_b varchar2(32767); -- is this a poor convention ?
begin
select a into v_a from tablex where id = 1;
v_b := 'Some arbitrary string: ' || v_a; -- ignore potential ORA-06502
insert into tabley(id, a) values(1, v_a); -- tablex.a and tabley.a types match
v_b := v_b || ' More arbitrary characters';
end;
/
Variable v_a is used to interface a database table column and therefore uses a %type attribute. But if I know the data type is varchar2 why shouldn't I use varchar2(4000) or varchar2(32767) that also guarantee the string read from database column will always fit to the PL/SQL variable ? Is there any other argument against this convention except the superiority of %type attribute ?
Variable v_b is only used in PL/SQL code and is usually returned to a JDBC client (Java/Python program, Oracle SOA/OSB etc.) or dumped into a flat file (with UTL_FILE). If the varchar2 presents e.g. csv-line why I should bother to calculate the exact maximum possible length (except to verify the line will fit into 32767 bytes in all cases so I don't need a clob) and re-calculate every time my data model changes ?
There is plenty of questions that covers varchar2 length semantics in SQL and explains why varchar2(4000) is a poor practice in SQL. Also the difference between SQL and PL/SQL varchar2-type is well covered:
What is the size limit for a varchar2 PL/SQL subprogram argument in Oracle?
VARCHAR(MAX) versus VARCHAR(n) in Oracle
Why does Oracle varchar2 have a mandatory size as a definition parameter?
Why does VARCHAR need length specification?
What is the default size of a varchar2 input to Oracle stored procedure, and can it be changed?
Why using anything else but VARCHAR2(4000) to store strings in an Oracle database?
Why does an oracle plsql varchar2 variable need a size but a parameter does not?
Ask Tom question: "I work with a modelers group, who would like to define every varchar2 field with the maximum length."
The only place where I have seen this issue discussed is the points #3 and #4 in an answer by APC:
The database uses the length of a variable when allocating memory for PL/SQL collections. As that memory comes out of the PGA supersizing the variable declaration can lead to programs failing because the server has run out of memory.
There are similar issues with the declaration of single variables in PL/SQL programs, it is just that collections tend to multiply the problem.
E.g. Oracle PL/SQL Programming, 5th Edition By Steven Feuerstein doesn't mention any drawbacks of declaring too long varchar2 variables, so it can't be a critical mistake, right ?
Update
After some more googling I found out that Oracle documentation has evolved during releases:
A quote from PL/SQL User's Guide and Reference 10g Release 2 Chapter 3 PL/SQL Datatypes:
Small VARCHAR2 variables are optimized for performance, and larger ones are optimized for efficient memory use. The cutoff point is 2000 bytes. For a VARCHAR2 that is 2000 bytes or longer, PL/SQL dynamically allocates only enough memory to hold the actual value. For a VARCHAR2 variable that is shorter than 2000 bytes, PL/SQL preallocates the full declared length of the variable. For example, if you assign the same 500-byte value to a VARCHAR2(2000 BYTE) variable and to a VARCHAR2(1999 BYTE) variable, the former takes up 500 bytes and the latter takes up 1999 bytes.
A quote from PL/SQL User's Guide and Reference 11g Release 1 Chapter 3 PL/SQL Datatypes:
For a CHAR variable, or for a VARCHAR2 variable whose maximum size is less than 2,000 bytes, PL/SQL allocates enough memory for the maximum size at compile time. For a VARCHAR2 whose maximum size is 2,000 bytes or more, PL/SQL allocates enough memory to store the actual value at run time. In this way, PL/SQL optimizes smaller VARCHAR2 variables for performance and larger ones for efficient memory use.
For example, if you assign the same 500-byte value to VARCHAR2(1999 BYTE) and VARCHAR2(2000 BYTE) variables, PL/SQL allocates 1999 bytes for the former variable at compile time and 500 bytes for the latter variable at run time.
But PL/SQL User's Guide and Reference 11g Release 2 Chapter 3 PL/SQL Datatypes doesn't mention memory allocation any more and I fail to find any other information about memory allocation at all. (I'm using this release so I check only 11.2 documentation.) The same holds also for PL/SQL User's Guide and Reference 12c Release 1 Chapter 3 PL/SQL Datatypes.
I also found an answer by Jeffrey Kemp that addresses this question too. However Jeffrey's answer refers to 10.2 documentation and the question is not about PL/SQL at all.
It looks like this is one of the areas where the PL/SQL functionality has evolved over releases when Oracle has implemented different optimizations.
Note this also means some of the answers listed in the OP are also release specific even that is not explicitly mentioned in those questions/answers. When the time pass by and use of older Oracle releases ends (me daydreaming ?) that information will became outdated (might take decades thought).
The conclusion above is backed with the following quote from chapter 12 Tuning PL/SQL Applications for Performance of PL/SQL Language Reference 11g R1:
Declare VARCHAR2 Variables of 4000 or More Characters
You might need to allocate large VARCHAR2 variables when you are not sure how big an expression result will be. You can conserve memory by declaring VARCHAR2 variables with large sizes, such as 32000, rather than estimating just a little on the high side, such as by specifying 256 or 1000. PL/SQL has an optimization that makes it easy to avoid overflow problems and still conserve memory. Specify a size of more than 4000 characters for the VARCHAR2 variable; PL/SQL waits until you assign the variable, then only allocates as much storage as needed.
This issue is no longer mentioned in 11g R2 nor 12c R1 version of the document. This is in line with the evolution of the chapter 3 PL/SQL Datatypes.
Answer:
Since 11gR2 it makes no difference from memory use of point of view to use varchar2(10) or varchar2(32767). Oracle PL/SQL compiler will take care of the dirty details for you in an optimal fashion !
For releases prior to 11gR2 there is a cutoff-point where different memory management strategies are used and this is clearly documented in each release's PL/SQL Language Reference.
The above only applies to PL/SQL-only variables when there is no natural length restriction that can be derived from the problem domain. If a varchar2-variable represents a GTIN-14 then one should declare that as varchar2(14).
When PL/SQL-variable interfaces with a table column use %type-attribute as that is the zero-effort way to keep you PL/SQL-code and database structure in sync.
Memory test results:
I run a memory analysis in Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 with the following results:
str_size iterations UGA PGA
-------- ---------- ----- ------
10 100 65488 0
10 1000 65488 65536
10 10000 65488 655360
32767 100 65488 0
32767 1000 65488 65536
32767 10000 65488 655360
Because the PGA changes are identical and depend only on iterations and not str_size I conclude the varchar2 declared size doesn't matter. The test might be too naïve though - comments welcome !
The test script:
-- plsql_memory is a convenience package wrapping sys.v_$mystat s and
-- sys.v_$statname tables written by Steven Feuerstein and available in the
-- code-zip file accompanying his book.
set verify off
define str_size=&1
define iterations=&2
declare
type str_list_t is table of varchar2(&str_size);
begin
plsql_memory.start_analysis;
declare
v_strs str_list_t := str_list_t();
begin
for i in 1 .. &iterations
loop
v_strs.extend;
v_strs(i) := rpad(to_char(i), 10, to_char(i));
end loop;
plsql_memory.show_memory_usage;
end;
end;
/
exit
Test run example:
$ sqlplus -SL <CONNECT_STR> #memory-test.sql 32767 10000
Change in UGA memory: 65488 (Current = 1927304)
Change in PGA memory: 655360 (Current = 3572704)
PL/SQL procedure successfully completed.
$
Maybe it's because I came of age in the era when the most memory any system had was 48k and then, happy days, up to a full 64k. And it was not virtual, what you allocated is what you got (wyaiwyg) (WAY-WIG?). I see in a lot of younger programmers a tendency to be lazy in their design and hide design flaws by throwing more memory at it.
If we get into the habit of just typing varchar2(MAX) whenever we define a string variable, we stop thinking about the length. But sometimes length matters. If we haven't already done so, as soon as we type the ( then we should stop and put some thought into how big it really needs to be. Does the size matter here? If so, what is a reasonable maximum (or minimum)? This forces us to look beyond the bytes and fields and indexes to the actual "thing" we are trying to work with. That is never a bad idea.
Discipline is hard. We should be developing habits to enforce it whenever we can. Good data and code design is difficult by nature. There are tools and techniques we can use to make it a little easier, but we shouldn't be doing anything just because it is easier. That's the path to the Dark Side and it catches up to us sooner or later.
I think the problem is the same as in any other piece of software: allocating too much memory will decrease performance and likeliness of failing due to the facts it has occupied all memory.
In my opinion, use %type as much as possible since it prevents mistakes when changing data type or length, and makes clear what the origin is from that variable.
Related
In PLSQL where do we store the global variables declared in the package specification
Oracle stores global variables in memory structures that are part of the Program Global Area. You can read about the PGA in the Memory Architecture chapter of the Oracle Concepts Guide.
Variables are only accessible through PL/SQL, we can't access them through data dictionary views.
Session variables can only fit in PGA, they cannot spill to disk, so we may have to be careful about loading too much data. We need to avoid storing large tables in variables, which we can often do by processing cursors with FOR loops or using a LIMIT clause.
For example, the following code loads a lot of data into a simple collection.
--Load 20,000 large strings. Takes about 10 seconds.
declare
type string_nt is table of varchar2(4000);
v_strings string_nt := string_nt();
begin
for i in 1 .. 20000 loop
v_strings.extend;
v_strings(v_strings.count) := lpad('A', 4000, 'A');
end loop;
null;
end;
/
We can't view the variable data in the data dictionary, but we can check the size of the data through the data dictionary. In this case, it takes about 105MB of memory to store 80MB of raw data:
--Maximum session PGA memory.
select value/1024/1024 mb
from v$sesstat
join v$statname
on v$sesstat.statistic# = v$statname.statistic#
where v$statname.name = 'session pga memory max'
order by value desc;
(My answer is based on the assumption that you're asking because you're worried about storing lots of data. If my assumption is wrong, please update the question to explain precisely what you're looking for.)
I have a question regarding how queries executed through
'execute immediate' is treated in the library cache (We use Oracle 11).
Let's say I have a function like this:
FUNCTION get_meta_map_value (
getfield IN VARCHAR2,
searchfield IN VARCHAR2,
searchvalue IN VARCHAR2
) RETURN VARCHAR2 IS
v_outvalue VARCHAR2(32767);
sql_stmt VARCHAR2(2000) := 'SELECT '||getfield||' FROM field_mapping, metadata '||
'WHERE field_mapping.metadataid = metadata.metadataid AND rownum = 1 AND '||searchfield||' = :1';
BEGIN
EXECUTE IMMEDIATE sql_stmt INTO v_outvalue USING searchvalue;
...
The getfield and searchfield are in one installation always the same (but has other values in another installation, so that is why we use dynamic sql)
So this leaves us with an sql that only differs in the searchvalue (which is a parameter).
This function is called in a loop that executes x times, from inside another stored procedure.
The stored procedure is executed y times during the connection life time, through ODBC connection.
And there are z connections, but each of them uses the same database login.
Now let us also assume that the searchvalue changes b times during one loop.
Question 1:
When calculating how many copies of the sql will be kept in the library cache,
can we disregard the different values the searchvalue can have (b), as the value is sent as a parameter to execute immediate?
Question 2:
Will the loop cause a hard parse of the query x times (query will be created in library cache x times), or can Oracle reuse the query?
(We assume that the searchvalue is the same for all calls in this question here, for simplicity)
Question 3:
Does the y (number of times the stored procedure is called from odbc during the lifetime of one connection)
also multiply the amount of copies of the query that are kept in library cache?
Question 4:
Does the z (number of simultaneous connections with same db login)
multiply the amount of copies of the query that are kept in library cache?
Main question:
What behaviour should I expect here?
Is the behaviour configurable?
The cause for this question is that we have had this code is production for 4 years, and now one of our customer gets back to us and says "This query fills our whole SGA, and Oracle says it's your fault".
The number of different combinations of getfield and searchfield should determine how many "copies" there will be. I use teh word "copies" cautiously because Oracke will treat each variation as distinct. Since you are using a bind variable for searchvalue so however many values you have for this will not add to the query count.
In short, it looks like your code is OK.
Number of connections should not increase the hard parses.
Ask for a AWR report to see exactly how many of these queries are in the SGA, and how many hard parses are being triggered.
I will disagree that the number of connections will not increase the hard parse count for the posted code because the last I knew dynamic SQL cannot be shared between sessions. Since the generated SQL uses a bind variable it should generate a reusable statement by the session, but it will not be sharable between user sessions. As a general rule dynamic SQL should be used only for infrequently executed statements. You may want to refer to the following:
- -
Designing applications for performance and scalability An Oracle White Paper July 2005
https://www.oracle.com/technetwork/database/performance/designing-applications-for-performa-131870.pdf
- -
enter code here
I want to insert the same long string into all cells in certain column, which is CLOB type.
It said I should use "bind variables" to do it. So i Googled this:
variable xmlstuff CLOB;
exec :xmlstuff := '<?xml version="1.0"?> ... really long xml...';
UPDATE TABLE_NAME SET COLUMN_NAME = '&&xmlstuff';
Now it still says
The string literal is longer than 4000 characters.
What is the proper use of bind variable in this case?
If you are programming using C# or Java - just use the OracleCLOB object, and it will do all the necessary steps.
If you want to use a CLOB in SQL or PL/SQL,
you need to allocate it, and release it after using.search for DBMS_LOB information.
Regarding the 4000 bytes limit - this is a varchar2 limit within SQL.
to bypass this - you can use PL/SQL - which limits you to a varchar of 32KB, which is not as near to the 4GB that you can hold in a CLOB, but that is the limit for "automatic" creation of a CLOB.
if your string is longer than 32K - you'll have to use a DBMS_LOB to load the data into the clog object, by using append on the clob object.
this is the fastest link I found about how to do it: http://geekswithblogs.net/robertphyatt/archive/2010/03/24/write-read-and-update-oracle-clobs-with-plsql.aspx
I wanted to answer fast, so please let me know if you cannot solve your issue after getting this information - and I'll try to explain it better.
I have a requirement to use the built in REPLACE function on a CLOB variable as part of a larger PL/SQL process. I'm using Oracle 11g R2 and the function works OK, in that it does the replace as required, but as the procedure runs (there are around 2.5 millions records to process), it slows down badly - as in:
first 20,000 records: ~12 minutes
second 20,000 records: ~24 minutes
third 20,000 records: ~37 minutes
fourth 20,000 records: ~52 minutes
etc...
Checking V$TEMPORARY_LOBS during operation shows that the value for CACHE_LOBS increases with every row processed - my assumption is that this implies that memory associated with LOBS (CLOBS in this case) is not getting released once it has been used...?
Stepping through the code using PL/SQL debugger reveals that the value for CACHE_LOBS increases by 2 for every call to the REPLACE function. The function calls are along the lines of:
clobRTFText CLOB;
...
dbms_lob.createtemporary(clobRTFText, TRUE, dbms_lob.call);
...
clobRTFText := REPLACE(clobRTFText, '<CR>', '\par '); <== Causes CACHE_LOBS to increase by 2
...
dbms_lob.freetemporary(clobRTFText); <== Doesn't seem to cause CACHE_LOBS to decrease
It's as though the third line of code above is creating further CLOB variables on the fly. Is that because there is some kind of implicit type conversion occurring due to the REPLACE function expecting a VARCHAR2 parameter? I've tried using dbms_lob.copy instead of "clobRTFText := REPLACE...etc", but it actually was worse (i.e. CACHE_LOBS increased even quicker). Whatever the reason, the call to dbms_lob.freetemporary doesn't seem to make any difference to the value of CACHE_LOBS.
I've gone through the PL/SQL Semantics for LOBs section of the Oracle documentation - it mentions the way CLOB and VARCHAR2 variables can be used in built-in functions but I can't find anything about doing so potentially causing extra memory usage.
Does anyone have any ideas why this is happening or how I could do it (i.e. use REPLACE with a CLOB) without it failing to releasing memory (assuming that is indeed what is happening)?
Thanks
Why do this procedurally? It seems a declarative approach fulfills the requirements.
UPDATE clob_table SET clob_column = REPLACE(clob_column, '<CR>', '\par ');
You can supply whatever WHERE clause that suits you.
I have a statement such as below but its padded out to do 1000 calls at a time. Anything over that throws a PLS-123 error Program Too Large Diana Nodes
begin
sp_myprocedure(....)
sp_myprocedure(....)
sp_myprocedure(....)
sp_myprocedure(....)
end
We are moving to 11g and I was wondering if this limitation could be increased to 2000 for example.
Thanks
"I have a statement such as below but its padded out to do 1000 calls
at a time"
This is a very bad programming strategy. Writing the same thing multiple times is a code smell. Anytime we find ourselves programming with cut'n'paste and then a bit of editing is a time when we should stop and ask ourselves, 'hmmm is there a better way to do this?'
"The parameters are different for each stored procedure call"
Yes, but the parameters have to come from somewhere. Presumably at the moment you are hard-coding them one thousand times. Yuck.
A better solution would be to store them in a table. Then you could write a simple loop. Like this:
for prec in ( select p1, p2 from my_parameters
order by id -- if ordering is important
)
loop
sp_myprocedure(prec.p1, prec.p2);
end loop;
Because you are storing the parameters in a table you can have as many calls to that proc as you like, and you are not bound by the Diana node limit.
True you will have to move your parameter values to a table, but it is not harder to maintain data in a table than it is to maintain hardcoded values in source code.
If you're just moving from 10g then I don't believe the limit has changed. So, if you're having problems now then you'll have them again in 11g. Take a look at this Ask Tom article. A general suggestion is to put your procedure in a package. Or, break it down into smaller blocks. If you're only getting the error when running the block which calls the procedure 1000 times and in the procedure on its own then I suggest you try as APC says and loop through it instead as this should reduce the number of nodes.