Passing large BLOBs to Stored Procedure - oracle

I have a simple (example) script to upload a file into the database (Oracle, if it matters):
<cfscript>
param string filename;
if ( FileExists( filename ) )
{
result = new StoredProc(
datasource = "ds",
procedure = "FILE_UPLOAD",
result = "NA",
parameters = [
{ value = FileReadBinary( filename ), type = "in", cfsqltype = "CF_SQL_BLOB" }
]
).execute();
}
</cfscript>
However, the ColdFusion CFML Reference states for FileReadBinary( filepath ):
Note:
This action reads the file into a variable in the local Variables scope. It is not intended for use with large files, such as logs, because they can bring down the server.
If I should not use FileReadBinary( filepath ), how should I upload a large (0.5 - 1Tb) file?

If using Java is an option, then you can pass an InputStream object to a PreparedStatement for filling a Blob field. Something like this, exception handling and all other stuff to be added:
Connection con = someDataSource.getConnection();
String sql = "INSERT INTO MY_TABLE(MY_BLOB) VALUES(?)";
PreparedStatement ps = con.prepareStatement(sql);
InputStream fis = new FileInputStream("MyBigFile.big");
ps.setBlob(1, fis);
ps.executeUpdate();
I think Java will do it using buffers, and not load the whole file into memory.

As suggested by #Galcoholic, you can utilise the underlying Java classes and use CallableStatement.setBlob( int, InputStream ):
<cfscript>
param string filename;
// Get the necessary Java classes:
Files = createObject( 'java', 'java.nio.file.Files' );
Paths = createObject( 'java', 'java.nio.file.Paths' );
// Do not timeout the request
setting requesttimeout = 0;
try {
input = Files.newInputStream( Paths.get( filename, [] ), [] );
connection = createObject( 'java', 'coldfusion.server.ServiceFactory' )
.getDataSourceService()
.getDataSource( 'ds' )
.getConnection()
.getPhysicalConnection();
statement = connection.prepareCall( '{call FILE_UPLOAD(?)}' );
statement.setBlob( JavaCast( 'int', 1 ), input );
statement.executeUpdate()
}
finally
{
if ( isDefined( "statement" ) )
statement.close();
if ( isDefined( "connection" ) )
connection.close();
}
</cfscript>
Note:
Every argument must be supplied for a Java method; so for methods with variable number of arguments then the VARARGS arguments must be passed as an array (or an empty array for no additional arguments).
ColdFusion numeric values will not be implicitly coerced to Java numeric literals so JavaCast( 'int', value ) is required.
Error handling is not included in the above example.
If the files have been uploaded then the "Maximum size of post data" and "Request Throttle Memory" settings in the Admin console will need to be increased from the default sizes to an appropriate limit for the size of the files being uploaded (otherwise coldfusion.util.MemorySemaphore will throw out-of-memory exceptions when it handles the upload before the script gets parsed).

Related

What is the maximum length of Azure blob metadata values?

What is the maximum size for the values in Windows Azure blob metadata?
I can see that the web server will impose a practical upper limit of about 4k.
Max. metadata size is 8K as per the documentation here: https://learn.microsoft.com/en-us/rest/api/storageservices/Setting-and-Retrieving-Properties-and-Metadata-for-Blob-Resources
Metadata names must adhere to the naming rules for C# identifiers.
Names are case-insensitive but are case-insensitive when set or read.
If two or more metadata headers with the same name are submitted for a resource, the Blob service returns status code 400 (Bad Request).
The metadata consists of name/value pairs.
The total size of all metadata pairs can be up to 8KB in size.
Metadata name/value pairs are valid HTTP headers, and so they adhere to all restrictions governing HTTP headers.
Additionally:
As Metadata Names must be valid C# identifiers - though there isn't a formal length limit on identifiers in C#, versions of Visual C# prior to Visual Studio 2015 would complain if an identifier exceeded 511 characters in length.
There is no formal or defined limit to the length of HTTP header names nor HTTP header values - though many web-servers will block requests that have a total header size exceeding 16KB, or even 8KB in some cases.
Each blob metadata entry is represented by its own separate HTTP header with a name prefix of the form x-ms-meta-{yourMetadataName}, so that's 10 characters added for each metadata name.
Here's a client-side validation check you could use:
static void ValidateMetadata( IEnumerable< KeyValuePair<String,String> > blobMetadata )
{
const int AZURE_MD_NAME_PREFIX_LENGTH = 10; // "x-ms-meta-"
Int32 totalLength = 0;
foreach( KeyValuePair<String,String> md in blobMetadata )
{
totalLength += AZURE_MD_NAME_PREFIX_LENGTH + md.Key.Length + m.Value.Length;
if( !IsValidMetadataName( md.Key ) )
{
throw new ArgumentException( message: "Metadata name \"" + md.Key + "\" is invalid." );
}
if( md.Value.Any( c => !IsValidHttpHeaderValueChar( c ) ) || md.Value.Contains("\r\n") )
{
throw new ArgumentException( message: "Metadata value \"" + md.Value + "\" is invalid." );
}
}
if( totalLength > 8192 )
{
throw new ArgumentException( message: "Total length of metadata names and values " + totalLength +" exceeds 8KiB limit." );
}
}
private static Boolean IsValidMetadataName( String name )
{
// https://stackoverflow.com/questions/47687379/what-characters-are-allowed-in-http-header-values
if( String.IsNullOrWhiteSpace( name ) ) return false;
// The intersection of valid HTTP Header Names and C# Identifiers means:
// * First character must be a letter.
// * All other characters must be ASCII letters or digits.
// * Underscores are technically legal, but many HTTP systems reject them: https://stackoverflow.com/questions/22856136/why-http-servers-forbid-underscores-in-http-header-names - this method disallows underscores to be safe, though in practice it will probably work fine.
if( !Char.IsLetter( name[0] ) ) return false;
foreach( Char c in name )
{
bool validChar = Char.IsLetterOrDigit( c ) && c < 127;
if( !validChar ) return false;
}
return true;
}
private static Boolean IsValidHttpHeaderChar( Char c )
{
// Technically a quoted-string can contain almost any character ("quoted-string" in the HTTP spec), but it's unclear if Azure Blob storage supports that or not.
bool isCtl = ( 0 <= c && c <= 31 ) || ( c == 127 );
if( isCtl ) return false;
return true; // This method checks individual chars, so it cannot check for \r\n.
}

mariadb jdbc driver blob update not supported

After I replaced mysql jdbc driver 5.1 with mariadb jdbc driver 1.1.5 and tested the existing code base that connected with MySQL Server 5.0 and MariaDB Server 5.2, everything works fine except a JDBC call to update a blob field in a table.
The blob field contains XML configuration file. It can be read out, and convert to xml and insert some values.
Then convert it to ByteArrayInputStream object, and call the method
statement.updateBinaryStream(columnLabel, the ByteArrayInputStream object, its length)
but an exception is thrown:
Perhaps you have some incorrect SQL syntax?
java.sql.SQLFeatureNotSupportedException: Updates are not supported
at
org.mariadb.jdbc.internal.SQLExceptionMapper.getFeatureNotSupportedException(SQLExceptionMapper.java:165)
at
org.mariadb.jdbc.MySQLResultSet.updateBinaryStream(MySQLResultSet.java:1642)
at
org.apache.commons.dbcp.DelegatingResultSet.updateBinaryStream(DelegatingResultSet.java:511)
I tried updateBlob method, the same exception was thrown.
The code works well with mysql jdbc driver 5.1.
Any suggestions on how to work around with this situation?
See the ticket updating blob with updateBinaryStream, which in commnet states that it isn't supported.
A workaround would be to use two SQL statements. One which is used to select the data and other to update the data. Something like this:
final Statement select = connection.createStatement();
try {
final PreparedStatement update = connection.prepareStatement( "UPDATE table SET blobColumn=? WHERE idColumn=?" );
try {
final ResultSet selectSet = select.executeQuery( "SELECT idColumn,blobColumn FROM table" );
try {
final int id = selectSet.getInt( "idColumn" );
final InputStream stream = workWithSTreamAndRetrunANew( selectSet.getBinaryStream( "blobColumn" ) ) );
update.setBinaryStream( 1,stream );
update.setInt( 2,id );
update.execute();
}
finally {
if( selectSet != null )
selectSet.close();
}
}
finally {
if( update != null )
update.close();
}
}
finally {
if( select != null )
select.close();
}
But be aware that you need some information how to uniquely identify a table entry, in this example the column idColumn was used for that purpose. Furthermore is you stored empty stream in the
database you might get an SQLException.
A simpler work around is using binary literals (like X'2a4b54') and concatenation (UPDATE table SET blobcol = blobcol || X'2a4b54') like this:
int iBUFSIZ = 4096;
byte[] buf = new byte[iBUFSIZ];
int iLength = 0;
int iUpdated = 1;
for (int iRead = stream.read(buf, 0, iBUFSIZ);
(iUpdated == 1) && (iRead != -1) && (iLength < iTotalLength);
iRead = stream.read(buf, 0, iBUFSIZ))
{
String sValue = "X'" + toHex(buf,0,iRead) + "'";
if (iLength > 0)
sValue = sBlobColumn + " || " + sValue;
String sSql = "UPDATE "+sTable+" SET "+sBlobColumn+"= "+sValue;
Statement stmt = connection.createStatement();
iUpdated = stmt.executeUpdate(sSql);
stmt.close();
}

Concatenate variables and create MD5

Using JMeter, I would like to take values from CSV file, concatenate the values and do a MD5 hash on them and then send the value as part of HTTP request using HTTP Request Sampler.
I tried the following but did not get the correct result:
created CSV Data Set Config and added the variables csvVal1,csvVal2,csvVal3;
in the jp#gc-Dummy Sampler i added the following:
${__MD5(${csvval1}+${csvval2}+${csvval3})}
This did not work, what is the right way?
I ended up using BeanShell Preporcessor and used the following script
import java.security.MessageDigest;
String val1 = vars.get("csv_val1");
String val2 = vars.get("csv_val2");
String val3 = vars.get("csv_val3");
String totalString = val1+val2+val3;
MessageDigest md = MessageDigest.getInstance("MD5");
byte[] md5hash = new byte[32];
md.update(totalString.getBytes("utf-8"), 0, totalString.length());
md5hash = md.digest();
StringBuffer sb = new StringBuffer();
for (int i=0;i<md5hash.length;i++) {
String sval = Integer.toHexString((int) md5hash[i] & 0xFF);
if(sval.length()== 1)
{
sval = "0"+sval;
}
sb.append(sval);
}
log.info("tktest: "+ sb);
vars.putObject("MD5Signature", sb.toString());
There's a new function __digest, currently in nightly builds
In your case to save in MD5Signature variable the result of 3 variable use the following:
${__digest(MD5,${csv_val1}${csv_val2}${csv_val3},,,MD5Signature)}

try-with-resources: Does Java make any guarantees about the order of calls to .close()?

When using a try-with-resources in Java 7, are there any guarantees about the order in which .close() is called?
Here's some sample code from Oracle showing this feature:
try (
java.util.zip.ZipFile zf = new java.util.zip.ZipFile(zipFileName);
java.io.BufferedWriter writer = java.nio.file.Files.newBufferedWriter(outputFilePath, charset)
) {
// Enumerate each entry
for (java.util.Enumeration entries = zf.entries(); entries.hasMoreElements();) {
// Get the entry name and write it to the output file
String newLine = System.getProperty("line.separator");
String zipEntryName = ((java.util.zip.ZipEntry)entries.nextElement()).getName() + newLine;
writer.write(zipEntryName, 0, zipEntryName.length());
}
Both zf.close() and writer.close() will be called. Is the order guaranteed?
It is in the opposite order of declaration, closing from the inside to the outside.

Calling any file/script with Russian name from VBScript fails

I am reading a name of the file/script from an utf-8 encoded XML file. This is then passed to VBScript. VBScript starts a second program to which this passed program/script is provided as an argument. When the argument is in English, the VBScript executes successfully.
But if the name read from XML is non-english( Russian in my case ), VBScript fails to find that file.
I am running VBScript from Java code only using "cscript" as I am running it on Windows.
However, if I copy the command fired by Java program to run VBScript, and paste it on command prompt, it executes normally despite the argument name is in non-english language.
I then hardcoded file/script name in VBScript. Changed encoding of VBScript to UCS2-LE, and directly run it from command prompt. It executed normally. It failed to execute for any other encoding used for VBScript. Also the non-english text is displayed as ? in any other encoding than UCS2-LE.
Then I tried to encode file/script name into UTF16-LE in Java and then passed it to VBScript. Irrespective of which encoding was used in VBScript, it fails. Again if I copy the command printed on standard output from Java program and run it from cmd, it executes.
The command printed from Java displays non-english text correctly.
Can anyone please help me to resolve the issue?
Any relative help would be greatly appreciated.
This is what I am doing currently. I need to pass an argument contatining Russian Text to VBScript from Java.
I tried to use two different approaches.
First approach in the code below writes the Russian text in a file using encoding UnicodeLittle. File is found to be in encoding UCS-2LE. And then VBScript reads the value from that file, and script is executed successfully.
In second approach, I tried to directly pass encoded Russian text as argument to script. VbScript fails saying that script can't be opened.This is the approach I want solution for.
Below is the Java code attached.
Any help would be greatly appreciated.
public class CallProgram
{
private static String encodeType = "UnicodeLittle";
private File scriptName = new File( "F:\\Trial Files\\scriptName.txt" );
public static void main(String[] args)
{
CallProgram obj = new CallProgram();
Runtime rt = Runtime.getRuntime();
try
{
**//Approach1 - Writes text to file and calls vbscript which reads text from file and uses it as an argument to a program**
String sName = "D:\\CheckPoints_SCRIPTS\\Менеджер по качеству"; //Russian Text
byte [] encodedByte= sName.getBytes( encodeType );
String testCase = new String( encodedByte, encodeType ); //New string containing russian text in UnicodeLittle encoding...
obj.writeToFile( testCase ); //Writing russian string to file...
String mainStr = "cscript /nologo \"D:\\Program Files\\2.0.1.3\\Adapter\\bin\\scriptRunner_FileRead_Write.vbs\"";
Process proc1 = rt.exec( mainStr );
int exit = proc1.waitFor();
System.out.println( "Exit Value = " + exit );
**//Approach 2 - Passing encoded Russian text directly to VbScript...**
//This is not working for me...
String [] arrArgs = { "cscript", "/nologo", "\"D:\\Program Files\\IBM\\Rational Adapters\\2.0.1.3\\QTPAdapter\\bin\\scriptRunner.vbs\"", testcase };
ProcessBuilder process = new ProcessBuilder( arrArgs );
Process proc2 = process.start();
proc2.waitFor();
}
catch (IOException e)
{
e.printStackTrace();
}
catch ( InterruptedException intue )
{
intue.printStackTrace();
}
}
//Function to write Russian text to file using encoding UnicodeLittle...
private void writeToFile( String testCase )
{
FileOutputStream fos = null;
Writer out = null;
try
{
fos = new FileOutputStream( this.scriptName );
out = new OutputStreamWriter( fos, encodeType );
out.write( testCase );
out.close();
fos.close();
}
catch (FileNotFoundException e)
{
e.printStackTrace();
}
catch ( IOException ioe )
{
ioe.printStackTrace();
}
finally
{
try
{
if ( fos != null )
{
fos.close();
fos = null;
}
if ( out != null)
{
out.close();
out = null;
}
}
catch( IOException ioe )
{
fos = null;
out = null;
}
}
} // End of method writeToFile....
}
I've resolved similar problems before by using the short 8.3-style filename instead of the long filename. I get this short name using the ShortPath method of FileSystemObject. Here's a VBScript example... you may want to try something similar in Java.
Function GetShortPath(strLongPath)
Dim FSO
Dim objFolder
Dim objFile
Dim strShortPath
Set FSO = CreateObject("Scripting.FileSystemObject")
' Is it a file or a folder?
If FSO.FolderExists(strLongPath) Then
' It's a folder.
Set objFolder = FSO.GetFolder(strLongPath)
strShortPath = objFolder.ShortPath
ElseIf FSO.FileExists(strLongPath) Then
' It's a file.
Set objFile = FSO.GetFile(strLongPath)
strShortPath = objFile.ShortPath
Else
' File not found.
strShortPath = ""
End If
GetShortPath = strShortPath
End Function
For example,
Debug.Print GetShortPath("C:\öêåéèüø.çõâ")
returns C:\B373~1, which can be used in place of the long filename with non-English characters. Example with dir /x (reveals the short filename) and notepad:
C:\sandbox>dir /x
Volume in drive C has no label.
Volume Serial Number is BC90-DF37
Directory of C:\sandbox
13/01/2011 15:12 <DIR> .
13/01/2011 15:12 <DIR> ..
13/01/2011 14:52 22 NEWTEX~1.TXT New Text Document.txt
13/01/2011 15:05 0 C7F0~1.TXT öêåéèüø.txt
13/01/2011 15:05 0 B373~1 öêåéèüø.çõâ
3 File(s) 22 bytes
2 Dir(s) 342,158,913,536 bytes free
C:\sandbox>notepad B373~1

Resources