Delete old azure blobs - azure-blob-storage

I want to use the following method to delete 30 day old blobs.
However, it seems that the part of "sourceBlob.getProperties().getLastModified().getTime();" generates an exception.
What could be the possible solutions?
The exception message shows null only.
The Azure Storage type is Storage (general purpose v1)
public static void deleteOldBlobs(String source) {
try {
System.out.println("deleteOldBlobs started");
CloudStorageAccount storageAccount = CloudStorageAccount
.parse(PropertyUtil.getProperty("storageConnectionString"));
CloudBlobClient blobClient = storageAccount.createCloudBlobClient();
CloudBlobContainer sourceContainer = blobClient.getContainerReference(source);
long daysBack = 30;
System.out.println(daysBack);
long cutoff = (daysBack * (24 * 60 * 60 * 1000));
for (ListBlobItem blobItem : sourceContainer.listBlobs()) {
String sourceFileName = new File(blobItem.getUri().toString()).getName();
System.out.println(sourceFileName);
CloudBlockBlob sourceBlob = sourceContainer.getBlockBlobReference(sourceFileName);
System.out.println(sourceBlob.getProperties().getLastModified().getTime());
long diff = new Date().getTime()- sourceBlob.getProperties().getLastModified().getTime();
if (diff > cutoff) {
sourceBlob.deleteIfExists();
}
}
System.out.println("deleteOldBlobs ended");
} catch (Exception ex) {
System.out.println(ex.getMessage());
} finally {
}
}

You'll need to call downloadAttributes() method to populate the properties of the blob.
Your following line of code:
CloudBlockBlob sourceBlob = sourceContainer.getBlockBlobReference(sourceFileName);
is simply creating an instance of CloudBlockBlob with the properties set with default values. When you call downloadAttributes method, a network call will be made and blob's attributes will be fetched.
So your code would be:
CloudBlockBlob sourceBlob = sourceContainer.getBlockBlobReference(sourceFileName);
sourceBlob.downloadAttributes();
System.out.println(sourceBlob.getProperties().getLastModified().getTime());
Considering you have already listed the blobs, you can cast your blobItem as a CloudBlob and then you should not have to fetch the attributes (which makes a network call and will make the entire process much slower and error prone).

Related

How to get speed of network, whether it is fast or slow in xamarin iOS?

How to get speed of network, whether it is fast or slow in xamarin iOS?
I used NetworkReachability, But it is giving result that url is reachable or not?
I want to get speed of network , fast or poor?` private static NetworkReachability _defaultRouteReachability;
public static event EventHandler ReachabilityChanged;
public static bool IsNetworkAvailable(string url)
{
if (_defaultRouteReachability == null)
{
_defaultRouteReachability = new NetworkReachability(url);
_defaultRouteReachability.SetNotification(OnChange);
_defaultRouteReachability.Schedule(CFRunLoop.Current, CFRunLoop.ModeDefault);
}
NetworkReachabilityFlags flags;
return _defaultRouteReachability.TryGetFlags(out flags) &&
IsReachableWithoutRequiringConnection(flags);
}
private static bool IsReachableWithoutRequiringConnection(NetworkReachabilityFlags flags)
{
// Is it reachable with the current network configuration?
bool isReachable = (flags & NetworkReachabilityFlags.Reachable) != 0;
// Do we need a connection to reach it?
bool noConnectionRequired = (flags & NetworkReachabilityFlags.ConnectionRequired) == 0;
// Since the network stack will automatically try to get the WAN up,
// probe that
if ((flags & NetworkReachabilityFlags.IsWWAN) != 0)
noConnectionRequired = true;
return isReachable && noConnectionRequired;
}
private static void OnChange(NetworkReachabilityFlags flags)
{
var h = ReachabilityChanged;
if (h != null)
h(null, EventArgs.Empty);
}`
Something like this can help you, unless you want to use a library. This basically gives you the technical definition of internet speed, but the real number will be a little bigger. It's very similar to the solution suggested by #Martheen
public async Task<string> CheckInternetSpeed()
{
//DateTime Variable To Store Download Start Time.
DateTime dt1 = DateTime.Now;
string internetSpeed;
try
{
// Create Object Of WebClient
var client = new HttpClient();
//Number Of Bytes Downloaded Are Stored In ‘data’
byte[] data = await client.GetByteArrayAsync("https://www.example.com/");
//DateTime Variable To Store Download End Time.
DateTime dt2 = DateTime.Now;
//To Calculate Speed in Kb Divide Value Of data by 1024 And Then by End Time Subtract Start Time To Know Download Per Second.
Console.WriteLine("ConnectionSpeed: DataSize (kb) " + data.Length / 1024);
Console.WriteLine("ConnectionSpeed: ElapsedTime (secs) " + (dt2 - dt1).TotalSeconds);
internetSpeed = "ConnectionSpeed: (kb/s) " + Math.Round((data.Length / 1024) / (dt2 - dt1).TotalSeconds, 2);
}
catch (Exception ex)
{
internetSpeed = "ConnectionSpeed:Unknown Exception-" + ex.Message;
}
Console.WriteLine(internetSpeed);
return internetSpeed;
}
Xamarin Essentials library can't get you Internet speed, but if you use dependency injection, you can use the native api s like this.

Create AIScene instance from the file's content

I'm writing a Java web service where it is possible to upload a 3D object, operate on it and store it.
What I'm trying to do is creating an AIScene instance using a byte[] as an input parameter which is the file itself (it's content).
I have found no way to do this in the docs, all import methods require a path.
Right now I'm taking a look at both the lwjgl java version of Assimp as well as the C++ version. It doesn't matter which one is used to solve the issue.
Edit: the code I'm trying to get done:
#Override
public String uploadFile(MultipartFile file) {
AIFileIO fileIo = AIFileIO.create();
AIFileOpenProcI fileOpenProc = new AIFileOpenProc() {
public long invoke(long pFileIO, long fileName, long openMode) {
AIFile aiFile = AIFile.create();
final ByteBuffer data;
try {
data = ByteBuffer.wrap(file.getBytes());
} catch (IOException e) {
throw new RuntimeException();
}
AIFileReadProcI fileReadProc = new AIFileReadProc() {
public long invoke(long pFile, long pBuffer, long size, long count) {
long max = Math.min(data.remaining(), size * count);
memCopy(memAddress(data) + data.position(), pBuffer, max);
return max;
}
};
AIFileSeekI fileSeekProc = new AIFileSeek() {
public int invoke(long pFile, long offset, int origin) {
if (origin == Assimp.aiOrigin_CUR) {
data.position(data.position() + (int) offset);
} else if (origin == Assimp.aiOrigin_SET) {
data.position((int) offset);
} else if (origin == Assimp.aiOrigin_END) {
data.position(data.limit() + (int) offset);
}
return 0;
}
};
AIFileTellProcI fileTellProc = new AIFileTellProc() {
public long invoke(long pFile) {
return data.limit();
}
};
aiFile.ReadProc(fileReadProc);
aiFile.SeekProc(fileSeekProc);
aiFile.FileSizeProc(fileTellProc);
return aiFile.address();
}
};
AIFileCloseProcI fileCloseProc = new AIFileCloseProc() {
public void invoke(long pFileIO, long pFile) {
/* Nothing to do */
}
};
fileIo.set(fileOpenProc, fileCloseProc, NULL);
AIScene scene = aiImportFileEx(file.getName(),
aiProcess_JoinIdenticalVertices | aiProcess_Triangulate, fileIo); // ISSUE HERE. file.getName() is not a path, just a name. so is getOriginalName() in my case.
try{
Long id = scene.mMeshes().get(0);
AIMesh mesh = AIMesh.create(id);
AIVector3D vertex = mesh.mVertices().get(0);
return mesh.mName().toString() + ": " + (vertex.x() + " " + vertex.y() + " " + vertex.z());
}catch(Exception e){
e.printStackTrace();
}
return "fail";
}
When debugging the method I get an access violation in the method that binds to the native:
public static long naiImportFileEx(long pFile, int pFlags, long pFS)
this is the message:
#
A fatal error has been detected by the Java Runtime Environment:
#
EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000007400125d, pid=6400, tid=0x0000000000003058
#
JRE version: Java(TM) SE Runtime Environment (8.0_201-b09) (build 1.8.0_201-b09)
Java VM: Java HotSpot(TM) 64-Bit Server VM (25.201-b09 mixed mode windows-amd64 compressed oops)
Problematic frame:
V [jvm.dll+0x1e125d]
#
Failed to write core dump. Minidumps are not enabled by default on client versions of Windows
#
An error report file with more information is saved as:
C:\Users\ragos\IdeaProjects\objectstore3d\hs_err_pid6400.log
#
If you would like to submit a bug report, please visit:
http://bugreport.java.com/bugreport/crash.jsp
#
It is possible if we use the aiImportFileFromMemory method.
The approach I wanted to follow was copied from a github demo and actually copies the buffer around unnecessarily.
The reason for the access violation was the use of indirect buffers (for more info why that is a problem, check this out).
The solution is not nearly as complicated as the code I initially pasted:
#Override
public String uploadFile(MultipartFile file) throws IOException {
ByteBuffer buffer = BufferUtils.createByteBuffer((int) file.getSize());
buffer.put(file.getBytes());
buffer.flip();
AIScene scene = Assimp.aiImportFileFromMemory(buffer,aiProcess_Triangulate, (ByteBuffer) null);
Long id = scene.mMeshes().get(0);
AIMesh mesh = AIMesh.create(id);
AIVector3D vertex = mesh.mVertices().get(0);
return mesh.mName().dataString() + ": " + (vertex.x() + " " + vertex.y() + " " + vertex.z());
}
Here I create a direct buffer with the appropriate size, load the data and flip it (this part is a must.) After that let Assimp do its magic so you get pointers to the structure. With the return statement I just check if I got the valid data.
edit
As in the comments it was pointed out, this implementation is limited to a single file upload and assumes it gets everything that is necessary from that one MultipartFile, it won't work well with referenced formats. See docs for more detail.
The demo that was linked in the question's comments which was used in the question as a base has a different use case to my original one.

Azure CloudBlockBlob.downloadText() operation failing with Blob hash mismatch (integrity check failed), Expected value is xxxx, retrieved xxx

Azure Storage Java SDK CloudBlockBlob.downloadText() operation failing with error: "Blob hash mismatch (integrity check failed), Expected value is xxxx, retrieved xxx".
I'm using azure storage sdk version: azure-storage:8.0.0. Here is the code snippet.
List<BlockEntry> blockList = new ArrayList<BlockEntry>();
File parent = new File(System.getProperty("java.io.tmpdir"));
File temp = new File(parent, "sample-file.txt");
String blockText="hellow world";
if (temp.exists()) {
temp.delete();
}
try {
temp.createNewFile();
CloudBlockBlob blob = container.getBlockBlobReference(temp.getName());
blob.uploadFromFile(temp.getAbsolutePath());
//Add Block to the created blob
String blockId1 = Base64.encode("B1".getBytes());
InputStream inputStream1 = new ByteArrayInputStream(blockText.getBytes());
long len = -1;
BlockEntry block1 = new BlockEntry(blockId1);
blockList.add(block1);
blockBlob.uploadBlock(blockId1, inputStream1, len);
blockBlob.commitBlockList(blockList);
//Print the blob content
System.out.println("Printing the block content" + blob.downloadText());
} catch (IOException ex) {
ex.printStackTrace();
}

Upload large files to Azure Media Services in Xamarin

I am trying to upload a .mp4 file, selected from the user's iOS or Android device, to my Azure Media Services account.
This code works for small files ( less than ~95MB):
public static async Task<string> UploadBlob(string blobContainerSasUri, string blobName, byte[] blobContent, string path)
{
string responseString;
int contentLength = blobContent.Length;
string queryString = (new Uri(blobContainerSasUri)).Query;
string blobContainerUri = blobContainerSasUri.Split('?')[0];
string requestUri = string.Format(System.Globalization.CultureInfo.InvariantCulture, "{0}/{1}{2}", blobContainerUri, blobName, queryString);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestUri);
request.Method = "PUT";
request.AllowWriteStreamBuffering = false;
request.Headers.Add("x-ms-blob-type", "BlockBlob");
request.ContentLength = contentLength;
request.Timeout = Int32.MaxValue;
request.KeepAlive = true;
int bufferLength = 1048576; //upload 1MB at time, useful for a simple progress bar.
Stream requestStream = request.GetRequestStream();
requestStream.WriteTimeout = Int32.MaxValue;
ProgressViewModel progressViewModel = App.Locator.GetProgressBar(App.Locator.MainViewModel.currentModuleItemId);
MyVideosPage myVideosPage = App.Locator.GetVideosPage(App.Locator.MainViewModel.currentModuleItemId);
FileStream fileStream = new FileStream(path, FileMode.Open, FileAccess.Read);
int nRead = 0;
int currentPos = 0;
while ((nRead = fileStream.Read(blobContent, currentPos, bufferLength)) > 0)
{
await requestStream.WriteAsync(blobContent, currentPos, nRead);
currentPos += nRead;
}
fileStream.Close();
requestStream.Close();
HttpWebResponse objHttpWebResponse = null;
try
{
// this is where it fails for large files
objHttpWebResponse = (HttpWebResponse)request.GetResponse();
Stream responseStream = objHttpWebResponse.GetResponseStream();
StreamReader stream = new StreamReader(responseStream);
responseString = stream.ReadToEnd();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
if (objHttpWebResponse != null)
objHttpWebResponse.Close();
}
return responseString;
}
An exception is thrown after this line is called:
(HttpWebResponse)request.GetResponse();
The exception message is "The request body is too large and exceeds the maximum permissible limit."
The exception StatusCode is "RequestEntityTooLarge".
How can I upload large files? Is this a problem with HttpWebRequest, or Azure Media Services?
Azure Storage supports one shot upload (aka PutBlob API) up to 256MB if you are using the new REST API versions. But since you are not specifying the REST API version, you're defaulting to a very old version where the maximum supported size of one shot upload is 100MB.
Use x-ms-version: 2018-03-28 header to be able to upload up to 256MB in one HTTP request.
If you have to deal with larger files, you will need to use block & commit upload. You will need to use PutBlock API to stage blocks from the source file. Blocks can be up to 100MB each. Then you need to commit all the blocks using the PutBlockList API. If you don't have to deal with this logic yourself, simply use the Azure Storage SDK for .NET (supports Xamarin) and use the uploadFromFile method. It is simple, and resilient.

ResultSet.getBlob() Exception

The Code:
ResultSet rs = null;
try {
conn = getConnection();
stmt = conn.prepareStatement(sql);
rs = stmt.executeQuery();
while (rs.next()) {
Blob blob = rs.getBlob("text");
byte[] blobbytes = blob.getBytes(1, (int) blob.length());
String text = new String(blobbytes);
The result:
java.sql.SQLException: Invalid column type: getBLOB not implemented for class oracle.jdbc.driver.T4CClobAccessor
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:111)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:145)
at oracle.jdbc.driver.Accessor.unimpl(Accessor.java:357)
at oracle.jdbc.driver.Accessor.getBLOB(Accessor.java:1299)
at oracle.jdbc.driver.OracleResultSetImpl.getBLOB(OracleResultSetImpl.java:1280)
at oracle.jdbc.driver.OracleResultSetImpl.getBlob(OracleResultSetImpl.java:1466)
at oracle.jdbc.driver.OracleResultSet.getBlob(OracleResultSet.java:1978)
I have class12_10g.zip in my class path. I've googled and have found essentially only one site on this particular problem, and it wasn't helpful at.
Does anyone have any ideas on this?
A little background:
We were converting one of our databases from MySQL to Oracle. Within the MySQL DB, one of the fields is a longtext which is treated as a BLOB in the code. The SQL developer workbench by default converts longtext to CLOB (make sense to me) but the code was expecting Blob. I guess the error wasn't that nice: oracle.jdbc.driver.T4CClobAccessor (though it does mention Clob).
When I tried the following:
rs = stmt.executeQuery();
while (rs.next()) {
byte[] blobbytes = rs.getBytes("text");
String text = new String(blobbytes);
}
it threw an unsupported exception - all I had to do in the first place was compare the types in the newly created Oracle DB with what the code was expecting (unfortunately I just assumed they would match).
Sorry guys! Not that I've put much thought into it, now I have to figure out why the original developers used BLOB types for longtext
Not sure about making the Blob object work -- I typically skip the Blob step:
rs = stmt.executeQuery();
while (rs.next()) {
byte[] blobbytes = rs.getBytes("text");
String text = new String(blobbytes);
}
try to use the latest version of the drivers (10.2.0.4). Try also the drivers for JDK 1.4/1.5 since classes12 are for JDK 1.2/1.3.
Try...
PreparedStatement stmt = connection.prepareStatement(query);
ResultSet rs = stmt.executeQuery();
rs.next();
InputStream is = rs.getBlob(columnIndex).getBinaryStream();
...instead?
I have a utility method in a DAO superclass of all my DAOs:
protected byte[] readBlob(oracle.sql.BLOB blob) throws SQLException {
if (blob != null) {
byte[] buffer = new byte[(int) blob.length()];
int bufsz = blob.getBufferSize();
InputStream is = blob.getBinaryStream();
int len = -1, off = 0;
try {
while ((len = is.read(buffer, off, bufsz)) != -1) {
off += len;
}
} catch (IOException ioe) {
logger.debug("IOException when reading blob", ioe);
}
return buffer;
} else {
return null;
}
}
// to get the oracle BLOB object from the result set:
oracle.sql.BLOB blob= (oracle.sql.BLOB) ((OracleResultSet) rs).getBlob("blobd");
Someone will now say "why didn't you just do XYZ", but there was some issue at the time that made the above more reliable.
When JDBC returns a ResultSet from an Oracle database it always returns an OracleResultSet. If you are typing it as a ResultSet, java upcasts it to the standard SQL ResultSet.
OracleResultSet overrides most of the data type methods, because Oracle datatypes are not standard SQL types.
In other words, that worked because you cast the rs as an OracleResultSet, and used it's getBlob method.

Resources