hdfs delete a file being written - hadoop

Acording the blog, hdfs uses the lease mechanism to avoid two client writing the same file. So I think one can not delete the file which is written by the other client. Howerver, it's wrong.
When client A is writing the lock.txt, client B can delete lock.txt immediately. Client A can continue writing although the file no longer exists. Just when closing the stream, A will encounter the exception:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /user/lock.txt (inode 643845185): File does not exist. Holder DFSClient_NONMAPREDUCE_-1636584585_1 does not have any open files**
Why this happen? My haddop version is 2.7.3.
================================
This is my test codeļ¼š
// write process
object Create {
private val filePath = "/user/lock.txt"
def main(args: Array[String]): Unit = {
println("Create start!")
val fs = FileSystem.get(new Configuration())
val os = Option(fs.create(new Path(filePath), false))
if (os.isDefined) {
println(s"Create result! $os", System.currentTimeMillis())
0.until(300).foreach(index => {
os.get.write(100)
os.get.flush()
println("Writing...")
Thread.sleep(1000)
})
println("pre close" + System.currentTimeMillis())
os.get.close()
println("close success!" + System.currentTimeMillis())
}
}
}
// delete process
object Delete {
private val filePath = "/user/lock.txt"
def main(args: Array[String]): Unit = {
println("Delete start!")
val fs = FileSystem.get(new Configuration())
while (!fs.exists(new Path(filePath))) {
println("File no exist!")
Thread.sleep(1000)
}
println("File exist!")
while (true) {
println("try delete!")
val tmp = Option(fs.delete(new Path(filePath), false))
if (tmp.isDefined) {
println(s"delete result:${tmp.get}!" + System.currentTimeMillis())
}
println("Try recover")
if (fs.asInstanceOf[DistributedFileSystem].recoverLease(new Path(filePath))) {
println("Recover lease success!")
val res = Option(fs.delete(new Path(filePath), false))
println(s"File delete success:${res.get}")
} else {
println("Recover lease failed!")
}
Thread.sleep(1000)
}
}
}

Related

Kotlin : copy image to gallery

I am trying to copy an image that is stored in my application folder to a predefined folder in my gallery.
I started from an image sharing code..
This is my code :
val extension = when (requireNotNull(pictureResult).format) {
PictureFormat.JPEG -> "jpg"
PictureFormat.DNG -> "dng"
else -> throw RuntimeException("Unknown format.")
}
val timestamp = System.currentTimeMillis()
val namePhoto = "picture_"+timestamp+"."+extension;
val destFile = File(filesDir, namePhoto)
val folder = "/CustomFolder"
CameraUtils.writeToFile(requireNotNull(pictureResult?.data), destFile) { file ->
if (file != null) {
// Code to share - it works
/*
val context = this#PicturePreviewActivity
val intent = Intent(Intent.ACTION_SEND)
intent.type = "image/*"
val uri = FileProvider.getUriForFile(context, context.packageName + ".provider", file)
intent.putExtra(Intent.EXTRA_STREAM, uri)
intent.addFlags(Intent.FLAG_GRANT_READ_URI_PERMISSION)
startActivity(intent)
*/
*/
// Code to save image to gallery - doesn't work :(
val photoDirectory = File(Environment.DIRECTORY_PICTURES+folder, namePhoto)
val sourcePath = Paths.get(file.toURI())
Log.i("LOG","sourcePath : "+sourcePath.toString()) // /data/user/0/com.app.demo/files/picture_1663772068143.jpg
val targetPath = Paths.get(photoDirectory.toURI())
Log.i("LOG","targetPath : "+targetPath.toString()) // /Pictures/CustomFolder/picture_1663772068143.jpg
Files.move(sourcePath, targetPath, StandardCopyOption.REPLACE_EXISTING)
// Error here but I don't know why
} else {
Toast.makeText(this#PicturePreviewActivity, "Error while writing file.", Toast.LENGTH_SHORT).show()
}
}
How do I copy the image to a predefined folder?
Ok, I did it !
Solution :
val folder = "/CustomFolder/" // name of your folder
val timestamp = System.currentTimeMillis()
val namePicture = "picture_"+timestamp+"."+extension;
try {
val path = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_PICTURES+folder);
if (!path.exists()) {
path.mkdir();
}
val pathImage = File(path,namePicture)
val stream = FileOutputStream(pathImage)
stream.write(imageByteArray).run {
stream.flush()
stream.close()
}
} catch (e: FileNotFoundException) {
e.printStackTrace()
}

Kotlin-native execute command and get the output

I'd like to know if there's a way in kotlin native to call a command via posix and receive it's terminal output. For example, I'd like to get the "git diff" command working without having to create a temporary file, write output to it and then read from that file.
On SO I've only found solutions requiring ProcessBuilder, which isn't available on kotlin-native, as it's a Java library.
I found a working piece of code I wanted to use, so I'm posting it here for future viewers!
fun executeCommand(command: String): String{
val fp: CPointer<FILE>? = popen(command, "r")
val buffer = ByteArray(4096)
val returnString = StringBuilder()
/* Open the command for reading. */
if (fp == NULL) {
printf("Failed to run command\n" )
exit(1)
}
/* Read the output a line at a time - output it. */
var scan = fgets(buffer.refTo(0), buffer.size, fp)
if(scan != null) {
while (scan != NULL) {
returnString.append(scan!!.toKString())
scan = fgets(buffer.refTo(0), buffer.size, fp)
}
}
/* close */
pclose(fp)
return returnString.trim().toString()
}
It's an improved version of exec command for Kotlin Native posted by mg-lolenstine, it throws an exception with command stderr instead of just returning exit(1) (which not always desirable behavior itself), also trim is now optional
import kotlinx.cinterop.*
import platform.posix.*
fun executeCommand(
command: String,
trim: Boolean = true,
redirectStderr: Boolean = true
): String {
val commandToExecute = if (redirectStderr) "$command 2>&1" else command
val fp = popen(commandToExecute, "r") ?: error("Failed to run command: $command")
val stdout = buildString {
val buffer = ByteArray(4096)
while (true) {
val input = fgets(buffer.refTo(0), buffer.size, fp) ?: break
append(input.toKString())
}
}
val status = pclose(fp)
if (status != 0) {
error("Command `$command` failed with status $status${if (redirectStderr) ": $stdout" else ""}")
}
return if (trim) stdout.trim() else stdout
}
Usually you can use the POSIX api and use fork and wait and some I/O related functions for your purpose
fun main() {
val childPid: pid_t = fork()
if (childPid == 0) {
val commands = listOf("git", "diff", "HEAD^1", "$projectDir/path/to/file", null)
val cwd = "$projectDir"
chdir(cwd)
memScoped {
execvp(commands[0], allocArrayOf(commands.map { it?.cstr?.ptr }))
}
} else {
wait(null)
}
}
Of course, this needs to deal with a lot of c-style code, so I also wrote a more practical library for this
repositories {
mavenCentral()
}
// add dependencies into your native target sourceSet
dependencies {
implementation("com.kgit2:kommand:0.1.4")
}
It is also very simple to use
fun main(args: Array<String>) {
val diffResult = Command("git")
.args("diff", "HEAD^1", "$projectDir/path/to/file")
.cwd("$projectDir")
.spawn()
.output()
}
playing around I used above's answers to create a working gradle kotlin native/jvm multiplatform multiproject that runs arbitrary local Processes/Commands:
here's my result:
https://github.com/hoffipublic/minimal_kotlin_multiplatform
import kotlinx.cinterop.refTo
import kotlinx.cinterop.toKString
import platform.posix.fgets
import platform.posix.pclose
import platform.posix.popen
actual object MppProcess : IMppProcess {
actual override fun executeCommand(
command: String,
redirectStderr: Boolean
): String? {
val commandToExecute = if (redirectStderr) "$command 2>&1" else command
val fp = popen(commandToExecute, "r") ?: error("Failed to run command: $command")
val stdout = buildString {
val buffer = ByteArray(4096)
while (true) {
val input = fgets(buffer.refTo(0), buffer.size, fp) ?: break
append(input.toKString())
}
}
val status = pclose(fp)
if (status != 0) {
error("Command `$command` failed with status $status${if (redirectStderr) ": $stdout" else ""}")
}
return stdout
}
}
on jvm
import java.util.concurrent.TimeUnit
actual object MppProcess : IMppProcess {
actual override fun executeCommand(
command: String,
redirectStderr: Boolean
): String? {
return runCatching {
ProcessBuilder(command.split(Regex("(?<!(\"|').{0,255}) | (?!.*\\1.*)")))
//.directory(workingDir)
.redirectOutput(ProcessBuilder.Redirect.PIPE)
.apply { if (redirectStderr) this.redirectError(ProcessBuilder.Redirect.PIPE) }
.start().apply { waitFor(60L, TimeUnit.SECONDS) }
.inputStream.bufferedReader().readText()
}.onFailure { it.printStackTrace() }.getOrNull()
}
}

How to make file downloadable from server to Front End in Spring Boot project, Created Rest API for file downloading but not functioning as expected?

In general, I have created rest controller for both uploading and downloading a file from front end (React) to our file system. Uploading works great, as expected. However, downloading rest api not functioning properly. it downloads only 12kb or smth like that of the file. Is there anything [configs] that I am missing in my project or what? Please help! any comment or suggestion would be appreciated thanks in advance
#GetMapping("/get")
fun getFile(#Valid data: FileDeleteDTO): ResponseEntity<Resource>{
val header = HttpHeaders()
val fileGetFromDb = baseFileUploaderAttachmentService.getByUid(data.qquuid)
if (!fileGetFromDb.isPresent)
throw FileNotFoundException()
val pathFileName = fileGetFromDb.get().filename + '.' + fileGetFromDb.get().extension
val originalFileName = fileGetFromDb.get().originalName + '.' + fileGetFromDb.get().extension
// val filePath = UPLOAD_ROOT_FOLDER + fileGetFromDb.get().path + pathFileName
val filePath = UPLOAD_ROOT_FOLDER + fileGetFromDb.get().path
// val file = File(filePath)
header.contentType = (MediaType.valueOf(fileGetFromDb.get().mime_type!!))
header.contentLength = fileGetFromDb.get().size
header.set("Content-Disposition", "attachment; filename=$originalFileName")
return ResponseEntity.ok()
.headers(header)
.body(loadFileAsResource(pathFileName,Paths.get(filePath)))
}
fun loadFileAsResource(fileName: String, fileStorageLocation: Path): Resource {
try {
val filePath = fileStorageLocation.resolve(fileName).normalize()
val resource = UrlResource(filePath.toUri())
return if (resource.exists()) {
resource
} else {
throw FileNotFoundException("File not found $fileName")
}
} catch (ex: MalformedURLException) {
throw Exception("File not found $fileName", ex)
}
}
I have no idea what happened but i have solved above problem by adding some header elements like below:
#RequestMapping(value = ["/download"], method = [RequestMethod.GET])
#Throws(IOException::class)
fun downloadFile(#Valid data: FileDeleteDTO): ResponseEntity<Any> {
val fileGetFromDb = baseFileUploaderAttachmentService.getByUid(data.qquuid)
val pathFileName = fileGetFromDb.get().filename + '.' + fileGetFromDb.get().extension
val originalFileName = fileGetFromDb.get().originalName + '.' + fileGetFromDb.get().extension
val filePath = UPLOAD_ROOT_FOLDER + fileGetFromDb.get().path + pathFileName
val file = File(filePath)
val resource = InputStreamResource(FileInputStream(file))
val headers = HttpHeaders()
headers.add("Content-Disposition", String.format("attachment; filename=\"%s\"", originalFileName))
headers.add("Cache-Control", "no-cache, no-store, must-revalidate")
headers.add("Pragma", "no-cache")
headers.add("Expires", "0")
return ResponseEntity.ok().headers(headers).contentLength(
file.length()
).contentType((MediaType.valueOf(fileGetFromDb.get().mime_type!!))).body(resource)
}
Just wondering does Cache control has impact on this issue? Where can i learn about headers?

Unzip a file in kotlin script [.kts]

I was toying with the idea of rewriting some existing bash scripts in kotlin script.
One of the scripts has a section that unzips all the files in a directory. In bash:
unzip *.zip
Is there a nice way to unzip a file(s) in kotlin script?
The easiest way is to just use exec unzip (assuming that the name of your zip file is stored in zipFileName variable):
ProcessBuilder()
.command("unzip", zipFileName)
.redirectError(ProcessBuilder.Redirect.INHERIT)
.redirectOutput(ProcessBuilder.Redirect.INHERIT)
.start()
.waitFor()
The different approach, that is more portable (it will run on any OS and does not require unzip executable to be present), but somewhat less feature-full (it will not restore Unix permissions), is to do unzipping in code:
import java.io.File
import java.util.zip.ZipFile
ZipFile(zipFileName).use { zip ->
zip.entries().asSequence().forEach { entry ->
zip.getInputStream(entry).use { input ->
File(entry.name).outputStream().use { output ->
input.copyTo(output)
}
}
}
}
If you need to scan all *.zip file, then you can do it like this:
File(".").list { _, name -> name.endsWith(".zip") }?.forEach { zipFileName ->
// any of the above approaches
}
or like this:
import java.nio.file.*
Files.newDirectoryStream(Paths.get("."), "*.zip").forEach { path ->
val zipFileName = path.toString()
// any of the above approaches
}
this code is for unziping from Assets
1.for unzping first u need InputStream
2.put it in ZipInputStream
3.if directory is not exist u have to make by .mkdirs()
private val BUFFER_SIZE = 8192//2048;
private val SDPath = Environment.getExternalStorageDirectory().absolutePath
private val unzipPath = "$SDPath/temp/zipunzipFile/unzip/"
var count: Int
val buffer = ByteArray(BUFFER_SIZE)
val context: Context = this
val am = context.getAssets()
val stream = context.getAssets().open("raw.zip")
try {
ZipInputStream(stream).use { zis ->
var ze: ZipEntry
while (zis.nextEntry.also { ze = it } != null) {
var fileName = ze.name
fileName = fileName.substring(fileName.indexOf("/") + 1)
val file = File(unzipPath, fileName)
val dir = if (ze.isDirectory) file else file.getParentFile()
if (!dir.isDirectory() && !dir.mkdirs())
throw FileNotFoundException("Invalid path: " + dir.getAbsolutePath())
if (ze.isDirectory) continue
val fout = FileOutputStream(file)
try {
while ( zis.read(buffer).also { count = it } != -1)
fout.write(buffer, 0, count)
} finally {
val fout : FileOutputStream =openFileOutput(fileName, Context.MODE_PRIVATE)
fout.close()
}
}
for unziping from externalStorage:
private val sourceFile= "$SDPath/unzipFile/data/"
ZipInputStream zis = null;
try {
zis = new ZipInputStream(new BufferedInputStream(new
FileInputStream(sourceFile)));
ZipEntry ze;
int count;
byte[] buffer = new byte[BUFFER_SIZE];
while ((ze = zis.getNextEntry()) != null) {
String fileName = ze.getName();
fileName = fileName.substring(fileName.indexOf("/") + 1);
File file = new File(destinationFolder, fileName);
File dir = ze.isDirectory() ? file : file.getParentFile();
if (!dir.isDirectory() && !dir.mkdirs())
throw new FileNotFoundException("Invalid path: " +
dir.getAbsolutePath());
if (ze.isDirectory()) continue;
FileOutputStream fout = new FileOutputStream(file);
try {
while ((count = zis.read(buffer)) != -1)
fout.write(buffer, 0, count);
} finally {
fout.close();
}
}
} catch (IOException ioe) {
Log.d(TAG, ioe.getMessage());
return false;
} finally {
if (zis != null)
try {
zis.close();
} catch (IOException e) {
}
}
return true;

how to move files inside my spark streaming application

here I streaming the data from streaming directory and the write it to a output location. I am also trying to implement the process of moving hdfs files from a input folder to the streaming directory. This move happens one time before the streaming context starts. But I want this move to get executed every time for each Batch of Dstream. is that even possible?
val streamed_rdd = ssc.fileStream[LongWritable, Text, TextInputFormat](streaming_directory, (t:Path)=> true , true).map { case (x, y) => (y.toString) }
streamed_rdd.foreachRDD( rdd => {
rdd.map(x =>x.split("\t")).map(x => x(3)).foreachPartition { partitionOfRecords =>
val connection: Connection = connectionFactory.createConnection()
connection.setClientID("Email_send_module_client_id")
println("connection started with active mq")
val session: Session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE)
println("created session")
val dest = session.createQueue("dwEmailsQueue2")
println("destination queue name = dwEmailsQueue2")
val prod_queue = session.createProducer(dest)
connection.start()
partitionOfRecords.foreach { record =>
val rec_to_send: TextMessage = session.createTextMessage(record)
println("started creating a text message")
prod_queue.send(rec_to_send)
println("sent the record")
}
connection.close()
}
}
)
**val LIST = scala.collection.mutable.MutableList[String]()
val files_to_move = scala.collection.mutable.MutableList[String]()
val cmd = "hdfs dfs -ls -d "+load_directory+"/*"
println(cmd)
val system_time = System.currentTimeMillis
println(system_time)
val output = cmd.!!
output.split("\n").foreach(x => x.split(" ").foreach(x => if (x.startsWith("/user/hdpprod/")) LIST += x))
LIST.foreach(x => if (x.toString.split("/").last.split("_").last.toLong < system_time) files_to_move += x)
println("files to move" +files_to_move)
var mv_cmd :String = "hdfs dfs -mv "
for (file <- files_to_move){
mv_cmd += file+" "
}
mv_cmd += streaming_directory
println(mv_cmd)
val mv_output = mv_cmd.!!
println("moved the data to the folder")**
if (streamed_rdd.count().toString == "0") {
println("no data in the streamed list")
} else {
println("saving the Dstream at "+System.currentTimeMillis())
streamed_rdd.transform(rdd => {rdd.map(x => (check_time_to_send+"\t"+check_time_to_send_utc+"\t"+x))}).saveAsTextFiles("/user/hdpprod/temp/spark_streaming_output_sent/sent")
}
ssc.start()
ssc.awaitTermination()
}
}
I tried doing same stuff in java implementation as below. you can call this method from foreachPartion on rdd
public static void moveFiles(final String moveFilePath,
final JavaRDD rdd) {
for (final Partition partition : rdd.partitions()) {
final UnionPartition unionPartition = (UnionPartition) partition;
final NewHadoopPartition newHadoopPartition = (NewHadoopPartition)
unionPartition.parentPartition();
final String fPath = newHadoopPartition.serializableHadoopSplit()
.value().toString();
final String[] filespaths = fPath.split(":");
if ((filespaths != null) && (filespaths.length > 0)) {
for (final String filepath : filespaths) {
if ((filepath != null) && filepath.contains("/")) {
final File file = new File(filepath);
if (file.exists() && file.isFile()) {
try {
File destFile = new File(moveFilePath + "/" +
file.getName());
if (destFile.exists()) {
destFile = new File(moveFilePath + "/" +
file.getName() + "_");
}
java.nio.file.Files.move((file
.toPath()), destFile.toPath(),
StandardCopyOption.REPLACE_EXISTING);
} catch (Exception e) {
logger.error(
"Exception while moving file",
e);
}
}
}
}
}
}
}

Resources