Read file from Azure Databricks DBFS REST 2.0 API - azure-databricks

Question

I am working on an application to read and write files using Azure Databricks DBFS API 2.0. Reference Documentation for this API:
https://docs.azuredatabricks.net/api/latest/dbfs.html#read
I am able to upload file (lets say 1.4MB file) by converting it into base64 then divided into 1MB chunks of data.
As the read length is limited to 1MB, I am iterating a loop to read data from offset 0 to 1000000 and 1000001 to end of the file. Now the first iteration of data, 0 - 1000000, is valid and I can confirm from the original file which i used for upload.
But the second and later data iterations, the base64 data is completely different and not present in the original base64 file.
Following is my test code:
Second iteration - 0.4MB
const axios = require('axios')
const fs = require('fs')
axios({
method: 'get',
url: 'https://********.azuredatabricks.net/api/2.0/dbfs/read',
data: {
path: '/Test/backlit-beach-clouds-1684881.jpg',
offset: 0,
length: 1000000
},
headers: {
'Authorization': `Bearer ****`
}
}).then(({data}) => {
if (data) {
console.log('Success', data.bytes_read)
fs.writeFile('./one.txt', data.data, function (err) {
console.log('done', err)
})
} else {
console.log('failed')
}
})
First iteration - 1MB
const axios = require('axios')
const fs = require('fs')
axios({
method: 'get',
url: 'https://********.azuredatabricks.net/api/2.0/dbfs/read',
data: {
path: '/Test/backlit-beach-clouds-1684881.jpg',
offset: 1000001,
length: 1000000
},
headers: {
'Authorization': `Bearer ****`
}
}).then(({data}) => {
if (data) {
console.log('Success', data.bytes_read)
fs.writeFile('./two.txt', data.data, function (err) {
console.log('done', err)
})
} else {
console.log('failed')
}
})
Here, *** are replaced with relevant domain and tokens.
As you can see, the above test code samples will generate one.txt and two.txt. By using cat one.txt two.txt > final.txt I can get final.txt which I will use to decode original file.
As this is just a testing code, I haven't used any loops or better coding format. This is just to understand what went wrong.
I stuck with this for over 1 week now. I am referring other code samples written for python, but no help.
I am not trying to waste anyone's time. But please, someone help me to figure out what went wrong or any other standard procedure that I can follow?

Answer 1

I've run the same type of issue and solve it with an iterative extraction.
However in order to make it work I had to extract from 0 to 1000000 and 1000000 to end of the file.
Otherwise the base64 decoding part was not properly considering the beginning of the second slice.
I found that you don't need to add this extra byte from one slice to another.