Azure Databricks API: import entire directory with notebooks - azure-databricks

Question

I need to import many notebooks (both Python and Scala) to Databricks using Databricks REST API 2.0
My source path (local machine) is ./db_code and destination (Databricks workspace) is /Users/dmitriy#kagarlickij.com
I'm trying to build 2.0/workspace/import call, so my body is: { "content": "$SOURCES_PATH", "path": "$DESTINATION_PATH", "format": "SOURCE", "language": "SCALA", "overwrite": true }
However I'm getting error: Could not parse request object: Illegal character and as per documentation content must be The base64-encoded content
Should I encode all notebooks to base64?
Maybe there're some examples of importing directory to Databricks using API?

Answer 1

After some research I've managed to get it work:
Write-Output "Task: Create Databricks Directory Structure"
Get-ChildItem "$SOURCES_PATH" -Recurse -Directory |
Foreach-Object {
$DIR = $_.FullName.split("$WORKDIR_PATH/")[1]
$BODY = #{
"path" = "$DESTINATION_PATH/$DIR"
}
$BODY_JSON = $BODY | ConvertTo-Json
Invoke-RestMethod -Method POST -Uri "https://$DATABRICKS_REGION.azuredatabricks.net/api/2.0/workspace/mkdirs" -Headers $HEADERS -Body $BODY_JSON | Out-Null
}
Write-Output "Task: Deploy Scala notebooks to Databricks"
Get-ChildItem "$SOURCES_PATH" -Recurse -Include *.scala |
Foreach-Object {
$NOTEBOOK_NAME = $_.FullName.split("$WORKDIR_PATH/")[1]
$NOTEBOOK_BASE64 = [Convert]::ToBase64String([IO.File]::ReadAllBytes("$_"))
$BODY = #{
"content" = "$NOTEBOOK_BASE64"
"path" = "$DESTINATION_PATH/$NOTEBOOK_NAME"
"language" = "SCALA"
"overwrite" = "true"
"format" = "SOURCE"
}
$BODY_JSON = $BODY | ConvertTo-Json
Invoke-RestMethod -Method POST -Uri "https://$DATABRICKS_REGION.azuredatabricks.net/api/2.0/workspace/import" -Headers $HEADERS -Body $BODY_JSON | Out-Null
}

Answer 2

Yes, the notebooks must be encoded to base64. You can use Powershell to achieve this. Try the below.
$BinaryContents = [System.IO.File]::ReadAllBytes("$SOURCES_PATH")
$EncodedContents = [System.Convert]::ToBase64String($BinaryContents)
The body for the REST call would look like this.
{
"format": "SOURCE",
"content": "$EncodedContents",
"path": "$DESTINATION_PATH",
"overwrite": "true",
"language": "Scala"
}
With respect to importing whole directories, you could loop through the files and folders in your local directory using Powershell and make a REST call for each notebook.