I am not to access files in databricks file system. Databricks Runtime version 11.1 (includes Apache spark 3.3.0,scala 2.12) I have multiple csv files in file path like folder1/folder2/ledger1.csv
To mount DBFS you need to enable DBFS file browser in Workspace settings.
To enable DBFS file browser follow this path: Settings> Workspace settings>DBFS File Browser
Nested mounts are not supported. For example, the following structure is not supported:
• folder1 mounted as /mnt/folder1
• folder2 mounted as /mnt/ folder1/ folder2
Databricks recommends creating separate mount entries for each storage object:
• folder1 mounted as /mnt/folder1
• folder2 mounted as /mnt/folder2
Refer - https://learn.microsoft.com/en-us/azure/databricks/data/databricks-file-system#--mount-object-storage-to-dbfs
Related
I'm wondering if rclone is able to donwload file from the shared folder of Google Drive. If yes, what is the command to do it?
rclone sync cloud_name:(what is the shared folder name?)file_name destination_path
You need to use rclone config to create a remote: for the Google 'Shared Drive'. See https://rclone.org/drive/
The line Configure this as a Shared Drive (Team Drive)?
Then the sync would be:
rclone sync SharedDriveName:"Directory/Directory" YourOtherRemote:"Directory/Directory"
Useful flags are:
-P or --progress (see progress during transfer)
-vv (see detailed logs)
--create-empty-src-dirs (to recreate empty directories)
-u or --update (Skip files that are newer on the destination)
--drive-server-side-across-configs (if you want to sync native Google docs)
--dry-run (as a practise)
I'm using the below ansible-playbook code to archive multiple folders under IBM folder.
Below is my absolute path Directory structure:
/app
|-- /IBM
|--/test
|--/log
|--/common
|--/api
|--/<unknown folders>
The number of folders and files under IBM folder is unknown hence it seems impossible to specify all the files / folders under IBM under exclude_path attribute.
I wish to build an archive (gz) that has only IBM folder containing only two folders i.e common and api folders while ignoring the rest of the folders and files.
Thus, I wrote the below playbook:
- name: Creating the archive
archive:
path:
- /was/IBM/common
- /was/IBM/api
dest: /var/backup/mysetup.tar.gz
exclude_path:
- /was/IBM/common/log
- /was/IBM/api/tmp
format: gz
This gives me the archive file mysetup.tar.gz.
I want the mysetup.tar.gz file to have a folder called IBM which should have the two folders common and api. Thus, I'm expecting the below in the mysetup.tar.gz no matter what other files are under the IBM folder.
IBM
|--/common
|--/api
But, the mysetup.tar.gz does not have IBM folder but has common and api folders.
I was not specific hence my question did not get answered here: How to archive multiple folders under one folder using Ansible
Can you please guide me as to how I can get the archive to have both the folders inside the IBM folder inside the mysetup.tar.gz?
Requirement you have can't be achieved straight using archive module. Two specific reasons are listed below:
If you specify the path say /app/IBM then the content of the zip file will have the path after IBM something like common/, api/. To get IBM as a root directory inside the tar.gz file, you need to specify path as /app. This brings to the next point about the usage of exclude_path.
exclude_path doesn't work as you would expect. Say, if the path is define as /app and you want to exclude /app/IBM/test, this wouldn't work. exclude_path works only with direct sub folder/file of a defined path (with wildcard) which means that exclusion of /app/IBM/test would work if path defined as /app/IBM/* but that is not what is expected and brings back the previous point. There is already an issue reported on this topic.
So you probably better off using regular tar command using command/shell module.
I have a web application, using Alibaba Cloud OSS, which is synchronized to the cloud from the local folder in Linux server, I see the files are uploading onto the Linux server but not synchronized to OSS
I have reconfigured the entire setup using the ossutil and the ossfs but still the same issue
The below is the error I face when I try to run the command
ossfs -ourl=http://oss-ap-south-1.aliyuncs.com
ossfs: There is no enough disk space for used as cache(or temporary) directory by ossfs.
Did you follow this guide?
For me OSS mount on Linux works when I type in command line:
ossfs bucketname /mnt/directory -ourl=http://oss-your-region.aliyuncs.com
If your Linux machine is in Alibaba Cloud you can use
-ourl=http://oss-your-region-internal.aliyuncs.com
You need to mount the OSS bucket to the specified directory as follows to synchronize Linux server and OSS.
To mount the OSS bucket to the directory:
ossfs bucket mountpoint -ourl=http://oss-your-region.aliyuncs.com
For instance, mount the bucket bucketName to the /tmp/ossfs directory. The AccessKeyId is abcdef, the AccessKeySecret is 123456, and the OSS endpoint is http://oss-cn-hangzhou.aliyuncs.com.
echo bucketName:abcdef:123456 > /etc/passwd-ossfs
chmod 640 /etc/passwd-ossfs
mkdir /tmp/ossfs
ossfs bucketName /tmp/ossfs -ourl=http://oss-cn-beijing.aliyuncs.com
Note: Permissions must be set correctly.
I'm using the Copy module to transfer a 10G file from my machine to the remote /tmp dir. However, Copy uses an intermediate folder inside home and I need to transfer the file directly to /tmp because /home doesn't have enough space.
Is it possible to control the src path used by the Copy module?
Thanks
In your ansible.cfg change the "remote_tmp" to a location where sufficient space available or you run your playbook as below:
ANSIBLE_REMOTE_TEMP=/dir1/some_dir/large_space/ ansible-playbook copy.yml
Official Documentation for ANSIBLE_REMOTE_TEMP with shell plugin
The default for uploaded images appears to be /storage/app/public/profiles/, but the frontend generated URL of /storage/profiles/image isn't being found.
I tried to find any sort of routing for that directory and haven't been able to do so. Any help would be appreciated.
You must link to the storage directory. From the documentation:
Once Spark is installed, you should link the public/storage directory to your storage/app/public directory. Otherwise, user profile photos stored on the local disk will not be available
And the relevant code:
ln -s /path/to/storage/app/public /path/to/public/storage
Just delete the storage folder inside public folder and run the command using absolute paths(from root folder).