I have set up a ADF pipeline that gets a set of .Zip files from Azure Storage, and iterates through each Zip file's folders and files to land them in an output container with preserved hierarchy.
Get Metadata:
For Each:
Issue:
The issue is that there is a specific .PDF file (ASC_NTS.pdf) that is embedded within each .Zip file that has the same name:
It is causing this error when trying to run the pipeline:
Error
Operation on target ForEach1 failed: Activity failed because an inner activity failed; Inner activity name: Copy data1, Error: ErrorCode=AdlsGen2OperationFailedConcurrentWrite,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error occurred when trying to upload a file. It's possible because you have multiple concurrent copy activities runs writing to the same file 'FAERS_output/ascii/ASC_NTS.pdf'. Check your ADF configuration.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ADLS Gen2 operation failed for: Operation returned an invalid status code 'PreconditionFailed'. Account: 'asastgssuaefdbhdg2dbc4'. FileSystem: 'curated'. Path: 'FAERS_output/ascii/ASC_NTS.pdf'. ErrorCode: 'LeaseIdMissing'. Message: 'There is currently a lease on the resource and no lease ID was specified in the request.'. RequestId: 'b21022a6-b01f-0031-641a-453ab6000000'. TimeStamp: 'Thu, 31 Mar 2022 16:15:56 GMT'..,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.Azure.Storage.Data.Models.ErrorSchemaException,Message=Operation returned an invalid status code 'PreconditionFailed',Source=Microsoft.DataTransfer.ClientLibrary,'
Is there a workaround for this pipeline setup that allows me to filter within the For Each loop? I just need the .TXT files, the .PDF files can be discarded.
This was the closest reference I could find, but does not address my use case:
Filter out file using wildcard path azure data factory
Have you tried using an If Condition activity? You can set the expression to check for the correct file extension.
Following this example and using Javascript, get the following error when run my tests. The error is generated on the following command:
enter code heredriver.executeScript('mobile: stopPerfRecord', [
{ profileName: 'Leaks' },
]);
Error:
Got this error “Request failed with status 500 due to An unknown server-side error occurred while processing the command. Original error: There is no .trace file found for performance profile ‘Time Profiler’ and device XXXX. Make sure the selected profile is supported on this device”
I am getting an error while running a code in TensorFlow for data pre-processing.
NotFoundError: NewRandomAccessFile failed to Create/Open: C:\Users\353\tensorflow_datasets\mnist\1.0.0\mnist-train.tfrecord-00005-of-00010 : The system cannot find the file specified.
; No such file or directory [Op:IteratorGetNextSync]
In my NiFi workflow I need to download .zip file from SOAP web-server, save it on machine (optional) and unpack content to sub-folder. Everything works on my local Win 10 machine, but issues occur when I try to move to remote Linux server. Here is the part of my flow when error happens:
So we have a FlowFile entered UpdateAttribute where filename attributed is set with required name and .zip extension. The file is correct as can be seen in queue after starting the processor.
Problems start happen when I pass the FlowFile to PutFile processors. I tried different scenarios based on selected directory:
relative to NiFi main folder ./out:
12:30:01 MSK ERROR
PutFile[id=05788ae5-64e5-32af-bb40-88d50d4c886c] Penalizing StandardFlowFileRecord[uuid=3e0c5e38-76f8-4ce3-b911-90f6901c35a4,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1586337594333-49, container=default, section=49], offset=2335, length=375628606],offset=0,name=,size=375628606] and transferring to failure due to /opt/nifi/nifi-1.11.4/./out: java.nio.file.DirectoryNotEmptyException: /opt/nifi/nifi-1.11.4/./out
12:30:01 MSK ERROR
PutFile[id=05788ae5-64e5-32af-bb40-88d50d4c886c] Unable to remove temporary file /opt/nifi/nifi-1.11.4/./out/. due to /opt/nifi/nifi-1.11.4/./out/.: Invalid argument: java.nio.file.FileSystemException: /opt/nifi/nifi-1.11.4/./out/.: Invalid argument
Full path /opt/nifi/nifi-1.11.4/out/file/ :
12:32:45 MSK ERROR
PutFile[id=0171102b-c82d-149d-c9ae-ea4da99b1750] Penalizing StandardFlowFileRecord[uuid=0573803f-8407-46e4-93f0-e52a5fc35a07,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1586337594333-49, container=default, section=49], offset=2335, length=375628606],offset=0,name=,size=375628606] and transferring to failure due to Failed to export StandardFlowFileRecord[uuid=0573803f-8407-46e4-93f0-e52a5fc35a07,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1586337594333-49, container=default, section=49], offset=2335, length=375628606],offset=0,name=,size=375628606] to /opt/nifi/nifi-1.11.4/out/file/. due to java.io.FileNotFoundException: /opt/nifi/nifi-1.11.4/out/file/. (No such file or directory): org.apache.nifi.processor.exception.FlowFileAccessException: Failed to export StandardFlowFileRecord[uuid=0573803f-8407-46e4-93f0-e52a5fc35a07,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1586337594333-49, container=default, section=49], offset=2335, length=375628606],offset=0,name=,size=375628606] to /opt/nifi/nifi-1.11.4/out/file/. due to java.io.FileNotFoundException: /opt/nifi/nifi-1.11.4/out/file/. (No such file or directory)
So it adds dot ('.') to path which causes exception. All folders are created and permissions granted. I tried to run simple test flow with file of 42B and the same path (GenerateFlowFile -> PutFile) and everything is OK.
What am I doing wrong?
The problem was in Linux system rights: there was 'nifi' user for flow execution and one more user for accessing Linux filesystem.
Assigning rwxrwxrwx for used folders solved the issue.
When running firefox on centos. I am getting the following error. How do i resolve this?
XPCOMGlueLoad error for file /usr/local/firefox/libxul.so:
libdbus-glib-1.so.2: cannot open shared object file: No such file or directory
Couldn't load XPCOM.