I was having a little trouble with reading files using the FTP adapter. Several warnings and errors were found in the application log, and it basically boiled down to this message:
A stream read or write operation failed because the stream is in an error state.
Upon inspecting the log, I was surprised to find:
< 125 List started.
< BEBI 827693 24/09/14 15:53:51 *STMF mes9_ corrigido2.csv
< 250 List completed.
> TYPE I
< 200 Representation type is binary IMAGE.
> PORT 192,168,205,82,213,48
< 200 PORT subcommand request successful.
> RETR corrigido2.csv
so, there is just a single file with name mes9_ corrigido2.csv (notice the space) and BizTalk tries to retrieve a file corrigido2.csv. Is this normal behaviour?
Related
I'm sending binary .gz files from Linux to z/OS via ftps. The file transfers seem to be fine, but when the mainframe folks pkunzip the file, they get a warning:
PEX013W Record(s) being truncated to lrecl= 996. Record# 1 is 1000 bytes.
Currently I’m sending the site commands:
SITE TRAIL
200 SITE command was accepted
SITE CYLINDERS PRIMARY=50 SECONDARY=50
200 SITE command was accepted
SITE RECFM=VB LRECL=1000 BLKSIZE=32000
200 SITE command was accepted
SITE CONDDISP=delete
200 SITE command was accepted
TYPE I
200 Representation type is Image
...
250 Transfer completed successfully.
QUIT
221 Quit command received. Goodbye.
They could read the file after the pkunzip, but having a warning is not a good thing.
Output from pkunzip:
SDSF OUTPUT DISPLAY RMD0063A JOB22093 DSID 103 LINE 25 COLUMNS 02- 81
COMMAND INPUT ===> SCROLL ===> CSR
PCM123I Authorized services are unavailable.
PAM030I INPUT Archive opened: TEST.FTP.SOA5021.GZ
PAM560I ARCHIVE FASTSEEK processing is disabled.
PDA000I DDNAME=SYS00001,DISP_STATUS=MOD,DISP_NORMAL=CATALOG,DISP_ABNORMAL=
PDA000I SPACE_TYPE=TRK,SPACE_TYPE=CYL,SPACE_TYPE=BLK
PDA000I SPACE_PRIMARY=4194304,SPACE_DIRBLKS=5767182,INFO_ALCFMT=00
PDA000I VOLUMES=DPPT71,INFO_CNTL=,INFO_STORCLASS=,INFO_MGMTCLASS=
PDA000I INFO_DATACLASS=,INFO_VSAMRECORG=00,INFO_VSAMKEYOFF=0
PDA000I INFO_COPYDD=,INFO_COPYMDL=,INFO_AVGRECU=00,INFO_DSTYPE=00
PEX013W Record(s) being truncated to lrecl= 996. Record# 1 is 1000 bytes.
PEX002I TEST.FTP.SOA5021
PEX003I Extracted to TEST.FTP.SOA5021I.TXT
PAM140I FILES: EXTRACTED EXCLUDED BYPASSED IN ERROR
PAM140I 1 0 0 0
PMT002I PKUNZIP processing complete. RC=00000004 4(Dec) Start: 12:59:48.86 End
Is there a better set of site commands to transfer a .gz file from Linux to z/OS to avoid this error?
**** Update ****
Using SaggingRufus's answer below, it turns out it doesn't much matter how you send the .gz file, as long as it's binary. His suggestion pointed us to the parameters sent to the pkunzip for the output file, which was VB and was truncating 4 bytes off the record.
Because it is a variable block file, there are 4 bytes allocated to the record attributes. Allocate the file with an LRECL of 1004 and it will be fine.
Rather than generating a .zip file, perhaps generate a .tar.gz file and transfer it to z/OS UNIX? Tar is shipped with z/OS by default, and Rocket Software provides a port of gzip that is optimized for z/OS.
I've created a simple go script: https://gist.github.com/kbl/86ed3b2112eb80522949f0ce574a04e3
It's fetching some xml from the internet and then starts X goroutines. The X depends on file content. In my case it was 1700 goroutines.
My first execution finished with:
$ go run mathandel1.go
2018/01/27 14:19:37 Get https://www.boardgamegeek.com/xmlapi/boardgame/162152?pricehistory=1&stats=1: dial tcp 72.233.16.130:443: socket: too many open files
2018/01/27 14:19:37 Get https://www.boardgamegeek.com/xmlapi/boardgame/148517?pricehistory=1&stats=1: dial tcp 72.233.16.130:443: socket: too many open files
exit status 1
I've tried to increase ulimit to 2048.
Now I'm getting different error, script is the same thou:
$ go build mathandel1.go
# command-line-arguments
/usr/local/go/pkg/tool/linux_amd64/link: flushing $WORK/command-line-arguments/_obj/exe/a.out: write $WORK/command-line-arguments/_obj/exe/a.out: file too large
What is causing that error? How can I fix that?
You ran ulimit 2048 which changed the maximum file size.
From man bash(1), ulimit section:
If no option is given, then -f is assumed.
This means that you now set the maximum file size to 2048 bytes, that's probably not enough for.... anything.
I'm guessing you meant to change the limit for number of open file descriptors. For this, you want to run:
ulimit -n 2048
As for the original error (before changing the maximum file size), you're launching 1700 goroutines, each performing a http get. Each creates a connection, using a tcp socket. These are covered by the open file descriptor limit.
Instead, you should be limiting the number of concurrent downloads. This can be done with a simple worker pool pattern.
Recently I needed to check for a huge list of filenames if they exist on a server. I did this by running a for loop which tried to wget each of those files. That was efficient enough, but took about 30 minutes in this case. I wonder if there is a faster way to check whether a file exists or not (since wget is for downloading files and not performing thousands of requests).
I don't know if that information is relevant, but it's an Apache server.
Curl would be the best option in a for loop and here is a straight forward simple way, run this in your forloop
curl -I --silent http://www.yoururl/linktodetect | grep -m 1 -c 404
What this simply does is check the http response header for a 404 returned on the link and if its detected as a missing file/link throwing a 404 then the command line output will display you a number 1; otherwise, if the file/link is valid and does not return a 404 then the command line output will display you a number 0.
I have an FTP server running on an AWS virtual server. We have about 100 users connecting to it over the course of a day, uploading images and other files. All but one are working perfectly. Files come in, not a problem.
We have 1 single user that causes the following log lines to be generated - the username and IP has been removed intentionally and the "***" has been added to highlight the error line:
> 227 Entering Passive Mode (54,79,122,6,195,96)
> STOR media/UV1358A_3.jpg
> 150 Opening data channel for file upload to server of "/media/UV1358A_3.jpg"
> 226 Successfully transferred "/media/UV1358A_3.jpg"
> PASV
> 227 Entering Passive Mode (54,79,122,6,195,141)
> STOR media/UV1358A_4.jpg
> 150 Opening data channel for file upload to server of "/media/UV1358A_4.jpg"
> PASV
> 227 Entering Passive Mode (54,79,122,6,195,136)
> 226 Successfully transferred ""
> STOR media/UV1358A_5.jpg
***********************************************
> 503 Bad sequence of commands.
***********************************************
> PASV
> 227 Entering Passive Mode (54,79,122,6,195,80)
> PORT 122,99,115,5,212,227
> 200 Port command successful
> PORT 122,99,115,5,226,227
> 200 Port command successful
> PORT 122,99,115,5,130,124
> 200 Port command successful
> STOR media/UV1358A_9.jpg
> 150 Opening data channel for file upload to server of "/media/UV1358A_9.jpg"
> PORT 122,99,115,5,152,62
> 200 Port command successful
> STOR media/UV1358A_10.jpg
> 150 Opening data channel for file upload to server of "/media/UV1358A_10.jpg"
> PORT 122,99,115,5,161,49
> 200 Port command successful
We're using FileZilla Server 0.9.55 on a Windows 2012 box.
My question, as stated in the title is essentially.. Is this our issue on the server end, or is this theirs?
Is this 503 error always caused by the FTP client screwing something up or is there the possibility that the FTP server is interpreting something wrongly?
I'm happy to go back to the customer and say "It is our issue", but I suspect it's not at our end.
Thanks
The client sends the PASV command to initiate another file transfer before waiting for the previous transfer (the STOR command) to be finished (226 response):
The first transfer starts:
> PASV
< 227 Entering Passive Mode (54,79,122,6,195,141)
> STOR media/UV1358A_4.jpg
< 150 Opening data channel for file upload to server of "/media/UV1358A_4.jpg"
The PASV command for another transfer before the first transfer finished:
> PASV
< 227 Entering Passive Mode (54,79,122,6,195,136)
The first transfer finishes only now. The filename in the message is missing, because the FileZilla Server resets a file transfer data (including the file name) upon processing of the out-of-order PASV command (it actually should have better rejected the PASV command already with the 503).
> 226 Successfully transferred ""
A request for the another transfer. It fails because the FileZilla server forgets the out-of-order PASV command upon completion of the first file transfer.
> STOR media/UV1358A_5.jpg
< 503 Bad sequence of commands.
After successful FTP file transfer, the the response is used to be "226 File send OK", but suddenly, it has changed to be "226 Transfer complete"
I have below questions:
Does FTP response code has any standard?
Can we customize FTP output response for a specific status code?
Find the standard FTP response for file transfer
$ ftp canopus
Connected to canopus.austin.century.com.
220 canopus.austin.century.com FTP server (Version 4.1 Sat Nov 23 12:52:09 CST 1991) ready.
Name (canopus:eric): dee
331 Password required for dee.
Password:
230 User dee logged in.
ftp> pwd
257 "/home/dee" is current directory.
ftp> cd desktop
250 CWD command successful.
ftp> type ascii
200 Type set to A.
ftp> send typescript
200 PORT command successful.
150 Opening data connection for typescript (128.114.4.99,1412).
226 File send OK.
ftp> cdup
250 CWD command successful.
ftp> bye
221 Goodbye.
Note: suddenly the response text 226 File send OK has changed to 226 Transfer complete
Find the details about FTP responses on wikipedia
RFC 959, 4.2. FTP REPLIES:
An FTP reply consists of a three digit number (transmitted as
three alphanumeric characters) followed by some text. The number
is intended for use by automata to determine what state to enter
next; the text is intended for the human user. It is intended
that the three digits contain enough encoded information that the
user-process (the User-PI) will not need to examine the text and
may either discard it or pass it on to the user, as appropriate.
In particular, the text may be server-dependent, so there are
likely to be varying texts for each reply code.