Use case:
I use socat to stream traffic between some app and external world via Squid (app->socat->Squid). To authenticate in SQUID I use mTLS.
My socat usage:
socat -d -d tcp-listen:3128,reuseaddr,fork \
openssl-connect:<SQUID_IP>:3128,cert=client-cert-key.pem,cafile-squid=squid-ca.crt,openssl-commonname=<SQUID_CN>-prd,keepalive
where
content of the PEM and CERT filles rotate.
Problem: If I put some trash into squid-ca.crt file socat after a couple of seconds catches the change and logs errors:
socat[72] E SSL_connect(): error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed
socat[72] N exit(1)
socat[9] N childdied(): handling signal 17
, and when I recover squid-ca.crt file socat ater some time (several to several dozen seconds) fetches the change and starts working again with with the recovered data.
Is there an option to control the time on files probing by socat?
Socat with your command waits for client connections and forks a new sub process for each one. Only in these sub processes the OpenSSL module is initialized and the certificate loaded.
So it is not some timing but just depends on the next TCP connection to arrive.
We are getting errors while loading data(large volume) in Greenplum through data stage jobs.
There are multiple jobs running sequentially. there is no particular job that fails. it is randomly. if on the 1st day Job1 fails, on the 2nd day Job2 gets fail.
We have also observed that it only impact the jobs that have to load a high volume of data.
Please find the error, we have got so fer.
day 1----------
Message:
STG_DEPS_G,0: The following SQL statement failed: INSERT INTO GPCC_ET_20211114015751397_14836_2 SELECT DEPT, DEPT_NAME, BUYER, MERCH, PROFIT_CALC_TYPE, PURCHASE_TYPE, GROUP_NO, BUD_INT, BUD_MKUP, TOTAL_MARKET_AMT, MARKUP_CALC_TYPE, OTB_CALC_TYPE, MAX_AVG_COUNTER, AVG_TOLERANCE_PCT, DEPT_VAT_INCL_IND, CREATE_ID, CREATE_DATETIME FROM staging.STG_DEPS. The statement reported the following reason: [SQLCODE=08S01][Native=373,254] [IBM (DataDirect OEM)][ODBC Greenplum Wire Protocol driver][Greenplum]ERROR: http response code 501 from gpfdist (gpfdist://DDCETLMIG:8000/DDCETLMIG_14836_gpw_11_3_20211114015751366): HTTP/1.0 501 pipe is busy, close the pipe and try again (seg0 192.168.199.10:6000 pid=25824)(File url_curl.c; Line 474; Routine check_response; ) (CC_GPCommon::checkThreadStatusThrow, file CC_GPCommon.cpp, line 808)
The following SQL statement failed: INSERT INTO GPCC_ET_20211114015751397_14836_2 SELECT DEPT, DEPT_NAME, BUYER, MERCH, PROFIT_CALC_TYPE, PURCHASE_TYPE, GROUP_NO, BUD_INT, BUD_MKUP, TOTAL_MARKET_AMT, MARKUP_CALC_TYPE, OTB_CALC_TYPE, MAX_AVG_COUNTER, AVG_TOLERANCE_PCT, DEPT_VAT_INCL_IND, CREATE_ID, CREATE_DATETIME FROM staging.STG_DEPS. The statement reported the following reason: [SQLCODE=08S01][Native=373,254] [IBM (DataDirect OEM)][ODBC Greenplum Wire Protocol driver][Greenplum]ERROR: http response code 501 from gpfdist (gpfdist://DDCETLMIG:8000/DDCETLMIG_14836_gpw_11_3_20211114015751366): HTTP/1.0 501 pipe is busy, close the pipe and try again (seg0 192.168.199.10:6000 pid=25824)(File url_curl.c; Line 474; Routine check_response; ) (CC_GPCommon::checkThreadStatusThrow, file CC_GPCommon.cpp, line 808)
day 2
STG_RPM_ZONE,0: The following SQL statement failed: INSERT INTO GPCC_ET_20211114093430218_8212_0 SELECT ZONE_ID, ZONE_DISPLAY_ID, ZONE_GROUP_ID, NAME, CURRENCY_CODE, BASE_IND, LOCK_VERSION FROM STAGING.STG_RPM_ZONE. The statement reported the following reason: [SQLCODE=08S01][Native=373,254] [IBM (DataDirect OEM)][ODBC Greenplum Wire Protocol driver][Greenplum]ERROR: http response code 501 from gpfdist (gpfdist://DDCETLMIG:8004/DDCETLMIG_8212_gpw_0_0_20211114093430186): HTTP/1.0 501 pipe is busy, close the pipe and try again (seg1 192.168.199.11:6000 pid=26726)(File url_curl.c; Line 474; Routine check_response; ) (CC_GPCommon::checkThreadStatusThrow, file CC_GPCommon.cpp, line 808)
The following SQL statement failed: INSERT INTO GPCC_ET_20211114093430218_8212_0 SELECT ZONE_ID, ZONE_DISPLAY_ID, ZONE_GROUP_ID, NAME, CURRENCY_CODE, BASE_IND, LOCK_VERSION FROM STAGING.STG_RPM_ZONE. The statement reported the following reason: [SQLCODE=08S01][Native=373,254] [IBM (DataDirect OEM)][ODBC Greenplum Wire Protocol driver][Greenplum]ERROR: http response code 501 from gpfdist (gpfdist://DDCETLMIG:8004/DDCETLMIG_8212_gpw_0_0_20211114093430186): HTTP/1.0 501 pipe is busy, close the pipe and try again (seg1 192.168.199.11:6000 pid=26726)(File url_curl.c; Line 474; Routine check_response; ) (CC_GPCommon::checkThreadStatusThrow, file CC_GPCommon.cpp, line 808)
day 3
Event type:Fatal
Timestamp:11/15/2021 9:27:36 AM
Message:
SUB_CLASS,3: APT_PMMessagePort::dispatch:ERROR: header = 04F02E20SUBPROC_SUPPORT_EOW, savedDispatchPosition = 04F02E20, currentDispatchPosition_ = 04F02E1FS, currentInputPosition_ = 04F02E58, buffer_ = 04F02E20, this = 04EEA1E0
Day 4
Message:
STG_GROUPS_G,0: The following SQL statement failed: INSERT INTO GPCC_ET_20211115015013039_2400_0 SELECT GROUP_NO, GROUP_NAME, BUYER, MERCH, DIVISION, CREATE_ID, CREATE_DATETIME FROM staging.STG_GROUPS. The statement reported the following reason: [SQLCODE=08S01][Native=373,254] [IBM (DataDirect OEM)][ODBC Greenplum Wire Protocol driver][Greenplum]ERROR: http response code 501 from gpfdist (gpfdist://DDCETLMIG:8009/DDCETLMIG_2400_gpw_1_1_20211115015013023): HTTP/1.0 501 pipe is busy, close the pipe and try again (seg5 192.168.199.12:6001 pid=1167)(File url_curl.c; Line 474; Routine check_response; ) (CC_GPCommon::checkThreadStatusThrow, file CC_GPCommon.cpp, line 808)
The following SQL statement failed: INSERT INTO GPCC_ET_20211115015013039_2400_0 SELECT GROUP_NO, GROUP_NAME, BUYER, MERCH, DIVISION, CREATE_ID, CREATE_DATETIME FROM staging.STG_GROUPS. The statement reported the following reason: [SQLCODE=08S01][Native=373,254] [IBM (DataDirect OEM)][ODBC Greenplum Wire Protocol driver][Greenplum]ERROR: http response code 501 from gpfdist (gpfdist://DDCETLMIG:8009/DDCETLMIG_2400_gpw_1_1_20211115015013023): HTTP/1.0 501 pipe is busy, close the pipe and try again (seg5 192.168.199.12:6001 pid=1167)(File url_curl.c; Line 474; Routine check_response; ) (CC_GPCommon::checkThreadStatusThrow, file CC_GPCommon.cpp, line 808)
A couple of questions:
You indicated there are multiple jobs running sequentially? Does that mean there are multiple concurrent jobs each running a set of sequential jobs? Or is there one job running multiple sequential steps? If it is the first, I would check to make sure no jobs are using the same gpfdist ports.
Is DataStage connecting through a proxy server? That could potentially be causing a problem (common cause of 501 errors according to google lookup).
Finally, what is the throughput speed of the NIC or NICs being used to connect DataStage/gpfdist to Greenplum? Also, what is the interconnect speed between each segment/segment host in Greenplum? If either of these is less than 10gig, your network may not be able to support really high throughput.
I have a Protobuf serialized data that is being sent over UDP multicast. I read it using socat from bash as follows
socat UDP4-RECVFROM:1277,ip-add-membership=239.195.1.2:192.168.254.5,fork,reuseaddr -
I want to chain this with protoc in order to keep decoding data in every packet. How can I do that ?
When I do
socat UDP4-RECVFROM:1277,ip-add-membership=239.195.1.2:192.168.254.5,fork,reuseaddr - | protoc --decode=proto.Message ./path/to/proto/message.proto
The terminal just hangs. Not sure how to proceed.
The problem turned out to be in socat. remove the fork portion and it works great.
So essentially
socat UDP4-RECVFROM:1277,ip-add-membership=239.195.1.2:192.168.254.5,reuseaddr - | protoc --decode=proto.Message ./path/to/proto/message.proto
I've created a simple go script: https://gist.github.com/kbl/86ed3b2112eb80522949f0ce574a04e3
It's fetching some xml from the internet and then starts X goroutines. The X depends on file content. In my case it was 1700 goroutines.
My first execution finished with:
$ go run mathandel1.go
2018/01/27 14:19:37 Get https://www.boardgamegeek.com/xmlapi/boardgame/162152?pricehistory=1&stats=1: dial tcp 72.233.16.130:443: socket: too many open files
2018/01/27 14:19:37 Get https://www.boardgamegeek.com/xmlapi/boardgame/148517?pricehistory=1&stats=1: dial tcp 72.233.16.130:443: socket: too many open files
exit status 1
I've tried to increase ulimit to 2048.
Now I'm getting different error, script is the same thou:
$ go build mathandel1.go
# command-line-arguments
/usr/local/go/pkg/tool/linux_amd64/link: flushing $WORK/command-line-arguments/_obj/exe/a.out: write $WORK/command-line-arguments/_obj/exe/a.out: file too large
What is causing that error? How can I fix that?
You ran ulimit 2048 which changed the maximum file size.
From man bash(1), ulimit section:
If no option is given, then -f is assumed.
This means that you now set the maximum file size to 2048 bytes, that's probably not enough for.... anything.
I'm guessing you meant to change the limit for number of open file descriptors. For this, you want to run:
ulimit -n 2048
As for the original error (before changing the maximum file size), you're launching 1700 goroutines, each performing a http get. Each creates a connection, using a tcp socket. These are covered by the open file descriptor limit.
Instead, you should be limiting the number of concurrent downloads. This can be done with a simple worker pool pattern.
I have used cocoa pod and try to archive. After that, I got this error.
I looked solution from here but it doesn't work too. How shall I do?
how to fix the issue "Command /bin/sh failed with exit code 1" in iphone
sent 29 bytes received 20 bytes 98.00 bytes/sec
total size is 0 speedup is 0.00
building file list ... done
rsync: push_dir#3 "/Users/khantthulinn/Library/Developer/Xcode/DerivedData/SMRTConnect-emozrftvltrpagajdyydwrdgnacg/Build/Intermediates/ArchiveIntermediates/SMRTConnect/InstallationBuildProductsLocation" failed: No such file or directory (2)
rsync error: errors selecting input/output files, dirs (code 3) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/main.c(580) [receiver=2.6.9]
rsync: connection unexpectedly closed (8 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/io.c(453) [sender=2.6.9]
Command /bin/sh failed with exit code 12