Fetching image with curl is smaller than original - image
I found the problem in some PHP code I'm working on but I can reproduce it using wget and curl, from the linux terminal. When I use the respective programmes to fetch the same image file from my server, the file sizes are different and the smaller curl file is corrupt. This happens on all image files, but doesn't always corrupt the file.
0.0.0.0 - - [18/Aug/2013:23:41:48 +0200] "GET /test.png HTTP/1.1" 200 555247 "-" "Wget/1.13.4 (linux-gnu)"
0.0.0.0 - - [18/Aug/2013:23:42:53 +0200] "GET /test.png HTTP/1.1" 200 555191 "-" "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"
You will notice the png file downloaded with curl is 56bytes smaller, which is enough to corrupt the image.
The server in question is my own server running ubuntu server, and apache, as far as I'm aware there is nothing none standard on it, or my curl\wget setup. (And yes I fudged the IP address for security)
Any help most appreciated.
Related
wget on Debian Server gets 302 Found while wget on Manjaro gets 200 OK
I'm using wget to retrieve the Instagram JSON from the URL https://www.instagram.com/instagram/?__a=1. Running wget from my local Manjaro setup returns a 200 OK and the proper JSON response, but running it from a Debian server retrieves a 302 found. At first I thought it could be because of the wget version differences, but running curl locally also works while wget doesn't work properly. Is there anything that I should be setting up on my server to get a proper response? My guess is that the HTTPS connection is refusing my server from connecting properly.
So, this is a weird quirk of the Instagram servers. Nothing you can do about it. The problem is that Instagram responds differently depending on whether you connect to their server over IPv4 or IPv6. Why they would do that is beyond me, but I can reliably reproduce the result by controlling for only this variable. IPv4: $ wget -O/dev/null -4 "https://www.instagram.com/instagram/?__a=1" --2020-09-03 14:22:15-- https://www.instagram.com/instagram/?__a=1 Resolving www.instagram.com (www.instagram.com)... 157.240.27.174 Connecting to www.instagram.com (www.instagram.com)|157.240.27.174|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 118552 (116K) [application/json] Saving to: ‘/dev/null’ 100%[================================================================================================================================>] 118,552 306KB/s in 0.4s 2020-09-03 14:22:17 (306 KB/s) - ‘/dev/null’ saved [118552/118552] IPv6: $ wget -O/dev/null -6 "https://www.instagram.com/instagram/?__a=1" --2020-09-03 14:22:54-- https://www.instagram.com/instagram/?__a=1 Resolving www.instagram.com (www.instagram.com)... 2a03:2880:f23f:e5:face:b00c:0:4420 Connecting to www.instagram.com (www.instagram.com)|2a03:2880:f23f:e5:face:b00c:0:4420|:443... connected. HTTP request sent, awaiting response... 302 Found Cookie coming from www.instagram.com attempted to set domain to www.instagram.com Cookie coming from www.instagram.com attempted to set domain to www.instagram.com Location: https://www.instagram.com/accounts/login/?next=/instagram/%3F__a%3D1 [following] --2020-09-03 14:22:54-- https://www.instagram.com/accounts/login/?next=/instagram/%3F__a%3D1 Reusing existing connection to [www.instagram.com]:443. HTTP request sent, awaiting response... 200 OK Cookie coming from www.instagram.com attempted to set domain to www.instagram.com Cookie coming from www.instagram.com attempted to set domain to www.instagram.com Length: 48094 (47K) [text/html] Saving to: ‘/dev/null’ 100%[================================================================================================================================>] 48,094 --.-K/s in 0.04s 2020-09-03 14:22:54 (1.28 MB/s) - ‘/dev/null’ saved [48094/48094] This is the same thing you see in your debug logs. On Manjaro, it makes a IPv4 connection, while on Debian it makes a IPv6 connection leading to the differences. Welcome to the world of crazy webservers :) In any case, the answer to your question then is to use only a IPv4 connection
H2O Import files via REST API from local server fails
I am trying to use h2o REST API to import CSV files that I have on my local server.
Command:
curl -v -X GET
'http://127.0.0.1:54321/3/ImportFiles?path=http://127.0.0.1:8083/datasets/tables/csv/RDsTWgcvAjHeWJFnbhCKTCE5rn6aLCjJ.csv'
Result in following log:
Trying 127.0.0.1... Connected to 127.0.0.1 (127.0.0.1) port 54321 (#0)
GET
/3/ImportFiles?path=http://127.0.0.1:8083/datasets/tables/csv/RDsTWgcvAjHeWJFnbhCKTCE5rn6aLCjJ.csv HTTP/1.1 Host: 127.0.0.1:54321 User-Agent: curl/7.47.0 Accept: /
HTTP/1.1 200 OK X-h2o-build-project-version: 3.16.0.2
X-h2o-rest-api-version-max: 3 X-h2o-cluster-id: 1512722051559
X-h2o-cluster-good: true X-h2o-context-path: / Content-Type:
application/json Content-Length: 349 Server: Jetty(8.y.z-SNAPSHOT)
Connection #0 to host 127.0.0.1 left intact
{"__meta":{"schema_version":3,"schema_name":"ImportFilesV3","schema_type":"ImportFiles"},"_exclude_fields":"","path":"http://127.0.0.1:8083/datasets/tables/csv/RDsTWgcvAjHeWJFnbhCKTCE5rn6aLCjJ.csv","pattern":null,"files":[],"destination_frames":[],"fails":["http://127.0.0.1:8083/datasets/tables/csv/RDsTWgcvAjHeWJFnbhCKTCE5rn6aLCjJ.csv"],"dels":[]}
H2O log on TRACE level shows only:
12-08 15:41:59.951 10.8.128.101:54321 36013 #4756-331 INFO: GET
/3/ImportFiles, parms:
{path=http://127.0.0.1:8083/datasets/tables/csv/RDsTWgcvAjHeWJFnbhCKTCE5rn6aLCjJ.csv}
Is there any way to debug while importing fails? h2o does not asking local server at all.
Commands from other servers work well:
curl -v -X GET "http://127.0.0.1:54321/3/ImportFiles?path=http://s3.amazonaws.com/h2o-public-test-data/smalldata/flow_examples/arrhythmia.csv.gz"
curl -v -X GET "https://raw.github.com/h2oai/h2o/master/smalldata/logreg/prostate.csv"
In general, trying to interact directly with the H2O REST API isn't easy. The vast majority of people use a pre-made API client like Python or R.
But if you really want to do this, I would debug it by comparing with something that's working. Like the R client for H2O.
Write an R program that does this:
h2o.init()
h2o.startLogging()
h2o.importFile("/path/to/data.csv")
The startLogging() call will produce a detailed log file with all the REST API requests and responses. Look at that and try to mimic it.
You can also refer to the autogenerated REST API documentation (http://docs.h2o.ai/h2o/latest-stable/h2o-docs/rest-api-reference.html), but I would caution that if you tried to write a working client just based on the docs it would be hard.
Looking at a logged conversation from an already working client is by far your best bet.
library(h2o)
h2o.init()
h2o.startLogging()
h2o.importFile("http://localhost:8082/datasets/tables/csv/vPrzC5TOQr6JTvnAYrU5AKyz8SP4ao8p.csv")
Time: 2017-12-11 11:55:09.237
GET http://localhost:54321/3/Cloud?skip_ticks=true postBody:
curlError: FALSE curlErrorMessage: httpStatusCode: 200
httpStatusMessage: OK millis: 7
{"__meta":{"schema_version":3,"schema_name":"CloudV3","schema_type":"Iced"},"_exclude_fields":"","skip_ticks":true,"version":"3.16.0.2","branch_name":"rel-wheeler","build_number":"2","build_age":"10 days","build_too_old":false,"node_idx":0,"cloud_name":"H2O_started_from_R_vasiliy_gey658","cloud_size":1,"cloud_uptime_millis":306486,"cloud_healthy":true,"bad_nodes":0,"consensus":true,"locked":true,"is_client":false,"nodes":[{"__meta":{"schema_version":3,"schema_name":"NodeV3","schema_type":"Iced"},"h2o":"localhost/127.0.0.1:54321","ip_port":"127.0.0.1:54321","healthy":true,"last_ping":1512982506643,"pid":97891,"num_cpus":4,"cpus_allowed":4,"nthreads":4,"sys_load":2.0917969,"my_cpu_pct":-1,"sys_cpu_pct":-1,"mem_value_size":17408,"pojo_mem":12224512,"free_mem":1896688640,"max_mem":1908930560,"swap_mem":0,"num_keys":56,"free_disk":0,"max_disk":0,"rpcs_active":0,"fjthrds":[-1,1,1,1,1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,1,-1,1,0,0,0,0,0,0,0],"fjqueue":[-1,0,0,0,0,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,0,-1,0,0,0,0,0,0,0,0],"tcps_active":0,"open_fds":-1,"gflops":4.598999977111816,"mem_bw":6.423728128E9}],"internal_security_enabled":false}
Time: 2017-12-11 11:55:09.251
GET
http://localhost:54321/3/ImportFiles?path=http%3A%2F%2Flocalhost%3A8082%2Fdatasets%2Ftables%2Fcsv%2FvPrzC5TOQr6JTvnAYrU5AKyz8SP4ao8p.csv&pattern=
postBody:
curlError: FALSE curlErrorMessage: httpStatusCode: 200
httpStatusMessage: OK millis: 6
{"__meta":{"schema_version":3,"schema_name":"ImportFilesV3","schema_type":"ImportFiles"},"_exclude_fields":"","path":"http://localhost:8082/datasets/tables/csv/vPrzC5TOQr6JTvnAYrU5AKyz8SP4ao8p.csv","pattern":"","files":[],"destination_frames":[],"fails":["http://localhost:8082/datasets/tables/csv/vPrzC5TOQr6JTvnAYrU5AKyz8SP4ao8p.csv"],"dels":[]}
This is my import information.
NullPointerException while analyzing a project in SonarQube
I am getting a "Synchronization issues have encountered a problem" along with a NullPointerException while analyzing a project in SonarQube 6.2. I am using Eclipse Mars 4.5.2 and I tried using SonarLint but didnt find it very useful because it didnt show me SonarQube Issues view and all the errors were not listed in there. So, I instead downloaded the SonarQube plugin (http://downloads.sonarsource.com/eclipse/eclipse/) I see the below issues in the access.log (in SonarQube directory) - 127.0.0.1 - - [15/Feb/2017:11:40:50 -0500] "GET /api/resources?resource=ffsUser%3AffsUser&verbose=false& HTTP/1.1" 200 237 "-" "Apache-HttpClient/4.3.6 (java 1.5)" "AVpCmoaydo39Zfz7AAAv" 127.0.0.1 - - [15/Feb/2017:11:40:50 -0500] "GET /api/issues/search?componentRoots=ffsUser:ffsUser&pageIndex=1&pageSize=-1&resolved=false HTTP/1.1" 200 - "-" "Java/1.8.0_45" "AVpCmoaydo39Zfz7AAAw"
It's not surprising that you're encountering problems attempting to run the old Eclipse plugin against a current version of SonarQube. There are reasons that plugin as abandoned. Your best bet would be to try again with SonarLint, which is designed to help you keep new code clean and to clean up the code you're already working on anyway
G-WAN v7.12.6 can't visit the static contents
G-WAN out-of-box is running no problem with the dynamic contents but can't visit the static contents. I ran G-WAN with kk user(not root, no sudo). All files and directories are of kk user/group. I installed it in localhost. When I typed 127.0.0.1:8080 in the browser, it returned: Server not found Firefox can't find the server at www.index.html. It is strange that the returned server name was www.index.html When I typed 127.0.0.1:8080/index.html, it returned 404 file not found. How can I workaround these, until a new release is out? This time, there is no error message in the log file. (I installed G-WAN in Ubuntu 15.10) UPDATE: ------------------------ There are two strange info from the served_from.c, as below: This page was processed... Using get_env(): by the Server: 0.0.0.0:8080:8080(hostname: 127.0.0.1) ^^^^^^^^^^^^^^^^^(Should it be only one '8080'?) Virtual Host: /home/kk/dev/gwan_v7.12.6/0.0.0.0:8080/#0.0.0.0 HTTP method: GET HTTP request: /?served_from HTTP query: served_from HTTP entity: (null) Content-Encoding: 0 Content-Length: 0 Content-Type: 0 for the Client: 127.0.0.1:43199 Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:47.0) Gecko/20100101 Firefox/47.0 Using HTTP Headers to get the same information: by the Server: 27.0.0.1:8080 (hostname: 27.0.0.1) ^^^^^^^^^^^^^(Should it be 127.0.0.1 ? The leading '1' is missing.) HTTP method: GET HTTP entity: - Content-Encoding: 0 Content-Length: 0 Content-Type: 0 for the Client: 127.0.0.1:43199 ozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:47.0) Gecko/20100101 Firefox/47.0 Acpt-Language: n-US,en;q=0.5 Acpt-Encoding: 3: |GZIP|DEFLATE Cookies: - Is there anything wrong in v7.12.6? UPDATE 2 ================================ with the following steps, G-WAN v7.12.6 can't show static contents: 1. all files and directories are of the same user/group (i.e. kk) 2. out-of-box, listen to 8080 3. run ./gwan cannot visit static contents. with following changes, G-WAN works normally: 1. change directory 0.0.0.0.8080 to 0.0.0.0:80 2. run sudo ./gwan -d:kk:kk everything is normal. sorry for my mis-typing in the UPDATE2. It should be 0.0.0.0:80 (emphasize the colon). Actually, I typed it right in the directory name. Repeat: ./gwan should work according to G-WAN's document, since the user launching the program is owning all the files and directories.
If you can query the server for scripts then G-WAN is running fine. The problem is obviously a file permission issue, hence the error 404 file not found you got for the query 127.0.0.1:8080/index.html. This system configuration problem will affect any program, not specifically G-WAN. Consult our dedicated FAQ to find how to correct it. For the served_from.c script, feel free to display the IP address in the "correct" order and to cut the string at the expected position to avoid the listener port repetition. UPDATE: Your latest update says "changing directory 0.0.0.0.8080 to 0.0.0.0:80 works". This was the problem: "0.0.0.0.8080" has never been a valid syntax for G-WAN.
magento logs out everytime I run an import
Everytime I run an import profile in Magento after about 2 - 3 minutes it logs me out. I have checked the apache error logs, this is the lead up to the log out: [13/Sep/2012:14:58:41 +0100] "POST /index.php/admin/system_convert_gui/batchRun/key/151d60f07b9e68b0023da294f315c3cc/?isAjax=true HTTP/1.1" 200 52 [13/Sep/2012:14:58:42 +0100] "POST /index.php/admin/system_convert_gui/batchRun/key/151d60f07b9e68b0023da294f315c3cc/?isAjax=true HTTP/1.1" 200 52 [13/Sep/2012:14:58:44 +0100] "POST /index.php/admin/system_convert_gui/batchRun/key/151d60f07b9e68b0023da294f315c3cc/?isAjax=true HTTP/1.1" 200 148 [13/Sep/2012:14:58:44 +0100] "GET /index.php/admin/index/login/key/70f0f08a622f871979b3890c5f769bf4/ HTTP/1.1" 200 1001 In the system log I get: 2012-09-13T14:13:00+00:00 DEBUG (7): Cannot send headers; headers already sent in /var/www/vhosts/httpdocs/app/code/core/Mage/Adminhtml/Block/System/Convert/Profile/Run.php, line 278 2012-09-13T14:13:00+00:00 DEBUG (7): HEADERS ALREADY SENT: [0] /var/www/vhosts/httpdocs/app/code/core/Mage/Core/Controller/Response/Http.php:44 Nothing in the exception log. I have researched the headers already sent issue but nothing has fixed this. Any suggestions would be greatly appreciated. Magento version 1.4.0.1
It's an annoyance. I get it on ee 1.8 during any import/export process that counts the items processed as it goes. I solved it by adding ob_start() just before that line throwing the error. I realize I should override the function, but I'm using an old version and will upgrade soon. Hopefully this is fixed in 1.12