FTP upload no data error - ftp

I'm getting a weird error that only appears when I try to run this code on a Linux server, but it works perfectly on my local Mac. I'm not sure if anyone else has ran into this issue and can give me insight.
import ftplib
import sys, os
files_to_drop = set()
server = ""
username = ""
password = ""
ftp_connection = ftplib.FTP(server, username, password)
remote_path = "/incoming/"
ftp_connection.cwd(remote_path)
local_path = '/var/www/analytics/'
def deliver_via_ftp():
fh = open(local_path + files_to_drop[0], 'rb')
ftp_connection.storbinary('STOR {}'.format(files_to_drop[0]), fh)
fh.close()
The error I'm receiving on the Linux machine is below:
*cmd* 'CWD /incoming/'
*put* 'CWD /incoming/\r\n'
*get* '250 Directory successfully changed.\n'
*resp* '250 Directory successfully changed.'
starting to drop files.
*cmd* 'TYPE I'
*put* 'TYPE I\r\n'
*get* '500 OOPS: vsf_sysutil_recv_peek: no data\n'
*resp* '500 OOPS: vsf_sysutil_recv_peek: no data'
Traceback (most recent call last):
File "manage_batches.py", line 258, in <module>
m.run()
File "manage_batches.py", line 253, in run
self.__deliver_via_ftp()
File "manage_batches.py", line 215, in __deliver_via_ftp
self.ftp_connection.storbinary('STOR {}'.format(file_name), fh)
File "/usr/lib/python3.5/ftplib.py", line 502, in storbinary
self.voidcmd('TYPE I')
File "/usr/lib/python3.5/ftplib.py", line 277, in voidcmd
return self.voidresp()
File "/usr/lib/python3.5/ftplib.py", line 250, in voidresp
resp = self.getresp()
File "/usr/lib/python3.5/ftplib.py", line 245, in getresp
raise error_perm(resp)
ftplib.error_perm: 500 OOPS: vsf_sysutil_recv_peek: no data
I found a thread here (https://github.com/mscdex/node-ftp/issues/50) regarding the problem, but seems to point to repeatedly opening and closing of the connection. I'm reusing the same connection for all files and it works on my Mac.
Debug trace on Mac:
*cmd* 'CWD /incoming/'
*put* 'CWD /incoming/\r\n'
*get* '250 Directory successfully changed.\n'
*resp* '250 Directory successfully changed.'
*cmd* 'TYPE I'
*put* 'TYPE I\r\n'
*get* '200 Switching to Binary mode.\n'
*resp* '200 Switching to Binary mode.'
*cmd* 'PASV'
*put* 'PASV\r\n'
*get* '227 Entering Passive Mode (<hidden IP>,11,133).\n'
*resp* '227 Entering Passive Mode (<hidden IP>,11,133).'
*cmd* 'STOR 11062017_0812_AM.txt'
*put* 'STOR 11062017_0812_AM.txt\r\n'
*get* '150 Ok to send data.\n'
*resp* '150 Ok to send data.'
*get* '226 Transfer complete.\n'
*resp* '226 Transfer complete.'
file dropped /Users/f/Desktop/bucket/11062017_0812_AM.txt
Can someone point me in the right direction?
Thanks
EDIT: I cleaned up the code to remove the class structure and I added the debug output set to level 2.

Related

Paho on_connect works if connection OK but is not called when error. Why?

Why does my paho-mqtt (1.5.1) on_connect work if connection OK but it is not called when an error. For testing I'm using Linux Lite 4.2 based on Ubuntu 18.04 LTS with Xfcs running in a VM (VBox).
class subscribemqtt:
.....
def on_connect(self, client, userdata, flags, rc):
print ("ZZZZZZZZZZZZZ in on_connect")
connectErrs = {.........}
self.connectRc = rc
self.connectReason = connectErrs[str(rc)]
print ("$$$$$$$$$$$$$", self.connectRc, self.connectReason)
return
def subscribe(self, arguments):
...........
self.client = paho.Client(self.config['CLIENT_ID'])
self.client.on_message = self.on_subscribe
self.client.on_connect = self.on_connect
print ("#############", self.on_connect)
print ("XXXXXXXXXXXX calling self.client.connect(...)
self.client.connect(self.config['HOST'],self.config['PORT'])
print ("YYYYYYYYYYYYY calling self.client.loop_start()")
self.client.loop_start()
print ("AAAAAAAAAAAAA", self.connected)
while not self.connected:
time.sleep(0.1)
print ("BBBBBBBBBBBBB", self.connected, self.connectRc)
When all the parameters are correct, on_connect gets called:
############# <bound method subscribemqtt.on_connect of <__main__.subscribemqtt object at 0x7f32065a6ac8>>
XXXXXXXXXXXX calling self.client.connect(self.config['HOST'],self.config['PORT']
YYYYYYYYYYYYY calling self.client.loop_start()
AAAAAAAAAAAAA**ZZZZZZZZZZZZZ in on_connect**
False$$$$$$$$$$$$$
0 Connection successful
BBBBBBBBBBBBB True 0
When I set the host address to an invalid address (to create an error to test my error handling) I get:
subscribemqtt.subscribe:topic= Immersion Dummy
############# <bound method subscribemqtt.on_connect of <__main__.subscribemqtt object at 0x7ffb942ae0b8>>
XXXXXXXXXXXX calling self.client.connect(self.config['HOST'],self.config['PORT']
Traceback (most recent call last):
File "/home/linuxlite/Desktop/EMS/sendroutines/subscribemqtt.py", line 275, in <module>
(retcode, reason, topic) = subscribeObj.subscribe([None, topic])
File "/home/linuxlite/Desktop/EMS/sendroutines/subscribemqtt.py", line 191, in subscribe
self.client.connect(self.config['HOST'],self.config['PORT'])
File "/usr/local/lib/python3.6/dist-packages/paho/mqtt/client.py", line 941, in connect
return self.reconnect()
File "/usr/local/lib/python3.6/dist-packages/paho/mqtt/client.py", line 1075, in reconnect
sock = self._create_socket_connection()
File "/usr/local/lib/python3.6/dist-packages/paho/mqtt/client.py", line 3546, in _create_socket_connection
return socket.create_connection(addr, source_address=source, timeout=self._keepalive)
File "/usr/lib/python3.6/socket.py", line 704, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
>>>
Thanks for reading.
Alan
PS. I just tried:
try:
self.client.connect(self.config['HOST'],self.config['PORT'])
except:
print ("**************** Error exception calling self.client.connect")
And that works but my understanding is that on_connect should be called for errors.
From the docs:
on_connect()
on_connect(client, userdata, flags, rc)
Called when the broker responds to our connection request.
The important part there is "when the broker responds". But in the example you have shown the hostname provided can not be resolved, so broker never responds because it is never actually contacted.
on_connect() will be called if connection succeeds or if it fails because the username/password is wrong or a unavailable protocol version (e.g. requesting MQTTv5 from a broker that only supports v3)

HUE File Manager, able to create HDFS subdirectory/folder but unable to upload file to HDFS

I am getting an:
Error: Undefined message in HUE every time I try to upload a file.
I am able to create a subdirectory/folder in HDFS but file uploads are not working.
I tried copying a file to HDFS from the linux CLI using the hadoop user, and it works.
Hue user is hadoop
HDFS directory owner is hadoop:hadoop
Edit: Adding the error
ERROR Internal Server Error: /filebrowser/upload/file
Traceback (most recent call last):
File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.20-py2.7.egg/django/core/handlers/exception.py", line 41, in inner
response = get_response(request)
File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.20-py2.7.egg/django/core/handlers/base.py", line 249, in _legacy_get_response
response = self._get_response(request)
File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.20-py2.7.egg/django/core/handlers/base.py", line 178, in _get_response
response = middleware_method(request, callback, callback_args, callback_kwargs)
File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.20-py2.7.egg/django/middleware/csrf.py", line 300, in process_view
request_csrf_token = request.POST.get('csrfmiddlewaretoken', '')
File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.20-py2.7.egg/django/core/handlers/wsgi.py", line 126, in _get_post
self._load_post_and_files()
File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.20-py2.7.egg/django/http/request.py", line 299, in _load_post_and_files
self._post, self._files = self.parse_file_upload(self.META, data)
File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.20-py2.7.egg/django/http/request.py", line 258, in parse_file_upload
return parser.parse()
File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.20-py2.7.egg/django/http/multipartparser.py", line 269, in parse
self._close_files()
File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.20-py2.7.egg/django/http/multipartparser.py", line 316, in _close_files
handler.file.close()
AttributeError: 'NoneType' object has no attribute 'close'
[12/Apr/2020 22:48:51 -0700] upload DEBUG HDFSfileUploadHandler receive_data_chunk
[12/Apr/2020 22:48:51 -0700] upload ERROR Not using HDFS upload handler:
[12/Apr/2020 22:48:51 -0700] resource ERROR All 1 clients failed: {'http://IRedactedMyinstanceIdentHere.ap-southeast-1.compute.internal:14000/webhdfs/v1': u'500 Server Error: Internal Server Error for url: http://IRedactedMyinstanceIdentHere.ap-southeast-1.compute.internal:14000/webhdfs/v1/user/hadoop/Test-Data?op=CHECKACCESS&fsaction=rw-&user.name=hue&doas=hadoop\n{"RemoteException":{"message":"java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.http.client.HttpFSFileSystem.Operation.CHECKACCESS","exception":"QueryParamException","javaClassName":"com.sun.jersey.api.ParamException$QueryParamException"}}\n'}
[12/Apr/2020 22:48:51 -0700] resource ERROR Caught exception from http://IRedactedMyinstanceIdentHere.ap-southeast-1.compute.internal:14000/webhdfs/v1: 500 Server Error: Internal Server Error for url: http://IRedactedMyinstanceIdentHere.ap-southeast-1.compute.internal:14000/webhdfs/v1/user/hadoop/Test-Data?op=CHECKACCESS&fsaction=rw-&user.name=hue&doas=hadoop
{"RemoteException":{"message":"java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.http.client.HttpFSFileSystem.Operation.CHECKACCESS","exception":"QueryParamException","javaClassName":"com.sun.jersey.api.ParamException$QueryParamException"}}
(error 500)
As you can see from the error message, it reports that there is no match found for the query param passed when hue tries to perform the CHECKACCESS operation.
http://IRedactedMyinstanceIdentHere.ap-southeast-1.compute.internal:14000/webhdfs/v1/user/hadoop/Test-Data?op=CHECKACCESS&fsaction=rw-&user.name=hue&doas=hadoop
{"RemoteException":{"message":"java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.http.client.HttpFSFileSystem.Operation.CHECKACCESS","exception":"QueryParamException","javaClassName":"com.sun.jersey.api.ParamException$QueryParamException"}}
This implementation seems to be missing in some of the hadoop versions and is a known bug HTTPFS - CHECKACCESS operation missing.

geocoder Service error in python

I am having a service error as I am trying to run geocoders from geopy to get a location coordinates in python.I have set my proxy config, given it an api_key.
Im not sure why I am getting this error. From the search that I have done on this, it seems that this might be a proxy issue but I have that set up already.
What could be the problem here?
This is my code :
from geopy import geocoders
proxies={'http': 'http://location:port', 'https': 'http://localhost:port'}
api_key = '.......'
g = geocoders.GoogleV3(api_key=api_key,proxies=proxies, timeout=10)
location = 'Mountain View, CA'
try:
place, (lat, lng) = g.geocode(location)
except ValueError as error_message:
print("Error: geocode failed on input %s with message %s" % (location, error_message))
this is my error output:
Traceback (most recent call last):
File "/Users/aqm1152/anaconda/lib/python3.5/urllib/request.py", line 1254, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "/Users/aqm1152/anaconda/lib/python3.5/http/client.py", line 1106, in request
self._send_request(method, url, body, headers)
File "/Users/aqm1152/anaconda/lib/python3.5/http/client.py", line 1151, in _send_request
self.endheaders(body)
File "/Users/aqm1152/anaconda/lib/python3.5/http/client.py", line 1102, in endheaders
self._send_output(message_body)
File "/Users/aqm1152/anaconda/lib/python3.5/http/client.py", line 934, in _send_output
self.send(msg)
File "/Users/aqm1152/anaconda/lib/python3.5/http/client.py", line 877, in send
self.connect()
File "/Users/aqm1152/anaconda/lib/python3.5/http/client.py", line 1260, in connect
server_hostname=server_hostname)
File "/Users/aqm1152/anaconda/lib/python3.5/ssl.py", line 377, in wrap_socket
_context=self)
File "/Users/aqm1152/anaconda/lib/python3.5/ssl.py", line 752, in __init__
self.do_handshake()
File "/Users/aqm1152/anaconda/lib/python3.5/ssl.py", line 988, in do_handshake
self._sslobj.do_handshake()
File "/Users/aqm1152/anaconda/lib/python3.5/ssl.py", line 633, in do_handshake
self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:645)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/aqm1152/anaconda/lib/python3.5/site-packages/geopy/geocoders/base.py", line 143, in _call_geocoder
page = requester(req, timeout=(timeout or self.timeout), **kwargs)
File "/Users/aqm1152/anaconda/lib/python3.5/urllib/request.py", line 163, in urlopen
return opener.open(url, data, timeout)
File "/Users/aqm1152/anaconda/lib/python3.5/urllib/request.py", line 466, in open
response = self._open(req, data)
File "/Users/aqm1152/anaconda/lib/python3.5/urllib/request.py", line 484, in _open
'_open', req)
File "/Users/aqm1152/anaconda/lib/python3.5/urllib/request.py", line 444, in _call_chain
result = func(*args)
File "/Users/aqm1152/anaconda/lib/python3.5/urllib/request.py", line 1297, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/Users/aqm1152/anaconda/lib/python3.5/urllib/request.py", line 1256, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error EOF occurred in violation of protocol (_ssl.c:645)>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/aqm1152/Documents/TestingCode/ACERT/test_1.py", line 12, in <module>
place, (lat, lng) = g.geocode(location)
File "/Users/aqm1152/anaconda/lib/python3.5/site-packages/geopy/geocoders/googlev3.py", line 217, in geocode
self._call_geocoder(url, timeout=timeout), exactly_one
File "/Users/aqm1152/anaconda/lib/python3.5/site-packages/geopy/geocoders/base.py", line 171, in _call_geocoder
raise GeocoderServiceError(message)
geopy.exc.GeocoderServiceError: EOF occurred in violation of protocol (_ssl.c:645)
I modified your code as below, as I don't use proxy and apply an API, and it works in my machine.
from geopy import geocoders
#proxies={'http': 'http://location:port', 'https': 'http://localhost:port'}
#api_key = '.......'
g = geocoders.GoogleV3()
location = 'Mountain View, CA'
try:
place, (lat, lng) = g.geocode(location)
except ValueError as error_message:
print("Error: geocode failed on input %s with message %s" % (location, error_message))
print (place, lat, lng)
Here is the result.
Mountain View, CA, USA 37.3860517 -122.0838511
From the trackback, it looks all the errors are related to SSL. You may try to add below code at the beginning to disable SSL Certification verification as an alternative.
import ssl
# Disable SSL certificate verification
try:
_create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
# Legacy Python that doesn't verify HTTPS certificates by default
pass
else:
# Handle target environment that doesn't support HTTPS verification
ssl._create_default_https_context = _create_unverified_https_context
I have encountered similar problem in Anaconda running Python 2.7 in a env with the root running Python 3.6. The problem wasn't encountered in Anaconda with both root and env running Python 2.7 in another machine.
In addition, this SO post about proxy server set up for geopy using os module may be good reference.
Hope this help.

pydrive.auth.RefreshError: No refresh_token found

I am trying to download file from google drive with pydrive. code is modified from Automating pydrive verification process
And it is giving following error.
$ python driveDownload.py a a.xls
Google Drive Token Expired, Refreshing
Traceback (most recent call last):
File "driveDownload.py", line 26, in <module>
gauth.Refresh()
File "build/bdist.linux-x86_64/egg/pydrive/auth.py", line 369, in Refresh
pydrive.auth.RefreshError: No refresh_token found.Please set access_type of OAuth to offline.
Hence I followed instructions as per stackoverflow here
Unable to get it working. Am I missing something basic ?
my settings.yaml is below
client_config_backend: settings
client_config:
client_id: ###
client_secret: ###
save_credentials: True
save_credentials_backend: file
save_credentials_file: credentials.json
get_refresh_token: True
oauth_scope:
- https://www.googleapis.com/auth/drive.file
- https://www.googleapis.com/auth/drive.install
I am getting following error :
$ python driveDownload.py a a.xls
Google Drive Token Expired, Refreshing
Traceback (most recent call last):
File "driveDownload.py", line 26, in <module>
gauth.Refresh()
File "build/bdist.linux-x86_64/egg/pydrive/auth.py", line 369, in Refresh
pydrive.auth.RefreshError: No refresh_token found.Please set access_type of OAuth to offline.
my code is driveDownload.py :
from pydrive.auth import GoogleAuth
import os
from pydrive.drive import GoogleDrive
import sys
#USAGE: python driveDownload.py googleDriveName localFileName
#authentication via the browser
#note... client_secrets.json is required for this that leads
#to the user authentication
fileName2Download = sys.argv[1]
#myLocalPath is hard coded and
myLocalPath = sys.argv[2]
# gauth = GoogleAuth()
gauth = GoogleAuth()
# Try to load saved client credentials
gauth.LoadCredentialsFile("GoogleDriveCredentials.txt")
if gauth.credentials is None:
# Authenticate if they're not there
gauth.LocalWebserverAuth()
elif gauth.access_token_expired:
# Refresh them if expired
print "Google Drive Token Expired, Refreshing"
gauth.Refresh()
else:
# Initialize the saved creds
gauth.Authorize()
# Save the current credentials to a file
gauth.SaveCredentialsFile("GoogleDriveCredentials.txt")
#accessing the drive
drive = GoogleDrive(gauth)
# Auto-iterate through all files that matches this query
file_list = drive.ListFile().GetList()
for file1 in file_list:
#download the file hello.txt ad myDownload.txt
print 'title: %s, id: %s , mimeType = %s' % (file1['title'], file1['id'],file1['mimeType'])
if file1['title'] == fileName2Download:
print ' file Title identiefied :%s'% (file1['title'])
# contentString = file1.GetContentString()
# print 'contentString : %s' % (contentString)
myFile = drive.CreateFile({'id': file1['id']})
if os.path.exists(myLocalPath):
os.remove(myLocalPath)
myFile.GetContentFile(myLocalPath,mimetype='application/vnd.openxmlformats-officedocument.spreadsheetml.sheet')
print '%s downloaded ' %(myLocalPath)
break

Django : error: [Errno 10053] An established connection was aborted by the software in your host machine

I got error when i perform ajax request. Without authentication lines user = authenticate(username = user_username, password = user_password) in the views.py, success function is called. And if i add, error function is called with Errno 10053.
I am using MySql in Wamp. Why it is happening ?
views.py
class LoginVerify(View):
print('login')
def post(self,request,*args,**kwargs):
if request.is_ajax():
print("post called")
user_email = request.POST.get('email',False)
user_password = request.POST.get('pswd',False)
print(user_email)
try:
user_username = User.objects.get(email=user_email).username
user = authenticate(username = user_username, password = user_password)
except:
print("error occured")
return HttpResponse("response form server")
def get(self, request,*args,**kwargs):
print("get method")
return render(request,'feeds/feeds_home.html')
ajax request:
$(document).ready(function(){
$("#submit").on("click",function(){
var $email = $("#signin-email").val();
var $pswd = $("#signin-password").val();
alert($pswd);
$.ajax({
url : '{% url "feeds:login_view" %}',
type: "POST",
data: {csrfmiddlewaretoken :"{{ csrf_token }}", pswd : $pswd, email: $email},
success: function(data){
location.reload();
},
error: function(){
alert("fails");
}
});
});
Tracebrack
post called
vivek.ananthan.m.s#gmail.com
[19/Apr/2015 11:10:22] "POST / HTTP/1.1" 200 12
Traceback (most recent call last):
File "C:\Python27\lib\wsgiref\handlers.py", line 86, in run
self.finish_response()
File "C:\Python27\lib\wsgiref\handlers.py", line 127, in finish_response
self.write(data)
File "C:\Python27\lib\wsgiref\handlers.py", line 210, in write
self.send_headers()
File "C:\Python27\lib\wsgiref\handlers.py", line 268, in send_headers
self.send_preamble()
File "C:\Python27\lib\wsgiref\handlers.py", line 192, in send_preamble
'Date: %s\r\n' % format_date_time(time.time())
File "C:\Python27\lib\socket.py", line 324, in write
self.flush()
File "C:\Python27\lib\socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 10053] An established connection was aborted by the software in your host machine
Traceback (most recent call last):
File "C:\Python27\lib\SocketServer.py", line 582, in process_request_thread
self.finish_request(request, client_address)
File "C:\Python27\lib\SocketServer.py", line 323, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "C:\Python27\lib\site-packages\django-1.7-py2.7.egg\django\core\servers\basehttp.py", line 129, in __init__
super(WSGIRequestHandler, self).__init__(*args, **kwargs)
File "C:\Python27\lib\SocketServer.py", line 640, in __init__
self.finish()
File "C:\Python27\lib\SocketServer.py", line 693, in finish
self.wfile.flush()
File "C:\Python27\lib\socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 10053] An established connection was aborted by the software in your host machine
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 52490)
Please explain where i made the mistake and why it is happening.
Thanks in advance !!
Reference: https://stackoverflow.com/a/17854758/3940406
From the Windows Sockets Error Codes list:
WSAECONNABORTED 10053 Software caused connection abort. An established
connection was aborted by the software in your host computer, possibly
due to a data transmission time-out or protocol error.
There was a timeout or other network-level error. This is your operating system closing the socket, nothing to do with Python, django or Flask, really.
It could be the remote browser stopped responding, the network connection died, or a firewall closed the connection because it was open too long, or any other number of reasons.
I came by the question when I was trying to research a problem with regards to running a mysql code using the pymysql as the python sql client. I happen to have this same issue. Renaming the config setting for mysql and then restarting your system or running mysql server. Works to fix this issue

Resources