I'm trying to parse the following gml as geopandas dataframe, which did work when trying to parse a csv string as pandas dataframe.
import geopandas as gpd
from io import StringIO
gml = '<gml:MultiPolygon srsName="EPSG:4326"><gml:polygonMember><gml:Polygon> <gml:outerBoundaryIs><gml:LinearRing><gml:coordinates cs="," ts=" ">-65.84480689655171659,-20.69851241379309315 -65.84480689655171659,-21.53409718390803818 -65.00922212643676801,-20.69851241379309315 -65.74453672413791594,-20.69851241379309315 -65.84480689655171659,-20.69851241379309315</gml:coordinates></gml:LinearRing></gml:outerBoundaryIs></gml:Polygon></gml:polygonMember></gml:MultiPolygon>'
sh = gpd.read_file(StringIO(gml))
This however throws the following error
TypeError: Constructor argument must be a file opened in binary mode or bytes/bytearray.
Does anyone have any suggestions how I can parse this gml string to a geopandas dataframe?
Related
I'm trying to setup a .py plugin that will save decoded Protobuf responses to file, but whatever I do, the result is always file in byte format (not decoded). I have also tried to do the same by using "w" in Mitmproxy - although on screen I saw decoded data, in the file it was encoded again.
Any thoughts how to do it correctly?
Sample code for now:
import mitmproxy
def response(flow):
# if flow.request.pretty_url.endswith("some-url.com/endpoint"):
if flow.request.pretty_url.endswith("some-url.com/endpoint"):
f = open("test.log","ab")
with decoded(flow.response)
f.write(flow.request.content)
f.write(flow.response.content)
Eh, I'm not sure this helps, but what happens if you don't open the file in binary mode
f = open("test.log","a")
?
Hy,
some basic things that I found.
Try replacing
f.write(flow.request.content)
with
f.write(flow.request.text)
I read it on this website
https://discourse.mitmproxy.org/t/modifying-https-response-body-not-working/645/3
Please read and try this to get the requests and responses assembled.
MITM Proxy, getting entire request and response string
Best of luck with your project.
I was able to find the way to do that. Seems mitmdump or mitmproxy wasn't able to save raw decoded Protobuf, so I used:
mitmdump -s decode_script.py
with the following script to save the decoded data to a file:
import mitmproxy
import subprocess
import time
def response(flow):
if flow.request.pretty_url.endswith("HERE/IS/SOME/API/PATH"):
protobuffedResponse=flow.response.content
(out, err) = subprocess.Popen(['protoc', '--decode_raw'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate(protobuffedResponse)
outStr = str(out, 'utf-8')
outStr = outStr.replace('\\"', '"')
timestr = time.strftime("%Y%m%d-%H%M%S")
with open("decoded_messages/" + timestr + ".decode_raw.log","w") as f:
f.write(outStr)
i have a telnet-server running here (from a vendor) which is implemented in a barcode reader. to get the data, i simply need to connect to the telnetserver port 23 with my telnetclient from commandline.
What i want to do is doing this with ruby or python and write the output to a file.
so whats work until now:
i can connect to the telnet-server.
import sys
import telnetlib
tn = telnetlib.Telnet("10.0.0.138")
tn.close
output = tn.read_all()
# write to a file
with open("logging.txt", "wb") as f:
f.write(output)
# Check that the file wrote correctly.
with open("logging.txt", "rb") as f:
print(f.read())
What is not working:
writing the output from the telnetserver to a textfile.
it doesnt matter for me if i do it with python or ruby, both languages are fine.
my codesample is python here. just for trying.
thanks for reading.
Using pandas 18.1...
I'm trying to iterate through a folder of CSVs to read each CSV and send it to an Oracle database table. There is a non-ascii character lurking in one of my many CSVs (more like reveling in my anguish). I keep getting this error:
UnicodeEncodeError: 'ascii' codec can't encode character '\xab' in position 8:
ordinal not in range(128)
Here's the code:
import pandas as pd
import pandas.io.sql as psql
from sqlalchemy import create_engine
import cx_Oracle as cx
engine = create_engine('oracle+cx_oracle://schema:'+pwd+'#server:port/service_name'
,encoding='latin1')
name='table'
path=r'path_to_folder'
filelist = os.listdir(path)
for file in filelist:
df = pd.read_csv(pathc+'\\'+file,encoding='latin1',index_col=0)
df=df.astype('unicode')
df['date'] = pd.to_datetime(df['date'])
df['date'] = pd.to_datetime(df['Contract_EffDt'],format='%YYYY-%mm-%dd')
df.to_sql(name, engine, if_exists = 'append')
I've tried the following:
encoding=utf-8 (in the engine, if I do that in read_csv, it throws an error)
Adding ?encoding=utf8 after "service_name" in the engine
Using df=df.astype('unicode') (and not)
What I want to do:
Replace the unreadable character with something else and, most importantly, proceed with sending data to Oracle.
Note:
The data file I'm using are from the cms.gov site. Here's a zip file with an example. I'm using the "contracts_info" file.
Thanks in advance!
You need to set the NLS_LANG environment variable like this:
os.environ['NLS_LANG']= 'AMERICAN_AMERICA.AL32UTF8'
Then the error won't occur.
I encoded string fields to utf-8 individually and this may have helped (a new error occurred, but I assume it is not related to this):
dfc['Organization Type'] = dfc['Organization Type'].str.encode('utf-8')
New error:
DatabaseError: (cx_Oracle.DatabaseError) ORA-00904: "Contract_ID": invalid identifier
This was because "Contract_ID" was not set as the index. Once I did that, all went well (except for being slower than molasses, which begins my next adventure).
I have been trying to read a jpeg file using PIL in python 3.4. I need to save this file into string format. Although some options are provided on this site but I have tried a few but it is not working. Following is my code snippet which i have found on this site only:-
from io import StringIO
fp = Image.open("images/login.jpg")
output = StringIO()
fp.save(output, format="JPEG")
contents = output.getvalue()
output.close()
But i am facing the following error :-
TypeError: string argument expected, got 'bytes'
Could you please suggest what I have done wrong and how to get this working?
In python 3 you should use a BytesIO,
whereas as read in python docs:
StringIO is a native in-memory unicode container
.
Thanks a lot for the hint. I Actually have a found a different way of reading the image file and storing in string object in python2.x . Here is the code. Please let me know if there is any disadvantage of using this.
imgText = open("images/login.jpg", 'rb')
imgTextStr = imgText.read()
imgText.close()
I want to embed a small python script inside a bash script so that I can send a json object to a socket.
I have tried the following code:
python -c "$(cat << 'EOF'
import socket
import json
data = {'ip':192.168.1.150', 'status':'up'}
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('127.0.0.1', 13373))
s.send(json.dumps(data))
result = json.loads(s.recv(1024))
print result
s.close()
EOF
)"
and this:
python -c "
import socket
import json
data = {'ip':192.168.1.150', 'status':'up'}
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('127.0.0.1', 13373))
s.send(json.dumps(data))
result = json.loads(s.recv(1024))
print result
s.close()
"
But I keep getting the following error:
data = {'ip':192.168.1.150', 'status':'up'}
^
SyntaxError: invalid syntax
I'm assuming this is because bash is interpreting this, not python. I've tested the code in a python script, and it works.
Also, do I need the -c option?
Apologies, I'm completely inexperienced in python, I've written some quite extensive bash scripting for the project I'm working on and need to send the output of these to sockets as json objects. Such a small snipped of embedded Python code seems by far the simplest answer, unless anyone has other suggestions.
Python version installed on CentOS Python 2.6.6
The problem that you're having that results in the SyntaxError is that you don't have an opening single quote on the IP value in the data dict. You have:
data = {'ip':192.168.1.150', 'status':'up'}
You need:
data = {'ip':'192.168.1.150', 'status':'up'}