How can I access an unsupported Wand API via the Python Wand interface? For example, I wish to call the Wand API MagickAddNoiseImage but it is not available in the Python interface.
Accessing unsupported APIs is pretty easy with wand.api, but you will need to open-up ImageMagick's docs/header files for reference.
from wand.api import library
import ctypes
# Re-create NoiseType enum
NOISE_TYPES = ('undefined', 'uniform', 'gaussian', 'multiplicative_gaussian',
'impulse', 'laplacian', 'poisson', 'random')
# Map API i/o
library.MagickAddNoiseImage.argtypes = [ctypes.c_void_p,
ctypes.c_uint]
library.MagickAddNoiseImage.restype = ctypes.c_int
# Extend wand's Image class with your new API
from wand.image import Image
class MySupportedImage(Image):
def add_noise(self, noise_type):
"""My MagickAddNoiseImage"""
if noise_type not in NOISE_TYPES:
self.raise_exception()
return library.MagickAddNoiseImage.argtypes(self.resource,
NOISE_TYPES.index(noise_type))
If your solution works out, think about submitting your solution back to the community (after you've created a solid unit test.)
Related
I'm trying to add tag to a folder using Windows API, and I've stumbled upon this article.
According in this article,
to create an ANSI simple property set, you would call IPropertySetStorage::Create to create the property set, specifying PROPSETFLAG_ANSI (simple is the default type of property set), then write to it with a call to IPropertyStorage::WriteMultiple. To read the property set, you would call IPropertyStorage::ReadMultiple.
I tried doing it as the code below but get stuck because I can't find where the IPropertySetStorage and IPropertyStorage is.
from ctypes import sizeof
STGM_READWRITE = 0x00000002
def addTag(folder):
shell32 = ctypes.windll.shell32
_clsid = None # should to be the CLSID from the SHFOLDERCUSTOMSETTINGS
# ↓↓↓↓↓↓ I can't find where should I get IPropertySetStorage
shell32.IPropertySetStorage.Create("{F29F85E0-4FF9-1068-AB91-08002B27B3D9}", _clsid, 0, STGM_READWRITE)
rglpwstrName = ["TestTag"]
rgpropid = ["prop5"]
cpropid = sizeof(rgpropid)
# ↓↓↓↓↓↓ I can't find where should I get IPropertyStorage
shell32.IPropertyStorage.WritePropertyNames(cpropid, rgpropid, rglpwstrName)
addTag(r"C:\Users\Agustin\Desktop\New folder")
I tried searching for answers here and in the Microsoft documentation but (Reading and writing Windows "tags" with Python 3) is the closest example I've seen. The example there is using pywin32 but I want to find out if this can be pulled off using only ctypes.
My Question: Where can I find the IPropertySetStorage and IPropertyStorage? And how can I add tag to a folder using only the Windows API with ctypes.
Any help will be appreciated.
When I used the API from Forms Recognizer, it returned a JSON file. Now, I am using Form Recognizer with SDK and Python, and it returns a data type that seems to be specific from the library azure.ai.formrecognizer.
Does anyone know how to save the data acquired from Form Recognizer SDK Python in a JSON file like the one received from Form Recognzier API?
from azure.ai.formrecognizer import FormRecognizerClient
from azure.identity import ClientSecretCredential
client_secret_credential = ClientSecretCredential(tenant_id, client_id, client_secret)
form_recognizer_client = FormRecognizerClient(endpoint, client_secret_credential)
with open(os.path.join(path, file_name), "rb") as fd:
form = fd.read()
poller = form_recognizer_client.begin_recognize_content(form)
form_pages = poller.result()
Thanks for your question! The Azure Form Recognizer SDK for Python provides helper methods like to_dict and from_dict on the models to facilitate converting the data type in the library to and from a dictionary. You can use the dictionary you get from the to_dict method directly or convert it to JSON.
For your example above, in order to get a JSON output you could do something like:
poller = form_recognizer_client.begin_recognize_content(form)
form_pages = poller.result()
d = [page.to_dict() for page in form_pages]
json_string = json.dumps(d)
I hope that answers your question, please let me know if you need more information related to the library.
Also, there's more information about our models and their methods on our documentation page here. You can use the dropdown to select the version of the library that you're using.
From my reading of the LightGBM document, one is supposed to define categorical features in the Dataset method. So I have the following code:
cats=['C1', 'C2']
d_train = lgb.Dataset(X, label=y, categorical_feature=cats)
However, I received the following error message:
/app/anaconda3/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:1243: UserWarning: Using categorical_feature in Dataset.
warnings.warn('Using categorical_feature in Dataset.')
Why did I get the warning message?
I presume that you get this warning in a call to lgb.train. This function also has argument categorical_feature, and its default value is 'auto', which means taking categorical columns from pandas.DataFrame (documentation). The warning, which is emitted at this line, indicates that, despite lgb.train has requested that categorical features be identified automatically, LightGBM will use the features specified in the dataset instead.
To avoid the warning, you can give the same argument categorical_feature to both lgb.Dataset and lgb.train. Alternatively, you can construct the dataset with categorical_feature=None and only specify the categorical features in lgb.train.
Like user andrey-popov described you can use the lgb.train's categorical_feature parameter to get rid of this warning.
Below is a simple example with some code how you could do it:
# Define categorical features
cat_feats = ['item_id', 'dept_id', 'store_id',
'cat_id', 'state_id', 'event_name_1',
'event_type_1', 'event_name_2', 'event_type_2']
...
# Define the datasets with the categorical_feature parameter
train_data = lgb.Dataset(X.loc[train_idx],
Y.loc[train_idx],
categorical_feature=cat_feats,
free_raw_data=False)
valid_data = lgb.Dataset(X.loc[valid_idx],
Y.loc[valid_idx],
categorical_feature=cat_feats,
free_raw_data=False)
# And train using the categorical_feature parameter
lgb.train(lgb_params,
train_data,
valid_sets=[valid_data],
verbose_eval=20,
categorical_feature=cat_feats,
num_boost_round=1200)
This is less of an answer to the original OP and more of an answer to people who are using sklearn API and encounter this issue.
For those of you who are using sklearn API, especially using one of the cross_val methods from sklearn, there are two solutions you could consider using.
Sklearn API solution
A solution that worked for me was to cast categorical fields into the category datatype in pandas.
If you are using pandas df, LightGBM should automatically treat those as categorical. From the documentation:
integer codes will be extracted from pandas categoricals in the
Python-package
It would make sense for this to be the equivalent in the sklearn API to setting categoricals in the Dataset object.
But keep in mind that LightGBM does not officially support virtually any of the non-core parameters for sklearn API, and they say so explicitly:
**kwargs is not supported in sklearn, it may cause unexpected issues.
Adaptive Solution
The other, more sure-fire solution to being able to use methods like cross_val_predict and such is to just create your own wrapper class that implements the core Dataset/Train under the hood but exposes a fit/predict interface for the cv methods to latch onto. That way you get the full functionality of lightGBM with only a little bit of rolling your own code.
The below sketches out what this could look like.
class LGBMSKLWrapper:
def __init__(self, categorical_variables, params):
self.categorical_variables = categorical_variables
self.params = params
self.model = None
def fit(self, X, y):
my_dataset = ltb.Dataset(X, y, categorical_feature=self.categorical_variables)
self.model = ltb.train(params=self.params, train_set=my_dataset)
def predict(self, X):
return self.model.predict(X)
The above lets you load up your parameters when you create the object, and then passes that onto train when the client calls fit.
What is the best way to use Dash with Websockets to build a real-time dashboard ? I would like to update a graph everytime a message is received but the only thing I've found is calling the callback every x seconds like the example below.
import dash
import dash_core_components as dcc
import dash_html_components as html
import dash_daq as daq
from dash.dependencies import Input, Output
import plotly
import plotly.graph_objs as go
from websocket import create_connection
from tinydb import TinyDB, Query
import json
import ssl
# Setting up the websocket and the necessary web handles
ws = create_connection(address, sslopt={"cert_reqs": ssl.CERT_NONE})
app = dash.Dash(__name__)
app.layout = html.Div(
[
dcc.Graph(id='live-graph', animate=True),
dcc.Interval(
id='graph-update',
interval=1*1000,
n_intervals=0)
]
)
#app.callback(Output('live-graph', 'figure'),
[Input('graph-update', 'n_intervals')])
def update_graph_live(n):
message = ws.recv()
x=message.get('data1')
y=message.get('data2')
.....
fig = go.Figure(
data = [go.Bar(x=x,y=y)],
layout=go.Layout(
title=go.layout.Title(text="Bar Chart")
)
)
)
return fig
if __name__ == '__main__':
app.run_server(debug=True)
Is there a way to trigger the callback everytime a message is received (maybe storing them in a database before) ?
This forum post describes a method to use websocket callbacks with Dash:
https://community.plot.ly/t/triggering-callback-from-within-python/23321/6
Update
Tried it, it works well. Environment is Windows 10 x64 + Python 3.7.
To test, download the .tar.gz file and run python usage.py. It will complain about some missing packages, install these. Might have to edit the address from 0.0.0.0 to 127.0.0.1 in usage.py. Browse to http://127.0.0.1:5000 to see the results. If I had more time, I'd put this example up on GitHub (ping me if you're having trouble getting it to work, or the original gets lost).
I had two separate servers: one for dash, the other one as a socket server. They are running on different ports. On receiving a message, I edited a common json file to share information to dash's callback. That's how I did it.
Is there a way to save and recover a trained Neural Network in PyBrain, so that I don't have to retrain it each time I run the script?
PyBrain's Neural Networks can be saved and loaded using either python's built in pickle/cPickle module, or by using PyBrain's XML NetworkWriter.
# Using pickle
from pybrain.tools.shortcuts import buildNetwork
import pickle
net = buildNetwork(2,4,1)
fileObject = open('filename', 'w')
pickle.dump(net, fileObject)
fileObject.close()
fileObject = open('filename','r')
net = pickle.load(fileObject)
Note cPickle is implemented in C, and therefore should be much faster than pickle. Usage should mostly be the same as pickle, so just import and use cPickle instead.
# Using NetworkWriter
from pybrain.tools.shortcuts import buildNetwork
from pybrain.tools.customxml.networkwriter import NetworkWriter
from pybrain.tools.customxml.networkreader import NetworkReader
net = buildNetwork(2,4,1)
NetworkWriter.writeToFile(net, 'filename.xml')
net = NetworkReader.readFrom('filename.xml')
The NetworkWriter and NetworkReader work great. I noticed that upon saving and loading via pickle, that the network is no longer changeable via training-functions. Thus, I would recommend using the NetworkWriter-method.
NetworkWriter is the way to go. Using Pickle you can't retrain network as Jorg tells.
You need something like this:
from pybrain.tools.shortcuts import buildNetwork
from pybrain.tools.customxml import NetworkWriter
from pybrain.tools.customxml import NetworkReader
net = buildNetwork(4,6,1)
NetworkWriter.writeToFile(net, 'filename.xml')
net = NetworkReader.readFrom('filename.xml')