Is there a way to save and recover a trained Neural Network in PyBrain, so that I don't have to retrain it each time I run the script?
PyBrain's Neural Networks can be saved and loaded using either python's built in pickle/cPickle module, or by using PyBrain's XML NetworkWriter.
# Using pickle
from pybrain.tools.shortcuts import buildNetwork
import pickle
net = buildNetwork(2,4,1)
fileObject = open('filename', 'w')
pickle.dump(net, fileObject)
fileObject.close()
fileObject = open('filename','r')
net = pickle.load(fileObject)
Note cPickle is implemented in C, and therefore should be much faster than pickle. Usage should mostly be the same as pickle, so just import and use cPickle instead.
# Using NetworkWriter
from pybrain.tools.shortcuts import buildNetwork
from pybrain.tools.customxml.networkwriter import NetworkWriter
from pybrain.tools.customxml.networkreader import NetworkReader
net = buildNetwork(2,4,1)
NetworkWriter.writeToFile(net, 'filename.xml')
net = NetworkReader.readFrom('filename.xml')
The NetworkWriter and NetworkReader work great. I noticed that upon saving and loading via pickle, that the network is no longer changeable via training-functions. Thus, I would recommend using the NetworkWriter-method.
NetworkWriter is the way to go. Using Pickle you can't retrain network as Jorg tells.
You need something like this:
from pybrain.tools.shortcuts import buildNetwork
from pybrain.tools.customxml import NetworkWriter
from pybrain.tools.customxml import NetworkReader
net = buildNetwork(4,6,1)
NetworkWriter.writeToFile(net, 'filename.xml')
net = NetworkReader.readFrom('filename.xml')
Related
I am trying to find the best Alpha for a Ridge model without CV, using Yellowbrick ManualAlphaSelection API. My code is pretty basic and it has been taken from the yellowbrick´s documentation. Even though it does not work:
from yellowbrick.regressor import ManualAlphaSelection
from sklearn.linear_model import Ridge
model = ManualAlphaSelection(Ridge(), scoring='neg_mean_squared_error')
model.fit(X_train, y_train)
model.show()
Python raises the message: 'Ridge' is not a CV regularization model; try ManualAlphaSelection instead.
But this message is wrong because the ManualAlphaSelection is already being used.
This actually appears to be a bug in our library 😅
Would you mind opening up a bug report on GitHub so we can be sure to fix it? Thank you for checking out Yellowbrick!
What is the best way to use Dash with Websockets to build a real-time dashboard ? I would like to update a graph everytime a message is received but the only thing I've found is calling the callback every x seconds like the example below.
import dash
import dash_core_components as dcc
import dash_html_components as html
import dash_daq as daq
from dash.dependencies import Input, Output
import plotly
import plotly.graph_objs as go
from websocket import create_connection
from tinydb import TinyDB, Query
import json
import ssl
# Setting up the websocket and the necessary web handles
ws = create_connection(address, sslopt={"cert_reqs": ssl.CERT_NONE})
app = dash.Dash(__name__)
app.layout = html.Div(
[
dcc.Graph(id='live-graph', animate=True),
dcc.Interval(
id='graph-update',
interval=1*1000,
n_intervals=0)
]
)
#app.callback(Output('live-graph', 'figure'),
[Input('graph-update', 'n_intervals')])
def update_graph_live(n):
message = ws.recv()
x=message.get('data1')
y=message.get('data2')
.....
fig = go.Figure(
data = [go.Bar(x=x,y=y)],
layout=go.Layout(
title=go.layout.Title(text="Bar Chart")
)
)
)
return fig
if __name__ == '__main__':
app.run_server(debug=True)
Is there a way to trigger the callback everytime a message is received (maybe storing them in a database before) ?
This forum post describes a method to use websocket callbacks with Dash:
https://community.plot.ly/t/triggering-callback-from-within-python/23321/6
Update
Tried it, it works well. Environment is Windows 10 x64 + Python 3.7.
To test, download the .tar.gz file and run python usage.py. It will complain about some missing packages, install these. Might have to edit the address from 0.0.0.0 to 127.0.0.1 in usage.py. Browse to http://127.0.0.1:5000 to see the results. If I had more time, I'd put this example up on GitHub (ping me if you're having trouble getting it to work, or the original gets lost).
I had two separate servers: one for dash, the other one as a socket server. They are running on different ports. On receiving a message, I edited a common json file to share information to dash's callback. That's how I did it.
I was trying to import the data from one hbase(v0.98.4) to another hbase(v0.98.13).
I have exported the data using the below command -
hbase org.apache.hadoop.hbase.mapreduce.Driver export 'tblname' /path/
But I am not able to import it using the below command -
hbase org.apache.hadoop.hbase.mapreduce.Driver import 'tblname' /hdfs/path/
I get the below deprecation messages as well as an Exception thrown -
Is it becoz of version conflicts between source db and destination db?
I happen to solve it. All I had to do was create an empty table with same metadata and then import it. :)
Try using the commands here for Hbase versions above 0.94. May be you are using generalized Map reduce class and giving export and import as arguments, when the actual classes Export and Import are present. Hope it helps. Happy coding
How can I access an unsupported Wand API via the Python Wand interface? For example, I wish to call the Wand API MagickAddNoiseImage but it is not available in the Python interface.
Accessing unsupported APIs is pretty easy with wand.api, but you will need to open-up ImageMagick's docs/header files for reference.
from wand.api import library
import ctypes
# Re-create NoiseType enum
NOISE_TYPES = ('undefined', 'uniform', 'gaussian', 'multiplicative_gaussian',
'impulse', 'laplacian', 'poisson', 'random')
# Map API i/o
library.MagickAddNoiseImage.argtypes = [ctypes.c_void_p,
ctypes.c_uint]
library.MagickAddNoiseImage.restype = ctypes.c_int
# Extend wand's Image class with your new API
from wand.image import Image
class MySupportedImage(Image):
def add_noise(self, noise_type):
"""My MagickAddNoiseImage"""
if noise_type not in NOISE_TYPES:
self.raise_exception()
return library.MagickAddNoiseImage.argtypes(self.resource,
NOISE_TYPES.index(noise_type))
If your solution works out, think about submitting your solution back to the community (after you've created a solid unit test.)
I am attempting to use the following code to save a image from URL using python:
image = urllib.URLopener()
image.retrieve("http://example.com/image.jpg","image.jpg")
The image saves as expected, I was wondering whether it would be possible to set assign a User-agent using the urllib method?
i dont think you can add custom headers while using urllib
but i know there are multiple ways to do it using urllib2
one way you could is like this:
import urllib2
headers = { 'User-Agent' : 'Mozilla/5.0' }
req = urllib2.Request('http://example.com/image.jpg', None, headers)
html = urllib2.urlopen(req).read()
with open('download.jpg','r+') as f:
f.write(html)
this will download the image but the 'download.jpg' has to already exist
there are more ways to do it i would take a look at this Setting the User-Agent
also take a look at this Question
Good Luck!