I'm curious of model parallelism, and I've read the code from Yaroslav Bulatov. And in that example, we should partition our model (or in tensorflow we called Graph) manually to different partition (left_network & right_network).
So, I was wondering if I have to make partitions manually, what's the simple_placer.cc and graph_partition.cc have done to the whole graph? And I'm still not clear as all.
In my thought(let me know if anythong wrong):
If the graph has 8 partitions(subgraph) which can be seen as 8 jobs, and 4 workers,How the partitions distributed to workers can be done through:
explicit annotations via tf.device(), or
distributed training, tf.train.replica_device_setter()
share the variables across parameter servers, and otherwise put all
ops on the worker device
But how's the graph make partitions?
I want to trace what's the subgraph (op-nodes set) looks like?
Can I dump the result or I need to trace/modified which code file?
Please let me know if any concepts is wrong or vague.
I'm a rookie of these, any opinion is appreciated.
In the code below, is matmul a op-node, would it be partition into
different jobs?
y_ = tf.placeholder(tf.float32, [None, 10])
x = tf.placeholder(tf.float32, [None, 784])
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))
y = tf.matmul(x, W) + b
You can get the result of the placement algorithm by passing additional options when you call tf.Session.run()
# ...
y = tf.matmul(x, W) + b
sess = tf.Session()
options = tf.RunOptions(output_partition_graphs=True)
metadata = tf.RunMetadata()
sess.run(y, options=options, run_metadata=metadata)
# `metadata` now contains information about what happened during the `run()` call.
for partition in metadata.partition_graphs:
# `partition` is a `tf.GraphDef` representing all the nodes that ran on a single
# device. All nodes in `partition` have the same `device` value.
device = partition.node[0].device
for node in partition.node:
# e.g. print each node or store it in a dictionary for further analysis.
# ...
Related
I am new to ML and TensorFlow. I am trying to build a CNN to categorize a good image against corrupted images, similar to rock paper scissor tutorials in tensor flow, except for only two categories.
The Colab Notebook
Model Architecture
train_generator = training_datagen.flow_from_directory(
TRAINING_DIR,
target_size=(150,150),
class_mode='categorical'
)
validation_generator = validation_datagen.flow_from_directory(
VALIDATION_DIR,
target_size=(150,150),
class_mode='categorical'
)
model = tf.keras.models.Sequential([
# Note the input shape is the desired size of the image 150x150 with 3 bytes color
# This is the first convolution
tf.keras.layers.Conv2D(64, (3,3), activation='relu', input_shape=(150, 150, 3)),
tf.keras.layers.MaxPooling2D(2, 2),
# The second convolution
tf.keras.layers.Conv2D(64, (3,3), activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
# The third convolution
tf.keras.layers.Conv2D(128, (3,3), activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
# The fourth convolution
tf.keras.layers.Conv2D(128, (3,3), activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
# Flatten the results to feed into a DNN
tf.keras.layers.Flatten(),
tf.keras.layers.Dropout(0.5),
# 512 neuron hidden layer
tf.keras.layers.Dense(512, activation='relu'),
tf.keras.layers.Dense(2, activation='softmax')
])
model.summary()
model.compile(loss = 'categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
history = model.fit_generator(train_generator, epochs=25, validation_data = validation_generator, verbose = 1)
model.save("rps.h5")
Only Change I made was turning input shape to (150,150,1) to (150,150,3) and changed last layers output to 2 neurons from 3. The training gave me consistently accuracy of 90 above for data set of 600 images in each class. But when I am making a prediction using code in the tutorial, it gives me highly wrong values even for data in the data set.
PREDICTION
Original code in TensorFlow tutorial
for file in onlyfiles:
path = fn
img = image.load_img(path, target_size=(150, 150,3)) # changed target_size to (150, 150,3)) from (150,150 )
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
images = np.vstack([x])
classes = model.predict(images, batch_size=10)
print(fn)
print(classes)
I changed target_size to (150, 150,3)) from (150,150) in my belief that since my input is a 3 channel image,
Result
It gives very wrong values [0,1][0,1] for even images in which are in dataset
But when I changed the code to this
for file in onlyfiles:
path = fn
img = image.load_img(path, target_size=(150, 150,3))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x /= 255.
classes = model.predict(images, batch_size=10)
print(fn)
print(classes)
In this case values come like
[[9.9999774e-01 2.2242968e-06]]
[[9.9999785e-01 2.1864464e-06]]
[[9.9999785e-01 2.1641024e-06]]
one or two errors are there but it is very much correct
So my question even though the last activation is softmax, why it is now coming in decimal values, is there any logical mistake in the way I am making predictions.? I tried binary also, but couldn't find much difference.
Please note -
When you are changing output classes from 2 to 3, you are asking the model to categorise into 3 classes. This would contradict your problem statement which separates good and corrupted ones i.e 2 output classes (a binary problem). I think it can be reversed from 3 to 2 if I have understood the question correctly.
Second the output you are getting is perfectly correct, the neural network models outputs probabilities instead of absolute class values like 0 or 1. By probability, it tells how likely it belongs to say class 0 or class 1.
Also , as mentioned above by #BBloggsbott - you just have to use np.argmax on the output array which will tell you the probability of belonging to class 1 (Positive class) by default.
Hope this helps.
Thanks.
Softmax returns probability distributions for the vector it gets as input. So, the fact that you are getting decimal values is not a problem. If you want to find the exact class each image belongs to, try using the argmax function on the predictions.
I am wondering if there are any reasonable ways to generate clients data sets for federated learning simulation using tff core code? In the tutorial for the federated core, it uses the MNIST database with each client has only one distinct label in his data set. In this case, there are only 10 different labels available. If I want to have more clients, how can I do that? Thanks in advance.
If you want to create a dataset from scratch you can use tff.simulation.FromTensorSlicesClientData to covert tensors to tff clientdata object. Just you need to pass dictionary having client id as key and dataset as value.
client_train_dataset = collections.OrderedDict()
for i in range(1, split+1):
client_name = "client_" + str(i)
start = image_per_set * (i-1)
end = image_per_set * i
print(f"Adding data from {start} to {end} for client : {client_name}")
data = collections.OrderedDict((('label', y_train[start:end]), ('pixels', x_train[start:end])))
client_train_dataset[client_name] = data
train_dataset = tff.simulation.FromTensorSlicesClientData(client_train_dataset)
you can check my complete implementation here, where i have splitted mnist into 4 clients.
There are preprocessed simulation datasets in TFF that should serve this purpose quite nicely. Take, for example, loading EMNIST where the images are partitioned by writer, corresponding to user, as opposed to label. This can be loaded into a Python runtime rather simply (here creating train data with 100 clients):
source, _ = tff.simulation.datasets.emnist.load_data()
def map_fn(example):
return {'x': tf.reshape(example['pixels'], [-1]), 'y': example['label']}
def client_data(n):
ds = source.create_tf_dataset_for_client(source.client_ids[n])
return ds.repeat(10).map(map_fn).shuffle(500).batch(20)
train_data = [client_data(n) for n in range(100)]
There are existing datasets partitioned in a similar way for extended MNIST (IE, includes handwritten characters in addition to digits), Shakespeare plays (partitioned by character), and Stackoverflow posts (partitioned by user). Documentation on these datasets can be found here.
If you wish to create your own custom dataset, please see the answer here.
I have the following function:
import random
lst = []
for i in range(100):
lst.append(random.randint(1, 10))
print(lst)
buffer = []
# This is the peace of code which I am interested to convert into tensorflow.
for a in lst:
buffer.append(a)
if len(buffer) > 5:
buffer.pop(0)
if len(buffer) == 5:
print(buffer)
So, from the code, I need to create a buffer (that could be a variable in tensorflow). This buffer should hold the extracted features from the last conv layer. The variable will be an input to an RNN in my case.
The advantage of this approach is that when we have large images, and when we need to feed a RNN with a (batch of images) * (sequence length) * (size of 1 image), that will require a very big batch of images to be loaded into the main memory. On the other hand, according to the code above, we will be feeding 1 image at a time using the Datasets from tensorflow, or an input queue or any other alternative. As a result, we will be storing in memory the feature of size: batch_size * sequence_length * feature space .In addition, we can say:
if len(buffer) == n:
# empty out the buffer after using its elements
buffer = [] # Or any other alternative way
I am aware that I can feed my network batches of images, but I need to accomplish the mentioned code based on some literature.
Any help is much appreciated!!
I try to regenerate your output using tf.FIFOQueue (https://www.tensorflow.org/api_docs/python/tf/FIFOQueue). I have given my code below with the comments where necessary.
BATCH_SIZE = 20
lst = []
for i in range(BATCH_SIZE):
lst.append(random.randint(1, 10))
print(lst)
curr_data = np.reshape(lst, (BATCH_SIZE, 1)) # reshape the tensor so that [BATCH_SIZE 1]
# queue starts here
queue_input_data = tf.placeholder(tf.int32, shape=[1]) # Placeholder for feed the data
queue = tf.FIFOQueue(capacity=50, dtypes=[tf.int32], shapes=[1]) # Queue define here
enqueue_op = queue.enqueue([queue_input_data]) # enqueue operation
len_op = queue.size() # chek the queue size
#check the length of the queue and dequeue one if greater than 5
dequeue_one = tf.cond(tf.greater(len_op, 5), lambda: queue.dequeue(), lambda: 0)
#check the length of the queue and dequeue five elemts if equals to 5
dequeue_many = tf.cond(tf.equal(len_op, 5), lambda:queue.dequeue_many(5), lambda: 0)
with tf.Session() as session:
for i in range(BATCH_SIZE):
_ = session.run(enqueue_op, feed_dict={queue_input_data: curr_data[i]}) # enqueue one element each ietaration
len = session.run(len_op) # check the legth of the queue
print(len)
element = session.run(dequeue_one) # dequeue the first element
print(element)
However, following two problems are associated with the above code,
Only the dequeue one and dequeue many operations are available and you cannot see the elements inside the queue (I don't think you will need this since you are looking something like a pipeline).
I think that tf.cond is the only way to implement a conditional operation (I couldn't find any other suitable function similar to that). However, since its similar to the if-then-else statement, its mandatory to define an operation when the statement is false also (not just having only if statement without the else). Since Tensorflow is all about building a graph I think its necessary to include the two branches (when the condition is true and false).
Moreover, a good explanation for Tensorflow input pipelines can be found here (http://ischlag.github.io/2016/11/07/tensorflow-input-pipeline-for-large-datasets/).
Hope this helps.
vcftools --vcf ALL.chr1.phase3_shapeit2_mvncall_integrated_v5.20130502.genotypes.vcf --weir-fst-pop POP1.txt --weir-fst-pop POP2.txt --out fst.POP1.POP2
The above script computes Fst distances on 1000 Genomes population data using Weir and Cokerham's 1984 formula. This formula uses 3 variance components, namely a,b,c (between populations; between individuals within populations; between gametes within individuals within populations).
The output directly provides the result of the formula but not the components that the program calculated to arrive at the final result. How can I ask Vcftools to output the values for a,b,c?
If you can get the data into the format for hierfstat, you can get the variance components from varcomp.glob. What I normally do is:
use vcftools with --012 to get genotypes
convert 0/1/2/-1 to hierfstat format (eg., 11/12/22/NA)
load the data into hierfstat and compute (see below)
R example:
library(hierfstat)
data = read.table("hierfstat.txt", header=T, sep="\t")
levels = data.frame(data$popid)
loci = data[,2:ncol(data)]
res = varcomp.glob(levels=levels, loci=loci, diploid=T)
print(res$loc)
print(res$F)
Fst for each locus (row) therefore is (without hierarchical design), from res$loc: res$loc[1]/sum(res$loc). If you have more complicated sampling, you'll need to interpret the variance components differently.
--update per your comment--
I do this in Pandas, but any language would do. It's a text replacement exercise. Just get your .012 file into a dataframe and convert as below. I read in row by row into numpy b/c I have tons of snps, but read_csv would work, too.
import pandas as pd
import numpy as np
z12_data = []
for i, line in enumerate(open(z12_file)):
line = line.strip()
line = [int(x) for x in line.split("\t")]
z12_data.append(np.array(line))
if i % 10 == 0:
print i
z12_data = np.array(z12_data)
z12_df = pd.DataFrame(z12_data)
z12_df = z12_df.drop(0, axis=1)
z12_df.columns = pd.Series(z12_df.columns)-1
hierf_trans = {0:11, 1:12, 2:22, -1:'NA'}
def apply_hierf_trans(series):
return [hierf_trans[x] if x in hierf_trans else x for x in series]
hierf = df.apply(apply_hierf_trans)
hierf.to_csv("hierfstat.txt", header=True, index=False, sep="\t")
Then, you'd read that file hierfstat.txt into R, these are your loci. You'd need to specify your levels in your sampling design (e.g., your population). Then call varcomp.glob() to get the variance components. I have a parallel version of this here if you want to use it.
Note that you are specifying 0 as the reference allele, in this case. May be what you want, maybe not. I often calculate minor allele frequency and make 2 the minor allele, but it depends on your study goal.
Is it possible to alter the following code to put Child_4 at the same horizontal level as Grandchild_0 (thereby pushing Grandchild_4 to its own level)?
import networkx as nx
import matplotlib.pyplot as plt
G = nx.DiGraph()
G.add_node("ROOT")
for i in xrange(5):
G.add_node("Child_%i" % i)
G.add_node("Grandchild_%i" % i)
G.add_edge("ROOT", "Child_%i" % i)
G.add_edge("Child_%i" % i, "Grandchild_%i" % i)
pos=nx.graphviz_layout(G,prog='dot')
nx.draw(G,pos,arrows=False)
plt.show()
The above code produces the following layout, which I'd like to alter by shifting a child down one level to be horizontally aligned with the grandchildren:
Within the Python network library networkx, I'm using graphviz's dot engine to render a tree (following this recommendation). I would like to control the y-position of the nodes by specifying which nodes should have the same height. The nodes might be at different depths in the tree.
I know I could control the node height if I wrote my own graphviz code through using the rank=same command (e.g., {rank=same; n4 -> p2;} [ex.]). However, I am relying on networkx.graphviz_layout() [doc | source] to generate the node positions, and graphviz_layout can send only command line arguments to pygraphviz. My attempts to use variants of nx.graphviz_layout(G, prog='dot', args="-Grank=same; n4 -> p2;") have failed. Is it possible to describe the desired node heights within the NetworkX wrapper for pygraphviz, or do I need to write my own wrapper around pygraphviz? Edit: The answer provides a new wrapper around pygraphviz. It would significantly simplify things to send the rank information within the existing NetworkX wrapper for pygraphviz. I'll change my accepted answer if someone can tell me how that might be possible.
I can't find a way to achieve this through the original networkx wrapper.
Instead, I've written a new wrapper for pygraphviz, with most lines copied from the source code. It adds a parameter sameRank = [] for a list of nodes-of-the-same-rank lists and a for loop around an invocation of pygraphviz.add_subgraph(listOfNodes,rank="same").
def graphviz_layout_with_rank(G, prog = "neato", root = None, sameRank = [], args = ""):
## See original import of pygraphviz in try-except block
## See original identification of root through command line
A = nx.to_agraph(G)
for sameNodeHeight in sameRank:
if type(sameNodeHeight) == str:
print("node \"%s\" has no peers in its rank group" %sameNodeHeight)
A.add_subgraph(sameNodeHeight, rank="same")
A.layout(prog=prog, args=args)
## See original saving of each node location to node_pos
return node_pos
In the question example, Child_4 can be pushed to the same horizontal level as Grandchild_0 through the line:
pos=graphviz_layout_with_rank(G, prog='dot',sameRank=[["Child_4","Grandchild_0"]])