Is it possible to inject a signal with waveform A and then recover it with a different waveform?
*Question asked by Nirban Bose on the bilby slack channel.
Yes, a good example of this is the Reduced Order Quadrature example. The key is to generate the data with a different waveform_generator to the one used in the search. Practically, this amounts to passing a different waveform_generator to the interferometer data generated and to the likelihood.
Here is a pseudo code demonstration which injects a binary black hole and recovers a binary neutron star waveform
# Make some injection parameters
injection_parameters = dict(
chirp_mass=36., mass_ratio=0.9, a_1=0.4, a_2=0.3, tilt_1=0.0, tilt_2=0.0,
phi_12=1.7, phi_jl=0.3, luminosity_distance=1000., iota=0.4, psi=0.659,
phase=1.3, geocent_time=1126259642.413, ra=1.375, dec=-1.2108)
# Make the injection waveform generator
waveform_arguments = dict(waveform_approximant='IMRPhenomPv2',
reference_frequency=20., minimum_frequency=20.)
injection_waveform_generator = bilby.gw.WaveformGenerator(
duration=duration, sampling_frequency=sampling_frequency,
frequency_domain_source_model=bilby.gw.source.lal_binary_black_hole,
waveform_arguments=waveform_arguments)
# Setup the interferometer list
ifos = bilby.gw.detector.InterferometerList(['H1', 'L1', 'V1'])
ifos.set_strain_data_from_power_spectral_densities(
sampling_frequency=sampling_frequency, duration=duration,
start_time=injection_parameters['geocent_time'] - 3)
# Here we inject a signal using the injection_waveform_generator
ifos.inject_signal(waveform_generator=injection_waveform_generator,
parameters=injection_parameters)
# Now create a search waveform generator (i.e. the one used for PE)
# Here you can specify anything you like, e.g. a different waveform approximant,
# in this example, I've just used a binary neutron star instead
waveform_arguments = dict(waveform_approximant='IMRPhenomPv2',
reference_frequency=20., minimum_frequency=20.)
search_waveform_generator = bilby.gw.WaveformGenerator(
duration=duration, sampling_frequency=sampling_frequency,
frequency_domain_source_model=bilby.gw.source.lal_binary_neutron_star,
waveform_arguments=waveform_arguments)
# Set up your priors
priors = ...
# Now set up the likelihood - here we pass in the search waveform generator
likelihood = bilby.gw.likelihood.GravitationalWaveTransient(
interferometers=ifos, waveform_generator=search_waveform_generator,
prior=priors)
# Run the sampler
result = bilby.run_sampler(
likelihood=likelihood, priors=priors, ...)
This should be adaptable to a good number of situations.
I have the following function:
import random
lst = []
for i in range(100):
lst.append(random.randint(1, 10))
print(lst)
buffer = []
# This is the peace of code which I am interested to convert into tensorflow.
for a in lst:
buffer.append(a)
if len(buffer) > 5:
buffer.pop(0)
if len(buffer) == 5:
print(buffer)
So, from the code, I need to create a buffer (that could be a variable in tensorflow). This buffer should hold the extracted features from the last conv layer. The variable will be an input to an RNN in my case.
The advantage of this approach is that when we have large images, and when we need to feed a RNN with a (batch of images) * (sequence length) * (size of 1 image), that will require a very big batch of images to be loaded into the main memory. On the other hand, according to the code above, we will be feeding 1 image at a time using the Datasets from tensorflow, or an input queue or any other alternative. As a result, we will be storing in memory the feature of size: batch_size * sequence_length * feature space .In addition, we can say:
if len(buffer) == n:
# empty out the buffer after using its elements
buffer = [] # Or any other alternative way
I am aware that I can feed my network batches of images, but I need to accomplish the mentioned code based on some literature.
Any help is much appreciated!!
I try to regenerate your output using tf.FIFOQueue (https://www.tensorflow.org/api_docs/python/tf/FIFOQueue). I have given my code below with the comments where necessary.
BATCH_SIZE = 20
lst = []
for i in range(BATCH_SIZE):
lst.append(random.randint(1, 10))
print(lst)
curr_data = np.reshape(lst, (BATCH_SIZE, 1)) # reshape the tensor so that [BATCH_SIZE 1]
# queue starts here
queue_input_data = tf.placeholder(tf.int32, shape=[1]) # Placeholder for feed the data
queue = tf.FIFOQueue(capacity=50, dtypes=[tf.int32], shapes=[1]) # Queue define here
enqueue_op = queue.enqueue([queue_input_data]) # enqueue operation
len_op = queue.size() # chek the queue size
#check the length of the queue and dequeue one if greater than 5
dequeue_one = tf.cond(tf.greater(len_op, 5), lambda: queue.dequeue(), lambda: 0)
#check the length of the queue and dequeue five elemts if equals to 5
dequeue_many = tf.cond(tf.equal(len_op, 5), lambda:queue.dequeue_many(5), lambda: 0)
with tf.Session() as session:
for i in range(BATCH_SIZE):
_ = session.run(enqueue_op, feed_dict={queue_input_data: curr_data[i]}) # enqueue one element each ietaration
len = session.run(len_op) # check the legth of the queue
print(len)
element = session.run(dequeue_one) # dequeue the first element
print(element)
However, following two problems are associated with the above code,
Only the dequeue one and dequeue many operations are available and you cannot see the elements inside the queue (I don't think you will need this since you are looking something like a pipeline).
I think that tf.cond is the only way to implement a conditional operation (I couldn't find any other suitable function similar to that). However, since its similar to the if-then-else statement, its mandatory to define an operation when the statement is false also (not just having only if statement without the else). Since Tensorflow is all about building a graph I think its necessary to include the two branches (when the condition is true and false).
Moreover, a good explanation for Tensorflow input pipelines can be found here (http://ischlag.github.io/2016/11/07/tensorflow-input-pipeline-for-large-datasets/).
Hope this helps.
I'm curious of model parallelism, and I've read the code from Yaroslav Bulatov. And in that example, we should partition our model (or in tensorflow we called Graph) manually to different partition (left_network & right_network).
So, I was wondering if I have to make partitions manually, what's the simple_placer.cc and graph_partition.cc have done to the whole graph? And I'm still not clear as all.
In my thought(let me know if anythong wrong):
If the graph has 8 partitions(subgraph) which can be seen as 8 jobs, and 4 workers,How the partitions distributed to workers can be done through:
explicit annotations via tf.device(), or
distributed training, tf.train.replica_device_setter()
share the variables across parameter servers, and otherwise put all
ops on the worker device
But how's the graph make partitions?
I want to trace what's the subgraph (op-nodes set) looks like?
Can I dump the result or I need to trace/modified which code file?
Please let me know if any concepts is wrong or vague.
I'm a rookie of these, any opinion is appreciated.
In the code below, is matmul a op-node, would it be partition into
different jobs?
y_ = tf.placeholder(tf.float32, [None, 10])
x = tf.placeholder(tf.float32, [None, 784])
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))
y = tf.matmul(x, W) + b
You can get the result of the placement algorithm by passing additional options when you call tf.Session.run()
# ...
y = tf.matmul(x, W) + b
sess = tf.Session()
options = tf.RunOptions(output_partition_graphs=True)
metadata = tf.RunMetadata()
sess.run(y, options=options, run_metadata=metadata)
# `metadata` now contains information about what happened during the `run()` call.
for partition in metadata.partition_graphs:
# `partition` is a `tf.GraphDef` representing all the nodes that ran on a single
# device. All nodes in `partition` have the same `device` value.
device = partition.node[0].device
for node in partition.node:
# e.g. print each node or store it in a dictionary for further analysis.
# ...
This is an algorithmic question about a somewhat complex problem. The foundation is this:
A scheduling system based on available slots and reserved slots. Slots have certain criteria, let's call them tags. A reservation is matched to an available slot by those tags, if the available slot's tag set is a superset of the reserved slot.
As a concrete example, take this scenario:
11:00 12:00 13:00
+--------+
| A, B |
+--------+
+--------+
| C, D |
+--------+
Between the times of 11:00 to 12:30 reservations for the tags A and B can be made, from 12:00 to 13:30 C and D is available, and there's an overlap from about 12:00 to 12:30.
11:00 12:00 13:00
+--------+
| A, B |
+--------+
+--------+
| C, D |
+--------+
xxxxxx
x A x
xxxxxx
Here a reservation for A has been made, so no other reservations for A or B can be made between 11:15-ish and 12:00-ish.
That's the idea in a nutshell. There are no specific limitations for the available slots:
an available slot can contain any number of tags
any number of slots can overlap at any time
slots are of arbitrary length
reservations can contain any number of tags
The only rule that needs to be obeyed in the system is:
when adding a reservation, at least one remaining available slot must match all the tags in the reservation
To clarify: when there are two available slots at the same time with, say, tag A, then two reservations for A can be made at that time, but no more.
I have that working with a modified implementation of an interval tree; as a quick overview:
all available slots are added to the interval tree (duplicates/overlaps are preserved)
all reserved slots are iterated and:
all available slots matching the time of the reservation are queried from the tree
the first of those matching the reservation's tags is sliced and the slice removed from the tree
When that process is finished, what's left are the remaining slices of available slots, and I can query whether a new reservation can be made for a particular time and add it.
Data structures look something like this:
{
type: 'available',
begin: 1497857244,
end: 1497858244,
tags: [{ foo: 'bar' }, { baz: 42 }]
}
{
type: 'reserved',
begin: 1497857345,
end: 1497857210,
tags: [{ foo: 'bar' }]
}
Tags are themselves key-value objects, a list of them is a "tag set". Those could be serialised if it helps; so far I'm using a Python set type which makes comparison easy enough. Slot begin/end times are UNIX time stamps within the tree. I'm not particularly married to these specific data structures and can refactor them if it's useful.
The problem I'm facing is that this doesn't work bug-free; every once in a while a reservation sneaks its way into the system that conflicts with other reservations, and I couldn't yet figure out how that can happen exactly. It's also not very clever when tags overlap in a complex way where the optimal distribution needs to be calculated so all reservations can be fit into the available slots as best as possible; in fact currently it's non-deterministic how reservations are matched to available slots in overlapping scenarios.
What I want to know is: interval trees are mostly great for this purpose, but my current system to add tag set matching as an additional dimension to this is clunky and bolted-on; is there a data structure or algorithm that can handle this in an elegant way?
Actions that must be supported:
Querying the system for available slots that match certain tag sets (taking into account reservations that may reduce availability but are not themselves part of said tag set; e.g. in the example above querying for an availability for B).
Ensuring no reservations can be added to the system which don't have a matching available slot.
Your problem can be solved using constraint programming. In python this can be implemented using the python-constraint library.
First, we need a way to check if two slots are consistent with each other. this is a function that returns true if two slots share a tag and their rimeframes overlap. In python this can be implemented using the following function
def checkNoOverlap(slot1, slot2):
shareTags = False
for tag in slot1['tags']:
if tag in slot2['tags']:
shareTags = True
break
if not shareTags: return True
return not (slot2['begin'] <= slot1['begin'] <= slot2['end'] or
slot2['begin'] <= slot1['end'] <= slot2['end'])
I was not sure whether you wanted the tags to be completely the same (like {foo: bar} equals {foo: bar}) or only the keys (like {foo: bar} equals {foo: qux}), but you can change that in the function above.
Consistency check
We can use the python-constraint module for the two kinds of functionality you requested.
The second functionality is the easiest. To implement this, we can use the function isConsistent(set) which takes a list of slots in the provided data structure as input. The function will then feed all the slots to python-constraint and will check if the list of slots is consistent (no 2 slots that shouldn't overlap, overlap) and return the consistency.
def isConsistent(set):
#initialize python-constraint context
problem = Problem()
#add all slots the context as variables with a singleton domain
for i in range(len(set)):
problem.addVariable(i, [set[i]])
#add a constraint for each possible pair of slots
for i in range(len(set)):
for j in range(len(set)):
#we don't want slots to be checked against themselves
if i == j:
continue
#this constraint uses the checkNoOverlap function
problem.addConstraint(lambda a,b: checkNoOverlap(a, b), (i, j))
# getSolutions returns all the possible combinations of domain elements
# because all domains are singleton, this either returns a list with length 1 (consistent) or 0 (inconsistent)
return not len(problem.getSolutions()) == 0
This function can be called whenever a user wants to add a reservation slot. The input slot can be added to the list of already existing slots and the consistency can be checked. If it is consistent, the new slot an be reserverd. Else, the new slot overlaps and should be rejected.
Finding available slots
This problem is a bit trickier. We can use the same functionality as above with a few significant changes. Instead of adding the new slot together with the existing slot, we now want to add all possible slots to the already existing slots. We can then check the consistency of all those possible slots with the reserved slots and ask the constraint system for the combinations that are consistent.
Because the number of possible slots would be infinite if we didn't put any restrictions on it, we first need to declare some parameters for the program:
MIN = 149780000 #available time slots can never start earlier then this time
MAX = 149790000 #available time slots can never start later then this time
GRANULARITY = 1*60 #possible time slots are always at least one minut different from each other
We can now continue to the main function. It looks a lot like the consistency check, but instead of the new slot from the user, we now add a variable to discover all available slots.
def availableSlots(tags, set):
#same as above
problem = Problem()
for i in range(len(set)):
problem.addVariable(i, [set[i]])
#add an extra variable for the available slot is added, with a domain of all possible slots
problem.addVariable(len(set), generatePossibleSlots(MIN, MAX, GRANULARITY, tags))
for i in range(len(set) +1):
for j in range(len(set) +1):
if i == j:
continue
problem.addConstraint(lambda a, b: checkNoOverlap(a, b), (i, j))
#extract the available time slots from the solution for clean output
return filterAvailableSlots(problem.getSolutions())
I use some helper functions to keep the code cleaner. They are included here.
def filterAvailableSlots(possibleCombinations):
result = []
for slots in possibleCombinations:
for key, slot in slots.items():
if slot['type'] == 'available':
result.append(slot)
return result
def generatePossibleSlots(min, max, granularity, tags):
possibilities = []
for i in range(min, max - 1, granularity):
for j in range(i + 1, max, granularity):
possibleSlot = {
'type': 'available',
'begin': i,
'end': j,
'tags': tags
}
possibilities.append(possibleSlot)
return tuple(possibilities)
You can now use the function getAvailableSlots(tags, set) with the tags for which you want the available slots and a set of already reserved slots. Note that this function really return all the consistent possible slots, so no effort is done to find the one of maximum lenght or for other optimalizations.
Hope this helps! (I got it to work as you described in my pycharm)
Here's a solution, I'll include all the code below.
1. Create a table of slots, and a table of reservations
2. Create a matrix of reservations x slots
which is populated by true or false values based on whether that reservation-slot combination are possible
3. Figure out the best mapping that allows for the most Reservation-Slot Combinations
Note: my current solution scales poorly with very large arrays as it involves looping through all possible permutations of a list with size = number of slots. I've posted another question to see if anyone can find a better way of doing this. However, this solution is accurate and can be optimized
Python Code Source
Part 1
from IPython.display import display
import pandas as pd
import datetime
available_data = [
['SlotA', datetime.time(11, 0, 0), datetime.time(12, 30, 0), set(list('ABD'))],
['SlotB',datetime.time(12, 0, 0), datetime.time(13, 30, 0), set(list('C'))],
['SlotC',datetime.time(12, 0, 0), datetime.time(13, 30, 0), set(list('ABCD'))],
['SlotD',datetime.time(12, 0, 0), datetime.time(13, 30, 0), set(list('AD'))],
]
reservation_data = [
['ReservationA', datetime.time(11, 15, 0), datetime.time(12, 15, 0), set(list('AD'))],
['ReservationB', datetime.time(11, 15, 0), datetime.time(12, 15, 0), set(list('A'))],
['ReservationC', datetime.time(12, 0, 0), datetime.time(12, 15, 0), set(list('C'))],
['ReservationD', datetime.time(12, 0, 0), datetime.time(12, 15, 0), set(list('C'))],
['ReservationE', datetime.time(12, 0, 0), datetime.time(12, 15, 0), set(list('D'))]
]
reservations = pd.DataFrame(data=reservation_data, columns=['reservations', 'begin', 'end', 'tags']).set_index('reservations')
slots = pd.DataFrame(data=available_data, columns=['slots', 'begin', 'end', 'tags']).set_index('slots')
display(slots)
display(reservations)
Part 2
def is_possible_combination(r):
return (r['begin'] >= slots['begin']) & (r['end'] <= slots['end']) & (r['tags'] <= slots['tags'])
solution_matrix = reservations.apply(is_possible_combination, axis=1).astype(int)
display(solution_matrix)
Part 3
import numpy as np
from itertools import permutations
# add dummy columns to make the matrix square if it is not
sqr_matrix = solution_matrix
if sqr_matrix.shape[0] > sqr_matrix.shape[1]:
# uhoh, there are more reservations than slots... this can't be good
for i in range(sqr_matrix.shape[0] - sqr_matrix.shape[1]):
sqr_matrix.loc[:,'FakeSlot' + str(i)] = [1] * sqr_matrix.shape[0]
elif sqr_matrix.shape[0] < sqr_matrix.shape[1]:
# there are more slots than customers, why doesn't anyone like us?
for i in range(sqr_matrix.shape[0] - sqr_matrix.shape[1]):
sqr_matrix.loc['FakeCustomer' + str(i)] = [1] * sqr_matrix.shape[1]
# we only want the values now
A = solution_matrix.values.astype(int)
# make an identity matrix (the perfect map)
imatrix = np.diag([1]*A.shape[0])
# randomly swap columns on the identity matrix until they match.
n = A.shape[0]
# this will hold the map that works the best
best_map_so_far = np.zeros([1,1])
for column_order in permutations(range(n)):
# this is an identity matrix with the columns swapped according to the permutation
imatrix = np.zeros(A.shape)
for row, column in enumerate(column_order):
imatrix[row,column] = 1
# is this map better than the previous best?
if sum(sum(imatrix * A)) > sum(sum(best_map_so_far)):
best_map_so_far = imatrix
# could it be? a perfect map??
if sum(sum(imatrix * A)) == n:
break
if sum(sum(imatrix * A)) != n:
print('a perfect map was not found')
output = pd.DataFrame(A*imatrix, columns=solution_matrix.columns, index=solution_matrix.index, dtype=int)
display(output)
The suggested approaches by Arne and tinker were both helpful, but not ultimately sufficient. I came up with a hybrid approach that solves it well enough.
The main problem is that it's a three-dimensional issue, which is difficult to solve in all dimensions at once. It's not just about matching a time overlap or a tag overlap, it's about matching time slices with tag overlaps. It's simple enough to match slots to other slots based on time and even tags, but it's then pretty complicated to match an already matched availability slot to another reservation at another time. Meaning, this scenario in which one availability can cover two reservations at different times:
+---------+
| A, B |
+---------+
xxxxx xxxxx
x A x x A x
xxxxx xxxxx
Trying to fit this into constraint based programming requires an incredibly complex relationship of constraints which is hardly manageable. My solution to this was to simplify the problem…
Removing one dimension
Instead of solving all dimensions at once, it simplifies the problem enormously to largely remove the dimension of time. I did this by using my existing interval tree and slicing it as needed:
def __init__(self, slots):
self.tree = IntervalTree(slots)
def timeslot_is_available(self, start: datetime, end: datetime, attributes: set):
candidate = Slot(start.timestamp(), end.timestamp(), dict(type=SlotType.RESERVED, attributes=attributes))
slots = list(self.tree[start.timestamp():end.timestamp()])
return self.model_is_consistent(slots + [candidate])
To query whether a specific slot is available, I take only the slots relevant at that specific time (self.tree[..:..]), which reduces the complexity of the calculation to a localised subset:
| | +-+ = availability
+-|------|-+ xxx = reservation
| +---|------+
xx|x xxx|x
| xxxx|
| |
Then I confirm the consistency within that narrow slice:
#staticmethod
def model_is_consistent(slots):
def can_handle(r):
return lambda a: r.attributes <= a.attributes and a.contains_interval(r)
av = [s for s in slots if s.type == SlotType.AVAILABLE]
rs = [s for s in slots if s.type == SlotType.RESERVED]
p = Problem()
p.addConstraint(AllDifferentConstraint())
p.addVariables(range(len(rs)), av)
for i, r in enumerate(rs):
p.addConstraint(can_handle(r), (i,))
return p.getSolution() is not None
(I'm omitting some optimisations and other code here.)
This part is the hybrid approach of Arne's and tinker's suggestions. It uses constraint-based programming to find matching slots, using the matrix algorithm suggested by tinker. Basically: if there's any solution to this problem in which all reservations can be assigned to a different available slot, then this time slice is in a consistent state. Since I'm passing in the desired reservation slot, if the model is still consistent including that slot, this means it's safe to reserve that slot.
This is still problematic if there are two short reservations assignable to the same availability within this narrow window, but the chances of that are low and the result is merely a false negative for an availability query; false positives would be more problematic.
Finding available slots
Finding all available slots is a more complex problem, so again some simplification is necessary. First, it's only possible to query the model for availabilities for a particular set of tags (there's no "give me all globally available slots"), and secondly it can only be queried with a particular granularity (desired slot length). This suits me well for my particular use case, in which I just need to offer users a list of slots they can reserve, like 9:15-9:30, 9:30-9:45, etc.. This makes the algorithm very simple by reusing the above code:
def free_slots(self, start: datetime, end: datetime, attributes: set, granularity: timedelta):
slots = []
while start < end:
slot_end = start + granularity
if self.timeslot_is_available(start, slot_end, attributes):
slots.append((start, slot_end))
start += granularity
return slots
In other words, it just goes through all possible slots during the given time interval and literally checks whether that slot is available. It's a bit of a brute-force solution, but works perfectly fine.
I would like to create a form in Elm that takes 4 required inputs:
3 floating point values
1 input which can take the values of "long" or "short" (presumably) this would be a drop-down
When the values are entered, a computation occurs that yields a single line of output based on these values.
I have this working as a command-line Python program:
#!/usr/bin/env python
from __future__ import print_function
# core
import logging
# pypi
import argh
# local
logging.basicConfig(
format='%(lineno)s %(message)s',
level=logging.WARN
)
def main(direction, entry, swing, atr):
entry = float(entry)
swing = float(swing)
atr = float(atr)
if direction == 'long':
sl = swing - atr
tp = entry + (2 * atr)
elif direction == 'short':
sl = swing + atr
tp = entry - (2 * atr)
print("For {0} on an entry of {1}, SL={2} and TP={3}".format(
direction, entry, sl, tp))
if __name__ == '__main__':
argh.dispatch_command(main)
but want to use Elm to create a web UI for it.
This isn't an ideal first introduction to Elm, since it takes you through a few of Elm's more difficult areas: mailboxes and the Graphics.Input library. There is also not an easy-to-deploy number input, so I've only implemented the dropdown; you'd use Graphics.Input.Field (which can get particularly dicey).
A mailbox is basically a signal that the dropdown input can send a message to. I've chosen for that message to be a function, specifically, how to produce a new state from the old one. We define state to be a record type (like a Python named tuple), so saving the direction involves storing the new direction in the record.
The text is rendered using a convenience function that makes it monospace. There's a whole text library you can use, but Elm 0.15 does not have string interpolation, so you're stuck with the quite ugly appends.
Finally, there's not much on an Elm community on SO, but you're welcome to join the mailing list where questions like this are welcomed. That being said, you really should try to dive into Elm and learn the basics so you can ask a more specific question. This is true of almost any language, library or framework - you have to do some basic exercises before you can write "useful" code.
import Graphics.Input as Input
import Graphics.Element as Element exposing (Element)
mailbox = Signal.mailbox identity
type Direction = Short | Long
type alias State = {entry : Float, swing : Float, atr : Float, dir : Direction}
initialState : State
initialState = State 0 0 0 Short
dropdown : Element
dropdown =
Input.dropDown
(\dir -> Signal.message mailbox.address (\state -> {state| dir <- dir}))
[("Short", Short), ("Long", Long)]
state : Signal State
state = Signal.foldp (<|) initialState mailbox.signal
render : State -> Element
render state =
dropdown `Element.above`
Element.show ("For "++(toString state.dir)++" on an entry of "++(toString state.entry) ++
", SL="++(toString (sl state))++" and TP="++(toString (tp state)))
main = Signal.map render state
-- the good news: these are separate, pure functions
sl {entry, swing, atr, dir} =
case dir of
Long -> swing - atr
Short -> swing + atr
tp {entry, swing, atr, dir} =
case dir of
Long -> entry + (2*atr)
Short -> entry - (2*atr)
Update: I wrote an essay on mailboxes and how to use them.