Please help understand what are the meaningful/significant differences among different ways to create kubeflow pipeline components and the reason for having so many ways?
from kfp.components import func_to_container_op
#func_to_container_op
def add_op(a: float, b: float) -> float:
"""Returns sum of two arguments"""
return a + b
from kfp.v2.dsl import component
#component
def add_op(a: float, b: float) -> float:
"""Returns sum of two arguments"""
return a + b
from kfp.components import create_component_from_func
#create_component_from_func
def add_op(a: float, b: float) -> float:
"""Returns sum of two arguments"""
return a + b
Great question! And I fully agree that KFP in the past has often offered multiple ways to do similar or even same tasks, which causes lots of user confusion.
In KFP SDK 2.0 (currently at beta stage with latest being 2.0.0b5), both func_to_container_op and create_component_from_func are deprecated.
The only option to create a component from a Python function without building a container image is to use #component. For more information, see https://www.kubeflow.org/docs/components/pipelines/v2/author-a-pipeline/components/
Related
In tensorflow CIFAR-10 tutorial in cifar10_inputs.py line 174 it is said you should randomize the order of the operations random_contrast and random_brightness for better data augmentation.
To do so the first thing I think of is drawing a random variable from the uniform distribution between 0 and 1 : p_order. And do:
if p_order>0.5:
distorted_image=tf.image.random_contrast(image)
distorted_image=tf.image.random_brightness(distorted_image)
else:
distorted_image=tf.image.random_brightness(image)
distorted_image=tf.image.random_contrast(distorted_image)
However there are two possible options for getting p_order:
1) Using numpy which disatisfies me as I wanted pure TF and that TF discourages its user to mix numpy and tensorflow
2) Using TF, however as p_order can only be evaluated in a tf.Session()
I do not really know if I should do:
with tf.Session() as sess2:
p_order_tensor=tf.random_uniform([1,],0.,1.)
p_order=float(p_order_tensor.eval())
All those operations are inside the body of a function and are run from another script which has a different session/graph. Or I could pass the graph from the other script as an argument to this function but I am confused.
Even the fact that tensorflow functions like this one or inference for example seem to define the graph in a global fashion without explicitly returning it as an output is a bit hard to understand for me.
You can use tf.cond(pred, fn1, fn2, name=None) (see doc).
This function allows you to use the boolean value of pred inside the TensorFlow graph (no need to call self.eval() or sess.run(), hence no need of a Session).
Here is an example of how to use it:
def fn1():
distorted_image=tf.image.random_contrast(image)
distorted_image=tf.image.random_brightness(distorted_image)
return distorted_image
def fn2():
distorted_image=tf.image.random_brightness(image)
distorted_image=tf.image.random_contrast(distorted_image)
return distorted_image
# Uniform variable in [0,1)
p_order = tf.random_uniform(shape=[], minval=0., maxval=1., dtype=tf.float32)
pred = tf.less(p_order, 0.5)
distorted_image = tf.cond(pred, fn1, fn2)
I would like to create a form in Elm that takes 4 required inputs:
3 floating point values
1 input which can take the values of "long" or "short" (presumably) this would be a drop-down
When the values are entered, a computation occurs that yields a single line of output based on these values.
I have this working as a command-line Python program:
#!/usr/bin/env python
from __future__ import print_function
# core
import logging
# pypi
import argh
# local
logging.basicConfig(
format='%(lineno)s %(message)s',
level=logging.WARN
)
def main(direction, entry, swing, atr):
entry = float(entry)
swing = float(swing)
atr = float(atr)
if direction == 'long':
sl = swing - atr
tp = entry + (2 * atr)
elif direction == 'short':
sl = swing + atr
tp = entry - (2 * atr)
print("For {0} on an entry of {1}, SL={2} and TP={3}".format(
direction, entry, sl, tp))
if __name__ == '__main__':
argh.dispatch_command(main)
but want to use Elm to create a web UI for it.
This isn't an ideal first introduction to Elm, since it takes you through a few of Elm's more difficult areas: mailboxes and the Graphics.Input library. There is also not an easy-to-deploy number input, so I've only implemented the dropdown; you'd use Graphics.Input.Field (which can get particularly dicey).
A mailbox is basically a signal that the dropdown input can send a message to. I've chosen for that message to be a function, specifically, how to produce a new state from the old one. We define state to be a record type (like a Python named tuple), so saving the direction involves storing the new direction in the record.
The text is rendered using a convenience function that makes it monospace. There's a whole text library you can use, but Elm 0.15 does not have string interpolation, so you're stuck with the quite ugly appends.
Finally, there's not much on an Elm community on SO, but you're welcome to join the mailing list where questions like this are welcomed. That being said, you really should try to dive into Elm and learn the basics so you can ask a more specific question. This is true of almost any language, library or framework - you have to do some basic exercises before you can write "useful" code.
import Graphics.Input as Input
import Graphics.Element as Element exposing (Element)
mailbox = Signal.mailbox identity
type Direction = Short | Long
type alias State = {entry : Float, swing : Float, atr : Float, dir : Direction}
initialState : State
initialState = State 0 0 0 Short
dropdown : Element
dropdown =
Input.dropDown
(\dir -> Signal.message mailbox.address (\state -> {state| dir <- dir}))
[("Short", Short), ("Long", Long)]
state : Signal State
state = Signal.foldp (<|) initialState mailbox.signal
render : State -> Element
render state =
dropdown `Element.above`
Element.show ("For "++(toString state.dir)++" on an entry of "++(toString state.entry) ++
", SL="++(toString (sl state))++" and TP="++(toString (tp state)))
main = Signal.map render state
-- the good news: these are separate, pure functions
sl {entry, swing, atr, dir} =
case dir of
Long -> swing - atr
Short -> swing + atr
tp {entry, swing, atr, dir} =
case dir of
Long -> entry + (2*atr)
Short -> entry - (2*atr)
Update: I wrote an essay on mailboxes and how to use them.
Here is my problem.
I'm using sympy and a complex matrix P (all elements of P are complex valued).
I wanna extract the real/imaginary part of the first row.
So, I use the following sequence:
import sympy as sp
P = sp.Matrix([ [a+sp.I*b,c-sp.I*d], [c-sp.I*d,a+sp.I*b] ])
Row = P.row(0)
Row.as_mutable()
Re_row = sp.re(Row)
Im_row = sp.im(Row)
But the code returns me the following error:
"AttributeError: ImmutableMatrix has no attribute as_coefficient."
The error occurs during the operation sp.re(Row) and sp.im(Row)...
Sympy tells me that Row is an Immutable matrix but I specify that I want a mutable one...
So I'm in a dead end, and I don't have the solution...
Could someone plz help me ?
thank you very much !
Most SymPy functions won't work if you just pass a Matrix to them directly. You need to use the methods of the Matrix, or if there is not such method (as is the case here), use applyfunc
In [34]: Row.applyfunc(re)
Out[34]: [re(a) - im(b) re(c) + im(d)]
In [35]: Row.applyfunc(im)
Out[35]: [re(b) + im(a) -re(d) + im(c)]
(I've defined a, b, c, and d as just ordinary symbols here, if you set them as real the answer will come out much simpler).
I've searched for the use of #specialized in the source code of the standard library of Scala 2.8.1. It looks like only a handful of traits and classes use this annotation: Function0, Function1, Function2, Tuple1, Tuple2, Product1, Product2, AbstractFunction0, AbstractFunction1, AbstractFunction2.
None of the collection classes are #specialized. Why not? Would this generate too many classes?
This means that using collection classes with primitive types is very inefficient, because there will be a lot of unnecessary boxing and unboxing going on.
What's the most efficient way to have an immutable list or sequence (with IndexedSeq characteristics) of Ints, avoiding boxing and unboxing?
Specialization has a high cost on the size of classes, so it must be added with careful consideration. In the particular case of collections, I imagine the impact will be huge.
Still, it is an on-going effort -- Scala library has barely started to be specialized.
Specialized can be expensive ( exponential ) in both size of classes and compile time. Its not just the size like the accepted answer says.
Open your scala REPL and type this.
import scala.{specialized => sp}
trait S1[#sp A, #sp B, #sp C, #sp D] { def f(p1:A): Unit }
Sorry :-). Its like a compiler bomb.
Now, lets take a simple trait
trait Foo[Int]{ }
The above will result in two compiled classes. Foo, the pure interface and Foo$1, the class implementation.
Now,
trait Foo[#specialized A] { }
A specialized template parameter here gets expanded/rewritten for 9 different primitive types ( void, boolean, byte, char, int, long, short, double, float ). So, basically you end up with 20 classes instead of 2.
Going back to the trait with 5 specialized template parameters, the classes get generated for every combination of possible primitive types. i.e its exponential in complexity.
2 * 10 ^ (no of specialized parameters)
If you are defining a class for a specific primitive type, you should be more explicit about it such as
trait Foo[#specialized(Int) A, #specialized(Int,Double) B] { }
Understandably one has to be frugal using specialized when building general purpose libraries.
Here is Paul Phillips ranting about it.
Partial answer to my own question: I can wrap an array in an IndexedSeq like this:
import scala.collection.immutable.IndexedSeq
def arrayToIndexedSeq[#specialized(Int) T](array: Array[T]): IndexedSeq[T] = new IndexedSeq[T] {
def apply(idx: Int): T = array(idx)
def length: Int = array.length
}
(Ofcourse you could still modify the contents if you have access to the underlying array, but I would make sure that the array isn't passed to other parts of my program).
In his book programming in scala (Chapter 5 Section 5.9 Pg 93)
Odersky mentioned this expression "bills !*&^%~ code!"
In the footnote on same page:
"By now you should be able to figure out that given this code,the Scala compiler would
invoke (bills.!*&^%~(code)).!()."
That's a bit to cryptic for me, could someone explain what's going on here?
What Odersky means to say is that it would be possible to have valid code looking like that. For instance, the code below:
class BadCode(whose: String, source: String) {
def ! = println(whose+", what the hell do you mean by '"+source+"'???")
}
class Programmer(who: String) {
def !*&^%~(source: String) = new BadCode(who, source)
}
val bills = new Programmer("Bill")
val code = "def !*&^%~(source: String) = new BadCode(who, source)"
bills !*&^%~ code!
Just copy&paste it on the REPL.
The period is optional for calling a method that takes a single parameter, or has an empty parameter list.
When this feature is utilized, the next chunk after the space following the method name is assumed to be the single parameter.
Therefore,
(bills.!*&^%~(code)).!().
is identical to
bills !*&^%~ code!
The second exclamation mark calls a method on the returned value from the first method call.
I'm not sure if the book provides method signatures but I assume it's just a comment on Scala's syntactic sugar so it assumes if you type:
bill add monkey
where there is an object bill which has a method add which takes a parameter then it automatically interprets it as:
bill.add(monkey)
Being a little Scala rusty, I'm not entirely sure how it splits code! into (code).!() except for a vague tickling of the grey cells that the ! operator is used to fire off an actor which in compiler terms might be interpretted as an implicit .!() method on the object.
The combination of the '.()' being optional with method calls (as Wysawyg explained above) and the ability to use (almost) whatever characters you like for naming methods, makes it possible to write methods in Scala that look like operator overloading. You can even invent your own operators.
For example, I have a program that deals with 3D computer graphics. I have my own class Vector for representing a 3D vector:
class Vector(val x: Double, val y: Double, val z: Double) {
def +(v: Vector) = new Vector(x + v.x, y + v.y, z + v.z)
// ...etc.
}
I've also defined a method ** (not shown above) to compute the cross product of two vectors. It's very convenient that you can create your own operators like that in Scala, not many other programming languages have this flexibility.