Pyspark: Parallelize a UDF - parallel-processing

I want to loop through two lists, pass the the combos to a function, and get the following output:
ru = ['a', 'b', 'c']
ni = ['x', 'y', 'z']
def my_func(ru, ni):
print("{} + {}".format(ru, ni))
for i in ru:
for j in ni:
my_func(i,j)
# Output
a + x
a + y
a + z
b + x
b + y
b + z
c + x
c + y
c + z
Since this is Pyspark, I would like to parallelize it, since each iteration of the function can run independently.
Note: My actual function is a long complicated algorithm in pyspark. Just wanted to post an easy example to generalize.
What is the best way to do this?
​

Use cartesian:
ru = sc.parallelize(['a', 'b', 'c'])
ni = sc.parallelize(['x', 'y', 'z'])
print(ru.cartesian(ni).collect())
For your case,
ru.cartesian(ni).map(some_func)
Or:
def my_func(ru, ni):
print("{} + {}".format(ru, ni))
ru.cartesian(ni).foreach(lambda t: my_func(t[0], t[1]))
a + z
a + y
a + x
b + y
b + x
b + z
c + y
c + x
c + z

Related

Mathematica Code with Module and If statement

Can I simply ask the logical flow of the below Mathematica code? What are the variables arg and abs doing? I have been searching for answers online and used ToMatlab but still cannot get the answer. Thank you.
Code:
PositiveCubicRoot[p_, q_, r_] :=
Module[{po3 = p/3, a, b, det, abs, arg},
b = ( po3^3 - po3 q/2 + r/2);
a = (-po3^2 + q/3);
det = a^3 + b^2;
If[det >= 0,
det = Power[Sqrt[det] - b, 1/3];
-po3 - a/det + det
,
(* evaluate real part, imaginary parts cancel anyway *)
abs = Sqrt[-a^3];
arg = ArcCos[-b/abs];
abs = Power[abs, 1/3];
abs = (abs - a/abs);
arg = -po3 + abs*Cos[arg/3]
]
]
abs and arg are being reused multiple times in the algorithm.
In a case where det > 0 the steps are
po3 = p/3;
b = (po3^3 - po3 q/2 + r/2);
a = (-po3^2 + q/3);
abs1 = Sqrt[-a^3];
arg1 = ArcCos[-b/abs1];
abs2 = Power[abs1, 1/3];
abs3 = (abs2 - a/abs2);
arg2 = -po3 + abs3*Cos[arg1/3]
abs3 can be identified as A in this answer: Using trig identity to a solve cubic equation
That is the most salient point of this answer.
Evaluating symbolically and numerically may provide some other insights.
Using demo inputs
{p, q, r} = {-2.52111798, -71.424692, -129.51520};
Copyable version of trig identity notes - NB a, b, p & q are used differently in this post
Plot[x^3 - 2.52111798 x^2 - 71.424692 x - 129.51520, {x, 0, 15}]
a = 1;
b = -2.52111798;
c = -71.424692;
d = -129.51520;
p = (3 a c - b^2)/3 a^2;
q = (2 b^3 - 9 a b c + 27 a^2 d)/27 a^3;
A = 2 Sqrt[-p/3]
A == abs3
-(b/3) + A Cos[1/3 ArcCos[
-((b/3)^3 - (b/3) c/2 + d/2)/Sqrt[-(-(b^2/9) + c/3)^3]]]
Edit
There is also a solution shown here
TRIGONOMETRIC SOLUTION TO THE CUBIC EQUATION, by Alvaro H. Salas
Clear[a, b, c]
1/3 (-a + 2 Sqrt[a^2 - 3 b] Cos[1/3 ArcCos[
(-2 a^3 + 9 a b - 27 c)/(2 (a^2 - 3 b)^(3/2))]]) /.
{a -> -2.52111798, b -> -71.424692, c -> -129.51520}
10.499

For integers A>0, B>0, N>0, find integers x>0,y>0 such that N-(Ax+By) is smallest non-negative

Example :
A=5, B=2, N=12
Then let x=2, y=1, so 12 - (5(2) + 2(1)) = 0.
Another example:
A=5, B=4, N=12
Here x=1, y=1 is the best possible. Note x=2, y=0 would be better except that x=0 is not allowed.
I'm looking for something fast.
Note it's sufficient to find the value of Ax+By. It's not necessary to give x or y explicitly.
If gcd(A,B)|N, then N is your maximal value. Otherwise, it's the greatest multiple of gcd(A,B) that's smaller than N. Using 4x+2y=13 as an example, that value is gcd(4,2)*6=12 realized by 4(2)+2(2)=12 (among many solutions).
As a formula, your maximal value is Floor(N/gcd(A,B))*gcd(A,B).
Edit: If both x and y must be positive, this may not work. However, won't even be a solution if A+B>N. Here's an algorithm for you...
from math import floor, ceil
def euclid_wallis(m, n):
col1 = [1, 0, m]
col2 = [0, 1, n]
while col2[-1] != 0:
f = -1 * (col1[-1] // col2[-1])
col2, col1 = [x2 * f + x1 for x1, x2 in zip(col1, col2)], col2
return col1, col2
def positive_solutions(A, B, N):
(x, y, gcf), (cx, cy, _) = euclid_wallis(A, B)
f = N // gcf
while f > 0:
fx, fy, n = f*x, f*y, f*gcf
k_min = (-fx + 0.) / cx
k_max = (-fy + 0.) / cy
if cx < 0:
k_min, k_max = k_max, k_min
if floor(k_min) + 1 <= ceil(k_max) - 1:
example_k = int(floor(k_min) + 1)
return fx + cx * example_k, fy + cy * example_k, n
if k_max <= 1:
raise Exception('No solution - A: {}, B: {}, N: {}'.format(A, B, N))
f -= 1
print positive_solutions(5, 4, 12) # (1, 1, 9)
print positive_solutions(2, 3, 6) # (1, 1, 5)
print positive_solutions(23, 37, 238) # (7, 2, 235)
A brute-force O(N^2 / A / B) algorithm, implemented in plain Python3:
import math
def axby(A, B, N):
return [A * x + B * y
for x in range(1, 1 + math.ceil(N / A))
for y in range(1, 1 + math.ceil(N / B))
if (N - A * x - B * y) >= 0]
def bestAxBy(A, B, N):
return min(axby(A, B, N), key=lambda x: N - x)
This matched your examples:
In [2]: bestAxBy(5, 2, 12)
Out[2]: 12 # 5 * (2) + 2 * (1)
In [3]: bestAxBy(5, 4, 12)
Out[3]: 9 # 5 * (1) + 4 * (1)
Have no idea what algorithm that might be, but I think you need something like that (C#)
static class Program
{
static int solve( int a, int b, int N )
{
if( a <= 0 || b <= 0 || N <= 0 )
throw new ArgumentOutOfRangeException();
if( a + b > N )
return -1; // Even x=1, y=1 still more then N
int x = 1;
int y = ( N - ( x * a ) ) / b;
int zInitial = a * x + b * y;
int zMax = zInitial;
while( true )
{
x++;
y = ( N - ( x * a ) ) / b;
if( y <= 0 )
return zMax; // With that x, no positive y possible
int z = a * x + b * y;
if( z > zMax )
zMax = z; // Nice, found better
if( z == zInitial )
return zMax; // x/y/z are periodical, returned where started, meaning no new values are expected
}
}
static void Main( string[] args )
{
int r = solve( 5, 4, 12 );
Console.WriteLine( "{0}", r );
}
}

I have been trying to write a program python

I have been trying to write a program a simple program without importing any library. I simply want to print the following strings in this array in vertical form without using any complex algorithm. I will be glad if anyone can help me please.
['San Francisco', 'Christchurch ', 'Sydney ', 'Bangkok ', 'Copenhagen ']
This can be done using some built-in functions, like max(), len() and zip():
L = ['San Francisco', 'Christchurch ', 'Sydney ', 'Bangkok ', 'Copenhagen ']
max_length = len(max(L, key = lambda x : len(x)))
new_L = []
for e in L:
new_L.append(e + ' ' * (max_length - len(e)))
for e in zip(*new_L):
for el in e:
if el != ' ':
print el,
Output:
S C S B C a h y a o n r d n p i n g e F s e k n r t y o h a c k a n h g c u e i r n s c c h o
The lines:
new_L = []
for e in L:
new_L.append(e + ' ' * (max_length - len(e)))
can be written with list comprehension like:
new_L = [e + ' ' * (max_length - len(e)) for e in L]
Edit:
L = ['San Francisco', 'Christchurch ', 'Sydney ', 'Bangkok ', 'Copenhagen ']
# Get the maximum length of a string in the list
max_length = len(max(L, key = lambda x : len(x)))
#print max(L, key = lambda x : len(x)) # get the maximum of the list based in length
#print max_length
# Iterate through indices of max_length: 0, 1, 2, 3 ...
for i in range(max_length):
# Iterate through each city in the list
for city in L:
# If the index is < than the length of the city
if i < len(city) and city[i] != ' ':
print city[i],

how to add expressions and do numerical calculation in Maxima?

I would like to ask:
how I can add expressions in Maxima? i.e. I have:
A = x + y;
B = 2*x + 2*y;
How to get Maxima to give me (A + B)?
how I can do numerical calculation in Maxima? I want to assign
x = 1;
b = 2;
How to get the numerical value of (A + B)?
(1) assignment in Maxima uses the colon symbol (i.e., ":") not the equal sign ("=").
(2) there are a couple of ways to evaluate with specific values.
(2a) subst([x = ..., y = ...], foo) where foo is some expression such as foo : A + B.
(2b) ev(foo, x = ..., y = ...)
So:
(%i1) A : x + y;
(%o1) y + x
(%i2) B : 2*x + 2*y;
(%o2) 2 y + 2 x
(%i3) foo : A + B;
(%o3) 3 y + 3 x
(%i4) subst ([x = 1, y = 2], foo);
(%o4) 9
(%i5) ev (foo, x = 1, y = 2);
(%o5) 9
Yet another way to substitute values into a formula is with the '' operator as follows:
(%i57) A : 2*a+b ; B : a-b;
(%o57) b + 2 a
(%o58) a - b
(%i59) a : 4; b : 10;
(%o59) 4
(%o60) 10
(%i61) A;
(%o61) b + 2 a
(%i62) ''A;
(%o62) 18
(%i63) ''B;
(%o64) - 6
(%i65) ''A + ''B;
(%o65) 12
(%i66) ''(A+B);
(%o66) 12

Convert boolean expression to 3 input NOR

I was taking a look at this link http://lizarum.com/assignments/boolean_algebra/chapter3.html to try and solve an equation I have. The original equation is:
H = MC + MC' + CRD + M'CD'
I simplified it to
H = M + CRD + M'CD'
Here is my attempt:
H = ((M + CRD + M'CD')')'
H = ((M)' * (CRD)' * (M'CD')')'
H = (((M)')' + ((CRD)')' + ((M'CD')')'
H = ((M')' + (C'+ R' + D')' + (M + C' + D)')'
Is that final equation a 3 input NOR equation? I have a feeling that I'm missing a step that makes the first parentheses into three variables.
As a first step, notice that M C + M C' = M
This simplifies your equation to
H = M + CRD + M'CD'
You can then leave out the M'. If it were false, M would be true and thus H.
H = M + CRD + CD'
This allows you to factor-out C:
H = M + C(RD + D')
Term D in the parentheses can be left out (same argument as above)
H = M + C(R + D')
The final result is:
H = M + CR + CD'
You could have arrived at this result using a Karnaugh-Veitch map.
Convince yourself asking WolframAlpha.

Resources