I’m trying to use the BLEU score from NLTK for quality evaluation of the machine translation. I wanted to check this code with two identical sentences, here I’m using method1 as a Smoothing function because I’m comparing two sentences and not corpora. I set 4-grams and weights 0.25 (1/4). But as a result, I’m getting 0.0088308. What am I doing wrong? Two identical sentences should get a score 1.0. I'm coding on Python 3, Windows 7, in PyCharm.
My code:
import nltk
from nltk import word_tokenize
from nltk.translate.bleu_score import SmoothingFunction
ref = 'You know that it would be untrue You know that I would be a liar If I was to say to you Girl, we couldnt get much higher.'
cand = 'You know that it would be untrue You know that I would be a liar If I was to say to you Girl, we couldnt get much higher.'
smoothie = SmoothingFunction().method1
reference = word_tokenize(ref)
candidate = word_tokenize(cand)
weights = (0.25, 0.25, 0.25, 0.25)
BLEUscore = nltk.translate.bleu_score.sentence_bleu(reference, candidate, weights, smoothing_function=smoothie)
print(BLEUscore)
My result:
0.008830895300928163
Process finished with exit code 0
BLEU allows to compare set of references with a candidate, so if you want to use it you should set the list of lists of sentences as a list of references. In other words, even if you take only one reference it should be a list of lists (in my example reference should be [reference]:
BLEUscore = nltk.translate.bleu_score.sentence_bleu([reference], candidate, weights, smoothing_function=smoothie)
When I put reference in [] I've got 1.0.
Related
I have a matrix addition with several terms that I want to display in a Jupyter Notebook. I need the order of terms to match the standard notation - in my case, of linear regression. But, the terms do not, by default, appear in the correct order for my purpose, and I would like to ask how to control the order of display of matrices in a matrix addition (MatAdd) term in Sympy. For example, here we see that Sympy selects a particular order for the terms, that appears to be based on the values in the Matrix.
from sympy import MatAdd, Matrix
A = Matrix([1])
B = Matrix([0])
print(MatAdd(A, B, evaluate=False))
This gives
Matrix([[0]]) + Matrix([[1]])
Notice the matrix terms do not follow the order of defintion or the variable names.
Is there anything I can do to control the print output order of Matrix terms in a MatAdd expression?
You can use init_printing to chose from a few options. In particular, the order keyword should control how things are shown on the screen vs how things are stored in SymPy objects.
Now comes the differences: by setting init_printing(order="none") printers behave differently. I believe this is some bug.
For example, I usually use Latex rendering when using Jupyter Notebook:
from sympy import MatAdd, Matrix, init_printing
init_printing(order="none")
A = Matrix([1])
B = Matrix([0])
add = MatAdd(A, B, evaluate=False)
print(add)
# out: Matrix([[0]]) + Matrix([[1]])
display(add)
# out: [1] + [0]
Here you can see that the latex printer is displaying the elements as they are stored (check add.args), whereas the string printer is not following that convention...
I am trying to convert a matrix to the type that can be received by gensim. AuthorTopic Model, which means I should convert a matrix to a sparse vector. I have already tried several functions in gensim like gensim.matutils.full2sparse and gensim.matutils.any2sparse. But there is something wrong:
my code:
matrix=numpy.array([[1,0 ,1],[0,1,1]])
mycorpus=any2sparse(matrix)
print(matrix)
print(mycorpus)
the output:
[[1 0 1]
[0 1 1]]
[(0, 1.0), (0, 1.0), (1, 0.0), (1, 0.0)] #mycorpus
accoring to the tutorial, mycorpus should be like:
[[(0,1),(2,1)]
[(1,1),(2,1)]]
I have no idea what's wrong. I really appreciate if anyone could give me some advise.
The Gensim AuthorTopicModel docs describe its desired corpus-format as iterable of list of (int, float).
Those int values would be word-ids, and ideally be accompanied by the id2word dict which idntifies which int means which word.
What's the source of your matrix, & do you know if it's the rows or the columns that represent words, and have a mapping of indexes to words? That will drive the conversion.
Also, as the docs mention, "The model is closely related to LdaModel. The AuthorTopicModel class inherits LdaModel, and its usage is thus similar.
Have you reviewed guides to Gensim LDA usage to see how they prepare their corpus, such as the multiple Usage Examples, to see if that helps suggest steps & necessary formats?
Or, is your corpus still available as texts, so you can directly use the examples there as a model to turn the text into the BoW format (rather than your already-processed matrix)?
If you're still having problems, you should expand your question text with more details, especially how the true corpus matrix that you have was created, and which errors you've encountered (& how you triggered them) that convince you things aren't working.
I have noticed that my gensim Doc2Vec (DBOW) model is sensitive to document tags. My understanding was that these tags are cosmetic and so they should not influence the learned embeddings. Am I misunderstanding something? Here is a minimal example:
from gensim.test.utils import common_texts
from gensim.models.doc2vec import Doc2Vec, TaggedDocument
import numpy as np
import os
os.environ['PYTHONHASHSEED'] = '0'
reps = []
for a in [0,500]:
documents = [TaggedDocument(doc, [i + a])
for i, doc in enumerate(common_texts)]
model = Doc2Vec(documents, vector_size=100, window=2, min_count=0,
workers=1, epochs=10, dm=0, seed=0)
reps.append(np.array([model.docvecs[k] for k in range(len(common_texts))])
reps[0].sum() == reps[1].sum()
This last line returns False. I am working with gensim 3.8.3 and Python 3.5.2. More generally, is there any role that the values of the tags play (assuming they are unique)? I ask because I have found that using different tags for documents in a classification task leads to widely varying performance.
Thanks in advance.
First & foremost, your test isn't even comparing vectors corresponding to the same texts!
In run #1, the vector for the 1st text in in model.docvecs[0]. In run #2, the vector for the 1st text is in model.docvecs[1].
And, in run #2, the vector at model.docvecs[0] is just a randomly-initialized, but never-trained, vector - because none of the training texts had a document tag of (int) 0. (If using pure ints as the doc-tags, Doc2Vec uses them as literal indexes - potentially leaving any unused slots less than your highest tag allocated-and-initialized, but never-trained.)
Since common_texts only has 11 entries, by the time you reach run #12, all the vectors in your reps array of the first 11 vectors are garbage uncorrelated with any of your texts/
However, even after correcting that:
As explained in the Gensim FAQ answer #11, determinism in this algorithm shouldn't generally be expected, given many sources of potential randomness, and the fuzzy/approximate nature of the whole approach. If you're relying on it, or testing for it, you're probably making some unwarranted assumptions.
In general, tests of these algorithms should be evaluating "roughly equivalent usefulness in comparative uses" rather than "identical (or even similar) specific vectors". For example, a test whether apple and orange are roughly at the same positions in each others' nearest-neighbor rankings makes more sense than checking their (somewhat arbitrary) exact vector positions or even cosine-similarity.
Additionally:
tiny toy datasets like common_texts won't show the algorithm's usual behavior/benefits
PYTHONHASHSEED is only consulted by the Python interpreter at startup; setting it from Python can't have any effect. But also, the kind of indeterminism it introduces only comes up with separate interpreter launches: a tight loop within a single interpreter run like this wouldn't be affected by that in any case.
Have you checked the magnitude of the differences?
Just running:
delta = reps[0].sum() - reps[1].sum()
for the aggregate differences results with -1.2598932e-05 when I run it.
Comparison dimension-wise:
eps = 10**-4
over = (np.abs(diff) <= eps).all()
Returns True on a vast majority of the runs which means that you are getting quite reproducible results given the complexity of the calculations.
I would blame numerical stability of the calculations or uncontrolled randomness. Even though you do try to control the random seed, there is a different random seed in NumPy and different in random standard library so you are not controlling for all of the sources of randomness. This can also have an influence on the results but I did not check the actual implementation in gensim and it's dependencies.
Change
import os
os.environ['PYTHONHASHSEED'] = '0'
to
import os
import sys
hashseed = os.getenv('PYTHONHASHSEED')
if not hashseed:
os.environ['PYTHONHASHSEED'] = '0'
os.execv(sys.executable, [sys.executable] + sys.argv)
My current understanding:
I have tried reading a few papers and links regarding NMF. It all talks about how we can split a MxN matrix into MxR and RxN matrices(R
Question:
I have a list of users(U) and some assignments(A) for each user. Now I split this matrix(UxA) using NMF. I get 2 Matrices UxR and RxA. How do I use these to predict what assignments(A') a new user(U') must have?
Any help would be appreciated as I couldn't understand this after trying to search for the answer.
Side question and opinion based:
Also if anyone can tell me with their experience, how do they chose R, specially when the number of assignments are in the order of 50,000 or perhaps a hundred thousand. I have been trying these with the scikit-learn library
Edit:
This can simply be done using model.inverse_transform(model.transform(User'))
You can try think this problem as recommender. you want to approximate decompose matrix X into two nonnegative matrix U and V.
see https://cambridgespark.com/content/tutorials/implementing-your-own-recommender-systems-in-Python/index.html
For pyothn scikit-learn, you can use:
http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.NMF.html
from sklearn.decomposition import NMF
model = NMF(n_components=2, init='random', random_state=0)
W = model.fit_transform(X)
H = model.components_
Where X is the matrix you want to decomose. W and H is the nonnegative factor
To predict what assignments(A') a new user(U'), you just use WH' to complete the maitrx
I'm taking the CalTech online course Learning From Data, and I'm stumped with creating a Perceptron in Scala. I chose Scala because I'm learning it and wanted to challenge myself. I understand the theory, and I also understand others' solutions in Python and Ruby. But I can't figure out why my own Scala code doesn't work.
For a background in the Perceptron code: Learning_algorithm
I'm running Scala 2.11 on OSX 10.10.
Per the algorithm, I start off with weights (0.0, 0.0, 0.0), where weight[2] is a learned bias component. I've already generated a test set in the space [-1, 1],[-1,1] on the X-Y plane. I do this by a) picking two random points and drawing a line through them, then b) generating some other random points and calculating if they are on one side of the line or the other. As far as I can tell by plotting it in Python, this generates linearly separable data.
My next step is to take my initialized weights and check against every point to find miss-classified points, i.e. points that don't generate the right +1 or -1 result. Here is the code that simply calculates dot-product of the weight and the vector x:
def h(weight:List[Double], p:Point ): Double = if ( (weight(0)*p.x + weight(1)*p.y + weight(2)) > 0) 1 else -1
It's the initial weights, so they are all miss-classified. I then update the weights, like so:
def newH(weight:List[Double], p:Point, y:Double): List[Double] = {
val newWt = scala.collection.mutable.ArrayBuffer[Double](0.0, 0.0, 0.0)
newWt(0) = weight(0) + p.x*y
newWt(1) = weight(1) + p.y*y
newWt(2) = weight(2) + 1*y
return newWt.toList
}
Then I identify miss-classified points again by checking the test set against the value output by h() above, and continue iterating.
This follows the algorithm (or is supposed to, at least) that Prof Yaser shows here: Library
The problem is that the algorithm never converges. My weights -- the third component of which is the bias -- keep getting more negative or more positive. My weight vector after every adjustment resembles this:
Weights: List(16.43341624736786, 11627.122008800507, -34130.0)
Weights: List(15.533397436141968, 11626.464265227318, -34131.0)
Weights: List(14.726969361305237, 11626.837346673012, -34132.0)
Weights: List(14.224745154380798, 11627.646470665932, -34133.0)
Weights: List(14.075232982635498, 11628.026384592056, -34134.0)
I'm a Scala newbie so my code is probably atrocious. But am I missing something in Scala, e.g. reassignment, that could be causing my weight to be messed up? Or have I completely misunderstood how the Perceptron even operates? Is my weight update just wrong?
Thanks for any help you can give me on this!
Thanks Till. I've discovered the two problems with my code and I'll share them, but to address your point: Someone else asked about this on the class's forum and it looks like what the Wiki formula does is simply to change the learning rate. Alpha can be picked randomly, and y-h(weight, p) would give you weights like
-1-1 = 2
In the case that y=-1 and h()=1, or
1-(-1) = 2
In the case that y=1 and h()=-1
My/the class formula takes 1*p.x instead of alpha*2, which seems to be a matter of different learning rates. Hope that makes sense.
My two problems were as follows:
The y value passed into the recalculation formula newH needs to be the target value of y, that is, the "correct y" that was discovered while generating the test points. I was passing in the y that was generated through h(), which is the guessed-at function. This makes sense obviously since we are looking to correc the weight by using the target y, not the incorrect y.
I was doing a comparison of target y and h()=yin Scala, but was comparison an element obtained from a map through .get(). My Scala map looks like Map[Point,Double] where the Double value refers to the y value generated during the test set creation. But doing a .get() gives you Option[Double] and not a Double value at all. This is explained in Scala Map#get and the return of Some() and makes a lot of sense now. I did map.get(<some Point>).get() for now, since I was focusing on debugging and not code perfection, and then I was accurately able to compare two Double values.