I followed the example in this link and ran the following script to process the latest english wikipedia articles:
https://radimrehurek.com/gensim/wiki.html
$ python -m gensim.scripts.make_wiki
The result of running the script after 9 hours is that I now have .mm and .txt files. I want to train a word2vec model but all the examples found start from the .bz2 file.
How do I train a word2vec model using the .mm files as input instead of the raw bz2 file? The link below shows how to train an LDA model. Can someone pls share syntax?
https://radimrehurek.com/gensim/wiki.html
Thanks!
Related
I am new to deep learning and I am trying to play with a pretrained word embedding model from a paper. I downloaded the following files:
1)sa-d300-m2-fasttext.model
2)sa-d300-m2-fasttext.model.trainables.syn1neg.npy
3)sa-d300-m2-fasttext.model.trainables.vectors_ngrams_lockf.npy
4)sa-d300-m2-fasttext.model.wv.vectors.npy
5)sa-d300-m2-fasttext.model.wv.vectors_ngrams.npy
6)sa-d300-m2-fasttext.model.wv.vectors_vocab.npy
If in case these details are needed
sa - sanskrit
d300 - embedding dimension
fastText - fastText
I dont have a prior experience with gensim, how can load the model into gensim or into tensorflow.
I tried
from gensim.models.wrappers import FastText
FastText.load_fasttext_format('/content/sa/300/fasttext/sa-d300-m2-fasttext.model.wv.vectors_ngrams.npy')
FileNotFoundError: [Errno 2] No such file or directory: '/content/sa/300/fasttext/sa-d300-m2-fasttext.model.wv.vectors_ngrams.npy.bin'
That set of multiple files looks like it was saved from Gensim's FastText implementation, using Gensim's save() method - and thus is not in Facebook's original 'fasttext_format'.
So, try loading them with the following instead:
from gensim.models.fasttext import FastText
model = FastText.load('/content/sa/300/fasttext/sa-d300-m2-fasttext.model')
(Upon loading that main/root file, it will find the subsidiary related files in the same directory, as long as they're all present.)
The source where you downloaded these files should have included clear instructions for loading them nearby!
We have an EBCDIC Mainframe format file which is already loaded into Hadoop HDFS Sytem. The File has the Corresponding COBOL structure as well. We have to Read this file from HDFS, Convert the file data into ASCII format and need to split the data into Dataframe based on its COBOL Structure. I've tried some options which didn't seem to work. Could anyone please suggest us some proven or working ways.
For python, take a look at the Copybook package (https://github.com/zalmane/copybook). It supports most features of Copybook includes REDEFINES and OCCURS as well as a wide variety of PIC formats.
pip install copybook
root = copybook.parse_file('sample.cbl')
For parsing into a PySpark dataframe, you can use a flattened list of fields and use a UDF to parse based on the offsets:
offset_list = root.to_flat_list()
disclaimer : I am the maintainer of https://github.com/zalmane/copybook
Find the COBOL Language Reference manual and research functions DISPLAY-OF and National-Of. The link : https://www.ibm.com/support/pages/how-convert-ebcdic-ascii-or-ascii-ebcdic-cobol-program.
I've trained a net using Darkflow and now have the .pb files. I was wondering if it's possible (and if it is, how can it be done) to convert the files to .weight files for Darknet? I'd like to use Darknet with these files to classify images on Raspberry Pi.
I've been Googling but I see most people want to do the opposite.
Instead of weight file try .pb:
sudo ./darknet detector test cfg/coco.data cfg/yolov2.cfg <yourfile>.pb data/dog.jpg
In order to use the Latent semantic indexation method from gensim, I want to begin with a small "classique" example like :
import logging, gensim, bz2
id2word = gensim.corpora.Dictionary.load_from_text('wiki_en_wordids.txt')
mm = gensim.corpora.MmCorpus('wiki_en_tfidf.mm')
lsi = gensim.models.lsimodel.LsiModel(corpus=mm, id2word=id2word, num_topics=400)
etc..
My question is : How to get the corpus iterator 'wiki_en_tfidf.mm' ? Must I download it from somewhere ? I have searched on the Internet but I did not find anything. Help please ?
The first page of search results includes a link to:
https://radimrehurek.com/gensim/wiki.html
which says "First let’s load the corpus iterator and dictionary, created in the second step above."
Step 2 is
Convert the articles to plain text (process Wiki markup) and store the result as sparse TF-IDF vectors. In Python, this is easy to do
on-the-fly and we don’t even need to uncompress the whole archive to
disk. There is a script included in gensim that does just that, run:
$ python -m gensim.scripts.make_wiki
How could I read Garmin's .fit file on Linux. I'd like to use it for some data analysis but the file is a binary file.
I have visited http://garmin.kiesewetter.nl/ but the website does not seem to work.
Thanks
You can use GPSbabel to do this. It's a command-line tool, so you end up with something like:
gpsbabel -i garmin_fit -f {filename}.fit -o csv -F {output filename}.csv
and you'll get a text file with all the lat/long coordinates.
What's trickier is getting out other data, ie: if you want speed, time, or other information from the .fit file. You can easily get those into a .gpx, where they're in xml and human-readable, but I haven't yet found a single line solution for getting that data into a csv.
The company that created ANT made an SDK package available here:
https://www.thisisant.com/resources/fit
When unzipping this, there is a java/FitCSVTool.jar file. Then:
java -jar java/FitCSVTool.jar -b input.fit output.csv
I tested with a couple of files and it seems to work really well. Then of course the format of the csv can be a little bit complex.
For example, latitude and longitude are stored in semicircles, so it should be multiplied by 180/(2^31) to give GPS coordinates.
You need to convert the file to a .csv, the Garmin repair tool at http://garmin.kiesewetter.nl/ will do this for you. I've just loaded the site fine, try again it may have been temporarily down.
To add a little more detail:
"FIT or Flexible and Interoperable Data Transfer is a file format used for GPS tracks and routes. It is used by newer Garmin fitness GPS devices, including the Edge and Forerunner." From the OpenStreetMap Wiki http://wiki.openstreetmap.org/wiki/FIT
There are many tools to convert these files to other formats for different uses, which one you choose depends on the use. GPSBabel is another converer tool that may help. gpsbabel.org (I can't post two links yet :)
This page parses the file and lets you download it as tables. https://www.fitfileviewer.com/ The fun bit is converting the timestamps from numbers to readable timestamps Garmin .fit file timestamp