Dears,
I need your help , i need to create a product this product will make the user enter a domain/email and the system will return the results that are related to the value that is entered by a user.
My product/platform will have more than 5000 Text files in multi-format content (i.e some files the content will be line by line, others will be comma by comma ..etc) and I have more than 1000 .SQL files (Database files).
The expected size for the files (txt & SQL ) is approx 5 terabytes.
So what is your recommendation, what I should use in my product? i.e how can I do a collective with my all files?
Must use a Syslog/Siem-like elastic search/OpenSearch to do that? or something else.
Please advise me.
Related
I have a internal webpage that makes data from excel searchable and readable from a 3rd party excel export file. The webpage allows for the uploading of multiple excel files in which the data gets read and stored in a MySQL database.
We want to update the application to keep a history of the uploaded data (it's data that has monthly values) so we can easily search, filter and generate graphs from the uploaded data.
So I am using Laravel 5.4 and have maatwebsite\excel to import and parse the excel file.
The Excel file always consists of the following columns (Dummy File)
| Item group | item # | item name | Item Currency | <month> <year> |
After Item Currency there is always 36 columns for the past 3 years of data from the current month so a column would be named like dec 2017
Now in Laravel I have created a Model for the item named Item and a model for the monthly values named ItemMonthly
Now I am able to read the file and create columns dynamically in the database but I feel like this is very ugly and not efficient at all:
(Gist) Code for Models and Excel Function
Biggest problem
Because I need to read all the monthly data and since I need them in order of month I can't really rename all the columns as far as I know. I need to be able to get all the columns to render in a Highchart graph and in a Datatable. and some items don't have the same monthly data (some only go till 2015 for example.
Needed advice
I've read a couple of solutions here some of them saying instead of creating columns in MySQL just store the monthly data as a json object in a single column.
Some answers just completely advice on changing from MySQL to MongoDB
I am kind of at a loss to find the best approach for this, and am sincerely wondering if MySQL is the right way to go. The solutions I have been trying so far all seem to involve really hacky ways of doing this.
If there is more info needed please let me know. I don't want to write an immense wall of text but I also want to provide the correct amount of information.
Many thanks!
Some questions arises when using MAGMI generic SQL datasource. Magmi 0.7.18 displays the following input information when using that plugin:
I tried several times with two approaches.
-the first one as described on the image was a direct query to the mysql database containing all rows to feed magento database using magmi. (no files on genericsql/requests)
-the second one using .sql file exported from my database and placing that file into (genericsql/requests).
in both cases I received the following statement: 1 warning(s) found Hide Details -> "No Records returned by datasource"
I read that some folks suggest to use input DB Initial Statement: SELECT COUNT(*) AS cnt FROM tablename but in my case it was the same.
question one: using mysql I can query mysql databse directy using the input db information (type, host, name, user, password) or I have to place the sql file in genericsql/requests too? is that my error?
question two: based on the fact that mysql cannot attach files - as MS sql can - which information do I have to place when magmi request user- pass for that sql file?
any help appreciated, I´m stuck with this issue and CSV is not suitable for my needs. brgds
by dweeves:
Your SELECT has to be put in a .sql file in the directory listed in red. (name it as you want as long as it ends with .sql extension)
The "Initial Statements" is a field that is meant to hold the "connection" time statements (like SET NAMES 'UTF8').
For the "quick count" , you might also add a .sql.count file in the same dir with the same name that the request you want to achieve.
By default magmi will find the count using a
SELECT COUNT(*) FROM (your request here)
see Generic SQL Datasource plugin documentation.
I have installed and made some successful product imports in to Magento using Magmi, but as soon as I try to import any data where the spreedsheet columns have commas [,] Magmi will not perform the import.
For example when I save the data in this speadsheet as a CSV file Magmi successfully imports the data;
http://i.imgur.com/PpDt0PS.png
However, Magmi refuses to import the data in the table below, where you can see in column F I have added data that include 'commas'.
http://i.imgur.com/MtGJPCw.png
Can anyone advise. I am using an Apple Mac with OpenOffice to prepare and save my data.
Is the data not importing entirely, or is just the visibility column not being set?
Visibility is a Magento core attribute which Magmi can set by using exact numerical option id value.
Generally, the option values you want to use for the visibility field are as follows:
Not Visible Individually = 1
Catalog = 2
Search = 3
Catalog, Search = 4
So in your case, if you want to set these products to Catalog, Search, you can set the visibility column value to 4.
To double-check that the above mapping is correct for your instance of Magento, the easiest way is as follows:
Go edit any product
Look for the Visibility drop down field, and right click > inspect element
In the developer tools, take note the values associated to each label.
Below an example of the process and what to look for.
Axel is correct, you should set the data to the numerical value 4.
But I do also recommend you explore a better way to export CSV content from Open Office. You may have to start a new document because I find I only see the dialogue below once and then I never see it again. Create a new document, paste your data into it. Choose save-as, select CSV, and save it. Eventually you should see the dialogue below. Change the encoding to UTF 8, the text delimiter to " and tick the 'Quote all text cells' box.
Then you should be able to have any cells with commas or other things in them. Always ensure you CSV files are quoted. "like","this","so you, can","have commas, in them". It is worth inspecting your CSV file in a text editor to see the format is as expected before uploading it to MAGMI.
I have one field when importing that can contain large data, it seems that CSV has unofficial limitation of about 65000 (likely 65535*) character. as both libreoffice calc and magento truncating the data for that particular field. I have investigated well and I'm certain it is not because of a special character or quotes. the data pretty straight forward, the lines are similar in format to each other.
Question: How to increase that size? or at least where I should look to find it?
Note: I counted in libreoffice writer and it was about 65040. but probably with carriage return characters it could reach 65535
I change:
1) in table catalog_category_entity_text
type of field "value" from "text" to "longtext"
2) in file app/code/core/Mage/ImportExport/Model/Import/Entity/Abstract.php
const DB_MAX_TEXT_LENGTH = 65536;
to
const DB_MAX_TEXT_LENGTH = 16777215;
and all OK
You are right, there is a limitation in Magento, because it sets texts fields as TEXT in MySQL database and, according to MySQL docs, this kind of field supports a maximum of 65535 chars.
http://dev.mysql.com/doc/refman/5.0/es/storage-requirements.html
So you could change the column type in your Magento database to use MEDIUMTEXT. I guess the correct place is in the catalog_product_entity_text table, where you should modify the 'value' field type to match your needs. But please, keep in mind this is dangerous. Make a full backup before trying. And you may even need to play with core files... not recommended!
I'm having the same issue with 8 products from a list of more than 400, and I think I'm not going to mess with Magento core and database, we can reduce the description strings for those few products.
The CSV could care less. Due to Microsoft Access allowing Memo fields which can contain quite a bit of data, I've exported 2-3k descriptions in CSV format to be imported into Magento quite successfully.
Your limitation is either because you are using a spreadsheet that has a cell limitation or export limitation on cells or because the field you are trying to import into has a maximum character limitation set in its table for that field.
You can determine the latter by using phpMyAdmin to see what the maximum character setting is for that field.
I have got ASCII files and want to convert them into maybe excel or tab/csv delimited text file. The file is a table with field name and field attributes. It also includes index name, table name and field(s) to index if required depending on the software. I don't think it is necessary to think of this. Well, field name and field attributes are enough, I hope so. I just want the information hidden inside. Can you all experts help me to get this done.
The lines are something like this:
10000001$"WORD" WORD$10001890$$$$495.7$$$N$$
10000002$11-word-word word$10000002$$$$$$$Y$$
10000003$11-word word word$10033315$0413004$$$$$$N$$
10000004$11-word word word$10033315$$$$$$$Y$017701$
The general answer, before knowing your ascii file in details, operating system, and so on, would be:
1 - cut the top n-lines, that containg the information you don't want. Leave the filds names, if you want to.
2 - check if the fields are separated by a common character, for example, one comma ,
3 - import the file inside a spreadsheet program, like Excel or OpenOffice Calc. In OOCalc, choose to import the file, then select the correct separating character
that's all.