I would like to download all property listings from a RETS server, including all photo URLs. I'm using DMQL2 and the PHRETS library. Properties and photo objects are stored in the RETS server in different tables.
To get all the photos, I know I can download the list of properties, then loop through each and retrieve the photos for each property like this:
$results = $rets->Search($resource, $class, $query);
foreach ($results as $r) {
$photos = $rets->GetObject('Property', 'Photo', $r->get('ListingID'), '*', 1);
foreach ($photos as $p) {
// Save the photo locations somewhere…
}
}
This is incredibly slow because there are many thousands of properties.
Is it possible to request all photos along with properties in a single query by joining the property and object tables (like a LEFT JOIN in MySQL)?
Or, is there a way to download all photo objects in one request, so I can correlate them to properties using their ListingID key?
Any other suggestions for getting all the data more quickly?
Partially possible, but then currently can't download entire photos along with properties in a single query.
We can download multiple listing's images in a single query. Check the sample query below.
$photos = $rets->GetObject("Property", "Photo", "12345,12346,12347", "*", 1);
foreach ($photos as $p) {
$listingId = $p['Content-ID']; // save photo based on each listingIds
//continue
}
Here 3rd parameter we can provide comma separated listingIds. Thereby you can pass the listingIds as batches.
Note: Still MLS holds the extraction speed. Some MLS boards may increase the bandwidth temporarily on request. Within that time try to extract full images.
You can download all the photos at once. Some servers also support media resources and classes that allow you to get the URLs to all the photos for a listing or group of listings. If your server supports that, I would use it because it's typically faster.
Using phrets, I think downloading all the images at one time looks like this:
$photos = $rets->GetObject("Property", "Photo", $record[ListingID], "*", 1);
I copied that from this SO answer: PHRets: Using PHP to download real estate listing photos
Related
I am querying a large data sets from the table and then iterating through a loop for creating a json file.
$user = App\User::all();
foreach($user as $val){
// logic goes here for creating the json file
}
Now the problem i am facing is that when iterating through the loop it is consuming memory and i am getting error 'Allowed memory size exhausted'.And also the cpu usage of the server becomng so high.
My question how i should use the laravel lazy collections to get rid of this issue.I have gone through the offcial docs but couldnt find the way.
Just replace the all method with the cursor one.
$user = App\User::cursor();
foreach($user as $val){
// logic goes here for creating the json file
}
For more informations about the methods you can chain, refer to the official documentation
I am loading data from excel. In foreach I am checking for each record if it does exist in database:
$recordExists = $this->checkIfExists($record);
function checkIfExists($record) {
$foundRecord = $this->repository->newQuery()
->where(..., $record[...])
->where(..., $record[...])
...
->get();
}
When the excel contains up to 1000 values which is relatively small piece of data - the code runs around 2 minutes. I am guessing this is very inefficient way to do it.
I was thinking of passing the array of loaded data to the method checkIfExists but then I could not query on the data.
What would be a way to proceed?
You can use laravel queue if you want to do a lot of work within a very short time. Your code will run on backend. Client can not recognize the process. just show a message to client that this process is under queue. Thats it
You can check the Official Documentation From Below Url
https://laravel.com/docs/5.8/queues
If you passes all the data from the database to the function (so no more queries to the database), you can use laravel collections functions to filter.
On of them is where => https://laravel.com/docs/5.8/collections#method-where
function checkIfExists($record, Collection $fetchedDataFromDatabase) {
// laravel collectons 'where' function
$foundRecord = $fetchedDataFromDatabase
->where(..., $record[...])
->where(..., $record[...]);
}
other helpful functions.
filter
contains
I am generating images on fly from DICOM files using:
public ActionResult Generatemage()
{
FileContentResult data;
.....
objImage = im.Bitmap(outputSize, PixelFormat.Format24bppRgb, m);
using (var memStream = new MemoryStream())
{
objImage.Save(memStream, ImageFormat.Png);
data = this.File(memStream.GetBuffer(), "image/png");
}
return data;
}
Can I store the image as a session variable so I can modify it using Point3D?
I tried to use:
Bitmap data = (Bitmap)Session["newimage"];
Got these two errors:
Cannot implicitly convert type 'System.Drawing.Bitmap' to 'System.Web.Mvc.FileContentResult' and
A local variable named 'data' is already defined in this scope
I would appreciate your suggestions, thanks in advance.
Can I store the image as a session variable so I can modify it using
Point3D?
I suggest to not do that. If you have not read Nathanael's post on image resizing pitfalls then I suggest you do so now. It may be talking about resizing but it also give hints on working with images in general. On point #3 it says:
Serving a file from disk by loading it into memory. Think about how
much RAM your server has, how large a single image is, how long it has
to stay in memory before users fi downloading it, and how many
users you have requesting images.
In your particular case you can replace "before users finish downloading it" with "before Point3D finish processing the image". So, what I suggest is that you get a handle to that file, say maybe there's an Id that uniquely identifies a file per user, use that Id to retrieve the file when it's time to process it with Point3D, load it into a MemoryStream (assuming Point3D can work with mem. stream), process it, then dispose of it. In that manner you are only holding on to the image for the duration of "Point3D processing".
Cannot implicitly convert type 'System.Drawing.Bitmap' to
'System.Web.Mvc.FileContentResult' and A local variable named 'data'
is already defined in this scope
That is most probably because you have defined data as such:
FileContentResult data;
and then you are doing a:
Bitmap data = (Bitmap)Session["newimage"];
same variable of two different types within the same scope.
I'm working with Magento CE 1.6 in a project where we need an easy way to fill the movie info in DVDs and Blu-Rays products for a reseller ecommerce. I'm set to use the Rotten Tomatoes API wich seems very adequate for our purposes, but here's the thing: We don't want to have to input every single detail of the movie in the New Product dialog, oppositely, we want to fetch the info automatically using the movie name as hint (the API perfectly supports this). I though that we could achieve this by two means:
Having the administrator to enter only the names of the movies and
create and run periodically a script that fetches the rest of the
info with the API and updates the data directly in the DB. I've been
watching the DB changes when a product is saved and would'nt like to
do that.
Editing the Magento code to make the new product form auto fillable,
maybe with ajax, once a movie name is entered. Zend framework isn't
my strong and seems kind of hard too.
Am I seeing this problem from the rigth angle? Is there maybe another API? Or a Magento extension? Or another ecommerce?!
I would suggest a little different approach. Enhancing the Admin interface is difficult, but possible. Here is an easier way.
Method #1 - Quick and Easy
Create yourself a script that would go through a list of products. You can select them by attribute type, category, or even just select them all! Then, loop through that collection, and for each product, grab the title, query the movie API, and set the product's attributes. Then, save the product, and move to the next one. Something like this:
Note: Be sure to create your custom attributes in the admin and assign them to the attribute set.
<?php
require_once 'app/Mage.php';
umask(0);
Mage::app('default');
function getVideoDataFromAPI($title)
{
// get your data from the API here...
return $data;
}
$collection = Mage::getResourceModel('catalog/product_collection')
->addAttributeToFilter('attribute_set_id', $yourAttributeSetId)
->addAttributeToFilter('year', ''); // <-- Set a field here that will be empty by default, and filled by the API. This is '' because it won't be null.
foreach ( $collection->getAllIds() as $id ) {
$product = Mage::getModel('catalog/product')->load($id);
$videoData = getVideoDataFromAPI($product->getName());
if ( empty($videoData) ) { continue; }
$product->setYear($videoData['year'])
->setRating($videoData['rating'])
->save();
}
?>
Method #2 - Do the above, but in a custom extension
I always like extensions over scripts. They are more secure and more powerful. With an extension you could have an admin list of the products, can filter them how ever you would like, and have a mass action to pull the video data manually. You could also set it up on a cron job to pull regularly.
So, I'm not quite sure how I should structure this in CakePHP to work correctly in the proper MVC form.
Let's, for argument sake, say I have the following data structure which are related in various ways:
Team
Task
Equipment
This is generally how sites are and is quite easy to structure and make in Cake. For example, I would have the a model, controller and view for each item set.
My problem (and I'm sure countless others have had it and already solved it) is that I have a level above the item sets. So, for example:
Department
Team
Task
Equipment
Department
Team
Task
Equipment
Department
Team
Task
Equipment
In my site, I need the ability for someone to view the site at an individual group level as well as move to view it all together (ie, ignore the groups).
So, I have models, views and controls for Depart, Team, Task and Equipment.
How do I structure my site so that from the Department view, someone can select a Department then move around the site to the different views for Team/Task/Equipment showing only those that belong to that particular Department.
In this same format, is there a way to also move around ignoring the department associations?
Hopefully the following example URLs clarifies anything that was unclear:
// View items while disregarding which group-set record they belong to
http://www.example.com/Team/action/id
http://www.example.com/Task/action/id
http://www.example.com/Equipment/action/id
http://www.example.com/Departments
// View items as if only those associated with the selected group-set record exist
http://www.example.com/Department/HR/Team/action/id
http://www.example.com/Department/HR/Task/action/id
http://www.example.com/Department/HR/Equipment/action/id
Can I get the controllers to function in this manner? Is there someone to read so I can figure this out?
Thanks to those that read all this :)
I think I know what you're trying to do. Correct me if I'm wrong:
I built a project manager for myself in which I wanted the URLs to be more logical, so instead of using something like
http://domain.com/project/milestones/add/MyProjectName I could use
http://domain.com/project/MyProjectName/milestones/add
I added a custom route to the end (!important) of my routes so that it catches anything that's not already a route and treats it as a "variable route".
Router::connect('/project/:project/:controller/:action/*', array(), array('project' => '[a-zA-Z0-9\-]+'));
Whatever route you put means that you can't already (or ever) have a controller by that name, for that reason I consider it a good practice to use a singular word instead of a plural. (I have a Projects Controller, so I use "project" to avoid conflicting with it.)
Now, to access the :project parameter anywhere in my app, I use this function in my AppController:
function __currentProject(){
// Finding the current Project's Info
if(isset($this->params['project'])){
App::import('Model', 'Project');
$projectNames = new Project;
$projectNames->contain();
$projectInfo = $projectNames->find('first', array('conditions' => array('Project.slug' => $this->params['project'])));
$project_id = $projectInfo['Project']['id'];
$this->set('project_name_for_layout', $projectInfo['Project']['name']);
return $project_id;
}
}
And I utilize it in my other controllers:
function overview(){
$this->layout = 'project';
// Getting currentProject id from App Controller
$project_id = parent::__currentProject();
// Finding out what time it is and performing queries based on time.
$nowStamp = time();
$nowDate = date('Y-m-d H:i:s' , $nowStamp);
$twoWeeksFromNow = $nowDate + 1209600;
$lateMilestones = $this->Project->Milestone->find('all', array('conditions'=>array('Milestone.project_id' => $project_id, 'Milestone.complete'=> 0, 'Milestone.duedate <'=> $nowDate)));
$this->set(compact('lateMilestones'));
$currentProject = $this->Project->find('all', array('conditions'=>array('Project.slug' => $this->params['project'])));
$this->set(compact('currentProject'));
}
For your project you can try using a route like this at the end of your routes.php file:
Router::connect('/:groupname/:controller/:action/*', array(), array('groupname' => '[a-zA-Z0-9\-]+'));
// Notice I removed "/project" from the beginning. If you put the :groupname first, as I've done in the last example, then you only have one option for these custom url routes.
Then modify the other code to your needs.
If this is a public site, you may want to consider using named variables. This will allow you to define the group on the URL still, but without additional functionality requirements.
http://example.com/team/group:hr
http://example.com/team/action/group:hr/other:var
It may require custom routes too... but it should do the job.
http://book.cakephp.org/view/541/Named-parameters
http://book.cakephp.org/view/542/Defining-Routes
SESSIONS
Since web is stateless, you will need to use sessions (or cookies). The question you will need to ask yourself is how to reflect the selection (or not) of a specific department. It could be as simple as putting a drop down selection in the upper right that reflects ALL, HR, Sales, etc. When the drop down changes, it will set (or clear) the Group session variable.
As for the functionality in the controllers, you just check for the Session. If it is there, you limit the data by the select group. So you would use the same URLs, but the controller or model would manage how the data gets displayed.
// for all functionality use:
http://www.example.com/Team/action/id
http://www.example.com/Task/action/id
http://www.example.com/Equipment/action/id
You don't change the URL to accommodate for the functionality. That would be like using a different URL for every USER wanting to see their ADDRESS, PHONE NUMBER, or BILLING INFO. Where USER would be the group and ADDRESS, PHONE NUMBER< and BILLING INFO would be the item sets.
WITHOUT SESSIONS
The other option would be to put the Group filter on each page. So for example on Team/index view you would have a group drop down to filter the data. It would accomplish the same thing without having to set and clear session variables.
The conclusion is and the key thing to remember is that the functionality does not change nor does the URLs. The only thing that changes is that you will be working with filtered data sets.
Does that make sense?