I start with describing the problem itself. Rather than a problem I'm looking for a better solution. I have a asp.net page which has a bunch of images and a link underneath it, Each image is infact the latest rendering of the link underneath it.
I scheduled a bat script which runs every hour to fetch the images through IECapt a web page rendering capture utility. One thing am annoyed about this utility is it takes a lot of time for the 20 images I have and for few because of the flash content it misses to take the actual screenshot of the website.
Now I like to know can this rendering be done by traditional programming am not interested in using any utilities. I'm interested in trying this. The solution need not be necessarily a C# based am ready to try in any other language. Because it gives me a chance to learn.
Thank you.
You should probably look at moz-headless-screenshot
You should be able to embed the functionality you need.
http://blog.mozilla.com/ted/2010/07/29/moz-headless-screenshot/
he also provided a sample embedding client application called moz-headless-screenshot.
This is a simple command line tool that takes a URL, image size, and output filename
and generates a PNG screenshot of the webpage.
You should look into browser shots:
http://browsershots.org/
They do what you want to do for lots of different browsers. It is even open source.
There's no simple-simple solution for what you're asking to do. This is because rendering HTML, CSS, and Flash is actually a very sophisticated process.
If you're up for quite a bit of coding, you can use the Gecko engine (which powers firefox) or another open-source web-browser core (ie Dillo) to render the page onto a custom canvas. Then save that canvas to a file. Unless you implement support for browser plug-ins, you won't get Flash this way, though. You could try using Gnash or its like. Good luck with that.
I don't know of an open-source project that already does this. It would be neat, though :-). If you write something, please push it to the world; it would be really cool to have a "get a screencap of this URL" tool.
One way is to use IRobotSoft web scraper. You can design a robot to go to the URL every hour, and capture the whole web page as an image via a function CapturePage(imagefile).
I am not sure if it will be better than IECapt though.
We have used ACA WebThumb ActiveX Control (http://www.acasystems.com/en/web-thumb-activex/) quite successfully to capture parts or whole of a web page in the web server and then to write them to a file, just passing in the url. It performs fast enough for our need.
I am not familiar with IECapt, but this might be something you might want to have a look at.
Related
I am trying to make a website similar to this and I'd like to know how this website was made. The website is also run on desktop and has a mobile app too so there's no HTML or anything in the back end.
Any ideas?
You can use Wappalyzer to identify the technologies used for this Website.
https://www.wappalyzer.com/
Result for https://realtimeboard.com/signup/
as mentioned you can use https://www.wappalyzer.com/ or http://builtwith.com to know the technologies used, further more you can view the webpage source or use the inspect element tool to have a better idea of what is going on. and ofcourse there is HTML used in that page whether it was automatically generated or manually
I was thinking about the applications of web scraping (still quite new to it) and came up with a question. Can you get an image from a page if there are advertisements on the page (like can you avoid advertisements and only look for the correct image content on the page)? Also, if the image is also a link to another page, can you say go to the next page and get that image (and then go from there until you either reach a certain amount or get all of the images)? This would mean avoiding going to the advertisements pages.
Absolutely. If you use a tool like kimonolabs.com this can be relatively easy. You click the data that you want on the page, so instead of getting all images including advertisements, Kimono uses the CSS selectors of the data you clicked to know which data to scrape.
You can use Kimono to scrape data within links as well. It's actually a very common use. Here's a break-down of that strategy: https://help.kimonolabs.com/hc/en-us/articles/203438300-Source-URLs-to-crawl-from-another-kimono-API
This might be a helpful solution for you, especially if you're not a programmer because it doesn't require coding experience. It's a pretty powerful tool.
I think if you are ok with PHP programming then give a look into php simple html dome parser. I have used it a lot and scrapped number of websites.
I have an April fools prank in mind and I will probably need some time to figure it out, so that's why I'm early:
I want to install an extension in the browser (needs to target botch FF and IE, so a cross browser implementation would be best) of my friend so that if he hits the news page he is visiting every morning, a fake image will make him think that he has been selected. The problem is, that I cannot just swap the images. I have to use the actual image from that website and put his head on the body of someone else.
I do have some graphics guys that can do that sort of stuff. My problem over here is to hook into the rendering process and do my own alterations to the image before it gets rendered. That is, taking the image, doing stuff to it and then passing in the modified image to the browser so that it is being redered instead of the original one.
Is that possible using extensions? If it may be only possible with eihter FF or IE, I might get him to use the browser of choice but I'd highly appreciate your suggestions, code snippets and starting points for research. Are there browser extensions that can do similar stuff?
Cheers everyone. There is a lot of reputation in that game so I don't care if it takes weeks or even months to complete the job.
Thank you guys, looking forward to suggestions!
Max
I think your intentions are harmless, but I must warn you: not all people may find your prank funny. But if you are sure that your friend will enjoy such a prank, it's really easy to set up.
You can use BFilter with site-specific filter to replace the image URL (it is very easy, just look at its examples and documentation). So when the user tries to open the web-page he will see your image instead of original picture. BFilter can be used as transparent proxy. I do not know how to setup transparent proxy in Windows, so you have to figure this part yourself. Alternatively, you can configure all installed browsers to use your filtering proxy.
You can use any other filtering proxy instead of BFilter.
Can anyone point me in the direction of such a script? It should also be able to work when called into another ajax window. This is the type of gallery i am going for:
http://dageniusmarketer.com/DigitalWonderland/pages/DemoGalleryExample.html
It should go on this page:
dageniusmarketer.com/DigitalWonderland/
Portfolio section.
This script should be real simple to use with minimal extra files to make it work. I also should be able to just drop images in a gallery folder and it populates the gallery automatically with thumbnails....I shouldnt have to write code for each image in my html. Should be all dynamic.
I also would like to know how I could go about a window effect where every time I open up a new section via my navigation, the window shrinks closed with the old content, then expands open with the new content. the window effect should be vertical (top to bottom shrink into center, expand from center top to bottom)
Please Let me know. Thanks
JQuery is one of my personal favorite javascript libraries (along with 99% of this site apparently!)
But it will have a learning curve, as your requirements seem pretty specific, and you will have to read some documentation to pull it off.
Try Spry from Adobe. They have a very similar demo. Also, the other common frameworks for this would be prototype/scriptaculous, dojo, mootools, jquery. In many cases they have extensions that would provide the exact thing you are looking for. For example, try
shadowbox extension which is framework agnostic. Best of luck!
Imago looks promising:
http://imago.codeboje.de/
Just discovered the very awesome-looking jQuery Tools library today. Meets your "simple and minimal" requirements and could probably pull off what you've sketched, with just the "tooltips" and "scrollable" components.
I also should be able to just drop images in a gallery folder and it populates the gallery automatically with thumbnails
My instinct is that you'd be better off writing server-side code to handle this part of your requirement.
So, I've been tasked with making a kiosk for the office for showing statistics about our SCRUM progress, build server status, rentability and so forth. It should ideally run a slideshow with bunch of different pages, some of them showing text, some showing graphs and so on.
What is the best approach for this? I first thought of powerpoint, but It should be able to take the images from a webserver so I can automate the graph generation procedure. I would also like to take text from an external source when showing "Who broke the build" or some page like that.
I have no doubt that ready-made systems exist, but I don't really know where to look for them.
Is this easy/hard in powerpoint? Or are there an ubiquous app that everybody but me knows about?
I would recommend creating it as a series of web-pages, which uses Javascript or the meta refresh tag to cycle though the different pages. Simply full-screen the browser on a spare machine, and connect it to a projector/monitor/big TV.
This has lots of benefits:
it's trivial to display images from an external server (an <img> tag)
it will cost nothing to setup (it can run on basically any functioning machine), and runs in a browser
it is quick to do (you do not have to worry about cross-browser compatibility, or different screen resolutions as you know the exact machine you are developing for
it's expandable - while what you describe is probably possible within Powerpoint, but if you do it as a web-page, you can use Javascript (or a JS framework like jQuery), and it's very easy to serve the pages via a web-server, then you can use any server-side scripting language.
Basically, you would have a series of files, say slide001.htm, slide002.htm and slide003.htm. Slide 1 would redirect to slide002 after 30 seconds, slide002 to slide 003, and slide003 would redirect to slide001..
The specific things you mention: graph generation and "Who broke the build" text:
Not sure which CI tool you use, but many of them generate graphs anyway, so that would be required is having one "slide" with something like <img src="http://hudson.abc/job/proj042/buildTimeGraph">
For the who-broke-the-build text, you would be easiest to run the slides as .php files served though a web-server, using XAMMP.
Then you would have a function that scrapes your CI server for whoever broke the last build, and in one of the slides, you would have <?PHP echo(who_broke_build()); ?>
(Obviously if you know some other language/system better, use that!)
The final benefit I can think of is that, if you serve the files through a web-server, you can allow people display it locally, say as their browsers home-page.
Thanks. I found jqS5 which did most of what you mentioned.
It requires 1 document where every h2 becomes a new slide.
I can then use the meta-refresh to reload to next page every 10 seconds. When I reach the end of the slides, I pull data from an aggregated RSS feed from all the different systems in order to pull information.
http://staticfree.info/projects/jqs5/