I am getting data from a broken RSS feed that gives me wrong link. I wanted to fix this link so I made this code:
<link.*>(.*)&.*tid(.*)</link>
and the link could be like:
www.somedomain.com/?value=50&burrrdurrrr;tid=120
But the real working link is in this form:
www.somedomain.com/?value=50&tid=120
The thing that I'm asking is if my measure thing looks like this:
[FeedURL]
Measure=Plugin
Plugin=Plugins\WebParser.dll
Url=[Feed]
StringIndex=2 ;now I only get www.somedomain.com/?value=50
Substitute=#SubstituteFeed#
How am I supposed to concatenate the strings together to complete the url?
I'm guessing rather than &burrrdurrrr;, the link has &, which is how you have to write & in an HTML or XML file.
If that's the case, you just need to set the DecodeCharacterReference option, as described in this handy-looking tutorial. Another option mentioned there is Substitute, which would be able to strip it out even if it really was &burrrdurrrr;.
None of this is a particularly sensible way of dealing with HTML or XML - a much better approach would be a plugin which actually parsed the document structure and let you reference nodes using XPath or CSS rules - but you work with what you've got, I guess. (I've never heard of this "Rainmeter" before, despite its claim to be "the best known and most popular desktop customization program for Windows"; maybe because nobody else calls their program that, instead almost universally using the word "widget"?)
What's a good way to parse HTML in AppleScript?
I haven't dabbled in AppleScript in quite some time, and even when I did it was very minimal and uninvolved, so I don't really think naturally in the language quite yet. But I need to do some string manipulation and parse some HTML (basically some simple screen scraping).
Naturally, I'd like to avoid common pitfalls of HTML parsing. However, this is a temporary script and doesn't need to be particularly robust or supportable. I really just need to scrape specific substrings (from a known starting substring to the next known character) into a file.
I've done plenty of string manipulation in C# and similar languages, but AppleScript is an interesting change of pace to say the least. Can somebody point me to some good resources (Google searches on this subject seem to have a high noise-to-signal ratio), or help me out with some sample code snippets?
The ultimate goal of what I'm doing is to take a pre-determined list of pages, open each one in Safari (I'm doing everything through tell application "Safari"), parse out links which fit a certain pattern, and store all of those links in a file. Then go through that file, open each of those links, parse out more links which fit another pattern, and store all of those links in a file.
(The site is actually owned by someone we're working with, so don't worry about me violating any terms of service or anything like that. But for reasons outside the scope of this question, I'm doing some page scraping in AppleScript.)
I can't say enough good things about Matt Neuburg's AppleScript: the Definitive Guide. Without a doubt the most complete documentation of AppleScript ever done. Matt's also one of my favorite tech writers.
I would also check out this article. It contains a tutorial on how to do this; the example provided there parses HTML data from only one source, but I think it's worth looking at.
What are the best algorithms for recognizing structured data on an HTML page?
For example Google will recognize the address of home/company in an email, and offers a map to this address.
A named-entity extraction framework such as GATE has at least tackled the information extraction problem for locations, assisted by a gazetteer of known places to help resolve common issues. Unless the pages were machine generated from a common source, you're going to find regular expressions a bit weak for the job.
If you have the markup proper—and not just the text from the page—I second the Beautiful Soup suggestion above. In particular, the address tag should provide the lowest of low-hanging fruit. Also look into the adr microformat. I'd only falll back to regexes if the first two didn't pull enough info or I didn't have the necessary data to look for the first two.
If you also have to handle international addresses, you're in for a world of headaches; international address formats are amazingly varied.
I'd guess that Google takes a two step approach to the problem (at least that's what I would do). First they use some fairly general search pattern to pick out everything that could be an address, and then they use their map database to look up that string and see if they get any matches. If they do it's probably an address if they don't it probably isn't. If you can use a map database in your code that will probably make your life easier.
Unless you can limit the geographic location of the addresses, I'm guessing that it's pretty much impossible to identify a string as an address just by parsing it, simply due to the huge variation of address formats used around the world.
Do not use regular expressions. Use an existing HTML parser, for example in Python I strongly recommend BeautifulSoup. Even if you use a regular expression to parse the HTML elements BeautifulSoup grabs.
If you do it with your own regexs, you not only have to worry about finding the data you require, you have to worry about things like invalid HTML, and lots of other very non-obvious problems you'll stumble over..
What you're asking is really quite a hard problem if you want to get it perfect. While a simple regexp will get it mostly right most of them time, writing one that will get it exactly right everytime is fiendishly hard. There are plenty of strange corner cases and in several cases there is no single unambiguous answer. Most web sites that I've seen to a pretty bad job handling all but the simplest URLs.
If you want to go down the regexp route your best bet is probably to check out the sourcecode of
http://metacpan.org/pod/Regexp::Common::URI::http
Again, regular expressions should do the trick.
Because of the wide variety of addresses, you can only guess if a string is an address or not by an expression like "(number), (name) Street|Boulevard|Main", etc
You can consider looking into some firefox extensions which aim to map addresses found in text to see how they work
You can check this USA extraction example http://code.google.com/p/graph-expression/wiki/USAAddressExtraction
It depends upon your requirement.
for email and contact details regex is more than enough.
For addresses regex alone will not help. Think about NLP(NER) & POS tagging.
For finding people related information you cant do anything without NER.
If you need information like paragraphs get the contents by using tags.
Some of my colleagues use special comments on their bug fixes, for example:
// 2008-09-23 John Doe - bug 12345
// <short description>
Does this make sense?
Do you comment bug fixes in a special way?
Please let me know.
I don't put in comments like that, the source control system already maintains that history and I am already able to log the history of a file.
I do put in comments that describe why something non-obvious is being done though. So if the bug fix makes the code less predictable and clear, then I explain why.
Over time these can accumulate and add clutter. It's better to make the code clear, add any comments for related gotchas that may not be obvious and keep the bug detail in the tracking system and repository.
I tend not to comment in the actual source because it can be difficult to keep up to date.
However I do put linking comments in my source control log and issue tracker. e.g. I might do something like this in Perforce:
[Bug-Id] Problem with xyz dialog.
Moved sizing code to abc and now
initialise later.
Then in my issue tracker I will do something like:
Fixed in changelist 1234.
Moved sizing code to abc and now
initialise later.
Because then a good historic marker is left. Also it makes it easy if you want to know why a particular line of code is a certain way, you can just look at the file history. Once you've found the line of code, you can read my commit comment and clearly see which bug it was for and how I fixed it.
Only if the solution was particularly clever or hard to understand.
I usually add my name, my e-mail address and the date along with a short description of what I changed, That's because as a consultant I often fix other people's code.
// Glenn F. Henriksen (<email#company.no) - 2008-09-23
// <Short description>
That way the code owners, or the people coming in after me, can figure out what happened and they can get in touch with me if they have to.
(yes, unfortunately, more often than not they have no source control... for internal stuff I use TFS tracking)
While this may seem like a good idea at the time, it quickly gets out of hand. Such information can be better captured using a good combination of source control system and bug tracker. Of course, if there's something tricky going on, a comment describing the situation would be helpful in any case, but not the date, name, or bug number.
The code base I'm currently working on at work is something like 20 years old and they seem to have added lots of comments like this years ago. Fortunately, they stopped doing it a few years after they converted everything to CVS in the late 90s. However, such comments are still littered throughout the code and the policy now is "remove them if you're working directly on that code, but otherwise leave them". They're often really hard to follow especially if the same code is added and removed several times (yes, it happens). They also don't contain the date, but contain the bug number which you'd have to go look up in an archaic system to find the date, so nobody does.
Comments like this are why Subversion lets you type a log entry on every commit. That's where you should put this stuff, not in the code.
I do it if the bug fix involves something that's not straightforward, but more often than not if the bugfix requires a long explanation I take it as a sign that the fix wasn't designed well. Occasionally I have to work around a public interface that can't change so this tends to be the source of these kinds of comments, for example:
// <date> [my name] - Bug xxxxx happens when the foo parameter is null, but
// some customers want the behavior. Jump through some hoops to find a default value.
In other cases the source control commit message is what I use to annotate the change.
Whilst I do tend to see some comments on bugs inside the code at work, my personal preference is linking a code commit to one bug. When I say one I really mean one bug. Afterwards you can always look at the changes made and know which bug these were applied to.
That style of commenting is extremely valuable in a multi-developer environment where there is a range of skills and / or business knowledge across the developers (e.g. - everywhere).
To the experienced knowledgable developer the reason for a change may be obvious, but for newer developers that comment will make them think twice and do more investigation before messing with it. It also helps them learn more about how the system works.
Oh, and a note from experience about the "I just put that in the source control system" comments:
If it isn't in the source, it didn't happen.
I can't count the number of times the source history for projects has been lost due to inexperience with the source control software, improper branching models etc. There is
only one place the change history cannot be lost - and that's in the source file.
I usually put it there first, then cut 'n paste the same comment when I check it in.
No I don't, and I hate having graffiti like that litter the code. Bug numbers can be tracked in the commit message to the version control system, and by scripts to push relevant commit messages into the bug tracking system. I do not believe they belong in the source code, where future edits will just confuse things.
Often a comment like that is more confusing, as you don't really have context as to what the original code looked like, or the original bad behavior.
In general, if your bug fix now makes the code run CORRECTLY, just simply leave it without comments. There is no need to comment correct code.
Sometimes the bug fix makes things look odd, or the bug fix is testing for something that is out of the ordinary. Then it might be appropriate to have a comment - usually the comment should refer back to the "bug number" from your bug database. For example, you might have a comment that says "Bug 123 - Account for odd behavior when the user is in 640 by 480 screen resolution".
If you add comments like that after a few years of maintaining the code you will have so many bug fix comments you wouldn't be able to read the code.
But if you change something that look right (but have a subtle bug) into something that is more complicated it's nice to add a short comment explaining what you did, so that the next programmer to maintain this code doesn't change it back because he (or she) thinks you over-complicated things for no good reason.
No. I use subversion and always enter a description of my motivation for committing a change. I typically don't restate the solution in English, instead I summarize the changes made.
I have worked on a number of projects where they put comments in the code when bug fixes were made. Interestingly, and probably not coincidentally, these were projects which either didn't use any sort of source control tool or were mandated to follow this sort of convention by fiat from management.
Quite honestly, I don't really see the value in doing this for most situations. If I want to know what changed, I'll look at the subversion log and the diff.
Just my two cents.
If the code is corrected, the comment is useless and never interesting to anybody - just noise.
If the bug isn't solved, the comment is wrong. Then it makes sense. :) So just leave such comments if you didn't really solved the bug.
To locate ones specific comment we use DKBUGBUG - which means David Kelley's fix and reviewer can easily identity, Ofcourse we will add Date and other VSTS bug tracking number etc along with this.
Don't duplicate meta data that your VCS is going to keep for you. Dates and names should be in the automatically added by the VCS. Ticket numbers, manager/user names that requested the change, etc should be in VCS comments, not the code.
Rather than this:
//$DATE $NAME $TICKET
//useful comment to the next poor soul
I would do this:
//useful comment to the next poor soul
If the code is on a live platform, away from direct access to the source control repository, then I will add comments to highlight the changes made as a part of the fix for a bug on the live system.
Otherwise, no the message that you enter at checkin should contain all the info you need.
cheers,
Rob
When I make bugfixes/enhancements in third party libraries/component I often make some comments. This makes it easier find and move the changes if I need to use a newer version of the library/component.
In my own code I seldom comments bugfixes.
I don't work on multi-person projects, but I sometimes add comments about a certain bug to a unit test.
Remember, there's no such thing as bugs, just insufficient testing.
Since I do as much TDD as possible (everything else is social suicide, because every other method will force you to work endless hours), I seldomly fix bugs.
Most of the time I add special remarks like this one to the code:
// I KNOW this may look strange to you, but I have to use
// this special implementation here - if you don't understand that,
// maybe you are the wrong person for the job.
Sounds harsh, but most people who call themselves "developers" deserve no other remarks.