This may be a question for Survey Monkey, but I felt that someone here may have encountered something like this in past experiences. Is there a way to work with the API of Survey Monkey (SM), to add the information from the survey straight into a database of my own? I realize that I can generate the information into output files, but I was wondering if there was a way to directly access the information from the SM database. I feel like this might cause some privacy concerns for SM. Has anyone attempted this, or would the best option of mine be to create my own surveys without a third party website?
I had a similar issue and here's my solution.
I was doing health related surveys which contain HIPPA protected Personal Health Info. Zapier is NOT HIPAA safe, so the "zap the results over to Google Drive" solution didn't work.
So I wanted a quick n dirty way to grab SM survey data and begin to design a data structure to analyse and store this data. I figured that I would start with <1000 results, sort it out, then build out a bigger/fancier structure as needed.
I just downloaded CSV's of the SM individual responses, munged the downloaded CSV files to make a Python CSV reader happy, then wrote a Python 3.5 script to grab the survey data and spit it out into a couple of output CSV files designed for different analytic purposes.
It was really quick and easy to alter the Python script to deliver different subsets of data to different output files, and really quick and easy to see if these output (CSV or XLS) files really told me what I wanted to know.
This is a really quick and easy way to start analysing right away without spending too much time on procedural overhead. You can alter CSV (or XLS ) tables really quickly and easily, so you can mix and match data / derivative data as much as you want. A wise person once told me "don't think, do." So the more you analyse on small runs of data, the better your final Big Buildout In The Sky will look.
Yah, you can spend a lot of time writing and API and setting up a dbase, but if you are not completely happy with what you want out of the SM data, start small. Hope this helps.
Related
I know basic SQL, and SQL is all I know when it comes to storing and retrieving data. I want to create 1 .exe and it should contain all ~100,000 key-value pairs (i have the data in .txt files) and maybe an extra attribute for description (this I would add myself - like a note to myself).
I also would like to write it in a new language I don't know yet; like python or C# (I have made desktop apps written in Java & VB.net all with SQL databases). So language will not be an issue and I would appreciate suggestions.
These key-value pairs might not need to be updated and I'm willing to re-compile/repackage the code to make 1 change in the data. The key is 6 letters long and 2 numbers at the end like hxnaaa01. Each of these letters represent or describe something about itself so I would also need to search for a specific letter on a specific position to get exactly what I need.
I know that regex would work well with what I need but all I mentioned is all I know. I don't know enough and I don't know what keywords to google.
I have read about XML and CSV. I don't really know what they are and I'm not sure how all of this would fit in 1 executable.
To summarize, I need:
1 executable (Windows Desktop App)
Search function ~100k KVP+1more attribute (using regex?)
no database
with GUI
ability to add a "note" to each KVP
should be fast and lightweight
1 executable (Windows Desktop App), no database
Data persistence will require either additional files, or a database. It's pretty much unavoidable, you can store data in memory, but it's only persisted for as long as it resides there.
You have another requirement: "fast and lightweight".
To achieve this requirement, you'll need to really think about your solution, what technology you use and how you can improve it in future.
Although searching through data is pretty trivial, an efficient solution is not. It requires upfront research into algorithms, data structures and general practices. (which is a rabbit hole itself).
In the case of JSON [1], you'll need to create an additional file to contain all your key/value pairs, you can use C# to create the extra file (on first launch, for example).
JSON promises to be lightweight, I tend to agree, some may not. When dealing with the filesystem, I think it can be agreed is often far from lightweight solution.
JSON is very readable though:
{
"key": "value",
"comment": "oh this is cool"
}
There's a lot of factors that play into something being fast and lightweight, so there's a need for some research on your part.
Honestly, depending on your experience, I wouldn't focus so much on the fast, I'd focus more on it working, then refactor that into something that's fast if it's too slow. [2]
And again, depending on your experience, I'd stick to opening the file, using a for/loop to find my key and do something with the data found, plus reward myself for having something that works.
TL;DR: you need either a file, or database for truly persistent storage, JSON or a remotely hosted MySQL would work. Try not to focus too much on fast before you have something that works.
https://www.json.org/json-en.html [1]
https://stackoverflow.com/a/5581595/2932298
https://stackify.com/premature-optimization-evil/ [2]
first question I've asked and I'm not sure how to ask it clearly, or if there will be an answer that I want to hear ;)
tl;dr: "I want to import a file into my application at work but I don't know the input format. How can I discover it?"
Forgive any pending wordiness and/or redaction.
In my work I depend on an unsupported (and proprietary) application written in Pascal. I have no experience with pascal (yet...) and naturally have no source code access. It is an excellent (and very secret/NDA sort of deal I think) application that allows us to deal with inventory and financial issues in my employer's organization. It is quite feature-comprehensive, reasonably stable and robust, and kind of foistered (word?) on us by a higher power.
One excellent feature that it has is the ability to load up "schedules" into our corporate system. This feature should be saving us hundreds of hours in data entry.
But it isn't.
The problem is, the schedules we receive are written in a legacy format intended for human eyes. The "new" system can't interpret them.
Our current information (which I have to read and then re-enter into the database by hand) is send in a sort of rich-text flat-file format, which would be easy to parse with the string library of probably any mainstream language.
So I want to write a converter to convert our data into a format that the new software can interpret.
By feeding certain assorted files into the system, I have learned a little bit about what kind of file it expects:
I "import" a zero-byte file. Nothing happens (same as printing a report with no data)
I "import" an XML file that I guess might look like the system expects. It responds with an exception dialog and a stacktrace. Apparently the string <?xml contains illegal characters or something
I "import" a jpeg image -- similar result to #2.
So I think that my target wants a flat-file itself. The file would need to contain a "document number" along with {entries with "incident IDs" and descriptions and numeric values}.
But I don't know this for certain.
Nobody is able to tell me exactly what these files should look like. Someone in the know said that they have seen the feature demonstrated -- somewhere out there is a utility that creates my importable schedules. But for now, the utility is lost and I am on my own.
What methods can I use to figure out the input file format? I know nothing about debugging pascal, but I assume that that is probably my best bet. Or do I have to keep on with brute force until I can afford a million monkey-operated typewriters? Do I have to decompile the target application? I don't know if I can get away with that, let alone read the decompiled source.
My google-fu has failed me.
Has anyone done something like this before or could they point me in the right direction? Are there any guides on this subject?
Thanks in advance.
PS: I am certain that I am not breaking any laws at this point, although I will have to check to find out if decompilation would get me into trouble or not, and that might be outside of my technical competence anyway.
If you have an example file you can try to take a hexdump utility and try to see if there things you can identify. Any additional info that you have (what should in the file) helps with that. Better even, if you know a program that can edit the file, you can use the editor to make minimal changes and then compare the file before and after.
IOW standard tricks of binary file format reverse engineering.
...If you have no existing files whatsoever, then reverse engineering the binary is your only option, and that is not pretty. Decompilation of native binaries is a black art that requires considerable time and skill. Read the various decompilation FAQs on the net.
First and for all, I would try to contact the authors of the program. Source code are options 1,2,3 and you only go with other options if there is really, really, really no hope whatsoever of obtaining source or getting normal support.
The application I am working on has "Default lists" one of which is already created in the app currently. The list has events and events touch 2-3 other models. Which would make seeding, etc very time consuming due to the complexity of the lists and the associated models the list has data in
Due to the complexity of the lists I would prefer to build the lists though the UI and then extracting it for later use.
Is there any worthwhile way of extracting the aforementioned list object and for lack of a better term "bootstrap it"
Thanks for your help in advance.
I think what you are trying to get at is seed data. Take a look at this railscats on just that.
Solution: https://github.com/rhalff/seed_dump
I highly enjoy the comments on the github page
It mainly exists for people who are too lazy writing create statements
in db/seeds.rb themselves and need something (seed_dump) to dump data
from the table(s) into seeds.rb
My response to that is "work smart, not hard" no need for me to spend a day or 2 writing out long seeds instead of doing actual work.
Unless i'm hungover then i'll just pretend seed_dump is on the fritz ;)
At my new job, we sell imported stuff. In order to be able to sell said stuff, currently the following things need to happen for every incoming shipment:
Invoice arrives, in the form of an email attachment, Excel spreadsheet
Monkey opens invoice, copy-pastes the relevant part of three columns into the relevant parts of a spreadsheet template, where extremely complex calculations happen, like =B2*550
Monkey sends this new spreadsheet to boss (email if lucky, printer otherwise), who sets the retail price
Monkey opens the reply, then proceeds to input the data into the production database using a client program that is unusable on so many levels it's not even worth detailing
Monkey fires up HyperTerminal, types in "AT", disconnect
Monkey sends text messages and emails to customers using another part of the horrible client program, one at a time
I want to change Monkey from myself to software wherever possible. I've never written anything that interfaces with email, Excel, databases or SMS before, but I'd be more than happy to learn if it saves me from this.
Here's my uneducated wishlist:
Monkey asks Thunderbird (mail server perhaps?) for the attachment
Monkey tells Excel to dump the spreadsheet into a more Jurily-friendly format, like CSV or something
Monkey parses the output, does the complex calculations
Monkey sends a link to the boss with a web form, where he can set the prices
Monkey connects to the database, inserts data
Monkey spams costumers
Is all this feasible? If yes, where do I start reading? How would you improve it? What language/framework do you think would be ideal for this? What would you do about the boss?
There are lots of tools that you could apply here, including Python, Excel macros, VB Script, etc.
In this case, PowerShell seems like an excellent choice, as it naturally combines COM access to Office, .NET, and scripting, and is all-around-awesome. If you already know a suitable technology, you'll get the job done fastest with what you know. Othewise, PowerShell.
(C# 4.0 is also reasonable, although earlier versions suck when interacting with Office's COM interfaces.)
Don't get carried away trying to solve the whole problem at once. Start by picking a small, easy part that gets you a lot of value right away. You are more likely to succeed this way. (To get your boss to agree, you need success fast. If you aren't telling your boss, you need success even faster!). Once you have that done, you can use your new-found free time (maybe only a few minutes per day) to extend your tools and skills to the next bite-sized piece. Success will accelerate success.
In time you will replace monkey with code, and either get a promotion or quit in disgust and get a better job.
The big parts are Excel and email. Excel can be handled with either COM or some sort of interaction with OpenOffice.org. Email, well, there's dozens of ways to do it. My hammer of choice is Python, along with pywin32 or PyUNO, and poplib and smtplib.
Boss... will always be boss. Not always very much you can do about the icky wetware stuff.
I'd start by asking myself the following questions
Does the invoice have to come via email or can there be a web form where the users can enter the data? There is a easy way to put a form on google docs so you can download the response in excel format in a common format set by you. I'm sure there are better ways too.
Does the boss need to create a new spreadsheet of can you provide him with a database app where he can view your form, enter the price, check "approved" and have that fire off the process that puts it in the production db?
Can the interface to the client program be worked around? Can you have some other app call the client
Can the text to the end user be sent by you and not the client app? If so, ca you automate that part
Just some thoughts.
One solution to #1 is to send email to a Unix server (instead of Exchange) and use procmail to dump out attachments (see http://gimpel.ath.cx/howto_fetch_proc_metamail.html for an example of how)
As for boss, have a nice web page which you can email him a link to. And send him a short email (3 lines or less) telling him that using that page will save him 30 mins of work over the course of a month and you 2 hours of work in a month. Just be prepared to back up the #s.
However, very high level, un less you're prepared to do the whole automation thing on your own time, you better be able to sell your boss that overall time savings x6 months are less than time to develop this. 'cause may be monkey salary in his eyes is low enough that the cost of software is just not worth it - and sadly he just may be right depending on how complicated a bulletproof robust solution is.
As I noted above, your last question is probably the most salient. It is probably best approached as a personal skunkwork project where you show the boss a completed product one day, collect your innovation bonus and then get fired because a stupider monkey can now do your job instead of you.
I've read about systems which use the Flickr database of photos to fill in gaps in photos (http://blogs.zdnet.com/emergingtech/?p=629).
How feasible is a system like this? I was toying with the idea (not just a way of killing time but as a good addition to something I am coding) of using Flickr to get photos of a certain entity (in this case, race tracks) and reconstruct a model. My biggest concern is that there aren't enough photos of a particular track and even then, it would be difficult to tell if two photos are of the same part of the racetrack, in which case one of them may be irrelevant.
How feasible is something like this? Is it worth attempting by a sole developer?
Sounds like you're wanting to build a Photosynth style system - check out Blaise Aguera y Arcas' demo at TED back in 2007. There's a section about 4 minutes in where he builds a model of the Sagrada FamÃlia from photographs.
I say +1 for photosynth answer, its a great tool. Not sure how well you could incorporate it into your own app though.
Its definately feasable. Anything is possible. And yes, doable for a single developer, just depends how much free time you have. It would be great to see something like this integrated into Virtual Earth or Google Maps Street View. Someone who could nail some software like this could help 3D model the entire world based purely on photographs. That would be a great product and make any single developer rich and famous.
So get coding. :)
I have plenty of free time, as I am in between jobs.
One way to do it is to get an overhead view of the track layout, make a blueprint based on this model, and then get one photo of the track and mimic the track's road colour. That would be a start.
LINQ to Flickr on codeplex has a great API and would be helpful for your task.