I am creating a survey in Qualtrics with many photos, say 1000. I want to have each survey participant answer, say 6, questions per photo. Each participant will see 5 photos that are randomly assigned.
Before looking into things, I assumed that there would be a way to upload the 1000 photos, create one block in Qualtrics (with the 6 questions) and then simply randomize the photo that occurs and have this be repeated this 5 times.
But it seems like this is either not possible or not obvious. I called Qualtrics and they said that I would manually need to create 1000 blocks (each block would be exactly the same with the exception of the title and the photo). I would then need to go into the Survey Flow and use the Randomizer there and manually add all 1000 blocks and have it randomly present 5 of the elements.
I really hope that there is a better way. This will take a ton of time if I have to do it this way.
If not, is there any way to automate anything?
Creating new blocks and automatically populating the photos. I know python and could possibly write a script to generate blocks, BUT the photo names are changed from their original names into some complicated code that Qualtrics generates.
Loading the photos into Qualtrics all at once (it currently requires one to load photos one at a time).
It turns out that there is a much better faster way to do this than the 1000 blocks fix.
There is a bunch of stuff going on to accomplish it, but it is possible.
First, one needs to put the photos into Qualtrics through the Graphics Library. The best way to do this is to simply drag and drop the photos into the desired location. Luckily one does not have to do this one-by-one. Make sure that they are in the order you want.
Second, create a block with a "question" where you want the random photo to appear. This block should also have all 6 questions.
Third, create a column in a spreadsheet (in, eg. Excel) of the URLs corresponding to the photos. This should be in order. One way to do this is mentioned at the bottom.
Fourth, go to the Loop and Merge option for this block. Copy and paste the column of URLs to, say, Field 1. Luckily this option exists and one does not have to do this one-by-one either. A sidenote is that if one changes the numbers in the gray boxes to the left of the rows, this changes what appears in the results. But there is no apparent way to change these more than one-by-one at a time.
Then you should be all set.
Finally, a little bit about how to get the URLs of the photos. Once again, make sure the photos in the library are in the order you want. Then you can use web scraping to scrape the image names, which can then be put into the proper URL. I used Python's Selenium and BeautifulSoup to accomplish this. Here is what I did, using a mac. The code at least gives you the idea:
from bs4 import BeautifulSoup
import codecs
import os
from selenium import webdriver
import re
chromedriver = "File path to /chromedriver"
os.environ["webdriver.chrome.driver"] = chromedriver
driver = webdriver.Chrome(chromedriver)
*In the Chrome browser that has appeared, manually navigate to the photos library page, then:
abc = driver.find_elements_by_css_selector(".thumbframe")
file = codecs.open('outputURLs.txt', 'w', encoding = 'utf-8')
urls = {}
for i in range(0,len(abc)):
h = abc[i].get_attribute("innerHTML")
soup = BeautifulSoup(h)
t = soup.find_all("img", attrs={"p4":re.compile('.*')})
urls[i] = t[0]['p1']
file.write("<img src=*Qualtrics Path/Graphic.php?IM=" + urls[i] + "/> + '\n')
One can find the proper first part to stick in "Qualtrics Path" by, eg. going to the Qualtrics Survey Editor, inserting a photo using Rich HTML Editing (or something similar), inserting the photo, clicking on View Source, and then looking at the pattern file path to use. It may begin with something like https://qualtrics.com/...
Then copy the results into a spreadsheet program and you should be ready to copy and paste.
Related
I am generating and storing PDFs in a database.
The pdf data is stored in a text field using Convert.ToBase64String(pdf.ByteArray)
If I generate the same exact PDF that already exists in the database, and compare the 2 base64strings, they are not the same. A big portion is the same, but it appears about 5-10% of the text is different each time.
What would make 2 pdfs different if both were generated using the same method?
This is a problem because I can't tell if the PDF was modified since it was last saved to the db.
Edit: The 2 pdfs visually appear exactly the same when viewing the actual pdf, but the base64string of the bytes are different
Two PDFs that look 100% the same visually can be completely different under the covers. PDF producing programs are free to write the word "hello" as a single word or as five individual letters written in any order. They are also free to draw the lines of a table first followed by the cell contents, or the cell contents first, or any combination of these such as one cell at a time.
If you are actually programmatically creating the PDFs and you create two PDFs using completely identical code you still won't get files that are 100% identical. There's a couple of reasons for this, the most obvious is that PDFs support creation and modification dates. These will obviously change depending on when they are created. You can override these (and confuse everyone else so I don't recommend this) using something like this:
var info = writer.Info;
info.Put(PdfName.CREATIONDATE, new PdfDate(new DateTime(2001,01,01)));
info.Put(PdfName.MODDATE, new PdfDate(new DateTime(2001,01,01)));
However, PDFs also support a unique identifier in the trailer's /ID entry. To the best of my knowledge iText has no support for overriding this parameter. You could duplicate your PDF, change this manually and then calculate your differences and you might get closer to a comparison.
Then there's fonts. When subsetting fonts, producers create a unique internal name based on the original name and an arbitrary selection of six uppercase ASCII letters. So for the font Calibri the font's name could be JLXWHD+Calibri one time and SDGDJT+Calibri another time. iText doesn't support overriding of this because you'd probably do more harm than good. These internal names are used to avoid font subset collisions.
So the short answer is that unless you are comparing two files that are physical duplicates of each other you can't perform a direct comparison on their binary contents. The long answer is that you can tweak some of the PDF entries to remove unique parts for comparison only but you'd probably be doing more work than it would take to just re-store the file in the database.
I'm using Firefox 53, and have Scrapbook X and want to save a lot of pages using the Save Multiple URLs feature, but before I do that, I want to extract a specific list of URLs without having to do so manually.
The site I'm looking at extracting data from is www.address-data.co.uk - namely this page.
What I want to do is extract only the URLs and subpages within that page but not the privacy policy or contact us page and all the sub-pages with the EH postcodes.
Is there a way to do this online, or any tool for Mac OS X that can find all related URLs before I copy them into Scrapbook's Save Multiple URLs (where I save them in a subfolder of Scrapbook)?
I assume that EH45 is typical of those you want to extract from the page you mentioned. Like its siblings it's of the form https://address-data.co.uk/postcode-district-EH<postcode number>.
This means that you can make a complete list of the urls if you have a list of the numbers, or of the postcodes.
My main difficulty in answering is that I don't know what tools (especially programming tools) you might have at your disposal. I will assume only that you have, or can obtain, access to an editor that can do macros or that can edit columns. On Windows I would use Emerald (ow known as Crimson).
Then copy the contents of the table in the EH page (not the table headings) and remove everything except the first column. Finally, prepend every item in the column with 'https://address-data.co.uk/postcode-district-'.
PS: This might also be a good question to put on SuperUser.
I am using Mathematica to enhance and thin images. I used it for single image, now i want to use it for multiple images. so I have to import 6 images, do the thining and store them in a list for example. Can any one show me how to do that??
The images will be used for biometrics identification system.
Since you want a list as a result you might think of using either Table or Map. Either of those can do n things, one after another, and put the result into your final list.
Since you didn't show the steps you used for processing a single list it is a little difficult to tell you exactly how to wrap Table or Map around this.
If you have a list of image file names then you could use Map to process those names one at a time. The processing could either be a compound function to Import the image and then enhance and thin and the output of that function would be a single processed image. Map would then do the repetition over all the names.
Table might work in a similar way, but you use each iteration to get the file name, do the processing and store the result in your desired list.
The St. Louis Federal Reserve Bank has a great set of data available on a variety of their web pages, such as:
http://research.stlouisfed.org/fred2/series/OILPRICE/downloaddata?cid=32217
http://www.federalreserve.gov/releases/h10/summary/default.htm
http://research.stlouisfed.org/fred2/series/DGS20
The data sets get updated, some as often as daily. I tend to have an interest in the daily data (see the above settings on the URLS)
I'd like to import these kinds of price or rate data streams (accessible as CSV or Excel files at the above URLs) directly into Mathematica.
I've looked at the documentation on Importing[] but I find scant documentation (actually none) on how to go about something like this.
It looks like I need to navigate to the pages, send some data to select specific files and formats, trigger the download, then access the downloaded data from my own machine. Even better if I could access the data directly from the sites.
I had hoped Wolfram Alpha might make this sort thing easy, but I haven't had any success.
FinancialData[] would seem natural for this sort of thing, but I don't see anyway to do it. Financial data has lots of features, but I don't see a way yo get this sort of thing.
Does anyone have any experience with this or can someone point me in the right direction?
You can Import directly from a URL. For example, the data from federalreserve.gov can be obtained and visualized as follows.
url = "http://www.federalreserve.gov/datadownload/Output.aspx?";
url = url<>"rel=H10&series=a660e724c705cea4b7bd1d1b85789862&lastObs=&";
url = url<>"from=&to=&filetype=csv&label=include&layout=seriescolumn";
data = Import[url, "CSV"];
DateListPlot[data[[7 ;;]], Joined -> True]
I broke up url for convenience, since it's so long. I had to examine the contents of data before I knew exactly how to plot it - a step that is typically necessary. I'm sure that the data from stlouisfed.org can be obtained in a similar way, but it requires the use of an API with key to access it.
As Mark said, you can get the data directly from a URL. Your oil data can be imported from a different URL than you had:
http://research.stlouisfed.org/fred2/data/OILPRICE.txt
With that URL, you can do this:
oil = Import["http://research.stlouisfed.org/fred2/data/OILPRICE.txt",
"Table", "HeaderLines" -> 12, "DateStringFormat" -> {"Year", "Month", "Day"}];
DateListPlot[oil, Joined -> True, PlotRange -> All]
Note that "HeaderLines"->12 option strips off the header text in the first 12 lines (you have to count the header lines to know how many to remove). I've also specified the date format.
To find that URL, do as you did before, but click on a data series and then choose View Data from the menu on the left when you see the chart.
The documentation has a short example on extracting data out of a webpage:
http://reference.wolfram.com/mathematica/howto/CleanUpDataImportedFromAWebsite.html
Of course, what actually needs to be done will vary significantly from page to page.
discussion on how to do this with your API key here:
http://library.wolfram.com/infocenter/MathSource/7583/
the function is based on the API documentation. I haven't looked at the code for a couple of years and from memory I put it together rather quickly but I have used it regularly for over 2 years without problems. Here is an example for monthly non seasonally adjusted retail sales from early 1992 to now:
wolfram alpha also uses FRED data so you could use that as an alternative to direct import but it is more tricky to get the query right. I prefer to use FRED directly. Also from memory the data is only available on alpha the day after the release, which is not what you would typically want.
I have an AppleScript program which creates XML tags and elements within an Adobe InDesign document. The data is in tables, and tagging each cell takes .5 seconds. The entire script takes several hours to complete.
I can post the inner loop code, but I'm not sure if SO is supposed to be generic or specific. I'll let the mob decide.
[edit]
The code builds a list (prior to this loop) which contains one item per row in the table. There is also a list containing one string for each column in the table. For each cell, the program creates an XML element and an XML tag by concatenating the items in the [row]/[column] positions of the two lists. It also associates the text in that cell to the newly-created element.
I'm completely new to AppleScript so some of this code is crudely modified from Adobe's samples. If the code is atrocious I won't be offended.
Here's the code:
repeat with columnNumber from COL_START to COL_END
select text of cell ((columnNumber as string) & ":" & (rowNumber as string)) of ThisTable
tell activeDocument
set thisXmlTag to make XML tag with properties {name:item rowNumber of symbolList & "_" & item columnNumber of my histLabelList}
tell rootXmlElement
set thisXmlElement to make XML element with properties {markup tag:thisXmlTag}
end tell
set contents of thisXmlElement to (selection as string)
end tell
end repeat
EDIT: I've rephrased the question to better reflect the correct answer.
The problem is almost certainly the select. Is there anyway you could extract all the text at once then iterate over internal variables?
I figured this one out.
The document contains a bunch of data tables. In all, there are about 7,000 data points that need to be exported. I was creating one root element with 7,000 children.
Don't do that. Adding each child to the root element got slower and slower until at about 5,000 children AppleScript timed out and the program aborted.
The solution was to make my code more brittle by creating ~480 children off the root, with each child having about 16 grandchildren. Same number of nodes, but the code now runs fast enough. (It still takes about 40 minutes to process the document, but that's infinitely less time than infinity.)
Incidentally, the original 7,000 children plan wasn't as stupid or as lazy as it appears. The new solution is forcing me to link the two tables together using data in the tables that I don't control. The program will now break if there's so much as a space where there shouldn't be one. (But it works.)
I can post the inner loop code, but I'm not sure if SO is supposed to be generic or specific. I'll let the mob decide.
The code you post as an example can be as specific as you (or your boss) is comfortable with - more often than not, it's easier to help you with more specific details.
If the inner loop code is a reasonable length, I don't see any reason you can't post it. I think Stack Overflow is intended to encompass both general and specific questions.
Are you using InDesign or InDesign Server? How many pages is your document (or what other information can you tell us about your document/ID setup)?
I do a lot of InDesign Server development. You could be seeing slow-downs for a couple of reasons that aren't necessarily code related.
Right now, I'm generating 100-300 page documents almost completely from script/xml in about 100 seconds (you may be doing something much larger).