I've tried countless combinations of paths to try and get my macro working to analyze a very large set of images using JACoP. Any suggestions on what might be the issue?
this iteration got me closest (I think) to getting it to work. It even loads images, as shown in attached photos, but won't actually analyze the images themselves. Im thinking it might have something to do with me improperly setting up the loop to repeat itself? also maybe I'm designating channels incorrectly? pretty lost on what to do next, and there aren't any answers I've found on the web.
## File (label = "Input directory", style = "directory") input
## File (label = "Output directory", style = "directory") output
## String (label = "File suffix", value = ".nd2") suffix
// See also Process_Folder.py for a version of this code
// in the Python scripting language.
input = "/Users/alexandrapowell/Desktop/221223B31A3/"
output = "/Users/alexandrapowell/Desktop/221223B31A3batchh/"
processFolder(input);
// function to scan folders/subfolders/files to find files with correct suffix
function processFolder(input) {
list = getFileList(input);
fileList = Array.sort(list);
for (i = 0; i < list.length; i++) {
if(File.isDirectory(input + File.separator + list[i]))
processFolder(input + File.separator + list[i]);
if(endsWith(list[i], suffix))
processFile(input, output, list[i]);
}
}
function processFile(input, output, file) {
open(list[i])
run("JACoP ", "imga=0 imgb=1 thra=15000 thrb=3000 get_pearsons get_manders get_overlap");
// Do the processing here by adding your own code.
// Leave the print statements until things work, then remove them.
print("Processing: " + input + File.separator + file);
saveAs("Text", output + File.separator + "_coloc.txt")
print("Saving to: " + output);
run("Close");
run("Close All");
}
Related
i have a case where i enter a searchTerm to a search field. Then I want to count the results shown to randomly select one of the entries. But I cannot realize it with cypress
let countOfElements = "";
cy.get(this.SEARCH_RESULT + ' > a').then($elements => {
countOfElements = $elements.length;
cy.log(countOfElements)
cy.log("Found " + countOfElements + " results for search term + " + searchTerm)
});
cy.get(this.SEARCH_RESULT + ' > a').invoke('val').as('searchEntries')
//This is obviosly not working, but I don't get how to fix this.
let randomNumber = this.getRandomNumberBetweenTwoValues(0, cy.get('#searchEntries')));
cy.get(this.SEARCH_RESULT + ' > a').eq(randomNumber).click()
I tried different things, like storing the value with .as() but I never seem to have access to the value outside a .then() block. So how I can use the value in my "getRandomNumber..." function to decide with entry in the result list shall be selected?
Pls help. thx
let countOfElements = "";
cy.get(this.SEARCH_RESULT + ' > a').then($elements => {
countOfElements = $elements.length;
cy.log(countOfElements)
cy.log("Found " + countOfElements + " results for search term + " + searchTerm)
});
cy.get(this.SEARCH_RESULT + ' > a').invoke('val').as('searchEntries')
//This is obviosly not working, but I don't get how to fix this.
let randomNumber = getRandomNumberBetweenTwoValues(0, this.searchEntries);
cy.get(this.SEARCH_RESULT + ' > a').eq(randomNumber).click()
Something like this could work. If this.searchEntries give undefined error then make the block function(){} instead of ()=>{}
All punctuations in the HTML that was brought over by pasting from MS word, show as a little square instead of " or ' The characters show as "FS" and "GS" in notepad++ and in plain HTML.
I tried to use the "DejaVu Sans" font but it did not help at all.
Any advice?
Ended replacing all offending characters.
String.prototype.replaceAll = function(str1, str2, ignore)
{
return this.replace(new RegExp(str1.replace(/([\,\!\\\^\$\{\}\[\]\(\)\.\*\+\?\|\<\>\-\&])/g, function(c){return "\\" + c;}), "g"+(ignore?"i":"")), str2);
};
Then for each cell of text:
function cleanHTML(input) {
var output = input.replaceAll('“','"');
output = output.replaceAll('”', '"');
output = output.replaceAll("’", "'");
output = output.replaceAll("‘", "'");
output = output.replaceAll("", "'");
output = output.replaceAll("", "-");
output = output.replaceAll("", "'");
return output;
}
Whatever other charter will show as square later, I will add a replace for it.
Hope someone will benefit from that.
I have a DRF model created in h2o flow that is supposed to be binomial and flow indicates that it is binomial
but I am having a problem where, importing it into h2o steam and deploying it to the prediction service, the model does not seem to be recognized as binomial. The reason I think this is true is shown below. The reason this is a problem is because I think it is what is causing the prediction service to NOT show the confidence value for the prediction (this reasoning is also shown below).
In the prediction service, I can get a prediction label, but no values filled in the index-label-probability table.
Using the browser inspector (google chrome), the prediction output seems to depend on a file called predict.js.
In order to get the prediction probability values to show in the prediction service, it seems like this block of code needs to run to get to this line. Opening the predict.js file within the inspector on the prediction service page and adding some debug output statements at some of the top lines (indicated with DEBUG/ENDDEBUG comments in the code below), my showResults() function then looks like:
function showResult(div, status, data) {
////////// DEBUG
console.log("showResult entered")
////////// ENDDEBUG
var result = '<legend>Model Predictions</legend>'
////////// DEBUG
console.log(data)
console.log(data.classProbabilities)
console.log("**showResult: isBinPred=" + isBinaryPrediction)
////////// ENDDEBUG
if (data.classProbabilities) {
////////// DEBUG
console.log("**showResult: data.classProbabilities not null")
////////// ENDDEBUG
// binomial and multinomial
var label = data.label;
var index = data.labelIndex;
var probs = data.classProbabilities;
var prob = probs[index];
result += '<p>Predicting <span class="labelHighlight">' + label + '</span>';
if (probs.length == 2) {
result += ' based on max F1 threshold </p>';
}
result += ' </p>';
result += '<table class="table" id="modelPredictions"> \
<thead> \
<tr> \
<th>Index</th> \
<th>Labels</th> \
<th>Probability</th> \
</tr> \
</thead> \
<tbody> \
';
if (isBinaryPrediction) {
var labelProbabilitiesMapping = [];
outputDomain.map(function(label, i) {
var labelProbMap = {};
labelProbMap.label = outputDomain[i];
labelProbMap.probability = probs[i];
if (i === index) {
labelProbMap.predicted = true;
}
labelProbMap.originalIndex = i;
labelProbabilitiesMapping.push(labelProbMap);
});
labelProbabilitiesMapping.sort(function(a, b) {
return b.probability - a.probability;
});
var limit = labelProbabilitiesMapping.length > 5 ? 5 : labelProbabilitiesMapping.length;
for (var i = 0; i < limit; i++) {
if (labelProbabilitiesMapping[i].predicted === true) {
result += '<tr class="rowHighlight">'
} else {
result += '<tr>'
}
result += '<td>' + labelProbabilitiesMapping[i].originalIndex + '</td><td>' + labelProbabilitiesMapping[i].label + '</td> <td>' + labelProbabilitiesMapping[i].probability.toFixed(4) + '</td></tr>';
}
} else {
for (var label_i in outputDomain) {
if (parseInt(label_i) === index ){
result += '<tr class="rowHighlight">'
} else {
result += '<tr>'
}
result += '<td>' + label_i + '</td><td>' + outputDomain[label_i] + '</td> <td>' + probs[label_i].toFixed(4) + '</td></tr>';
}
}
result += '</tbody></table>';
}
else if ("cluster" in data) {
// clustering result
result = "Cluster <b>" + data["cluster"] + "</b>";
}
else if ("value" in data) {
// regression result
result = "Value <b>" + data["value"] + "</b>";
}
else if ("dimensions" in data) {
// dimensionality reduction result
result = "Dimensions <b>" + data["dimensions"] + "</b>";
}
else {
result = "Can't parse result: " + data;
}
div.innerHTML = result;
}
and clicking the "predict" in the prediction service now generates the console output:
If I were to add a statement isBinaryPrediction = true to forcec the global variable to true (around here) and run the prediction again, the console shows:
indicating that the variable outputDomain is undefined. The variable outputDomain seems to be set in the function showModel. This function appears to run when the page loads, so I can't edit it in the chrome inspector to see what the variable values are. If anyone knows how to fix this problem (getting the prediction probability values to show up for h2o steam's prediction service for binomial models) it would a big help. Thanks :)
The UI has not been updated to handle MOJOs yet and there seems to be a bug. You're welcome to contribute: https://github.com/h2oai/steam/blob/master/CONTRIBUTING.md
My solution is very hacky, but works for my particular case (ie. I have a DRF, binomial model in h2o steam that is not being recognized as a binary model (how I know this is shown in this answer)).
Solution:
In my original post, there was a variable outputDomain that was undefined. Looking at the source code, that variable is set to (what is supposed to be) the domain labels of the output response for the model, here. I changed this line from outputDomain = domains[i1]; to outputDomain = domains[i1-1];. My output after clicking the predict button looks like:
From the official linux download for h2o steam, you can access the prediction service predict.js file by opening steam-1.1.6-linux-amd64/var/master/assets/ROOT.war/extra/predict.js, then saving changes and relaunching the jetty server $ java -Xmx6g -jar var/master/assets/jetty-runner.jar var/master/assets/ROOT.war.
Causes?:
I suspect the problem has something to do with that fact that the global variable isBinaryPrediction in predict.js seems to remain false for my model. The reason that isBinaryPrediction is false seems to be because in the function showInputParameters(), data.m has no field _problem_type. Using console.dir(data, {depth: null}) in the inspector console to see the fields of data.m, I see that the expectedd field data.m._problem_type does not exist and so returns undefined, thus isBinaryPrediction is never set true (here).
Why this is happening, I do not know. I have only used DRF models in steam so far and this may be a problem with that model, but I have not tested. If anyone knows why this may be happening, please let me know.
I am trying to parse a huge file (approx 23 MB) using the code below, wherein I populate a multiprocessing.manager.list with all the lines read from the file . In the target routine (parse_line) for each process, I pop a line and parse it to create a defaultdict object with certain parsed attributes and finally push each of these objects into another multiprocessing.manager.list.
class parser(object):
def __init__(self):
self.manager = mp.Manager()
self.in_list = self.manager.list()
self.out_list = self.manager.list()
self.dict_list,self.lines, self.pcap_text = [],[],[]
self.last_timestamp = [[(999999,0)]*32]*2
self.num = Word(nums)
self.word = Word(alphas)
self.open_brace = Suppress(Literal("["))
self.close_brace = Suppress(Literal("]"))
self.colon = Literal(":")
self.stime = Combine(OneOrMore(self.num + self.colon) + self.num + Literal(".") + self.num)
self.date = OneOrMore(self.word) + self.num + self.stime
self.is_cavium = self.open_brace + (Suppress(self.word)) + self.close_brace
self.oct_id = self.open_brace + Suppress(self.word) + Suppress(Literal("=")) \
+ self.num + self.close_brace
self.core_id = self.open_brace + Suppress(self.word) + Suppress(Literal("#")) \
+ self.num + self.close_brace
self.ppm_id = self.open_brace + self.num + self.close_brace
self.oct_ts = self.open_brace + self.num + self.close_brace
self.dump = Suppress(Word(hexnums) + Literal(":")) + OneOrMore(Word(hexnums))
self.opening = Suppress(self.date) + Optional(self.is_cavium.setResultsName("cavium")) \
+ self.oct_id.setResultsName("octeon").setParseAction(lambda toks:int(toks[0])) \
+ self.core_id.setResultsName("core").setParseAction(lambda toks:int(toks[0])) \
+ Optional(self.ppm_id.setResultsName("ppm").setParseAction(lambda toks:int(toks[0])) \
+ self.oct_ts.setResultsName("timestamp").setParseAction(lambda toks:int(toks[0]))) \
+ Optional(self.dump.setResultsName("pcap"))
def parse_file(self, filepath):
self.filepath = filepath
with open(self.filepath,'r') as f:
self.lines = f.readlines()
for lineno,line in enumerate(self.lines):
self.in_list.append((lineno,line))
processes = [mp.Process(target=self.parse_line) for i in range(mp.cpu_count())]
[process.start() for process in processes]
[process.join() for process in processes]
while self.in_list:
(lineno, len) = self.in_list.pop()
print mp.current_process().name, "start"
dic = defaultdict(int)
result = self.opening.parseString(line)
self.pcap_text.append("".join(result.pcap))
if result.timestamp or result.ppm:
dic['oct'], dic['core'], dic['ppm'], dic['timestamp'] = result[0:4]
self.last_timestamp[result.octeon][result.core] = (result.ppm,result.timestamp)
else:
dic['oct'], dic['core'] = result[0:2]
dic['ppm'] = (self.last_timestamp[result.octeon][result.core])[0]
dic['ts'] = (self.last_timestamp[result.octeon][result.core])[1]
dic['line'] = lineno
self.out_list.append(dic)
However this entire process takes approximately 3 minutes to complete.
My question is, if there is a better way to make this faster ?
I am using pyparsing module to parse each line, if it makes any difference.
PS: Made changes in the routine Paul McGuire's advice
Not a big performance issue, but learn to iterate over files directly, instead of using readlines(). In place of this code:
self.lines = f.readlines()
for lineno,line in enumerate(self.lines):
self.in_list.append((lineno,line))
You can write:
self.in_list = list(enumerate(f))
A hidden performance killer is using while self.in_list: (lineno,line) = list.pop(). Each call to pop removes the 0'th element from the list. Unfortunately, Python's lists are implemented as arrays. To remove the 0'th element, the 1..n-1'th elements have to be moved up one slot in the array. You don't really have to destroy self.in_list as you go, just iterate over it:
for lineno, line in self.in_list:
<Do something with line and line no. Parse each line and push into out_list>
If you are thinking that consuming self.in_list as you go is a memory-saving measure, then you can avoid the array-shifting inefficiency of Python lists by using a deque instead (from Python's provided collections module). deque's are implemented internally as linked lists, so that pushing or popping to and from either end is very fast, but indexed access is slow. To use a deque, replace the line:
self.in_list = list(enumerate(f))
with:
self.in_list = deque(enumerate(f))
Then replace the call in your code self.in_list.pop() with self.in_list.popleft().
But MUCH more likely to be the performance issue is the pyparsing code you are using to process each line. But since you didn't post the parser code, there is not much help we can provide there.
To get an idea about where the time is going, try leaving all your code, and then comment out the <Do something with line and line no. Parse each line and push into out_list> code (you may have to add a pass statement for the for loop), and then run against your 23MB file. This will give you a rough idea about how much of your 3 minutes is being spent in reading and iterating over the file, and how much is being spent doing the actual parsing. Then post back in another question when you find where the real performance issues lie.
I'm not a groovy expert, just use it from time to time. One of the latest goals was to generate a very simple file containing some random data. I created the following script:
out = new File('sampledata.txt')
Random random = new Random();
java.util.Date dt = new java.util.Date();
for (int i=0; i<100000; ++i) {
dt = new java.util.Date();
out << dt.format('yyyMMdd HH:mm:ss.SSS') + '|box|process|||java.lang.Long|' + random.nextInt(100) + '|name\n'
}
Now, I'm really puzzled with its performance. It takes around 1.5 minutes to complete whilst the same code written in Java or Ruby takes less than a second.
Similar code in Ruby (takes around 1 second to execute):
require "time"
File.open("output.txt", "w") do |file|
100000.times do
line = Time.now.strftime("%Y%m%d %H:%M:%S.%L") + '|box|process|||java.lang.Long|' + rand(100).to_s + '|name'
file.puts line
end
end
Any ideas how groovy's processing speed could be improved?
The left shift operator opens the file, jumps to the end, appends the text, and closes the file again...
Instead, try:
Random random = new Random();
// Open the file and append to it.
// If you want a new file each time, use withWriter instead of withWriterAppend
new File('sampledata.txt').withWriterAppend { w ->
100000.times {
w.writeLine "${new Date().format('yyyMMdd HH:mm:ss.SSS')}|box|process|||java.lang.Long|${random.nextInt(100)}|name"
}
}
(this is also much more like what the Ruby code is doing)