Need help to insert some text to my c++ file - bash
I want to insert some text to my c++ file after certain line ( pattern ). The following is my file structure.
38 #include "stdlib.h"
39 #include "string.h"
40 #include "malloc.h"
41
...
324 void DMProcMon::threadManagerMonitorThread(DMProcMon* dmProcMon)
325 {
...
338 while (dmState == DVProcMon::Active &&
339 DmManService::getDCMRestartingFlag() == 0){
340 try{
342 setupTimerVerification(dmProcMon);
343 setupSignalVerification(dmProcMon);
344
....
360 }
I want to add code coverage macros using gcov. So basically what I need to achieve is
Add below text after all the #include statements.
45 #ifdef GCOV
46 extern "C"
47 void _gcov_flush();
48 #endif
Add the below text after the while statement in the threadManagerMonitorThread function
#ifdef GCOV
_gcov_flush();
#endif
So final code will loos like as below.
38 #include "stdlib.h"
39 #include "string.h"
40 #include "malloc.h"
41
45 #ifdef GCOV
46 _gcov_flush();
47 #endif
...
324 void DMProcMon::threadManagerMonitorThread(DMProcMon* dmProcMon)
325 {
...
338 while (dmState == DVProcMon::Active &&
339 DmManService::getDCMRestartingFlag() == 0){
340 try{
342 #ifdef GCOV
343 _gcov_flush();
344 #endif
346 setupTimerVerification(dmProcMon);
347 setupSignalVerification(dmProcMon);
348
....
360 }
What is the best way to do this. I would like to do this with either bash or pythyon.
Thanks
~S
You might use sed to insert a line after/before a line containing a certain pattern. See this: https://stackoverflow.com/a/11695086/2749648.
Also here is the explanation on how to insert a line after a block of code (this will help you with the #include block) but i am not sure on how to fix the while statement. If while statement is on a single line, it would be easy, but that is hard to guarantee.
Related
Missing ',' in line when Biopython reads a nexus tree
I want to edit a tree that I got from BEAST2 treeannotator in nexus-format. Usually I use the module Phylo from Biopython for such work but Phylo.read(r"filename.tree", "nexus") gave me the next exception: --------------------------------------------------------------------------- NexusError Traceback (most recent call last) Input In [29], in <cell line: 1>() ----> 1 Phylo.read(r"filename.tree", "nexus") File ~\miniconda3\lib\site-packages\Bio\Phylo\_io.py:60, in read(file, format, **kwargs) 58 try: 59 tree_gen = parse(file, format, **kwargs) ---> 60 tree = next(tree_gen) 61 except StopIteration: 62 raise ValueError("There are no trees in this file.") from None File ~\miniconda3\lib\site-packages\Bio\Phylo\_io.py:49, in parse(file, format, **kwargs) 34 """Parse a file iteratively, and yield each of the trees it contains. 35 36 If a file only contains one tree, this still returns an iterable object that (...) 46 47 """ 48 with File.as_handle(file) as fp: ---> 49 yield from getattr(supported_formats[format], "parse")(fp, **kwargs) File ~\miniconda3\lib\site-packages\Bio\Phylo\NexusIO.py:40, in parse(handle) 32 def parse(handle): 33 """Parse the trees in a Nexus file. 34 35 Uses the old Nexus.Trees parser to extract the trees, converts them back to (...) 38 eventually change Nexus to use the new NewickIO parser directly.) 39 """ ---> 40 nex = Nexus.Nexus(handle) 42 # NB: Once Nexus.Trees is modified to use Tree.Newick objects, do this: 43 # return iter(nex.trees) 44 # Until then, convert the Nexus.Trees.Tree object hierarchy: 45 def node2clade(nxtree, node): File ~\miniconda3\lib\site-packages\Bio\Nexus\Nexus.py:668, in Nexus.__init__(self, input) 665 self.options["gapmode"] = "missing" 667 if input: --> 668 self.read(input) 669 else: 670 self.read(DEFAULTNEXUS) File ~\miniconda3\lib\site-packages\Bio\Nexus\Nexus.py:718, in Nexus.read(self, input) 716 break 717 if title in KNOWN_NEXUS_BLOCKS: --> 718 self._parse_nexus_block(title, contents) 719 else: 720 self._unknown_nexus_block(title, contents) File ~\miniconda3\lib\site-packages\Bio\Nexus\Nexus.py:759, in Nexus._parse_nexus_block(self, title, contents) 757 for line in block.commandlines: 758 try: --> 759 getattr(self, "_" + line.command)(line.options) 760 except AttributeError: 761 raise NexusError("Unknown command: %s " % line.command) from None File ~\miniconda3\lib\site-packages\Bio\Nexus\Nexus.py:1144, in Nexus._translate(self, options) 1142 break 1143 elif c != ",": -> 1144 raise NexusError("Missing ',' in line %s." % options) 1145 except NexusError: 1146 raise NexusError: Missing ',' in line 1 AB298157.1_2015_-7.9133750332192605_114.8086828279248, 2 AB298158.1_2007_-8.41698974207… Using Nexus.read(Nexus(), input=r"filename.tree") gave the same result. Please could anyone help with this? I cannot understand the reason of this error because nexus file looks correct.
The reason is that Biopython cannot read nexus trees with links, constituent from translations & a newick tree. So it is required previously to convert this to the form with full names into the tree (as hereinbelow). Begin tree TREE1 = (((your,tree),(in,(the, newick))),format); End; P.S. It is allowed in the newick format to surround the label with quotes, & some programmes or scripts add them to those names that have ambiguous characters. But it can lead to exceptions during the following phylogenetic analysis, for instance, in BEAST. I wish you would be careful with this.
ncurses: init_color() has no effect
Trying to define color pairs, I was getting strange results. All 256 colors are already defined, and attempt to change any color with init_color() has no affect. I'm using Putty with 256-colors enabled and TERM=xterm-256color (also putty-256color), ncurses 6.0 compiled with --enable-widec and --enable-ext-colors. This shows all colors are defined and the init_color() doesn't change anything even though it succeeds: init_scr(); start_color(); if (has_colors() && COLORS == 256 && can_change_color()) { NCURSES_COLOR_T f; for (f = 1; f < 256; f++) { if (init_pair(f, f, COLOR_BLACK) == ERR) break; attron(COLOR_PAIR(f)); printw("(%d)", f); attroff(COLOR_PAIR(f)); refresh(); } getch(); clear(); for (f = 1; f < 256; f++) { if (init_color(f, 0, 0, f*3) == ERR) break; if (init_pair(f, f, COLOR_BLACK) == ERR) break; attron(COLOR_PAIR(f)); printw("(%d)", f); attroff(COLOR_PAIR(f)); refresh(); } getch(); clear(); } I've read that the default colors can't be changed, but only refers to COLOR_BLACK, etc (0-7). Where are these 256 default colors defined and why can't I change them? If they can't be changed, I could make use of the colors defined, but only if I can rely on them being the same on any 256-color capable terminal.
short: PuTTY doesn't do that, ncurses can't tell if PuTTY can... long: In ncurses, the init_color function checks its parameters (in the example given, those appear okay if your $TERM is "xterm-256color"), as well as checking if the terminal description has the initc (initialize_color) capability. If that is missing or cancelled, ncurses returns an error. However, that's only the terminal description. ncurses cannot tell if you have chosen an incorrect or inappropriate terminal description. In a quick check, PuTTY doesn't respond to the control sequence which is used in initc. This is a known limitation, as indicated in the (more appropriate) terminal description putty-256color provided by ncurses: putty-256color|PuTTY 0.58 with xterm 256-colors, use=xterm+256setaf, use=putty, That xterm+256setaf is used for terminals whose palette is hard-coded. PuTTY is not the only terminal which both sets TERM=xterm` and lacks the ability to change its palette. If you happen to be using an old version of the terminal database, you may be misled, since that error was fixed in 2014: # 2014-03-30 # * cancel ccc in putty-256color and konsole-256color for consistency # with the cancelled initc capability (patch by Sven Zuhlsdorf). # * add xterm+256setaf building block for various terminals which only # get the 256-color feature half-implemented -TD # * updated "st" entry (leaving the 0.1.1 version as "simpleterm") to # 0.4.1 -TD # Like the other terminals whose developers set TERM=xterm (or TERM=xterm-256color), there are differences between those and xterm. Further reading: Why not just use TERM set to "xterm"?
Couple of things I discovered. First, yes I was apparently referencing an old putty-256color terminfo that had "ccc", allowing can_change_color() to succeed, but then init_color() would fail. But the same Putty window using "xterm-256color" would init_color() OK and color_content() even shows the new values, but nothing changed on the screen. What was really confusing is sometimes the colors I set would appear and other times seemingly random colors appeared instead. Here's what I found: putty-256color xterm-256color gnome-256color xterm-256color (putty) (putty) (gnome-terminal) (MobaXterm) -------------- -------------- ---------------- -------------- change_color OK/ERR OK OK OK init_color ERR ERR OK OK color_content ERR OK/NOCH OK OK color changed? NO NO YES YES So there's basically no way to determine whether colors can be changed or not. But I did find that every terminal had already defined the standard 256 xterm colors, whether they could be changed or not. So, now, I just define the colors I want to use using the same color numbers as in the xterm palette. That way, the colors I expect will appear whether I needed to define them or not. So, to use "PaleGreen3", I just use: init_color(77, 372, 843, 372) If it works, it works, and if not, its probably already defined. For reference, I converting all the Xwindow/xterm colors from GUI hex notation to the ncurses (0-1000) values: # Name Tk Ncurses --- ---------------- ------- ------------- 16 Grey0 #000000 0,0,0 17 NavyBlue #00005f 0,0,372 18 DarkBlue #000087 0,0,529 19 Blue3 #0000af 0,0,686 20 Blue3 #0000d7 0,0,843 21 Blue1 #0000ff 0,0,1000 22 DarkGreen #005f00 0,372,0 23 DeepSkyBlue4 #005f5f 0,372,372 24 DeepSkyBlue4 #005f87 0,372,529 25 DeepSkyBlue4 #005faf 0,372,686 26 DodgerBlue3 #005fd7 0,372,843 27 DodgerBlue2 #005fff 0,372,1000 28 Green4 #008700 0,529,0 29 SpringGreen4 #00875f 0,529,372 30 Turquoise4 #008787 0,529,529 31 DeepSkyBlue3 #0087af 0,529,686 32 DeepSkyBlue3 #0087d7 0,529,843 33 DodgerBlue1 #0087ff 0,529,1000 34 Green3 #00af00 0,686,0 35 SpringGreen3 #00af5f 0,686,372 36 DarkCyan #00af87 0,686,529 37 LightSeaGreen #00afaf 0,686,686 38 DeepSkyBlue2 #00afd7 0,686,843 39 DeepSkyBlue1 #00afff 0,686,1000 40 Green3 #00d700 0,843,0 41 SpringGreen3 #00d75f 0,843,372 42 SpringGreen2 #00d787 0,843,529 43 Cyan3 #00d7af 0,843,686 44 DarkTurquoise #00d7d7 0,843,843 45 Turquoise2 #00d7ff 0,843,1000 46 Green1 #00ff00 0,1000,0 47 SpringGreen2 #00ff5f 0,1000,372 48 SpringGreen1 #00ff87 0,1000,529 49 MediumSpringGreen #00ffaf 0,1000,686 50 Cyan2 #00ffd7 0,1000,843 51 Cyan1 #00ffff 0,1000,1000 52 DarkRed #5f0000 372,0,0 53 DeepPink4 #5f005f 372,0,372 54 Purple4 #5f0087 372,0,529 55 Purple4 #5f00af 372,0,686 56 Purple3 #5f00d7 372,0,843 57 BlueViolet #5f00ff 372,0,1000 58 Orange4 #5f5f00 372,372,0 59 Grey37 #5f5f5f 372,372,372 60 MediumPurple4 #5f5f87 372,372,529 61 SlateBlue3 #5f5faf 372,372,686 62 SlateBlue3 #5f5fd7 372,372,843 63 RoyalBlue1 #5f5fff 372,372,1000 64 Chartreuse4 #5f8700 372,529,0 65 DarkSeaGreen4 #5f875f 372,529,372 66 PaleTurquoise4 #5f8787 372,529,529 67 SteelBlue #5f87af 372,529,686 68 SteelBlue3 #5f87d7 372,529,843 69 CornflowerBlue #5f87ff 372,529,1000 70 Chartreuse3 #5faf00 372,686,0 71 DarkSeaGreen4 #5faf5f 372,686,372 72 CadetBlue #5faf87 372,686,529 73 CadetBlue #5fafaf 372,686,686 74 SkyBlue3 #5fafd7 372,686,843 75 SteelBlue1 #5fafff 372,686,1000 76 Chartreuse3 #5fd700 372,843,0 77 PaleGreen3 #5fd75f 372,843,372 78 SeaGreen3 #5fd787 372,843,529 79 Aquamarine3 #5fd7af 372,843,686 80 MediumTurquoise #5fd7d7 372,843,843 81 SteelBlue1 #5fd7ff 372,843,1000 82 Chartreuse2 #5fff00 372,1000,0 83 SeaGreen2 #5fff5f 372,1000,372 84 SeaGreen1 #5fff87 372,1000,529 85 SeaGreen1 #5fffaf 372,1000,686 86 Aquamarine1 #5fffd7 372,1000,843 87 DarkSlateGray2 #5fffff 372,1000,1000 88 DarkRed #870000 529,0,0 89 DeepPink4 #87005f 529,0,372 90 DarkMagenta #870087 529,0,529 91 DarkMagenta #8700af 529,0,686 92 DarkViolet #8700d7 529,0,843 93 Purple #8700ff 529,0,1000 94 Orange4 #875f00 529,372,0 95 LightPink4 #875f5f 529,372,372 96 Plum4 #875f87 529,372,529 97 MediumPurple3 #875faf 529,372,686 98 MediumPurple3 #875fd7 529,372,843 99 SlateBlue1 #875fff 529,372,1000 100 Yellow4 #878700 529,529,0 101 Wheat4 #87875f 529,529,372 102 Grey53 #878787 529,529,529 103 LightSlateGrey #8787af 529,529,686 104 MediumPurple #8787d7 529,529,843 105 LightSlateBlue #8787ff 529,529,1000 106 Yellow4 #87af00 529,686,0 107 DarkOliveGreen3 #87af5f 529,686,372 108 DarkSeaGreen #87af87 529,686,529 109 LightSkyBlue3 #87afaf 529,686,686 110 LightSkyBlue3 #87afd7 529,686,843 111 SkyBlue2 #87afff 529,686,1000 112 Chartreuse2 #87d700 529,843,0 113 DarkOliveGreen3 #87d75f 529,843,372 114 PaleGreen3 #87d787 529,843,529 115 DarkSeaGreen3 #87d7af 529,843,686 116 DarkSlateGray3 #87d7d7 529,843,843 117 SkyBlue1 #87d7ff 529,843,1000 118 Chartreuse1 #87ff00 529,1000,0 119 LightGreen #87ff5f 529,1000,372 120 LightGreen #87ff87 529,1000,529 121 PaleGreen1 #87ffaf 529,1000,686 122 Aquamarine1 #87ffd7 529,1000,843 123 DarkSlateGray1 #87ffff 529,1000,1000 124 Red3 #af0000 686,0,0 125 DeepPink4 #af005f 686,0,372 126 MediumVioletRed #af0087 686,0,529 127 Magenta3 #af00af 686,0,686 128 DarkViolet #af00d7 686,0,843 129 Purple #af00ff 686,0,1000 130 DarkOrange3 #af5f00 686,372,0 131 IndianRed #af5f5f 686,372,372 132 HotPink3 #af5f87 686,372,529 133 MediumOrchid3 #af5faf 686,372,686 134 MediumOrchid #af5fd7 686,372,843 135 MediumPurple2 #af5fff 686,372,1000 136 DarkGoldenrod #af8700 686,529,0 137 LightSalmon3 #af875f 686,529,372 138 RosyBrown #af8787 686,529,529 139 Grey63 #af87af 686,529,686 140 MediumPurple2 #af87d7 686,529,843 141 MediumPurple1 #af87ff 686,529,1000 142 Gold3 #afaf00 686,686,0 143 DarkKhaki #afaf5f 686,686,372 144 NavajoWhite3 #afaf87 686,686,529 145 Grey69 #afafaf 686,686,686 146 LightSteelBlue3 #afafd7 686,686,843 147 LightSteelBlue #afafff 686,686,1000 148 Yellow3 #afd700 686,843,0 149 DarkOliveGreen3 #afd75f 686,843,372 150 DarkSeaGreen3 #afd787 686,843,529 151 DarkSeaGreen2 #afd7af 686,843,686 152 LightCyan3 #afd7d7 686,843,843 153 LightSkyBlue1 #afd7ff 686,843,1000 154 GreenYellow #afff00 686,1000,0 155 DarkOliveGreen2 #afff5f 686,1000,372 156 PaleGreen1 #afff87 686,1000,529 157 DarkSeaGreen2 #afffaf 686,1000,686 158 DarkSeaGreen1 #afffd7 686,1000,843 159 PaleTurquoise1 #afffff 686,1000,1000 160 Red3 #d70000 843,0,0 161 DeepPink3 #d7005f 843,0,372 162 DeepPink3 #d70087 843,0,529 163 Magenta3 #d700af 843,0,686 164 Magenta3 #d700d7 843,0,843 165 Magenta2 #d700ff 843,0,1000 166 DarkOrange3 #d75f00 843,372,0 167 IndianRed #d75f5f 843,372,372 168 HotPink3 #d75f87 843,372,529 169 HotPink2 #d75faf 843,372,686 170 Orchid #d75fd7 843,372,843 171 MediumOrchid1 #d75fff 843,372,1000 172 Orange3 #d78700 843,529,0 173 LightSalmon3 #d7875f 843,529,372 174 LightPink3 #d78787 843,529,529 175 Pink3 #d787af 843,529,686 176 Plum3 #d787d7 843,529,843 177 Violet #d787ff 843,529,1000 178 Gold3 #d7af00 843,686,0 179 LightGoldenrod3 #d7af5f 843,686,372 180 Tan #d7af87 843,686,529 181 MistyRose3 #d7afaf 843,686,686 182 Thistle3 #d7afd7 843,686,843 183 Plum2 #d7afff 843,686,1000 184 Yellow3 #d7d700 843,843,0 185 Khaki3 #d7d75f 843,843,372 186 LightGoldenrod2 #d7d787 843,843,529 187 LightYellow3 #d7d7af 843,843,686 188 Grey84 #d7d7d7 843,843,843 189 LightSteelBlue1 #d7d7ff 843,843,1000 190 Yellow2 #d7ff00 843,1000,0 191 DarkOliveGreen1 #d7ff5f 843,1000,372 192 DarkOliveGreen1 #d7ff87 843,1000,529 193 DarkSeaGreen1 #d7ffaf 843,1000,686 194 Honeydew2 #d7ffd7 843,1000,843 195 LightCyan1 #d7ffff 843,1000,1000 196 Red1 #ff0000 1000,0,0 197 DeepPink2 #ff005f 1000,0,372 198 DeepPink1 #ff0087 1000,0,529 199 DeepPink1 #ff00af 1000,0,686 200 Magenta2 #ff00d7 1000,0,843 201 Magenta1 #ff00ff 1000,0,1000 202 OrangeRed1 #ff5f00 1000,372,0 203 IndianRed1 #ff5f5f 1000,372,372 204 IndianRed1 #ff5f87 1000,372,529 205 HotPink #ff5faf 1000,372,686 206 HotPink #ff5fd7 1000,372,843 207 MediumOrchid1 #ff5fff 1000,372,1000 208 DarkOrange #ff8700 1000,529,0 209 Salmon1 #ff875f 1000,529,372 210 LightCoral #ff8787 1000,529,529 211 PaleVioletRed1 #ff87af 1000,529,686 212 Orchid2 #ff87d7 1000,529,843 213 Orchid1 #ff87ff 1000,529,1000 214 Orange1 #ffaf00 1000,686,0 215 SandyBrown #ffaf5f 1000,686,372 216 LightSalmon1 #ffaf87 1000,686,529 217 LightPink1 #ffafaf 1000,686,686 218 Pink1 #ffafd7 1000,686,843 219 Plum1 #ffafff 1000,686,1000 220 Gold1 #ffd700 1000,843,0 221 LightGoldenrod2 #ffd75f 1000,843,372 222 LightGoldenrod2 #ffd787 1000,843,529 223 NavajoWhite1 #ffd7af 1000,843,686 224 MistyRose1 #ffd7d7 1000,843,843 225 Thistle1 #ffd7ff 1000,843,1000 226 Yellow1 #ffff00 1000,1000,0 227 LightGoldenrod1 #ffff5f 1000,1000,372 228 Khaki1 #ffff87 1000,1000,529 229 Wheat1 #ffffaf 1000,1000,686 230 Cornsilk1 #ffffd7 1000,1000,843 231 Grey100 #ffffff 1000,1000,1000 232 Grey3 #080808 31,31,31 233 Grey7 #121212 70,70,70 234 Grey11 #1c1c1c 109,109,109 235 Grey15 #262626 149,149,149 236 Grey19 #303030 188,188,188 237 Grey23 #3a3a3a 227,227,227 238 Grey27 #444444 266,266,266 239 Grey30 #4e4e4e 305,305,305 240 Grey35 #585858 345,345,345 241 Grey39 #626262 384,384,384 242 Grey42 #6c6c6c 423,423,423 243 Grey46 #767676 462,462,462 244 Grey50 #808080 501,501,501 245 Grey54 #8a8a8a 541,541,541 246 Grey58 #949494 580,580,580 247 Grey62 #9e9e9e 619,619,619 248 Grey66 #a8a8a8 658,658,658 249 Grey70 #b2b2b2 698,698,698 250 Grey74 #bcbcbc 737,737,737 251 Grey78 #c6c6c6 776,776,776 252 Grey82 #d0d0d0 815,815,815 253 Grey85 #dadada 854,854,854 254 Grey89 #e4e4e4 894,894,894 255 Grey93 #eeeeee 933,933,933
Apk unzipping. Not a valid zip file, but unzip worked perfect
I have problem with golang zip reader. We have apk builder which build apk and then encrypted this apk with dex protector. When i try to open apk file, go return err "zip: not a valid zip file". But when i try to open with unzip or zipinfo, worked fine. (Also I noticed something interesting: in php 5.5 apk file opened with ZipArchive fine, but in 5.6 return error like golang) I try open to debug golang archive/zip pckage and found that error returned in this line https://github.com/golang/go/blob/master/src/archive/zip/reader.go#L269. Then i try to debug this line like this: fmt.Printf("%+v", f) fmt.Println( "len(f.Extra):", len(f.Extra), "len(b):", len(b), "tag:", tag, "size:", size, ) And have this results: &{FileHeader:{ Name:META-INF/MANIFEST.MF CreatorVersion:20 ReaderVersion:20 Flags:2056 Method:8 ModifiedTime:23093 ModifiedDate:18230 CRC32:2569174240 CompressedSize:359 UncompressedSize:570 CompressedSize64:359 UncompressedSize64:570 Extra:[] ExternalAttrs:0 Comment: } zipr:0xc208038038 zipsize:276893 headerOffset:0} &{FileHeader:{ Name:res/drawable/ic_launcher.png CreatorVersion:10 ReaderVersion:10 Flags:2048 Method:0 ModifiedTime:22030 ModifiedDate:18230 CRC32:2310989215 CompressedSize:20418 UncompressedSize:20418 CompressedSize64:20418 UncompressedSize64:20418 Extra:[] ExternalAttrs:0 Comment: } zipr:0xc208038038 zipsize:276893 headerOffset:2066} ... ---------------->PROBLEM HERE:-------------------=> &{FileHeader:{ Name:AndroidManifest.xml CreatorVersion:10 ReaderVersion:10 Flags:2048 Method:0 ModifiedTime:22030 ModifiedDate:18230 CRC32:1929033187 CompressedSize:7268 UncompressedSize:7268 CompressedSize64:7268 UncompressedSize64:7268 Extra:[204 94 136 27 148 14 49 95 154 4 231 53 127 242 4 199 243 207 229 161 103 90 116 69 45 31 54 220 113 43 172 4 159 109 211 57 217 2 161 17 196 16 168 19 174 138 107 15 36 181 223 22 15 21 106 153] ExternalAttrs:0 Comment: } zipr:0xc208038038 zipsize:276893 headerOffset:22954} len(f.Extra): 56 len(b): 52 tag: 24268 size: 7048 Problem in extra headers (i think this extra headers add dex protector when encrypt apk). Size more then length of extra bytes. But unzip util and ZipArchive in php5.5 woked perfect with this extras. How to fix this ? I want to rewrite this piece in go archive/zip package, so that the extras do not interfere golang zip reader. It's very important for me, because i rewrite build apk from php to golang and I came across a problem that can not decide :(. Please help me.
Calculation of application speedup using gnuplot and awk
Here's the problem: Speedup formula: S(p) = T(1)/T(p) = (avg time for one process / avg time for p processes) There are 5 logs, from which one wants to extract the information. cg.B.1.log contains the execution times for one process, so we do the calculation of the average time to obtain T(1). The other log files contain the execution times for 2, 4, 8 and 16 processes. Averages of those times must also be calculated, since they are T(p). Here's the code that calculates the averages: tavg(n) = "awk 'BEGIN { FS = \"[ \\t]*=[ \\t]*\" } /Time in seconds/ { s += $2; c++ } /Total processes/ { if (! CP) CP = $2 } END { print s/c }' cg.B.".n.".log ".(n == 1 ? ">" : ">>")." tavg.dat;" And the code that calculates the speedup: system "awk 'NR==1{n=$0} {print n/$0}' tavg.dat > speedup.dat;" How do I combine those two commands so that the output 'speedup.dat' is produced directly without using file tavg.dat? Here are the contents of files, the structure of all log files is identical. I attached only the first two executions for abbreviation purposes. cg.B.1.log -/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/- Start in 16:45:15--25/12/2014 NAS Parallel Benchmarks 3.3 -- CG Benchmark Size: 75000 Iterations: 75 Number of active processes: 1 Number of nonzeroes per row: 13 Eigenvalue shift: .600E+02 iteration ||r|| zeta 1 0.30354859861452E-12 59.9994751578754 2 0.11186435488267E-14 21.7627846142536 3 0.11312258511928E-14 22.2876617043224 4 0.11222160585284E-14 22.5230738188346 5 0.11244234177219E-14 22.6275390653892 6 0.11330434819384E-14 22.6740259189533 7 0.11334259623050E-14 22.6949056826251 8 0.11374839313647E-14 22.7044023166872 9 0.11424877443039E-14 22.7087834345620 10 0.11329475190566E-14 22.7108351397177 11 0.11337364093482E-14 22.7118107121341 12 0.11379928308864E-14 22.7122816240971 13 0.11369453681794E-14 22.7125122663243 14 0.11430390337015E-14 22.7126268007594 15 0.11400318886400E-14 22.7126844161819 16 0.11352091331197E-14 22.7127137461755 17 0.11350923439124E-14 22.7127288402000 18 0.11475378864565E-14 22.7127366848296 19 0.11366777929028E-14 22.7127407981217 20 0.11274243312504E-14 22.7127429721364 21 0.11353930792856E-14 22.7127441294025 22 0.11299685800278E-14 22.7127447493900 23 0.11296405041170E-14 22.7127450834533 24 0.11381975597887E-14 22.7127452643881 25 0.11328127301663E-14 22.7127453628451 26 0.11367332658939E-14 22.7127454166517 27 0.11283372178605E-14 22.7127454461696 28 0.11384734158863E-14 22.7127454624211 29 0.11394011989719E-14 22.7127454713974 30 0.11354294067640E-14 22.7127454763703 31 0.11412988029103E-14 22.7127454791343 32 0.11358088407717E-14 22.7127454806740 33 0.11263266152515E-14 22.7127454815316 34 0.11275183080286E-14 22.7127454820131 35 0.11328306951409E-14 22.7127454822840 36 0.11357880314891E-14 22.7127454824349 37 0.11332687790488E-14 22.7127454825202 38 0.11324108818137E-14 22.7127454825684 39 0.11365065523777E-14 22.7127454825967 40 0.11361185361321E-14 22.7127454826116 41 0.11276519820716E-14 22.7127454826202 42 0.11317183424878E-14 22.7127454826253 43 0.11236007481770E-14 22.7127454826276 44 0.11304065564684E-14 22.7127454826296 45 0.11287791356431E-14 22.7127454826310 46 0.11297028000133E-14 22.7127454826310 47 0.11281236869666E-14 22.7127454826314 48 0.11277254075548E-14 22.7127454826317 49 0.11320327289847E-14 22.7127454826309 50 0.11287655285563E-14 22.7127454826321 51 0.11230503422400E-14 22.7127454826324 52 0.11292089094944E-14 22.7127454826313 53 0.11366728396408E-14 22.7127454826315 54 0.11222618466968E-14 22.7127454826310 55 0.11278193276516E-14 22.7127454826315 56 0.11244624896030E-14 22.7127454826316 57 0.11264508872685E-14 22.7127454826318 58 0.11255583774760E-14 22.7127454826314 59 0.11227129146723E-14 22.7127454826314 60 0.11189480800173E-14 22.7127454826318 61 0.11163241472678E-14 22.7127454826315 62 0.11278839424218E-14 22.7127454826318 63 0.11226804133008E-14 22.7127454826313 64 0.11222456601361E-14 22.7127454826317 65 0.11270879524310E-14 22.7127454826308 66 0.11303771390006E-14 22.7127454826319 67 0.11240101357287E-14 22.7127454826319 68 0.11240278884391E-14 22.7127454826321 69 0.11207748067718E-14 22.7127454826317 70 0.11178755187571E-14 22.7127454826327 71 0.11195935245649E-14 22.7127454826313 72 0.11260715126337E-14 22.7127454826322 73 0.11281677964997E-14 22.7127454826316 74 0.11162340034815E-14 22.7127454826318 75 0.11208709203921E-14 22.7127454826310 Benchmark completed VERIFICATION SUCCESSFUL Zeta is 0.2271274548263E+02 Error is 0.3128387698896E-15 CG Benchmark Completed. Class = B Size = 75000 Iterations = 75 Time in seconds = 88.72 Total processes = 1 Compiled procs = 1 Mop/s total = 616.64 Mop/s/process = 616.64 Operation type = floating point Verification = SUCCESSFUL Version = 3.3 Compile date = 25 Dec 2014 Compile options: MPIF77 = mpif77 FLINK = $(MPIF77) FMPI_LIB = -L/usr/lib/openmpi/lib -lmpi -lopen-rte -lo... FMPI_INC = -I/usr/lib/openmpi/include -I/usr/lib/openm... FFLAGS = -O FLINKFLAGS = -O RAND = randi8 Please send the results of this run to: NPB Development Team Internet: npb#nas.nasa.gov If email is not available, send this to: MS T27A-1 NASA Ames Research Center Moffett Field, CA 94035-1000 Fax: 650-604-3957 Finish in 16:46:46--25/12/2014 -/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/- -/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/- Start in 17:03:13--25/12/2014 NAS Parallel Benchmarks 3.3 -- CG Benchmark Size: 75000 Iterations: 75 Number of active processes: 1 Number of nonzeroes per row: 13 Eigenvalue shift: .600E+02 iteration ||r|| zeta 1 0.30354859861452E-12 59.9994751578754 2 0.11186435488267E-14 21.7627846142536 3 0.11312258511928E-14 22.2876617043224 4 0.11222160585284E-14 22.5230738188346 5 0.11244234177219E-14 22.6275390653892 6 0.11330434819384E-14 22.6740259189533 7 0.11334259623050E-14 22.6949056826251 8 0.11374839313647E-14 22.7044023166872 9 0.11424877443039E-14 22.7087834345620 10 0.11329475190566E-14 22.7108351397177 11 0.11337364093482E-14 22.7118107121341 12 0.11379928308864E-14 22.7122816240971 13 0.11369453681794E-14 22.7125122663243 14 0.11430390337015E-14 22.7126268007594 15 0.11400318886400E-14 22.7126844161819 16 0.11352091331197E-14 22.7127137461755 17 0.11350923439124E-14 22.7127288402000 18 0.11475378864565E-14 22.7127366848296 19 0.11366777929028E-14 22.7127407981217 20 0.11274243312504E-14 22.7127429721364 21 0.11353930792856E-14 22.7127441294025 22 0.11299685800278E-14 22.7127447493900 23 0.11296405041170E-14 22.7127450834533 24 0.11381975597887E-14 22.7127452643881 25 0.11328127301663E-14 22.7127453628451 26 0.11367332658939E-14 22.7127454166517 27 0.11283372178605E-14 22.7127454461696 28 0.11384734158863E-14 22.7127454624211 29 0.11394011989719E-14 22.7127454713974 30 0.11354294067640E-14 22.7127454763703 31 0.11412988029103E-14 22.7127454791343 32 0.11358088407717E-14 22.7127454806740 33 0.11263266152515E-14 22.7127454815316 34 0.11275183080286E-14 22.7127454820131 35 0.11328306951409E-14 22.7127454822840 36 0.11357880314891E-14 22.7127454824349 37 0.11332687790488E-14 22.7127454825202 38 0.11324108818137E-14 22.7127454825684 39 0.11365065523777E-14 22.7127454825967 40 0.11361185361321E-14 22.7127454826116 41 0.11276519820716E-14 22.7127454826202 42 0.11317183424878E-14 22.7127454826253 43 0.11236007481770E-14 22.7127454826276 44 0.11304065564684E-14 22.7127454826296 45 0.11287791356431E-14 22.7127454826310 46 0.11297028000133E-14 22.7127454826310 47 0.11281236869666E-14 22.7127454826314 48 0.11277254075548E-14 22.7127454826317 49 0.11320327289847E-14 22.7127454826309 50 0.11287655285563E-14 22.7127454826321 51 0.11230503422400E-14 22.7127454826324 52 0.11292089094944E-14 22.7127454826313 53 0.11366728396408E-14 22.7127454826315 54 0.11222618466968E-14 22.7127454826310 55 0.11278193276516E-14 22.7127454826315 56 0.11244624896030E-14 22.7127454826316 57 0.11264508872685E-14 22.7127454826318 58 0.11255583774760E-14 22.7127454826314 59 0.11227129146723E-14 22.7127454826314 60 0.11189480800173E-14 22.7127454826318 61 0.11163241472678E-14 22.7127454826315 62 0.11278839424218E-14 22.7127454826318 63 0.11226804133008E-14 22.7127454826313 64 0.11222456601361E-14 22.7127454826317 65 0.11270879524310E-14 22.7127454826308 66 0.11303771390006E-14 22.7127454826319 67 0.11240101357287E-14 22.7127454826319 68 0.11240278884391E-14 22.7127454826321 69 0.11207748067718E-14 22.7127454826317 70 0.11178755187571E-14 22.7127454826327 71 0.11195935245649E-14 22.7127454826313 72 0.11260715126337E-14 22.7127454826322 73 0.11281677964997E-14 22.7127454826316 74 0.11162340034815E-14 22.7127454826318 75 0.11208709203921E-14 22.7127454826310 Benchmark completed VERIFICATION SUCCESSFUL Zeta is 0.2271274548263E+02 Error is 0.3128387698896E-15 CG Benchmark Completed. Class = B Size = 75000 Iterations = 75 Time in seconds = 87.47 Total processes = 1 Compiled procs = 1 Mop/s total = 625.43 Mop/s/process = 625.43 Operation type = floating point Verification = SUCCESSFUL Version = 3.3 Compile date = 25 Dec 2014 Compile options: MPIF77 = mpif77 FLINK = $(MPIF77) FMPI_LIB = -L/usr/lib/openmpi/lib -lmpi -lopen-rte -lo... FMPI_INC = -I/usr/lib/openmpi/include -I/usr/lib/openm... FFLAGS = -O FLINKFLAGS = -O RAND = randi8 Please send the results of this run to: NPB Development Team Internet: npb#nas.nasa.gov If email is not available, send this to: MS T27A-1 NASA Ames Research Center Moffett Field, CA 94035-1000 Fax: 650-604-3957 Finish in 17:04:43--25/12/2014 -/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/-/- tavg.dat 88.3055 45.1482 37.7202 37.4035 53.777 speedup.dat 1 1.9559 2.34107 2.36089 1.64207
You can do it all in one awk script that processes all the log files: #!/usr/bin/awk -f BEGIN { FS="=" } lfname != FILENAME { lfname = FILENAME; split(FILENAME, a, "."); fnum=a[3] } /Time in seconds/ { tsecs[fnum] += $2; tcnt[fnum]++ } /Total processes/ { cp[fnum] = int($2) } END { tavg1 = tsecs[1]/tcnt[1] for( k in tsecs ) { tavgk = tsecs[k]/tcnt[k] if( tavgk > 0 ) { print k OFS cp[k] OFS tavgk OFS tavg1/tavgk } } } If you put that in a file called awk.script and make it executable with chmod +x awk.script you can run it in bash like: ./awk.script cg.B.*.log If you're using GNU awk, the output will be ordered( extra steps may be needed to ensure the output is ordered using other awk flavors ). Where I generated a 2nd and 3rd file, the output is like: 1 1 88.095 1 2 2 68.095 1.29371 3 4 49.595 1.77629 where the unnamed columns are like: file number, # processes, avg per file, speedup. You could get just the speedups by changing the print in the END block to be like print tavg1/tavgk. Here's a breakdown of the script: Use a simpler field separator in BEGIN lfname != FILENAME - parse out file number from the filename as fnum but only when the FILENAME changes. /Time in seconds/ - store the values in tsecs and tcnt arrays with an fnum key. Use int() function to strip whitespace from processes value. /Total processes/ - store the process in the cp array with an fnum key END - Calculate the average for fnum 1 as tavg1, loop through the keys in tsecs and calculate the average by fnum key as tavgk. When tavgk > 0 print the output as described above.
You have figured out all the difficult parts already. You don't need the tavg.dat file at all. Create your tavg(n) function directly as a system call: tavg(n) = system("awk 'BEGIN { FS = \"[ \\t]*=[ \\t]*\" } \ /Time in seconds/ { s += $2; c++ } /Total processes/ { \ if (! CP) CP = $2 } END { print s/c }' cg.B.".n.".log") And a speedup(n) function as speedup(n)=tavg(n)/tavg(1) Now you can set print to write to a file: set print "speedup.dat" do for [i=1:5] { print speedup(i) } unset print
Why is RLIMIT_NOFILE rlim_max of -1 on BSD?
In the following code: 139 struct rlimit limit; 140 141 method = "rlimit"; 142 if (getrlimit(RLIMIT_NOFILE, &limit) < 0) { 143 perror("calling getrlimit"); 144 exit(1); 145 } 146 147 /* set the current to the maximum or specified value */ 148 if (max_desired_fds) 149 limit.rlim_cur = max_desired_fds; 150 else { 151 limit.rlim_cur = limit.rlim_max; 152 } 153 154 if (setrlimit(RLIMIT_NOFILE, &limit) < 0) { 155 perror("calling setrlimit"); 156 exit(1); 157 } the setrlimit line fails (I get the error "calling setrlimit"). Further investigation shows that limit.rlim_max is -1, which is not a valid value. Any ideas why would this be? This is on Mac OSX.
If setrlimit fails, try again with rlim_cur set to OPEN_MAX. For example, see http://source.winehq.org/source/libs/wine/loader.c#L653. (The comment mentioning Leopard means that Leopard first introduced that behavior. Read it as Leopard-and-later.) ETA: See the note in COMPATIBILITY in the setrlimit(2) man page.