Why is RLIMIT_NOFILE rlim_max of -1 on BSD? - macos

In the following code:
139 struct rlimit limit;
140
141 method = "rlimit";
142 if (getrlimit(RLIMIT_NOFILE, &limit) < 0) {
143 perror("calling getrlimit");
144 exit(1);
145 }
146
147 /* set the current to the maximum or specified value */
148 if (max_desired_fds)
149 limit.rlim_cur = max_desired_fds;
150 else {
151 limit.rlim_cur = limit.rlim_max;
152 }
153
154 if (setrlimit(RLIMIT_NOFILE, &limit) < 0) {
155 perror("calling setrlimit");
156 exit(1);
157 }
the setrlimit line fails (I get the error "calling setrlimit"). Further investigation shows that limit.rlim_max is -1, which is not a valid value. Any ideas why would this be? This is on Mac OSX.


If setrlimit fails, try again with rlim_cur set to OPEN_MAX. For example, see http://source.winehq.org/source/libs/wine/loader.c#L653. (The comment mentioning Leopard means that Leopard first introduced that behavior. Read it as Leopard-and-later.)
ETA: See the note in COMPATIBILITY in the setrlimit(2) man page.

Related

Cannot run pool.amap() twice in pathos.multiprocessing

I want to parallelize simulation of multiple agents. Because I want my results to be instances of a class, to avoid issues with serialization I use pathos.multiprocessing instead of multiprocessing.
I do it like this:
import pathos.multiprocessing as mp
sim_agent(T): #simulate single agent for T periods
ag = agent()
for t in range(T):
ag.step()
return ag
def simulate_parallel(N, T):
if __name__ == '__main__':
pool = mp.ProcessPool()
results = pool.amap(sim_agent, [T]*N)
agents = results.get()
return agents
I can run simulate_parallel once. When I do it again, I get an error:
--> 142 agents = results.get()
~/anaconda3/anaconda3/lib/python3.7/site-packages/multiprocess/pool.py in get(self, timeout)
655 return self._value
656 else:
--> 657 raise self._value
658
659 def _set(self, i, obj):
~/anaconda3/anaconda3/lib/python3.7/site-packages/multiprocess/pool.py in _handle_tasks(taskqueue, put, outqueue, pool, cache)
429 break
430 try:
--> 431 put(task)
432 except Exception as e:
433 job, idx = task[:2]
~/anaconda3/anaconda3/lib/python3.7/site-packages/multiprocess/connection.py in send(self, obj)
207 self._check_closed()
208 self._check_writable()
--> 209 self._send_bytes(_ForkingPickler.dumps(obj))
210
211 def recv_bytes(self, maxlength=None):
~/anaconda3/anaconda3/lib/python3.7/site-packages/multiprocess/connection.py in _send_bytes(self, buf)
394 n = len(buf)
395 # For wire compatibility with 3.2 and lower
--> 396 header = struct.pack("!i", n)
397 if n > 16384:
398 # The payload is large so Nagle's algorithm won't be triggered
error: 'i' format requires -2147483648 <= number <= 2147483647
I also get the same error when I try to use pool.amap() in other functions. Even when I make N and T small, it is still the same behavior: I can run simulate_parallel once and cannot do it for the second time. Why could that be? Thanks!

Win Error 5 has occurred when I tried to execute Pytorch code

I'm new to Anaconda. I tried to execute Pytorch Adversarial Neural Network in anaconda. It shows some error that I have no clue. Here is the code that downloads dataset
# MNIST Test dataset and dataloader declaration
test_loader = torch.utils.data.DataLoader(datasets.MNIST('../data', train=False, download=True, transform=transforms.Compose([
transforms.ToTensor(),])),batch_size=1, shuffle=True)
This is the error message I got :
PermissionError Traceback (most recent call last)
<ipython-input-4-59310f6a37f8> in <module>
41
42 # MNIST Test dataset and dataloader declaration
43 test_loader = torch.utils.data.DataLoader(datasets.MNIST('../data', train=False, download=True, transform=transforms.Compose([transforms.ToTensor(),])),batch_size=1, shuffle=True)
44
45 # Define what device we are using
~\anaconda3\lib\site-packages\torchvision\datasets\mnist.py in __init__(self, root,
train,
transform, target_transform, download)
77
78 if download:
79 self.download()
80
81 if not self._check_exists():
~\anaconda3\lib\site-packages\torchvision\datasets\mnist.py in download(self)
138 return
139
140 os.makedirs(self.raw_folder, exist_ok=True)
141 os.makedirs(self.processed_folder, exist_ok=True)
142
~\anaconda3\lib\os.py in makedirs(name, mode, exist_ok)
211 if head and tail and not path.exists(head):
212 try:
213 makedirs(head, exist_ok=exist_ok)
214 except FileExistsError:
215 # Defeats race condition when another thread created the path
~\anaconda3\lib\os.py in makedirs(name, mode, exist_ok)
211 if head and tail and not path.exists(head):
212 try:
213 makedirs(head, exist_ok=exist_ok)
214 except FileExistsError:
215 # Defeats race condition when another thread created the path
~\anaconda3\lib\os.py in makedirs(name, mode, exist_ok)
221 return
222 try:
223 mkdir(name, mode)
224 except OSError:
225 # Cannot rely on checking for EEXIST, since the operating system
PermissionError: [WinError 5] Access is denied: '../data' .

How about changing the '../data' to another directory, like '/Users/***/Downloads/data' or somewhere else.

AWS Elasticsearch TransportError(413, '')

We are using the python elasticsearch API version "elasticsearch-6.8.0"
With AWS Elasticsearch version 6.7
It only happens sometimes, on some machines (even when we make the queries within seconds of each other from different machines, one will succeed and the other will throw the error). Thus this is difficult to reproduce. We are using a paginated search like this:
def scrolled_search_query(self, index, query, size=es_config.ES_SIZE,
scroll=es_config.ES_SCROLL,
request_timeout=es_config.ES_TIMEOUT,
docs_only=False):
data = self.es_conn.search(index=index, size=size, scroll=scroll, body=query,
request_timeout=request_timeout)
while len(data['hits']['hits']) > 0:
for hit in data['hits']['hits']:
if docs_only:
yield hit['_source']
else:
yield hit
data = self.es_conn.scroll(scroll_id=data['_scroll_id'], scroll=scroll,
request_timeout=request_timeout)
The error message is utterly useless, just basically saying that there's a 413. like this:
TransportError: TransportError(413, '')
From looking around at forums, it seems the problem is because ES is attempting to return a large payload. However, since we're using scroll pagination, I don't know why this would be an issue because each payload should be well below the 100MB limit. Since the error message is so terrible, I can't really see what it's trying to do in the database. It seems like the database is calculating the total data for the query and rejecting it without taking into account pagination. It's normal for us to request large payloads, and paginate them.
The full exception looks like this, what else do you need?
---------------------------------------------------------------------------
TransportError Traceback (most recent call last)
<ipython-input-3-a3fd964bf999> in <module>
41 blah=blah,
42 blah=blah,
---> 43 start_date=start_date)
44
45 df = df.drop(['#blah', '#blah'], axis=1)
~/Documents/Repositories/blah/blah/downloader.py in get_data(self, data_type, blah, blah, device, blah, start_date, end_date)
186 return self.gatorEs.get_blah(es_requests=self.es_requests, blah=blah, blah=blah, blah=blah,
--> 188 start_date=start_date, end_date=end_date)
189
190 if data_type == 'blah':
~/Documents/Repositories/gator_access/gator_access/es_gator_api.py in get_blah(self, es_requests, blah, blah, blah, blah, blah, start_date, end_date)
180 logger=self.logger)
181 if not data: continue
--> 182 es_data += data
183
184 self.logger.info(" ".join(["Download took {t} seconds for p:'{p}' d:'{d}'",
~/Documents/Repositories/gator_core/gator_core/elasticsearch/es_requests.py in scrolled_search_query(self, index, query, size, scroll, request_timeout, docs_only)
144
145 data = self.es_conn.scroll(scroll_id=data['_scroll_id'], scroll=scroll,
--> 146 request_timeout=request_timeout)
147
148
~/anaconda3/envs/prognostic/lib/python3.6/site-packages/elasticsearch/client/utils.py in _wrapped(*args, **kwargs)
82 if p in kwargs:
83 params[p] = kwargs.pop(p)
---> 84 return func(*args, params=params, **kwargs)
85
86 return _wrapped
~/anaconda3/envs/prognostic/lib/python3.6/site-packages/elasticsearch/client/__init__.py in scroll(self, scroll_id, body, params)
1353 """
1354 return self.transport.perform_request(
-> 1355 "GET", _make_path("_search", "scroll", scroll_id), params=params, body=body
1356 )
1357
~/anaconda3/envs/prognostic/lib/python3.6/site-packages/elasticsearch/transport.py in perform_request(self, method, url, headers, params, body)
351 headers=headers,
352 ignore=ignore,
--> 353 timeout=timeout,
354 )
355
~/anaconda3/envs/prognostic/lib/python3.6/site-packages/elasticsearch/connection/http_requests.py in perform_request(self, method, url, params, body, timeout, ignore, headers)
139 raw_data,
140 )
--> 141 self._raise_error(response.status_code, raw_data)
142
143 self.log_request_success(
~/anaconda3/envs/prognostic/lib/python3.6/site-packages/elasticsearch/connection/base.py in _raise_error(self, status_code, raw_data)
160
161 raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
--> 162 status_code, error_message, additional_info
163 )
TransportError: TransportError(413, '')

ncurses: init_color() has no effect

Trying to define color pairs, I was getting strange results. All 256 colors are already defined, and attempt to change any color with init_color() has no affect.
I'm using Putty with 256-colors enabled and TERM=xterm-256color (also putty-256color), ncurses 6.0 compiled with --enable-widec and --enable-ext-colors. This shows all colors are defined and the init_color() doesn't change anything even though it succeeds:
init_scr();
start_color();
if (has_colors() && COLORS == 256 && can_change_color()) {
NCURSES_COLOR_T f;
for (f = 1; f < 256; f++) {
if (init_pair(f, f, COLOR_BLACK) == ERR) break;
attron(COLOR_PAIR(f));
printw("(%d)", f);
attroff(COLOR_PAIR(f));
refresh();
}
getch();
clear();
for (f = 1; f < 256; f++) {
if (init_color(f, 0, 0, f*3) == ERR) break;
if (init_pair(f, f, COLOR_BLACK) == ERR) break;
attron(COLOR_PAIR(f));
printw("(%d)", f);
attroff(COLOR_PAIR(f));
refresh();
}
getch();
clear();
}
I've read that the default colors can't be changed, but only refers to COLOR_BLACK, etc (0-7).
Where are these 256 default colors defined and why can't I change them? If they can't be changed, I could make use of the colors defined, but only if I can rely on them being the same on any 256-color capable terminal.

short: PuTTY doesn't do that, ncurses can't tell if PuTTY can...
long:
In ncurses, the init_color function checks its parameters (in the example given, those appear okay if your $TERM is "xterm-256color"), as well as checking if the terminal description has the initc (initialize_color) capability. If that is missing or cancelled, ncurses returns an error.
However, that's only the terminal description. ncurses cannot tell if you have chosen an incorrect or inappropriate terminal description.
In a quick check, PuTTY doesn't respond to the control sequence which is used in initc. This is a known limitation, as indicated in the (more appropriate) terminal description putty-256color provided by ncurses:
putty-256color|PuTTY 0.58 with xterm 256-colors,
use=xterm+256setaf, use=putty,
That xterm+256setaf is used for terminals whose palette is hard-coded. PuTTY is not the only terminal which both sets TERM=xterm` and lacks the ability to change its palette. If you happen to be using an old version of the terminal database, you may be misled, since that error was fixed in 2014:
# 2014-03-30
# * cancel ccc in putty-256color and konsole-256color for consistency
# with the cancelled initc capability (patch by Sven Zuhlsdorf).
# * add xterm+256setaf building block for various terminals which only
# get the 256-color feature half-implemented -TD
# * updated "st" entry (leaving the 0.1.1 version as "simpleterm") to
# 0.4.1 -TD
#
Like the other terminals whose developers set TERM=xterm (or TERM=xterm-256color), there are differences between those and xterm.
Further reading:
Why not just use TERM set to "xterm"?

Couple of things I discovered. First, yes I was apparently referencing an old putty-256color terminfo that had "ccc", allowing can_change_color() to succeed, but then init_color() would fail. But the same Putty window using "xterm-256color" would init_color() OK and color_content() even shows the new values, but nothing changed on the screen. What was really confusing is sometimes the colors I set would appear and other times seemingly random colors appeared instead.
Here's what I found:
putty-256color xterm-256color gnome-256color xterm-256color
(putty) (putty) (gnome-terminal) (MobaXterm)
-------------- -------------- ---------------- --------------
change_color OK/ERR OK OK OK
init_color ERR ERR OK OK
color_content ERR OK/NOCH OK OK
color changed? NO NO YES YES
So there's basically no way to determine whether colors can be changed or not. But I did find that every terminal had already defined the standard 256 xterm colors, whether they could be changed or not. So, now, I just define the colors I want to use using the same color numbers as in the xterm palette. That way, the colors I expect will appear whether I needed to define them or not. So, to use "PaleGreen3", I just use:
init_color(77, 372, 843, 372)
If it works, it works, and if not, its probably already defined. For reference, I converting all the Xwindow/xterm colors from GUI hex notation to the ncurses (0-1000) values:
# Name Tk Ncurses
--- ---------------- ------- -------------
16 Grey0 #000000 0,0,0
17 NavyBlue #00005f 0,0,372
18 DarkBlue #000087 0,0,529
19 Blue3 #0000af 0,0,686
20 Blue3 #0000d7 0,0,843
21 Blue1 #0000ff 0,0,1000
22 DarkGreen #005f00 0,372,0
23 DeepSkyBlue4 #005f5f 0,372,372
24 DeepSkyBlue4 #005f87 0,372,529
25 DeepSkyBlue4 #005faf 0,372,686
26 DodgerBlue3 #005fd7 0,372,843
27 DodgerBlue2 #005fff 0,372,1000
28 Green4 #008700 0,529,0
29 SpringGreen4 #00875f 0,529,372
30 Turquoise4 #008787 0,529,529
31 DeepSkyBlue3 #0087af 0,529,686
32 DeepSkyBlue3 #0087d7 0,529,843
33 DodgerBlue1 #0087ff 0,529,1000
34 Green3 #00af00 0,686,0
35 SpringGreen3 #00af5f 0,686,372
36 DarkCyan #00af87 0,686,529
37 LightSeaGreen #00afaf 0,686,686
38 DeepSkyBlue2 #00afd7 0,686,843
39 DeepSkyBlue1 #00afff 0,686,1000
40 Green3 #00d700 0,843,0
41 SpringGreen3 #00d75f 0,843,372
42 SpringGreen2 #00d787 0,843,529
43 Cyan3 #00d7af 0,843,686
44 DarkTurquoise #00d7d7 0,843,843
45 Turquoise2 #00d7ff 0,843,1000
46 Green1 #00ff00 0,1000,0
47 SpringGreen2 #00ff5f 0,1000,372
48 SpringGreen1 #00ff87 0,1000,529
49 MediumSpringGreen #00ffaf 0,1000,686
50 Cyan2 #00ffd7 0,1000,843
51 Cyan1 #00ffff 0,1000,1000
52 DarkRed #5f0000 372,0,0
53 DeepPink4 #5f005f 372,0,372
54 Purple4 #5f0087 372,0,529
55 Purple4 #5f00af 372,0,686
56 Purple3 #5f00d7 372,0,843
57 BlueViolet #5f00ff 372,0,1000
58 Orange4 #5f5f00 372,372,0
59 Grey37 #5f5f5f 372,372,372
60 MediumPurple4 #5f5f87 372,372,529
61 SlateBlue3 #5f5faf 372,372,686
62 SlateBlue3 #5f5fd7 372,372,843
63 RoyalBlue1 #5f5fff 372,372,1000
64 Chartreuse4 #5f8700 372,529,0
65 DarkSeaGreen4 #5f875f 372,529,372
66 PaleTurquoise4 #5f8787 372,529,529
67 SteelBlue #5f87af 372,529,686
68 SteelBlue3 #5f87d7 372,529,843
69 CornflowerBlue #5f87ff 372,529,1000
70 Chartreuse3 #5faf00 372,686,0
71 DarkSeaGreen4 #5faf5f 372,686,372
72 CadetBlue #5faf87 372,686,529
73 CadetBlue #5fafaf 372,686,686
74 SkyBlue3 #5fafd7 372,686,843
75 SteelBlue1 #5fafff 372,686,1000
76 Chartreuse3 #5fd700 372,843,0
77 PaleGreen3 #5fd75f 372,843,372
78 SeaGreen3 #5fd787 372,843,529
79 Aquamarine3 #5fd7af 372,843,686
80 MediumTurquoise #5fd7d7 372,843,843
81 SteelBlue1 #5fd7ff 372,843,1000
82 Chartreuse2 #5fff00 372,1000,0
83 SeaGreen2 #5fff5f 372,1000,372
84 SeaGreen1 #5fff87 372,1000,529
85 SeaGreen1 #5fffaf 372,1000,686
86 Aquamarine1 #5fffd7 372,1000,843
87 DarkSlateGray2 #5fffff 372,1000,1000
88 DarkRed #870000 529,0,0
89 DeepPink4 #87005f 529,0,372
90 DarkMagenta #870087 529,0,529
91 DarkMagenta #8700af 529,0,686
92 DarkViolet #8700d7 529,0,843
93 Purple #8700ff 529,0,1000
94 Orange4 #875f00 529,372,0
95 LightPink4 #875f5f 529,372,372
96 Plum4 #875f87 529,372,529
97 MediumPurple3 #875faf 529,372,686
98 MediumPurple3 #875fd7 529,372,843
99 SlateBlue1 #875fff 529,372,1000
100 Yellow4 #878700 529,529,0
101 Wheat4 #87875f 529,529,372
102 Grey53 #878787 529,529,529
103 LightSlateGrey #8787af 529,529,686
104 MediumPurple #8787d7 529,529,843
105 LightSlateBlue #8787ff 529,529,1000
106 Yellow4 #87af00 529,686,0
107 DarkOliveGreen3 #87af5f 529,686,372
108 DarkSeaGreen #87af87 529,686,529
109 LightSkyBlue3 #87afaf 529,686,686
110 LightSkyBlue3 #87afd7 529,686,843
111 SkyBlue2 #87afff 529,686,1000
112 Chartreuse2 #87d700 529,843,0
113 DarkOliveGreen3 #87d75f 529,843,372
114 PaleGreen3 #87d787 529,843,529
115 DarkSeaGreen3 #87d7af 529,843,686
116 DarkSlateGray3 #87d7d7 529,843,843
117 SkyBlue1 #87d7ff 529,843,1000
118 Chartreuse1 #87ff00 529,1000,0
119 LightGreen #87ff5f 529,1000,372
120 LightGreen #87ff87 529,1000,529
121 PaleGreen1 #87ffaf 529,1000,686
122 Aquamarine1 #87ffd7 529,1000,843
123 DarkSlateGray1 #87ffff 529,1000,1000
124 Red3 #af0000 686,0,0
125 DeepPink4 #af005f 686,0,372
126 MediumVioletRed #af0087 686,0,529
127 Magenta3 #af00af 686,0,686
128 DarkViolet #af00d7 686,0,843
129 Purple #af00ff 686,0,1000
130 DarkOrange3 #af5f00 686,372,0
131 IndianRed #af5f5f 686,372,372
132 HotPink3 #af5f87 686,372,529
133 MediumOrchid3 #af5faf 686,372,686
134 MediumOrchid #af5fd7 686,372,843
135 MediumPurple2 #af5fff 686,372,1000
136 DarkGoldenrod #af8700 686,529,0
137 LightSalmon3 #af875f 686,529,372
138 RosyBrown #af8787 686,529,529
139 Grey63 #af87af 686,529,686
140 MediumPurple2 #af87d7 686,529,843
141 MediumPurple1 #af87ff 686,529,1000
142 Gold3 #afaf00 686,686,0
143 DarkKhaki #afaf5f 686,686,372
144 NavajoWhite3 #afaf87 686,686,529
145 Grey69 #afafaf 686,686,686
146 LightSteelBlue3 #afafd7 686,686,843
147 LightSteelBlue #afafff 686,686,1000
148 Yellow3 #afd700 686,843,0
149 DarkOliveGreen3 #afd75f 686,843,372
150 DarkSeaGreen3 #afd787 686,843,529
151 DarkSeaGreen2 #afd7af 686,843,686
152 LightCyan3 #afd7d7 686,843,843
153 LightSkyBlue1 #afd7ff 686,843,1000
154 GreenYellow #afff00 686,1000,0
155 DarkOliveGreen2 #afff5f 686,1000,372
156 PaleGreen1 #afff87 686,1000,529
157 DarkSeaGreen2 #afffaf 686,1000,686
158 DarkSeaGreen1 #afffd7 686,1000,843
159 PaleTurquoise1 #afffff 686,1000,1000
160 Red3 #d70000 843,0,0
161 DeepPink3 #d7005f 843,0,372
162 DeepPink3 #d70087 843,0,529
163 Magenta3 #d700af 843,0,686
164 Magenta3 #d700d7 843,0,843
165 Magenta2 #d700ff 843,0,1000
166 DarkOrange3 #d75f00 843,372,0
167 IndianRed #d75f5f 843,372,372
168 HotPink3 #d75f87 843,372,529
169 HotPink2 #d75faf 843,372,686
170 Orchid #d75fd7 843,372,843
171 MediumOrchid1 #d75fff 843,372,1000
172 Orange3 #d78700 843,529,0
173 LightSalmon3 #d7875f 843,529,372
174 LightPink3 #d78787 843,529,529
175 Pink3 #d787af 843,529,686
176 Plum3 #d787d7 843,529,843
177 Violet #d787ff 843,529,1000
178 Gold3 #d7af00 843,686,0
179 LightGoldenrod3 #d7af5f 843,686,372
180 Tan #d7af87 843,686,529
181 MistyRose3 #d7afaf 843,686,686
182 Thistle3 #d7afd7 843,686,843
183 Plum2 #d7afff 843,686,1000
184 Yellow3 #d7d700 843,843,0
185 Khaki3 #d7d75f 843,843,372
186 LightGoldenrod2 #d7d787 843,843,529
187 LightYellow3 #d7d7af 843,843,686
188 Grey84 #d7d7d7 843,843,843
189 LightSteelBlue1 #d7d7ff 843,843,1000
190 Yellow2 #d7ff00 843,1000,0
191 DarkOliveGreen1 #d7ff5f 843,1000,372
192 DarkOliveGreen1 #d7ff87 843,1000,529
193 DarkSeaGreen1 #d7ffaf 843,1000,686
194 Honeydew2 #d7ffd7 843,1000,843
195 LightCyan1 #d7ffff 843,1000,1000
196 Red1 #ff0000 1000,0,0
197 DeepPink2 #ff005f 1000,0,372
198 DeepPink1 #ff0087 1000,0,529
199 DeepPink1 #ff00af 1000,0,686
200 Magenta2 #ff00d7 1000,0,843
201 Magenta1 #ff00ff 1000,0,1000
202 OrangeRed1 #ff5f00 1000,372,0
203 IndianRed1 #ff5f5f 1000,372,372
204 IndianRed1 #ff5f87 1000,372,529
205 HotPink #ff5faf 1000,372,686
206 HotPink #ff5fd7 1000,372,843
207 MediumOrchid1 #ff5fff 1000,372,1000
208 DarkOrange #ff8700 1000,529,0
209 Salmon1 #ff875f 1000,529,372
210 LightCoral #ff8787 1000,529,529
211 PaleVioletRed1 #ff87af 1000,529,686
212 Orchid2 #ff87d7 1000,529,843
213 Orchid1 #ff87ff 1000,529,1000
214 Orange1 #ffaf00 1000,686,0
215 SandyBrown #ffaf5f 1000,686,372
216 LightSalmon1 #ffaf87 1000,686,529
217 LightPink1 #ffafaf 1000,686,686
218 Pink1 #ffafd7 1000,686,843
219 Plum1 #ffafff 1000,686,1000
220 Gold1 #ffd700 1000,843,0
221 LightGoldenrod2 #ffd75f 1000,843,372
222 LightGoldenrod2 #ffd787 1000,843,529
223 NavajoWhite1 #ffd7af 1000,843,686
224 MistyRose1 #ffd7d7 1000,843,843
225 Thistle1 #ffd7ff 1000,843,1000
226 Yellow1 #ffff00 1000,1000,0
227 LightGoldenrod1 #ffff5f 1000,1000,372
228 Khaki1 #ffff87 1000,1000,529
229 Wheat1 #ffffaf 1000,1000,686
230 Cornsilk1 #ffffd7 1000,1000,843
231 Grey100 #ffffff 1000,1000,1000
232 Grey3 #080808 31,31,31
233 Grey7 #121212 70,70,70
234 Grey11 #1c1c1c 109,109,109
235 Grey15 #262626 149,149,149
236 Grey19 #303030 188,188,188
237 Grey23 #3a3a3a 227,227,227
238 Grey27 #444444 266,266,266
239 Grey30 #4e4e4e 305,305,305
240 Grey35 #585858 345,345,345
241 Grey39 #626262 384,384,384
242 Grey42 #6c6c6c 423,423,423
243 Grey46 #767676 462,462,462
244 Grey50 #808080 501,501,501
245 Grey54 #8a8a8a 541,541,541
246 Grey58 #949494 580,580,580
247 Grey62 #9e9e9e 619,619,619
248 Grey66 #a8a8a8 658,658,658
249 Grey70 #b2b2b2 698,698,698
250 Grey74 #bcbcbc 737,737,737
251 Grey78 #c6c6c6 776,776,776
252 Grey82 #d0d0d0 815,815,815
253 Grey85 #dadada 854,854,854
254 Grey89 #e4e4e4 894,894,894
255 Grey93 #eeeeee 933,933,933

Apk unzipping. Not a valid zip file, but unzip worked perfect

I have problem with golang zip reader. We have apk builder which build apk and then encrypted this apk with dex protector.
When i try to open apk file, go return err "zip: not a valid zip file". But when i try to open with unzip or zipinfo, worked fine.
(Also I noticed something interesting: in php 5.5 apk file opened with ZipArchive fine, but in 5.6 return error like golang)
I try open to debug golang archive/zip pckage and found that error returned in this line https://github.com/golang/go/blob/master/src/archive/zip/reader.go#L269.
Then i try to debug this line like this:
fmt.Printf("%+v", f)
fmt.Println(
"len(f.Extra):", len(f.Extra),
"len(b):", len(b),
"tag:", tag,
"size:", size,
)
And have this results:
&{FileHeader:{
Name:META-INF/MANIFEST.MF CreatorVersion:20 ReaderVersion:20 Flags:2056 Method:8
ModifiedTime:23093 ModifiedDate:18230 CRC32:2569174240
CompressedSize:359 UncompressedSize:570 CompressedSize64:359 UncompressedSize64:570
Extra:[] ExternalAttrs:0 Comment:
} zipr:0xc208038038 zipsize:276893 headerOffset:0}
&{FileHeader:{
Name:res/drawable/ic_launcher.png CreatorVersion:10 ReaderVersion:10 Flags:2048 Method:0
ModifiedTime:22030 ModifiedDate:18230 CRC32:2310989215
CompressedSize:20418 UncompressedSize:20418 CompressedSize64:20418 UncompressedSize64:20418
Extra:[] ExternalAttrs:0 Comment:
} zipr:0xc208038038 zipsize:276893 headerOffset:2066}
...
---------------->PROBLEM HERE:-------------------=>
&{FileHeader:{
Name:AndroidManifest.xml CreatorVersion:10 ReaderVersion:10 Flags:2048 Method:0
ModifiedTime:22030 ModifiedDate:18230 CRC32:1929033187
CompressedSize:7268 UncompressedSize:7268 CompressedSize64:7268 UncompressedSize64:7268
Extra:[204 94 136 27 148 14 49 95 154 4 231 53 127 242 4 199 243 207 229 161 103 90 116 69 45 31 54 220 113 43 172 4 159 109 211 57 217 2 161 17 196 16 168 19 174 138 107 15 36 181 223 22 15 21 106 153]
ExternalAttrs:0 Comment:
} zipr:0xc208038038 zipsize:276893 headerOffset:22954}
len(f.Extra): 56
len(b): 52
tag: 24268
size: 7048
Problem in extra headers (i think this extra headers add dex protector when encrypt apk). Size more then length of extra bytes.
But unzip util and ZipArchive in php5.5 woked perfect with this extras. How to fix this ?
I want to rewrite this piece in go archive/zip package, so that the extras do not interfere golang zip reader.
It's very important for me, because i rewrite build apk from php to golang and I came across a problem that can not decide :(. Please help me.

Resources