def deflate(string, level)
z = Zlib::Deflate.new(level)
dst = z.deflate(string, Zlib::NO_FLUSH)
z.close
return dst
end
def inflate(string)
zstream = Zlib::Inflate.new
buf = zstream.inflate(string)
zstream.finish
zstream.close
return buf
end
a = deflate("asasasas",6)
p a
p inflate(a)
Gives me a buffer error on line
zstream.finish
Why is that? Ruby 1.8.7 I believe.
The documentation on deflate says:
Normally the parameter flush is set to Z_NO_FLUSH, which allows
deflate to decide how much data to accumulate before producing output,
in order to maximize compression.
By using Zlib::NO_FLUSH the returned value is not the entire deflated stream. Use Zlib::FINISH instead.
The functions you've written are the ones provided in the Zlib::Deflate / Zlib::Inflate documentation. You could replace all the code you've written with:
a = Zlib::Deflate.deflate("asasasas", 6)
p a
p Zlib::Inflate.inflate(a)
Related
I would like to have a formatter for the buildin string type of the nim language, but somehow I fail at providing it. Nim compilis to c, and the c representation of the string type you see here:
#if defined(__GNUC__) || defined(__clang__) || defined(_MSC_VER)
# define SEQ_DECL_SIZE /* empty is correct! */
#else
# define SEQ_DECL_SIZE 1000000
#endif
typedef char NIM_CHAR;
typedef long long int NI64;
typedef NI64 NI;
struct TGenericSeq {NI len; NI reserved; };
struct NimStringDesc {TGenericSeq Sup; NIM_CHAR data[SEQ_DECL_SIZE]; };
and here is the output of what I have tried in the lldb session:
(lldb) frame variable *longstring
(NimStringDesc) *longstring = {
Sup = (len = 9, reserved = 15)
data = {}
}
(lldb) frame variable longstring->data
(NIM_CHAR []) longstring->data = {}
(lldb) type summary add --summary-string "${&var[0]%s}" "NIM_CHAR []"
(lldb) frame variable longstring->data
(NIM_CHAR []) longstring->data = {}
(lldb) type summary add --summary-string "${var%s}" "NIM_CHAR *"
(lldb) frame variable longstring->data
(NIM_CHAR []) longstring->data = {}
(lldb) frame variable &longstring->data[0]
(NIM_CHAR *) &[0] = 0x00007ffff7f3a060 "9 - 3 - 2"
(lldb) frame variable *longstring
(lldb) type summary add --summary-string "${var.data%s}" "NimStringDesc"
(lldb) frame variable *longstring
(NimStringDesc) *longstring = NIM_CHAR [] # 0x7ffff7f3a060
(lldb) type summary add --summary-string "${&var.data[0]%s}" "NimStringDesc"
(lldb) frame variable *longstring
(NimStringDesc) *longstring = {
Sup = (len = 9, reserved = 15)
data = {}
}
(lldb)
I simply can't manage, that the output will just be data interpreted as a '\0' terminated c-string
The summary string syntax you've tried is (by design) not as syntax rich as C.
And since you're using a zero-sized array, I don't think we have any magic provision to treat that as a pointer-to string. You might want to file a bug about it, but in this case, it's arguable whether it would help you. Since your string is length-encoded it doesn't really need to be zero-terminated, and that is the only hint LLDB would understand out of the box to know when to stop reading out of a pointer-to characters.
In your case, you're going to have to resort to Python formatters
The things you need are:
the memory location of the string buffer
the length of the string buffer
a process to read memory out of
This is a very small Python snippet that does it - I'll give you enhancement suggestions as well, but let's start with the basics:
def NimStringSummary(valobj,stuff):
l = valobj.GetChildMemberWithName('Sup').GetChildMemberWithName('len').GetValueAsUnsigned(0)
s = valobj.GetChildMemberWithName('data').AddressOf()
return '"%s"'% valobj.process.ReadMemory(s.GetValueAsUnsigned(0),l,lldb.SBError())
As you can see, first of all it reads the value of the length field;
then it reads the address-of the data buffer; then it uses the process that the value comes from to read the string content, and returns it in quotes
Now, this is a proof of concept. If you used it in production, you'd quickly run into a few issues:
What if your string buffer hasn't been initialized yet, and it says the size of the buffer is 20 gigabytes? You're going to have to limit the size of the data you're willing to read. For string-like types it has builtin knowledge of (char*, std::string, Swift.String, ...) LLDB prints out the truncated buffer followed by ..., e.g.
(const char*) buffer = "myBufferIsVeryLong"...
What if the pointer to the data is invalid? You should check that s.GetValueAsUnsigned(0) isn't actually zero - if it is you might want to print an error message like "null buffer".
Also, here I just passed an SBError that I then ignore - it would be better to pass one and then check it
All in all, you'd end up with something like:
import lldb
import os
def NimStringSummary(valobj,stuff):
l = valobj.GetChildMemberWithName('Sup').GetChildMemberWithName('len').GetValueAsUnsigned(0)
if l == 0: return '""'
if l > 1024: l = 1024
s = valobj.GetChildMemberWithName('data').AddressOf()
addr = s.GetValueAsUnsigned(0)
if addr == 0: return '<null buffer>'
err = lldb.SBError()
buf = valobj.process.ReadMemory(s.GetValueAsUnsigned(0),l,err)
if err.Fail(): return '<error: %s>' % str(err)
return '"%s"' % buf
def __lldb_init_module(debugger, internal_dict):
debugger.HandleCommand("type summary add NimStringDesc -F %s.NimStringSummary" % os.path.splitext(os.path.basename(__file__))[0])
The one extra trick is the __lldb_init_module function - this function is automatically called by LLDB whenever you 'command script import' a python file. This will allow you to add the 'command script import ' to your ~/.lldbinit file and automatically get the formatter to be picked up in all debug sessions
Hope this helps!
redim = 2;
# Loading data
iris_data = readdlm("iris_data.csv");
iris_target = readdlm("iris_target.csv");
# Center data
iris_data = broadcast(-, iris_data, mean(iris_data, 1));
n_data, n_dim = size(iris_data);
Sw = zeros(n_dim, n_dim);
Sb = zeros(n_dim, n_dim);
C = cov(iris_data);
classes = unique(iris_target);
for i=1:length(classes)
index = find(x -> x==classes[i], iris_target);
d = iris_data[index,:];
classcov = cov(d);
Sw += length(index) / n_data .* classcov;
end
Sb = C - Sw;
evals, evecs = eig(Sw, Sb);
w = evecs[:,1:redim];
new_data = iris_data * w;
This code just does LDA (linear discriminant analysis) for the iris_data.
Reduct the dimensions of the iris_data to 2.
It will takes about 4 seconds, but Python(numpy/scipy) only takes about 0.6 seconds.
Why?
This is from the first page, second paragraph of the introduction in the Julia Manual:
Because Julia’s compiler is different from the interpreters used for languages like Python or R, you may find that Julia’s performance is unintuitive at first. If you find that something is slow, we highly recommend reading through the Performance Tips section before trying anything else. Once you understand how Julia works, it’s easy to write code that’s nearly as fast as C.
Excerpt:
Avoid global variables
A global variable might have its value, and therefore its type, change at any point. This makes it difficult for the compiler to optimize code using global variables. Variables should be local, or passed as arguments to functions, whenever possible.
Any code that is performance critical or being benchmarked should be inside a function.
We find that global names are frequently constants, and declaring them as such greatly improves performance
Knowing that the script (all procedural top level code) style is so pervasive among many scientific computing users, I would recommend you to at least wrap the whole file inside a let expression for starters (let introduces a new local scope), ie:
let
redim = 2
# Loading data
iris_data = readdlm("iris_data.csv")
iris_target = readdlm("iris_target.csv")
# Center data
iris_data = broadcast(-, iris_data, mean(iris_data, 1))
n_data, n_dim = size(iris_data)
Sw = zeros(n_dim, n_dim)
Sb = zeros(n_dim, n_dim)
C = cov(iris_data)
classes = unique(iris_target)
for i=1:length(classes)
index = find(x -> x==classes[i], iris_target)
d = iris_data[index,:]
classcov = cov(d)
Sw += length(index) / n_data .* classcov
end
Sb = C - Sw
evals, evecs = eig(Sw, Sb)
w = evecs[:,1:redim]
new_data = iris_data * w
end
But I would also urge you to refactor that into small functions and then compose a main function that calls the rest, something like this, notice how this refactor makes your code general and reusable (and fast):
module LinearDiscriminantAnalysis
export load_data, center_data
"Returns data and target Matrices."
load_data(data_path, target_path) = (readdlm(data_path), readdlm(target_path))
function center_data(data, target)
data = broadcast(-, data, mean(data, 1))
n_data, n_dim = size(data)
Sw = zeros(n_dim, n_dim)
Sb = zeros(n_dim, n_dim)
C = cov(data)
classes = unique(target)
for i=1:length(classes)
index = find(x -> x==classes[i], target)
d = data[index,:]
classcov = cov(d)
Sw += length(index) / n_data .* classcov
end
Sb = C - Sw
evals, evecs = eig(Sw, Sb)
redim = 2
w = evecs[:,1:redim]
return data * w
end
end
using LinearDiscriminantAnalysis
function main()
iris_data, iris_target = load_data("iris_data.csv", "iris_target.csv")
result = center_data(iris_data, iris_target)
#show result
end
main()
Notes:
You don't need all those semicolons.
anonymous functions are currently slow but that will change in v0.5. You can use FastAnonymous for now, if performance is critical.
In summary read carefully and take into account all the performance tips.
main is just a name, it could be anything else you like.
I need to access all memory of a running process in my local Windows 7-64bit. I am new to winapi.
Here is my problem; Whenever I try to Open a process and reads its memory, I get Access is Denied error.
I searched and found something. It is said that If I run the main process as Administrator and use PROCESS_ALL_ACCESS on OpenProcess, I would have enough right to do it as it is said. OK, I did it. but nothing is changed. On reading memory, I still get Access is Denied.
So, I kept searching and found another thing which is enabling SeDebugPrivilege. I have also done that but nothing is changed. I still get the error.
I've read the quest and his answer here;
Windows Vista/Win7 Privilege Problem: SeDebugPrivilege & OpenProcess .
But as I said, I am really new to winapi. I could not solve my problem yet. Is there anything which which I need to configure in my local operating system?
Here is my Python code with pywin32;
from _ctypes import byref, sizeof, Structure
from ctypes import windll, WinError, c_buffer, c_void_p, create_string_buffer
from ctypes.wintypes import *
import win32security
import win32api
import gc
import ntsecuritycon
from struct import Struct
from win32con import PROCESS_ALL_ACCESS
from struct import calcsize
MEMORY_STATES = {0x1000: "MEM_COMMIT", 0x10000: "MEM_FREE", 0x2000: "MEM_RESERVE"}
MEMORY_PROTECTIONS = {0x10: "PAGE_EXECUTE", 0x20: "PAGE_EXECUTE_READ", 0x40: "PAGEEXECUTE_READWRITE",
0x80: "PAGE_EXECUTE_WRITECOPY", 0x01: "PAGE_NOACCESS", 0x04: "PAGE_READWRITE",
0x08: "PAGE_WRITECOPY"}
MEMORY_TYPES = {0x1000000: "MEM_IMAGE", 0x40000: "MEM_MAPPED", 0x20000: "MEM_PRIVATE"}
class MEMORY_BASIC_INFORMATION(Structure):
_fields_ = [
("BaseAddress", c_void_p),
("AllocationBase", c_void_p),
("AllocationProtect", DWORD),
("RegionSize", UINT),
("State", DWORD),
("Protect", DWORD),
("Type", DWORD)
]
class SYSTEM_INFO(Structure):
_fields_ = [("wProcessorArchitecture", WORD),
("wReserved", WORD),
("dwPageSize", DWORD),
("lpMinimumApplicationAddress", DWORD),
("lpMaximumApplicationAddress", DWORD),
("dwActiveProcessorMask", DWORD),
("dwNumberOfProcessors", DWORD),
("dwProcessorType", DWORD),
("dwAllocationGranularity", DWORD),
("wProcessorLevel", WORD),
("wProcessorRevision", WORD)]
class PyMEMORY_BASIC_INFORMATION:
def __init__(self, MBI):
self.MBI = MBI
self.set_attributes()
def set_attributes(self):
self.BaseAddress = self.MBI.BaseAddress
self.AllocationBase = self.MBI.AllocationBase
self.AllocationProtect = MEMORY_PROTECTIONS.get(self.MBI.AllocationProtect, self.MBI.AllocationProtect)
self.RegionSize = self.MBI.RegionSize
self.State = MEMORY_STATES.get(self.MBI.State, self.MBI.State)
# self.Protect = self.MBI.Protect # uncomment this and comment next line if you want to do a bitwise check on Protect.
self.Protect = MEMORY_PROTECTIONS.get(self.MBI.Protect, self.MBI.Protect)
self.Type = MEMORY_TYPES.get(self.MBI.Type, self.MBI.Type)
ASSUME_ALIGNMENT = True
class TARGET:
"""Given a ctype (initialized or not) this coordinates all the information needed to read, write and compare."""
def __init__(self, ctype):
self.alignment = 1
self.ctype = ctype
# size of target data
self.size = sizeof(ctype)
self.type = ctype._type_
# get the format type needed for struct.unpack/pack.
while hasattr(self.type, "_type_"):
self.type = self.type._type_
# string_buffers and char arrays have _type_ 'c'
# but that makes it slightly slower to unpack
# so swap is for 's'.
if self.type == "c":
self.type = "s"
# calculate byte alignment. this speeds up scanning substantially
# because we can read and compare every alignment bytes
# instead of every single byte.
# although if we are scanning for a string the alignment is defaulted to 1 \
# (im not sure if this is correct).
elif ASSUME_ALIGNMENT:
# calc alignment
divider = 1
for i in xrange(4):
divider *= 2
if not self.size % divider:
self.alignment = divider
# size of target ctype.
self.type_size = calcsize(self.type)
# length of target / array length.
self.length = self.size / self.type_size
self.value = getattr(ctype, "raw", ctype.value)
# the format string used for struct.pack/unpack.
self.format = str(self.length) + self.type
# efficient packer / unpacker for our own format.
self.packer = Struct(self.format)
def get_packed(self):
"""Gets the byte representation of the ctype value for use with WriteProcessMemory."""
return self.packer.pack(self.value)
def __str__(self):
return str(self.ctype)[:10] + "..." + " <" + str(self.value)[:10] + "..." + ">"
class Memory(object):
def __init__(self, process_handle, target):
self._process_handle = process_handle
self._target = target
self.found = []
self.__scann_process()
def __scann_process(self):
"""scans a processes pages for the target value."""
si = SYSTEM_INFO()
psi = byref(si)
windll.kernel32.GetSystemInfo(psi)
base_address = si.lpMinimumApplicationAddress
max_address = si.lpMaximumApplicationAddress
page_address = base_address
while page_address < max_address:
page_address = self.__scan_page(page_address)
if len(self.found) >= 60000000:
print("[Warning] Scan ended early because too many addresses were found to hold the target data.")
break
gc.collect()
return self.found
def __scan_page(self, page_address):
"""Scans the entire page for TARGET instance and returns the next page address and found addresses."""
information = self.VirtualQueryEx(page_address)
base_address = information.BaseAddress
region_size = information.RegionSize
next_region = base_address + region_size
size = self._target.size
target_value = self._target.value
step = self._target.alignment
unpacker = self._target.packer.unpack
if information.Type != "MEM_PRIVATE" or \
region_size < size or \
information.State != "MEM_COMMIT" or \
information.Protect not in ["PAGE_EXECUTE_READ", "PAGEEXECUTE_READWRITE", "PAGE_READWRITE"]:
return next_region
page_bytes = self.ReadMemory(base_address, region_size)
for i in xrange(0, (region_size - size), step):
partial = page_bytes[i:i + size]
if unpacker(partial)[0] == target_value:
self.found.append(base_address + i)
del page_bytes # free the buffer
return next_region
def ReadMemory(self, address, size):
cbuffer = c_buffer(size)
success = windll.kernel32.ReadProcessMemory(
self._process_handle,
address,
cbuffer,
size,
0)
assert success, "ReadMemory Failed with success == %s and address == %s and size == %s.\n%s" % (
success, address, size, WinError(win32api.GetLastError()))
return cbuffer.raw
def VirtualQueryEx(self, address):
MBI = MEMORY_BASIC_INFORMATION()
MBI_pointer = byref(MBI)
size = sizeof(MBI)
success = windll.kernel32.VirtualQueryEx(
self._process_handle,
address,
MBI_pointer,
size)
assert success, "VirtualQueryEx Failed with success == %s.\n%s" % (
success, WinError(win32api.GetLastError())[1])
assert success == size, "VirtualQueryEx Failed because not all data was written."
return PyMEMORY_BASIC_INFORMATION(MBI)
def AdjustPrivilege(priv):
flags = win32security.TOKEN_ADJUST_PRIVILEGES | win32security.TOKEN_QUERY
p = win32api.GetCurrentProcess()
htoken = win32security.OpenProcessToken(p, flags)
id = win32security.LookupPrivilegeValue(None, priv)
newPrivileges = [(id, win32security.SE_PRIVILEGE_ENABLED)]
win32security.AdjustTokenPrivileges(htoken, 0, newPrivileges)
win32api.CloseHandle(htoken)
def OpenProcess(pid=win32api.GetCurrentProcessId()):
# ntsecuritycon.SE_DEBUG_NAME = "SeDebugPrivilege"
AdjustPrivilege(ntsecuritycon.SE_DEBUG_NAME)
phandle = windll.kernel32.OpenProcess( \
PROCESS_ALL_ACCESS,
0,
pid)
assert phandle, "Failed to open process!\n%s" % WinError(win32api.GetLastError())[1]
return phandle
PID = 22852
process_handle = OpenProcess(PID)
Memory(process_handle, TARGET(create_string_buffer("1456")))
Here is the error I always get;
AssertionError: ReadMemory Failed with success == 0 and address == 131072 and size == 4096.
[Error 5] Access is denied.
I do not know what information else about my code and my personal Windows 7 operating system, I should provide to you. If you need to know more, please ask it from me, I will provide it to solve that problem.
I guess, this is about a lack of configuration in my operating system , not about pywin32. I'll be waiting for your solutions.
TCP sockets are streams, not messages, so berkeley sockets send() function on some systems may send less data than required. Since Ruby Socket is very thin wrapper over berkeley sockets, AFAIK Socket#send will behave exactly like berkeley sockets send(). So what is the correct way to send a complete message via Ruby TCP sockets? In python it's a special function for that called sendall(). But in Ruby i need to manually write code like that:
while (sent = sock.send( data, 0 ) < data.length do
data = data[ sent..-1 ]
end
To expand on what danielnegri says, IO.write ends up calling io_binwrite in io.c
The relevant bit of the ruby source is below (n and len are initially set to the length of your data and offset to 0)
retry:
arg.offset = offset;
arg.length = n;
if (fptr->write_lock) {
r = rb_mutex_synchronize(fptr->write_lock, io_binwrite_string, (VALUE)&arg);
}
else {
long l = io_writable_length(fptr, n);
r = rb_write_internal(fptr->fd, ptr+offset, l);
}
if (r == n) return len;
if (0 <= r) {
offset += r;
n -= r;
errno = EAGAIN;
}
if (rb_io_wait_writable(fptr->fd)) {
rb_io_check_closed(fptr);
if (offset < RSTRING_LEN(str))
goto retry;
}
return -1L;
As you can see, until it has written all the data it will keep on doing goto retry and trying again. rb_io_wait_writable basically checks that the value of errno is such that one should try again (as opposed to something more fatal) and then calls select to avoid busy-waiting.
Recently I used 'write' method:
require "socket"
server = TCPServer.new('localhost', 20000)
loop do
Thread.start(server.accept) do |s|
print(s, " is accepted\n")
server.write(Time.now)
print(server, " is gone\n")
server.close
end
end
Try:
require 'socket'
sock = Socket.open(Socket::PF_INET,Socket::SOCK_STREAM,Socket::IPPROTO_TCP)
#data = "anyThing"
#addr = pack_sockaddr_in(port, host)
sock.connect(#addr) #make the connection
sock.send(#data, 0)
You may also want to try using the TCPSocket class. I haven't used any of this Ruby code, so I'm not used to this particular library; please let me know if I got this all wrong.
require 'socket'
sock = TCPSocket.new(host, port)
#data = "anyThing"
sock.send(#data, 0)
I have the following code that I put together for a simple Ruby TFTP server. It works fine in that it listens to port 69 and my TFTP client connects to it and I am able to write the packets to the test.txt, but instead of just writing packets, I want to be able to TFTP a file from my client to the /temp directory.
Thanks in advance for your help!
require 'socket.so'
class TFTPServer
def initialize(port)
#port = port
end
def start
#socket = UDPSocket.new
#socket.bind('', #port)
while true
packet = #socket.recvfrom(1024)
puts packet
File.open('/temp/test.txt', 'w') do |p|
p.puts packet
end
end
end
end
server = TFTPServer.new(69)
server.start
Instead of writing to the /temp/test.txt you can use ruby's Tempfile class
So in your example:
require 'socket.so'
require 'tempfile'
class TFTPServer
def initialize(port)
#port = port
end
def start
#socket = UDPSocket.new
#socket.bind('', #port)
while true
packet = #socket.recvfrom(1024)
puts packet
Tempfile.new('tftpserver') do |p|
p.puts process_packet( packet )
end
end
end
end
server = TFTPServer.new(69)
server.start
This will create a guaranteed unique temporary file in your /tmp directory with a name based off of 'tftpserver'.
EDIT: I noticed you wanted to write to /temp (not /tmp) to do this you can do Tempfile.new('tftpserver', '/temp') to specify a specific temporary directory.
Edit 2: For anyone interested there is a library that will do this https://github.com/spiceworks/net-tftp
you'll not get it so easily, the tftp protocol is relatively easy, but put/get is not stateless, or at least if the file does not fit in a single packet, that is something like 512, but some extensions allow a bigger packet
the file on the wire is splited and you'll get a sequence of packets
each packet has a sequence number so the other end can send error for a specific packet
you should take a look at wikipedia page:
http://en.wikipedia.org/wiki/Trivial_File_Transfer_Protocol
here a sample code I wrote in 2005 but id does the specular thing (it sends a file)
it's python but reasonably similar to ruby :D
def send_file(self, addr, filesend, filesize, blocksize):
print '[send_file] Sending %s (size: %d - blksize: %d) to %s:%d' % (filesend, filesize, blocksize, addr[0], addr[1])
fd = open(filesend, 'rb')
for c in range(1, (filesize / blocksize) + 2):
hdr = pack('!H', DATA) + pack('!H', c)
indata = fd.read(blocksize)
if debug > 5: print '[send_file] [%s] Sending block %d size %d' % (filesend, c, len(indata))
self.s.sendto(hdr + indata, addr)
data, addr = self.s.recvfrom(1024)
res = unpack('!H', data[:2])[0]
data = data[2:]
if res != ACK:
print '[send_file] Transfer aborted: %s' % errors[res]
break
if debug > 5:
print '[send_file] [%s] Received ack for block %d' % (filesend, unpack('>H', data[:2])[0] + 1)
fd.close()
## End Transfer
pkt = pack('!H', DATA) + pack('>H', c) + NULL
self.s.sendto(pkt, addr)
if debug: print '[send_file] File send Done (%d)' % c
you can find constants in arpa/tftp.h (you need a unix or search online)
the sequence number is a big endian (network order) short !H format for struct pack
ruby has something like python struct in String class