I'd like to check the option to debug my kernel driver installed over remote physical machine (since I don't have firewire cables). Reading the relevant documentation, I haven't seen any limitation about remote physical debugging medium, so I deduced both firewire cables and ip over wireless network should work.
I thought that lldb remote connection using kdp-remote <machine-ip> would do the trick, but I don't get any response.
From remote VM however, it succeed even though the VM can be located on remote physical machine.
My boot-args configuration are keepsyms=1 debug=0x144 -v
We figured out the problem in the comments (item 2 below), but for posterity, here's a list of things to check if xnu kernel debugging isn't working:
The target machine must have a physical ethernet port which is connected via PCIe or Thunderbolt, or you must use a direct firewire connection (optionally via Thunderbolt). USB to ethernet adapters will not work on the target end. The client machine is less fussy, you can use wifi or USB-ethernet there.
The protocol uses UDP, not TCP. Have you got a firewall running on your lldb machine which might be blocking UDP packets? (You could try sending udp packets from target to client with the nc (netcat) tool while the machine is not crashed.)
Is the ARP entry correct on the client machine? arp <target ip> should yield the target interface's MAC address.
The crashed Mac will NOT respond to pings, only to KDP packets via UDP. So not getting pings back doesn't mean anything.
As far as I'm aware the machine won't request a new DHCP lease when it crashes, so that shouldn't be the problem, but you can always try setting a static IP address just to be sure.
Did you reboot after setting the boot-args? They only take effect on a fresh boot.
If SIP is active, you can only set nvram variables from the recovery environment from OS X/macOS 10.11 onwards. You can run nvram boot-args to verify that the settings stuck.
My personal recommendation is to use FireWire for kernel debugging if possible, it seems to be the fastest and most reliable in my experience.
I'm trying to determine which application or system program is using a particular port on a Windows 2008 R2 machine.
I've run
netstat -a -n -o
And have determined that PID is holding open port 445, which I'm interested in.
But when I run tasklist or Microsoft's pslist program, it tells me that the process holding the port open is simply named 'Sys'.
Is there another tool or approach I can use to find which is the real process holding it open?
PID 4 is the system process - if PID 4 is holding a port open, it means that some device driver has opened the port. Given that it's port 445, my guess is that it's the CIFS network filesystem or server. Try doing a "net stop srv" and "net stop rdr" from an elevated command prompt - that should shut down the service using the port.
The sysinternals tool procexp (process explorer) shows both processes and if the process is a service - it can show which services are running in the same process. (Windows service processes can contain a number of service threads).
Port 445 is normally the SMB port for Windows domain activities and file sharing and so on.
Have you looked at TCPView http://technet.microsoft.com/en-us/sysinternals/bb897437? It's another tool from sysinternals.
There is a free tool on Nirsoft's website called "CPORTS" with both 32 and 64 bit versions that might help you. Port 445 is used by Server 2008 R2 and later for communicating with other systens using SAMBA /TCP. I got the list below from "cyberciti.biz"
■netbios-ns - 137/tcp # NETBIOS Name Service
■netbios-dgm - 138/tcp # NETBIOS Datagram Service
■netbios-ssn - 139/tcp # NETBIOS session service
■microsoft-ds - 445/tcp # if you are using Active Directory
■Port 389 (TCP) - for LDAP (Active Directory Mode)
■Port 445 (TCP) - NetBIOS was moved to 445 after 2000 and beyond, (CIFS)
■Port 901 (TCP) - for SWAT service (not related to client communication
If you can run "grep" this is their recommended format: "$ grep -i NETBIOS /etc/services". If not, "AstroGrep" is a little more "user-friendly" Both can be gotten from SourceForge. My own interest lies in a solution to why some systems are unable to map network drives to a server 2008 R2 box yet they have no problem making a VPN connection and running the SQL software on the same server. Port 445 is the one I am most closely looking at due to NETBIOS needs but I really don't have a clue. One person can be unable to map a drive letter from their laptop on their home internet, but then use their cellphone as a tethered modem and map it with no problem. Same system same everything else.
I hope one of these helps you, as my problem is still ongoing but the Nirsoft tool is the easiest to use by far and the listed use of grep did give me other information that may be of use to you. The tool from Nirsoft provides an excellent map of all ports in use and plenty of other information. Requires no installation and small enough to keep handy on a flashdrive. Grep or Astrogrep from SourceForge.
Happy Holidays
My device, which is a Linux based IP in-liner is transparent to the network peripherals, that is, no IP address assigned to any of its interfaces.
For the sake of the conversation, let's use ADSL connection as an example, while the device is inspecting the bi-directional traffic, the network is behaving same as if device was not there, attached to the wire (see Physical setup at the attached diagram).
I wonder if I can enclosed that "device" within a Windows machine and have it operated virtually so it still seats inline between the ADSL router and the Windows netwroking interface by using virtual NICs, (or whatever their name is in windows), and inspecting the traffic, same as if it was on a separate physical device, the drawing under "Virtual Setup" in the attached diagram show what I am trying to achieve.
Reading a bit on the VirtualBox docs, seems like binding the right side is relatively simple, perhaps I should have one network adapter set as Bridge Networking and VirtualBox will connect it to the physical NIC on the host machine, and network packets are exchanged directly, circumventing the host operating system's network stack (WinXP in my case).
However, I have no idea how to achieve the left side of my diagram, which requires adding virtual NICs to windows and configure them correctly in a way to make that pipeline possible.
I would appreciate any help.
by the way, if that is not possible with VirtualBox but with other virtualization solution (e.g. VMWare), I would accept the other as well.
components setup diagram http://img715.imageshack.us/img715/810/physicalvsvirtualsetup2.png
It depends on whether you want ( or more if it can ) your linix ( *nix ) inline device to "route" the packets that go through it possibly from one network to another and whether you want this "app" to essentially have no access directly out of your virtual environment ( the green box in your diagram ). If so, what might work is something like:
your "app" is run inside a virtual machine ( say another windows xp instance ) with a NIC configured on a virtual internal or "private" network ( I'm positive both virtualbox and vmware server have this option ) but with the same IP subnet and as your actual network 192.168.1.0
the left side - eth0 - of your *nix device ( which would have to be a virtual machine ) is also set up on this internal / private network
the right side - eth1 - of your *nix device is set up on a bridged network so it has real work access into your real network
This will make it slightly easier to see if traffic / packets are infact going through your *nix device or not, as the app will not have access outside of the virtual internal/private network. NOTE when i say internal/private network I'm not talking about VPNs, just a private network that the host nor your network can reach and the virtual machines can't contact the host or your network.
Without knowing what your *nix device is doing its a little hard. if it can't route or act as a pass through device of some description then the dhcp broadcasts wont get to your dhcp server - which is what I'm guessing your doing something around given that you mentioned it in your diagram.
If your *nix device is a real device then obviously this wont work as I'm assuming its a virtual machine of some description. If this is the case let me know and I'll give some advise on how to work with that situation.
Good luck
Is there a problem using VMware on Windows to host a virtual linux box running iptables? I have a configuration that seems to work on physical hardware but is flaky under VMware.
I'm using VMware to run a virtual linux 2.6.24 machine on a Windows 2003 Server host. The linux application is essentially a NATting router that runs iptables. The rules in the nat table include:
Chain foo_pre
target prot opt in out source destination
LOG all -- * * 0.0.0.0/0 0.0.0.0/0 [options here]
LOG all -- * * 0.0.0.0/0 10.10.1.33 [options here]
DNAT all -- * * 0.0.0.0/0 10.10.1.33 tcp dpt:80 to:192.168.0.33:8080
Chain PREROUTING
target prot opt in out source destination
foo_pre all -- * * 0.0.0.0/0 0.0.0.0/0
I'm seeing the incoming packets to 10.10.1.33:80 using tcpdump, and the first LOG generates messages, but neither the DNAT or the second LOG show the packets registering on their packet counters, the second LOG generates no messages, and tcpdump doesn't show the packets to 192.168.0.33.
The eth0 adapter is on the 10.10.0.0/16 network with a default gateway of 10.10.1.1; it has a secondary address of 10.10.1.33/32. /proc/sys/net/ipv4/config/eth0/forwarding is set to 1.
Is VMware the culprit, or am I missing something? Thanks!
Update: we've simplified the test environment. No NAT rules at all, just a linux VM running under a Win2k3 Server host. Test steps:
VM is bridged to host NIC. VM and host are on the same subnet, with the same default gateway as above.
VM communicates with devices both on and off its subnet: ICMP, TCP, UDP. Communication is bidirectional: it doesn't matter which equipment initiates it.
Engineer power-cycled the default gateway while poking at the system.
VM now communicates only with devices on its subnet. Any attempt to communicate through the gateway to the same equipment from Step 2 fails to put packets on the wire. tcpdump on eth0 on the VM shows outgoing packets with no response; WireShark on the host shows nothing on the physical NIC.
Stopping and restarting the VM does not change its behavior. Stopping the VM and replacing it with a different VM with appropriate IP address, etc. does not change the behavior.
The Win2k3 host continues to communicate normally, both on and off its subnet.
I can only conclude from this that "something happens" between the VM and the host: in the VMware drivers, or in the host's network stacks. I'm off to scour the web again.... it's hard to imagine we're the first to observe this.
Updates as they come. Thanks for your thoughts and discussion.
Your second log line is trying to match packets sent to 10.10.1.33, but you changed the destination address to 192.168.0.33 on the line above it.
I'm not sure why you don't see the outgoing packets in tcpdump yet. I assume you're running tcpdump on the linux VM itself. Is the VM sending packets on the same interface it's receiving, or is there a second virtual ethernet adapter? What machines are the various IP addresses assigned to (other than 10.10.1.33).
Regarding update:
I gather you're not using DHCP (people usually don't bother when using static IP addresses). Also, it sounds like the gateway sees one NIC using two IP addresses. Normally that should be ok, but it's always the details that get you.
Is it possible the gateway will only assign one IP address to the NIC and is ignoring traffic from the VM?
After your edit, I suggest an experiment: on your physical machine, configure your NIC to disable all hardware acceleration.
I often run into problems where I can't get something to connect to something else. I usually forget to check something obvious. Can you help with:
A tip/technique for diagnosing a connection issue
The name of a tool or application that can help (and the situation in which it's useful)
I know the question is a little non-specific, but hopefully the answers can form a useful starting point for anybody who's stuck trying to get computers/programs talking to each other.
Please can you give one answer per answer so the best ones can be voted up.
Simple checks to run when debugging network problems:
Has each machine got an IP address, Go to command prompt and run ipconfig. Key things to check here are the interfaces and ensuring the appropriate ones have IP addresses.
Check both machines IP addresses are in the same range and subnet if you are running it on an internal or Virtual network.
Try pinging each machine from the other to see if they can communicate with each other. Note that some firewalls will block ping requests.
If Pinging fails then check to see if firewalls are active. If the communication is within a 'safe' internal network then try disabling the firewalls and re-pinging.
If the connections are over a wireless network then check signal strength.
If pinging fails and you are connecting through several networks then try running a tracert to see at which will may show you where on the network the connection is failing.
If you are able to ping but not connect then check firewall settings and network connection settings. Windows 2000+ has the capability of setting port an ip access on a connection properties.
Try drawing a network diagram of the connections to help in visualising the problem.
If you are connecting through routers, firewalls and loadbalancers then check that all devices are not tied to any specific ip addresses and that the IP address redirection (if in place) is correct. Also check any NAT logs to see if connections are being received and properly re-directed.
Wireshark
Latest versions of ProcMon
netstat
Wireshark www.wireshark.org
Wireshark is a network protocol analyzer for Unix and Windows.
Features:
Deep inspection of hundreds of protocols, with more being added all the time
Live capture and offline analysis
Standard three-pane packet browser
Multi-platform: Runs on Windows, Linux, OS X, Solaris, FreeBSD, NetBSD, and many others
Captured network data can be browsed via a GUI, or via the TTY-mode TShark utility
The most powerful display filters in the industry
Rich VoIP analysis
Read/write many different capture file formats: tcpdump (libpcap), Catapult DCT2000, Cisco Secure IDS iplog, Microsoft Network Monitor, Network General Sniffer® (compressed and uncompressed), Sniffer® Pro, and NetXray®, Network Instruments Observer, Novell LANalyzer, RADCOM WAN/LAN Analyzer, Shomiti/Finisar Surveyor, Tektronix K12xx, Visual Networks Visual UpTime, WildPackets EtherPeek/TokenPeek/AiroPeek, and many others
Capture files compressed with gzip can be decompressed on the fly
Live data can be read from Ethernet, IEEE 802.11, PPP/HDLC, ATM, Bluetooth, USB, Token Ring, Frame Relay, FDDI, and others (depending on your platfrom)
Decryption support for many protocols, including IPsec, ISAKMP, Kerberos, SNMPv3, SSL/TLS, WEP, and WPA/WPA2
Coloring rules can be applied to the packet list for quick, intuitive analysis
Output can be exported to XML, PostScript®, CSV, or plain text.
work the OSI model from the bottom up
Physical (Do you have a network adapter/connection)
Link layer (arp, ethernet port blocked by network team (I've seen this where locked down environments see two MAC addresses coming from one workstation port and shut down the port)
Network layer (ipconfig, tracert, ping,)
Do you have a network address (DHCP, fixed)
Are you on a proper subnet/have routing between subnets
Is something in the middle blocking you
firewalls, routing tables
When in doubt, check to see if the windows firewall is messing with your communications. 8 times out of 10, it's at fault.
Using tracert is a good start to see how far along the chain you are getting.
For virtual machines it's usally a good idea to make sure you have the loopback adapter set correctly in the Host os.
Most frequently used tool is the ping. It can be used both to test your connection and the availability of a target
Second tool is the tracert if you want to see where the packets get lost.
For more advanced debugging I use the following tools: nmap, wireshark, etc.
Windows has a netstat utility which is pretty similar to the Unix netstat and can do a number of different things that might help you solve network issues.
Random example:
netstat -r displays routing information
netstat /? for usage information
Since you said you're using 2 virtual machines I would hazard a guess that both machines are setup in a NAT configuration (rather than a unique network device) -- In the NAT configuration, neither machine would (typically) be able to ping the other.
If you're familiar with the command line, you can try the "netstat" command.
You can also try "arp -a" to list all the IP/MAC addresses known to your PC.
The "tracert [ip address]" command will show you how many gateways/routers your packets jump through on their way to their destination. (This is probably not helpful if both machines are on the same network, though.)
And don't forget to check your Windows firewall settings.
Otherwise, if you want to get down and dirty, you can try the packet sniffer known as Wireshark: http://www.wireshark.org/ (aka. Ethereal)
Pull the network cable out
If you can get some communications to a device (eg a ping), but can't get your program to talk to a service on the computer. Then, try pulling the network cable out and see if the ping stops. This will verify you're communicating with the computer you really think you are.
On windows i user PortQueryUI : http://www.microsoft.com/en-us/download/details.aspx?id=24009
DNS activity: Portable DNS Cache and Firewall;
General network activity: Wireshark, Network Monitor;
Windows utilities: ping, netstat, nslookup.
You need to be use the process of elimination, for example if you can ping the ip address but not the hostname then there's DNS issues. If you can ping the system but not connect to a share etc.
DNS out of sync
If you're using a virtual machine and you perform a roll-back on it, then it could become out of sync with the DNS (Domain name server). Try to remove and re-add the machine to the domain, or if you've got access to the DNS machine, then get it to flush its cache.