I'll try to be as clear as possible (sorry for any inconvenience)
At job we have an old C program which works with industrial hand terminals from Honeywell.
That terminal has its own ssh client to connect to a linux redhat 6.6 server.
Once it is connected to the linux box (using a certain user), a C program is launched by the bash shell with the following parameters
export TERM=vt200
stty raw icrnl -echo
$APLI_EXEC/program param1 param2
so the flow is like => client ssh --> ssh server-> bash --> c program
The application (or it seems) is working fine but sometimes (1-3-5 times per week) a randomly terminal stops receiving data from the server but the application receives the inputs from it. It is like if you writes Ctrl+S in a shell
Debuging the application and the ssh process using strace I realized about something strange:
The app strace is fine
write(1, "1", 7) = 1
but the strace of the ssh process is not fine (I think.. and yes I saw the ioctl no echo param, but...)
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
read(3, "\227\316\242\350\261\330)\300e\210\352\367\2VX\24\305\2474\272\371\34\273n{\323p.\211\17H\327"..., 16384) = 48
select(14, [3 9], [11], NULL, {900, 0}) = 1 (out [11], left {899, 999996})
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
write(11, "1", 1) = 1
ioctl(11, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 -opost -isig -icanon -echo ...}) = 0
select(14, [3 9], [], NULL, {900, 0} <<<<
file descriptor used by the ssh process:
lr-x------ 1 root root 64 Feb 15 17:12 9 -> pipe:[383586491]
lr-x------ 1 root root 64 Feb 15 17:12 8 -> /var/lib/sss/mc/group
lrwx------ 1 root root 64 Feb 15 17:12 7 -> socket:[383586484]
lrwx------ 1 root root 64 Feb 15 17:12 6 -> socket:[383586478]
lrwx------ 1 root root 64 Feb 15 17:12 5 -> socket:[383586458]
lrwx------ 1 root root 64 Feb 15 17:12 4 -> socket:[383586457]
lrwx------ 1 root root 64 Feb 15 17:12 3 -> socket:[383585929]
lrwx------ 1 root root 64 Feb 15 17:12 2 -> /dev/null
lrwx------ 1 root root 64 Feb 15 17:12 14 -> /dev/ptmx
lrwx------ 1 root root 64 Feb 15 17:12 13 -> /dev/ptmx
lrwx------ 1 root root 64 Feb 15 17:12 11 -> /dev/ptmx
l-wx------ 1 root root 64 Feb 15 17:12 10 -> pipe:[383586491]
lrwx------ 1 root root 64 Feb 15 17:12 1 -> /dev/null
lrwx------ 1 root root 64 Feb 15 17:12 0 -> /dev/null
In the select call, I miss fd #11 or fd #13 in there
Comparing this with another call
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
read(3, "\365\354\354C\10|\336-\4\342\327B0P\275&\213)\367\32\24\333)#\364\355V\3\237\337\33\204"..., 16384) = 52
select(14, [3 9 13], [11], NULL, {900, 0}) = 1 (out [11], left {899, 999997})
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
write(11, "a", 1) = 1
ioctl(11, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig -icanon -echo ...}) = 0
select(14, [3 9 13], [], NULL, {900, 0} <<<
What it is going on with fd #13 in the other call?
Is it possible the C program is doing something to lock the file descriptor of ssh? I don't think so because the ssh process is owned by root and the C program is running by a normal user, but who knows
It is possible the hand terminal sends a combination of ctrl keys which they 'hangs' the standard output?
I ran out of ideas.. Can anybody drive me to the right direction?
Thanks in advance
Nacho.
Related
I'm working on a project for which I'm using beaglebone black which is connected via USB (on MAC) to Mini-USB (on beaglebone side).
So for me to read or write any data on the serial port on beaglebone side - I would need to know the port number. Is there any way I can find the TTY port assigned for MINI-USB? I know it's one among /dev/tty.
Thanks!
debian#beaglebone:~$ ls -l /dev/ | grep tty
crw-rw-rw- 1 root tty 5, 2 Jun 17 16:08 ptmx
crw-rw-rw- 1 root tty 5, 0 Jun 17 13:18 tty
crw--w---- 1 root tty 4, 0 Jun 17 13:18 tty0
crw--w---- 1 root tty 4, 1 Jun 17 13:18 tty1
crw--w---- 1 root tty 4, 10 Jun 17 13:18 tty10
crw--w---- 1 root tty 4, 11 Jun 17 13:18 tty11
crw--w---- 1 root tty 4, 12 Jun 17 13:18 tty12
crw--w---- 1 root tty 4, 13 Jun 17 13:18 tty13
crw--w---- 1 root tty 4, 14 Jun 17 13:18 tty14
crw--w---- 1 root tty 4, 15 Jun 17 13:18 tty15
crw--w---- 1 root tty 4, 16 Jun 17 13:18 tty16
crw--w---- 1 root tty 4, 17 Jun 17 13:18 tty17
crw--w---- 1 root tty 4, 18 Jun 17 13:18 tty18
crw--w---- 1 root tty 4, 19 Jun 17 13:18 tty19
dmesg | grep serial
will return something like:
44e09000.serial: ttyS0 at MMIO 0x44e09000 (irq = 158, base_baud = 3000000) is a 8250
48022000.serial: ttyS1 at MMIO 0x48022000 (irq = 159, base_baud = 3000000) is a 8250
48024000.serial: ttyS2 at MMIO 0x48024000 (irq = 160, base_baud = 3000000) is a 8250
cat /proc/tty/driver/serial
will also provide useful information:
serinfo:1.0 driver revision:
0: uart:8250 mmio:0x44E09000 irq:158 tx:4380 rx:0 RTS|CTS|DTR|DSR
1: uart:8250 mmio:0x48022000 irq:159 tx:0 rx:0 CTS|DSR|CD|RI
2: uart:8250 mmio:0x48024000 irq:160 tx:0 rx:0 CTS|DSR
3: uart:unknown port:00000000 irq:0
4: uart:unknown port:00000000 irq:0
5: uart:unknown port:00000000 irq:0
in both cases you will find out all active serial ports and will need to figure out which one you want to use.
I get the below error message when I try to rsync from a local hard disk to a USB disk mounted at E: on Windows 10.
rsync: failed to set times on "/cygdrive/e/.": Invalid argument (22)
My rsync command is as below (path shortened for brevity):
rsync -rtv --delete --progress --modify-window=5 /cygdrive/d/path/to/folder/ /cygdrive/e/
I actually need to set modification times (on directories as well) and rsync actually sets modification times perfectly. It only fails to set times on root of the USB disk.
I experienced exactly the same problem.
I created a dir containing one text file and when trying to rsync it to an removable (USB) drive, I got the error. However, the file was copied to the destination. The problem is not reproducible if the destination is a folder (other than root) on the removable drive
I then repeated the process using a fixed drive as destination, and the problem was not reproducible
The 1st difference that popped up between the 2 drives, was the file system (for more details, check [MS.Docs]: File Systems Technologies):
FAT32 - on the removable drive
NTFS - on the fixed one
So this was the cause of my failure. Formatting the USB drive as NTFS fixed the problem:
The USB drive formatted as FAT32 (default):
cfati#cfati-e5550-0 /cygdrive/e/Work/Dev/StackOverflow/q045006385
$ ll /cygdrive/
total 20
dr-xr-xr-x 1 cfati None 0 Jul 14 17:58 .
drwxrwx---+ 1 cfati None 0 Jun 9 15:04 ..
d---r-x---+ 1 NT SERVICE+TrustedInstaller NT SERVICE+TrustedInstaller 0 Jul 13 22:21 c
drwxrwx---+ 1 SYSTEM SYSTEM 0 Jul 14 13:19 e
drwxr-xr-x 1 cfati None 0 Dec 31 1979 n
drwxr-xr-x 1 cfati None 0 Dec 31 1979 w
cfati#cfati-e5550-0 /cygdrive/e/Work/Dev/StackOverflow/q045006385
$ rsync -rtv --progress --modify-window=5 ./dir/ /cygdrive/w
sending incremental file list
rsync: failed to set times on "/cygdrive/w/.": Invalid argument (22)
./
a.txt
3 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/2)
sent 111 bytes received 111 bytes 444.00 bytes/sec
total size is 3 speedup is 0.01
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1196) [sender=3.1.2]
cfati#cfati-e5550-0 /cygdrive/e/Work/Dev/StackOverflow/q045006385
$ ll /cygdrive/
total 20
dr-xr-xr-x 1 cfati None 0 Jul 14 17:58 .
drwxrwx---+ 1 cfati None 0 Jun 9 15:04 ..
d---r-x---+ 1 NT SERVICE+TrustedInstaller NT SERVICE+TrustedInstaller 0 Jul 13 22:21 c
drwxrwx---+ 1 SYSTEM SYSTEM 0 Jul 14 13:19 e
drwxr-xr-x 1 cfati None 0 Dec 31 1979 n
drwxr-xr-x 1 cfati None 0 Dec 31 1979 w
After formatting the USB drive as NTFS:
cfati#cfati-e5550-0 /cygdrive/e/Work/Dev/StackOverflow/q045006385
$ ll /cygdrive/
total 24
dr-xr-xr-x 1 cfati None 0 Jul 14 17:59 .
drwxrwx---+ 1 cfati None 0 Jun 9 15:04 ..
d---r-x---+ 1 NT SERVICE+TrustedInstaller NT SERVICE+TrustedInstaller 0 Jul 13 22:21 c
drwxrwx---+ 1 SYSTEM SYSTEM 0 Jul 14 13:19 e
drwxr-xr-x 1 cfati None 0 Dec 31 1979 n
drwxrwxrwx+ 1 Administrators Administrators 0 Jul 14 17:59 w
cfati#cfati-e5550-0 /cygdrive/e/Work/Dev/StackOverflow/q045006385
$ rsync -rtv --progress --modify-window=5 ./dir/ /cygdrive/w
sending incremental file list
./
a.txt
3 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/2)
sent 111 bytes received 38 bytes 298.00 bytes/sec
total size is 3 speedup is 0.02
cfati#cfati-e5550-0 /cygdrive/e/Work/Dev/StackOverflow/q045006385
$ ll /cygdrive/
total 24
dr-xr-xr-x 1 cfati None 0 Jul 14 17:59 .
drwxrwx---+ 1 cfati None 0 Jun 9 15:04 ..
d---r-x---+ 1 NT SERVICE+TrustedInstaller NT SERVICE+TrustedInstaller 0 Jul 13 22:21 c
drwxrwx---+ 1 SYSTEM SYSTEM 0 Jul 14 13:19 e
drwxr-xr-x 1 cfati None 0 Dec 31 1979 n
drwxrwxrwx+ 1 Administrators Administrators 0 Jul 14 13:19 w
As a side note, when I was at step #2., I was an idiot and kept the --delete arg, so til I hit Ctrl + C, it deleted some data. Luckily, it didn't get to delete crucial files / folders.
I have one process (PID1) that does:
exec 3<>/dev/tcp/127.0.0.1/12713
And when I do:
$ ls -lh /proc/self/fd/
lrwx------ 1 0 0 64 Mar 24 12:19 0 -> /dev/pts/9
lrwx------ 1 0 0 64 Mar 24 12:19 1 -> /dev/pts/9
lrwx------ 1 0 0 64 Mar 24 12:19 2 -> /dev/pts/9
lrwx------ 1 0 0 64 Mar 24 12:20 255 -> /dev/pts/9
lrwx------ 1 0 0 64 Mar 24 12:19 3 -> socket:[83968639]
Now let's say I have a second process PID2, is it possible to read the socket opened through the PID1?
I have tried:
exec 1>/proc/PID1/fd/3
but i get the error message: No such device or address
My scenario has the PID1 writing to the socket and PID2 reading it. (basically for experimentation with the file descriptors)
I have two UART devices on an FPGA exposed to Linux on an Altera Cyclone V SoC. I have modified the DTS to incorporate these devices, and Linux picks them up on boot:
[ 0.879942] (NULL device *): ttyAL0 at MMIO 0xff200400 (irq = 41, base_baud = 3125000) is a Altera UART
[ 0.890050] (NULL device *): ttyAL1 at MMIO 0xff200420 (irq = 44, base_baud = 3125000) is a Altera UART
Resulting in a ttyAL0 and ttyAL1 in /dev/. The devices also appear in the relevant device subdirectory in /sys/devices/soc/ with the driver symlink present, for example:
lrwxrwxrwx 1 root root 0 Jun 20 10:36 driver -> ../../../bus/platform/drivers/altera_uart
-rw-r--r-- 1 root root 4096 Jun 20 10:36 driver_override
-r--r--r-- 1 root root 4096 Jun 20 10:36 modalias
drwxr-xr-x 2 root root 0 Jun 20 10:36 power
lrwxrwxrwx 1 root root 0 Jun 20 10:36 subsystem -> ../../../bus/platform
-rw-r--r-- 1 root root 4096 Jun 20 10:36 uevent
However if I try to open the port either programmatically, or with cat or setserial, there is a 20s stall before the RCU scheduler throws an exception:
[ 202.242133] INFO: rcu_sched detected stalls on CPUs/tasks: {} (detected by 0, t=2102 jiffies, g=124, c=123, q=254)
[ 202.252516] INFO: Stall ended before state dump start
[ 223.252109] INFO: rcu_sched self-detected stall on CPU { 0} (t=2100 jiffies g=125 c=124 q=229)
[ 223.260843] Task dump for CPU 0:
[ 223.264066] klogd R running 0 954 1 0x00000002
[ 223.270566] [<c0017984>] (unwind_backtrace) from [<c00137e0>] (show_stack+0x20/0x24)
[ 223.278319] [<c00137e0>] (show_stack) from [<c004b6cc>] (sched_show_task+0xb0/0x104)
[ 223.286045] [<c004b6cc>] (sched_show_task) from [<c004e34c>] (dump_cpu_task+0x48/0x4c)
[ 223.293941] [<c004e34c>] (dump_cpu_task) from [<c006ae60>] (rcu_dump_cpu_stacks+0xa0/0xcc)
[ 223.302188] [<c006ae60>] (rcu_dump_cpu_stacks) from [<c006e520>] (rcu_check_callbacks+0x488/0x790)
[ 223.311137] [<c006e520>] (rcu_check_callbacks) from [<c0072db0>] (update_process_times+0x50/0x70)
[ 223.319982] [<c0072db0>] (update_process_times) from [<c0083258>] (tick_sched_timer+0x78/0x27c)
[ 223.328656] [<c0083258>] (tick_sched_timer) from [<c00735f4>] (__run_hrtimer+0x90/0x1bc)
[ 223.336719] [<c00735f4>] (__run_hrtimer) from [<c0073ef4>] (hrtimer_interrupt+0x140/0x31c)
[ 223.344955] [<c0073ef4>] (hrtimer_interrupt) from [<c0016b58>] (twd_handler+0x40/0x50)
[ 223.352867] [<c0016b58>] (twd_handler) from [<c00669bc>] (handle_percpu_devid_irq+0x90/0x124)
[ 223.361364] [<c00669bc>] (handle_percpu_devid_irq) from [<c0062684>] (generic_handle_irq+0x3c/0x4c)
[ 223.370377] [<c0062684>] (generic_handle_irq) from [<c0062948>] (__handle_domain_irq+0x6c/0xb4)
[ 223.379042] [<c0062948>] (__handle_domain_irq) from [<c00086b0>] (gic_handle_irq+0x34/0x6c)
[ 223.387362] [<c00086b0>] (gic_handle_irq) from [<c0014380>] (__irq_svc+0x40/0x54)
[ 223.394811] Exception stack(0xded29cf8 to 0xded29d40)
[ 223.399842] 9ce0: 00000001 c06cb200
[ 223.407986] 9d00: 00000000 00000000 c0687b34 00000000 00000082 00000001 df418800 c06c416c
[ 223.416128] 9d20: ded28000 ded29d9c 00000000 ded29d40 c06cb200 c0029330 200f0113 ffffffff
[ 223.424285] [<c0014380>] (__irq_svc) from [<c0029330>] (__do_softirq+0xc4/0x2f0)
[ 223.431656] [<c0029330>] (__do_softirq) from [<c00297f8>] (irq_exit+0x88/0xc0)
[ 223.438851] [<c00297f8>] (irq_exit) from [<c006294c>] (__handle_domain_irq+0x70/0xb4)
[ 223.446649] [<c006294c>] (__handle_domain_irq) from [<c00086b0>] (gic_handle_irq+0x34/0x6c)
[ 223.454965] [<c00086b0>] (gic_handle_irq) from [<c0014380>] (__irq_svc+0x40/0x54)
[ 223.462412] Exception stack(0xded29e08 to 0xded29e50)
[ 223.467443] 9e00: dfbd3540 df782ac0 00000000 0000996f df59d6c0 dfbd3540
[ 223.475584] 9e20: c0695e20 00000000 df59c1c0 df59c540 ded28030 ded29e6c ded29e70 ded29e50
[ 223.483725] 9e40: c047bad0 c004756c 600f0013 ffffffff
[ 223.488762] [<c0014380>] (__irq_svc) from [<c004756c>] (finish_task_switch+0x78/0x11c)
[ 223.496661] [<c004756c>] (finish_task_switch) from [<c047bad0>] (__schedule+0x230/0x5f4)
[ 223.504726] [<c047bad0>] (__schedule) from [<c047bed4>] (schedule+0x40/0x8c)
[ 223.511746] [<c047bed4>] (schedule) from [<c0061a58>] (do_syslog+0x51c/0x5a8)
[ 223.518855] [<c0061a58>] (do_syslog) from [<c0061b00>] (SyS_syslog+0x1c/0x20)
[ 223.525968] [<c0061b00>] (SyS_syslog) from [<c000f820>] (ret_fast_syscall+0x0/0x30)
I don't know why this is happening but I have noticed two interesting (i.e. wrong) things about how Linux sees my devices. The first is that their IRQs, even though correctly reported during boot and any bind/unbind operations, are not listed in /proc/interrupts (they would appear as ff200400.serial2 and ff200420.serial3):
CPU0 CPU1
29: 47565 47091 GIC 29 twd
74: 0 0 GIC 74 0009
75: 0 0 GIC 75 000A
76: 0 0 GIC 76 000A
77: 0 0 GIC 77 0004
78: 0 0 GIC 78 0003
79: 0 0 GIC 79 0006
80: 0 0 GIC 80 0011
81: 0 0 GIC 81 0011
82: 0 0 GIC 82 0010
171: 10554 0 GIC 171 dw-mci
186: 0 0 GIC 186 dw_spi65535
190: 0 0 GIC 190 ffc04000.i2c
191: 0 0 GIC 191 ffc05000.i2c
192: 0 0 GIC 192 ffc06000.i2c
193: 0 0 GIC 193 ffc07000.i2c
194: 465 0 GIC 194 serial
199: 0 0 GIC 199 timer0
207: 0 0 GIC 207 fpga-mgr
IPI0: 0 0 CPU wakeup interrupts
IPI1: 0 0 Timer broadcast interrupts
IPI2: 591 3015 Rescheduling interrupts
IPI3: 0 0 Function call interrupts
IPI4: 1 5 Single function call interrupts
IPI5: 0 0 CPU stop interrupts
IPI6: 0 0 IRQ work interrupts
IPI7: 0 0 completion interrupts
Err: 0
The other observation is that in /sys/class/tty, the ttyAL* entries are links to virtual devices instead of the physical ones:
...
lrwxrwxrwx 1 root root 0 Jun 20 10:49 tty8 -> ../../devices/virtual/tty/tty8
lrwxrwxrwx 1 root root 0 Jun 20 10:49 tty9 -> ../../devices/virtual/tty/tty9
lrwxrwxrwx 1 root root 0 Jun 20 10:49 ttyAL0 -> ../../devices/virtual/tty/ttyAL0
lrwxrwxrwx 1 root root 0 Jun 20 10:49 ttyAL1 -> ../../devices/virtual/tty/ttyAL1
lrwxrwxrwx 1 root root 0 Jun 20 10:49 ttyS0 -> ../../devices/soc/ffc02000.serial0/tty/ttyS0
lrwxrwxrwx 1 root root 0 Jun 20 10:49 ttyS1 -> ../../devices/soc/ffc03000.serial1/tty/ttyS1
lrwxrwxrwx 1 root root 0 Jun 20 10:49 ttyp0 -> ../../devices/virtual/tty/ttyp0
lrwxrwxrwx 1 root root 0 Jun 20 10:49 ttyp1 -> ../../devices/virtual/tty/ttyp1
...
You can see the other two physical devices ttyS0 and ttyS1 ('real' UARTs on the ARM part of the SoC), I expected my devices to be in the same format. If you refer to the /sys/devices/soc/ device subdirectory listing above, you'll notice that it does not have a corresponding tty subdirectory - presumably part of the reason why I have a virtual TTY associated with the device.
So my question is: Why is my physical serial device appearing as virtual, and is that the reason I'm suffering kernel stalls?
In case I am missing vital information in the DTS, here are my UART additions:
uart2: serial2#ff200400 {
compatible = "altr,uart-1.0";
reg = <0xff200400 0x20>;
interrupts = <0 9 4>;
clock-frequency = <50000000>;
current-speed = <115200>;
};
uart3: serial3#ff200420 {
compatible = "altr,uart-1.0";
reg = <0xff200420 0x20>;
interrupts = <0 12 4>;
clock-frequency = <50000000>;
current-speed = <115200>;
};
They are child nodes of a soc node where the interrupt controller is specified.
I finally discovered the issue, and it's unsurprising judging from the RCU scheduler stack trace: My IRQs are wrong.
I don't quite understand the exact mechanics of it as I'm not a firmware engineer, but the UART modules were on a IRQ offset of 40, so the their IRQs were not 9 and 12 as I thought, but 49 and 52. Updating the DTS to match caused everything to work as expected.
I've a file like below.
ab13p29im-sss29511
0
Jan 12 22:43
ab13p29im-sss29531
0
Jan 12 22:43
ab13p29im-sss29512
0
Feb 2 16:11
ab13p29im-sss29522
0
Feb 2 16:12
ab13p29im-sss29532
0
Feb 2 16:12
ab21p30im-sss30511
0
Jan 12 22:43
ab21p30im-sss30531
0
Jan 12 22:43
ab21p30im-sss30512
0
Feb 2 16:13
ab21p30im-sss30522
3
Feb 2 16:12
i want to print this is below format.
ab13p29im-sss29511 0 Jan 12 22:43
ab21p30im-sss30522 0 Feb 2 16:12
ab21p30im-sss30531 0 Jan 12 22:43
I'm using the command paste - - - < inputfile.But if any of the value is null, the format is all messed up like below?
ab13p29im-sss29511 0 Jan 12 22:43
ab21p30im-sss30522 0 ab21p30im-sss30531
0 Jan 12 22:43 ab21p30im-sss30523.
Like if there's no date for any host or if any value is null, it breaks the 3,3,3 pattern.
You like some like this:
awk 'ORS=NR%3?" ":RS' file
ab13p29im-sss29511 0 Jan 12 22:43
ab13p29im-sss29531 0 Jan 12 22:43
ab13p29im-sss29512 0 Feb 2 16:11
ab13p29im-sss29522 0 Feb 2 16:12
ab13p29im-sss29532 0 Feb 2 16:12
ab21p30im-sss30511 0 Jan 12 22:43
ab21p30im-sss30531 0 Jan 12 22:43
ab21p30im-sss30512 0 Feb 2 16:13
ab21p30im-sss30522 3 Feb 2 16:12
sed 'N;N;s/\n/ /g' YourFile
Load 2 lines, remove new line before printing it then cycle
you could secude by putting a pattern check to initiate the cycle like /[a-b0-9]\{9\}-[a-b0-9]\{8\}/!d; before first N
By using awk command and explicit concatenation:
$ awk 'NR % 3 == 1 { lines=$0 ; next } { lines=lines" "$0 } NR % 3 == 0 { print lines ; lines="" }' file
ab13p29im-sss29511 0 Jan 12 22:43
ab13p29im-sss29531 0 Jan 12 22:43
ab13p29im-sss29512 0 Feb 2 16:11
ab13p29im-sss29522 0 Feb 2 16:12
ab13p29im-sss29532 0 Feb 2 16:12
ab21p30im-sss30511 0 Jan 12 22:43
ab21p30im-sss30531 0 Jan 12 22:43
ab21p30im-sss30512 0 Feb 2 16:13
ab21p30im-sss30522 3 Feb 2 16:12