Debugging a SIGSEGV Backtrace Part 2

Today I found an effortless way to debug a sisseg backtrace using gdb:

The problem was this backtrace:
[17450 XX 23:31:34 (+18)][10237] <TWS_ERROR> {sigsafe} src/common.c@1303: [bt]: (5) /usr/lib64/twsch/libtwsch.so(ch_thread+0x1fd)[0x7f64a2467669]

The procedure:

1 – First of all we need to open a instance of the process with gdb, and run only to the begin of the main function to let the process dynamic load the libraries:

$ gdb ./your_buggy_app

(gdb) b main

2 – use the function name and the offset in this way

(gdb) info line *ott_thread+0x1fd

the result:

Line 426 of “src//ch.c” starts at address 0x7fffee99a65d <ott_thread+497> and ends at 0x7fffee99a68b <ch_thread+543>.

More clear? Throw it some water

via GIPHY

 

13 Reasons Why… Vim

Hello, do you have a moment to talk about Vim?
Today we bring to you THE ultimate text editor: Vim

Before Vim our life was sad, really sad, then we found it, and our life was sad and complicated :(, but then, we learnt how to use it, and now, we are ridiculously happy!!( except for the designer, he uses Windows and can’t use Vim like us, so he is still sad)
So, why would you want to use it?? Let’ s see:

1 – If you don’t have money to buy a mouse, don’t worry, you don’t need one!.
2 – It´s doesn’t consume 130% of RAM like other editors.
3 – It´s stable, Vim just … never gonna give you up, never gonna let you down .. really, no crashing.
4 – You will find Vi ( Vim´s father) included in just about any unix derived system.
5 – Ideal when you are ssh-ing into a linux-running server or something, where don’t even need an X.
6 – Works fine even using a poor ssh or network connection.
7 – It’s customizable and extensible, you can personalize it as you see fit.
8 – It´s free!
9 – Wherever you go, you can take with you your vim configuration easily.
10 – Very nice documentation, just need to use “:help”.
11 – Vim is really powerfull and incredibly fast once you get past the initial learning curve (which is really steep, don’t give up halfway!).
12 – It has regex!
13 – Vim is awesome! (yes, that’s a reason).

 

 

Join the Vim side of the Force …

htop: uptime (!)

Realizando una revisión de rutina en los equipos de un cliente, noté algo que no me había llamado la atención hasta ahora…

Al abrir nuestro amado y user-friendly htop 

Un tipico htop con mas de 40 hilos de ejecucion, se tuvo que hacer un poco de crop en la imagen

En la imagen podemos ver que junto a la no pequeña suma de 322 dias de uptime tenemos un (!). Esto parece ser que siempre estuvo, pero nunca le preste atención, al investigar un poco sobre que es, llegue al codigo del htop donde podemos ver:

static void UptimeMeter_updateValues(Meter* this, char* buffer, int len) {
   int totalseconds = Platform_getUptime();
   if (totalseconds == -1) {
      snprintf(buffer, len, "(unknown)");
      return;
   }
   int seconds = totalseconds % 60;
   int minutes = (totalseconds/60) % 60;
   int hours = (totalseconds/3600) % 24;
   int days = (totalseconds/86400);
   this->values[0] = days;
   if (days > this->total) {
      this->total = days;
   }
   char daysbuf[15];
   if (days > 100) {
      snprintf(daysbuf, sizeof(daysbuf), "%d days(!), ", days);
   } else if (days > 1) {
      snprintf(daysbuf, sizeof(daysbuf), "%d days, ", days);
   } else if (days == 1) {
      snprintf(daysbuf, sizeof(daysbuf), "1 day, ");
   } else {
      daysbuf[0] = '\0';
   }
   snprintf(buffer, len, "%s%02d:%02d:%02d", daysbuf, hours, minutes, seconds);
}

Y ahi nos damos cuenta que solamente era el htop felicitandonos(?) por haber sobrepasado el hito de 100 dias de uptime 😛

Ahorrando tiempo de compilación con make -j

A la hora de compilar una libreria, un programa o el propio kernel de Linux, el comando make por defecto utiliza un sólo procesador lógico (thread) del sistema.
Para acelerar los tiempos podemos utilizar el parámetro -j e indicar la cantidad de procesadores lógicos que queremos utilizar.

Ejemplo:

make -j 8

¿Cómo saber que cantidad de procesadores lógicos o threads tengo disponible?

cat /proc/cpuinfo | grep processor | wc -l

En el siguiente video les mostraré como compilar el kernel con un servidor Dell con doble procesador Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz

 

Aprovechamos este post para invitarlos a participar de la edición 2017 de NAB SHOW en Las Vegas.

Se pueden registrar en el siguiente link: http://www.3way.com.ar/nab2017.php

Acceso remoto por tunel SSH inverso cuando no hay acceso

Cuantas veces han querido tener acceso SSH a un servidor dentro de una red privada de una empresa y no han logrado que el encargado en redes les haga un NAT hacia el puerto SSH (22) del servidor requerido.

A veces por políticas de seguridad o por inconvenientes en la configuración no logran redirigir algún puerto público al puerto 22 del servidor que queremos tener acceso.

Hay una solución, no temas tener que caer en las manos de un TeamViewer sin licencia.

Si Mahoma no va a la montaña, la montaña va a Mahoma.
Hay una posibilidad de hacer un SSH inverso (o túnel SSH).

¿Cómo?

Solo hay que acceder una única vez al servidor (el día de la instalación, o pedirle a alguien que ejecute una línea) y tener desde la pc que estamos intentando conectarnos el servidor de SSH corriendo, como también conocer la ip pública.
Sabiendo

esto, sólo es necesario ejecutar un comando de SSH:

ssh -N -f -R {puerto_destino}:localhost:22 {ip_publica_nuestra}

Ej: ssh -N -f -R 22022:localhost:22 200.142.168.151

Este comando se conecta por SSH a nuestra PC (pide login a nuestra pc) y deja un túnel creado asociado al localhost de nuestra pc, para que si luego nos queremos conectar a ese servidor lo podamos hacer con este simple comando:

Ej: ssh -p 22022 root@localhost

De esta manera tenemos creado un túnel constante a nuestra pc y nos podemos conectar sin necesidad de que haya que redirigir ningún puerto 22 desde el router.

* El servidor debe tener configurada correctamente la salida a Internet.
* Debe estar corriendo el servidor SSH desde la ip pública que estamos saliendo a Internet.
* Dicho script remoto podría quedar configurado al inicio con un  certificado SSH para que la conexión se restablezca sola si el servidor se reinicia.

Showing what the threads are doing, trying not to interrupt the process

Humans are curious, perhaps that makes us humans[1], and you might be curious about what your program is doing.
Sometimes, the program has several threads running and sometimes you can’t completely stop it neither kill it to see what it is doing.

A situation like that could be when there is a service running on a client and it has some problem, you suspect a thread (one of several) is locked or waiting for something that will never happen, but all other threads look like they are running fine; so you don’t want to interrupt neither kill the process for now.

Don't stop me now
Don’t stop me now

What I do, is to use gdb and write a file with the commands I want to run and ending the file with the ‘q’ command (quit), making gdb quit so the process can continue its execution. I usually write a file called ‘commands’ with this:

thread apply all bt
q

That will execute ‘bt’ (backtrace) to all threads and then ‘q’ (quit) gdb after executing backtrace. Printing the backtrace for all threads shows me (more or less) what the threads are executing.
Using the ‘commands’ file I run gdb like this:
gdb -p <pid> -x commands > /tmp/threads

Being <pid> the process pid.
Notice I redirect the output to a file, that is because unless I redirect to a file, gdb will stop the output when the console is full and wait for me to press ‘return’; which will make the process stop for a while, something I don’t want to happen.

After seeing the threads, I can write another ‘commands’ files with another instructions to gdb, like printing some variable.

[1]:
[…]the ability to ask questions is probably the central cognitive element that distinguishes human and animal cognitive abilities[…] (https://en.wikipedia.org/wiki/Great_ape_language#Limitations_of_ape_language)

Build shared libraries with invalid instructions

Some background

We have to decode many videos in multiple encodings(mpeg2video, h264, etc). The decoding proccess consumes too much CPU, and sometimes in one server we decode up to 20 Full HD videos, to make this possible we aid the CPU with either Nvidia CUDA or Intel QSV.

We distribute our software in rpm packages, which are built in our R&D department and deployed to our clients, usually this means that the hardware specifications are not the same.

These days we dealt with an interesting problem, we wanted to unify our decoding library and make it’s compilation hardware independant. CUDA didn’t pose a problem because we install the libraries in all of our systems whether they have a video card or not. However, Intel QSV support proved to be a little bit more difficult… If Intel QSV is not supported by the processor, the application raises a SIGILL (Illegal Instruction), which means that the processor tried to execute an invalid instruction, and stops the execution.

We wanted to keep one package for the library that could be built in all of our processors (whether they support Intel QSV or not). Up to now, if we wanted Intel QSV, we had to compile the library in the specific processor that supported it, and we had a bunch of conditions in our Makefiles to detect the availabity of QSV and build the library with it. Since the decoding library changes very often, new rpm packages are uploaded almost daily by us, so if the CPU doesn’t support QSV the package is built without it, and when is installed in a CPU that does support it, it doesn’t use it D:

Challenge accepted
Wait.. what is the challenge?

We want to run an application that may have invalid instructions in some of the libraries it includes, and decide during execution time whether to call them or not…

The solution

So what did we do? First of all, we took away the Intel QSV decoding part, and made a new library with it, like a wrapper. You might be wondering what does this change, because our main decoding library would link the QSV library and it should throw a SIGILL same as before.

What I didn’t mention is how do we link this new library, we are not linking it as a shared object when compiling, we dynamically load it during execution. How do we do it? Using ldfcn(), this allow us to read object files, such as a shared object, and link it during execution. By linking in this way, we can check the processor during execution and link the QSV library only if supported, however we have to manually link every function we have in the library to function pointers in the main decoding library context using dlsym(). To sum up:

  1. Divide the main decoding library and the QSV part
  2. Build the latter as a shared object
  3. Check in the main library when is QSV supported
  4. Dynamically load the .so
spongebob
The solution
A few considerations

We need the headers of the QSV library because they provide the definitions of the structures, so the rpm package will be installed always, whether we have QSV support or not, this way we can include the header and compile the main decoding library without errors.

It is important not to link QSV when compiling, if we do so the application will compile but when we try to run it, a SIGILL will show up (when not supported) before the main() is even executed. This happens because the library has a constructor that is called when the library is loaded.

If the unsupported library has too many API functions, we will have to link each and every one of them manually, this can be tedious .__. And take care not to load the library when not supported because you will have a SIGILL immediately

tag tag tag!

The problem

Lot of branches, therefore much disorder. To solve this we can tag and archive.

To keep the git repo tidy we usually recommend to avoid leaving open branches that have already been merged to the main branch.


How to tag a branch and then close correctly on local and remote repos..

1 - Create the tag (locally)
     git tag archive/<branchname> <branchname>
2 - push the new tag to remote
     git push --tags
3 - delete branch locally
     git branch -d <branchname>
4 - delete branch on remote repo
     git push origin --delete <branch_name>
5 - go to master branch (or other branch)
     git checkout master

How to recover older tagged branch

1 - go to tag
     git checkout archive/<branchname>
2 - recreate branch
     git checkout -b new_branch_name

This is all for now! matta ne!

See which rpm package a file belongs to

We make rpm packages for most of our libraries and applications, and that brings a lot of benefits. However, when you have too many packages you might (and probably will) forget which package installs a specific file.

$ rpm -qf filename

For example:

[root@videus ~]# rpm -qf /bin/bash
bash-3.0-31

Then the file /bin/bash is installed by the package bash-3.0-31

allyoubase
1989 Zero Wing‘s broken english 

This is pretty straightforward and easy to remember

Reading logs with ‘tac’

Logs are usefull for debugging and tracing what your code is doing, and what has done in the past.

I started using ‘tac’ when reading logs, specially when I need to see what happened recently.
$ tac that_log.log | less

From man tac:
tac - concatenate and print files in reverse
So with ‘tac’ you get the last line first. Which is nice if you don’t need to see lines from long ago.

print files in reverse
print files in reverse

After using tac for a while, I feel it is better to read from end to beginning, you read the error first and then what happened before. I think reading the error first makes you be more alert to read the following lines paying more attention.

If just using less to read the last line, you has to press Shift-g and wait a moment to get the last line, and sometimes with large files less hangs for a while.
Another option is to use tail -n<lines> to get the last lines right away, but usually <lines> are not enough.