Tutorial: How to debug network problems

Tutorial: How to debug network problems
Posted in: Development Hacks
0 comments

Sometimes Magento website uses connections to external resources, which are initiated from the server side. This might be caused by using fopen, curl, sockets, xmlrpc, etc. This Magento SMTP extension is a bright example of the use of these external connections. In any case unavailability or partial availability of external resources might lead to unexpected website behavior.

For example if the licensing server is overloaded or unavailable, license check in extension might end up with an error. In the best case this would lead to execution delay, in the worst case (if the network errors are not handled correctly by the extension code) this might lead to unpredictable behavior or various errors that are hard to connect with the genuine reason.

If your website suddenly stopped working, probably you need to check the case related to the unavailability of external resources.

As an Introduction

For our example we used Magento extension that tries to connect Gmail servers instead of using «mail()» function to send e-mails. Since connections to port 25/tcp are prohibited for web-server, such behavior leads to delay or error in request processing.

Though the example is taken just from the head, it suits well to illustrate the techniques recommended. The techniques described use privileged system options and consequently require superuser permissions. In case you don’t have superuser permissions, you are very unlikely to be able to use the examples described.

 Take full control over Magento SMTP setup, apply pre-defined settings for email service providers and test your emails using debug mode.

Connections logging in firewall

One of the ways to understand that your server tries to connect external resources is to log such actions in the firewall. To activate logging of all external connections one needs to add the following rule in the end of OUTPUT chain:

# iptables -A OUTPUT -m state --state NEW -j LOG

If there is a new connection, the log would have the following records in syslog (usually /var/log/messages or /var/log/syslog):

Aug  1 15:54:46 p53m kernel: [1125878.818034] IN= OUT=venet0 SRC=192.168.100.10 DST=192.168.100.1 
LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=48656 DF PROTO=UDP SPT=37348 DPT=53 LEN=40

Aug  1 15:54:46 p53m kernel: [1125878.818502] IN= OUT=venet0 SRC=192.168.100.10 DST=74.125.136.109
LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=30704 DF PROTO=TCP SPT=36355 DPT=25 WINDOW=5840 RES=0x00 SYN URGP=0

The first record in the log shows request to DNS server (address 192.168.100.1, port 53/udp), the second record shows request to SMTP server (address 74.125.136.109, port 25/tcp).

If your server has many services that establish outbound connections, one can limit the list. This can be made, if one know user’s UID, under which PHP interpreter is launched. As usual this would be Apache (UID: 99), www-data (UID: 33) or files owner (UID can be learnt by the command id -u username). In case your server has Debian or Ubuntu, the required rule will be the following:

# iptables -A OUTPUT -m owner --uid-owner 33 -m state --state NEW -j LOG

Tracing interpreter process

More information about the changes that happen during request performance can be received by tracing PHP interpreter process. For tracing utility strace is used. In case it is not installed at your server, you can install it through package manager or compile it by yourself.

$ wget 'http://sourceforge.net/projects/strace/files/strace/4.8/strace-4.8.tar.xz/download' -O 
strace-4.8.tar.xz

$ tar -xf strace-4.8.tar.xz

$ cd strace-4.8/

$ ./configure

$ make

$ ./strace -V

Now to perform tracing, please add instruction sleep (60) in the beginning of the root file index.php;

<?php

sleep(60);

This would make the interpreter to delay execution by 60 seconds, which is enough for you to find the required process and attach strace to it. In order to attach to interpreter, one needs to know PID (Process ID). One can learn it by referring to mod_status http://your-site-name/server-status) or by studying processes list. If you know PID (let it be 12345 for example), you can attach to the process.

$ strace -o /tmp/strace.log -fF -e network -s 8192 -p 12345

After the request finishes, stop strace execution by pressing CTRL+C and check log file /tmp/strace.log. In this log you will see the list of the system calls performed by the interpreter during the request execution. Parameter «–e» stands for the choice of call class (in the example only calls related with network operations are logged). If you want to see all calls, exclude the following parameter.

12345  socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 36

12345  socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 38

12345  connect(38, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.100.1")}, 16)
= 0

12345  sendto(38, "5\362\1\0\0\1\0\0\0\0\0\0\4smtp\5gmail\3com\0\0\1\0\1", 32, MSG_NOSIGNAL, NULL, 0)
= 32

12345  recvfrom(38, "5\362\201\200\0\1\0\3\0\4\0\0\4smtp\5gmail\3com\0\0\1\0\1\300\f\0\5\0\1\0\0\0\325\
0\32\16gmail-smtp-msa\1l\6google\300\27\300,\0\1\0\1\0\0\0\325\0\4J}\210l\300,\0\1\0\1\0\0\0\325\0\4J}\
210m\300=\0\2\0\1\0\1R/\0\6\3ns1\300=\300=\0\2\0\1\0\1R/\0\6\3ns3\300=\300=\0\2\0\1\0\1R/\0\6\3ns2\300=\
300=\0\2\0\1\0\1R/\0\6\3ns4\300=", 1024, 0, {sa_family=AF_INET, sin_port=htons(53),
sin_addr=inet_addr("192.168.100.1")}, [16]) = 174

12345  connect(36, {sa_family=AF_INET, sin_port=htons(25), sin_addr=inet_addr("74.125.136.108")}, 16)
= -1 ECONNREFUSED (Connection refused)

12345  shutdown(34, 1 /* send */)

In this example DNS request to server is seen and then connection attempt to the host 74.125.136.108 with port 25/tcp, which ended with an error (ECONNREFUSED).

No matter which method you use in your debugging, don’t forget to check error logs. Usually web server logs are: /var/log/httpd/error_log in RHEL/CentOS and /var/log/apache2/error_log in Debian/Ubuntu.

August 22, 2014
September 16, 2014
June 12, 2014
Comments
Leave your comment

Your email address will not be published

This blog was created with Amasty Blog Pro

This blog was created with Amasty Blog Pro

© 2009-2023 Amasty. All Rights Reserved.