This is a Drupal website running Ubuntu 12.04 LTS on Linode. The server is just a website. It is not a DNS server or anything else.
Here are my issues. It seems they are all related to DNS issues. (I've been troubleshooting this with Linode for a couple days and I also got help here regarding iptables. Now I can focus on the issues in this question.)
Our Mollom spam protection is not working on the Drupal site because the server cannot reach the external service. (more details below)
Drupal reports, "Your system or network configuration does not allow Drupal to access web pages, resulting in reduced functionality." (more details below - our Drupal config did not change)
the server cannot ping the Internet
ping google.com ping: unknown host google.comOther network services fail. For example:
$ sudo ntpdate ntp.ubuntu.com Exiting, name server cannot be used: Temporary failure in name resolution (-3) 1 Jun 16:42:34 ntpdate[7420]: name server cannot be used: Temporary failure in name resolution (-3) apt-get update Err http://us.archive.ubuntu.com precise Release.gpg Temporary failure resolving 'us.archive.ubuntu.com' Err http://security.ubuntu.com precise-security Release.gpg Temporary failure resolving 'security.ubuntu.com'The server is running very poorly. Could errors like the following one be related to inability to access the Internet?
Out of memory: Kill process 2300 (mysqld) score 129 or sacrifice child Killed process 2300 (mysqld) total-vm:354780kB, anon-rss:53180kB, file-rss:0kB Out of memory: Kill process 5937 (mysqld) score 60 or sacrifice child Killed process 5937 (mysqld) total-vm:344040kB, anon-rss:78988kB, file-rss:0kB
Here are the Drupal errors in more detail. Drupal configuration was not changed before these errors started (afaik).
HTTP request status Fails Your system or network configuration does not allow Drupal to access web pages, resulting in reduced functionality. This could be due to your webserver configuration or PHP settings, and should be resolved in order to download information about available updates, fetch aggregator feeds, sign in via OpenID, or use other network-dependent services. If you are certain that Drupal can access web pages but you are still seeing this message, you may add $conf['drupal_http_request_fails'] = FALSE; to the bottom of your settings.php file.
Mollom API keys Service error The Mollom API keys could not be verified. Please try again later. This issue prevents members from registering for our site or posting comments.
Here is the information I think may be relevant:
root@example1:/etc# cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 789.456.123.6
nameserver 789.456.123.7
root@example1:/etc# cat /etc/hosts
#127.0.0.1 localhost
127.0.1.1 example1.local example1
192.168.150.8 example1 example1
123.456.789.55 example1.example.com example1 www.example.com
root@example1:/etc# cat /etc/nsswitch.conf
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.
passwd: compat
group: compat
shadow: compat
hosts: files dns
networks: files
protocols: db files
services: db files
ethers: db files
rpc: db files
netgroup: nis
root@example1:/etc# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 123.456.789.1 0.0.0.0 UG 100 0 0 eth0
123.456.789.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.100.0 0.0.0.0 255.255.128.0 U 0 0 0 eth0
root@example1:/etc# cat /etc/network/interfaces
# The loopback network interface
auto lo
iface lo inet loopback
# This line ensures that the interface will be brought up during boot.
auto eth0 eth0:0
iface eth0 inet static
address 123.456.789.55
netmask 255.255.255.0
gateway 123.456.789.1
dns-nameservers 789.456.123.6 789.456.123.7
# eth0:0 - Private IPs have no gateway (they are not publicly routable) so all you need to specify is the address and netmask.
iface eth0:0 inet static
address 192.168.150.8
netmask 255.255.128.0
root@example1:/etc# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN
link/ether 76:49:bf:95:98:68 brd ff:ff:ff:ff:ff:ff
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
inet 192.168.150.8/17 brd 192.168.255.255 scope global eth0:0
valid_lft forever preferred_lft forever
inet 123.456.789.55/24 brd 123.456.789.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 1000:aaaa::aaaa:aaaa:aaaa:aaa/64 scope global dynamic
valid_lft 2591730sec preferred_lft 604530sec
inet6 fe80::aaaa:aaa:fffff:ffff/64 scope link
valid_lft forever preferred_lft forever
4: teql0: <NOARP> mtu 1500 qdisc noop state DOWN qlen 100
link/void
5: tunl0: <NOARP> mtu 1480 qdisc noop state DOWN
link/ipip 0.0.0.0 brd 0.0.0.0
6: gre0: <NOARP> mtu 1476 qdisc noop state DOWN
link/gre 0.0.0.0 brd 0.0.0.0
7: sit0: <NOARP> mtu 1480 qdisc noop state DOWN
link/sit 0.0.0.0 brd 0.0.0.0
8: ip6tnl0: <NOARP> mtu 1452 qdisc noop state DOWN
link/tunnel6 :: brd ::
9: ip6gre0: <NOARP> mtu 1448 qdisc noop state DOWN
link/[823] 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00 brd 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
root@example1:/etc# ifconfig
eth0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet addr:123.456.789.55 Bcast:123.456.789.255 Mask:255.255.255.0
inet6 addr: 1000:aaaa::aaaa:aaaa:aaaa:aaa/64 Scope:Global
inet6 addr: fe80::aaaa:aaa:fffff:ffff/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:66713 errors:0 dropped:0 overruns:0 frame:0
TX packets:54198 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:7588512 (7.5 MB) TX bytes:67678447 (67.6 MB)
Interrupt:77
eth0:0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet addr:192.168.150.8 Bcast:192.168.255.255 Mask:255.255.128.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:77
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:13579 errors:0 dropped:0 overruns:0 frame:0
TX packets:13579 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:852371 (852.3 KB) TX bytes:852371 (852.3 KB)
root@example1:/etc#
Here's more info:
# ping -n 789.456.123.7
PING 789.456.123.7 (789.456.123.7) 56(84) bytes of data.
64 bytes from 789.456.123.7: icmp_req=1 ttl=63 time=2.46 ms
64 bytes from 789.456.123.7: icmp_req=2 ttl=63 time=1.80 ms
^C
--- 789.456.123.7 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.802/2.132/2.462/0.330 ms
ping -n 173.194.37.100 (NOTE: this is google.com)
PING 173.194.37.100 (173.194.37.100) 56(84) bytes of data.
64 bytes from 173.194.37.100: icmp_req=1 ttl=55 time=14.2 ms
64 bytes from 173.194.37.100: icmp_req=2 ttl=55 time=14.1 ms
64 bytes from 173.194.37.100: icmp_req=3 ttl=55 time=14.5 ms
^C
--- 173.194.37.100 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.188/14.325/14.513/0.168 ms
And more info:
root@example1:/etc# dig google.com
; <<>> DiG 9.8.1-P1 <<>> google.com
;; global options: +cmd
;; connection timed out; no servers could be reached
root@example1:/etc# dig www.google.com @8.8.8.8
; <<>> DiG 9.8.1-P1 <<>> www.google.com @8.8.8.8
;; global options: +cmd
;; connection timed out; no servers could be reached
root@example1:/etc# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_req=1 ttl=51 time=1.05 ms
64 bytes from 8.8.8.8: icmp_req=2 ttl=51 time=1.08 ms
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.058/1.073/1.089/0.036 ms
root@example1:/etc# traceroute google.com
google.com: Temporary failure in name resolution
Cannot handle "host" cmdline arg `google.com' on position 1 (argc 1)
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Although I cannot reach the Ubuntu repos to install software the usual way, I downloaded nmap locally and then copied the deb package to the server using SCP.
Here are the results of nmap -sS -sU -p 53:
Host is up (0.00085s latency).
PORT STATE SERVICE
53/tcp open domain
53/udp open domain
/etc/resolv.confand/etc/hosts. Could you post them, please?789.456.123.7which isn't a correct IP address (just curious)?/etc/hoststo the question./etc/resolv.confwas already there. I also added more test results such as dig. (I did a search/replace on the actual nameserver IPs before posting which is why you see789.456.123.7used. I was taught to alway obfuscate IP addresses, but I can post the real nameservers if needed.)789.456.123.6and789.456.123.7which supposedly are DNS servers but doesn't behave cooperatively. You can verify this by replacing the nameservers in/etc/resolv.confby208.67.222.222and208.67.220.220which are OpenDNS' resolvers.