Server lagging

Go to http://acmectf.com for more info and file downloads!

[angry]

28 June 2011 at about 12:56AM Eastern Time

It stopped about five minutes later, but everyone on the server was lagging.

Code:

Target Name: www.acmectf.com
         IP: 208.43.95.250
  Date/Time: 6/27/2011 9:56:27 PM

 1    2 ms    1 ms    1 ms    1 ms    1 ms  [192.168.1.1]
 2   31 ms   30 ms   32 ms   31 ms   31 ms  L100.LSANCA-DSL-33.verizon-gni.net [71.103.160.1]
 3   33 ms   32 ms   34 ms   33 ms   32 ms  P11-1.LSANCA-LCR-03.verizon-gni.net [130.81.45.156]
 4   35 ms   34 ms   35 ms   34 ms   34 ms  so-6-1-2-0.LAX01-BB-RTR1.verizon-gni.net [130.81.28.225]
 5   35 ms   35 ms   35 ms   35 ms   34 ms  0.so-2-2-0.XL3.LAX15.ALTER.NET [152.63.2.165]
 6  N/A     N/A      73 ms  121 ms   73 ms  0.xe-5-0-0.XL3.DFW7.ALTER.NET [152.63.0.50]
 7  N/A      71 ms   73 ms  N/A      72 ms  TenGigE0-4-1-0.GW4.DFW13.ALTER.NET [152.63.101.54]
 8   75 ms   75 ms  102 ms   74 ms   75 ms  internapGIGE1-gw.customer.alter.net [65.208.15.230]
 9   80 ms   73 ms   73 ms   74 ms   73 ms  border1.ge1-1-bbnet1.ext1a.dal.pnap.net [216.52.191.24]
10  193 ms  193 ms  193 ms   *      193 ms  softlayerexempt-9.border3.ext1.dal.pnap.net [63.251.44.50]
11  191 ms   *      191 ms  191 ms  191 ms  ae7.bbr02.eq01.dal03.networklayer.com [173.192.18.209]
12  N/A     206 ms   *      206 ms  216 ms  ae1.bbr01.tl01.atl01.networklayer.com [173.192.18.135]
13  174 ms  111 ms  110 ms  138 ms  244 ms  po5.bbr01.eq01.wdc02.networklayer.com [173.192.18.152]
14  110 ms  110 ms  111 ms  112 ms  111 ms  po2.cer01.sr01.wdc01.networklayer.com [173.192.18.197]
15  111 ms  113 ms  111 ms  112 ms  112 ms  po1.fcr01.sr01.wdc01.networklayer.com [208.43.118.134]
16  113 ms   *       *       *       *      208.43.108.254-static.reverse.softlayer.com [208.43.108.254]
17  216 ms  216 ms   *      217 ms   *      acmegamers.com [208.43.95.250]

Ping statistics for www.acmectf.com
Packets: Sent = 5, Received = 3, Lost = 2 (40.0%)
Round Trip Times: Minimum = 216ms, Maximum = 217ms, Average = 216ms[/size]

[WORM]

Sorry for the troubles. Softlayer has been upgrading routers and revising BGP routing tables, this could be part of your problem. They're getting a lot of complaints as this is affecting all their data centers, not just us, and your problem starts in Dallas before you get to the server.

I know some others are experiencing higher pings than what they were used to, and some others are seeing lower pings, as their routing changes. Nothing can be done about that except wait and see how traffic shakes out in the long run. 

For future incidents, you can access the Softlayer Looking Glass here: http://lg.softlayer.com
Click "I Agree" to their conditions of use, choose IPv4 traceroute, select WDC01 - Washington, DC for router/location and this will run a traceroute from the data center to you. Compare it to your traceroute to the server and it will tell you more about traffic problems.

p.s. Courtney Love explains BGP here: http://www.routergod.com/?p=40 

[angry]

For future incidents, you can access the Softlayer Looking Glass here: http://lg.softlayer.com
Click "I Agree" to their conditions of use, choose IPv4 traceroute, select WDC01 - Washington, DC for router/location and this will run a traceroute from the data center to you. Compare it to your traceroute to the server and it will tell you more about traffic problems.

Just did this. It's not promising or helpful:

Code:

Type escape sequence to abort.
Tracing the route to pool-71-103-161-224.lsanca.dsl-w.verizon.net (71.103.161.224)

  1 te1-6.cer02.wdc01.networklayer.com (208.43.118.145) [MPLS: Label 12588 Exp 0] 0 msec 0 msec 0 msec
  2 po1.bbr02.eq01.wdc02.networklayer.com (173.192.18.188) 4 msec 0 msec 0 msec
  3 xe-8-2-0.edge1.Washington4.Level3.net (4.53.112.25) 0 msec 0 msec 0 msec
  4 vlan90.csw4.Washington1.Level3.net (4.69.149.254) [AS LEVEL3] 4 msec
    vlan80.csw3.Washington1.Level3.net (4.69.149.190) [AS LEVEL3] 0 msec
    vlan70.csw2.Washington1.Level3.net (4.69.149.126) [AS LEVEL3] 0 msec
  5 ae-91-91.ebr1.Washington1.Level3.net (4.69.134.141) [AS LEVEL3] 0 msec
    ae-92-92.ebr2.Washington1.Level3.net (4.69.134.157) [AS LEVEL3] 0 msec
    ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) [AS LEVEL3] 0 msec
  6 ae-5-5.ebr2.Washington12.Level3.net (4.69.143.222) [AS LEVEL3] 4 msec 0 msec 0 msec
  7  *  *  *
  8 mci-level3.washingtondc4.level3.net (4.68.62.134) [AS LEVEL3] 52 msec 36 msec
    mci-level3.washingtondc4.level3.net (4.68.62.138) [AS LEVEL3] 40 msec
  9 0.ae2.RES-BB-RTR2.verizon-gni.net (152.63.34.74) [AS UUNET] 40 msec 40 msec 36 msec
 10 P12-0.LSANCA-LCR-04.verizon-gni.net (130.81.28.230) [AS VZGNI-TRANSIT] 80 msec 80 msec 76 msec
 11 P10-0.LSANCA-DSL-33.verizon-gni.net (130.81.45.155) [AS VZGNI-TRANSIT] 84 msec 80 msec 80 msec
 12  *  *  *
 13  *  *  *
 14  *  *  *
 15  *  *  *
 16  *  *  *
 17  *  *  *
 18  *  *  *
 19  *  *  *
 20  *  *  *
 21  *  *  *
 22  *  *  *
 23  *  *  *
 24  *  *  *
 25  *  *  *
 26  *  *  *
 27  *  *  *
 28  *  *  *
 29  *  *  *
 30  *  *  *

Code:

Type escape sequence to abort.
Tracing the route to pool-71-103-161-224.lsanca.dsl-w.verizon.net (71.103.161.224)

  1 te1-6.cer02.wdc01.networklayer.com (208.43.118.145) [MPLS: Label 12588 Exp 0] 0 msec 0 msec 0 msec
  2 po1.bbr02.eq01.wdc02.networklayer.com (173.192.18.188) 0 msec 0 msec 0 msec
  3 xe-8-2-0.edge1.Washington4.Level3.net (4.53.112.25) 4 msec 0 msec 0 msec
  4 vlan60.csw1.Washington1.Level3.net (4.69.149.62) [AS LEVEL3] 0 msec
    vlan80.csw3.Washington1.Level3.net (4.69.149.190) [AS LEVEL3] 0 msec 0 msec
  5 ae-82-82.ebr2.Washington1.Level3.net (4.69.134.153) [AS LEVEL3] 4 msec
    ae-72-72.ebr2.Washington1.Level3.net (4.69.134.149) [AS LEVEL3] 4 msec 0 msec
  6 ae-5-5.ebr2.Washington12.Level3.net (4.69.143.222) [AS LEVEL3] 0 msec
    ae-8-8.ebr1.Washington12.Level3.net (4.69.143.218) [AS LEVEL3] 0 msec 4 msec
  7  *  *  *
  8 mci-level3.washingtondc4.level3.net (4.68.62.134) [AS LEVEL3] 36 msec 40 msec
    mci-level3.washingtondc4.level3.net (4.68.62.138) [AS LEVEL3] 36 msec
  9 0.ae1.RES-BB-RTR2.verizon-gni.net (152.63.32.157) [AS UUNET] 40 msec
    0.ae2.RES-BB-RTR2.verizon-gni.net (152.63.34.74) [AS UUNET] 40 msec 36 msec
 10 P12-0.LSANCA-LCR-04.verizon-gni.net (130.81.28.230) [AS VZGNI-TRANSIT] 80 msec 128 msec 76 msec
 11 P10-0.LSANCA-DSL-33.verizon-gni.net (130.81.45.155) [AS VZGNI-TRANSIT] 80 msec 84 msec 76 msec
 12  *  *  *
 13  *  *  *
 14  *  *  *
 15  *  *  *
 16  *  *  *
 17  *  *  *
 18  *  *  *
 19  *  *  *
 20  *  *  *
 21  *  *  *
 22  *  *  *
 23  *  *  *
 24  *  *  *
 25  *  *  *
 26  *  *  *
 27  *  *  *
 28  *  *  *
 29  *  *  *
 30  *  *  *

[WORM]

They are helpful...shows you getting out of the data center without any problem.

Hop 8 should be a little better than it shows, but then, Level3 doesn't have the smoothest ride at anytime. 

[angry]

I must be missing something, but I don't see my subnet on the return traces at all.

[WORM]

That's normal. Looking Glass traceroute programs send UDP datagrams to some random high-numbered port where nothing is likely to be listening. Since nothing is answering on that port, an ICMP Port Unreachable error message is returned. If, by some weird chance, there is a program on that port, it still won't reply. That's why you don't see a reply from your IP.

Windows traceroute programs uses ICMP Echo Request packets rather than UDP datagrams and, unless your router is ignoring ICMP requests, will respond. I just ran traces to your from home and from Acmectf server using ICMP, you responded to both and both looked okay.

[angry]

Ahh, I get it now. Thanks WORM.

I am a lot of things. A network engineer is not one of them.