GoDaddy, observed May 12th 2011

Routes between LIFEGUARD's source in Korea and a GoDaddy router in Chicago had historically been going through Level3 in both directions.  When the source detected a problem, its traceroutes to the GoDaddy router terminated after Level3's San Jose PoP. However, Lifegaurd isolated the failure to the reverse direction. 

When the source sent a forward traceroute spoofing as a vantage point that could reach the GoDaddy router, it reached the destination via Level3's Los Angeles, Dallas, and Chicago PoPs, yielding seven hops past where the (non-spoofed) traceroute terminated. LIFEGUARD historical reverse traceroutes showed that the reverse path from the destination passed through Level3 in Chicago, Level3 in Denver, and then Level3's San Jose PoP. The source could still successfully ping the San Jose, Los Angeles, Dallas, and Denver PoPs, but the Chicago PoP was pingable only from other vantage points. Other vantage points could also ping the destination.  These results placed the "reachability horizon'' in Level3, with San Jose, Los Angeles, Denver, and Dallas on the reachable side (near the source) and Chicago on the unreachable side (near the destination).  These measurements suggested a failure affecting some paths through Level3 towards the Korean vantage point.