Sporadic Wesconnect availability

RESOLVED — February 21, 2014

[gear]

This issue is resolved. Wesconnect is working normally. We experienced typical traffic volumes this week and the site has performed as expected, with no further outages. For more information on the recent outages please see Moving Forward by iModules President Fred Weiss (Feb. 18), and A View From the Trenches by Director of Product Development Steve Williams (Feb. 21).

February 17–20, 2014

Wesconnect is working normally. We will keep a close eye on site performance and mark this issue as resolved once we are confident that the website is functioning normally under typical weekly traffic volumes. Please send any questions via the Alumni Helpdesk Support form, by email to alumnihelp@wesleyan.edu or ask us on Twitter or Facebook. We apologize for the inconvenience to our alumni, students, faculty and staff caused by these connection issues.

February 16, 2014

10:01 a.m.: We pointed DNS for wesconnect.wesleyan.edu back to iModules’ servers yesterday afternoon and Wesconnect appears to be operating consistently since that time. We will keep a close eye on availability and performance and mark this issue as resolved once we are confident that all Wesconnect traffic is successfully reaching iModules’ servers, and that the website is functioning normally under typical weekly traffic volumes.

February 15, 2014

5:21 p.m.: iModules has identified the root cause of the outages that have affected Wesconnect availability since the DDoS attack on their servers on February 3. We will soon switch DNS for wesconnect.wesleyan.edu back to iModules’ servers. Here’s part of the resolution announcement:

We have successfully identified and corrected the issue causing the outage during the last few days. While a more in-depth report on the issue will follow, to summarize, we had networking environment configurations that were sensitive to traffic volumes coming from individual source IP addresses. When we made the emergency switch to our DDoS filtering service, we essentially consolidated source IP addresses, and there were downstream configuration issues that blocked most traffic and caused the outage. We have been able to recreate the issue and confirm that our change resolves it.

iModules plans to monitor their new networking configurations closely but they are confident that this issue is now resolved. We will also monitor closely and will mark this issue as resolved once we are confident that all Wesconnect traffic is successfully reaching iModules’ servers and the website is functioning normally, under typical weekly traffic volumes.

8:45 a.m.: While Wesconnect availability has improved since Friday morning the root cause of the outages has not yet been identified. iModules continues to investigate. We are passing traffic through to the iModules webserver, but if their host does not respond after 30 seconds we are rerouting traffic to the Wesconnect Outage page. iModules will send an update on progress later today.

February 14, 2014

1:11 p.m.: After a period of consistent uptime this morning, yesterday’s intermittent Wesconnect access problems have returned. iModules continues to investigate. We are currently trying to pass traffic through to the iModules host, but if the host does not respond after 30 seconds we are rerouting traffic to the Wesconnect Outage page.

Here is some additional detail from the most recent iModules status communication:

We are proceeding down parallel paths to resolve the situation. The first path is that we have engaged additional resources and expertise to evaluate the situation and ultimately, resolve the outage in our current environment. The second path is looking at creating an additional environment for our U.S. client sites to transition to in the situation that the current issues are not solved in the near term.

We appreciate how frustrating this is for our users and apologize for the ongoing inconvenience.

10:04 a.m.: The Wesconnect outage continues. From the latest iModules status message:

After temporary site availability this morning, the outage affecting U.S.-Hosted Encompass sites continues as of 8:05 a.m. Central. We are actively investigating the cause and will provide updates as they become available.

7:33 a.m.: iModules technical staff believe they have resolved the problems leading to the outage and maintained consistent uptime on their hosts for several hours, so we are now routing traffic back to the main website. Here’s part of a recent email from iModules describing their work overnight:

Yesterday’s outage appears to be due to a particular mix of the load put onto the system between our firewalls, the public internet, and the devices that sit between them. We worked closely throughout the day yesterday and into the evening with multiple partners – our co-location facility, attack mitigation vendor, fire-wall provider, and load-balancer vendor, to review traffic and configurations among and between the systems. We ran sustained tests overnight that exceeded our typical traffic and are no longer able to recreate the circumstances that caused the outage. We are bringing in additional expertise to assess and evaluate the outage.

We will continue to monitor throughout the day and will provide any further updates here, as needed.

February 13, 2014

10:36 p.m.: iModules continues to work on the problem. No resolution as yet.

4:14 p.m.: We are now redirecting all Wesconnect traffic to an information page explaining the outage. When we are confident that service has been restored at iModules end we will re-route traffic to the usual website.

2:30 p.m.: We don’t have any additional information to report except that iModules continues to investigate. Here’s part of a recent email from iModules on the outage:

There are many potential scenarios that might be causing the current outage and we are actively working with our DDoS mitigation provider, firewall provider, and hosting facility to narrow down the problem and a resolution.

Please send any questions via the Alumni Helpdesk Support form, by email to alumnihelp@wesleyan.edu or ask us on Twitter or Facebook. We apologize for the ongoing inconvenience to our alumni, students, faculty and staff.

11:30 a.m.: Wesconnect, the alumni website, is responding sporadically. On campus we are seeing persistent “connection dropped by server” messages. Our platform vendor iModules is aware of the problem and is working to resolve it. We’ll post an update here when we know more. In the meantime, more information may be available at status.imodules.com.