WDY Enterprises, LLC Disaster Recovery Mode

On Tuesday, September 2, 2025, we experienced a critical failure of multiple components at our primary data center. This has put us in a degraded operational state.

What does this mean for you?

1. Your access to email, webmail, and user preference services is currently disrupted.
2. What specific impact can we discuss at this point in time?
   1. We have been receiving email mostly continuously at our secondary site.
   2. We moved primary email to our secondary site and it became functional roughtly 12:20am on 9/3/2025.
   3. Incoming mail has slowed down, suggesting that remote sites have also caught up.
   4. Some email has been lost, but we're unable to tell exactly how much.
      • There were no usable backups from the morning 2025-09-02 backup run (reflecging changes on 2025-09-01).
      • The last usable backup has been restored, from morning of 2025-09-01.
      • Mail received at the primary site after the 2025-09-01 backups ran (it finished before 6a eastern) is lost.
      • Logging data was also lost, so we don't have any idea what mail came in during that time.
   5. There have been a number of necessary software and hardware changes.
   • Webmail: Squirrelmail is beyond support; we'll use RoundCube mail
   • Mail Transfer Agent: replaced Sendmail with Postfix
   • Mail Services: upgraded Dovecot to current version
   • Web: replaced Apache HTTPD with Nginx
   • DB: replaced MySQL with MariaDB, rushing 80% ready upgrade plan into place
   • OS: moved to modern OS (Debian) rather than archaic Gentoo Linux
   • Quotas: discontinuing Linux aquota system; will transition to soft quotas with automated cleanup.
   • Anti-Spam: old version of SpamAssassin doesn't work; will have to upgrade
3. This web page is the best place to get information about this outage.

We ask that you remain patient, and check this page for updates.

This page will be updated as the situation evolves.

The recovery priorities are as follows:

1. 100% COMPLETE- Restore operation of email transmission protocols (ESMTP)
2. 100% COMPLETE - Restore email messages from backup
3. 85% COMPLETE - restore operation of email retrieval protocols (POP3, IMAP, SUBMIT)
4. restore normal website operations
5. restore web mail access applications
6. restore user preference applications
7. implement new spam filters
8. disk quota control
9. payment engine

Approximate Timeline of events (all times US/Eastern) (decreasing order):

• 20250903-2100: human factors - 2 hours sleep in almost 40 hours... will resume tomorrow AM.
• 20250903-1155: work suspended - primary job
• 20250903-1130: POP3, IMAP, and SUBMIT protocol answer, but I need to write a custom auth module to use them.
• 20250903-0729: resolution to ESMTP encryption issues
• 20250903-0700: restore-from-backup appear done: testing underway.
• 20250903-0120: began data restoration from email data backups
  • we've made some changes to the way disk storage works; what is goign on now is temporary (no disk quotas). Quotas will be implemented
• 20250903-0052: mail delivery functionality working without errors.
• 20250903-0020: email receipt integration appears complete.
• 20250902-2018: network encryption updates in place: shouldn't give more browser warnings on open.
• 20250902-1940: initial version of this webpage is created.
• 20250902-1820: confirmed hard failure due to worst possible kind of failure, storage controller.
  • All disk images are likely corrupted at the primary site.
  • Off-site data backup images are not corrupt; backups definitely ran on 9/1, may not be complete for 9/2.
  • rebuilding will likely go faster than trying to restore services at primary data server; outage likely to exceed 24 hours.
• 20250902-1725: second hard crash of data center services; this time, we got hardware diagnostic data.
• 20250902-1640: after on-site maintenance at the data center, we (only temporarily, it appears) restored services. Left site with external confirmation of normal functionality at 16:56.
• 20250902: between 11:00 and 15:00, the primary data center services went down. Departed home office for data center at 15:20.

We apologize for any inconvenience you may experience, and are working to restore services.