frii.site outage from 23rd of August to 28th of August

This blog article will be covering the longest outage in frii.site's history. We'll be discussing how this happened, why this happened, and what we'll do to avoid this in the future.

All of the timestamps are in the UTC+3 timezone.

The first signs of trouble

23/08

- 21:15

I get the first report that there is something wrong with frii.site from a user that was trying to use our API. I am currently not home, and assume that there's something wrong on the users side. Why would it be down? I have not received any emails from any providers (Namecheap, Cloudflare, or Radix)

- 21:53

I get home, and start looking into the issue. Firstly, I check Vercel's logs for both the backend and frontend, nothing suspicious. Next I try to go to api.frii.site, which returns an "Ip address not found". Next I try to access the frontend, which leads to the same error.

- 21:56

I'm suspecting that someone has exploited the DNS with a wildcard, but I can't find any suspicious domains. I check the last registered domain, which was completely normal. I log into the Cloudflare dashboard, and see that all of the DNS records are there, and frii.site is marked as active.

- 22:02

I discover that DNS dig tool can't resolve any domains. I start suspecting that there's something wrong with Namecheap's NS servers I try creating a new record, which does not work.

- 22:38

I discover that my domain is no longer pointing to Cloudflare's Nameservers, even though it looks like it on the namecheap dashboard.

-22:58

I start a support chat with namecheap's support, who responded quickly

- 23:15

A namecheap support agent tells me that my domain has been suspended for phishing, and they redirect me to Radix's unsuspension form

- 23:32

I fill out my first unsuspension request, and stop working for the day.

24/08

- 10:15

I check my email for any response from Radix, and I have not received any. This is almost 11 hours after the suspension.

- 10:21

I send an email to abuse@radixregistry.com

- 11:05

I start getting the first batch of (rightfully) angry users asking what happened to their subdomain.

- 12:25

Google marks frii.site as not a phishing scam

25/08

- 9:54

I send a tweet tagging Radix saying that frii.site has been suspended for an extended period of time

26/08

- 8:05

I send Radix another unsuspension form

27/08

~10:30

I send Namecheap support a query on how long should this take, their response was basically "They're a 3rd party and we don't know".

-10:52

I get the first response from Radix's support, which shows they didn't read my submission at all. This is their response:

"Thank you for contacting RADIX. The domain name <frii.site> has been suspended due to its blacklisting by Netcraft for phishing activities. Below listed is the URL where the phishing contents were found to be hosted. To get the domain unsuspended, please clear the phishing contents and take all necessary precautions to secure the website. Once done, kindly update us and we shall proceed with the unsuspension request. Please feel free to update us, should you have any queries."

The form I submitted: "Infringing subdomain has been deleted, and the user has been banned."

-12:07

I send them an email informing that I've done the necessary steps

28/08

- 17:44

Radix unsuspends our domain.

How did this happen?

One of our subdomains had been hosting phishing content, and since our domain monitoring isn't perfect, we ultimately missed them. As soon as we found out that this had happened, we immediately suspended the users domains, along with their account.

Why did this even happen

We had not contacted Radix about our own abuse form (frii.site/report), which caused them to suspend our entire domain.

How we'll avoid this form happening again

We have talked with Radix's support team for a few days, and here are some responses which they gave us:

"we host free subdomains, and abuse is inevitable. We have an abuse form, which users can report domains to. How do we get future reports to go here":

"1. Reporting Abuse: At Radix, we prioritize the safety and integrity of online spaces. When domains are identified as abusive, and are involved in malicious activities, or resembling domains with a history of abuse, we take swift action to suspend them. We send out suspension notifications to the respective registrar's post-suspension to ensure clarity and facilitate resolution. To ensure future abuse reports are directed correctly, please use our dedicated abuse form https://abuse.radix.website/unsuspension, which will ensure your reports are processed promptly.."

"how can we trust you after you took over 4 days to respond to our query?"

"We apologize for the delay in our response. Your request was raised during our non-working days, which impacted our response time. We are actively working to improve our processes to ensure more timely communication in the future."

"If in the future we get another abuse report, will out domain get suspended again?"

"If there is an abuse report in the future against your domain, we will not be suspending the reported domain name; instead, we will send you an email regarding the same."

Even if our request was sent during their "non-working days", having to wait 4 days for a multi-million dollar company to respond to our unsuspension request is simply ridiculous. I can't recommend purchasing a domain thats TLD is owned by Radix, just because their support is so bad.

What about compensation?

To be clear: we did not receive any form of compensation from this.

If you feel like this outage has severely impacted you or your users, send an email to contact@frii.site