August 06, 2010
This morning, our new IP address, which we announced on Wednesday, began suffering a Denial of Service attack just as the old one had. For those of you who recently switched your custom domains to the new IP, you'll find that your site is once again down (which is good news for those of us who milk cows).
We certainly knew there was a risk of this happening, but we were hoping that the attack on the old IP would be the end of it. This morning, our operations team responded by bringing up multiple new servers and reviving the old IP address, which is no longer under attack.
Getting Back Online
If you're a user of a .posterous.com subdomain, a custom domain you purchased through Posterous, or a custom domain that has not yet had its A record updated to the new IP, your site should now be up.
If, unfortunately, you were one of the people who responded promptly to Wednesday's outage (don't understand this one), you'll need to either wait for this attack to end or change your address again to get your site back online.
no offense to those effected, but i don't care about your ritzy "owned" domain problems, I've got my own posterous.com problems--little schadenfreude goes a long way.
At this point, to mitigate future issues as much as possible, we're recommending the following course of action for these users. It's slightly more complicated, but it should be somewhat more durable as well:
-
If you point your main domain to Posterous (ie: mydomain.com) and DO NOT receive email at that domain: We recommend that you point your www subdomain to posterous.com via a CNAME record, and do the same for your main domain (sometimes referred to as @) if your registrar allows the main domain to use a CNAME record. If yours does not, follow the next steps:
-
If you point your main domain to Posterous (ie: mydomain.com) and DO receive email at that domain: We recommend that you point your www subdomain to posterous.com via a CNAME record, and you point your main domain to 66.216.125.32 via an A record.
-
If you point a subdomain to Posterous (ie: blog.mydomain.com): we recommend that you point your subdomain to posterous.com via a CNAME record.
The new CNAME records should follow us as we make any changes to servers in the future, unlike an A record, which must be changed each time. We hope there won't be many changes in the future, but we thought that last week too.
Having trouble? It's understandable, since this stuff isn't always easy. If you can't get this figured out, send me an email personally, at help+jackson@posterous.com (nice guy) with the subject line "Custom Domain". (once again, if you can afford your own website, just get your butler to fix it).
If you can, include who your domain registrar (or other DNS host) is, and ideally provide a screenshot of the screen you're looking at. I'll respond back with detailed instructions tailored to your situation as quickly as I can. (yeah, sorry, no sympathy, and while you're getting your tailor to help, I wear a 40L. how bout something in a classic hounds tooth)
Bulking Up
If you're wondering about the new IP address, it's part of our plan to reinforce our services. Tonight, we'll have an outage at 10p PDT for 2-4 hours while we switch to a new host, datacenter, and significantly beefier servers. We'll also be with a host that has better capabilities for dealing with these types of attacks, and more expertise we could leverage to evade or prevent them in the future.
We're confident that the new move will mean a faster Posterous, higher reliability, and more resiliency to attacks like those we've experienced this week. Of course, this doesn't guarantee we'd survive more attacks, but it will give us a significantly better chance to weather the storm.
Identifying the Source
Many users have asked us about the cause of these attacks (i'd question them first. simple TV Detective Work, or you could play along and pretend like you're interested in their questions, but try to keep them on the line for longer than five minutes, that way you can narrow down their location. Also listen for audible clues, such as any background noise which you may think is insignificant, but which the Forensic DNS Expert will be the first to tell you can provide the greatest advantage in their case. RE: The guy who was trying to be helpful in the Twitter DNS Killer Whale Incident, was identified by his 140 word answering machine message.).
First, a quick explanation about Denial of Service (DoS) attacks. Wikipedia has more info, (be careful with the Wikipedia after a huge DoS Attack, some of the Dossers will actually EDIT the Page to their advantage and it may be a day or two before even the most rabid Wikipedians can file an Immediate Removal) but it's often hundreds or thousands of computers making as many requests as they can of a single website. Sometimes these computers are all in one place, but often they are distributed around the world, many times otherwise-innocent systems that had been infected with a virus. The people who control that virus can send instructions for the new attack when the infected systems phone home periodically. (Jesus, are you guys okay? I bet the wife doesn't use that tone of voice when her girlfriends ask what her husband does anymore...'oh, he's the CEO of a Blog'.
The motive of these attacks is often unclear (i like where you're going with it, but i've glanced ahead, and you might not want to tip your cards quite that much. NEXT TIME, try something like: DOES ANYONE HAVE A FRIEND WHO LIKE MAY HAVE SENT A MILLION EMAILS YESTERDAY...I MEAN, IN CASE YOU RECIEVED AN EMAIL WITH A CC ON IT THAT WAS REALLY LONG...?). It could have been an individual hacker/cracker (SORRY, the last person I'd suspect of having the inclination or the ability to pull this off is a Redneck, however, next time you find a COW on the Company's Roof, that might be a good guess), who wants to show off a bit ("Hey, Baby, You know that dude who was puttin down Skynard the other day on Posterous, I fuckin' shut his ass down!"). It could have been a piracy group that was upset about us removing abusive material. (Here again, this is one of the oldest stereotypes in the world...those guys are more into the Rum and Parrot Ships, or if you're talking about the African Ones, Rum and Parrot Ships) It could have been a foreign government (I was hoping I wasn't going to have to give you any Frank Sinatra advice about 'overdoing' the INTRIGUE. BELIEVE ME, YOU WOULD HAVE GOTTEN ROYALLY LAID BEFORE YOU ENDED WITH THE COLUMBO ROUTINE. Now, I'm worried, not that you won't get laid over this, but that you'll only have about 50% of the choice) wanting to silence someone using Posterous to protest (that's ridiculous. I'm not crazy about your Post Editor, but I believe you have a right to use it).
Tracking down the source of these kinds of attacks is a tricky challenge (maybe a little stronger word than 'tricky'). First, they often disguise their source addresses, and that's true in this case. We're getting hundreds of thousands of large requests every second (no marketing expert, but I'd say work the social-networking angle of that somehow. Ask that freaky Mashable guy with the absurdly square jaw who has "Mash This" tattooed on his penis what he would do) from systems that all identify themselves as 0.0.0.0 (first clue: LOW SELF-IMAGE!), which is an invalid address (ya think?). Secondly, our host quickly kills our address when the attacks start, so we have a very small sample of requests that made it through to analyze.
We've made a call to the relevant authorities to enlist their help, as forensic analysis of evidence of these attacks is a very specialized skill. ( BY all means Kiss THEIR ass in advance like that. Have you seen the SNL IT Guy Sketches? Those guys are FREAKS!)
So while we're hoping we do eventually find an answer, it could be that we'll never find out what exactly happened. For now, we'll concentrate on bulking up to stay as tough as possible against the wild west that still is the Internet. We'll do our best, we humbly apologize for our failures thus far, and we greatly thank you all for your continued support and patience.
UPDATE (4:53p PDT): At the moment, all attacks have ceased, and all Posterous sites should be up and running. While we still advise making the changes above, it's no longer necessary to do so.
Re: Update: While publishing this at (7:58p PDT): I got a 504 error and this priceless bit of DoS comedy was lost!
CONGRATULATIONS GUYS!
YOU ROCK~
LIMBS
p.s. YOU STILL HAVEN'T FIXED THE FUCKING HTML FOR BLOGGER AUTOPOST FOOTERS! LIKE THIS...
from Dogmeat
